06
How would you design a telemetry system to monitor GPU health across a large cluster?
Tap to write answer
0 words | 0 charsPress Enter ↵ to reveal
Your Attempt
0 wordsRefined Model Answer
ReferenceBefore designing it, I would clarify what metrics matter most: temperature, utilization, memory pressure, power draw, fan speed, error codes, and alerting thresholds. At a high level, I would have agents on each node collect metrics, send them to an ingestion service, store them in a time-series backend, and feed them into dashboards and alerts. I would keep the write path lightweight because telemetry has to be cheap and reliable at scale. The trade-off is between very high data frequency and storage cost, so I would probably aggregate some metrics locally before shipping them. I would also discuss retry handling, backpressure, and how to make sure I do not lose data during node failures.