Design Metrics Monitoring (Datadog/Prometheus)

A metrics monitoring service collects, aggregates, and alerts on time-series telemetry data from millions of application servers.

1. High-Level Design

Monitoring systems handle extreme write throughput, ingesting counters, gauges, and histograms from distributed fleets.

Agent Pull: Prometheus ---> Ingest Gateway ---> TSDB Storage
Agent Push: Datadog Agent ---> Ingest Gateway ---> Kafka ---> TSDB Storage

Components

1. Metrics Collection Agent: Runs on client servers and collects metrics (CPU, Memory, API latency).

2. Ingest Gateway: Authenticates incoming payloads and passes metrics logs to Kafka.

3. Time-Series Database (TSDB): Optimizes write operations and performs historical range queries.

4. Alerting Engine: Evaluates rule expressions (e.g., avg(cpu) > 90%) and dispatches notifications.

2. Potential Deep Dives

• Pull vs Push Collection Models:
• Pull Model (Prometheus): Server scrapes target endpoints. Simpler configuration and avoids overload.
• Push Model (Datadog): Agent pushes metrics to gateway. Better suited for dynamic autoscale fleets.
• TSDB Storage Optimization:

Metrics are grouped in 2-hour chunks, compressed using delta-of-delta pace algorithms, and written to disk as immutable blocks.

3. References & Tech Blogs

• Prometheus Architecture: Time-Series Database Internals
• Datadog Engineering: Ingesting Trillions of Points per Day