Design a Distributed Job Scheduler (Cron)

A distributed job scheduler manages and triggers background jobs at scheduled times with high availability.

1. High-Level Design

The scheduler must guarantee that a job runs exactly once at its configured execution window, even in the event of worker node crashes.

Job Store (PostgreSQL) ---> Scheduler Node (Acquires Lease) ---> Worker Queue (Kafka)

Components

1. Job Database: Persistent store for job definitions, metadata schedules, and run state records.

2. Leader Election Node: Uses Raft / ZooKeeper to elect a leader node responsible for polling tasks.

3. Task Queue: A distributed buffer queue (Kafka / Redis Streams) routing tasks to execution nodes.

4. Execution Worker: Pulls tasks and runs the actual payload script.

2. Potential Deep Dives

• Preventing Double Execution:

When a Scheduler Node selects a job for execution, it attempts to acquire a short-term distributed lock (lease) on the job ID in Redis with a 30-second TTL. If it succeeds, the job is published to the execution queue.

• handling Failed Workers:

If a worker node crashes mid-execution, the lease on that job expires in Redis. The reconciliation manager identifies the expired state and re-queues the job.

3. References & Tech Blogs

• Netflix Tech Blog: Dyno-Queues for Distributed Task Execution
• Google Cloud: Design Patterns for Distributed Job Schedulers