Design a Price Tracking Service

A price tracking service monitors product pricing changes across e-commerce platforms and alerts users during price drops.

1. High-Level Design

The challenges here include distributed web crawling and high-volume alert dispatching.

Scrapers (Proxy Pool) ---> Kafka Ingest ---> Parsing Engine ---> Time Series DB
                                                     |
                                               Alert dispatcher ---> Amazon SES

Components

1. Distributed Scraping Fleet: Crawlers scraping e-commerce sites. Uses proxy pools and cookie rotation to avoid bot blocks.

2. Scraper Ingestion (Kafka): Buffers scraped HTML/JSON payloads.

3. Price Parsing Engine: Extracts prices and logs historical data points to a Time-Series Database (TimescaleDB / InfluxDB).

4. Notification Engine: Triggers alert dispatches (emails, SMS) when prices drop below user-configured targets.

2. Potential Deep Dives

• Anti-Scraping Defenses:

Rotate IP addresses using premium proxy lists. Render dynamic Javascript payloads using headless browser clusters (Playwright / Puppeteer) running in serverless nodes.

• Historical Data Compression:

Use delta-of-delta compression (similar to Gorilla compression) to pack pricing time-series metrics.

3. References & Tech Blogs

• TimescaleDB: How to Store and Query Time-Series Data at Scale
• Google Cloud: Building a Scalable Distributed Scraping Architecture