Design a News Aggregator (Hacker News)

A news aggregator crawls, classifies, and ranks headlines and article links based on user engagement and chronologies.

1. High-Level Design

This architecture requires a news ingestion pipeline alongside a high-performance rank-calculation query engine.

Feeds Scraper ---> Parser (HTML) ---> Duplicate Filter (SimHash) ---> DB (PostgreSQL)
                                                                            |
                                                                    Ranking Cache (Redis)

Components

1. Scraping Engine: Connects to RSS feeds and scrapes news sites continuously.

2. De-duplication Engine: Uses SimHash algorithms to filter out near-duplicate articles.

3. Database Ledger: Stores article metadata (title, link, category, score, created_at).

4. Ranking Worker: A background service recalculating article positions based on votes and age.

2. Potential Deep Dives

• Hacker News Ranking Algorithm:

Score = (Points - 1) / (Age + 2)^gravity. Gravity controls how fast articles decay over time. Rankings are pre-computed every 30 seconds and stored in a Redis cache.

• Duplicate Detection (SimHash):

Extracts keyword tokens from articles, hashes them, and builds a 64-bit signature. If two articles share a Hamming distance of < 3 bits, they are flagged as duplicates.

3. References & Tech Blogs

• Hacker News: How the Ranking Algorithm Works
• Google Search: De-duplication Algorithms at Scale