Handling Large Blobs (Binary Large Objects)

High-scale architectures optimize binary file storage (videos, images, zip files) to prevent application server memory exhaustion.

1. High-Level Design

Never upload or download raw file binaries directly through application servers, as this consumes connection threads and exhausts CPU memory.

Client App ---> Gateway ---> Object Storage (S3 via Presigned URL)
                        |
                  Metadata DB

Ingestion Flow

1. Request Upload: Client requests an upload token.

2. Presigned URL: Application server generates a short-lived presigned URL pointing to Object Storage (AWS S3) and returns it.

3. Direct Upload: Client uploads binary data directly to S3 using the URL.

4. Notification: S3 triggers an event notification (via SQS/Lambda) to notify the metadata database of the new file.

2. Multipart Upload API

For large files (> 100 MB), use multipart upload APIs. The client splits the file into parallel segments, uploads them concurrently, and S3 reassembles the parts.

3. References & Tech Blogs

• AWS Developer Guide: S3 Multipart Upload API
• Dropbox Tech: Rabin Fingerprints and Chunking