File Sync Service

A file sync service keeps a folder identical across all of a user's devices: edit a file on your laptop and it appears on your phone moments later. It's a great interview problem because the naive answer ("upload the whole file on every change") falls apart immediately, and the real design is all about moving the minimum amount of data reliably. This leans heavily on Handling Large Blobs.

1. Requirements

Functional:

• Files added/edited/deleted on one device sync to all the user's other devices.
• Large files sync efficiently (don't re-upload a 1 GB file for a one-line change).
• Works across offline periods — a device catches up when it reconnects.

Non-functional:

• Efficient — minimise bytes transferred and stored.
• Reliable — never lose or corrupt a file; sync must survive interrupted uploads.
• Reasonably fast — changes propagate within seconds when online.
• Conflicts (same file edited on two devices) handled sanely, not silently lost.

2. The core idea: chunking

The key decision is to split every file into fixed-size chunks (say 4 MB) and treat the file as a list of chunk hashes. This unlocks everything:

• Sync only what changed. Edit one part of a large file and only the affected chunk's hash changes — you upload one chunk, not the whole file.
• Deduplication. Identical chunks (same hash) are stored once, even across files and users.
• Resumable uploads. An interrupted upload resumes at the next missing chunk.

file  →  [chunk1, chunk2, chunk3, ...]  →  manifest = ordered list of chunk hashes

3. The pieces

Three stores, each doing one job:

• Blob storage (S3-style object store) holds the actual chunk bytes, keyed by chunk hash.
• Metadata DB holds the file tree: for each file, its path, version, and its manifest (the ordered chunk hashes). This is the source of truth for "what the folder should look like."
• Sync service compares a device's known state to the metadata and tells it what changed.

device ─► upload changed chunks ─► blob store
       ─► update file manifest  ─► metadata DB
other devices ◄─ notified of change ◄─ metadata DB

Chunk bytes go directly to blob storage via presigned URLs, never through your servers — the app servers only move small metadata.

4. The sync loop

1. A device watches its folder; on a change it chunks the file and computes each chunk's hash.
2. It asks the metadata service which of those chunk hashes the server already has, and uploads only the missing chunks to blob storage.
3. It commits the new manifest (file version N+1) to the metadata DB.
4. Other devices, notified of the new version, download only the chunks they don't already have, then reassemble the file.

Notifying other devices is a real-time concern — a lightweight push/long-poll tells an online device "version changed, come sync"; offline devices reconcile on reconnect by asking "what changed since my last version?"

5. Conflicts

The same file edited on two devices before syncing is the hard case. Options:

• Last-write-wins — simplest, but silently discards one edit (risky for documents).
• Keep both — the pragmatic default: save the loser as file (conflicted copy from Device X) and let the user merge. No data is lost, which is the priority.

Versioning the metadata (each file has an incrementing version) lets you detect a conflict: a device committing version N+1 based on N discovers the server is already at N+1 from another device.

6. Deep dives

• Efficiency: content-defined chunking (variable boundaries) survives inserts better than fixed blocks, so inserting bytes at the start of a file doesn't shift every chunk.
• Metadata scale: the metadata DB holds many small rows per user; shard by user_id so a user's whole tree lives together (see Sharding).
• Integrity: the chunk hash is the content check — a corrupted chunk hashes wrong and is rejected.
• Compression/encryption: compress and encrypt chunks client-side before upload; the server just stores opaque blobs.

What to emphasise

Lead with chunking — it's the decision everything else follows from:

"I'd split each file into content-hashed chunks and represent the file as a manifest of chunk hashes. That means I only ever transfer the chunks that actually changed, I deduplicate identical chunks across the whole system, and uploads are resumable. Chunk bytes go straight to blob storage via presigned URLs; the metadata DB holds the file tree and manifests and is the source of truth. Other devices are pushed a 'version changed' signal and pull only the chunks they're missing. Conflicts keep both copies rather than silently dropping an edit."

Next up: Local Delivery Service — availability and ordering under real-world constraints.