HAMZA KAREEM / CONCEPTS hamzawi.dev

Lesson 802 min read

Presigned URLs (Direct-to-Storage Uploads)

Cloud infrastructurecloud infrastructure / storage

Symptom 1: the API server is the bottleneck

Your flow: browser → POST /upload (your API) → s3.put_object() → S3.

Fine for 2 MB avatars. Then users start uploading 500 MB videos:

  • pods OOMKilled — the framework buffers the body in RAM
  • p99 on unrelated endpoints jumps — 20 slow uploads pin every worker
  • transfer billed twice: in to your compute, out to S3

Reason it through

The bytes never needed to touch your server. Its only real job was authorization: "may user 42 write videos/42/clip.mp4?" You're spending RAM, threads and bandwidth to babysit a byte pipe.

The fix: sign, don't proxy

A presigned URL is an ordinary S3 URL with a signature of your server's credentials in the query string. Whoever holds it can perform exactly one operation (PUT that key), for a limited time, without ever having your keys.

url = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": "uploads",
            "Key": f"user/{uid}/{uuid4()}.mp4",
            "ContentType": "video/mp4"},
    ExpiresIn=900)   # 15 minutes

// browser
await fetch(url, {method: "PUT",
                  headers: {"Content-Type": "video/mp4"},
                  body: file})

Server RAM per upload: ~0. Worker threads held: 0.

Symptom 2: "works in curl, 403 in the browser"

SignatureDoesNotMatch. Causes, most common first:

1️⃣ Header mismatch. You signed Content-Type: video/mp4, the browser sent video/mp4; charset=utf-8. Every header in Params becomes part of the signature — send it back byte-identical, or don't sign it at all.

2️⃣ CORS. The PUT is cross-origin, so the bucket must allow PUT from your origin (and expose ETag for multipart). A missing rule often surfaces as a vague network error, not a clean 403.

3️⃣ Expiry / clock skew. The signer's clock drifting 10 min makes URLs dead on arrival.

4️⃣ The URL got mangled — re-encoded, truncated in a log, or passed through a redirect.

What practitioners actually get wrong

The URL is the authorization. Once issued you cannot revoke it, and nothing inspects the file afterwards.

  • Sign a key YOU generate. Never presign a client-supplied path, or I'll PUT to user/1/avatar.png.
  • A presigned PUT cannot cap file size. If that matters, use a POST policy with content-length-range.
  • Treat the object as untrusted until a post-upload step (S3 event → Lambda) checks real type/size and flips your DB row to ready. "Upload finished" ≠ "file is safe".
  • It's a bearer credential: don't log it, don't cache it in a CDN, keep ExpiresIn in minutes.
  • Same trick works for downloads — presigned GET beats streaming private files through your API.

Your exercise

Presign a PUT with ExpiresIn=60 and upload with:

curl -X PUT --upload-file ./test.mp4 "<url>"

Then (a) wait 90s and retry the same URL, and (b) inside the window, retry with Content-Type changed by one character.

Two different failures. Before you run them — can you predict which returns ExpiredToken vs SignatureDoesNotMatch, and which error body names the header that broke it?