HAMZA KAREEM / CONCEPTS hamzawi.dev

DAILY CONCEPTS

One concept
a day.

Short, practical lessons on the ideas behind systems, web, AI and infrastructure: the myth, what actually happens, the traps, and an exercise.

111 lessonssince 24 Jun 20266 topics

/ Latest

Lesson 111System design Last-Write-Wins & Clock Skew Your "Latest" Timestamp Is Lying updated_at doesn't measure when something happened. It measures what some machine's clock said when the code ran. Those are different numbers, and on a fleet of nodes they… Read the lesson

/ Archive

October 2026

  1. Last-Write-Wins & Clock SkewYour "Latest" Timestamp Is Lying System design
  2. Semantic Caching for LLM AppsBuild It In 12 Lines, Then See Where It Bites AI engineering
  3. Idle Timeout Mismatch (LB Keep-Alive vs Backend)The Silent Hang-Up Cloud infrastructure
  4. Property-Based Testing (Shrinking & Oracles)97% Coverage, One Missing Example Developer tooling
  5. srcset vs picture (Responsive Images)Who Actually Picks The Image Web development
  6. UUID v4 vs v7 Primary KeysThe Cost Isn't The 16 Bytes Databases
  7. LoRA (Low-Rank Adaptation)Fine-Tuning a Model in 6 Lines AI engineering
  8. Event Time vs Processing Time (Watermarks & Late Data)The Bug That Only Shows Up On The Hour System design
  9. PodDisruptionBudgets — Voluntary vs Involuntary DisruptionThe Permission Slip Nobody Has To Ask For Cloud infrastructure
  10. Shallow Clone vs Partial CloneTwo Ways to Not Download Your Repo Developer tooling
  11. INP & Long Tasks — Yielding to the Main ThreadYour Handler Was Fast, the Main Thread Wasn't Web development
  12. The Lock Queue (ALTER TABLE & Lock Waits)Why a "Safe" Migration Froze Your Whole Table Databases
  13. Prompt Caching — The Prefix Match You Keep BreakingThe Prefix Match You Keep Breaking AI engineering
  14. Shuffle Sharding — Blast-Radius MathBlast-Radius Math System design
  15. SNAT Port Exhaustion (NAT Gateway Outbound Limits)Why Outbound Calls Die At Scale Cloud infrastructure
  16. Build Cache Invalidation — Timestamps vs Content HashesTimestamps vs Content Hashes Developer tooling

September 2026

  1. Font Loading Strategy (font-display & Metric Overrides)font-display & Metric Overrides Web development
  2. Index-Only Scans & the Visibility MapDebug-Along Databases
  3. Hybrid Search & Reciprocal Rank Fusion (RRF)Build It in 5 Lines AI engineering
  4. Wire Compatibility & Schema Evolution (Protobuf field numbers)A Schema Change Is a Rollout, Not an Edit System design
  5. Spot Instances — Interruptions Are About Capacity, Not PriceInterruptions Are About Capacity, Not Price Cloud infrastructure
  6. Stdio Buffering — Line vs Block (Pipes Change Your Program)Why Your Logs Disappear The Moment You Pipe Them Developer tooling
  7. The Back/Forward Cache (bfcache)Your Page Isn't Reloading, It's Waking Up Web development
  8. Prepared Statements — Custom Plan vs Generic PlanCustom Plan vs Generic Plan Databases
  9. The Agent Loop — Tool Use Is a Contract, Not a Function CallTool Use Is a Contract, Not a Function Call AI engineering
  10. Cross-AZ Traffic — The Bill Line Nobody Can ExplainThe Bill Line Nobody Can Explain Cloud infrastructure
  11. Leases & Fencing Tokens — Why Your Distributed Lock Isn't SafeWhy Your Distributed Lock Isn't Safe System design
  12. Merge Queues & Semantic ConflictsPR #412 renames a helper: getUser(id) → fetchUser(id). CI green ✅ Developer tooling
  13. Service Workers — Caching & the Update LifecycleBuild One in 14 Lines, Then Meet the Trap Web development
  14. Reading EXPLAIN ANALYZE — Estimated vs Actual RowsEstimated vs Actual Rows Databases
  15. ANN Vector Indexes (HNSW) — Recall TuningRecall Is a Dial, Not a Guarantee AI engineering
  16. Presigned URLs (Direct-to-Storage Uploads)Your flow: browser → POST /upload (your API) → s3.put_object() → S3. Cloud infrastructure
  17. Fan-out on Write vs Fan-out on ReadYour social app has 400k users. Home feed loads in 40ms. Then a footballer with 2.1M followers signs up, posts once — and every write starts timing out. Not reads. Writes.… System design
  18. Mutation Testing — Coverage vs Mutation ScoreCoverage vs Mutation Score Developer tooling
  19. Resource Hints (preconnect / preload / prefetch)Preload does not create bandwidth. It only reorders the queue. Web development
  20. SELECT FOR UPDATE SKIP LOCKED (Postgres Job Queue)a real job queue in 5 lines of SQL Databases
  21. Continuous Batchingwhy your LLM server sits at 20% GPU and still feels slow AI engineering
  22. Container Memory Limits & OOMKilledA CPU limit is a speed bump. A memory limit is a cliff. Cloud infrastructure
  23. Refresh Token Rotation & Reuse DetectionA laptop gets stolen in March. The user changes their password that night. In July, your logs show that same person's account pulling data from an IP in another country — with… Web development
  24. Quorum Reads & Writes (R + W > N)R + W > N buys you exactly ONE thing: overlap. System design
  25. strace (Syscall Tracing)debugging a process that refuses to explain itself Developer tooling
  26. Tokenization (BPE)the unit your LLM actually charges, truncates and reasons in AI engineering
  27. Partitioning vs Shardingsame word "split", very different pain Databases
  28. Autoscaling on the Wrong Signal (CPU vs Queue Depth)CPU vs Queue Depth Cloud infrastructure
  29. Request Cancellation & Stale Responses (AbortController)"Search shows the wrong results. Type 'react' fast and sometimes you get results for 'rea'. Refreshing fixes it." Web development

August 2026

  1. Exponential Backoff with Jitterbuild a correct retry in 12 lines System design
  2. MVCC, Dead Tuples & Autovacuumwhy deleting rows made your DB bigger Databases
  3. OIDC Federation for CI (Keyless Cloud Credentials)keyless cloud credentials Cloud infrastructure
  4. Flame Graphs & Sampling Profilersreading a profiler without fooling yourself Developer tooling
  5. RAG Chunking Strategywhy your RAG "can't find" text that's definitely in the docs AI engineering
  6. Head-of-Line Blocking (HTTP/1.1 vs HTTP/2 vs HTTP/3)HTTP/2 fixed HOL blocking at the application layer but pushed it down to the transport (TCP) layer. On a lossy network it can be just as painful. HTTP/3 is what actually… Web development
  7. Little's Lawthe napkin math that sizes every pool you own System design
  8. Serverless Cold Starts & Provisioned Concurrencythe warm-up ping myth Cloud infrastructure
  9. CI Dependency Caching (cache keys & restore-keys)your cache is a hash table, not a folder Developer tooling
  10. Content Security Policy (nonce vs hash)You finally ship a CSP header to kill XSS. The page goes blank: Web development
  11. Database Deadlocks & Lock Orderingdebug one with me Databases
  12. LLM-as-a-Judgebuild one in 12 lines, then learn the 4 ways it lies AI engineering
  13. Transactional Outbox Patternwhy "save, then publish" quietly loses events System design
  14. DNS TTL & Failover Tailthe failover that took 45 minutes with a 60-second TTL Cloud infrastructure
  15. Constant-Time ComparisonWhy: your 5-liner is correct in Python, but a JIT or an optimizing C compiler is free to reintroduce a branch. The stdlib versions are written to survive that. Web development
  16. Alpine vs Debian-slim Base Imagesthe "small image" trade you probably got wrong Developer tooling
  17. Layout Thrashing (Forced Synchronous Reflow)why a loop that adds no work drops you from 60fps to 8fps Web development
  18. Speculative Decodinghow a 1B model makes your 70B model 2x faster AI engineering
  19. Tail Latency Amplificationwhy your p99 lies System design
  20. Terraform State & Driftthe file that thinks it knows your cloud Cloud infrastructure
  21. Replication Lag & Read-Your-WritesA read replica is a live stream of a football match. The stadium (primary) is the truth; the stream is the same match, faithfully, ~8 seconds late. You add replicas to get more… Databases
  22. Shell Strictness (set -euo pipefail)the deploy that "succeeded" Developer tooling
  23. Hydration (SSR vs Interactive)the myth that SSR makes your page interactive faster Web development
  24. Constrained Decoding (Structured Outputs)a debug-along AI engineering
  25. CPU Limits & CFS Throttlingwhy your pod is slow at 30% CPU usage Cloud infrastructure
  26. Backpressurebuild it in 5 lines System design

July 2026

  1. Expand-Contract Migrationshow to change a schema without downtime Developer tooling
  2. Write-Ahead Logging (WAL)how a database survives a crash mid-write Databases
  3. The Event Loop: Microtasks vs MacrotasksMost people say: 1, 2, 3, 4 — "setTimeout(0) runs next, right?" Web development
  4. The KV Cache (Prefill vs Decode)why your LLM's first token is slow and the rest fly AI engineering
  5. L4 vs L7 Load BalancingBoth spread traffic across servers. But they operate at different layers — and picking the wrong one quietly costs you features or latency. Cloud infrastructure
  6. Lockfiles & npm ciwhy `npm install` and `npm ci` are NOT the same Developer tooling
  7. Cache Stampede (Thundering Herd)the moment your cache HELPS the outage System design
  8. Tree Shakingwhy your "tiny" import still ships 70 KB Web development
  9. LLM Quantizationshrinking models without lobotomizing them AI engineering
  10. Dead Letter Queues (DLQ)where poison messages go to be studied, not to kill your consumer Cloud infrastructure
  11. Bloom Filtersa "probably yes / definitely no" set that fits in a few KB System design
  12. SameSite Cookieswhy your login works locally but dies in prod Web development
  13. Feature Flagsdecoupling "deployed" from "released" Developer tooling
  14. Reranking (Cross-Encoders in RAG)why your RAG retrieves the right doc but still answers wrong AI engineering
  15. Transaction Isolation Levelsthe dial that decides which "impossible" bugs are possible Databases
  16. Graceful Shutdown (SIGTERM handling)You ship a new version. Rolling deploy, zero errors in the dashboard... except your on-call gets 12 reports of "payment succeeded but order never created." Every deploy, a… Cloud infrastructure
  17. Circuit Breaker Patternwhy "just retry" makes outages worse System design
  18. Real-Time Updates: Long Polling vs SSE vs WebSocketsLong Polling vs SSE vs WebSockets Web development
  19. The N+1 Query Problemthe silent killer of ORM performance Databases
  20. Distributed Tracingfollowing one request across many services Developer tooling
  21. Prompt Injectionwhen your LLM app trusts the wrong text AI engineering
  22. Blue-Green vs Canary Deploymentstwo ways to ship without a maintenance window Cloud infrastructure
  23. HTTP Caching (Cache-Control & ETags)the deploy your users couldn't see Web development
  24. Token Bucket Rate Limitingbuild the real thing in ~12 lines System design
  25. Database Indexes (B-tree)debugging the query that got slow overnight Databases
  26. RAG vs Fine-tuningtwo ways to "teach" an LLM, and picking the wrong one costs you AI engineering

June 2026

  1. Saga Patternhow to "undo" across microservices that don't share a database System design
  2. git bisectbinary search for the commit that broke it Developer tooling
  3. CORS Preflight RequestsMake the server answer the OPTIONS with explicit permission: Web development
  4. Liveness vs Readiness ProbesTwo probes, almost identical YAML, completely different jobs. Mixing them up is one of the most common ways a healthy service ends up in a CrashLoopBackOff death spiral. Cloud infrastructure
  5. Optimistic Concurrency ControlTwo support agents open the same ticket. Both see "priority: low". Anna bumps it to High and saves. Bob, a second later, changes the assignee and saves. Anna's High is gone —… System design
  6. Offset vs Cursor PaginationTwo ways to page through a big list. At page 1 they look identical. At page 50,000 one of them quietly falls over. Knowing which — and why — is a classic intermediate line. System design
  7. Consistent HashingImagine a circle numbered 0 to 2^32-1, wrapping around like a clock. You hash each SERVER (by name/IP) to a point on the ring. You hash each KEY to a point too. A key belongs… System design
  8. Temperature 0 vs Deterministictemperature=0 makes sampling GREEDY: at each step the model picks the single highest-probability token instead of sampling from the distribution. That removes the randomness… AI engineering
  9. Connection PoolingYour API runs great in dev. You deploy to production on serverless (Lambda / Cloud Run), traffic hits ~300 req/s, and Postgres starts throwing: Cloud infrastructure
  10. Vector Embeddings & Semantic SearchAn embedding is a fixed-length list of numbers a model generates from a piece of text (or an image), placing similar meanings near each other in vector space. Semantic search… AI engineering
  11. Mobile Radio Networks (3G/4G/5G) & Web Performanceit's LATENCY, not "speed" Web development
  12. Idempotency KeysAn idempotency key is a unique client-generated token (often a UUID) sent with a write request so the server can safely retry it. If the same key arrives twice, the server… System design
  13. Docker Layer Cachingwhy every build reinstalls ALL your dependencies Developer tooling
  14. Debouncing vs Throttlingtwo fixes for "this fires way too often" Web development