Backend First Principles
Articles/Phase 4 — Data Persistence & Storage Engines

Caching — The Secret Behind High-Performance Backend Systems

From CDN Edge Nodes and DNS to CPU L1/L2/L3, RAM vs Disk, Redis Cache-Aside, Eviction (LRU/LFU/TTL), Database Query Caching, and In-Memory Rate Limiting.

Core Concept & First Principle

Caching is the fundamental engineering practice of storing a frequently requested subset of data in high-speed storage (RAM/Edge) to avoid repeated expensive computation or disk retrieval. Mastering multi-level caching—from CDNs and DNS resolution to hardware CPU caches, Redis Cache-Aside patterns, eviction policies (LRU/LFU/TTL), and in-memory rate limiting—transforms database bottlenecks into sub-millisecond responses.

1. What is Caching & Why It Exists: Core Mechanics & Mental Model

At its first principles level, Caching is the technique of storing a subset of original data in a high-speed, faster-access storage medium (RAM or Edge PoP) to drastically decrease the time and computational effort required for subsequent retrievals.

The Fundamental Caching Rule: Subset, Never Everything
A cache is strictly a temporary subset of authoritative data. High-speed memory (RAM) is finite and orders of magnitude more expensive per gigabyte than disk storage. Attempting to cache the entire database is an anti-pattern that defeats the architectural purpose of a cache.

01Trigger #1: Expensive Computation

When generating a result consumes significant CPU, GPU, or complex data aggregation (e.g. searching across billions of indexed documents, calculating trending hashtags, running machine learning ranking), re-computing identical queries for every user is wasteful. Calculate once → Cache result → Reuse across millions.

02Trigger #2: Expensive or Large Data Retrieval

When data resides on slow disk-based databases or distant geographic servers (e.g. streaming 4K video files, multi-table SQL joins, external 3rd-party currency APIs), fetching over the network repeatedly introduces latency and network congestion. Read once → Cache hot slice → Serve repeatedly.

Avoiding Repeated Index Ranking

Google Search

When millions of users search 'weather in Delhi' or 'cricket score', Google serves pre-computed search result cards from distributed in-memory cache clusters, skipping crawling and ranking pipelines.

Global Edge Content Delivery

Netflix Streaming

Instead of streaming 4K video from a central US origin server, Netflix encodes videos into multi-bitrate chunks (1080p, 720p, 480p) and caches popular regional movies at local CDN Edge Nodes (Open Connect).

Trending Topics Precomputation

Twitter / X

Analyzing millions of live tweets for viral trends requires heavy ML processing. Twitter precomputes trends periodically into Redis, allowing millions of timeline refreshes to hit in-memory RAM with zero latency.

Mental Model: Cache Hit vs. Cache Miss
INPUT / REQUEST
// Request 1 (Cache MISS):
GET /api/products/456
1. Check Redis: NOT FOUND (Miss)
2. Query PostgreSQL on NVMe Disk: ~120ms
3. Write to Redis (TTL = 3600s)
4. Return Response
OUTPUT / RESULT
// Request 2 (Cache HIT):
GET /api/products/456
1. Check Redis: FOUND (Hit) ~0.8ms
2. Return Response immediately (Bypassing PostgreSQL entirely!)
Takeaway:A Cache Hit saves over 99% of query latency and eliminates database CPU and connection pool consumption.

2. Multi-Layer Network & Edge Caching: CDNs and Multi-Level DNS

Caching begins long before a network packet reaches your backend server. Geographically distributed network caches eliminate long-distance speed-of-light latency.

01Content Delivery Networks (CDNs) & Edge Locations

A CDN is a globally distributed network of Edge Servers (Points of Presence - PoPs). Using Anycast BGP routing, requests for static assets (images, JavaScript, CSS, video segments) are intercepted and served from the server geographically closest to the user, bypassing the central origin server.

02Why CDNs Cache Popular Subsets, Not Everything

A platform with petabytes of catalog content (like Netflix or Amazon) cannot store all files in every edge node. Edge servers dynamically cache the most popular 10–20% of regional content, evicting cold assets back to central origin storage.

03Multi-Level DNS Caching Architecture

Translating example.com to an IP address without caching would require traversing the global DNS hierarchy on every single HTTP link click. DNS employs a 4-tier caching chain that halts resolution as early as possible.
Comparison Table↔ Scroll horizontally
DNS Cache LayerLocation & StorageTypical Resolution LatencyRole & Scope
1. Browser DNS CacheClient browser memory (Chrome/Firefox)< 1 msChecks if the active browser session has resolved the hostname within the last few minutes.
2. OS DNS CacheOperating System socket resolver cache< 2 msMaintained by Windows DNS Client / macOS mDNS / Linux systemd-resolved across all local apps.
3. Recursive Resolver CacheISP or Public DNS (1.1.1.1, 8.8.8.8)~10 – 30 msCaches domain records for thousands of neighboring users; queries Root/TLD servers only on miss.
4. Authoritative Name ServerDomain Registrar / Cloudflare DNS~50 – 150 msThe authoritative single source of truth containing official A/AAAA/CNAME records.
DNS TTL (Time-To-Live) Tradeoff
DNS records include a TTL (e.g. TTL = 3600). Long TTL maximizes cache hits and speeds up page loads, but slows down DNS failover if an IP address changes during server migrations. Short TTL allows fast failover but increases recursive DNS lookup latency.

3. Hardware-Level Caching: CPU L1/L2/L3 & RAM vs. Disk

The principle of caching is universal—it is implemented in silicon inside modern CPUs as well as across server memory hierarchies.

Comparison Table↔ Scroll horizontally
Memory LayerPhysical LocationTypical SizeAccess LatencyRelative Speed Factor
CPU L1 CacheDirectly on CPU Core32 KB – 64 KB per core~0.5 – 1 nanosecondInstantaneous (1x)
CPU L2 CacheDedicated Core Cache512 KB – 1 MB per core~3 – 5 nanoseconds~5x slower than L1
CPU L3 CacheShared across all Cores16 MB – 64 MB shared~10 – 20 nanoseconds~20x slower than L1
System RAM (Memory)DDR4 / DDR5 Modules16 GB – 512 GB~60 – 100 nanoseconds~100x slower than L1
NVMe SSD (Disk Storage)PCIe Flash Bus500 GB – 8 TB~50 – 150 microseconds~1,000x slower than RAM!
Why In-Memory Stores (Redis / Memcached) Win
Accessing data in RAM takes nanoseconds, while querying an SSD disk takes microseconds (and rotational HDDs take milliseconds). In-memory key-value stores like Redis operate purely in RAM, providing predictable sub-millisecond responses that traditional disk-bound databases cannot achieve under heavy load.

4. Software Caching Strategies: Cache-Aside vs. Write-Through

Selecting how the application coordinates reads and writes between the cache and the primary database is critical for balancing performance against data freshness.

Populate on Demand

Strategy 1: Cache-Aside (Lazy Caching)

The application checks the cache first. On a cache hit, it returns data immediately. On a cache miss, it reads from the database, writes the fetched record into the cache with a TTL, and returns it. Most popular and resilient pattern for read-heavy REST APIs.

Simultaneous Dual-Write

Strategy 2: Write-Through

Whenever data is created or updated, the application writes simultaneously to both the primary database and the cache. Guarantees the cache is always 100% fresh, but adds write latency and populates keys that may never be read again.

Comparison Table↔ Scroll horizontally
Architectural DimensionCache-Aside (Lazy Caching)Write-Through Caching
Cache Population TriggerOn Read Miss (GET request)On Write Mutation (POST/PUT/DELETE)
Read LatencySub-millisecond on hits; penalty on initial missAlways sub-millisecond (cache is pre-warmed)
Write LatencyMinimal (writes go straight to DB)Higher (must wait for DB + Redis write confirmation)
Memory EfficiencyHigh: Only actually requested data occupies RAMLower: Caches written data that might never be queried
Staleness RiskPossible if DB is modified without cache evictionZero: Cache is updated synchronously with DB

5. Cache Eviction Policies & Memory Management (LRU, LFU, TTL)

Because cache storage (RAM) is strictly bounded, when the cache reaches its memory limit (e.g. maxmemory 2gb), the engine must execute an Eviction Policy to discard old keys to accommodate new incoming data.

01LRU (Least Recently Used) — Recency-Based

Evicts the key that has not been read or accessed for the longest duration. Best for temporal locality where recently queried items are most likely to be requested again.

02LFU (Least Frequently Used) — Frequency-Based

Maintains an access counter for each key and evicts items with the lowest total request count. Protects frequently accessed 'evergreen' items from being evicted by sudden temporary spikes in one-off requests.

03TTL (Time-To-Live / volatile-ttl) — Expiration-Based

Evicts keys with an explicit expiration timestamp that are closest to expiring. Guarantees bounded data freshness for time-sensitive data (e.g. weather forecasts, stock prices).

04noeviction (Strict Fail-Safe)

Refuses to evict any data. When memory fills up, all incoming write commands (SET, HSET) return an OOM (Out of Memory) error while read operations continue to work.
Interview Cheat Sheet: LRU vs. LFU
LRU asks: 'Which item hasn't been used recently?' (tracks last accessed timestamp). LFU asks: 'Which item is used the least overall?' (tracks access frequency counter).

6. Practical Backend Use Cases & Real-World Patterns

In production backend engineering, caching solves four critical scalability bottlenecks:

E-Commerce Product Catalogs

1. Database Query Caching

Complex SQL queries with multiple table JOINs, aggregations, and category filtering (e.g. Amazon product pages) are cached in Redis. Millions of users read the cached JSON, protecting the PostgreSQL database from crashing during Black Friday sales.

User Authentication Stores

2. Session Management

After a user logs in, their profile, permissions, and session metadata are stored in Redis (sess_a8b9c0...). Verifying user identity on every API request becomes a 1ms RAM lookup rather than a disk query.

Third-Party Rate Limit & Cost Defense

3. External API Caching

Calling external APIs (e.g. Weather API, Stripe FX rates, AI models) consumes API quota and incurs monetary billing. Caching API responses with a 1-hour TTL cuts external API costs by over 95%.

DDoS & Abuse Mitigation

4. Rate Limiting

Using high-speed Redis atomic increments (INCR + EXPIRE) to track IP request counts (e.g. 50 req/sec) without burdening persistent relational databases with high-frequency counter writes.

Why Never Use Relational Databases for Rate Limiting
If you write an UPDATE users SET request_count = request_count + 1 WHERE ip = '...' to PostgreSQL on every incoming HTTP request, you generate massive disk write contention and lock bottlenecks. In-memory Redis INCR handles 100,000+ atomic operations per second with microsecond latency.

7. Multi-Language Code Implementations: Cache-Aside & Rate Limiting

Production-ready implementations of the Cache-Aside Pattern with fallback database lookups and Redis Rate Limiting Middleware across TypeScript, Go, and Rust:

1import { Request, Response, NextFunction } from 'express';
2import Redis from 'ioredis';
3
4const redis = new Redis(process.env.REDIS_URL || 'redis://localhost:6379');
5
6// 1. Generic Cache-Aside Pattern
7export async function getOrSetCache<T>(
8 key: string,
9 ttlSeconds: number,
10 fetchDb: () => Promise<T>
11): Promise<T> {
12 const cached = await redis.get(key);
13 if (cached) {
14 return JSON.parse(cached);
15 }
16
17 const freshData = await fetchDb();
18 await redis.set(key, JSON.stringify(freshData), 'EX', ttlSeconds);
19 return freshData;
20}
21
22// 2. High-Speed Rate Limiting Middleware (HTTP 429)
23export async function rateLimitGuard(req: Request, res: Response, next: NextFunction) {
24 const ip = req.ip || '127.0.0.1';
25 const key = `rate:${ip}`;
26
27 const requests = await redis.incr(key);
28 if (requests === 1) {
29 await redis.expire(key, 60); // 1-minute sliding window
30 }
31
32 res.setHeader('X-RateLimit-Limit', '50');
33 res.setHeader('X-RateLimit-Remaining', Math.max(0, 50 - requests).toString());
34
35 if (requests > 50) {
36 return res.status(429).json({
37 error: 'HTTP 429 Too Many Requests',
38 message: 'Rate limit exceeded. Please wait 60 seconds.'
39 });
40 }
41 next();
42}

8. The Fundamental Tradeoffs & The Cache Invalidation Dilemma

Phil Karlton famously observed: 'There are only two hard things in Computer Science: cache invalidation and naming things.' Understanding when NOT to cache is as critical as knowing how to cache.

Comparison Table↔ Scroll horizontally
ScenarioShould You Cache?Architectural Rationale
Read-Heavy Data (Read >>> Write)✅ Highly RecommendedHuge performance payoff; database load drops by 90%+.
Expensive Computations / Aggregations✅ Highly RecommendedSaves CPU/GPU resources by computing once and reusing results.
External API Responses with Rate Limits✅ Highly RecommendedProtects against third-party billing costs and rate limit exhaustion.
Rapidly Mutating Data (Write-Heavy)❌ Avoid / Be CautiousHigh mutation rate constantly invalidates cache, causing cache churn and low hit ratio.
Strict Consistency Requirements (Banking Balances)❌ Avoid Caching Authoritative StateServing stale financial balances or inventory stock causes double-spending and data corruption.
Production Checklist & Guidelines
Always set an explicit TTL on all cache keys to prevent memory leaks from abandoned data.
Use Cache-Aside as the default architectural starting point for read-heavy REST APIs.
Never cache unvalidated or sensitive PII unless encrypted with tenant-scoped keys.
Use in-memory Redis atomic operations (INCR/EXPIRE) for high-throughput rate limiting instead of relational DB writes.
Monitor your Cache Hit Ratio: A healthy production cache should maintain an 85%–98% hit rate.
Summary & Core Takeaways
Key Insights
1Caching stores a hot subset of data in high-speed storage (RAM/Edge) to reduce expensive computations and slow disk I/O.
2Multi-level caching exists across the entire stack: Network (CDN, DNS), Hardware (CPU L1/L2/L3, RAM), and Software (Redis, Memcached).
3Cache-Aside (Lazy) is the standard for read-heavy REST APIs; Write-Through maintains strict cache freshness during writes.
4When cache memory fills up, eviction algorithms (LRU for recency, LFU for frequency, TTL for expiration) protect system stability.
5Redis in-memory atomic operations power critical backend infrastructure including query caches, session stores, and HTTP 429 rate limiters.
Interactive Distributed Caching Simulator

Multi-Tier Cache & Eviction Engine

1. Client Browser
Local Memory Cache

Directly inside browser RAM. Eliminates network roundtrips completely.

Storage: Client RAM
2. CDN Edge Node
Point of Presence (PoP)

Geographically close edge server. Serves static chunks and images.

Storage: Edge SSD / RAM
3. Redis Cache
In-Memory RAM Store

Stores hot precomputed trends, user sessions, and database query results.

Storage: Server RAM (DDR5)
4. Primary Database
PostgreSQL Engine

Authoritative disk storage. High latency sequential scan or index lookup.

Storage: NVMe SSD Disk
Previous Lesson
Databases for Backend Engineers: PostgreSQL, Schemas, Migrations & Performance
Next Lesson
Background Jobs & Task Queues: The Architecture of Scalable & Asynchronous Backends