Track 1 · Module 1.7 · OS & Concurrency Fundamentals

Processes, Threads & Concurrency

How servers juggle millions of simultaneous requests. Understand the physical boundaries of memory, the hidden latency cost of CPU context switching, and why modern architectures chose Event Loops and Goroutines over traditional thread-per-request models.

⏱️ Reading Time: ~20 min

🎯 Core Skill: Server Concurrency Architecture

🎬 Format: Interactive Concurrency Simulator

⚡ Real-Time Simulator

Server Concurrency Model Stress Test

Scale connections to see RAM & context switch impact

1. Concurrent Connections: 2,000

1002,5005,00010,000 reqs

2. Server Architecture Model

RAM Consumption2,048 MB

Active OS Threads2,000

Context Switches / sec~140,000

System Status⚠️ Heavy Switching

In Thread-per-Request mode, each connection allocates a dedicated 1MB OS stack. At 2,000 connections, 2GB of RAM is spent solely on idle thread stacks. The CPU burns massive compute time saving and restoring registers across 2,000 threads.

🏢 1. What is a Process? (The Apartment Building)

🏠 Real-World Analogy

Imagine an apartment building. Each process is like a completely private, locked apartment. It has its own kitchen, bedroom, front door key, and power meter. People living in Apartment A cannot reach into Apartment B to grab a glass of water without ringing the doorbell (inter-process communication).

In computer hardware, a Process is an executing instance of a program. The Operating System creates a protective sandbox called a Virtual Address Space. Process A's memory address 0x00400000 points to completely different physical RAM chips than Process B's 0x00400000.

🛡️ The Blast Radius Advantage:If Spotify hits a segmentation fault bug and crashes, Chrome doesn't even notice. The OS kernel immediately reclaims Process B's memory pages without harming Process A.

👥 2. What is a Thread? (Roommates in the Same Apartment)

🏠 Real-World Analogy

If a process is an apartment, Threads are roommates living together inside that apartment. They share the kitchen fridge (the Heap) and the living room TV (the Code). However, each roommate has their own private backpack (their own Thread Stack & CPU Registers) to keep track of their personal tasks.

Dimension	Process	Thread
Memory Boundary	Completely isolated virtual address space.	Shares Heap, Code, & Data with sibling threads.
Creation Overhead	Heavy (~10–50ms, kernel page table cloning).	Lightweight (~10–50µs, just allocates a stack).
Communication (IPC)	Slow: Sockets, Unix Pipes, Shared Memory maps.	Ultra-fast: Direct pointer reads in shared Heap.
Failure Blast Radius	Isolated: 1 crashed process doesn't kill others.	Fatal: If 1 thread corrupts memory, entire process dies.

⏱️ 3. Concurrency vs Parallelism (The Master Chef)

🍳 The Kitchen Analogy

Engineers frequently confuse these two terms. Rob Pike (creator of Go) famously said: "Concurrency is about structure; Parallelism is about execution."

Concurrency (1 CPU Core · Interleaving)

One chef juggling 3 dishes. They chop onions for 5 seconds, flip the steak for 5 seconds, then stir the soup. Only 1 dish is actively touched at any microsecond, but all 3 make progress over time.

Parallelism (4 CPU Cores · Simultaneous)

Four chefs cooking 4 dishes simultaneously. Each chef has their own stove (CPU core). Physical work happens at the exact same instant in time.

🔄 4. Context Switching — The Hidden Latency Tax

📖 The Bookmark Problem

Imagine you are reading 10 dense textbooks at the same time. Every 2 minutes, someone yells "SWITCH!". You must stop, write down the exact page and sentence on a bookmark, put the book back on the shelf, grab the next book, open to its bookmark, and spend 15 seconds remembering what the chapter was about.

If you switch books 100 times an hour, you spend more time managing bookmarks than actually reading! This is exactly what happens when an OS runs too many threads.

1. Save Registers

The CPU stops Thread A and dumps registers (RAX, RIP, stack pointer) into kernel memory.

2. Kernel Scheduler

OS scheduler algorithm (Linux CFS) picks the highest-priority runnable thread from the run-queue.

3. TLB & Cache Invalidation

Switching processes flushes the TLB cache. CPU L1/L2 data caches get evicted (Cold Cache Penalty).

4. Restore State

Thread B's registers are reloaded into hardware and the instruction pointer jumps to Thread B's code.

📊 The Cost in Numbers:A single thread context switch costs ~1 to 5 microseconds. If your server runs 10,000 threads switching 1,000 times/sec, you lose 30% to 70% of your total CPU capacity purely to switching overhead!

🏛️ 5. Three Server Architectures (The Interview Core)

🔗 High-Level Design Decision

Every web framework in the world (Spring, Node.js, Go, Nginx, Django, Elixir) picks one of these three architectures. In a system design interview, you must justify why you chose one over the others.

Model 1: Thread-per-Request (Apache, Tomcat, Traditional Java)Sync / Blocking

Every incoming HTTP socket connection is assigned a dedicated OS thread from a pool. The thread executes your code synchronously — when it calls db.query(), the thread sits 100% blocked in sleep state until the database responds.

// Traditional Java / Spring Boot

User user = db.query("SELECT * FROM users WHERE id=1"); // Thread BLOCKS here

return new Response(user);

✅ Pros: Simple linear code, easy debugging, excellent for heavy CPU calculations.

❌ Cons: 1MB RAM stack per thread. Maxes out at ~2,000–5,000 connections before memory exhaustion.

Model 2: Single-Threaded Event Loop (Node.js, Nginx, Redis)Async / Non-Blocking (epoll)

ONE single OS thread handles all connections using OS kernel notification mechanisms (epoll on Linux, kqueue on macOS). When an I/O query starts, the thread registers a callback and immediately services the next user. When data arrives, the event loop triggers the callback.

// Node.js (Async / Await syntactic sugar over callbacks)

const user = await db.query("SELECT * FROM users WHERE id=1"); // Thread yields control to handle others

return res.json(user);

✅ Pros: Zero context switching, tiny memory footprint (<30MB). Handles 100,000+ idle WebSockets effortlessly.

❌ Cons: CPU-bound work freezes everything. If one user requests a CPU-heavy image resize, all other 99,999 users freeze!

Model 3: M:N Green Threads & Goroutines (Go Runtime, Erlang BEAM)Hybrid M:N Scheduler

Combines the clean synchronous coding style of Model 1 with the ultra-low memory efficiency of Model 2. The Go runtime creates lightweight Goroutines that start with only 2KB stack size (grows dynamically). The Go M:N scheduler multiplexes 1,000,000 goroutines onto a small pool of 8 real OS threads.

// Go (Goroutines)

user := db.QueryRow("SELECT * FROM users WHERE id=1") // Looks blocking, but only blocks the Goroutine, not the OS Thread!

return json(user)

✅ Pros: Code looks synchronous (resp := fetch()) without callback hell. Scales to millions of connections.

❌ Cons: Requires a dedicated language runtime scheduler (Go, Erlang).

⚠️ 6. Concurrency Bugs: Race Conditions & Deadlocks

💸 The Bank Account Bug

Because threads share the same Heap memory, concurrent access without synchronization causes silent, catastrophic data corruption.

The Classic Race Condition (Read-Modify-Write)

Database / Shared MemoryBalance: $100

Thread A (ATM 1)

T1Read: $100

T2Wait...

T3Subtract $50 (Local = $50)

T4Write: $50

Thread B (ATM 2)

T1Wait...

T2Read: $100 (Stale!)

T3Wait...

T5Write: $50 (Overwrites A!)

❌ Total withdrawn = $100, but account still has $50 left! Free money bug.

🔒 The Solution: Mutex (Mutual Exclusion)

Thread A acquires a hardware lock (mutex.Lock()) before reading. Thread B is forced to wait until Thread A finishes writing and unlocks. Only 1 thread touches the shared memory at a time.

mutex.Lock()

balance = balance - 50

mutex.Unlock()

💀 The Danger: Deadlocks

Thread A holds Lock 1 and waits for Lock 2.
Thread B holds Lock 2 and waits for Lock 1.
Both threads freeze forever waiting on each other.

Fix: Strict global lock acquisition ordering (always acquire Lock 1 before Lock 2).

🎯 7. Test Your Understanding

Scenario 1: The Memory Crash Under Load

Your Java Spring Boot server handles 500 concurrent users with 5% CPU usage. When traffic spikes to 5,000 users, CPU usage stays at only 20%, but response latency jumps to 15 seconds and the server crashes with OutOfMemoryError. What is the root cause?

Scenario 2: The Frozen Real-Time Chat

A single-instance Node.js chat server comfortably holds 50,000 open WebSockets. However, whenever a user requests a PDF invoice export (which computes a complex vector graphic in JavaScript), all 50,000 chat users experience a complete 3-second freeze. Why?

← Module 1.6: What Happens When You Type a URL Module 1.8: Thrashing & Memory Bottlenecks →