Latency, Throughput, Memory & Accuracy
The universal laws of software architecture: why tuning for one dimension forces compromises in others. Mastering Batching vs. Streaming, In-Memory Caching vs. Cold Tiering, and Exact SQL vs. HyperLogLog Probabilistic Approximation.
1. The Architectural Iron Quadrangle
In system design interviews, there is no universally “best” architecture. Every design decision represents a position inside the Four-Way Trade-Off Quadrangle:
Time taken to process one individual request (e.g. 5 milliseconds).
Total requests or data volume processed per second (e.g. 100,000 QPS).
Hardware RAM, disk, and cloud infrastructure cost per month.
Precision of computation (100% exact SQL count vs 99.2% approximation).
🏎️ The Formula 1 Car vs. City Bus Analogy
A Formula 1 car is tuned for extreme low latency: it transports 1 person across town at 200 MPH in 2 minutes. But its throughput is terrible (1 passenger per trip) and its cost is astronomical ($10,000,000).
A City Bus has higher latency (takes 30 minutes, stops at every light), but delivers massive throughput (transports 100 passengers per trip) at a tiny fraction of the cost per passenger!
2. Throughput vs. Latency: The Mechanics of Batching
Why does optimizing for maximum throughput almost always increase the latency of individual requests?
Immediate Streaming (Zero Batching)
Every incoming event triggers an immediate network round-trip and database transaction.
Micro-Batching (50ms / 500 items Buffer)
Events wait in an in-memory buffer and flush as a single bulk network packet / multi-row insert.
Kafka achieves millions of messages/sec per broker by tuning linger.ms=20 and batch.size=65536. Producers hold messages for up to 20 milliseconds to pack full TCP buffers, trading 20ms of artificial latency for a 50x surge in throughput!
3. Accuracy vs. Latency: Exact SQL vs. HyperLogLog
When counting 1,000,000,000 unique visitors (cardinality estimation), calculating the 100% exact number requires storing every single 64-bit user ID in a massive hash set (8GB+ RAM) and sorting/deduplicating:
| Method | Memory Footprint | Execution Latency | Accuracy |
|---|---|---|---|
| Exact Hash Set (SQL DISTINCT) | ~8.4 Gigabytes RAM | 42,000 ms (42 seconds) | 100.0% Exact |
| HyperLogLog (Redis PFADD) | 12 Kilobytes (Constant!) | 0.15 ms (Sub-millisecond) | 99.19% (0.81% Standard Error) |
🎯 When Is 99% Accuracy “Good Enough”?
If YouTube displays 14,284,912 views on a video, nobody cares if the true count was 14,291,200. Trading 0.8% precision allows YouTube to save millions of dollars in RAM and serve counts instantly from Redis! (Financial ledgers and bank balances, however, must always remain 100% exact).
4. Interactive 4-Way Architectural Trade-off Lab
Adjust batch buffers, memory cache allocation, and algorithm accuracy to observe how tuning one architectural knob impacts latency, throughput, cost, and precision in real time.
5. Adaptive Batching Buffer in 5 Languages
Below is an implementation of a thread-safe adaptive batching buffer that flushes when either the maximum batch size or the time window threshold is exceeded.
import asyncio
from typing import List, Any
class AdaptiveBatcher:
def __init__(self, max_batch_size: int = 500, max_latency_ms: float = 50.0):
self.max_batch_size = max_batch_size
self.max_latency_sec = max_latency_ms / 1000.0
self.buffer: List[Any] = []
self.lock = asyncio.Lock()
self.flush_task = None
async def add(self, item: Any):
async with self.lock:
self.buffer.append(item)
if len(self.buffer) >= self.max_batch_size:
await self._flush_internal()
elif len(self.buffer) == 1:
# First item in new batch: schedule deadline flush
self.flush_task = asyncio.create_task(self._schedule_flush())
async def _schedule_flush(self):
await asyncio.sleep(self.max_latency_sec)
async with self.lock:
if self.buffer:
await self._flush_internal()
async def _flush_internal(self):
if not self.buffer:
return
batch_to_send = self.buffer
self.buffer = []
if self.flush_task and not self.flush_task.done():
self.flush_task.cancel()
# Execute single bulk network write
await self.execute_bulk_insert(batch_to_send)
async def execute_bulk_insert(self, batch: List[Any]):
# Single bulk I/O call
pass
export class AdaptiveBatcher<T> {
private buffer: T[] = [];
private timer: NodeJS.Timeout | null = null;
constructor(
private readonly maxBatchSize = 500,
private readonly maxLatencyMs = 50,
private readonly flushHandler: (batch: T[]) => Promise<void>
) {}
public add(item: T): void {
this.buffer.push(item);
if (this.buffer.length >= this.maxBatchSize) {
this.flush();
} else if (this.buffer.length === 1) {
this.timer = setTimeout(() => this.flush(), this.maxLatencyMs);
}
}
public async flush(): Promise<void> {
if (this.timer) {
clearTimeout(this.timer);
this.timer = null;
}
if (this.buffer.length === 0) return;
const batch = [...this.buffer];
this.buffer = [];
await this.flushHandler(batch);
}
}
package main
import (
"sync"
"time"
)
type Batcher struct {
maxBatchSize int
maxDelay time.Duration
buffer []interface{}
mu sync.Mutex
timer *time.Timer
flushFunc func([]interface{})
}
func (b *Batcher) Add(item interface{}) {
b.mu.Lock()
defer b.mu.Unlock()
b.buffer = append(b.buffer, item)
if len(b.buffer) >= b.maxBatchSize {
b.flushLocked()
} else if len(b.buffer) == 1 {
b.timer = time.AfterFunc(b.maxDelay, func() {
b.mu.Lock()
defer b.mu.Unlock()
b.flushLocked()
})
}
}
func (b *Batcher) flushLocked() {
if b.timer != nil {
b.timer.Stop()
}
if len(b.buffer) == 0 {
return
}
batch := make([]interface{}, len(b.buffer))
copy(batch, b.buffer)
b.buffer = b.buffer[:0]
go b.flushFunc(batch)
}
import java.util.*;
import java.util.concurrent.*;
import java.util.function.Consumer;
public class AdaptiveBatcher<T> {
private final int maxBatchSize;
private final List<T> buffer = new ArrayList<>();
private final ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();
private final Consumer<List<T>> flushConsumer;
private ScheduledFuture<?> scheduledFlush;
public AdaptiveBatcher(int maxBatchSize, long maxLatencyMs, Consumer<List<T>> flushConsumer) {
this.maxBatchSize = maxBatchSize;
this.flushConsumer = flushConsumer;
}
public synchronized void add(T item) {
buffer.add(item);
if (buffer.size() >= maxBatchSize) {
flush();
} else if (buffer.size() == 1) {
scheduledFlush = scheduler.schedule(this::flush, 50, TimeUnit.MILLISECONDS);
}
}
public synchronized void flush() {
if (scheduledFlush != null) scheduledFlush.cancel(false);
if (buffer.isEmpty()) return;
List<T> batch = new ArrayList<>(buffer);
buffer.clear();
flushConsumer.accept(batch);
}
}
#include <vector>
#include <mutex>
#include <functional>
#include <chrono>
template <typename T>
class AdaptiveBatcher {
private:
size_t max_size;
std::chrono::milliseconds max_delay;
std::vector<T> buffer;
std::mutex mtx;
std::function<void(std::vector<T>)> flush_fn;
public:
AdaptiveBatcher(size_t size, std::chrono::milliseconds delay, std::function<void(std::vector<T>)> fn)
: max_size(size), max_delay(delay), flush_fn(fn) {}
void add(T item) {
std::lock_guard<std::mutex> lock(mtx);
buffer.push_back(item);
if (buffer.size() >= max_size) {
flush_internal();
}
}
void flush_internal() {
if (buffer.empty()) return;
std::vector<T> batch = std::move(buffer);
buffer.clear();
flush_fn(batch);
}
};