manbo/blog-post-backup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: manbo:main
Branches Tags
manbo:main manbo:markyan04-patch-1
..
compare: manbo:markyan04-patch-1
Branches Tags
manbo:main manbo:markyan04-patch-1

1 Commits
main ... markyan04-

Author SHA1 Message Date
Markyan04
1743af8b64 Update drafts/2026-01-24-profiling-rust-written-network-program.md 2026-01-24 15:01:56 +08:00
4 changed files with 2 additions and 1158 deletions
Expand all files Collapse all files

276
drafts/2026-01-24-profiling-rust-written-network-program.md
View File

@@ -192,7 +192,7 @@ move to smol::Executor + N threads (usually num_cpus)
or run multiple block_on() workers (careful: avoid accept() duplication)
## outcome oi
## outcome
### CPU hotspot
testing commands:
@@ -254,280 +254,8 @@ oi_dwarf.svg
### syscall-cost check
```bash
sudo strace -ff -C -p $(pidof oi) -o /tmp/oi.strace
sudo strace -ff -c -p $(pidof oi) -o /tmp/oi.strace
# run 15–30s under load, then Ctrl+C
tail -n +1 /tmp/oi.strace.*
```
## More real setup
traffic goes through real kernel routing + 2 TCP legs
Create namespaces + veth links:
```bash
sudo ip netns add ns_client
sudo ip netns add ns_server
sudo ip link add veth_c type veth peer name veth_c_ns
sudo ip link set veth_c_ns netns ns_client
sudo ip link add veth_s type veth peer name veth_s_ns
sudo ip link set veth_s_ns netns ns_server
sudo ip addr add 10.0.1.1/24 dev veth_c
sudo ip link set veth_c up
sudo ip addr add 10.0.0.1/24 dev veth_s
sudo ip link set veth_s up
sudo ip netns exec ns_client ip addr add 10.0.1.2/24 dev veth_c_ns
sudo ip netns exec ns_client ip link set veth_c_ns up
sudo ip netns exec ns_client ip link set lo up
sudo ip netns exec ns_server ip addr add 10.0.0.2/24 dev veth_s_ns
sudo ip netns exec ns_server ip link set veth_s_ns up
sudo ip netns exec ns_server ip link set lo up
sudo sysctl -w net.ipv4.ip_forward=1
```
Config to force redirect path:
```yaml
10.0.1.1 9000 10.0.0.2 9001
```
Start backend server in ns_server:
```bash
sudo ip netns exec ns_server iperf3 -s -p 9001
```
Run client in ns_client → forwarder → backend:
```bash
sudo ip netns exec ns_client iperf3 -c 10.0.1.1 -p 9000 -t 30 -P 8
```
perf report:
```text
sudo perf stat -p $(pidof oi) -e cycles,instructions,cache-misses,branches,branch-misses,context-switches,cpu-migrations -- sleep 33
Performance counter stats for process id '209785':
113,810,599,893 cpu_atom/cycles/ (0.11%)
164,681,878,450 cpu_core/cycles/ (99.89%)
102,575,167,734 cpu_atom/instructions/ # 0.90 insn per cycle (0.11%)
237,094,207,911 cpu_core/instructions/ # 1.44 insn per cycle (99.89%)
33,093,338 cpu_atom/cache-misses/ (0.11%)
5,381,441 cpu_core/cache-misses/ (99.89%)
20,012,975,873 cpu_atom/branches/ (0.11%)
46,120,077,111 cpu_core/branches/ (99.89%)
211,767,555 cpu_atom/branch-misses/ # 1.06% of all branches (0.11%)
245,969,685 cpu_core/branch-misses/ # 0.53% of all branches (99.89%)
1,686 context-switches
150 cpu-migrations
33.004363800 seconds time elapsed
```
flamegraph
### Add latency + small-packet tests
netperf (request/response)
Start netserver in backend namespace:
```bash
sudo ip netns exec ns_server netserver -p 9001
```
Run TCP_RR against forwarded port:
```bash
sudo ip netns exec ns_client netperf -H 10.0.1.1 -p 9000 -t TCP_RR -l 30 -- -r 32,32
```
## After opt
Here, we changed future_lite::io 8KiB buffer to a customized 16KiB buffer. (To avoid conflict, I changed binary name to oiopt).
```rust
async fn pump(mut r: TcpStream, mut w: TcpStream) -> io::Result<u64> {
// let's try 16KiB instead of future_lite::io 8KiB
// and do a profiling to see the outcome
let mut buf = vec![0u8; 16 * 1024];
let mut total = 0u64;
loop {
let n = r.read(&mut buf).await?;
if n == 0 {
// EOF: send FIN to peer
let _ = w.shutdown(Shutdown::Write);
break;
}
w.write_all(&buf[0..n]).await?;
total += n as u64;
}
Ok(total)
}
// And change the function call in handle_tcp_connection
let client_to_server = pump(client_stream.clone(), server_stream.clone());
let server_to_client = pump(server_stream, client_stream);
```
### outcomes
Still with `sudo ip netns exec ns_client iperf3 -c 10.0.1.1 -p 9000 -t 30 -P 8`
perf stat:
```text
sudo perf stat -p $(pidof oiopt) -e cycles,instructions,cache-misses,branches,branch-misses,context-switches,cpu-migrations -- sleep 33
Performance counter stats for process id '883435':
118,960,667,431 cpu_atom/cycles/ (0.05%)
131,934,369,110 cpu_core/cycles/ (99.95%)
100,530,466,140 cpu_atom/instructions/ # 0.85 insn per cycle (0.05%)
185,203,788,299 cpu_core/instructions/ # 1.40 insn per cycle (99.95%)
11,027,490 cpu_atom/cache-misses/ (0.05%)
2,123,369 cpu_core/cache-misses/ (99.95%)
19,641,945,774 cpu_atom/branches/ (0.05%)
36,245,438,057 cpu_core/branches/ (99.95%)
214,098,497 cpu_atom/branch-misses/ # 1.09% of all branches (0.05%)
179,848,095 cpu_core/branch-misses/ # 0.50% of all branches (99.95%)
2,308 context-switches
31 cpu-migrations
33.004555878 seconds time elapsed
```
system call check:
```bash
sudo timeout 30s strace -c -f -p $(pidof oiopt)
```
output:
```text
strace: Process 883435 attached with 4 threads
strace: Process 883438 detached
strace: Process 883437 detached
strace: Process 883436 detached
strace: Process 883435 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
57.80 14.590016 442121 33 epoll_wait
28.84 7.279883 4 1771146 sendto
13.33 3.363882 1 1771212 48 recvfrom
0.02 0.003843 61 62 44 futex
0.01 0.001947 12 159 epoll_ctl
0.00 0.000894 99 9 9 connect
0.00 0.000620 34 18 9 accept4
0.00 0.000503 14 34 timerfd_settime
0.00 0.000446 13 33 33 read
0.00 0.000271 15 18 ioctl
0.00 0.000189 21 9 write
0.00 0.000176 19 9 socket
0.00 0.000099 11 9 getsockopt
0.00 0.000079 4 18 shutdown
0.00 0.000049 2 18 close
------ ----------- ----------- --------- --------- ----------------
100.00 25.242897 7 3542787 143 total
```
## Further tests to explain why this huge
Changed 16KiB buffer to 64KiB, and named the binary to oiopt64
iperf3 throughput under `-P 8`, highest 54.1Gbits/sec, other threads are much higher than before(16KiB buffer)
perf stat:
```text
sudo perf stat -p $(pidof oiopt64) -e cycles,instructions,cache-misses,branches,branch-misses,context-switches,cpu-migrations -- sleep 33
Performance counter stats for process id '893123':
120,859,810,675 cpu_atom/cycles/ (0.15%)
134,735,934,329 cpu_core/cycles/ (99.85%)
79,946,979,880 cpu_atom/instructions/ # 0.66 insn per cycle (0.15%)
127,036,644,759 cpu_core/instructions/ # 0.94 insn per cycle (99.85%)
24,713,474 cpu_atom/cache-misses/ (0.15%)
9,604,449 cpu_core/cache-misses/ (99.85%)
15,584,074,530 cpu_atom/branches/ (0.15%)
24,796,180,117 cpu_core/branches/ (99.85%)
175,778,825 cpu_atom/branch-misses/ # 1.13% of all branches (0.15%)
135,067,353 cpu_core/branch-misses/ # 0.54% of all branches (99.85%)
1,519 context-switches
50 cpu-migrations
33.006529572 seconds time elapsed
```
system call check:
```bash
sudo timeout 30s strace -c -f -p $(pidof oiopt64)
```
output:
```text
strace: Process 893123 attached with 4 threads
strace: Process 893126 detached
strace: Process 893125 detached
strace: Process 893124 detached
strace: Process 893123 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
54.56 18.079500 463576 39 epoll_wait
27.91 9.249443 7 1294854 2 sendto
17.49 5.796927 4 1294919 51 recvfrom
0.01 0.003778 50 75 49 futex
0.01 0.002188 12 175 epoll_ctl
0.00 0.000747 83 9 9 connect
0.00 0.000714 17 40 timerfd_settime
0.00 0.000510 13 39 38 read
0.00 0.000452 25 18 9 accept4
0.00 0.000310 17 18 ioctl
0.00 0.000232 23 10 write
0.00 0.000200 22 9 socket
0.00 0.000183 20 9 getsockopt
0.00 0.000100 5 18 shutdown
0.00 0.000053 2 18 close
0.00 0.000020 20 1 mprotect
0.00 0.000015 15 1 sched_yield
0.00 0.000005 5 1 madvise
------ ----------- ----------- --------- --------- ----------------
100.00 33.135377 12 2590253 158 total
```
### Cleanup:
```bash
sudo ip netns del ns_client
sudo ip netns del ns_server
sudo ip link del veth_c
sudo ip link del veth_s
```

475
drafts/2026-01-25-ace-profiling-attorney-the-case-of-the-missing-gbits.md
View File

@@ -1,475 +0,0 @@
---
title: Ace Profiling Attorney - The Case of the Missing Gbits
categories: [Programming, Profiling]
tags: [Rust, kernel, networking]
---
> **Disclaimer:** This is not a language-war post. No “X vs Y”.
> This is a profiling detective story about my Rust TCP forwarder [`oi`](https://github.com/DaZuo0122/oxidinetd).
---
## 0) Prologue — The Courthouse Lobby
> **Me:** I wrote a Rust TCP port forwarder. It works. It forwards.
>
> **Inner Prosecutor (Phoenix voice):** *Hold it!* “Works” is not a metric. How fast?
>
> **Me:** Not fast enough under load.
>
> **Inner Prosecutor:** *Objection!* “Not fast enough” is an emotion. Bring evidence.
>
> **Me:** Fine. I’ll bring **perf**, **strace**, and a **flamegraph**.
>
> **Inner Prosecutor:** Good. This court accepts only facts.
## 1) The Crime Scene — Setup & Reproduction
**Me:** Single machine, Debian 13. No WAN noise, no tunnel bottlenecks.
**Inner Prosecutor:** *Hold it!* If it’s “single machine”, how do you avoid loopback cheating?
**Me:** Network namespaces + veth. Local, repeatable, closer to real networking.
### Environment
- Debian 13
- Kernel: `6.12.48+deb13-amd64`
- Runtime: `smol`
- Test topology: `ns_client → oi (root ns) → ns_server` via veth
### Reproduction commands
**Exhibit A: Start backend server in `ns_server`**
```bash
sudo ip netns exec ns_server iperf3 -s -p 9001
````
**Exhibit B: Run client in `ns_client` through forwarder**
```bash
sudo ip netns exec ns_client iperf3 -c 10.0.1.1 -p 9000 -t 30 -P 8
```
**Inner Prosecutor:** *Hold it!* Why `-P 8`?
**Me:** Because a forwarder can look fine in `-P 1` and fall apart when syscall pressure scales.
**Inner Prosecutor:** …Acceptable.
---
## 2) The Suspects — What Could Be Limiting Throughput?
**Me:** Four suspects.
1. **CPU bound** (pure compute wall)
2. **Kernel TCP stack bound** (send/recv path, skb, softirq, netfilter/conntrack)
3. **Syscall-rate wall** (too many `sendto/recvfrom` per byte)
4. **Runtime scheduling / contention** (wake storms, locks, futex)
**Inner Prosecutor:** *Objection!* That’s too broad. Narrow it down.
**Me:** That’s what the tools are for.
---
## 3) Evidence #1 — `perf stat` (The Macro View)
**Me:** First I ask: are we burning CPU, thrashing schedulers, or stalling on memory?
**Command:**
```bash
sudo perf stat -p $(pidof oi) -e \
cycles,instructions,cache-misses,branches,branch-misses,context-switches,cpu-migrations \
-- sleep 33
```
**What I’m looking for:**
* Huge `context-switches` → runtime thrash / lock contention
* Huge `cpu-migrations` → unstable scheduling
* Very low IPC + huge cache misses → memory stalls
* Otherwise: likely syscall/kernel path
Output:
```text
Performance counter stats for process id '209785':
113,810,599,893 cpu_atom/cycles/ (0.11%)
164,681,878,450 cpu_core/cycles/ (99.89%)
102,575,167,734 cpu_atom/instructions/ # 0.90 insn per cycle (0.11%)
237,094,207,911 cpu_core/instructions/ # 1.44 insn per cycle (99.89%)
33,093,338 cpu_atom/cache-misses/ (0.11%)
5,381,441 cpu_core/cache-misses/ (99.89%)
20,012,975,873 cpu_atom/branches/ (0.11%)
46,120,077,111 cpu_core/branches/ (99.89%)
211,767,555 cpu_atom/branch-misses/ # 1.06% of all branches (0.11%)
245,969,685 cpu_core/branch-misses/ # 0.53% of all branches (99.89%)
1,686 context-switches
150 cpu-migrations
33.004363800 seconds time elapsed
```
**Low context switching**:
- context-switches: 1,686 over ~33s → ~51 switches/sec
- cpu-migrations: 150 over ~33s → ~4.5/s → very stable CPU placement
**CPU is working hard**:
- 237,094,207,911 cpu_core instructions
- IPC: 1.44 (instructions per cycle) → not lock-bound or stalling badly
**Clean cache, branch metrics**:
- cache-misses: ~3.1M (tiny compared to the instruction count)
- branch-misses: 0.62%
**Inner Prosecutor:** *Hold it!* You didn’t show the numbers.
**Me:** Patience. The next exhibit makes the culprit confess.
---
## 4) Evidence #2 — `strace -c` (The Confession: Syscall Composition)
**Me:** Next: “What syscalls are we paying for?”
**Command:**
```bash
sudo timeout 30s strace -c -f -p $(pidof oi)
```
**What I expect if this is a forwarding wall:**
* `sendto` and `recvfrom` dominate calls
* call counts in the millions
Output (simplified):
```text
sendto 2,190,751 calls 4.146799s (57.6%)
recvfrom 2,190,763 calls 3.052340s (42.4%)
total syscall time: 7.200789s
```
(A) **100% syscall/copy dominated:**
- Almost all traced time is inside:
- sendto() (TCP send)
- recvfrom() (TCP recv)
(B) **syscall rate is massive**
- Total send+recv calls:
- ~4,381,500 syscalls in ~32s
- → ~137k `sendto` per sec + ~137k `recvfrom` per sec
- → ~274k syscalls/sec total
**Inner Prosecutor:** *Objection!* Syscalls alone don’t prove the bottleneck.
**Me:** True. So I brought a witness.
---
## 5) Evidence #3 — FlameGraph (The Witness)
**Me:** The flamegraph doesn’t lie. It testifies where cycles go.
**Commands:**
```bash
sudo perf record -F 199 --call-graph dwarf,16384 -p $(pidof oi) -- sleep 30
sudo perf script | stackcollapse-perf.pl | flamegraph.pl > oi.svg
```
**What the flamegraph showed (described, not embedded):**
* The widest towers were kernel TCP send/recv paths:
* `__x64_sys_sendto` → `tcp_sendmsg_locked` → `tcp_write_xmit` → ...
* `__x64_sys_recvfrom` → `tcp_recvmsg` → ...
* My userspace frames existed, but were comparatively thin.
* The call chain still pointed into my forwarding implementation.
**Inner Prosecutor:** *Hold it!* So you’re saying… the kernel is doing the heavy lifting?
**Me:** Exactly. Which means my job is to **stop annoying the kernel** with too many tiny operations.
---
## 6) The Real Culprit — A “Perfectly Reasonable” Copy Loop
**Me:** Here’s the original relay code. Looks clean, right?
```rust
let client_to_server = io::copy(client_stream.clone(), server_stream.clone());
let server_to_client = io::copy(server_stream, client_stream);
futures_lite::future::try_zip(client_to_server, server_to_client).await?;
```
**Inner Prosecutor:** *Objection!* This is idiomatic and correct.
**Me:** Yes. That’s why it’s dangerous.
**Key detail:** `futures_lite::io::copy` uses a small internal buffer (~8KiB in practice).
Small buffer → more iterations → more syscalls → more overhead.
If a forwarder is syscall-rate bound, this becomes a ceiling.
---
## 7) The First Breakthrough — Replace `io::copy` with `pump()`
**Me:** I wrote a manual pump loop:
* allocate a buffer once
* `read()` into it
* `write_all()` out
* on EOF: `shutdown(Write)` to propagate half-close
```rust
async fn pump(mut r: TcpStream, mut w: TcpStream, buf_sz: usize) -> io::Result<u64> {
let mut buf = vec![0u8; buf_sz];
let mut total = 0u64;
loop {
let n = r.read(&mut buf).await?;
if n == 0 {
let _ = w.shutdown(std::net::Shutdown::Write);
break;
}
w.write_all(&buf[..n]).await?;
total += n as u64;
}
Ok(total)
}
```
Run both directions:
```rust
let c2s = pump(client_stream.clone(), server_stream.clone(), BUF);
let s2c = pump(server_stream, client_stream, BUF);
try_zip(c2s, s2c).await?;
```
**Inner Prosecutor:** *Hold it!* That’s just a loop. How does that win?
**Me:** Not the loop. The **bytes per syscall**.
---
### 8) Exhibit C — The Numbers (8KiB → 16KiB → 64KiB)
### Baseline: ~8KiB (generic copy helper)
Throughput:
```text
17.8 Gbit/s
```
**Inner Prosecutor:** *Objection!* That’s your “crime scene” number?
**Me:** Yes. Now watch what happens when the kernel stops getting spammed.
### Pump + 16KiB buffer
Throughput:
```text
28.6 Gbit/s
```
`strace -c` showed `sendto/recvfrom` call count dropped:
```text
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
57.80 14.590016 442121 33 epoll_wait
28.84 7.279883 4 1771146 sendto
13.33 3.363882 1 1771212 48 recvfrom
0.02 0.003843 61 62 44 futex
0.01 0.001947 12 159 epoll_ctl
...
------ ----------- ----------- --------- --------- ----------------
100.00 25.242897 7 3542787 143 total
```
**Inner Prosecutor:** *Hold it!* That’s already big. But you claim there’s more?
**Me:** Oh, there’s more.
### Pump + 64KiB buffer
Throughput:
```text
54.1 Gbit/s (best observed)
```
`perf stat` output:
```text
Performance counter stats for process id '893123':
120,859,810,675 cpu_atom/cycles/ (0.15%)
134,735,934,329 cpu_core/cycles/ (99.85%)
79,946,979,880 cpu_atom/instructions/ # 0.66 insn per cycle (0.15%)
127,036,644,759 cpu_core/instructions/ # 0.94 insn per cycle (99.85%)
24,713,474 cpu_atom/cache-misses/ (0.15%)
9,604,449 cpu_core/cache-misses/ (99.85%)
15,584,074,530 cpu_atom/branches/ (0.15%)
24,796,180,117 cpu_core/branches/ (99.85%)
175,778,825 cpu_atom/branch-misses/ # 1.13% of all branches (0.15%)
135,067,353 cpu_core/branch-misses/ # 0.54% of all branches (99.85%)
1,519 context-switches
50 cpu-migrations
33.006529572 seconds time elapsed
```
`strace -c` output:
```text
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
54.56 18.079500 463576 39 epoll_wait
27.91 9.249443 7 1294854 2 sendto
17.49 5.796927 4 1294919 51 recvfrom
...
------ ----------- ----------- --------- --------- ----------------
100.00 33.135377 12 2590253 158 total
```
**Inner Prosecutor:** *OBJECTION!* `epoll_wait` is eating the time. That’s the bottleneck!
**Me:** Nice try. That’s a classic trap.
---
## 9) Cross-Examination — The `epoll_wait` Trap
**Me:** `strace -c` measures time spent *inside syscalls*, including time spent **blocked**.
In async runtimes:
* One thread can sit in `epoll_wait(timeout=...)`
* Other threads do `sendto/recvfrom`
* `strace` charges the blocking time to `epoll_wait`
So `epoll_wait` dominating **does not** mean “epoll is slow”.
It often means “one thread is waiting while others work”.
**What matters here:**
* `sendto` / `recvfrom` call counts
* and how they change with buffer size
---
## 10) Final Explanation — Why 64KiB Causes a “Nonlinear” Jump
**Inner Prosecutor:** *Hold it!* You only reduced syscall calls by ~some percent. How do you nearly triple throughput?
**Me:** Because syscall walls are **nonlinear**.
A forwarder’s throughput is approximately:
> **Throughput ≈ bytes_per_syscall_pair × syscall_pairs_per_second**
If you’re syscall-rate limited, increasing `bytes_per_syscall_pair` pushes you past a threshold where:
* socket buffers stay fuller
* the TCP window is better utilized
* each stream spends less time in per-chunk bookkeeping
* concurrency (`-P 8`) stops fighting overhead and starts helping
Once you cross that threshold, throughput can jump until the next ceiling (kernel TCP, memory bandwidth, iperf itself).
That’s why a “small” change can create a big effect.
---
## 11. Trade-offs: buffer size is not free
**Inner Prosecutor:** *Objection!* Bigger buffers waste memory!
**Me:** Sustained.
A forwarder allocates **two buffers per connection** (one per direction).
So for 64KiB:
* ~128KiB per connection (just for relay buffers)
* plus runtime + socket buffers
That’s fine for “few heavy streams”, but it matters if you handle thousands of concurrent connections.
In practice, the right move is:
* choose a good default (64KiB is common)
* make it configurable
* consider buffer pooling if connection churn is heavy
---
## Epilogue — Case Closed (for now)
**Inner Prosecutor:** So the culprit was…
**Me:** A perfectly reasonable helper with a default buffer size I didn’t question.
**Inner Prosecutor:** And the lesson?
**Me:** Don’t guess. Ask sharp questions. Use the tools. Let the system testify.
> **Verdict:** Guilty of “too many syscalls per byte.”
>
> **Sentence:** 64KiB buffers and a better relay loop.
---
## Ending
This was a good reminder that performance work is not guessing — it’s a dialogue with the system:
1. Describe the situation
2. Ask sharp questions
3. Use tools to confirm
4. Explain the results using low-level knowledge
5. Make one change
6. Re-measure
And the funniest part: the “clean” one-liner `io::copy` was correct, but its defaults were hiding a performance policy I didn’t want.
> **Inner Prosecutor:** “Case closed?”
>
> **Me:** “For now. Next case: buffer pooling, socket buffer tuning, and maybe a Linux-only `splice(2)` fast path — carefully, behind a safe wrapper.”
---

350
drafts/2026-03-03-building-babel-a-fuzzy-llm-vs-the-os.md
View File

@@ -1,350 +0,0 @@
---
title: Building Babel - a fuzzy LLM vs the OS
categories: [Thoughts]
tags: [os]
math: true
---
*A post about reliability, memory, and the compiler we didn’t mean to write.*
When people talk about “prompt engineering,” it often sounds like a bag of tricks: write clearer instructions, add examples, constrain the format, keep history short, and pray. But if you zoom out, the pattern looks less like copywriting and more like systems engineering. We’re trying to run workloads on a machine whose “CPU” is probabilistic, whose “RAM” is fixed-size, and whose caching behavior depends on keeping the same prefix intact.
That framing is useful, because it pushes us toward familiar tools: define an ISA, build a runtime, control errors with redundancy and verification, and manage memory with explicit lifetimes.
---
## 0 — The foundation: why the OS analogy is structurally correct
### 0.1 A minimal machine model
If you strip away the marketing, an LLM session is a constrained compute device with:
* a **finite internal working set** (context window),
* a **state-dependent transition function** (next-token generation),
* and an **acceleration cache** tied to previously seen prefixes (KV cache).
That is already enough to justify an OS framing, because OS thinking starts from exactly these questions:
* What is my *working set limit*?
* What is *resident* vs *non-resident* state?
* Where do failures originate: computation, I/O, or state corruption?
* How do I bound resource usage and isolate side effects?
So we don’t start with “LLM = OS.”
We start with “LLM session = state machine with bounded memory + caching,” and OS is the natural language we use to design such machines.
### 0.2 Formal mapping: LLM as a bounded-state transducer
A practical theoretical model looks like this:
* Let $X$ be the set of possible contexts (token sequences) with max length $N$.
* Let $Y$ be token outputs.
* The model implements a stochastic policy:
$$
\pi(y \mid x)
$$
where $x \in X$.
In each interaction, you append some new tokens to $x$, then the model emits tokens $y$, producing a new context $x' = \text{append}(x, y)$, then truncation/packing happens due to the context limit $N$.
From an OS perspective, the key point is not stochasticity. The key point is **boundedness**:
* The model is not a Turing machine with unlimited tape.
* It is a finite-memory device (finite context window) with expensive state transitions.
That’s why memory management dominates in practice.
### 0.3 “Main context ⇒ RAM” : the working-set equivalence
In OS terms, RAM is defined by three properties:
1. **bounded capacity**
2. **fast access for computation**
3. **content determines behavior** (program correctness depends on which pages are resident and what they contain)
An LLM context window has exactly those properties:
* bounded capacity: fixed token limit $N$
* fast access: everything in-context is “directly addressable” by attention
* content determines behavior: the probability distribution $\pi(\cdot\mid x)$ changes when $x$ changes
That’s enough to justify the equivalence “context behaves like RAM,” even though the representation isn’t bytes.
If you want a stronger OS-flavored claim: the LLM’s output is effectively a function of its **resident working set**. If important facts fall out of the window, the model behaves as if the memory page was evicted.
### 0.4 “Immutable prefix ⇒ kernel text” is about invariants and reentrancy
Kernel code in an OS is special for reasons that map cleanly:
* it establishes invariants (security model, syscalls, scheduling policy)
* it is expected to be stable/reentrant across workloads
* it is always “mapped” (logically present) and influences everything
Similarly, the immutable prefix (role, policies, high-level goals, tool contracts) is the part of context we want:
* stable across turns,
* consistently applied to all tasks,
* and reusable for performance.
This isn’t metaphorical; it’s a design constraint. If you mutate “kernel invariants” mid-execution, you get undefined behavior in both worlds.
### 0.5 “KV cache ⇒ TLB/cache” is justified by prefix-dependent reuse
KV cache reuse is not magic; it’s a computational caching mechanism:
* If the prefix is identical, the model can reuse previously computed key/value tensors for those tokens.
* If the prefix changes, cache reuse degrades.
That is the same structural property as a TLB/cache: stable mappings/prefixes produce high hit rates; churn kills locality.
From an OS dev perspective, this creates an optimization target: keep the “kernel prefix” stable to maximize cache locality across turns.
### 0.6 “Skills compiler” is justified by separation of concerns
OS devs separate *policy* from *mechanism*:
* policy decides what to do (scheduler policy, VM policy)
* mechanism executes it deterministically (context switch, page fault handler)
This is exactly what the compiler/runtime split is doing:
* **LLM is policy**: produce a plan from a messy spec
* **Sandbox runtime is mechanism**: execute primitives deterministically
So “NL → instructions compiler” isn’t just a nice idea; it’s literally applying a core OS design principle: isolate fuzzy/high-level decisions from low-level mechanisms, so failures can be contained and reasoned about.
### 0.7 Why binaries (success/failure) matter: observability and debugging
OS engineering relies on observability:
* syscalls return error codes
* page faults are explicit
* sched events are traceable
* invariants can be asserted
When you compile skills into basic instructions with binary status, you create **syscall-like boundaries**. That gives you:
* localized failure (which instruction failed)
* structured error types (timeout vs empty input vs permission)
* the ability to retry/repair minimally
This is the real foundation of the “ECC/control loop” later: error correction requires a detectable syndrome. Binary instruction results are your syndrome.
---
## Part 1 — Reliability: the fuzzy ALU problem, and an ECC-shaped solution
### The reliability issue: deterministic execution vs probabilistic compilation
In a classic machine, the critical property is that execution is deterministic: given the same instruction stream and machine state, you get the same result. That’s what makes debugging possible, and it’s why “bit flips” are an exceptional event handled by ECC, parity checks, and redundancy.
LLMs invert that. The core model is best understood as a conditional distribution $P(y \mid x)$: the next token depends on the prompt/context. Even if you force deterministic decoding, the *system-level behavior* remains fragile because the mapping from a messy human request to an internal strategy is not explicit and not stable. Small context changes, minor phrasing differences, or irrelevant baggage in the prompt can flip the “mode” the model enters. In practice, this looks like the ALU occasionally returning the wrong result, except the “wrongness” is semantic, not bit-level.
A direct way to improve reliability is to reduce the amount of “semantic work” the model must do while it is producing final outputs. Instead of asking the LLM to execute tasks in free-form language, we ask it to **compile** the request into a small set of **deterministic primitives**. Then we run those primitives in a runtime we control.
### The solution: compile to a tiny ISA (`read/write/grep/exec`) and execute in a strict sandbox
Assume we define a minimal instruction set:
* `read` — load data from a declared source
* `write` — store a value into a declared state slot
* `grep` — deterministically match/filter/extract
* `exec` — call a deterministic tool (curl, a parser, a local command) inside a sandbox
Now, a “skill” is no longer a blob of natural language. It becomes a program: a sequence of these instructions. Web search becomes “write the query → exec curl → read the response → grep for indicators → write the extracted results.” Local doc search becomes “read the corpus → grep for pattern → write the match list.”
You can visualize the pipeline like a compiler+runtime split:
```
User request (natural language)
|
v
+-------------------+
| Skills Compiler | (LLM: NL -> ISA plan)
+-------------------+
|
v
ISA Program: [write, exec, read, grep, ...]
|
v
+-------------------+
| Sandbox Runtime | (deterministic executor + logs)
+-------------------+
|
v
Structured result + trace (success/failure per instruction)
```
This architecture deliberately moves uncertainty into one place: compilation. Execution becomes observable and mostly deterministic.
### Why this increases reliability
Let $U$ be the user request (the “spec”), $C$ be the compiler (LLM), $P$ the produced plan (ISA program), $R$ the runtime, and $O$ the observed output. Let $V(U,O)\in{0,1}$ be a checker that says whether the output satisfies the request (even a weak checker helps).
Because the runtime is deterministic and instrumented, the overall success probability decomposes conceptually into:
* how often the compiler emits a correct plan, and
* how often correct plans actually pass in the runtime environment.
Informally:
$$
\Pr[V=1] \approx \Pr[P\ \text{correct}] \cdot \Pr[V=1\mid P\ \text{correct}]
$$
If your runtime is strict and your primitives are deterministic, $\Pr[V=1\mid P\ \text{correct}]$ is high. That’s the central win: you turn “LLM unpredictability everywhere” into “LLM uncertainty mainly at compilation time.” Once the failure surface is concentrated, you can apply ECC-like techniques there.
### ECC for compilation: redundancy + decoding
ECC works by adding redundancy and then decoding based on constraints that detect and correct errors. You can do the same for plans:
1. generate multiple candidate plans $P_1, …, P_k$
2. statically validate them (types, allowed effects, resource access)
3. partially execute cheap prefixes if needed
4. select the plan that passes checks / yields valid outputs
Even without strong independence assumptions, plan diversity (different “compilation prompts,” constraints, or decomposition templates) tends to decorrelate failure modes. With a checker, the selection step becomes the decoder.
Here’s an OS-style view of this redundancy:
```
+------------------+
U ----------> | Compiler (LLM) |
+------------------+
| | |
P1 P2 P3 ... (redundant plans)
\ | /
\ | /
v v v
+-------------------+
| Verifier/Decoder |
+-------------------+
|
v
choose P*
|
v
+----------------+
| Runtime Exec |
+----------------+
```
### Control theory as practical “ECC glue”: localize failures, repair minimally
Once every step is an instruction with a success/failure status, you gain something that free-form prompting almost never gives you: precise failure localization.
* `exec curl` failed → network/tool failure; retry/backoff or switch endpoint
* `grep` failed → query too narrow or wrong target; rewrite only the grep or the query
* `read` failed → missing resource; fix source selection
* `write` failed → schema/invariant violation; repair the state shape
This is where a control loop becomes natural. The “plant” is your plan+runtime; the “sensor” is the instruction trace; the “controller” decides whether to retry, patch the plan, or escalate. You don’t need to force PID math onto it, but the conceptual mapping is clean: proportional fixes handle immediate errors, integral memory of repeated errors adjusts defaults, and derivative-like logic prevents oscillations (e.g., endless retries).
---
## Part 2 — Memory: context is RAM, and we need a real allocator
### The memory issue: fixed context window behaves like physical memory
If the LLM is single-threaded, the context window is its working memory. It’s fixed size. It is expensive. It gets polluted. And when it overflows, you start doing swap-like hacks: aggressive summarization, pruning, or retrieval-augmented generation (RAG).
The OS analogy isn’t decorative here. It’s operationally accurate:
* **context window** is physical RAM (bounded)
* **immutable prefix** is kernel code mapped and hot
* **KV cache** behaves like a cache/TLB where prefix stability matters
* “adding more prompt” is “allocating memory,” and you can run out
So memory management isn’t just “write less.” It’s deciding what stays resident, what gets offloaded, and how to keep the working set small.
### The solution: arena allocator + staged lifetimes + lazy loading + offload
Rust’s arena allocator is a good mental model because it matches a core workflow truth: most intermediate data is useful only during a phase. You don’t need fine-grained frees; you need cheap allocation and bulk reclamation at phase boundaries.
We treat the context as an append-only arena with a bump pointer. We also define “stages” in a skill:
1. compile plan
2. execute primitives
3. validate
4. commit minimal results
5. free intermediates
At each stage start, we take a checkpoint. During the stage, we append logs and intermediate outputs. When the stage completes, we write a compact “commit record” (summary + hashes + pointers), then reset the arena to the checkpoint. That gives you explicit lifetimes and prevents context leaks.
Here is the memory layout you proposed, as an OS-style diagram:
```
+----------------------------------------------------------------------------------+
| LLM Context Window (fixed) |
+----------------------------------------------------------------------------------+
| Immutable Prefix | Disk Index (pointers) | Working Set (stage arena) | Free |
| (kernel text) | (skills/tools/docs) | (plan + outputs + trace) | |
+----------------------------------------------------------------------------------+
```
And here is the arena checkpoint mechanism:
```
Stage S begins:
bump_ptr = B
checkpoint[S] = B
During stage:
append(plan fragments)
append(exec outputs)
append(grep matches)
append(validation notes)
Stage S commits:
write(commit_record[S]) <-- compact, structured
bump_ptr = checkpoint[S] <-- bulk free (arena reset)
```
### Why this works
This approach gives two strong properties that are “theoretical” in the systems sense.
**(1) Bounded working set by construction.**
If you cap the amount of intermediate data a stage is allowed to keep (or the number/size of instruction traces), then your memory usage becomes bounded not by “how long the conversation is,” but by “how big the current stage is.” That’s exactly how a well-structured kernel avoids unbounded growth: scope lifetimes, control allocations, and reclaim aggressively when a phase ends.
**(2) Reduced need for lossy compaction.**
Most prompt-based “memory management” relies on summarization, which is effectively lossy compression. An arena+stages approach keeps the majority of the “bulk data” as structured runtime artifacts (commit records, hashes, pointers), not as raw narrative. That reduces how often you must compress meaning into fewer tokens. Prune and RAG become swap mechanisms rather than the default mode.
### Lazy loading: the disk index is your page table
The “disk index” part is important. It’s the page table of your LLM OS: it stores *addresses* (skill names, tool contracts, doc IDs), not the content itself. You keep this index resident because it’s small and highly reusable. When a plan needs a resource, it pages it in via deterministic steps:
* `exec(fetch_doc, id)`
* `read(output)`
* `grep(...)`
This is consistent with your “treat prune and RAG as last option” stance. The system shouldn’t drag content into RAM unless it’s demanded by the current working set.
### Prune & RAG: swap/compaction when you must, not when you feel anxious
Eventually you will hit memory pressure. When you do, you have two OS-like fallback moves:
* **pruning**: compact resident memory by removing older low-value regions
* **RAG/swap**: offload bulk to external storage and page it back when needed
The key is policy: these are last-resort mechanisms, triggered by pressure and guided by working-set logic, not something you do on every turn.
---
## Are we building the Tower of Babel?
In mood terms, yes: we are building a translation layer from human intent to machine action, and history says that “universal translators” are where complexity goes to multiply.
But there’s a difference between *Babel as confusion* and *Babel as a compiler toolchain*. The dangerous version is when natural language directly triggers actions without strict interfaces. That system becomes un-debuggable and hard to secure because the space of meanings is unbounded and the boundary between interpretation and execution is blurry.
The OS-shaped version is constrained:
* the target language is not “whatever the model feels like,” it’s a tiny ISA
* the runtime is deterministic and instrumented
* the boundary between compilation (fuzzy) and execution (controlled) is explicit
* failures are localized to instruction traces
* memory is managed with lifetimes and checkpoints, not hope
So yes, we’re building Babel—but the grown-up, systems version. Not a tower of endless languages, but a tower with a syscall table, an allocator, and something that looks suspiciously like ECC wrapped around the only fuzzy component: compilation.

59
zed-config/settings.json
View File

@@ -1,59 +0,0 @@
{
"telemetry": {
"metrics": false,
"diagnostics": false
},
"auto_update": false,
"ui_font_size": 16,
"buffer_font_size": 15,
"base_keymap": "JetBrains",
"buffer_font_family": "JetBrains Mono",
"theme": {
"mode": "dark",
"light": "One Light",
"dark": "One Dark"
},
"languages": {
"Python": {
"tab_size": 4
}
},
"inlay_hints": {
"enabled": true
},
"lsp": {
"rust-analyzer": {
"binary": {
"ignore_system_version": true
},
"initialization_options": {
"inlayHints": {
"maxLength": null,
"lifetimeElisionHints": {
"enable": "skip_trivial",
"useParameterNames": true
},
"closureReturnTypeHints": {
"enable": "always"
},
"closureCaptureHints": {
"enable": true
}
}
}
},
"hls": {
"initialization_options": {
"haskell": {
"formattingProvider": "fourmolu"
}
}
}
},
"disable_ai": true,
"icon_theme": {
"mode": "dark",
"light": "Zed (Default)",
"dark": "Zed (Default)"
}
}