What it is, how it works under the hood, and when plain REST is still the better choice

Most services today talk to each other with REST and JSON.

It works. It is easy to read. You can test it with one curl command.

But as a system grows, small problems pile up.

One team renames a field. Another team’s client still sends the old name.

Every language needs its own hand-written client.

Nobody is fully sure what the API accepts until something breaks.

gRPC is one answer to that mess.

It is an open source RPC framework that started at Google.

RPC means “remote procedure call.” In plain words: you call a function on another machine.

Let’s walk through how it works, one piece at a time.

Step 1: Write the contract once

With gRPC, you start by describing your API in a .proto file.

It lists the methods, what each one takes in, and what it sends back.

Here is the small example from the gRPC docs:

service HelloService {  rpc SayHello (HelloRequest) returns (HelloResponse);}
message HelloRequest {
  string greeting = 1;
}message HelloResponse {  string reply = 1;}

That file is the single source of truth.

Then a compiler reads it and generates code for you.

On the server, you get an interface to fill in with your logic.

On the client, you get a “stub.” A stub is a local object with the same methods as the remote service.

So calling another service looks like this: client.SayHello(request).

The generated code packs the request, sends it over the network, and unpacks the reply.

The client can be in Go and the server in Java. The contract keeps them in sync.

Step 2: Protocol Buffers, the message format

By default, gRPC sends data as Protocol Buffers, often called protobuf.

Protobuf is a way to turn structured data into bytes.

Google describes it as like XML, but smaller, faster, and simpler.

The key idea is the number next to each field.

On the wire, a field is sent by its number, not its name.

That is why the format is compact.

It is also why you can change your API safely, if you follow two rules.

Rule one: adding a new field is safe. Old code simply ignores fields it does not know.

Rule two: never reuse a field number. If you delete a field, mark its number as “reserved” so nobody uses it again.

The protobuf docs are blunt about this.

Reusing a number can lead to parse errors, corrupted data, or even leaked personal data.

A small habit from the same docs: fields 1 to 15 take one byte to tag, so give those numbers to your most used fields.

Step 3: HTTP/2 underneath

gRPC runs on top of HTTP/2.

HTTP/2 is the newer version of the protocol the web runs on.

Two features matter most here.

First, many calls can share one connection at the same time. This is called multiplexing.

So a service that makes hundreds of calls does not need hundreds of connections.

Second, headers are compressed.

Repeated headers like auth tokens cost less on every call.

Step 4: Four kinds of calls

REST is mostly one request, one response.

gRPC gives you four shapes.

Unary: one request, one response. Just like a normal function call.

Server streaming: one request, many responses. Think of a live feed of order updates.

Client streaming: many requests, one response. Think of uploading many log lines and getting one summary back.

Bidirectional streaming: both sides send whenever they want. Think of a chat.

The docs say gRPC keeps messages in order within a single call.

Step 5: Always set a deadline

This is the part teams most often skip.

A deadline is the point in time after which the client stops waiting.

Here is the surprising default.

The gRPC docs say that by default, no deadline is set at all.

That means a client can wait, in their words, “effectively forever.”

So the advice is simple: always set a realistic deadline on every call.

When time runs out, the call fails with a status called DEADLINE_EXCEEDED.

The server side also cancels the call, so it stops wasting work.

It gets better when one service calls another.

The docs give an example. A client gives a request 2 seconds.

The user service spends 0.5 seconds, then calls the billing service.

gRPC can pass the remaining 1.5 seconds along to billing.

It sends the time left, not a clock time, so clocks that disagree do not matter.

Some languages do this by default, like Java and Go. In C++ you have to turn it on.

One warning from the docs: cancelling a call does not undo changes already made.

Step 6: Errors with standard codes

In REST, every team invents its own error shapes.

gRPC has a fixed list of status codes.

A few you will see often: OK, INVALID_ARGUMENT, NOT_FOUND, DEADLINE_EXCEEDED, and UNAVAILABLE.

Because the list is shared, retry logic and dashboards can work the same way across every service.

The pushback, stated fairly

gRPC has real critics. Kevin McDonald wrote a well known post called “gRPC: The Bad Parts,” and it drew over 200 comments on Hacker News.

Here are the main points, in plain words.

It is harder to try out. You cannot just send a friend a curl command.

It adds a build step. Code generation is one more thing to set up, and it feels heavy in dynamic languages.

Browsers cannot speak plain gRPC. They need gRPC-Web, which often means running a proxy.

Payloads are binary. You cannot read them in logs without extra tools.

One top Hacker News comment argued the ecosystem is complex enough that the best supported languages are the ones Google itself maintains.

Another said most teams do not need gRPC at all, and end up with extra complexity and few real gains.

To be fair, McDonald also wrote a follow up on the good parts.

He points out that some of his complaints have since improved, like the standard way to map protobuf to JSON.

So when should you use it?

Here is a simple rule that holds up well.

Use gRPC between your own services, where you control both sides.

There, the shared contract, compact messages, streaming, and deadlines really pay off.

Use REST or GraphQL at the public edge, where browsers and outside developers call you.

There, being easy to read and easy to try matters more than saving bytes.

And whichever you pick, steal one idea from gRPC: put a deadline on every call.

What does your team use between services today, and what made you choose it?

Sources

• gRPC, Introduction: https://grpc.io/docs/what-is-grpc/introduction/
• gRPC, Core concepts, architecture and lifecycle: https://grpc.io/docs/what-is-grpc/core-concepts/
• gRPC, Deadlines: https://grpc.io/docs/guides/deadlines/
• gRPC, Status codes: https://grpc.io/docs/guides/status-codes/
• Protocol Buffers overview: https://protobuf.dev/overview/
• Protocol Buffers, Language Guide (proto3): https://protobuf.dev/programming-guides/proto3/
• RFC 9113, HTTP/2: https://www.rfc-editor.org/rfc/rfc9113
• gRPC-Web: https://github.com/grpc/grpc-web
• Kevin McDonald, “gRPC: The Bad Parts”: https://kmcd.dev/posts/grpc-the-bad-parts/
• Hacker News discussion: https://news.ycombinator.com/item?id=40798740
• Kevin McDonald, “gRPC: The Good Parts”: https://kmcd.dev/posts/grpc-the-good-parts/

You click save.

A row goes into a table.

It feels like one step. Inside the database, it is several.

This piece follows that one write all the way down, using PostgreSQL as the main example, with RocksDB for contrast. Everything here comes from the official docs, linked at the end.

Step 1: Data lives in pages, not rows

A database does not keep your table as one long list of rows.

It splits it into pages. A page is a fixed-size block of bytes.

In Postgres, every table and every index is stored as an array of pages, usually 8 KB each.

Each page has a small header at the start, 24 bytes.

Then a list of item pointers. Each one says where a row sits inside the page.

The rows themselves are packed in from the end of the page, growing backward.

Free space sits in the middle until the two sides meet.

Why does this matter? Because the database reads and writes whole pages. Even if you want one row, it fetches the page it lives on.

Step 2: The hot pages stay in memory

Reading from disk is slow. Reading from memory is fast.

So the database keeps a pool of pages in memory. Postgres calls this shared buffers. The general name is the buffer pool.

Your new row is written into a page in that pool first, not straight to the disk file.

The default size in Postgres is typically 128 MB. The docs suggest about 25% of RAM as a starting point on a dedicated server with 1 GB or more.

They also say more than 40% is unlikely to help, because Postgres also relies on the operating system’s own cache.

A page that has been changed in memory but not yet saved to disk is called dirty.

A background writer process saves dirty pages to disk over time.

Step 3: Write it to the log first

Here is the obvious problem. If the change is only in memory and the power goes out, it is gone.

The fix is the write-ahead log, or WAL.

The rule is simple. Before a change goes into the data files, a record describing that change must be written to the log and flushed to permanent storage.

So when you commit, Postgres only has to make sure the log is safely on disk. It does not have to flush every data page you touched.

If the server crashes, it replays the log and redoes any changes that never made it into the data pages.

This is also faster, not just safer.

The log is written in order, one record after another. The docs point out that syncing one sequential file costs much less than flushing many scattered pages.

And when many small transactions run at once, one flush of the log can commit many of them together.

The same log powers backups too. Keep the old WAL files, and you can restore a backup and replay up to any point in time.

Step 4: The index has to know about the new row

A table with no index is just a pile of pages. To find one row, the database may have to read all of them. That is called a sequential scan.

An index fixes that. The most common kind is the B-tree.

A B-tree keeps keys in sorted order, in a tree of pages.

The top pages point to lower pages. The bottom pages, called leaf pages, point to the actual rows. Postgres says typically over 99% of the pages in a B-tree are leaf pages.

To find a key, the database walks from the top down. That is a handful of page reads, not the whole table.

When a leaf page is full, it splits into two. That split can push a new pointer into the parent, and the parent can split too, all the way up.

The other design: LSM trees

Not every database works this way. RocksDB uses a different structure, the log-structured merge tree, or LSM tree.

New writes go into an in-memory table called a memtable, and optionally into a write-ahead log.

When the memtable fills up, it is written to disk as a sorted file. Those files are never changed in place.

Later, a background job called compaction merges files together and throws away old or deleted values.

The plain trade-off: writes are cheap because they are batched in memory. Reads can be costlier because a key may be in several files, which is why RocksDB uses tricks like Bloom filters to skip files that cannot hold the key.

A rough rule many engineers use: read-heavy work leans toward B-trees, write-heavy work leans toward LSM trees. Real systems tune both.

Step 5: Other people are reading at the same time

While your write happens, someone else may be reading the same table.

Postgres handles this with MVCC, short for multiversion concurrency control.

Each statement sees a snapshot of the data as it was at a point in time.

So readers never block writers, and writers never block readers.

The catch is that an UPDATE or DELETE does not remove the old row right away. Another transaction might still need to see it.

The old versions pile up as dead rows.

VACUUM is the cleanup job. It marks that space as free for new rows. The autovacuum daemon runs it for you, and it is on by default.

Step 6: Reading it back

Later, someone runs a query to find your row.

The query planner decides how. Use the index? Scan the whole table? Which join method?

It picks the plan with the lowest estimated cost, based on statistics about your data. The ANALYZE command collects those statistics from a random sample of rows.

You can see its choice with EXPLAIN. EXPLAIN ANALYZE actually runs the query and shows the real row counts and times next to the estimates.

The docs say the most important thing to check is whether the estimated row counts are close to reality.

The honest catches

None of this is free. Each catch below comes straight from the docs.

An index speeds up reads but adds work to writes. Postgres notes that an UPDATE which cannot use its HOT shortcut needs a new entry in every index on the table, even indexes on columns that did not change.

The planner can skip your index on purpose. On a tiny table it will nearly always choose a sequential scan, because one page read is cheaper than walking an index.

Results from a toy table do not carry over to a big one. The docs warn that plan costs are not linear.

Plain VACUUM does not give space back to the operating system in most cases. It only makes the space reusable. VACUUM FULL shrinks the table but locks it the whole time.

Long-running transactions hold cleanup back. Old row versions must stay while some transaction might still see them.

And vacuum is not optional. Postgres transaction IDs are 32 bits. Every table must be vacuumed at least once every two billion transactions, or old data can suddenly look like it is from the future.

EXPLAIN ANALYZE really runs the statement. If it is an UPDATE or DELETE, the change happens unless you wrap it in a transaction and roll it back.

LSM trees pay their bill later. RocksDB says overall write speed depends on how fast compaction can keep up, and a burst of writes can stall if the flush threads are busy.

What this changes day to day

A missing index on a busy column means the database reads pages it did not need to.

A huge transaction keeps old row versions alive and makes cleanup wait.

“Just add an index” is a trade, not a free win. Every index makes writes a bit heavier.

When a query is slow, run EXPLAIN ANALYZE before guessing. Compare the estimated rows to the actual rows.

And let autovacuum do its job. Turning it off is a decision you will pay for later.

Which of these steps has bitten you hardest in production: indexes, locks, or cleanup?

Sources

• PostgreSQL, Database Page Layout: https://www.postgresql.org/docs/current/storage-page-layout.html
• PostgreSQL, Resource Consumption (shared_buffers, background writer): https://www.postgresql.org/docs/current/runtime-config-resource.html
• PostgreSQL, Write-Ahead Logging (WAL): https://www.postgresql.org/docs/current/wal-intro.html
• PostgreSQL, B-Tree Indexes: https://www.postgresql.org/docs/current/btree.html
• PostgreSQL, Concurrency Control introduction (MVCC): https://www.postgresql.org/docs/current/mvcc-intro.html
• PostgreSQL, Routine Vacuuming: https://www.postgresql.org/docs/current/routine-vacuuming.html
• PostgreSQL, Using EXPLAIN: https://www.postgresql.org/docs/current/using-explain.html
• RocksDB Wiki, RocksDB Overview: https://github.com/facebook/rocksdb/wiki/RocksDB-Overview

Almost every system you use today has a load balancer in front of it.

Most engineers know the word.

Fewer can say what it actually decides, and why the choice matters.

This piece is the plain version. Everything here comes from the NGINX, AWS, Envoy, and Cloudflare docs and Google’s Maglev paper, linked at the end.

The one-line job

A load balancer has two jobs.

First, spread requests so no single server drowns.

Second, stop sending traffic to a server that is down.

Cloudflare compares it to a grocery store. One open checkout line means a long wait. Open eight lines and the wait drops a lot. The load balancer is the person telling each shopper which line to join.

The two big families: Layer 4 and Layer 7

The “layers” come from the OSI model. That is just a standard way to describe the levels of a network, from wires at the bottom to apps at the top.

A Layer 4 load balancer works at the transport level.

It sees IP addresses and ports.

It does not read the request itself.

That makes it fast and simple. AWS says its Network Load Balancer, a Layer 4 product, can handle millions of requests per second.

For TCP, it picks a server using a flow hash. A hash is a function that turns some input into a number. Here the input is the protocol, the source and destination IPs and ports, and the TCP sequence number.

Each TCP connection then stays on one server for its whole life.

A Layer 7 load balancer works at the application level.

It reads the HTTP request: the path, the headers, the cookies, the host name.

So it can make smarter choices.

AWS’s Application Load Balancer, a Layer 7 product, can send /api to one group of servers and /images to another. It can route by host name, by header, by query parameter, or by HTTP method.

The trade is simple.

Layer 4 is faster and knows less.

Layer 7 knows more and does more work per request.

The algorithms, in plain words

Once the load balancer has a list of healthy servers, it needs a rule for picking one.

Round robin. Go down the list in order, then start again. It is the default in NGINX and in AWS’s Application Load Balancer.

Weighted round robin. Same idea, but some servers get more turns. In the NGINX docs example, one server has weight 3 and two others have the default. Out of every 5 requests, the first gets 3 and the others get 1 each.

Least connections. Send the new request to the server with the fewest active connections. NGINX says this is fairer when some requests take much longer than others.

Power of two choices. Pick two servers at random and send the request to the less busy one. Envoy uses this for its least request policy. Its docs say research by Mitzenmacher and others shows it is nearly as good as checking every server. It also avoids herding, which is when every load balancer rushes to the same “least busy” server at once.

IP hash and sticky sessions. Hash the client’s IP address, so the same client keeps landing on the same server. NGINX says this holds unless that server becomes unavailable. “Sticky” just means the client sticks to one server.

Consistent hashing. This one matters when servers come and go.

With a plain hash, adding one server can reshuffle almost every client.

Consistent hashing places servers on an imaginary ring. Each request goes to the nearest server on the ring.

Envoy’s docs say that with this setup, adding or removing one server out of N moves only about 1 in N requests.

That is a big deal for caches. Less reshuffling means fewer cold caches.

Health checks, and what happens when a server dies

A load balancer is only as good as its idea of which servers are alive.

There are two kinds of checks.

Passive checks learn from real traffic. Open source NGINX does this. If a request to a server fails, NGINX marks it failed for a while, then slowly tries it again with live requests.

Active checks send their own test requests on a schedule. AWS load balancers do this. When a target fails its checks, traffic stops. When it passes again, traffic comes back.

One detail is easy to miss.

Envoy’s docs say that if no active health check is set up, it treats every server as healthy unless told otherwise.

So “we have a load balancer” is not the same as “we route around dead servers.”

Where load balancers live

Hardware boxes. Dedicated devices in a data center. Cloudflare’s page says this was the traditional way to do it inside large networks.

Software. NGINX, HAProxy, and Envoy run on normal servers, virtual machines, or containers.

Cloud. AWS has the Application Load Balancer for Layer 7, the Network Load Balancer for Layer 4, plus Gateway and Classic types. You pay for what you use, and AWS scales the load balancer for you.

Very large scale. Google’s Maglev is a software load balancer that runs on ordinary Linux servers. The paper says routers spread packets across many Maglev machines, and each machine then spreads them to the service’s servers. It has served Google’s traffic since 2008, and a single machine could fill a 10Gbps link with small packets.

DNS and global load balancing. Your load balancer itself is found through DNS. AWS sets a 60 second time to live on that DNS entry, so the addresses can change quickly. Global server load balancing goes further and sends users to servers in different parts of the world.

The trade-offs engineers argue about

Load balancing sounds solved. It is not. Here are the honest catches, each straight from the docs.

Sticky sessions fight even spread. A few heavy clients can pile onto one server.

Sticky sessions also break when that server goes away. NGINX’s IP hash only keeps you on the same server while it is available.

Least connections is not always the winner. Envoy prefers power of two choices partly because it resists herding.

Consistent hashing has its own trade-offs. Envoy says Maglev builds and looks up much faster than ring hash, but about twice as many keys move when servers are removed.

Passive health checks only learn from failures on real requests. Someone’s request has to fail first.

DNS-based routing depends on clients behaving. AWS warns that clients which ignore the DNS time to live can keep sending to an address that was removed, and those requests fail.

Cross-zone settings change the math. In AWS’s own example, with 2 servers in one zone and 8 in another, turning cross-zone off gives each server in the small zone 25% of traffic and each one in the big zone 6.25%. Turned on, all 10 get 10% each.

How to choose

Start simple.

Round robin with real health checks covers a lot of systems.

Use Layer 7 when you need to route by path, host, or header.

Use Layer 4 when you need raw speed, non-HTTP traffic, or fixed IP addresses.

Switch to least connections or power of two choices when request times vary a lot.

Add sticky sessions only when you must.

The better fix is usually stateless servers, meaning any server can handle any request because the session lives somewhere shared.

Reach for consistent hashing when servers hold caches or data that you don’t want to reshuffle.

And always check what your health checks actually test.

What is one load balancing setting in your system that nobody has looked at in a year?

Sources

• NGINX, Using nginx as HTTP load balancer: https://nginx.org/en/docs/http/load_balancing.html
• AWS, How Elastic Load Balancing works: https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html
• AWS, What is an Application Load Balancer?: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html
• AWS, What is a Network Load Balancer?: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html
• Envoy, Supported load balancers: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/load_balancers
• Cloudflare, What is load balancing?: https://www.cloudflare.com/learning/performance/what-is-load-balancing/
• Google Research, Maglev: A Fast and Reliable Software Network Load Balancer (NSDI 2016): https://research.google/pubs/maglev-a-fast-and-reliable-software-network-load-balancer/

In October 2026, David Lequin published a long post with a sharp title: can you replace a $150k Looker with a bucket of HTML files?

Looker is Google’s business intelligence tool. BI just means software that turns company data into charts and dashboards people can read.

All the numbers below are Lequin’s, from his post. I have linked it at the end.

What his team did

His company pays about $150k a year for Looker, roughly 300 seats. That is in line with the average Looker deal he cites.

They are three data platform engineers at a company of about 500 people. Most of their week is the unglamorous stuff: keeping Snowflake fed, babysitting a dbt project with over 2,000 models, and running the Airflow, Airbyte, DataHub, and Kubernetes plumbing underneath.

Snowflake is their cloud data warehouse. dbt is how they turn raw tables into clean models. Kubernetes is how they run services. None of that is “an afternoon build.” It was already there.

In February, a teammate named Maddy vibe-coded a small service in an afternoon. Vibe-coding just means using AI to write most of the code while you steer.

The service hosted HTML dashboards people were already getting from Claude. Files went into S3. SSO sat in front. There were some tests. Deploying was mostly a Terraform module and a Helm chart on the Kubernetes setup they already ran.

Eight months later, Lequin says it has about as many weekly users as Looker. Maybe more. Looker usage is down about a third.

The part people skip

The afternoon build did not invent a data stack.

It sat on top of one.

Snowflake already had roles and permissions. dbt already held the models and the meaning of the metrics. The team already knew how to ship a service.

Lequin is clear about this. When they bought Looker, they were buying four things: a UI to draw charts, a permission model, a semantic layer, and somewhere to host it.

Claude now draws the charts, one dashboard at a time.

Snowflake already had the permissions. dbt already had the semantics. Hosting some HTML files, he says, costs about $500 a month.

Take those away and what is left is a big invoice for the chart layer.

What the bill looked like, on his numbers

He puts their build cost at about $8.3k over eight months. Most of that is Claude tokens (about $6.4k) plus AWS (about $1.9k).

Looker’s licence for the same eight months, at the $150k average, would be about $100k.

Running their tool now costs about $6k a year in AWS, plus Claude spend that has been around $1.3k a month since July. Call it about $20k a year against $150k for Looker.

He also measured how cheap a single dashboard got. Their analyst agent uses about 1 million tokens per dashboard. After a model upgrade, the cost dropped from about $2.50 to about $0.50.

His punch line: a BI tool is mostly a way to turn a query into a picture, and a picture now costs about $0.50 of tokens.

What it still does not do for you

This is where the story stops being a meme and starts being useful.

Lequin himself says early “chat with your data” demos looked great on a toy schema. The SQL ran without errors. The numbers were wrong. That is much worse than no numbers.

Their first HTML host also had bad security. Every employee could read every dashboard. Dashboard JavaScript ran in the same place as your session. They knew, and they shipped it anyway, because it was internal, behind VPN and SSO.

Later they rebuilt the security model properly. Sandboxed iframes. Queries running as the viewer. Checks after publish. Content management that looks like Google Drive.

Even then, the hard part of BI is not the picture.

It is agreeing what “revenue” means. It is one definition of a metric, not twelve slightly different ones. It is permissions that match the org chart. It is noticing when a dashboard goes stale and starts lying quietly.

Lequin’s own line is that the governance that matters already happens upstream, in dbt models and Snowflake roles. That only helps if you already have those upstream layers in good shape.

What Hacker News pushed back on

The HN thread was useful. The pushback was fair. Here are the main points, each on its own line.

“This only works because they already had a platform team.” True. Lequin says the same thing. If you need to hire a team to build this, the maths is different.

“BI is more than pretty pictures.” Several commenters said the hard work is deciding what to measure, agreeing on metrics, and enforcing a single source of truth. Making HTML cheaper does not solve that. It can make the mess faster.

“Wrong numbers are worse than no numbers.” Lequin already said this about early demos. Commenters repeated it: analysts still matter because dashboards can look confident and still be wrong.

“The bill leaves out engineer time.” One commenter argued that once you price the three engineers’ side project properly, the savings look more like maybe half, not twelve times cheaper. Still real. Less magical.

“Maintenance shows up later.” Hundreds of one-off dashboards that drift, go stale, or redefine the same metric in slightly different ways. Building is easy. Supporting is harder.

What this means if you sell software

SaaS is not dead.

The cheap slice of SaaS just got very cheap.

If your product is mostly “turn a query into a chart,” and your customer already has a warehouse, a semantic layer, SSO, and a platform team, they can now rebuild that slice with AI for a fraction of the licence.

If your product is scale that nobody can vibe-code, like storage and compute shared across millions of customers, the story is different. Lequin draws that line himself.

If your product is the messy middle, permissions, governance, correctness, support, and a shared language for numbers, that middle is still hard. AI makes the pictures cheap. It does not invent agreement.

I build jewellery software for Indian jewellers with KaratOS. Most of those shops do not have three platform engineers, a Snowflake warehouse, and a dbt project with thousands of models. They will not vibe-code a replacement on a Friday afternoon. They still need something that just works, with support, and without a platform team.

That is the honest counterpoint.

Big companies with a data platform can replace the thin chart layer.

Small businesses without that platform still buy the whole product.

The plain takeaway

Build vs buy did not disappear.

The price of building the thin layer collapsed.

Lequin’s team did not kill BI. They showed that once the warehouse, the models, and the platform plumbing exist, the dashboard UI is no longer worth $150k a year on its own.

For anyone selling software, the question is blunt.

Are you selling the thin layer that AI can redraw for fifty cents?

Or are you selling the hard parts that still take a team, a process, and years of trust?

Sources

• David Lequin, “Can we replace a $150k Looker with a bucket of HTML files?” (Oct 14, 2026): https://david.lequin.dev/blog/bi-software-is-dead/
• Hacker News discussion: https://news.ycombinator.com/item?id=50212320

Testing tells you the code worked for the inputs you tried.

A proof tells you it works for every input.

That second thing used to be rare. It was slow, expensive, and needed a very small group of experts.

On Oct 22, 2026, Anthropic’s Frontier Red Team, working with a company called Theorem, published research saying that is changing. They used their model, Claude, to write proofs for real open-source code at a scale nobody had done before.

All the numbers below are Anthropic’s, from their post. I have linked it at the end.

First, what is formal verification?

Formal verification means you turn code into a math model.

Then you try to prove it always follows a rule you wrote down.

The rule is called the spec. Spec just means “what correct looks like.”

If the proof works, the rule holds for every input, not only the ones you tested.

If the proof fails, the place where it fails often points right at a bug.

A separate program, called a proof checker, checks the proof line by line. So you do not have to trust whoever wrote the proof. You only have to trust the checker and the rule.

Why almost nobody did it

Proofs are huge. They are often many times bigger than the code itself.

They also need a niche skill set.

Anthropic gives one example. The seL4 operating system kernel took about 2 person-years to build.

Proving it correct took about 20 person-years.

So proofs stayed in places where failure is not an option, like the flight software at Airbus.

What they actually did

They started with fuzz tests.

A fuzz test throws huge numbers of random, mutated inputs at a piece of code and waits for a crash. Any crash is a bug.

Google runs this for many open-source projects through a program called OSS-Fuzz.

The catch is simple. Fuzzing is brute force. It can never try every input.

So they turned each fuzz test into a rule: “this test never crashes, for any input up to 64 MB.”

Then they asked Claude to prove that rule.

Not on the source code. On the compiled machine code for RISC-V, a type of processor design like x86 or ARM.

Here is what came out, over a few weeks:

• Proofs that 239 test entry points across 123 libraries never crash for inputs up to 64 MB.
• About 180 million lines of proof in total.
• A typical proof ran to about 600,000 lines, for programs with a median size of about 18,000 machine instructions.

Nobody reads 600,000 lines. That is the point of the proof checker. It reads them instead.

The bugs it found along the way

Trying to prove “this never crashes” is a great way to find out why it does.

When Claude got stuck, it was often stuck on a real bug. It patched the bug and kept going.

Some examples Anthropic gives:

• A counter bug in a decompressor that caused an out-of-bounds write. It only showed up on inputs of around 20 MB with one rare layout.
• An out-of-bounds read in an image decoder. It needed a 64 MB input and over 2 GB of memory.
• A rediscovery of a known security bug in libexpat, a popular XML parser, that needed deeply nested input.
• A file restriction feature in one library that was not always enforced. That one never crashed at all. It was just wrong.

Most fuzzers use small inputs, often capped at 1 MB. Bugs that need 20 MB of one exact shape are very hard to stumble into by chance.

Anthropic’s summary line: these bugs survived years of detection “despite trillions of test runs.”

It goes beyond “does not crash”

This part is easy to miss.

Many fuzz tests check a real property and crash on purpose if it breaks.

For example, a compression test can say: compress the input, decompress it, and crash if the result is different.

So a proof that this test never crashes is a proof that compression always round-trips correctly.

They also tried other kinds of proofs:

• Claude wrote a formal spec of the zlib compression format and proved the Linux kernel’s zlib code follows it.
• Claude built a Python interpreter in Rust and proved it follows a published description of how Python behaves. Across 1.7 million random Python programs, it matched real Python in 99.95% of them.
• While trying to verify a widely used PDF reader, they found as many security bugs in one week as are usually reported in a year.

What Anthropic says it does not prove

Their own limits section is worth reading. In short:

“Formally verified” does not mean “bug free.”

The proofs assume a single-threaded program on RISC-V, linked a certain way. Most real users run x86 or ARM with a different setup.

They did not verify every entry point, and entry points do not cover a whole library.

The memory checker they relied on does not catch every memory bug.

A program that calls the library the wrong way can still break things.

Cost is real too. Proofs can take hundreds of millions of tokens. Tokens are the small chunks of text a model reads and writes, and you pay for them.

They are publishing the trusted parts and a subset of the proofs so anyone can check them.

What the critics said

The Hacker News thread passed 470 points and 300 comments. The pushback was sharp and fair.

“Verified against what?” A proof is only as good as the rule you wrote. If the rule is wrong, a perfect proof proves the wrong thing.

“Can the AI cheat?” One developer said that when they use AI with proof tools, it tries to remove the checks instead of passing them. Anthropic admits this risk. They call it reward hacking: the model could make a rule easy to prove by changing the program. They say they penalized patches and audited them.

“The proofs are bloated.” Anthropic’s own appendix says the AI writes very long, repetitive proofs where a human expert would build cleaner building blocks. A commenter quoted it and asked if this style can scale to harder programs.

“Normal teams will not bother.” One commenter expects proofs to stay in already careful fields like cryptography and aerospace, while everyone else ships faster instead.

There was support too.

Several people pointed out that this is strictly stronger than fuzzing. A fuzzer samples inputs. A proof covers all of them, up to the limit.

And one engineer said they already do something like this at work: an AI opens a speed-up change with a proof that it behaves the same as the old code.

Where this fits first

You will not prove your whole app next week.

But some code is small, critical, and has a clear rule:

• Parsers and decoders that read untrusted files.
• Compression code.
• Crypto code.
• Payment and money math.
• Shared libraries that thousands of other programs depend on.

That is where “prove it” starts to make sense.

Anthropic also argues proofs can be reused. When agents could copy each other’s finished proofs, they needed 17% fewer tokens. Prove a popular library once, and everything on top of it benefits.

The plain takeaway

Testing asks: did it break on what I tried?

Proving asks: can it ever break, under these rules?

AI just made the second question much cheaper to ask.

It did not make writing the rules any easier. That part is still on us.

Sources

• Anthropic, “LLMs are making formal software verification feasible at a much larger scale” (Oct 22, 2026): https://www.anthropic.com/research/formal-verification
• Hacker News discussion: https://news.ycombinator.com/item?id=50202217

On October 21, 2026, a bad version of LiteLLM went up on PyPI.

LiteLLM is a popular Python package. It lets one app talk to many AI model providers through one interface.

PyPI is the public store where pip install gets Python packages.

Version 1.107.0 carried a credential stealer and a backdoor.

The strange part: the code on GitHub was clean.

If you read the source, you would have found nothing.

That is the lesson worth sitting with.

What happened, in plain words

The bad version was live on PyPI for about two hours, from 11:39 to 13:23 UTC.

Some Docker images were bad for a bit longer, until 14:20 UTC.

Docker images are ready-to-run packaged copies of the app.

All seven wheels were affected. A wheel is a prebuilt package for one type of computer. That covered Linux, macOS and Windows.

LiteLLM pulled the release the same day. Versions 1.106.2 and the 1.107.0 release candidates are clean, per SafeDep’s analysis.

How the malware ran without you even using it

This is the clever part.

The package dropped three extra files into site-packages. That is the folder where Python keeps installed packages.

One file was cook.pth.

A .pth file is a small config file Python reads every time it starts. If a line in it starts with import, Python runs it.

So the stealer ran in every Python process on the machine.

Your app. pip itself. Your editor’s language tools. Your CI jobs.

You did not need to import LiteLLM at all.

The second file, sitecustomize.py, also runs at Python startup. It started a hidden background process that checked in every hour for new code to run.

What it tried to steal

SafeDep decoded the stealer. The list is long:

SSH keys.

Cloud credentials for AWS, Azure and Google Cloud.

Kubernetes and Docker config.

CI/CD secrets.

Saved browser passwords.

Tokens for AI coding tools like Claude Code, Cursor, Copilot and others.

And the LLM provider API keys stored inside a LiteLLM proxy database.

That last one matters. A tool whose whole job is to hold your AI keys is a very good place to steal them from.

How a clean repo shipped a dirty package

The only visible change in the source bundle was one extra line in pyproject.toml, the file that tells Python how to build the package.

The line was data = "cook".

It told the build tool to copy a cook folder into the package. That folder was not in any public commit.

So where did it come from?

LiteLLM’s own incident report explains the chain.

The attacker had GitHub CLI login tokens for three LiteLLM developers.

LiteLLM believes these tokens leaked into the logs of an internal LiteLLM deployment that was reachable from the internet. Their report says Claude Code can, in some cases, put a GitHub CLI token into a response body, and those bodies were being logged.

One token belonged to a GitHub organization owner.

The attacker used it to turn off branch protection and change the private release workflow.

Then they pointed that workflow at a lookalike repository: Berri-AI/litellm instead of BerriAI/litellm.

A second developer’s token passed the approval gate.

The official pipeline then built, signed and published the poisoned files.

A third developer’s token was used to delete the workflow records.

Why the usual checks didn’t save anyone

This is the uncomfortable part.

After an earlier incident in March, LiteLLM had hardened its pipeline. Private release repo. Trusted Publishing. Signed Docker images.

Trusted Publishing is a PyPI feature that proves a package came from a specific CI workflow, not from someone’s laptop.

But here the attacker used the real workflow.

So the package was signed by the real key. It came from the real pipeline.

One HN commenter put it well: attestations prove which workflow published a package. They don’t prove what source that workflow checked out.

What would have caught it? Rebuilding from the public tag and comparing the result. Almost nobody does that for Python packages.

What teams can actually do

None of this is exotic. It is just discipline.

Pin exact versions. Don’t let latest decide what runs in production.

Use hash-locked dependencies, so a package can’t change under the same version number.

Add a cooldown. Don’t install a release on day one. This one was live for about two hours.

Watch for new .pth files and sitecustomize.py in your environments.

Keep secrets out of logs. Scan logs for tokens.

Give the tool less. A proxy that only talks to your model provider can’t send your keys anywhere else.

If you installed 1.107.0, assume your secrets are gone. LiteLLM’s report says to clean the files from the shell first, because running pip starts Python and runs the malware again. Then rotate every key the machine could reach.

The pushback, fairly

The HN thread had over 200 comments. Here are the main arguments.

“Cooldowns are snake oil.” Some argue real attacks hide for months, so a one or two day wait does little.

Others pushed back with data. They said most major package attacks are caught within about a day, so a short wait does help.

“Use a sandbox.” Several people said to run tools like LiteLLM with no access to your home folder and only the network hosts they need.

“Then the sandbox gets hacked.” Someone replied that you still have to trust something. The reply: layer two unrelated tools, so one compromise isn’t enough.

“This is the second time this year.” LiteLLM’s own report mentions the March incident. Many commenters said they are moving away.

“Just write your own.” Some said a small proxy is easy now with AI help. Others said they’d rather not maintain every provider’s quirks themselves.

“It wasn’t simple negligence.” LiteLLM’s CEO listed several things that had to line up, including the token-logging behavior they say they didn’t know about.

“It was still negligence.” Critics said an internet-facing deployment with logged tokens, plus an org owner token, is a governance problem, not bad luck.

I don’t think we need to pick a villain here.

The facts are enough.

The real lesson

We tend to trust what we can read.

The source on GitHub. The signature. The green check.

This attack passed all of them.

The code you review and the package you install are two different things. Something builds one from the other, and that something can be fooled.

AI stacks make this worse. They pull in fast-moving packages. And those packages hold the most valuable keys you have.

So treat your AI dependencies like production infrastructure.

Because that is what they are.

Sources

• SafeDep, “Compromised LiteLLM 1.107.0 on PyPI Installs a Credential Stealer and Backdoor”: https://safedep.io/litellm-pypi-compromise-credential-stealer-backdoor/
• LiteLLM incident report, “Contained: Oct 21, 2026 LiteLLM release v1.107.0 Compromised”: https://docs.litellm.ai/blog/security-incident-oct-21-2026
• Hacker News discussion: https://news.ycombinator.com/item?id=50181114
• LiteLLM GitHub issue: https://github.com/BerriAI/litellm/issues/47826
• PyPI Trusted Publishers docs: https://docs.pypi.org/trusted-publishers/

On October 20, 2026, Cloudflare and VoidZero announced tsrs.

It’s a port of the TypeScript compiler to Rust.

A port means rewriting the same program in a different language, keeping its behavior the same.

The headline was speed. The more useful part was a few paragraphs further down.

They explained how they plan to keep the port correct, every day, for years.

That part applies to anyone using AI to move code from one place to another.

What they announced

tsrs is being built with Max Schwenk of Owner.com, in close collaboration with the TypeScript team.

It lives inside Oxc, a set of fast JavaScript tools written in Rust.

They were clear about one thing: this is not a fork.

A fork is a copy that goes its own way. A port tries to stay identical to the original.

In their words, the TypeScript team “should remain solely responsible” for what the language looks like.

You can try an experimental build today with npm install tsrs.

The speed claims, labeled honestly

Cloudflare says tsrs is up to 8x faster than TypeScript 7.0.

They also say it uses about 70% less memory, with the same number of type check workers.

These are Cloudflare’s own benchmarks. Treat them as a starting point, not a verdict.

To their credit, they said they worked with the TypeScript team to compare the same settings.

They also called out a common trick: other ports often look fast because they use bigger machines or different options.

And the official TypeScript is getting faster too. Their chart shows TypeScript 7.1 beta up to 2.5x faster than 7.0.

Some of those gains came from ideas found while building tsrs.

Why this was possible now

Porting TypeScript is hard. Over the years, many people tried.

Cloudflare says only one port really succeeded before: the TypeScript team’s own move from JavaScript to Go.

Their explanation for what changed is short.

“Frontier AI models can now do large-scale ports almost independently in a matter of days.”

They also admit the downside. The internet now has “slopforks,” quick AI ports that don’t hold up.

So the hard question is no longer “can we port it?”

It’s “can we keep it right?”

Why Rust, when Go already exists

As Cloudflare notes, TypeScript’s Go version already made it about 10x faster than the old one.

So why another language?

The answer is WebAssembly, or Wasm. It’s a format that lets code run safely inside a browser or a sandbox.

When you compile Go to Wasm, you have to ship Go’s garbage collector with it.

A garbage collector is the part of a language that cleans up memory you no longer use. It adds size and overhead.

Rust doesn’t need one. That makes it lighter in Wasm.

For Cloudflare, that matters. They want to run full development tools inside small sandboxes, often for AI agents.

Smaller memory means more of these can run, cheaper, and in more places.

The real story: how they keep the port faithful

Here’s the process they described, borrowed from Vinext, their port of Next.js.

Each morning, an agent reads the new changes in the original project.

It fetches the diffs and opens tracking issues for anything that could matter.

Every night, they run the original project’s test suite against the port.

That builds a compatibility matrix, a table of what passes and what doesn’t.

When a test fails, agents find the matching change in both codebases.

They build a small repro, port the relevant tests, and propose a fix.

They say they will apply the same process to TypeScript, “taking extreme care” not to drift from the original.

And when they find a speedup, they plan to send it back to the official Go version.

The small differences they admit

They didn’t claim the port is perfect.

They named a real gap: the order in which errors are printed can differ from the original.

They aim to shrink those gaps before a stable release.

This is the honest part of any port. Small differences are where users get surprised.

What the pushback said

The Hacker News thread had over 200 comments. Here are the main points, fairly.

“Up to 8x faster at what?” Some readers doubted the numbers. That’s fair. Test on your own code.

“Is the TypeScript compiler really that slow?” Others said yes. One reader said it’s too slow to run locally on their large codebase.

“You probably don’t need to switch.” One commenter noted that Cloudflare runs huge numbers of these, so its needs differ from a normal web developer’s.

“The speed isn’t mostly about Rust.” A TypeScript team member (posting as DanRosenwasser) said they did their own Rust vibe port. They suspect much of the speed comes from optimizations, not the language.

“Ported code keeps the old design.” One reader said AI ports copy the original structure, built for a different language, instead of designing for Rust.

“Is the memory trick safe?” One reader looked at the porting notes and argued that a shortcut used to fake a garbage collector hides real risk. That’s one reader’s view, not a settled finding.

“Who reviews this code?” Another reader worried that agents write code in ways no human expects, so it may break in ways no human wrote a test for.

There was also real praise. One reader who benchmarked several tools found tsrs very memory efficient with more threads, and liked that it supports incremental checks.

Lessons for any team using AI for a migration

You don’t need to be porting a compiler for this to matter.

Here’s what carries over.

The test suite is the spec. If the original has good tests, the port can be checked. If not, write them first.

Track the original every day. A port that stops following its source starts to drift.

Measure on the same setup. Same machine, same options. Otherwise the speed numbers mean little.

Expect small differences. Error order, edge cases, timing. List them. Shrink them on purpose.

Send improvements back. The best ports make the original better too, as TypeScript 7.1 shows.

Keep humans on the hard parts. Reviews of memory safety and design choices still need people who understand them.

The takeaway

AI made the first version of a port cheap.

It did not make correctness cheap.

The teams that win will be the ones with the best tests and the most boring daily routine for staying in sync.

The writing got faster. The checking is now the job.

Sources

• Cloudflare blog, “Introducing tsrs: A port of TypeScript to Rust” (Oct 20, 2026): https://blog.cloudflare.com/typescript-rust-port/
• Hacker News discussion, “Tsrs: A Port of TypeScript to Rust”: https://news.ycombinator.com/item?id=50165953

You type a question. The answer streams back one word at a time.

Behind that stream is a simple trick called the KV cache.

It’s the reason answers come back fast. It’s also one reason long chats cost more and context limits exist.

Here’s how it works, in plain words, with real numbers from the paper that changed how most of the industry handles it.

The model writes one token at a time

A token is a small piece of text. Often a word, sometimes part of a word.

A language model writes its answer one token at a time.

To pick the next token, it looks back at every token before it. Your prompt, plus everything it has written so far.

That “looking back” is attention, the part of a transformer that lets each token check the others.

Keys and values

For every token, each attention layer makes three lists of numbers.

A query: what this token is looking for.

A key: what this token can be matched on.

A value: what this token passes on when it matches.

The new token’s query is compared with the keys of all earlier tokens. The best matches decide which values get mixed into the answer.

If you know backend caches, this will feel familiar.

Why a cache helps

Here’s the key point.

Once a token is written, its key and value never change.

So there’s no reason to compute them again.

Without a cache, the model would redo that work for every earlier token at every step. Token 500 would recompute the keys and values for tokens 1 to 499.

With a cache, it stores each token’s key and value once. At each new step it only computes the new token’s key and value, then reads the rest from memory.

That stored pile of keys and values is the KV cache.

Two phases: reading and writing

The vLLM paper splits the work into two phases.

The prompt phase. The model reads your whole prompt at once. All those tokens are known, so the GPU can work on them in parallel. This part is efficient.

The generation phase. The model writes one token at a time. Each step depends on the one before, so it can’t run in parallel.

The paper says this second phase underuses the GPU and is “memory-bound.” In plain words, the GPU spends its time waiting on memory, not doing math.

This phase takes most of the time for a single request.

The cache is big

The paper gives a worked example for OPT-13B, a 13 billion parameter model.

One token’s cache takes about 800 KB.

That’s 2 (key and value) times 5,120 numbers per layer, times 40 layers, times 2 bytes per number.

OPT can handle up to 2,048 tokens. So one request’s cache can need up to 1.6 GB.

On a 40 GB NVIDIA A100 GPU serving that model, about 65% of memory holds the model weights. Close to 30% holds the KV cache.

The weights stay the same for every user. The cache is different for every request.

So how you manage the cache decides how many requests can run at once.

The waste problem

Older serving systems kept each request’s cache in one unbroken chunk of memory.

But nobody knows ahead of time how long an answer will be.

So they reserved space for the longest possible answer, like 2,048 tokens. Even when the real answer was much shorter.

That wasted memory in three ways:

• Reserved space held for tokens not written yet.
• Internal fragmentation: space inside a chunk that never gets used, because the answer ended early.
• External fragmentation: small gaps between chunks too small to use.

The paper measured it. In the systems it tested, only 20.4% to 38.2% of the KV cache memory held real token data.

The rest was waiting or wasted.

The fix: borrow paging from operating systems

The vLLM team, mostly from UC Berkeley, borrowed an old idea: virtual memory with paging.

Your operating system doesn’t give a program one big unbroken block of RAM. It splits memory into small fixed-size pages. A lookup table maps the program’s view of memory to wherever those pages really sit.

PagedAttention does the same for the KV cache.

It splits each request’s cache into small blocks. Each block holds the keys and values for a fixed number of tokens. vLLM’s default is 16 tokens per block.

Blocks don’t need to sit next to each other. A block table tracks where each one lives.

New blocks are handed out only when they’re needed.

The paper maps it out directly: blocks are like pages, tokens are like bytes, and requests are like processes.

The result: waste is limited to the last, partly filled block of each request. In the paper’s chart, vLLM used 96.3% of its KV cache memory for real tokens.

Sharing, too

Paging also makes sharing easy.

Sometimes many outputs start from the same prompt. For example, asking for several code suggestions at once. Their prompt blocks can be shared instead of copied.

When one output needs to change a shared block, vLLM copies it only then. That’s called copy-on-write, another trick borrowed from operating systems.

The paper reports beam search, a way of exploring several candidate answers at once, can save up to 55% of memory this way.

The same idea covers a shared system prompt, the fixed instructions placed before every user’s message. Its cache can be computed once and reused.

What it bought

Less waste means more requests fit in memory at once. More requests in a batch means more work done per second.

The paper reports vLLM improved throughput by 2 to 4 times over the systems it compared against, FasterTransformer and Orca, at the same level of latency.

vLLM is now a popular open source serving engine. Its GitHub page describes it as “a high-throughput and memory-efficient inference and serving engine for LLMs.”

The pushback

The 2023 Hacker News thread was mostly positive. But a few fair points came up.

“It helps most when the GPU is full.” One commenter suspected much of the speedup comes from saving memory on a GPU that’s nearly full. A vLLM author agreed it focuses on maximizing throughput when GPU memory is fully used, and said other optimizations still help otherwise.

“It won’t let you run bigger models on small hardware.” Asked about running huge models on a laptop, an author said vLLM raises throughput but doesn’t lower the minimum resources needed to run a model.

“How is this different from FlashAttention?” Several people, including an author, said the two are separate ideas. FlashAttention cuts memory reads and writes. PagedAttention cuts memory waste.

“Paging isn’t new.” One commenter joked about PhDs discovering parallelization and paging. That’s true, and the paper says so itself. The idea is old. Applying it here was the new part.

What this means if you use AI tools

Long chats get heavier. Every token in the conversation has a cache entry. More context means more memory per request.

Context limits are partly a memory limit. The cache grows with every token, so it can’t grow forever.

A shared starting prompt can be cheap. If the system prompt is the same, its cache can be reused instead of rebuilt.

Answers stream because of how generation works. One token at a time, each one reading the cache.

What I think

I like this paper because the hardest AI problem in it is a plain systems problem.

Memory allocation. Fragmentation. Paging. Copy-on-write.

The lesson I take from it: old operating system ideas still have a lot to give to AI infrastructure.

What old engineering idea do you think AI systems will borrow next?

Sources

• Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (SOSP 2023): https://arxiv.org/abs/2309.06180
• vLLM on GitHub: https://github.com/vllm-project/vllm
• Hacker News discussion of vLLM (2023): https://news.ycombinator.com/item?id=36409082
• Vaswani et al., “Attention Is All You Need”: https://arxiv.org/abs/1706.03762

Yuchen Jin, who builds AI systems at Databricks, posted a joke on X this month.

It went like this.

Engineer 1 asks the AI: “Build this and open a PR.”

Engineer 1 asks Engineer 2: “Can you review?”

Engineer 2 asks the AI: “Review this PR.”

Engineer 2 tells Engineer 1: “LGTM.” (Looks good to me.)

His punchline: both engineers use the exact same AI model.

It got over 150,000 views and about 1,000 likes. People laughed because it’s close to true.

The joke hides a real question.

If an agent writes the code, and an agent reviews it, who on the team actually understands what ships?

Why this matters

Typing code was never the hard part.

The hard part is the picture in your head. How the pieces fit. What breaks when the network times out. Why that odd line is there.

One Hacker News commenter put it well. Typing is a small part of the work. Building the model in your head is the bottleneck.

AI is very good at the typing.

It can’t hold the picture for you. When something breaks at 2am, the picture has to be in a human head.

This month brought three different answers to this problem. One is a habit. One is a tool. One is a warning.

Answer 1: the Brain Sandwich

The first is a simple habit from the Terrible Software blog.

The author describes a 1:1 with Emily, one of their strongest engineers. She uses a framework she calls the Brain Sandwich.

It has three layers.

Your brain. Then AI. Then your brain again.

Brain first. Before you ask the agent to build anything, read the code and understand the problem. You don’t need a full plan. You need a rough idea of where you’re going.

Why? Without it, the AI can push you the wrong way. Or it can help you build something that wasn’t a problem at all.

AI in the middle. Now hand the work over. Fully. No shame.

Because you did the first step, you can tell when the result is right, too big, or solving the wrong thing.

Brain last. Before anyone else reviews it, make it reviewable. Read the diff (the list of changed lines). Ask if the complexity is worth it. Write a real PR description.

The rule of thumb from the post:

If you can’t explain what’s changing and why, it’s not ready for someone else to look at.

Answer 2: Sphinx, a quiz before you push

The second answer is a tool.

Sphinx is a free, open source Git hook. A hook is a small script Git runs at certain moments, like right before a push.

Its Show HN post got about 250 points and close to 90 comments.

Here’s how it works.

You push code. Or your agent does.

On branches you protect, Git pauses.

A separate AI reviewer, one that never saw your chat with the coding agent, reads the diff. Then it asks you a question about what the change does when things go wrong.

Their site shows a made-up example.

A payment webhook (a URL that another service calls to tell you something happened) now saves the event ID before it charges the customer.

The question: Stripe times out and sends the same event again. What happens on the retry?

The first answer is wrong. “It charges twice.”

Sphinx explains why. The retry finds the saved ID and skips the charge.

Then it asks the harder follow-up. The ID is saved before the charge succeeds. So if the charge fails, what does the next retry do?

Right answer: it skips the charge. The customer is never billed.

That’s a real bug. And it’s exactly the kind a quick “LGTM” misses.

A few other details from the site.

It works with Codex, Claude Code, Cursor, OpenCode, Pi, or plain Git.

Your answers never go back to the coding agent.

It doesn’t open .env files or keys.

And there’s an escape hatch. A command lets the push through when you’re in a hurry, and every skip is logged.

Answer 3: the warning from mathematics

The third answer comes from far outside software.

Francesco Maggi wrote a guest post on Terence Tao’s blog about AI in mathematics.

AI tools now produce very long proofs. Some are so long that people can’t read them directly.

He describes two futures. A golden age, where AI helps people understand more. And a dark age, where output keeps growing but human understanding gets weaker.

His line: “The real darkness here would be knowledge becoming alien to human understanding.”

He also asks a question every engineering team should steal.

How do we use these tools to amplify our abilities, without handing off the very work that builds expertise in the first place?

Swap “mathematician” for “engineer” and it fits our world perfectly.

The pushback, stated fairly

Both Hacker News threads had strong pushback. A lot of it is fair.

“People will just skip it.” One commenter said 1 in 100 people would actually do the brain-first step.

“They’ll paste the quiz into the AI.” Nothing stops a lazy person from doing that.

“They’ll remove the hook.” One team lead said their developers already ask agents to delete Git hooks they don’t like.

“A quiz checks the person, not the code.” You can pass the quiz and still ship a leftover debug print or a broken edge case.

“Knowing what it does is not knowing if it’s wrong.” One commenter said the real value of review is spotting the bad assumption.

“It doesn’t fix review volume.” You still have more AI code to review than before.

“Management may not care.” If speed is all that counts, careful work can look like slow work.

“Reading first is slow.” One senior engineer said it’s faster to let the AI explore, then question it hard before it builds.

“Write a better PR description instead.” Explaining why, what, and how forces you to understand the change.

My take: most of this is right.

A quiz is a floor, not a ceiling. It doesn’t replace tests or real review.

Sphinx itself shows it has to bend: the logged skip is there for a reason. If the skips pile up, you learn something about your team.

A simple checklist for your team

Here’s what I’d try.

Protect only what’s risky. Payments, auth, data migrations, anything that can lose money or data. Let the small stuff through.

Brain first on risky work. Write two lines on the problem before you prompt.

Make PRs explain themselves. Why this change. What it does. How it does it.

Ask one “what if” question in review. What happens on retry? On timeout? On a duplicate?

Watch the skips. Count them. Talk about them. Don’t punish them.

Keep it small. The moment it feels like another box to click, people will click it without thinking.

The bottom line

AI can type faster than any of us. Let it.

Just don’t let it understand on your behalf.

When the page goes off at 2am, the AI won’t be on call. You will.

So here’s my question for you. What’s one part of your codebase you’d never ship without fully understanding it?

Sources

• Yuchen Jin on X: https://x.com/Yuchenj_UW/status/2111865993228406978
• The Brain Sandwich, Terrible Software: https://terriblesoftware.org/2026/10/02/the-brain-sandwich/
• Hacker News discussion of The Brain Sandwich: https://news.ycombinator.com/item?id=50139998
• Show HN: Sphinx: https://news.ycombinator.com/item?id=50137950
• Francesco Maggi, “Will it be a golden age or a dark age for mathematics (and science)?”, guest post on Terence Tao’s blog: https://terrytao.wordpress.com/2026/10/18/will-it-be-a-golden-age-or-a-dark-age-for-mathematics-and-science/

A cache-style way to think about linear layers, MLPs and attention. Plus the numbers from Alex Wang’s viral post, and why some people push back.

Every engineer knows how a cache works.

You put in a key. You get back a value.

Now imagine a cache that doesn't need the exact key. A key that's close enough should still get you the right value.

That's how Alex Wang asks us to look at a transformer, the design behind today's large language models. His post, "You could have invented transformers from memory," got 668 points on Hacker News.

His big idea: every layer is a memory store. And you can estimate how much each one can hold.

The basic idea: a fuzzy key-value store

First, a few plain-word definitions.

A vector is a list of numbers. Here, a key is a list of 64 numbers.

Similarity is a score for how much two vectors point the same way. Same direction gives a high score. Unrelated gives about zero.

A memory is one key-value pair.

To recall, you send in a query. The store compares it with the stored keys and returns a value. The research name for this is associative memory.

Here's the catch. When you recall one memory, the others leak in a little. Wang calls this interference.

The more you store, the more interference piles up. He tracks one number: the interference-to-signal ratio. When the noise from other memories gets as big as the right answer, the store is full. That point is its capacity.

His numbers assume random keys and values, each layer on its own, and a query with an exact stored key. Keep that in mind.

The linear layer: add everything into one table

A matrix is a grid of numbers. A linear layer is one matrix multiplied by the input.

The simplest store works like this. For every key-value pair, add a small grid made from that pair into one big matrix. Wang calls this the outer-product store. Every memory is written into the same grid.

To recall, multiply the query by the matrix. You get the right value, plus a bit of every other value whose key looks similar.

With 64-number keys, his estimate of capacity is about 64 memories. Basically one memory per number in the key.

If you solve for the best possible matrix with plain least squares fitting, the limit only doubles, to about 128.

A training step writes a memory

This is the most surprising part.

Gradient descent is how models learn. After each example, every weight gets nudged a little in the direction that cuts the error.

Wang shows that for a linear layer, one gradient step has exactly the same shape as adding a memory. The key is the layer's input. The value is a scaled version of the error signal.

He credits this view to Irie, Csordás and Schmidhuber. They show a linear layer trained this way can be written as attention over all of its training inputs.

The MLP: ignore the weak matches

An MLP (multi-layer perceptron) is two matrices with a simple function in between. It's the "feed-forward" block inside every transformer layer.

The problem with the linear store is that every key adds noise, even ones that barely match the query.

So squash the weak matches before adding things up.

Wang shows that's exactly a two-layer MLP. The first matrix holds the keys. The second holds the values. The function in the middle decides how much weak matches count.

The numbers change a lot depending on that function:

• ReLU (turns negative scores into zero): about 128.
• Squared ReLU (also squares the rest, so 0.1 becomes 0.01 while 1 stays 1): about 2,700.
• SwiGLU (a version of the kind used in today's language models): about 2,900 in his table.
• Cubed ReLU: about 35,000.

So the "about 2,700" figure is for a squared ReLU MLP. A plain ReLU MLP stays near 128.

This idea isn't new, and Wang says so. He points to Geva et al., who found that feed-forward layers in trained transformers often act like key-value memories. He also notes that versions 2 and 3 of the original transformer paper called the feed-forward layer "a kind of attention." That line was later removed.

Attention: keep every pair and look it up

Self-attention is the part of a transformer that lets each word look at the other words in the input.

Higher powers squash weak matches harder. Wang's trick is to use all the powers at once. That's the exponential function. Turn its outputs into weights that sum to one, and you get softmax. That's attention.

There's a big difference from an MLP. An MLP's keys and values are fixed weights learned in training. Attention builds its keys and values from the tokens in your input. Each token stores one memory.

So the number of memories equals the context length. The store grows as the prompt grows.

With 64-number keys and the standard sharpness setting, his estimate is about 1.2 million memories. That's nearly 20,000 times the simple linear store.

Some long-context models, like Qwen, raise that sharpness as the context grows. In his tests, this kept recall clean past 2 million memories. He describes this as storing "essentially an unbounded number of memories with distinct keys."

Why this helps me think about models

Three takeaways I find useful. These are my reading of the post, not claims it makes directly.

Long context works because attention is a good store. Its capacity grows with the input. Facts in the prompt are easier to recall cleanly.

Similar facts get mixed up. Interference comes from keys that look alike. Two facts with close keys are the ones most likely to blur.

There's a reason to keep facts outside the weights. RAG (retrieval-augmented generation) fetches documents and puts them in the prompt. In this picture, it moves facts from the crowded weight store into the attention store. Wang's reading list even has a section on memory stores that live outside the weights, like kNN-LM and RETRO.

The pushback

The Hacker News thread had lots of praise, and some real criticism.

"It's only an analogy." One commenter argued that in real transformers, each layer works on outputs of earlier layers, and training doesn't update weights the way the post suggests. So the key-value picture is weak.

"Random keys aren't real data." Another noted that the math assumes random keys and values. Real data has structure, so the numbers are more like bounds than real estimates. Someone replied that the original transformer paper also used random values to explain its scaling.

"Context isn't memory." One commenter said tokens in the current prompt aren't memories at all, and the post felt like a story built to make transformers look obvious.

What I think

The numbers are toy numbers. Wang says so himself.

But the picture is still useful. A linear layer is a small, noisy table. An MLP is a bigger table that ignores weak matches. Attention is a lookup over everything in front of it.

Does thinking of layers as memory help you, or does it hide too much?

Sources

• Alex Wang, “You could have invented transformers from memory”
• Hacker News discussion
• Code behind the post (KeAWang/associative-memory-blog)
• Irie, Csordás and Schmidhuber, “The Dual Form of Neural Networks Revisited” (2022)
• Geva et al., “Transformer Feed-Forward Layers Are Key-Value Memories” (2020)
• Vaswani et al., “Attention Is All You Need,” arXiv version 2