The Six Pains of a Codebase Without Memory

In early 2024, a malicious contributor known as "Jia Tan" was discovered to have spent over two years systematically introducing a backdoor into xz/liblzma — a compression library present in the majority of Linux distributions and deeply embedded in OpenSSH.

Part of what made the attack feasible: the project's original maintainer, Lasse Collin, had been under sustained pressure to move faster and hand off responsibility. The project's design rationale lived in one person's head. When Jia Tan gained maintainer trust, they inherited code whose original intent nobody else could clearly articulate or defend.

The backdoor was discovered only because a Microsoft engineer noticed an unexplained 500ms delay in SSH connections. By then, it had already shipped in Debian and Fedora test releases.

On August 1, 2012, Knight Capital lost $440 million in 45 minutes due to a piece of code originally used for test orders — called "Power Peg" — that had been repurposed years earlier.

When new software was deployed, one server wasn't updated. The old Power Peg code — no longer intended to run, its original constraints long forgotten — activated. It executed 4 million trades in 45 minutes against live market conditions it was never designed for. The firm was insolvent by the end of the trading session.

The code had drifted so far from its documented purpose that nobody remembered what enabling it actually meant in production.

When Log4Shell was disclosed, the immediate response from most affected organisations was not "let's patch this." It was: "do we even have Log4j in our stack?"

Teams didn't know. They had accumulated direct dependencies, transitive dependencies of those dependencies, and build artifacts nobody had inspected in years. The risk surface was invisible until it was in every security headline. Hundreds of engineering teams spent weeks doing emergency archaeology on their own codebases — not to fix a vulnerability, but simply to find out if they had it.

Heartbleed was a memory buffer overflow in OpenSSL — the encryption library used in roughly two-thirds of all web servers. At the time it was introduced (2011), OpenSSL had two core volunteer maintainers. A single engineer introduced the vulnerability while adding a new feature.

The patch went unreviewed in any meaningful way. The bug lived undetected for two years, exposing private keys, passwords, and session tokens for a significant fraction of the encrypted internet.

The project had bus factor 1 in the module that mattered most. No captured design reasoning. No reviewable intent for the change. No mechanism to detect that the new implementation deviated from the protocol's stated behaviour.

CrowdStrike deployed a content configuration update — not a code change, a sensor configuration file — to its Falcon security agent. Within 78 minutes, 8.5 million Windows machines crashed with Blue Screen of Death. Flights were grounded. Hospitals reverted to paper. Banks went offline.

The update modified memory that was null-dereferenced by the agent — a dependency that the team making the configuration change didn't have a complete picture of. Estimated economic damage: $5.4 billion.

The component had a high rate of operational changes, but the understanding of what those changes could trigger in production had not kept pace with the change velocity.

A wrongful death lawsuit against Toyota led to a court-appointed expert analysis of the Camry's Engine Control Module firmware. Expert witness Michael Barr testified that the codebase contained over 10,000 global variables, inadequate stack overflow protection, and safety-critical routines with single points of failure.

The highest-risk sections had been untouched for years — not because they were stable, but because nobody was confident enough in their understanding to safely change them. The "do not touch" attitude had become tacit organizational policy, applied to firmware that directly controlled vehicle safety behaviour.