Tailscale says deeply buried 16-year-old SQLite bug caused last year's outages

Users of peer-to-peer networking outfit Tailscale might have struggled through some surprising outages beginning late last year. After a six-month investigation, the team finally knows why: A bug in SQLite’s write-ahead log that had remained hidden for 16 years. 

The Tailscale team announced in a Wednesday blog post that it had finally addressed the issue with the help of SQLite maintainers, who even had to create a new tool (with Tailscale funding) to log virtual file system activity in order to track the thing down, which Tailscale software engineer Alex Chan described as resisting “all our initial attempts to find it.” 

According to Chan, the problem goes deep into the nature of SQLite – so deep that the database maintainers actually had to add code to reproduce it. 

To understand what happened, it’s necessary to know how Tailscale works. The service, based on the WireGuard VPN protocol, directly connects devices in a virtual private mesh network. It’s designed to be low complexity and easy to implement for everything from remotely accessing a NAS to connecting teams in a unified private network. Each mesh network, or "tailnet," lives on one of several servers, where a SQLite database manages all the information about the tailnets it houses. 

“We’ve used SQLite as our primary database since 2022, and we chose it because it's well-known, reliable, and widely used,” Chan wrote in the company’s post-mortem.

But a year ago, something went very wrong.

“In our current backup pipeline, we take a complete snapshot of the database every few minutes, then upload the entire SQLite file to an S3 bucket,” Chan said, but in August 2025 those backups began detecting database corruption, repeatedly, with no obvious common trigger.

The Tailscale team couldn’t reproduce the issue because there were no reliable triggers for it. No low-level code had been changed in months. A review of everything that touched SQLite turned up nothing.  

Working with the SQLite team, the Tailscalers tried to figure out what could be causing it – POSIX locks broken by close() calls? Nope. Mismanaged memory? Not that either. SQLite being used from multiple threads with thread safety disabled? Nuh-uh.

“After every incident, we gathered more data, added more diagnostics, and systematically ruled out these theories,” Chan explained. 

The WAL-Reset bug comes out of hiding

Suspicion was closing in on SQLite’s checkpointing process, which is how it takes new database entries out of a temporary hopper for addition to the master database file.

SQLite has an option to improve performance and concurrency known as the Write-Ahead Log (WAL), which serves as the aforementioned hopper. Writing the WAL to the database occurs in a process known as checkpointing. 

“In most deployments, SQLite itself decides when to do a checkpoint, and the process is invisible to the end user and developer,” Chan said. “In our control plane, we take manual control of the checkpoint process so we can run fast and consistent backups.”

The SQLite team wrote a new tool to take a closer look at the process: a virtual file system shim that extensively logs checkpointing activity. After waiting for the next corruption incident, the teams had their answer, dubbed the WAL-Reset bug. 

Described by Chan as “a rare data race in the SQLite source code between a checkpoint and write transaction,” it’s essentially a collision between checkpoints and writing data to the WAL. 

“If a write occurs at a specific time during a checkpoint, the checkpointing process gets confused — it thinks some of the pages have been copied from the WAL into the main database file, but they haven’t,” Chan said. Those pages are never written and are permanently lost, but pages that reference those pages are still written, corrupting the database and causing all hell to break loose. 

According to the SQLite team’s WAL-Reset writeup, the issue can be triggered only when WAL mode is active and multiple database connections are open on the same file, and because there has to be reading and writing going on at the same memory spot at the same time, it’s incredibly unlikely to happen in most situations. Tailscale’s decision to perform manual checkpoints was a rare exception. 

SQLite maintainers believe the bug was present going all the way back to version 3.7.0, released in July 2010; it’s now fixed, and the SQLite team recommends users update to a fixed version, though it stresses the bug is extremely unlikely to occur in ordinary use.

“This bug, though rare, does have serious consequences,” the SQLite WAL-Reset notice states. 

The incident contains a useful reminder for devs: Even the most boring, reliable software poses risks when operated in a non-standard fashion.

“Most people use SQLite in a standard configuration and never face this sort of issue,” Chan said. “By taking manual control of the checkpointing process and running at our own aggressive pace, we stepped off the well-trodden operational path.” ®