Rendered at 12:31:23 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
carlsverre 16 hours ago [-]
Hi! I'm the author and the person who was on a road trip when I decided to do this experiment back in July. I'm super excited to finally get the blog post out, and even better, alongside a post describing the root cause process that went into the project. You can read about Tailscale's story here: https://tailscale.com/blog/sqlite-wal-reset-bug
I'm only sad that I didn't put SQLite under test earlier in the year, or I would have found this issue right away. If you look at the workload[1], you can see how simple it is. Exactly the same kind of workload we write every day to help our customers find bugs.
If you have any questions about our process or how debugging with Antithesis works, please let me know! Thanks for reading!
How difficult would it have been to isolate that problem if you didn't already know the SQLite subsystem it was in? This feels, to someone relatively ignorant of the SQLite / Tailscale / Antithesis architectures as a "hindsight is 20/20" kind of thing, but I'm open to learning more.
carlsverre 15 hours ago [-]
Great question! The general approach we take with transactional systems like this is to put reachability statements throughout the complex stateful machinery and then stress-test them in Antithesis.
The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!).
Said differently, this is exactly the approach we take with all stateful transactional systems. I am only sad that I didn't do this experiment months earlier, as it would have saved Tailscale and the SQLite team a lot of time.
> This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!).
Suuure. So why exactly haven't you found this bug already a long time ago? I mean it is trivially obvious that this is where bugs in SQLite would be, right?
SQLite is open source and is one of the most popular databases, surely testing it to find bugs would also be obvious?
carlsverre 14 hours ago [-]
We put many things to the test, but it costs money, and until very recently we haven't had a way to do this without a lot of human oversight. Now that we have https://github.com/antithesishq/antithesis-skills and LLMs are getting pretty reliably good at writing workloads, we are looking at ways to open up our technology to the OSS community. We have already started down that path with Etcd last year (https://etcd.io/blog/2025/autonomus_testing_with_antithesis/) and are working with various OSS groups to put more systems under test in a responsible way.
I'm not gonna lie, though, you're hitting a sore point. I'm absolutely kicking myself for not putting SQLite under test when I joined. The first few projects I put under test after joining were all projects that had a few properties:
1. The project had a path to a commercial deal
2. The project likely had interesting bugs to find
TBH, I didn't expect that one of the most widely used and well-tested pieces of software in existence would have a bug our fuzzer could find in 15 mins. (To be clear, this is not a hit on SQLite, just that our fuzzer is stupid good at finding race conditions. SQLite remains one of my favorite pieces of software ever.)
With this in mind, are there other similar projects that I'm not thinking of that I should put under test? Before you ask, I'm already scaling up my SQLite harness to cover more of its state space and run against master. Maybe I'll find something novel in SQLite after all! Never too late to check.
skybrian 14 hours ago [-]
Other embedded databases like Dolt? DuckDB? git?
carlsverre 11 hours ago [-]
We've started testing git and DuckDB! Haven't played much with Dolt, though. We have some initial findings on Git that we are validating and will hopefully be making public soon!
awwaiid 11 hours ago [-]
Matrix, specifically synapse/rustMatrixSDK :)
wps 10 hours ago [-]
Test out Fossil SCM, it depends on SQLite and is also written in c89.
wwilson 14 hours ago [-]
Brutally honest answer: it only recently became easy to get random systems under test with Antithesis without a bunch of manual/expert effort. Expect us to be reporting a lot of bugs against open source projects in the near future.
NortySpock 9 hours ago [-]
I've been jealous of the work antithesis is doing and support you in spirit; the world needs deterministic, tested software.
If you want a challenging open source deterministic game to test, may I suggest Beyond All Reason, powered by the Spring / Recoil engine? It's an RTS game that's been in various incarnations of development for 20+ years. (Disclosure: I'm an occasional code contributor to the project).
carlsverre 8 hours ago [-]
Oh man, Beyond All Reason would be a killer demo. We are actually testing another RTS as we speak, but I won't spoil it for you :) Look forward to a really epic post coming soon.
HackerThemAll 37 minutes ago [-]
So you got information where the bug is, and now you brag that your AI friend found it quickly :joy:
Go find bugs that haven't been yet found, and then brag about them :joy:
skybrian 14 hours ago [-]
Nice article, but could you get someone to fix the blog layout? Having the author card pinned to the window rather than being able to scroll past it is distracting.
jauntywundrkind 6 hours ago [-]
you asserted that Antithesis would have found it right away, but it seems like from your write-up that you first set up instrumentation on the source, then told it to reproduce the WAL bug.
i don't feel like i gained a strong appreciation for what the actual Antithesis capabilities are or why you had confidence this was anything the Antithesis tools would have helped find? i want to know more about what kind of analysis or capabilities Antithesis has and how it could have helped find the issue. what's clear is that Antithesis could confirm an already found issue. and even there I don't feel like I gained any connection or knowledge or understanding of what the tooling was doing or how it would help. i'd really like more technical meat to this post to understand the value proposition.
nemothekid 8 hours ago [-]
I understand the author feels he missed an opportunity to catch a bug in Sqlite - but this still feels like catching something in hindsight. Just knowing the bug the is in Sqlite's concurrency engine is a huge win.
But reading over tailscale's blogpost, I'm not sure how I would connect the dots to "this issue I'm seeing in production" to "load this up in antithesis". It seemed half the battle was figuring out if the bug was in Sqlite, Tailscale, or Linux, and in that situation if you gave Antithesis all the context, would it reliably find the bug?
stavros 44 minutes ago [-]
Maybe it wouldn't, but being able to cheaply test SQLite within a few minutes is still very valuable, even if just to rule it out.
xyzsparetimexyz 16 minutes ago [-]
Omg the one thing I hate more than ai art is ai pixelart where the pixels are all different sizes
moomin 3 hours ago [-]
I feel like we’re in a weird P=NP age, where after someone finds an issue, someone points out a cheaper way to find the same issue, which inevitably involves knowing the issue is there in the first place and it’s approximate location.
uhohherewegoaga 15 hours ago [-]
as a long time lurker who usually enjoys antithesis posts, this was a pretty existentially sad read. we all know that company blog posts here always tend to be ads at their core, but historically afaicr antithesis generally toes the line well of keeping the "thing we sell is really good" as a side dish to the "i (author) did very interesting deep thinking and/or hard work to accomplish something cool" entree.
this one falls very flat on that front imo. the "antithesis finds rare bugs" is the overwhelming majority of the actual content here, where as the usual meat of the blog, the hard work, the deep thinking, really distills down to "typed this into claude mobile, typed that into claude mobile". yes, its a statement to the power of the technologies of today that's all it takes to find this bug, but man it's a sad signal for those of us looking for compelling technical reads instead of b2b sales pitches.
minimaltom 15 hours ago [-]
I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite.
I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test.
The closest I mention of the prompt I could find was:
> Then I asked it to write a simple workload which exercised the WAL insert and checkpoint code. Notably, this is a completely generic workload.
With a skeptical lens, unclear.
carlsverre 15 hours ago [-]
Hey, author here! Good question. The prompt was to use the Antithesis skills[1] to write a workload that exercises SQLite's insertion and checkpoint machinery across multiple processes using a specific version of SQLite (3.51.2). The agent who did the work was aware of the bug and knew I wanted to see if Antithesis could reproduce it. However, I specifically designed the workload the same way we approach testing other WAL checkpoint-style systems: by putting them under stress. The workload itself does not know about the bug.
My goal, which I believe I accomplished, was to determine if Antithesis would have found this bug if we had put the SQLite WAL under test.
Which is: "Given the knowledge of a kind of bug that can happen in one Raft implementation, can a similar workload find similar bugs in other Raft implementations?" Turns out it can.
minimaltom 14 hours ago [-]
Yeah 100%! And I'm sorry if I sound a little more critical and less eager, its just thats theres a world of difference between a priori finding the bug, and reproducing it, and the impression of the article (from my read) was the former.
But please keep writing, I know its super hard to put yourself out there and make content!
carlsverre 14 hours ago [-]
Thank you! I appreciate the note of support! I'd love to write more blog posts in the future.
gamegoblin 15 hours ago [-]
Their founder guy says[1] they found some other bugs while doing this that they will report soon, so if that is true, seems more plausibly like a pretty generic thing.
Looking forward to seeing the other bugs they found.
Which says they made two commits to make instrumentation work. Also, no fixes (which seems reasonable to me; need to figure out what's wrong first)
Presumably any bugs found are being sent upstream to fix.
carlsverre 15 hours ago [-]
Those are the commits from upstream that resolved the bug. Notice that most of them were committed by Dr. Richard Hipp himself.
biorach 15 hours ago [-]
oh! damn! I did not notice that. That does answer my question.
15 hours ago [-]
wwilson 16 hours ago [-]
We wanted to publish this ASAP since people are talking about this bug today. In a follow-up, we will show how our automated causality analysis could have saved Tailscale and SQLite from 6 months of root cause analysis. Stay tuned!
Carl (author) here. This is a poor reading of the blog post. Notably, I built a general-purpose workload that simply exercises the write/checkpoint machinery. The exact same workload we use to break all kinds of transactional systems. This generic workload found the bug without any special knowledge of the bug, due to how Antithesis works.
returningfory2 15 hours ago [-]
I think the issue is that you knew there was a bug to be found in the write and checkpoint interactions, which then determined which workload you built.
My understanding is that this workload is very uncommon: the Tailscale blog says they used a custom unusual configuration to have many checkpoints like this. So without knowing the bug, it seems unlikely one would build this workload and then find the bug. If that makes sense.
Edit: just want to say that you being able to repro it is awesome, but that the overall claim seems a little overstated to me.
carlsverre 15 hours ago [-]
I can see how the post comes across that way. I may need to edit it to somehow be clearer about how I approached the workload development. I'll refer you to my other replies to comments for more information on my approach:
Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.
quadrifoliate 3 hours ago [-]
I do feel like you're being ganged up on as well. IMO this is a very cool thing that Antithesis is able to find and reproduce this bug in this specific subsystem with a relatively generic workload.
I guess what would have been an even more cool thing is "we ran some more general testing with Antithesis, and it found five other bugs". Have you thought along those lines or explored something like that? There have to be other, similar bugs lurking in SQLite :)
karagenit 11 hours ago [-]
I’m sorry you’re feeling ganged up on, I thought it was a nice article and cool you were able to repro the bug with your tools.
Like some others have mentioned, one of my earliest thoughts was “how much of a hint was the LLM given about the bug?” I think if the prompt used was stated clearly/verbatim near the beginning of the article, that would probably dispel a good amount of the criticism.
grebc 16 hours ago [-]
I read your ad.
It states you replicated the bug once the SQLite team fixed it, and published it.
Not sure what’s difficult about replicating behaviour when it’s spelled out for you.
biorach 16 hours ago [-]
this is a deliberate mischaracterisation of the content of the article
grebc 15 hours ago [-]
You’re mischaracterising an ad, that’s also funnily enough full of it.
Mawr 15 hours ago [-]
Okay, so use the same methodology to find a new, unknown bug? That'd be actually impressive.
As-is, this is just 20/20 hindsight with concerns about leading the AI on through the prompt hand-waved away. Come on.
wwilson 16 hours ago [-]
As the post (and linked repo) pretty clearly indicate, zero info about the existence of the bug went into this work.
Oh btw, we also found some other ones… Stay tuned!
grebc 16 hours ago [-]
So you found this before SQLite published the fix?
I'm only sad that I didn't put SQLite under test earlier in the year, or I would have found this issue right away. If you look at the workload[1], you can see how simple it is. Exactly the same kind of workload we write every day to help our customers find bugs.
If you have any questions about our process or how debugging with Antithesis works, please let me know! Thanks for reading!
[1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...
The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!).
Said differently, this is exactly the approach we take with all stateful transactional systems. I am only sad that I didn't do this experiment months earlier, as it would have saved Tailscale and the SQLite team a lot of time.
[1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...
Suuure. So why exactly haven't you found this bug already a long time ago? I mean it is trivially obvious that this is where bugs in SQLite would be, right?
SQLite is open source and is one of the most popular databases, surely testing it to find bugs would also be obvious?
I'm not gonna lie, though, you're hitting a sore point. I'm absolutely kicking myself for not putting SQLite under test when I joined. The first few projects I put under test after joining were all projects that had a few properties: 1. The project had a path to a commercial deal 2. The project likely had interesting bugs to find
TBH, I didn't expect that one of the most widely used and well-tested pieces of software in existence would have a bug our fuzzer could find in 15 mins. (To be clear, this is not a hit on SQLite, just that our fuzzer is stupid good at finding race conditions. SQLite remains one of my favorite pieces of software ever.)
With this in mind, are there other similar projects that I'm not thinking of that I should put under test? Before you ask, I'm already scaling up my SQLite harness to cover more of its state space and run against master. Maybe I'll find something novel in SQLite after all! Never too late to check.
If you want a challenging open source deterministic game to test, may I suggest Beyond All Reason, powered by the Spring / Recoil engine? It's an RTS game that's been in various incarnations of development for 20+ years. (Disclosure: I'm an occasional code contributor to the project).
Go find bugs that haven't been yet found, and then brag about them :joy:
i don't feel like i gained a strong appreciation for what the actual Antithesis capabilities are or why you had confidence this was anything the Antithesis tools would have helped find? i want to know more about what kind of analysis or capabilities Antithesis has and how it could have helped find the issue. what's clear is that Antithesis could confirm an already found issue. and even there I don't feel like I gained any connection or knowledge or understanding of what the tooling was doing or how it would help. i'd really like more technical meat to this post to understand the value proposition.
But reading over tailscale's blogpost, I'm not sure how I would connect the dots to "this issue I'm seeing in production" to "load this up in antithesis". It seemed half the battle was figuring out if the bug was in Sqlite, Tailscale, or Linux, and in that situation if you gave Antithesis all the context, would it reliably find the bug?
this one falls very flat on that front imo. the "antithesis finds rare bugs" is the overwhelming majority of the actual content here, where as the usual meat of the blog, the hard work, the deep thinking, really distills down to "typed this into claude mobile, typed that into claude mobile". yes, its a statement to the power of the technologies of today that's all it takes to find this bug, but man it's a sad signal for those of us looking for compelling technical reads instead of b2b sales pitches.
I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test.
The closest I mention of the prompt I could find was:
> Then I asked it to write a simple workload which exercised the WAL insert and checkpoint code. Notably, this is a completely generic workload.
With a skeptical lens, unclear.
My goal, which I believe I accomplished, was to determine if Antithesis would have found this bug if we had put the SQLite WAL under test.
[1]: https://github.com/antithesishq/antithesis-skills
1. The specific bug isnt mentioned 2. (If youre game) a model with a knowledge-cutoff date before the report is used
Which is: "Given the knowledge of a kind of bug that can happen in one Raft implementation, can a similar workload find similar bugs in other Raft implementations?" Turns out it can.
But please keep writing, I know its super hard to put yourself out there and make content!
Looking forward to seeing the other bugs they found.
1. https://news.ycombinator.com/item?id=49278351
I'd be interested in seeing what the SQLite people have to say about them if/when they are submitted upstream
Which says they made two commits to make instrumentation work. Also, no fixes (which seems reasonable to me; need to figure out what's wrong first)
Presumably any bugs found are being sent upstream to fix.
My understanding is that this workload is very uncommon: the Tailscale blog says they used a custom unusual configuration to have many checkpoints like this. So without knowing the bug, it seems unlikely one would build this workload and then find the bug. If that makes sense.
Edit: just want to say that you being able to repro it is awesome, but that the overall claim seems a little overstated to me.
https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278521
Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.
I guess what would have been an even more cool thing is "we ran some more general testing with Antithesis, and it found five other bugs". Have you thought along those lines or explored something like that? There have to be other, similar bugs lurking in SQLite :)
Like some others have mentioned, one of my earliest thoughts was “how much of a hint was the LLM given about the bug?” I think if the prompt used was stated clearly/verbatim near the beginning of the article, that would probably dispel a good amount of the criticism.
It states you replicated the bug once the SQLite team fixed it, and published it.
Not sure what’s difficult about replicating behaviour when it’s spelled out for you.
As-is, this is just 20/20 hindsight with concerns about leading the AI on through the prompt hand-waved away. Come on.
Oh btw, we also found some other ones… Stay tuned!