writes and claim/ack flow. really depends on your journal mode and synchrnous mode as well.
notifs are extremely cheap, either in the old stat(2) mode or the new PRAGMA page_version (see my update on feeback comment). Some other comments mentioned that stat(2) is about 1µs.
The PR implements a three-layer polling architecture:
- PRAGMA data_version every 1ms
- stat every 100ms
- retry connection to handle blips
1. PRAGMA data_version every 1ms replaces stat-based (size, mtime) change detection. This is SQLite's own commit counter: monotonic, immune to clock skew, correctly handles WAL truncation and rolled-back transactions. ~3µs nonblocking query. Credit to ncruces for pointing to this. This is not done for performance but for correctness as it is slightly slower. tuo-lei also pointed out truncation risk, which turned out to be more real than i thought.
Interesting note: I found in testing that the C API's SQLITE_FCNTL_DATA_VERSION does not work cross-connection. So for now honker continues paying the cost of going through the VFS layer which vlovich123 pointed out and now we tradeoff explicitly.
2. Reconnect-on-error: if the data_version query fails (disk blip, NFS hiccup, corrupted connection), honker tries to reconnect and wakes subscribers as a precaution. zbentley pointed me in this direction.
3. stat identity check every 100ms: compares (dev, ino) against startup values to detect file replacement (atomic rename, litestream restore, volume remount). data_version can't catch this because it polls through the open fd, which follows the original inode even after replacement. Credit to zbentley for the file-replacement scenarios.
Again - thanks for the discussion, honker got better because of it and I learned some stuff. See you round
I actually looked at fstat, but the "check for deletions" piece, given I'm polling at 1kHZ, was the reason I decided not to use it. Older hardware actually made this a big issue but it's fast enough now I decided it wasn't a problem.
I'll ignore the malicious ones bc [out of scope declaration]. Object paranoia is an artifact of build trama and I respect that lmao.
I've just looked into the device number and system clock issues. I think what i'll end up doing is actually a combo of ncruces's above comment and your feedback: a 1kHZ data_version and a 10HZ stat() with version check. This gets around syscall load, avoid clock issues, avoids the WAL truncation issues that others have mentioned, and is both lighter weight and less bugabooable than my previous design.
One clarification: by "check for deletions" I didn't mean that you need to read back through the filesystem; you can check for deletions for free using fstat(2)'s result. The number of hard links to a file descriptor's underlying description returned by fstat includes the "existential" hard link of the file itself, and drops to zero when the file's deleted and the open handle is an orphan:
import os
import time
from threading import Thread, Event
f = '/tmp/foo.test'
ev = Event()
Thread(target=lambda: ev.wait() and os.unlink(f), daemon=True).start()
with open(f, 'w+') as fh:
print("before delete:", os.fstat(fh.fileno()).st_nlink)
ev.set()
time.sleep(1)
print("after delete:", os.fstat(fh.fileno()).st_nlink)
Damn it was real the whole time. I found Opus 4.7 to holistically underperform 4.6, and especially in how much wordiness there is. It's harder to work with so I just switched back to 4.6 + Kimi K2.6. Now GPT 5.5 is here and it's been excellent so far.
"a small proliferation" is a nice way to describe the cluster that is my side project habit. if you bump into any issues pls pull a PR or drop an issue on the repo!
Nope! The extension just functions as a shortcut for raw SQL. Litestream edits the wal file but only like a normal checkpoint. So not too bad. Although I haven’t tested it directly. Probably need to
See comment below - Darwin silently drops same-process notifs. I could change the behavior depending on same vs cross process and platform but I wanted to”just one thing to worry about”. Potentially a good optimization later. Would help reduce syscalls.
The specific thing I'm talking about is this: write events don't fire until the file handle is closed. [1] I didn't validate this myself btw, but my original design was certainly trying to use notify events rather than stat polling. My research (heavily AI assisted of course) led me away from that path as platforms differ in behavior and I wanted to avoid that.
If this has been fixed somewhere or there is a better alternative I'd love to use that over polling. Current plan is to move to polling data version for speed + occasional stat for safety. Getting rid of polling was my original goal but i compromised with syscalls.
I have no idea why they aren't using kqueue but that works on macOS and FreeBSD. It has for years.
You want EVFILT_VNODE with NOTE_WRITE. That's hooked up to VNOP_WRITE in the kernel, the call made to the relevant filesystem to actually perform the write.
notifs are extremely cheap, either in the old stat(2) mode or the new PRAGMA page_version (see my update on feeback comment). Some other comments mentioned that stat(2) is about 1µs.
reply