fosstodon.org is one of the many independent Mastodon servers you can use to participate in the fediverse.
Fosstodon is an invite only Mastodon instance that is open to those who are interested in technology; particularly free & open source software. If you wish to join, contact us for an invite.

Administered by:

Server stats:

11K
active users

AlexMillerDB

@penberg, does “libSQL will always be compatible with the SQLite file format” mean that adding checksums for on-disk data is fundamentally not a part of your roadmap because it’d break SQLite file format compatibility? Or should I interpret this as libsql can read SQLite, but SQLite might not be able to read libsql?

@AlexMillerDB We mean the latter: you should always be able to read and write your SQLite databases with libSQL, but there indeed may be opt-in libSQL extensions that won't work with SQLite. Checksums seem like a great example of something like that.

@penberg @AlexMillerDB in case of checksums, it would also be worth considering if it can be added in a way that's still readable by SQLite. E.g. page checksums can easily be stored in a separate file, and per-row checksums could be visible to SQLite as an additional regular column.

@penberg @sarna I’d feel a bit weird about the concept of “per row” checksums, as you have to decode the page to get to that point, but it would be a reasonable middle ground which still prevents returning corrupted data to users. SQLite wouldn’t update the separate file (or row, unless trigger?), so it’d still leave interop in a weird place.

Maybe it’d be reasonable to offer a codec implementation for SQLite which mirrors the same page checksum logic? (FoundationDB did checksums as a codec.)

@penberg @sarna in any case, I bring this up because my largest complaint about SQLite has been how it completely disavows any responsibility to verify the OS and hardware did their jobs right, even though we have decades of evidence to argue that corruption is a real issue. And especially since SQLite is often running specifically on consumer grade (or worse) storage, it’s just… extra disappointing to me. Dqlite, rqlite, etc, all transitively and ironically inherit this flaw as well…

@AlexMillerDB @penberg I agree, full interoperability will be hard to achieve. I think a good compromise is to write new potentially incompatible code in a way that can be read from SQLite if need be, but should be written via libSQL.

@AlexMillerDB @penberg @sarna

> I’d feel a bit weird about the concept of “per row” checksums

Could be useful to provide end-to-end guarantees from user perspective. E.g. row did not get corrupted in transition / on compaction / etc.

Good discussion of that in "End-to-end arguments in system design" (1984) paper

web.mit.edu/Saltzer/www/public

Oops!An unexpected error occurred.