Rendered at 18:27:01 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
dangoodmanUT 15 hours ago [-]
> DbOptions::durable()
> Appended to the WAL without a per-write sync
So… it’s not durable? Durable doesn’t mean “survives a process restart”, it means “durably saved to persistent storage”. For example, this “durable” mode wouldn’t survive power loss.
nijave 3 hours ago [-]
I give a little leeway to distributed systems that replicate and don't flush since there's a bit of middle ground assuming they're in different fault domains. Garage object storage defaults to that
However, this doesn't appear to be the case here...
Unsurprisingly, performance goes to crap when sync is enabled.
Absolutely. If they just dirty some pages in memory and return back to the client the benchmarks will look "insanely fast".
I have nothing against this being a non-default option in a db/kv engine but anything advertising to be durable and not fsyncing by default is something I would stay away from. To me it's like a litmus test of how well the author knows/cares data durability and not destroying users data.
rgbimbochamp 46 minutes ago [-]
durable() syncs periodically on flush, WAL rotation, and clean close; paranoid() is the sync-before-ack mode. This is clarified in the README, and the benchmarks report all three modes separately. Other KV-stores that you see on the market, do this too. It's a performance tradeoff most applications make. Sync on every write kills every optimization. See the benchmark table for example.
procaryote 11 hours ago [-]
Pretty much... paranoid() seems to be the real durable() which isn't a great look for a database project.
Being able to recover a db without corruption beyound losing the last few writes is a pretty useful feature, and buys a lot of performance, but it would be better to label that clearly, as a reasonable expectation on the durable() preset would be for it to be Durable.
stingraycharles 14 hours ago [-]
Yeah this should be benchmarked against other systems that have flush() disabled.
mmap is nice but it doesn’t support durable semantics in the way that we usually mean with databases.
if a write is acknowledged it should not be forgotten, which is not what this is.
muvlon 6 hours ago [-]
As of a couple years ago, mmap actually has a MAP_SYNC flag that makes it durable in the DB sense. The caveat is that it requires DAX on the file and so comes with a whole bunch of restrictions w.r.t. filesystem, storage media and even CPU architecture.
rgbimbochamp 14 hours ago [-]
You're right, that mode provides process crash recovery, not power-loss durability. The benchmark compares it against fjall’s equivalent buffered-WAL mode.
deathanatos 11 hours ago [-]
Word choice matters. Defaults matter. People will go "well it says durable right here" and while arguably, yes, they should RTFM, it would still be great if tool-builders did not set the shotgun's default state to State::AT_FOOT. It would be nice if every paragraph of technical writing that I have to do need not be burdened by a thousand asterisks of "durable in this context means something other than durable".
carlmr 10 hours ago [-]
>arguably, yes, they should RTFM
Agreed. Good design is when the things do what you expect them to do without reading the manual, don't reuse wording with other meaning in the wrong way. That way if you do encounter nee wording, you know you should read the manual.
If that's your design constraint, couldn't you speed it up by getting rid of the WAL?
oneshadab 12 hours ago [-]
You'd lose durability against process crashes.
If your system has a reasonable tolerance for power failure (multi-az multi-cloud), this can provide much better throughput
t098i3 12 hours ago [-]
Indeed, a common enough pattern for etcd is to run it backed by a RAMdisk and have multi-az availability + periodic backups + tolerance at a business level to be OK losing some recent data.
deepsun 12 hours ago [-]
Are we back to MongoDB -- no fsync() but webscale speed?
This is surprisingly common, from what I can tell.
pjmlp 3 hours ago [-]
Most of these "insanely fast" projects kind of feel like people rediscovering compiled languages after the scripting languages dark ages.
Insanely fast was making 8 bit games possible at all.
xtracto 1 hours ago [-]
More or less. I think the use of CPU specific instructions can make an compiled program different than "the rest". Although nowadays some compilers are clever enough to do better than manual optimization.
KAdot 5 hours ago [-]
The benchmark is setup to test 80 MiB dataset on a machine with 32 GiB RAM, which doesn't represent a typical database workload. How does the key-value store perform on larger than RAM datasets? MMAP is fast when the dataset fits in memory, but it can slow to a crawl when it doesn't, especially if the workload is mostly random point lookups.
boguscoder 14 hours ago [-]
Embedded could also mean no_std, which this is absolutely not. Still cool though
LoganDark 13 hours ago [-]
Yep, the correct term here is "embeddable", not embedded.
If someone asked me what a DB is, I'd probably start the conversation with the exact description "a file-backed map, like a hashmap".
riffraff 12 hours ago [-]
Well, dbm (database manager) is basically that and has been called that for almost 50 years
otabdeveloper4 5 hours ago [-]
dbm isn't a database either, and people who call it that are simply wrong.
rcxdude 4 hours ago [-]
it's not a relational database, but those aren't the only kind.
howerj 1 hours ago [-]
In fact the name "relational database" implies there are other kinds.
mikedelago 7 hours ago [-]
Why not?
mattrighetti 9 hours ago [-]
I like the fact that the first commits were about the logo, important things first :D
DarmokTanagra 8 hours ago [-]
Off topic, but why is tokio still independent of the rust async runtime?
It seems pretty ubiquitous yet not a part of the core rust libs.
kshri24 6 hours ago [-]
I prefer compio over tokio. So would not want libs to depend on a library that is essentially a runtime. Runtimes should be swappable.
insanitybit 8 hours ago [-]
The same reason as ever. Not everyone wants to use the same runtime.
DarmokTanagra 8 hours ago [-]
Sure, but now if my rust application isn't using tokio I have to include it as a new dependency because the author of this lib decided to use it as his async runtime?
I'm not trying to be pedantic, but this split over async runtimes was what originally turned me off of rust years ago and it still seems to be an issue.
vlovich123 4 hours ago [-]
Because Tokio, while great for lots of things (ie default decent performance with ease of use), is a performance bottleneck if you want extremely high performance.
For example, the best this DB can do is ~3.7Mkeys/s for non durable writes and 162k/s for durable. I have an equivalent DB that’s always durable and does 30M/s* (for 8 byte entries) because it doesn’t use Tokio among other things. For this dataset it would be saturating the disk I/O no problem so I would expect it to be running ~7-14M/s depending on how fast your SSD is (~2-4x faster than non durable mode and 40-80x faster than its “durable”)
* it was running at 70-100mhz at one point but the challenge is keeping the hot path at ~10ns as you add features and other things.
DarmokTanagra 2 hours ago [-]
Thank you for a complete answer.
Out of curiosity where do you stand on libraries like this imposing a runtime choice on the user?
Is it standard in the rust ecosystem to have multiple async runtimes in a project?
vlovich123 2 hours ago [-]
It’s fine to have multiple runtimes. There’s probably some additional performance overhead at the API boundary because you have to transfer the work to run on the other runtime but I haven’t benchmarked what that looks like. It’s probably on the order of 50ns-1us if done naiively and depending on contention (ie you’re looking at a max throughput of 20M-1M/s).
Use whatever you want. Tokio is a very good work stealing runtime which is what Apple’s GCD popularized 17 years ago. It’s a fine model but sacrifices total throughput for ease of use and “easy” multithreading. Thread per core with pinned cores is what you use when you prioritize throughput and absolute possible latency. Tail latencies can suffer if you make a mistake and have imbalanced work on a single thread
timschmidt 7 hours ago [-]
I'm using async on esp32 with embassy, for instance, where there's simply no room and no need for a runtime as complex as tokio.
insanitybit 7 hours ago [-]
You would only have to include it if the library uses `spawn`, as far as I am aware, or some tokio specific type, which is the same as any other library.
airstrike 6 hours ago [-]
The author of this library can leave the async library choice up to the user.
iced does that quite well.
tinco 7 hours ago [-]
It's not pedantry, you're trying to find something in Rust that's simply not there. If this sort of thing turns you off on Rust you should be looking at a programming language with other priorities.
derriz 7 hours ago [-]
I’ve had the same reaction. We’ve seen this play out in other language eco-systems (Java - JAXP, javax.validation, JPA, etc all ended up with a de facto single implementation) and the idea of pluggable implementations sounds appealing but rarely pays off. The price that Rust paid for this abstraction - which in fact ended up not being useful as tokio is the only reasonable choice - was high in terms of requiring ugly (to my eyes) changes to its type system.
VorpalWay 7 hours ago [-]
Tokio is definitely not the only reasonable choice. I use embassy[1] on microcontrollers (the Pi Pico and ESP32 in my case), where tokio can't run.
There is no price to pay on the type system that was imposed by tokio. I assume you're saying something like "I have to add 'static in generics" or something?
It has absolutely paid off, there are many people not using tokio.
dist-epoch 8 hours ago [-]
How does it compare to RocksDB?
Surac 9 hours ago [-]
Fast compared to what?
rollulus 11 hours ago [-]
Now that “blazing fast in Rust” has become a meme, is “insanely” the next thing?
carlos-menezes 10 hours ago [-]
Insanely/ridiculously.
ivolimmen 4 hours ago [-]
Ludacris
Tepix 11 hours ago [-]
Is an atomic get+delete operation planned?
vlovich123 4 hours ago [-]
It’s usually very difficult in a KV db to have an efficient operation that returns the item deleted. You’d need a transaction API to do it reliably. The challenge is concurrent writes are impossible to serialize against without transactions.
This DB doesn’t have a transaction API.
nine_k 15 hours ago [-]
I suppose the insane speed is due to this:
> TurboKV's persisted Bloom-filter format uses hardware AES.
Also, built-in LZ4 compression.
I would expect SIMD to be used for scans.
haberman 15 hours ago [-]
I assume this is for hashing. I've seen several hashing algorithms turn to hardware AES instructions before, but I haven't seen any evidence that this technique outperforms state-of-the-art hashes like RapidHash (https://github.com/Nicoshev/rapidhash) in either quality or speed.
itemize123 7 hours ago [-]
that's optimizing a pretty fast already portion of code. unlikely it's the difference maker.
Sesse__ 10 hours ago [-]
For any complex system, there's never one single trick or design choice that makes it fast. It's always a large amount of engineering (or exaggerations, of course).
rgbimbochamp 15 hours ago [-]
Those help but the main write speed gain is the WAL, that uses preallocated mmap segments to avoid a write(2) per durable mutation while preserving crash recovery.
AES hashing mainly helps Bloom filter point lookups and LZ4 mainly helps SSTable I/O. Scans benefit indirectly, but don’t yet use a custom SIMD merge loop.
bestouff 14 hours ago [-]
I said elsewhere this doesn't survive a power loss.
rgbimbochamp 12 hours ago [-]
paranoid() does survive power loss.
vlovich123 4 hours ago [-]
And has performance I believe worse than rocksdb
taneq 14 hours ago [-]
While it’s important to make this explicit, at what point do we just assume a high-reliability UPS is table stakes?
Of course, if you need SIL2 type reliability then you need to assume any given hardware component can spontaneously combust and become a total loss, at which point the data loss caused by a power cut is a rounding error.
toast0 12 hours ago [-]
> While it’s important to make this explicit, at what point do we just assume a high-reliability UPS is table stakes?
Several years after they become commercially available?
My experience with small UPSes is they tend to cook the batteries and you don't find out until they switch the load and the battery doesn't hold up.
Large facility UPSes tend to do better, but automatic transfer switches have a tendancy to fail ocassionally. If you're hosted in many locations, it's not unusual to have a couple ATS failures per decade.
All that said, unexpected power loss is certainly one reason that writes may be lost, but OSes crash too. Disk firmware can also crash, but if thst bricks the disk, writes in progress don't really matter. Sometimes cabling fails. Or you get a uncorrectable ECC error (which will typically cause an OS panic... unless you're running a very fancy OS, but if it's in dirty disk backed page, even a fancy OS wouldn't save you)
Plenty of applications don't need or want to pay the cost for full commit to disk, but calling something durable when it's not committed to disk is inaccurate.
And that's before we get into the whole thing where the OS and the disk like to return success when things haven't quite finished.
whilenot-dev 13 hours ago [-]
What's got this to do with a UPS? Not having a UPS is an external threat on the reliability of the power grid.
Doing a hard shutdown or tripping over power cords seem much likelier local scenarios than any spontaneous combustion of hardware components.
bestouff 2 hours ago [-]
If it doesn't survive a power loss it may not survive a kernel crash.
zbentley 6 hours ago [-]
A UPS won’t save you from a kernel panic.
imtringued 9 hours ago [-]
Not sure how you managed to do it but you got it completely backwards.
If someone demands that the database should use fsync and only respond with success once the write finished, it is not some arbitrarily high reliability demand that needs to be implemented using reliable hardware. In fact, the entire point of implementing the power loss protection in software is so that you don't need perfectly reliable hardware. The power loss event turns into a downtime event which is often completely acceptable.
The requirement to have infallible hardware only emerged because the software refused to do its job. Infallible hardware is not a requirement decided by the user, it's a requirement decided by the developer of TurboKV to intentionally restrict his software to exclusively operate in a reliable hardware environment.
The fact that the user specified durability of the KV store during power loss does not make the user obsessed over hardware reliability, the software shifted the burden onto the hardware and forced the user to deal with this mess.
I don't know how exactly TurboKV works so let's talk about a hypothetical software instead.
Let's say the software cannot survive a power loss event and just corrupts the database. If the user wants to operate the software, he is forced by the software to operate it in an infallible environment where power loss can never occur. Based on how the software was designed, power loss is a catastrophic event. The SIL2 type reliability you're talking about only makes sense in contexts with catastrophic events.
So how it went is that the user made a reasonable demand with bounded reliability: "please survive power loss with durable writes" and the author says, sure just run the software on a SIL2 type reliability hardware environment.
It's not the user who blew up the hardware requirements.
> Appended to the WAL without a per-write sync
So… it’s not durable? Durable doesn’t mean “survives a process restart”, it means “durably saved to persistent storage”. For example, this “durable” mode wouldn’t survive power loss.
However, this doesn't appear to be the case here...
Unsurprisingly, performance goes to crap when sync is enabled.
This is pretty old now but has some useful fsync/sec numbers which can be completely divorced from other I/O performance https://www.percona.com/blog/fsync-performance-storage-devic...
I have nothing against this being a non-default option in a db/kv engine but anything advertising to be durable and not fsyncing by default is something I would stay away from. To me it's like a litmus test of how well the author knows/cares data durability and not destroying users data.
Being able to recover a db without corruption beyound losing the last few writes is a pretty useful feature, and buys a lot of performance, but it would be better to label that clearly, as a reasonable expectation on the durable() preset would be for it to be Durable.
mmap is nice but it doesn’t support durable semantics in the way that we usually mean with databases.
if a write is acknowledged it should not be forgotten, which is not what this is.
Agreed. Good design is when the things do what you expect them to do without reading the manual, don't reuse wording with other meaning in the wrong way. That way if you do encounter nee wording, you know you should read the manual.
If your system has a reasonable tolerance for power failure (multi-az multi-cloud), this can provide much better throughput
Insanely fast was making 8 bit games possible at all.
https://github.com/ncruces/go-sqlite3
And I'm still having fun.
https://github.com/ncruces/go-sqlite3/pull/421
https://labs.tomasino.org/gnu-recutils/
It seems pretty ubiquitous yet not a part of the core rust libs.
I'm not trying to be pedantic, but this split over async runtimes was what originally turned me off of rust years ago and it still seems to be an issue.
For example, the best this DB can do is ~3.7Mkeys/s for non durable writes and 162k/s for durable. I have an equivalent DB that’s always durable and does 30M/s* (for 8 byte entries) because it doesn’t use Tokio among other things. For this dataset it would be saturating the disk I/O no problem so I would expect it to be running ~7-14M/s depending on how fast your SSD is (~2-4x faster than non durable mode and 40-80x faster than its “durable”)
* it was running at 70-100mhz at one point but the challenge is keeping the hot path at ~10ns as you add features and other things.
Out of curiosity where do you stand on libraries like this imposing a runtime choice on the user?
Is it standard in the rust ecosystem to have multiple async runtimes in a project?
Use whatever you want. Tokio is a very good work stealing runtime which is what Apple’s GCD popularized 17 years ago. It’s a fine model but sacrifices total throughput for ease of use and “easy” multithreading. Thread per core with pinned cores is what you use when you prioritize throughput and absolute possible latency. Tail latencies can suffer if you make a mistake and have imbalanced work on a single thread
iced does that quite well.
[1] https://embassy.dev/
It has absolutely paid off, there are many people not using tokio.
This DB doesn’t have a transaction API.
> TurboKV's persisted Bloom-filter format uses hardware AES.
Also, built-in LZ4 compression.
I would expect SIMD to be used for scans.
Of course, if you need SIL2 type reliability then you need to assume any given hardware component can spontaneously combust and become a total loss, at which point the data loss caused by a power cut is a rounding error.
Several years after they become commercially available?
My experience with small UPSes is they tend to cook the batteries and you don't find out until they switch the load and the battery doesn't hold up.
Large facility UPSes tend to do better, but automatic transfer switches have a tendancy to fail ocassionally. If you're hosted in many locations, it's not unusual to have a couple ATS failures per decade.
All that said, unexpected power loss is certainly one reason that writes may be lost, but OSes crash too. Disk firmware can also crash, but if thst bricks the disk, writes in progress don't really matter. Sometimes cabling fails. Or you get a uncorrectable ECC error (which will typically cause an OS panic... unless you're running a very fancy OS, but if it's in dirty disk backed page, even a fancy OS wouldn't save you)
Plenty of applications don't need or want to pay the cost for full commit to disk, but calling something durable when it's not committed to disk is inaccurate.
And that's before we get into the whole thing where the OS and the disk like to return success when things haven't quite finished.
Doing a hard shutdown or tripping over power cords seem much likelier local scenarios than any spontaneous combustion of hardware components.
If someone demands that the database should use fsync and only respond with success once the write finished, it is not some arbitrarily high reliability demand that needs to be implemented using reliable hardware. In fact, the entire point of implementing the power loss protection in software is so that you don't need perfectly reliable hardware. The power loss event turns into a downtime event which is often completely acceptable.
The requirement to have infallible hardware only emerged because the software refused to do its job. Infallible hardware is not a requirement decided by the user, it's a requirement decided by the developer of TurboKV to intentionally restrict his software to exclusively operate in a reliable hardware environment.
The fact that the user specified durability of the KV store during power loss does not make the user obsessed over hardware reliability, the software shifted the burden onto the hardware and forced the user to deal with this mess.
I don't know how exactly TurboKV works so let's talk about a hypothetical software instead.
Let's say the software cannot survive a power loss event and just corrupts the database. If the user wants to operate the software, he is forced by the software to operate it in an infallible environment where power loss can never occur. Based on how the software was designed, power loss is a catastrophic event. The SIL2 type reliability you're talking about only makes sense in contexts with catastrophic events.
So how it went is that the user made a reasonable demand with bounded reliability: "please survive power loss with durable writes" and the author says, sure just run the software on a SIL2 type reliability hardware environment.
It's not the user who blew up the hardware requirements.