I think I'm done with the redis filesystem for the moment. It does everything I need it to do, although I am curious to see how much faster it could go if it were to use non-blocking writes that isn't a major concern.
There is only one missing feature I'm planning to play with - and that is the ability to implement snapshots.
As a refresher redis is a key+value store, which mostly uses system memory. I've built a simple FUSE filesystem on top of that (now with symlink support!) and so all the file contents, meta-data, and similar is stored in memory.
Implementing snapshots could be done by two different routes:
- Copying all keys and their values, under a new name.
For example right now, by default, all the filesystem entries in the root directory are stored beneath the key "SKX:/" - where "SKX" is the key prefix.
Assume I copy each existing key, and the associated value(s), giving them a new prefix such as "SKX2:" I can mount the filesystem against that prefix - and we've got a point-in-time snapshot.
- Serialising all keys & values
Redis has a primitive which allows you to determine the names of keys at runtime. Given that all my filesystem keys have a prefix ("SKX:" by default) it wouldn't be difficult to find them, and serialise them.
This would require more effort to re-import and re-mount, but it should be portable across hosts.
Anyway assuming I get this right we'll have a filesystem which is replication-friendly and snapshot-able. A fun combination.
ObQuote: "We had been everywhere. We had really seen nothing. " - Lolita