joshaber / freezer Goto Github PK

We have a date attribute type, but the object we get out isn't a date. 😢

It'd be useful/cool to be able to say, "give me the database from yesterday", etc.

Currently the sqlite database just keeps adding things. That's less than ideal. We should have some way of trimming it.

Some things:

• We could trim down to the most recent level attribute-value for all keys.
• But it's tricky to know that someone's not holding on to an old database.
• How does this interact with #3?

Currently, getting a collection of size n requires (at least) n * 2 selects.

That's super gross.

Just laziness on my part, but we need to fix it.

Sqlite's awesome, LevelDB might be more awesome.

👍

• Faster than Sqlite
• Snapshots are a non-persistent version of our immutable databases.
• More thread-safety out of the box.
• Sorted keys could allow for a bit more flexibility.

👎

• Outside dependency.
• Snapshots can't be persisted.
• C++

Transactions exist currently in the database, but we don't expose them to users. It could be useful:

• Timestamp for changes.
• List of changes.

Related to #3.

Currently each attribute is its own table. We could instead implement each attribute as a column on one table. This could help address #5.

The downside is that we'd probably end up with a super wide table. I'm not sure how Sqlite feels about that.

Since FRZDatabases are immutable, it'd be super easy to cache lookups in process. Should we?

We should run VACUUM. Probably as part of trimming.

Reads and writes require that we look up attribute information (type, cardinality, etc.). We should cache that.

We'd get this for free if we did #8.

Probably easiest to leverage -keysWithAttribute and use some attribute as a belongs-to-this-collection tag.

We currently don't use any indexes. It'd probably make sense to index on key. Maybe attribute too?

Should Freezer provide more sophisticated querying?

• Most query APIs are heavily stringly typed.
• Could we use Sqlite functions to get more typing?

In light of #5, #12, #19, I think it'd make sense to move back to using a single table.

👍

• Simper, more consistent implementation.
• Faster key-based reads.

👎

• The value column will just be a blob, making it harder for us to leverage SQLite magic.
  • We could maybe mitigate that with custom functions which unpack the blob?
• Less space-efficient, presumably.

Add a way for users to remove an attribute.

Currently key-based reads require walking over each of attribute tables and checking if they have an entry for the key. This means it's gonna slow down as you add more attributes.

A couple thoughts on speeding it up:

• Partition keys so that we can known based on the they that it could only have certain attributes.
• Keep an index of the attributes which a key has.

From @mdiep.

I think the dissonance for me is that it feels like I'm mutating a dictionary, but the names don't convey that

I agree. It's tricky because dictionaries are key -> value, but Freezer's more like key -> (attribute, value).

From @mdiep.

It seems like a good idea to make it clearer what's happening.

Currently on both branches, inserting is O(n), n = number of objects in the previous "database." This is because they create a new list of objects for each insert. That's... less than ideal.