[question] compiler plugin/virtual table functionality? about joedb

[Remi-Coulom]

Hi,

Thanks for your interest in joedb.

I am not familiar with capn, and I am not really sure what you mean by a "virtual table" interface. From what I understand, my impression is there are two possible approaches:

1. Store the capn data as a string in joedb, and use capn software to manipulate it.
2. Convert the capn schema into a joedb schema, and manipulate everything as joedb data.

I don't wish to incorporate any capn-specific tool into the core of joedb. Supporting other data formats should be done as a separate independent tool, like I did for json and SQL exports. I guess it should be possible to produce a separate capn-to-joedb compiler that would produce a joedb schema, as well as convenient functions to insert a capn buffer into the corresponding joedb table.

I certainly don't have time or motivation to work on capn myself. But if you develop the tool you need by yourself, I'd be glad to link to it, or help you with any joedb-related question you might have. I am ready to contribute coding effort only if it improves joedb in a way that is independent of capn.

[liamstask]

Thanks for your response.

For the purposes of this discussion, capn proto is similar to protobuf - a message definition format with an encoding/decoding specification.

By 'virtual table' I have in mind something like the virtual table mechanism of sqlite. This would allow data to be stored in joedb in native capn proto or protobuf encoding (as a blob or string field as you suggest), but an implementation of this virtual table mechanism could be used to interpret that data in order to respond to standard joedb queries/accessors (avoiding the conversion to joedb data).

To be clear, no capn proto or protobuf specific functionality would be implied in joedb - the main requirement would this virtual table interface, which could be implemented for any interesting external data type.

Anyhow, I realize this would be potentially be a significant effort, but thought I would check to see if it's interesting, in which case I could explore ways to contribute.

[Remi-Coulom]

Thanks for explaining more. I understand better what you mean now.

For the future of joedb, I have in mind to offer additional storage options for tables. Right now, the only options are in-memory vectors of structs, or freedom keepers. I'd like to be able to also store data on disk in leveldb databases (or, why not, SQL databases), and make a generic plug-in system for storage options of joedbc. This would allow alternative storage options.

This would be a bit different from the virtual table mechanism of sqlite, because the virtual table is dynamic and interpreted, but the spirit is a bit similar, in a static, compiled approach. I'd call this a "storage engine" rather than a virtual table.

Such a storage engine could use the capn format, but it would be abstracted away from the top-level C++ interface generated by joedbc. So you would not be able to directly access the capn buffers. It might be possible to add a shortcut to the low-level storage, but that looks a bit dirty.

Anyway, as you wrote, this is a very significant programming effort, and I will not be able to do it in a long time. Unless you are willing to develop the storage engine framework of joedbc yourself, it does not seem possible in the short term.

If you really want to use joedb, I guess the simplest approach is to store your data in joedb format as I suggested in my previous message: write your own compiler that produces a joedb schema from a capn schema, as well as a function that uses the capn api to interpret capn buffers and store them in the compiled joedb database. It should be possible to do it without any change to joedb.

[liamstask]

Sounds good 👍 I think that solution would be close to ideal for my use case. I'll explore the current joedb implementation further, and keep an eye out for storage engine work in the future - thanks!