Faceting/Dynamic indexing (with MVs) about icedb

[danthegoodman1]

This becomes less useful the more merged files become honestly. E.g. if reducing a day to 3 files, then something probably appears in all 3 files, and unless we pull it out into its own column it's a full-scan of the files anyway.

[danthegoodman1]

Ultimately this would be more useful if we had lower-level control over the parquet file reading. For example if we could say it lives in {...} row groups in files {...} then we could just directly to those byte ranges, but we don't have this low-level control within duckdb or clickhouse to read that way. A custom parquet reader could probably do this, which could be written in python.

[danthegoodman1]

Instead of using a JSONB column and a GIN index, we could make a table IS and index. That way we could do sparse-indexing and probably be far more efficient with both size and cardinality.

[danthegoodman1]

moving to example as this is more a way to use icedb rather than a feature, and also really is just a materialized view because data should be copied to most efficiently handle this

[danthegoodman1]

Closed as these are just materialized views