joseph-isaacs
Summary: support time series data, i.e. tables sorted by a timestamp column with high-volume appends, range queries over time, and time-bucketed aggregations. ### Sortedness - [ ] Record a per-file sort key in the footer and fill `is_sorted` / `is_strict_sorted` in file stats. - [ ] Binary search or zone seeking on the sort key for time-range predicates. - [ ] Reverse scan for newest-first queries. - [ ] Infer DataFusion `output_ordering` from the file sort key instead of requiring `WITH ORDER`, and support ordering on the v2 table. - [ ] Let the compressor use `IsSorted` to pick Sequence or Delta without trial encodes. ### Aggregate pushdown - [ ] Implement partial aggregation pushdown support from `DataFusion` into vortex. ### Filter and limit pushdown - [ ] Allow filter and limit in the same scan and short-circuit once the limit is met. - [ ] Push limit from DataFusion when a filter is present. - [ ] Push TopK and join dynamic filters into zone pruning, matching DuckDB. - [ ] Push Interval and Duration literals and `now() - interval` style predicates from DataFusion. ### Better timestamp and other types - [ ] Add Duration and Interval dtypes and fix the Arrow mappings that currently panic. - [ ] Cast between timestamp units. - [ ] Coerce time units in comparison and pruning instead of erroring. - [ ] Finish DateTimeParts comparison kernels so sub-day range filters do not fall back to a full decode. ### New scalar functions - [ ] `date_trunc`, `date_bin`, `date_part` / `extract`, timestamp plus or minus interval. - [ ] Push these down from DataFusion as vortex-expr. ### Pruning extensions - [ ] Falsifiers for functions of the time column, starting with `date_trunc` / `date_bin`. - [ ] Multi-column predicates such as interval overlap. - [ ] Wire the existing Bloom filter aggregate into the default writer for tag columns. ### Adding delta - [ ] Enable Delta in an edition and fix its cast rule. - [ ] Delta-of-delta for near-regular timestamps. - [ ] Re-apply Delta to monotone children like RLE indices and string offsets. ### Benchmarks - [ ] Sorted ClickBench: hits sorted by `EventTime`, same queries. - [ ] TSBS DevOps or IoT style workload: last-point, time-range, group-by-bucket and newest-N queries. - [ ] Compression-ratio suite on regular and jittered timestamps, counters and sensor doubles against Parquet, Gorilla and Pco. - [ ] Ingest benchmark: streaming append throughput and time to first readable row.