VM Cold Storage is an experimental long-term storage layer for VictoriaMetrics.
The repository currently provides the storage foundation used by vmcoldarchive
and vmcoldselect: tenant-isolated object keys, an immutable range-readable cold
container, label/time indexes, hot/cold merging, downsample states, retention
configuration, and administrative commands.
go test ./...
go run ./cmd/vmcoldctl policy set --root ./data --tenant 1:2 --raw 8760h --hourly 26280h --daily 87600h
go run ./cmd/vmcoldctl policy list --root ./dataCI runs unit/integration tests, the race detector, and go vet. Local shortcuts
are available through make test, make test-race, make vet, and make bench.
Pack and range-read the included opaque-block fixture:
go run ./cmd/vmcoldarchive --root ./data --tenant 1:2 --manifest testdata/archive/manifest.json
go run ./cmd/vmcoldselect --root ./data --tenant 1:2 \
--start 1782864000000 --end 1782867599999 \
--label __name__ --op = --value demo_temperature_celsius --read
go run ./cmd/vmcoldctl object verify --root ./data --tenant 1:2
go run ./cmd/vmcoldctl object compact --root ./data --tenant 1:2 --target-bytes 268435456 --delete-sources
go run ./cmd/vmcoldctl retention apply --root ./data --tenant 1:2
go run ./cmd/vmcoldctl retention apply --root ./data --tenant 1:2 --execute
go run ./cmd/vmcoldctl archive status --root ./data
go run ./cmd/vmcoldctl catalog rebuild --root ./data --tenant 1:2
go run ./cmd/vmcoldctl generation status --root ./data --tenant 1:2
go run ./cmd/vmcoldctl generation gc --root ./data --tenant 1:2 --grace 24h
go run ./cmd/vmcoldctl generation gc --root ./data --tenant 1:2 --grace 24h --execute
go run ./cmd/vmcoldctl downsample build --root ./data --tenant 1:2 \
--start 1782864000000 --end 1782867599999 --resolution 1h --kind gauge --execute
go run ./cmd/vmcoldselect --root ./data --tenant 1:2 \
--start 1782864000000 --end 1782867599999 --step 1m \
--query 'avg_over_time(demo_temperature_celsius[1h])'Use s3://bucket/prefix as --root and set VMCOLD_S3_ENDPOINT,
VMCOLD_S3_REGION, VMCOLD_S3_ACCESS_KEY, and VMCOLD_S3_SECRET_KEY for an
S3-compatible backend. Multipart thresholds can be tuned with
VMCOLD_S3_MULTIPART_THRESHOLD and VMCOLD_S3_PART_SIZE.
An unmodified VictoriaMetrics native export can also be packed directly:
go run ./cmd/vmcoldarchive --root ./data --tenant 1:2 \
--native-file export.bin --codec-version vm-c620cb3The object layout is always scoped by VictoriaMetrics account and project IDs:
tenants/<AccountID>/<ProjectID>/objects/<data-yyyy>/<data-mm>/<sha256>.vmcold
See docs/architecture.md for the implemented format and the remaining integration milestones. The pinned upstream wire contract is documented in docs/upstream-victoriametrics.md.
