victoriametrics / metricsql

Standalone PromQL and MetricsQL parser

VictoriaMetrics

VictoriaMetrics is a fast, cost-saving, and scalable solution for monitoring and managing time series data. It delivers high performance and reliability, making it an ideal choice for businesses of all sizes.

Here are some resources and information about VictoriaMetrics:

• Documentation: docs.victoriametrics.com
• Case studies: Grammarly, Roblox, Wix,....
• Available: Binary releases, Docker images, Source code
• Deployment types: Single-node version, Cluster version, and Enterprise version
• Changelog: CHANGELOG, and How to upgrade
• Community: Slack, Twitter, LinkedIn, YouTube

Yes, we open-source both the single-node VictoriaMetrics and the cluster version.

Prominent features

VictoriaMetrics is optimized for timeseries data, even when old time series are constantly replaced by new ones at a high rate, it offers a lot of features:

• Long-term storage for Prometheus or as a drop-in replacement for Prometheus and Graphite in Grafana.
• Powerful stream aggregation: Can be used as a StatsD alternative.
• Ideal for big data: Works well with large amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various Enterprise workloads.
• Query language: Supports both PromQL and the more performant MetricsQL.
• Easy to setup: No dependencies, single small binary, configuration through command-line flags, but the default is also fine-tuned; backup and restore with instant snapshots.
• Global query view: Multiple Prometheus instances or any other data sources may ingest data into VictoriaMetrics and queried via a single query.
• Various Protocols: Support metric scraping, ingestion and backfilling in various protocol.
  • Prometheus exporters, Prometheus remote write API, Prometheus exposition format.
  • InfluxDB line protocol over HTTP, TCP and UDP.
  • Graphite plaintext protocol with tags.
  • OpenTSDB put message.
  • HTTP OpenTSDB /api/put requests.
  • JSON line format.
  • Arbitrary CSV data.
  • Native binary format.
  • DataDog agent or DogStatsD.
  • NewRelic infrastructure agent.
  • OpenTelemetry metrics format.
• NFS-based storages: Supports storing data on NFS-based storages such as Amazon EFS, Google Filestore.
• And many other features such as metrics relabeling, cardinality limiter, etc.

Enterprise version

In addition, the Enterprise version includes extra features:

• Anomaly detection: Automation and simplification of your alerting rules, covering complex anomalies found in metrics data.
• Backup automation: Automates regular backup procedures.
• Multiple retentions: Reducing storage costs by specifying different retentions for different datasets.
• Downsampling: Reducing storage costs and increasing performance for queries over historical data.
• Stable releases with long-term support lines (LTS).
• Comprehensive support: First-class consulting, feature requests and technical support provided by the core VictoriaMetrics dev team.
• Many other features, which you can read about on the Enterprise page.

Contact us if you need enterprise support for VictoriaMetrics. Or you can request a free trial license here, downloaded Enterprise binaries are available at Github Releases.

We strictly apply security measures in everything we do. VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. See Security page for more details.

Benchmarks

Some good benchmarks VictoriaMetrics achieved:

• Minimal memory footprint: handling millions of unique timeseries with 10x less RAM than InfluxDB, up to 7x less RAM than Prometheus, Thanos or Cortex.
• Highly scalable and performance for data ingestion and querying, 20x outperforms InfluxDB and TimescaleDB.
• High data compression: 70x more data points may be stored into limited storage than TimescaleDB, 7x less storage space is required than Prometheus, Thanos or Cortex.
• Reducing storage costs: 10x more effective than Graphite according to the Grammarly case study.
• A single-node VictoriaMetrics can replace medium-sized clusters built with competing solutions such as Thanos, M3DB, Cortex, InfluxDB or TimescaleDB. See VictoriaMetrics vs Thanos, Measuring vertical scalability, Remote write storage wars - PromCon 2019.
• Optimized for storage: Works well with high-latency IO and low IOPS (HDD and network storage in AWS, Google Cloud, Microsoft Azure, etc.).

Community and contributions

Feel free asking any questions regarding VictoriaMetrics:

• Slack Inviter and Slack channel
• Twitter
• Linkedin
• Reddit
• Telegram-en
• Telegram-ru
• Mastodon

If you like VictoriaMetrics and want to contribute, then please read these docs.

VictoriaMetrics Logo

Zip contains three folders with different image orientations (main color and inverted version).

Files included in each folder:

• 2 JPEG Preview files
• 2 PNG Preview files with transparent background
• 2 EPS Adobe Illustrator EPS10 files

Logo Usage Guidelines

Font used

• Lato Black
• Lato Regular

Color Palette

• HEX #110f0f
• HEX #ffffff

We kindly ask

• Please don't use any other font instead of suggested.
• To keep enough clear space around the logo.
• Do not change spacing, alignment, or relative locations of the design elements.
• Do not change the proportions for any of the design elements or the design itself. You may resize as needed but must retain all proportions.

metricsql's Issues

apply rule-based optimization

Some queries may be optmized by some rules to reduce latency and impove performance, such As

foo > 2 < 1     -> empty
sum(sum(sum(foo) by (a, b, c) by (a, b))) by (a)  -> sum(foo) by (a)

Support seting dynamic time duration

We want set dynamic time duration when generate recoding rules, support using variable during e.g.

• with (a = 2h) (up{job="samples"}[a])
• with (a = "2h") (up{job="samples"}[a])
• with (hours = (day_of_month() - 1) * 24 + hour() + "h") (up{job="samples"}[hours])

related issue: dynamic time duration

Support of authorization for setInitPush

That would be awesome if i can set Bearer token for setInitPush URL when I need to push data to some protected SaaSs

Any plan to support Unicode in metric name?

Original promql only support limited ascii characters, it would be helpful to support other UTF-8 characters for non-English users

This is my fork to support Chinese chars, It requires small changes to the lexer.

• jiacai2050@5cb9cb9

Support remove label

When I use PushdownBinaryOpFilters
Example:
sum(rabbitmq_queue_messages_unacknowledged)
change to
sum(rabbitmq_queue_messages_unacknowledged{job=''rabbitmq-exporter})
But before PushdownBinartOpFilters，I want remove the label of job
Example:
sum(rabbitmq_queue_messages_unacknowledged{job=''rabbitmq-exporter})
change to
sum(rabbitmq_queue_messages_unacknowledged)
Does this feature currently support it? It's really useful !

DurationExpr support set operation

in application code, i want to convert metricql foo{} to foo{}[1d], but the RollupExpr's field (DurationExpr) has no update operation. can we support it in the near future.

i is not equal to j

metricsql/lexer.go

Line 641 in a271ee2

if i == j || i == len(s) {

	if s[i] == '.' {
		j := i
		i++
		for i < len(s) && isDecimalChar(s[i]) {
			i++
		}
		if i == j || i == len(s) {
			return -1
		}
	}

I'm learning MetricsQL and I've noticed that here 'i' might not be equal to 'j' because 'i' undergoes an addition operation. Should it be j+1 here?

WITH templates collision between names of variables and functions #2383

Describe the bug
When using WITH templates, users can define variables with custom names:

WITH(
now = sum(rate(my_metric_total)),
before = sum(rate(my_metric_total) offset 1h)
)
now/before*100

This expression will fail starting from v1.70 where new MetricsQL function now() was introduced.
But for versions below v1.70 the expression will be valid.

To Reproduce
See https://play.victoriametrics.com/promql/expand-with-exprs?query=WITH%28%0D%0Anow+%3D+sum%28rate%28my_metric_total%29%29%2C%0D%0Abefore+%3D+sum%28rate%28my_metric_total%29+offset+1h%29%0D%0A%29%0D%0Anow%2Fbefore*100&commit=Expand

Expected behavior
Local variable names should not conflict with function names. Otherwise, adding new function to MetricsQL will always have a chance to affect existing user queries.

cc @valyala