itzg / mc-monitor

Monitor the status of Minecraft servers and provides Prometheus exporter and Influx line protocol output

Command/agent to monitor the status of Minecraft servers

Install module

go get github.com/itzg/go-mc-status

Usage

Subcommands:
	flags            describe all known top-level flags
	help             describe subcommands and their syntax
	version          Show version and exit

Subcommands for monitoring:
	export-for-prometheus  Registers an HTTP metrics endpoints for Prometheus export
	gather-for-telegraf  Periodically gathers to status of one or more Minecraft servers and sends metrics to telegraf over TCP using Influx line protocol

Subcommands for status:
	status           Retrieves and displays the status of the given Minecraft server
	status-bedrock   Retrieves and displays the status of the given Minecraft Bedrock Dedicated server

Examples

Checking the status of a server

To check the status of a Java edition server:

docker run -it --rm itzg/mc-monitor status --host mc.hypixel.net

To check the status of a Bedrock Dedicated server:

docker run -it --rm itzg/mc-monitor status-bedrock --host play.fallentech.io

where exit code will be 0 for success or 1 for failure.

Workarounds for some status errors

Some Forge servers may cause a string length out of bounds error during status messages due to how the FML2 protocol bundles the entire modlist for client compatibility check. If there are issues with status failing when it otherwise should work, you can try out the experimental --use-mc-utils flag below (enables the mcutils protocol library):

docker run -it --rm itzg/mc-monitor status --use-mc-utils --host play.fallentech.io

Monitoring a server with Telegraf

The following example is provided in examples/mc-monitor-telegraf

Given the telegraf config file:

[[inputs.socket_listener]]
  service_address = "tcp://:8094"

[[outputs.file]]
  files = ["stdout"]

...and a Docker composition of telegraf and mc-monitor services:

version: '3'

services:
  telegraf:
    image: telegraf:1.13
    volumes:
    - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
  monitor:
    image: itzg/mc-monitor
    command: gather-for-telegraf
    environment:
      GATHER_INTERVAL: 10s
      GATHER_TELEGRAF_ADDRESS: telegraf:8094
      GATHER_SERVERS: mc.hypixel.net

The output of the telegraf service will show metric entries such as:

minecraft_status,host=mc.hypixel.net,port=25565,status=success response_time=0.172809649,online=51201i,max=90000i 1576971568953660767
minecraft_status,host=mc.hypixel.net,port=25565,status=success response_time=0.239236074,online=51198i,max=90000i 1576971579020125479
minecraft_status,host=mc.hypixel.net,port=25565,status=success response_time=0.225942383,online=51198i,max=90000i 1576971589006821324

Monitoring a server with Prometheus

When using the export-for-prometheus subcommand, mc-monitor will serve a Prometheus exporter on port 8080, by default, that collects Minecraft server metrics during each scrape of /metrics.

The sub-command accepts the following arguments, which can also be viewed using --help:

  -bedrock-servers host:port
    	one or more host:port addresses of Bedrock servers to monitor, when port is omitted 19132 is used (env EXPORT_BEDROCK_SERVERS)
  -port int
    	HTTP port where Prometheus metrics are exported (env EXPORT_PORT) (default 8080)
  -servers host:port
    	one or more host:port addresses of Java servers to monitor, when port is omitted 25565 is used (env EXPORT_SERVERS)

The following metrics are exported

• minecraft_status_healthy
• minecraft_status_response_time_seconds
• minecraft_status_players_online_count
• minecraft_status_players_max_count

with the labels

• server_host
• server_port
• server_edition : java or bedrock
• server_version

An example Docker composition is provided in examples/mc-monitor-prom, which was used to grab the following screenshot:

mc-monitor's Issues

Display players nicknames in metrics when more than 0 players online

Hi guys,

Would it be possible to have a metric in Prometheus Exporter of MC-Monitor that lists nicknames of players connected.
It works perfectly to know how many users are connected, but I would love to be able to query "online players nicknames".

Thanks a lot !

Fix security issues with golang/stdlib < 1.20.11

Hello,

there are some security problems with golang/stdlib smaller than 1.20.11. I ask for a new release and increase of the dependency.

• HIGH: CVE-2023-45283
• HIGH CVE-2023-44487
• HIGH CVE-2023-39325
• MEDIUM CVE-2023-45284

Thank you very much

New Release build missing

the main branch has some changes, but a new release build is missing.

Thank you

String buffer overflow on status for long forge modlists

I'm running an Enigmatica6 server on Forge 1.16.5. FML2 handshakes grow linearly with the mod list as they use it to confirm client compatibility:

❯ docker run -it -e DEBUG=true --rm itzg/mc-monitor status --host [REDACTED] --port 25565
2022-09-20T13:10:25.277Z        DEBUG   mc-monitor/java_status.go:76    pinging
2022-09-20T13:10:25.435Z        DEBUG   mc-monitor/java_status.go:79    ping returned   {"error": "string length out of bounds", "info": null}
failed to ping [REDACTED]:25565 : string length out of bounds%

After flushing this one out for a while, it seems to be that once the handshake reply is beyond math.MaxInt16 or 65535 bytes, mc-pinger breaks and returns a slightly nebulous error. This makes my LivenessProbe fail and I want to use mc-monitor if I can. 😄

I'm mostly writing this issue as documentation of this quirk in case it saves another time-poor weekend server op some time. I have a quick and dirty way of fixing this issue, but it does involve a slightly less obvious change in a forthcoming PR. I'll also try to relay this upstream in case @raqbit feels up to tackling it or can direct us where to help out, though it appears there's some refactoring work that might need to be done.

Monitoring multiple servers/types

Hi, is there a way to monitor multiple server types with a single mc-monitor stack?

I've added the following in my compose:
- EXPORT_SERVERS=mc,mcb:19132

A second bedrock server (mcb) does appear in Grafana with some metrics but some data missing.

Is there something that needs doing with a sub-command server-type ? I'm running one Java (mc) and one Bedrock (mcb) instance.

docker-compose.yml

`version: "3.8"

services:
mc:
image: itzg/minecraft-server:java8
ports:
- 25565:25565/tcp
environment:
- EULA=TRUE
- MEMORY=2G
- TYPE=FORGE
- VERSION=1.16.5
- DEBUG=true
- MAX_PLAYERS=5
volumes:
- ./config:/data
- ./config/plugins:/plugins
# attach local host directory "mods" in same directory as this compose file
# all mods in this directory get copied into /data/mods at startup
- ./mods:/mods
restart: unless-stopped
tty: true
stdin_open: true
mcb:
image: itzg/minecraft-bedrock-server:latest
ports:
- 19132:19132/udp
environment:
- EULA=TRUE
- TZ=Australia/Perth
- GAMEMODE=creative
- DIFFICULTY=peaceful
- PACKAGE_BACKUP_KEEP=2
volumes:
- ./mcb/config:/data
restart: unless-stopped
tty: true
stdin_open: true
monitor:
image: itzg/mc-monitor
command: export-for-prometheus
environment:
- EXPORT_SERVERS=mc,mcb:19132
- DEBUG=true
depends_on:
- mc
- mcb
cadvisor:
image: gcr.io/cadvisor/cadvisor
ports:
- 8180:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
prometheus:
image: prom/prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus-tsdb:/prometheus
depends_on:
- monitor
grafana:
image: grafana/grafana-oss:${GRAFANA_VERSION:-8.3.3}
ports:
- 3000:3000
volumes:
- grafana-lib:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
- ./dashboards:/etc/grafana/dashboards
depends_on:
- prometheus
backups:
image: itzg/mc-backup
environment:
BACKUP_INTERVAL: "2h"
# instead of network_mode below, could declare RCON_HOST
# RCON_HOST: mc
volumes:
# mount the same volume used by server, but read-only
- ./config:/data:ro
# use a host attached directory so that it in turn can be backed up
# to external/cloud storage
- /mnt/backup/mc-backups:/backups
# share network namespace with server to simplify rcon access
network_mode: "service:mc"
depends_on:
- mc

volumes:
prometheus-tsdb: {}
grafana-lib: {}
`

Extend the prometheus exporter to include player metrics

Would like to see the prometheus exporter be extended to provide the info that is in /data/world/stats.

This is a pretty big enhancement, IMHO. But wanted to track it, discuss it, and explain how I am doing it today

My current custom implementation

I am going to explain what I have, just to give details. It is not the expected solution, as it requires many different tools that are not part of the minecraft system.

A custom solution that is a combination of bash on the host that runs a cronjob, a mongodb table to store the data, and a couple flows in nodered that put the data in the mongodb collection, and that serves up the metrics endpoint for prometheus.

This is a cronjob that runs every 5 minutes. It takes the contents of the stats directory and POSTs to a nodered endpoint that

#!/usr/bin/env bash

data_dir="/mnt/container_data/minecraft/curseforge/data";
stats_dir="${data_dir}/world/stats";

# loop all json files in $stats_dir
# the file name is the player uuid

for file in ${stats_dir}/*.json; do
  # get the player uuid
  echo "Processing ${file}";
  player_uuid=$(basename ${file} .json);
  
  stat_data=$(cat ${file});
  echo "Sending data to Node-RED";
  curl -X POST -H "Content-Type: application/json" -d "${stat_data}" "https://nodered.bit13.local/minecraft/player/${player_uuid}/stats/";
done

This posts the stats from each file, finds the user, from my whitelist, and then stores the full set of data in mongo.

var user_info = msg.payload[0];
var data = global.get("mc_stats")["stats"];
var timestamp = Date.now();
msg.payload = {
    "$set": {
        "uuid": user_info.uuid, // comes from whitelist lookup
        "user_id": user_info.user_id, // comes from whitelist lookup
        "username": user_info.username, // comes from whitelist lookup
        "stats": data,
        "modified": timestamp
    }
}

Besides the POST endpoint to collect the metrics, and store them in mongodb, there is a metrics endpoint that is consumed by prometheus.

This is what this flow looks like

The first line of that flow is the stats collection, and then the formatting that data to a prometheus exported data format.
The second line gets the installed mod version (ie: ATM8 1.0.15)
The third line gets the server stats, like max players, online players, etc.

The collect metrics step from the first line of the flow:

function safe_name(n) {
    return n.replace(/\:/gmi, "_");
}
function metric_name(group,metric,labels) {
    labels["metric"] = metric
    split_item = metric.split(":");
    if (split_item && split_item.length == 2) {
        labels["source"] = split_item[0];
        labels["item"] = split_item[1];
    }
    return `mcstats_${safe_name(group)}{${labels_out(labels)}}`.toLowerCase();
}
function labels_out(labels) {
    let outValue = "";
    for (let l in labels) {
        outValue += `${l}="${labels[l]}",`;
    }
    if (outValue.length >= 1) {
        outValue = outValue.slice(0, -1);
    }
    return outValue;
}
let metrics = {};

for (let x = 0; x < msg.payload.length; ++x) {
    var labels = {};
    var stats = msg.payload[x].stats;
    
    labels["uuid"] = msg.payload[x].uuid;
    labels["username"] = msg.payload[x].username;
    labels['user_id'] = msg.payload[x].user_id;
    for (let group in stats) {
        for (let metric in stats[group]) {
            metrics[metric_name(group,metric,labels)] = stats[group][metric];
        }
    }   
}
global.set("mc_stats_metrics", metrics);
msg.payload = metrics;
return msg;

What I have setup ties in to my whitelist, so I am able to get the username. I look up the uuid to get the username. I wouldn't expect it to be this level of info, as it requires a way to lookup the uuid -> username. (hitting an api for this would have to include caching). But having the username makes it very easy to filter by user on a grafana dashboard.

user_id: discord user id.
username: minecraft username
uuid: minecraft uuid
metric: the full metric name, defined in the stats
source: derived from the metric. what provides the metric
item: derived from the metric. what the metric is.

the prometheus metric name itself comes from the stats file. looping over the objects in the stats to create them. replacing : with _. Here is a screenshot of the metrics for a user.

The children of this become the prometheus exported metric.

mcstats_minecraft_killed{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="creeperoverhaul:hills_creeper",source="creeperoverhaul",item="hills_creeper"} 1
mcstats_minecraft_dropped{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:red_mushroom",source="minecraft",item="red_mushroom"} 2
mcstats_minecraft_used{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="create:brass_casing",source="create",item="brass_casing"} 22
mcstats_minecraft_custom{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="tombstone:kill_undead_boss",source="tombstone",item="kill_undead_boss"} 2
mcstats_minecraft_killed_by{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:ravager",source="minecraft",item="ravager"} 1
mcstats_minecraft_mined{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:cobbled_deepslate",source="minecraft",item="cobbled_deepslate"} 247
mcstats_minecraft_broken{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:stone_shovel",source="minecraft",item="stone_shovel"} 2
mcstats_minecraft_picked_up{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:enchanted_book",source="minecraft",item="enchanted_book"} 2
mcstats_minecraft_crafted{uuid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",username="AAAAAAAA",user_id="1111111111111111",metric="minecraft:stone_brick_stairs",source="minecraft",item="stone_brick_stairs"} 28

Here are some screenshots of the Grafana Dashboard. Almost all of the panel items are filterable in some way. By user, by source, or by item. dashboard json

how to setup MC Monitor by using grafana

Hi itzg, it's a perfect project! But I don't know how to setup mc-monitor/examples/mc-monitor-prom ,I went into this directory and successfully executed sudo docker-compose up -d and sudo docker-compose logs -f, but my Grafana doesn't receive any message, this is my configuration

docker-compose.yml :

prometheus.yml:

and docker run -it --rm itzg/mc-monitor status --host mc.hypixel.net command is work perfect.

Connection reset spam on legacy versions

1.5.2 spam in log using mc-monitor. Minecraft client doesn't spam while pinging

Fix security issues with golang/stdlib < 1.21.5

Hello,

there are some security problems with golang/stdlib smaller than 1.21.5, I ask for a new release and increase of the dependency.

HIGH: CVE-2023-45283

Thank you very much

Update cadvisor in example for ARM64 runtime

More of improvement request than issue.

1. cadvisor latest image is not working on ARM64 platform (OCI)
   Used zcube/cadvisor image as it has pre-compiled ARM64 platform. Works as expected.
   Googled that gcr.io/cadvisor/cadvisor:v0.45.0 should have ARM64. Didn't have time to check.
2. cadvisor permissions
   Required to add privileged: true on Ubuntu to be able to get metrics data. Not sure if it is required for all OS flavors and should go into commit.

Provide scriptable option to output server's player count

See itzg/docker-minecraft-server#1046 (comment)

Add support for proxy_protocol usage in bungeecord

Bungeecord servers can be configured to support HAProxy PROXY protocol, https://www.spigotmc.org/wiki/bungeecord-configuration-guide/, so the one-shot mode of mc-monitor needs to provide an option to send pings via PROXY.

Unknown Telegraph debug stats

I am having issues getting any more status than response time from mc-monitor. I found that passing an environment variable of debug:true to the container would change to debug mode.

Within my container, I only see a debug log for gathering, but it doesn't list what it may be gathering.

2021-02-08T14:47:55.451Z DEBUG gather project/telegraf.go:49 gathering {"host": "minecraft-creative", "port": "19132"}

For my commands in docker-compose I have the following:
command: gather-for-telegraf status-bedrock
environment:
- GATHER_INTERVAL=10s
- GATHER_TELEGRAF_ADDRESS=telegraf:8094
- GATHER_SERVERS=minecraft-creative:19132

mc-pinger dependency supports Timeouts

The mc-pinger dependency has been updated and now supports Timeouts and Contexts.
Maybe have a look at it, so that the java status command can fail after a certain time.
Currently the command will try to connect to the server indefinitely even if the server is offline.

If you don't have time I could also implement the feature and open a pull request. Just let me know if you would like to support this feature

Server versions before 1.7 don't support handshake->ping format

In 1.7 the server networking stack was re-implemented with Netty and a new handshake->ping sequence was introduced. When status is used on those servers, errors like the following are reported by the server:

java.io.IOException: Bad packet id 99
        at ew.a(SourceFile:194)
        at cm.i(SourceFile:248)
        at cm.c(SourceFile:16)
        at cn.run(SourceFile:93)
2021-11-24 18:02:54 [INFO] Disconnecting /172.17.0.1:38702: Protocol error
2021-11-24 18:02:54 [INFO] /172.17.0.1:38702 lost connection

Need to add support for legacy server list ping, which is compatible pre 1.12.