elonmir / filestat_exporter

Prometheus exporter gathering metrics about file size, modification time and other stats

File statistics exporter

Prometheus exporter gathering metrics about file size, modification time and other statistics.

Quickstart

Pre-built binaries are available on the GitHub release page.

Usage

Configure target files on command line, passing glob patterns in parameters

./filestat_exporter '*'

Optional flags:

• -config.file <yaml>: The path to the configuration file (use "none" to disable).
• -log.level <level>: Logging level [debug, info, warn, error]. (default: info)
• -version: Print the version of the exporter and exit.
• -web.listen-address <port>: Address to listen on for web interface and telemetry. (default: 9943)
• -web.telemetry-path <URL path>: Path under which to expose metrics. (default: /metrics)
• -web.config <web file>: Path to config yaml file that can enable TLS or authentication.
• -web.systemd-socket: Flag to use systemd socket activation listeners instead of port listeners (Linux only).
• -path.cwd <path>: Change working directory of path pattern collection.
• -metric.crc32: Generate CRC32 hash metric of files.
• -metric.nb_lines: Generate line number metric of files.
• -metric.namespace: Change the namespace for the metrics.

The exporter can read a config file in yaml format (filestat.yaml by default).

exporter:
  # Optional network parameters
  listen_address: ':9943'
  #metrics_path: /metrics
  
  # Optional working directory - overridden by parameter '-path.cwd'
  working_directory: "/path/to/my/project"
  # Default enable/disable of metrics - overridden if not set by parameter '-metric.*'
  enable_crc32_metric: true
  # enable_nb_line_metric: false
  
  # Set the namespace for metrics
  # namespace: "file"
  
  # list of patterns to apply - metrics can be enable/disabled for each group
  files:
    - patterns: ["*.html","assets/*.css","scripts/*.js"]
    - patterns: ["data/*.csv"]
      enable_nb_line_metric: true
      # namespace: "file"
    - patterns: ["archives/*.tar.gz"]
      enable_crc32_metric: false
      enable_nb_line_metric: false
      add_prefix_to_metric: ""

Note: if a file is matched by a pattern more than once, only the first match's config is used

Glob pattern format

Glob pattern uses the implementation of bmatcuk/doublestar project:

• Doublestar (**) can be used to match directories recursively. It should appear surrounded by path separators such as /**/.
• Usual Glob syntax is still supported.

Exported Metrics

Metric	Description	Labels
file_glob_match_number	Number of files matching pattern	pattern
file_stat_size_bytes	Size of file in bytes	path
file_stat_modif_time_seconds	Last modification time of file in epoch time	path
file_content_hash_crc32 (*)	CRC32 hash of file content	path
file_content_line_number (*)	Number of lines in file	path

Note: metrics with (*) are only provided if configured
Note: if namespace option is configured, the metric name is changed accordingly e.g. file_glob_match_number becomes namespace_glob_match_number

Building and running

Prerequisites:

• Go compiler - currently version v1.13
• Linux with make installed
• Essential build environment for dependencies

Building

go get github.com/michael-doubez/filestat_exporter
cd ${GOPATH-$HOME/go}/src/github.com/michael-doubez/filestat_exporter
make
./filestat_exporter <flags> <file glob patterns ...>

To see all available configuration flags:

./filestat_exporter -h

The Makefile provides several targets:

• make check: Running checks and tests
• make run: Run exporter from go
• make version: Print current version
• make build: Build exporter without checks

Cross compiled distribution

To build all distribustion packages

make dist

To build a specific os/architecture package

make dist-<os>-<archi>
make dist-linux-amd64
...

Using Docker

The filestat_exporter is designed to monitor files on the host system.

Try it out in minutes on Katakoda docker playground:

# create local file
docker container run --rm -d -v ~/my_files:/my_files --name my_files bash -c 'echo "Hello world" > /my_files/sample.txt'
# launch exporter watching the files
docker run -d -p 9943:9943 --name=filestats -v ~/my_files:/data mdoubez/filestat_exporter -path.cwd /data '*'
# see file metrics
curl -s docker:9943/metrics | grep file_

TLS and basic authentication

Filestat Exporter supports TLS and basic authentication. This enables better control of the various HTTP endpoints.

To use TLS and/or basic authentication, you need to pass a configuration file using the --web.config.file parameter. The format of the file is described in the exporter-toolkit repository.

License

Apache License 2.0, see LICENSE.