pcdshub / archstats

EPICS Archiver Appliance statistics IOC

archstats

EPICS Archiver Appliance statistics IOC and Grafana dashboards

Documentation

https://pcdshub.github.io/archstats/

Requirements

• Python 3.7+
• aiohttp
• elasticsearch
• inflection

Installation

pip install git+https://github.com/pcdshub/archstats

Running the Tests

$ pip install -r dev-requirements.txt
$ pytest -vv

archstats's Issues

Optionally change elasticsearch indices to monthly, or always write to the same index

Expected Behavior

• Monthly index changing

Current Behavior

• Daily index changing

Context

Optimal index sizing is on the order of 1GB-10GB. Archstats daily indices are likely on the order of 1MB.

Add ratios for ease of display and alerting in Grafana

The appliance reports: PV count, connected PV count.
It's useful to calculate the percentage of connected PVs, which is possible in Grafana with a transform.

The transform can be plotted for a status dashboard, but it's not possible to use transforms with alerts yet. This means that we can only set an absolute PV count threshold, which is far from ideal.

{lts,mts,sts}_time_copy_data_into_store[_percent] can be NaN

elasticsearch doesn't seem to like documents with math.nan for floating point values; need to look into this more.

Add other database backends

Prometheus, as suggested by Murali?

Grafana supports these out of the box:

Multi-target elastic indices

Grafana supports multi-target index-* configuration for elasticsearch data sources.

https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-index.html

May be worth replicating this for the database-backed PVGroups.

[Grafana] Note min interval of 1 minute in data source configuration

Minimum interval of 1 minute, as that's the rate the archiver is polled at.

[Grafana] Timezone complications

• At 4PM PST on 12-02, it's 12-03 according to grafana datasources configuration: Elasticsearch error: no such index [archiver_appliance_metrics_pscaa01-2020.12.03] - which is true, but it shouldn't be checking for tomorrow's index just yet
• Server time is correct; grafana settings only seem to show browser timezone

Seems likely to be an elasticsearch misconfiguration, but due to permissions I can't even take a look at them for now.

This appears to manifest itself in grafana, most annoyingly, as the following:

The fields no longer get listed - but if you know the full name, typing it still works.

Dependency doctr-versions-menu is now docs-version-menu

Dependencies and sphinx config need updating.
Old: https://pypi.org/project/doctr-versions-menu/
New: https://pypi.org/project/docs-versions-menu/
Repository: https://github.com/goerz/docs_versions_menu/

Consider reworking data flow: elasticsearch -> PVGroup

Currently:

• PVGroup startup method loops and queries the archiver appliance
  • Writes to PVs
  • Writes an elasticsearch document

Consider:

• caproto-agnostic asyncio application queries archiver appliance

  • Writes an elasticsearch document

• caproto PVGroup watches elasticsearch database for updates

  • Writes to PVs

Appliance metrics units can change

estimated_bytes_transferred_in_etl0_to1_mb
became
estimated_bytes_transferred_in_etl0_to1_gb

https://github.com/slacmshankar/epicsarchiverap/blob/481da884c5c3e850df1730cba3005b32b42d358f/src/main/org/epics/archiverappliance/etl/bpl/reports/ApplianceMetricsDetails.java#L83

Appliance reboot may cause available space to stop updating?

Between 6/9 and 7/1, after a PAMM archiver appliance maintenance and reboot, pscaa02 disk statistics did not update.
Other metrics were unaffected.

Unclear of the cause at the moment, but a reboot of ioc-archstats brought it back to life.