pcdshub / archstats Goto Github PK

Between 6/9 and 7/1, after a PAMM archiver appliance maintenance and reboot, pscaa02 disk statistics did not update.
Other metrics were unaffected.

Unclear of the cause at the moment, but a reboot of ioc-archstats brought it back to life.

elasticsearch doesn't seem to like documents with math.nan for floating point values; need to look into this more.

The appliance reports: PV count, connected PV count.
It's useful to calculate the percentage of connected PVs, which is possible in Grafana with a transform.

The transform can be plotted for a status dashboard, but it's not possible to use transforms with alerts yet. This means that we can only set an absolute PV count threshold, which is far from ideal.

Dependencies and sphinx config need updating.
Old: https://pypi.org/project/doctr-versions-menu/
New: https://pypi.org/project/docs-versions-menu/
Repository: https://github.com/goerz/docs_versions_menu/

Currently:

• PVGroup startup method loops and queries the archiver appliance
  • Writes to PVs
  • Writes an elasticsearch document

Consider:

• caproto-agnostic asyncio application queries archiver appliance

  • Writes an elasticsearch document

• caproto PVGroup watches elasticsearch database for updates

  • Writes to PVs

Expected Behavior

• Monthly index changing

Current Behavior

• Daily index changing

Context

Optimal index sizing is on the order of 1GB-10GB. Archstats daily indices are likely on the order of 1MB.

• At 4PM PST on 12-02, it's 12-03 according to grafana datasources configuration: Elasticsearch error: no such index [archiver_appliance_metrics_pscaa01-2020.12.03] - which is true, but it shouldn't be checking for tomorrow's index just yet
• Server time is correct; grafana settings only seem to show browser timezone

Seems likely to be an elasticsearch misconfiguration, but due to permissions I can't even take a look at them for now.

This appears to manifest itself in grafana, most annoyingly, as the following:

The fields no longer get listed - but if you know the full name, typing it still works.

Grafana supports multi-target index-* configuration for elasticsearch data sources.

https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-index.html

May be worth replicating this for the database-backed PVGroups.

Minimum interval of 1 minute, as that's the rate the archiver is polled at.

Prometheus, as suggested by Murali?

Grafana supports these out of the box:

estimated_bytes_transferred_in_etl0_to1_mb
became
estimated_bytes_transferred_in_etl0_to1_gb

https://github.com/slacmshankar/epicsarchiverap/blob/481da884c5c3e850df1730cba3005b32b42d358f/src/main/org/epics/archiverappliance/etl/bpl/reports/ApplianceMetricsDetails.java#L83