Write failed creating shard - read-only file system about influxdb

[davidby-influx]

The error message opening shard previously failed with indicates that a particular shard has been put on an in-memory list of bad shards. To avoid a doom loop of attempting to open a bad shard again and again, all subsequent opens will fail and return the original error.

The list of bad shards is only in memory. restarting the database will clear the list, after which the shard creation should succeed.

[salcinad]

@davidby-influx

I did many restarts of influxdb pod but it did not clean the list of bad shards. I guess under Kubernetes it is different, or is here meant something else with "restarting the database" ?

$ k scale deployment -n influxdb influxdb-prod --replicas=0
deployment.apps/influxdb-prod scaled
$ k get pods -n influxdb
No resources found in influxdb namespace.
$ k scale deployment -n influxdb influxdb-prod --replicas=1
deployment.apps/influxdb-prod scaled
$ k logs -n influxdb influxdb-prod-757fb75486-pkjq9 -f
influxdb 19:59:34.62 INFO  ==>
influxdb 19:59:34.62 INFO  ==> Welcome to the Bitnami influxdb container
influxdb 19:59:34.62 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
influxdb 19:59:34.62 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
influxdb 19:59:34.62 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
influxdb 19:59:34.62 INFO  ==>
influxdb 19:59:34.63 INFO  ==> ** Starting InfluxDB setup **
influxdb 19:59:34.64 INFO  ==> influx CLI configuration /opt/bitnami/influxdb/etc/config.yaml detected!
influxdb 19:59:34.64 INFO  ==> Deploying InfluxDB with persisted data
influxdb 19:59:34.65 INFO  ==> Stopping InfluxDB...
influxdb 19:59:34.66 INFO  ==> ** InfluxDB setup finished! **

influxdb 19:59:34.67 INFO  ==> ** Starting InfluxDB **
ts=2024-06-14T19:59:34.894448Z lvl=info msg="Welcome to InfluxDB" log_id=0pn5BZwG000 version=v2.7.6 commit=3c58c06 build_date=2024-06-14T19:59:34Z log_level=info
ts=2024-06-14T19:59:34.902063Z lvl=info msg="Resources opened" log_id=0pn5BZwG000 service=bolt path=/bitnami/influxdb/influxd.bolt
ts=2024-06-14T19:59:34.902140Z lvl=info msg="Resources opened" log_id=0pn5BZwG000 service=sqlite path=/bitnami/influxdb/influxd.sqlite
ts=2024-06-14T19:59:34.909481Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0pn5BZwG000 bolt_path=/bitnami/influxdb/influxd.bolt
ts=2024-06-14T19:59:34.909581Z lvl=info msg="Using data dir" log_id=0pn5BZwG000 service=storage-engine service=store path=/bitnami/influxdb/data
ts=2024-06-14T19:59:34.909600Z lvl=info msg="Compaction settings" log_id=0pn5BZwG000 service=storage-engine service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
ts=2024-06-14T19:59:34.909609Z lvl=info msg="Open store (start)" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open op_event=start
ts=2024-06-14T19:59:34.915540Z lvl=error msg="Failed to open shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=65 error="[shard 65] unlinkat ./MANIFEST321764486: read-only file system"
ts=2024-06-14T19:59:34.917669Z lvl=error msg="Failed to open shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=69 error="[shard 69] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T19:59:34.919534Z lvl=error msg="Failed to open shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=70 error="[shard 70] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T19:59:34.920048Z lvl=error msg="Failed to open shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=73 error="[shard 73] unlinkat ./MANIFEST3482556016: read-only file system"



ts=2024-06-14T19:59:54.877154Z lvl=error msg="Failed to open shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=66 error="[shard 66] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T19:59:54.877551Z lvl=error msg="Failed to open shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=67 error="[shard 67] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T19:59:54.878859Z lvl=info msg="index opened with 8 partitions" log_id=0pn5BZwG000 service=storage-engine index=tsi
ts=2024-06-14T19:59:54.879650Z lvl=info msg="loading changes (start)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
ts=2024-06-14T19:59:54.879686Z lvl=info msg="loading changes (end)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.039ms
ts=2024-06-14T19:59:54.880598Z lvl=error msg="Failed to open shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=68 error="[shard 68] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T19:59:54.880834Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/4241f41bb6774795/autogen/74/000000001-000000001.tsm id=0 duration=1.018ms
ts=2024-06-14T19:59:54.884633Z lvl=info msg="index opened with 8 partitions" log_id=0pn5BZwG000 service=storage-engine index=tsi
ts=2024-06-14T19:59:54.884887Z lvl=info msg="index opened with 8 partitions" log_id=0pn5BZwG000 service=storage-engine index=tsi
ts=2024-06-14T19:59:54.885278Z lvl=info msg="loading changes (start)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
ts=2024-06-14T19:59:54.885316Z lvl=info msg="loading changes (end)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.051ms
ts=2024-06-14T19:59:54.885530Z lvl=info msg="Reading file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=cacheloader path=/bitnami/influxdb/wal/4241f41bb6774795/autogen/76/_00001.wal size=10485881
ts=2024-06-14T19:59:54.885728Z lvl=info msg="loading changes (start)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
ts=2024-06-14T19:59:54.885833Z lvl=info msg="loading changes (end)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.132ms
ts=2024-06-14T19:59:54.887337Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/4241f41bb6774795/autogen/75/000000001-000000001.tsm id=0 duration=1.322ms
ts=2024-06-14T19:59:54.913901Z lvl=info msg="Opened shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/bitnami/influxdb/data/4241f41bb6774795/autogen/75 duration=36.700ms
ts=2024-06-14T19:59:54.929791Z lvl=info msg="Opened shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/bitnami/influxdb/data/4241f41bb6774795/autogen/74 duration=56.648ms
ts=2024-06-14T19:59:54.958359Z lvl=info msg="index opened with 8 partitions" log_id=0pn5BZwG000 service=storage-engine index=tsi
ts=2024-06-14T19:59:54.959412Z lvl=info msg="loading changes (start)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
ts=2024-06-14T19:59:54.959453Z lvl=info msg="loading changes (end)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.047ms
ts=2024-06-14T19:59:54.977619Z lvl=info msg="index opened with 8 partitions" log_id=0pn5BZwG000 service=storage-engine index=tsi
ts=2024-06-14T19:59:54.978740Z lvl=info msg="loading changes (start)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
ts=2024-06-14T19:59:54.978787Z lvl=info msg="loading changes (end)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.060ms
ts=2024-06-14T19:59:55.035098Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/df58e7766e336a14/autogen/48/000006579-000000005.tsm id=2 duration=56.111ms
ts=2024-06-14T19:59:55.043695Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/df58e7766e336a14/autogen/40/000014578-000000010.tsm id=4 duration=84.021ms
ts=2024-06-14T19:59:55.090435Z lvl=error msg="Failed to open shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=64 error="[shard 64] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T19:59:55.116707Z lvl=info msg="index opened with 8 partitions" log_id=0pn5BZwG000 service=storage-engine index=tsi
ts=2024-06-14T19:59:55.117832Z lvl=info msg="loading changes (start)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
ts=2024-06-14T19:59:55.117874Z lvl=info msg="loading changes (end)" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.051ms
ts=2024-06-14T19:59:55.162697Z lvl=info msg="Reading file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=cacheloader path=/bitnami/influxdb/wal/4241f41bb6774795/autogen/76/_00002.wal size=10485881
ts=2024-06-14T19:59:55.323403Z lvl=info msg="Reading file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=cacheloader path=/bitnami/influxdb/wal/4241f41bb6774795/autogen/76/_00003.wal size=5685781
ts=2024-06-14T19:59:55.409709Z lvl=info msg="Opened shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/bitnami/influxdb/data/4241f41bb6774795/autogen/76 duration=530.255ms
ts=2024-06-14T19:59:55.437832Z lvl=error msg="Failed to open shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open db_shard_id=72 error="[shard 72] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T19:59:56.975822Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/df58e7766e336a14/autogen/40/000014578-000000006.tsm id=0 duration=1993.968ms
ts=2024-06-14T19:59:59.654446Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/df58e7766e336a14/autogen/40/000014578-000000009.tsm id=3 duration=4610.689ms
ts=2024-06-14T20:00:04.025573Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/df58e7766e336a14/autogen/48/000006579-000000003.tsm id=0 duration=7047.813ms
ts=2024-06-14T20:00:04.118712Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/df58e7766e336a14/autogen/56/000005409-000000003.tsm id=1 duration=76.359ms
ts=2024-06-14T20:00:06.552249Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/df58e7766e336a14/autogen/40/000014578-000000008.tsm id=2 duration=11517.090ms
ts=2024-06-14T20:00:06.662144Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/df58e7766e336a14/autogen/40/000014578-000000007.tsm id=1 duration=11659.899ms
ts=2024-06-14T20:00:06.662372Z lvl=info msg="Opened shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/bitnami/influxdb/data/df58e7766e336a14/autogen/40 duration=11748.281ms
ts=2024-06-14T20:00:11.746507Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/df58e7766e336a14/autogen/56/000005409-000000002.tsm id=0 duration=7627.718ms
ts=2024-06-14T20:00:11.746739Z lvl=info msg="Opened shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/bitnami/influxdb/data/df58e7766e336a14/autogen/56 duration=16656.228ms
ts=2024-06-14T20:00:13.923193Z lvl=info msg="Opened file" log_id=0pn5BZwG000 service=storage-engine engine=tsm1 service=filestore path=/bitnami/influxdb/data/df58e7766e336a14/autogen/48/000006579-000000004.tsm id=1 duration=14268.669ms
ts=2024-06-14T20:00:13.923406Z lvl=info msg="Opened shard" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=/bitnami/influxdb/data/df58e7766e336a14/autogen/48 duration=18992.665ms
ts=2024-06-14T20:00:14.124488Z lvl=info msg="Open store (end)" log_id=0pn5BZwG000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=39214.872ms
ts=2024-06-14T20:00:14.124559Z lvl=info msg="Starting retention policy enforcement service" log_id=0pn5BZwG000 service=retention check_interval=30m
ts=2024-06-14T20:00:14.124581Z lvl=info msg="Starting precreation service" log_id=0pn5BZwG000 service=shard-precreation check_interval=10m advance_period=30m
ts=2024-06-14T20:00:14.130601Z lvl=info msg="Starting query controller" log_id=0pn5BZwG000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
ts=2024-06-14T20:00:14.135414Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0pn5BZwG000 max_select_point=0 max_select_series=0 max_select_buckets=0
ts=2024-06-14T20:00:14.155678Z lvl=info msg=Starting log_id=0pn5BZwG000 service=telemetry interval=8h
ts=2024-06-14T20:00:14.158802Z lvl=info msg=Listening log_id=0pn5BZwG000 service=tcp-listener transport=http addr=0.0.0.0:8086 port=8086
ts=2024-06-14T20:00:39.876319Z lvl=warn msg="Write failed creating shard" log_id=0pn5BZwG000 service=storage-engine service=write shard=72 error="opening shard previously failed with: [shard 72] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T20:00:40.053407Z lvl=warn msg="Write failed creating shard" log_id=0pn5BZwG000 service=storage-engine service=write shard=72 error="opening shard previously failed with: [shard 72] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T20:00:40.183322Z lvl=warn msg="Write failed creating shard" log_id=0pn5BZwG000 service=storage-engine service=write shard=72 error="opening shard previously failed with: [shard 72] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T20:00:40.185735Z lvl=warn msg="Write failed creating shard" log_id=0pn5BZwG000 service=storage-engine service=write shard=72 error="opening shard previously failed with: [shard 72] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T20:00:49.859132Z lvl=warn msg="Write failed creating shard" log_id=0pn5BZwG000 service=storage-engine service=write shard=72 error="opening shard previously failed with: [shard 72] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T20:00:50.050991Z lvl=warn msg="Write failed creating shard" log_id=0pn5BZwG000 service=storage-engine service=write shard=72 error="opening shard previously failed with: [shard 72] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T20:00:50.181388Z lvl=warn msg="Write failed creating shard" log_id=0pn5BZwG000 service=storage-engine service=write shard=72 error="opening shard previously failed with: [shard 72] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T20:00:50.195613Z lvl=warn msg="Write failed creating shard" log_id=0pn5BZwG000 service=storage-engine service=write shard=72 error="opening shard previously failed with: [shard 72] unlinkat ./L0-00000051.tsl: read-only file system"
ts=2024-06-14T20:00:51.123886Z lvl=warn msg="Write failed creating shard" log_id=0pn5BZwG000 service=storage-engine service=write shard=72 error="opening shard previously failed with: [shard 72] unlinkat ./L0-00000051.tsl: read-only file system"
^C

$

[davidby-influx]

It looks like your file system is read-only:

ts=2024-06-14T20:00:39.876319Z lvl=warn msg="Write failed creating shard" log_id=0pn5BZwG000 service=storage-engine service=write shard=72 error="opening shard previously failed with: [shard 72] unlinkat ./L0-00000051.tsl: read-only file system"

[davidby-influx]

The list of bad shards is cleared on restart of the database, but the problem re-occurs, because your file system is read-only.