flux account-update-db: ERROR:root:unable to open database file about flux-accounting

[garlick]

When we updated fluke to flux-accounting 0.20, the update script tried to run but failed with the above when running as the flux user. Probably because it is running from flux cron in the broker's current working directory which is /:

[root@fluke1:flux]# ps -fu flux|grep broker
flux        2067       1  0 10:32 ?        00:00:55 broker --config-path=/etc/fl
[root@fluke1:flux]# ls -l /proc/2067/cwd
lrwxrwxrwx 1 flux flux 0 Oct  5 12:44 /proc/2067/cwd -> /

@grondo captured the cron error:

ate-fshare; /bin/flux account-priority-update"": "flux-accounting database out of date; please update DB with 'flux account-update-db' before running commands"
te-fshare; /bin/flux account-priority-update"": "ERROR:root:unable to open database file"
ate-fshare; /bin/flux account-priority-update"": "flux-accounting database out of date; updating DB with 'flux account-update-db' before sending infomation to plugin"
te-fshare; /bin/flux account-priority-update"": "Traceback (most recent call last):"
te-fshare; /bin/flux account-priority-update"": "  File "/usr/libexec/flux/cmd/flux-account-priority-update.py", line 139, in <module>"
te-fshare; /bin/flux account-priority-update"": "    main()"
te-fshare; /bin/flux account-priority-update"": "  File "/usr/libexec/flux/cmd/flux-account-priority-update.py", line 135, in main"
te-fshare; /bin/flux account-priority-update"": "    bulk_update(path)"
te-fshare; /bin/flux account-priority-update"": "  File "/usr/libexec/flux/cmd/flux-account-priority-update.py", line 68, in bulk_update"
te-fshare; /bin/flux account-priority-update"": "    conn = est_sqlite_conn(path)"
te-fshare; /bin/flux account-priority-update"": "  File "/usr/libexec/flux/cmd/flux-account-priority-update.py", line 59, in est_sqlite_conn"
te-fshare; /bin/flux account-priority-update"": "    subprocess.run(["flux", "account-update-db", "-p", path], check=True)"
te-fshare; /bin/flux account-priority-update"": "  File "/usr/lib64/python3.6/subprocess.py", line 438, in run"
te-fshare; /bin/flux account-priority-update"": "    output=stdout, stderr=stderr)"
te-fshare; /bin/flux account-priority-update"": "subprocess.CalledProcessError: Command '['flux', 'account-update-db', '-p', '/var/lib/flux/FluxAccounting.db']' returned non-zero exit status 1."
t-priority-update"": Failed: Exit 1

He was able to get jobs running again by running the update script manually. But anyway it sounds like this needs a fix more urgently.

[cmoussa1]

I'm about to post a PR that hopefully fixes this issue using the Python module you referenced in an above comment. The module enables the creation of the temporary database inside /tmp/ that's used to upgrade the old DB.

[garlick]

A suggestion would be to create a unique temporary directory in /tmp for this.

This class may be helpful. Since the database contains user data, the directory and its contents should be removed afterwards on all exit paths.