repo: integrate flux-accounting with Flux install about flux-accounting

[grondo]

What part of flux-accounting will need to be a broker module? I had thought it made use of the job-archive module and operated on the sqlite DB out of band of the actual Flux instance?

Perhaps you mean to install a utility such that it is picked up by the flux(1) command driver, e.g. flux accounting?

Also, I think we still need to figure out how the multifactor priority generated by flux-accounting will be incorporated into the job-manager priority in flux-core. I think we had talked about a plugin interface, but not sure if we got much farther than that...

[dongahn]

@cmoussa1:

Thank you for creating the issue and we can chat more at today's scheduled meeting.

I maybe wrong, but I think flux-accounting will less likely install flux "modules" that show up with flux module command.

Instead, front-end tools such as your recent accounting cli should be installed so that scheduler admins can use them to get to the user accounting database. For example, in this case, your standalone cli tools will be called as subcommand via the flux command: e.g., flux account ...

If you look at Fluxion, you may get some feel for how third party components can be installed to a pre-existing flux.

[dongahn]

Also, I think we still need to figure out how the multifactor priority generated by flux-accounting will be incorporated into the job-manager priority in flux-core. I think we had talked about a plugin interface, but not sure if we got much farther than that...

I fully agree.

As I see this, there are two major integration points of flux-accounting to the rest of flux.

1. Install front-end tools so that admins can use them seamlessly through the co-installed version. (Where should the accounting DB file reside though?)
2. Once the admin populate the DB, A) the multi-factor plugin should allow job-manager to sort its queue based on accounting policies (let's believe this is the right path for now; as opposed to fluxion does this); B) per account "limit" info may allow Fluxion to enforce them. We haven't decided whether per-account limit will be required though.

Maybe, working on the initial level integration (installation issues and such) first gives @cmoussa1 a more concrete idea about how flux-accounting should interface w/ the rest of flux and users.

[SteVwonder]

It looks like pip supports a --target argument similar to --prefix for configure: https://stackoverflow.com/a/19404371

[SteVwonder]

Oh. It was just some random path that I chose. Sorry for the confusion. Feel free to use whichever path you'd like to install flux into on your local system. Just something other than ~/.local/ to demonstrate compatibility with --prefix.

[cmoussa1]

Hm, interesting. I only see the flux-accounting executable. The contents of lib/:

[moussa1@machine:accounting-target]$ ls -l lib
total 2348
drwx------ 5 moussa1 moussa1    4096 Jun 11 14:31 flux
-rwxr-xr-x 1 moussa1 moussa1    1029 Jun 11 14:30 libflux-core.la
lrwxrwxrwx 1 moussa1 moussa1      21 Jun 11 14:30 libflux-core.so -> libflux-core.so.2.0.0
lrwxrwxrwx 1 moussa1 moussa1      21 Jun 11 14:30 libflux-core.so.2 -> libflux-core.so.2.0.0
-rwxr-xr-x 1 moussa1 moussa1 1979720 Jun 11 14:30 libflux-core.so.2.0.0
-rwxr-xr-x 1 moussa1 moussa1     974 Jun 11 14:30 libflux-idset.la
lrwxrwxrwx 1 moussa1 moussa1      22 Jun 11 14:30 libflux-idset.so -> libflux-idset.so.1.0.0
lrwxrwxrwx 1 moussa1 moussa1      22 Jun 11 14:30 libflux-idset.so.1 -> libflux-idset.so.1.0.0
-rwxr-xr-x 1 moussa1 moussa1   79136 Jun 11 14:30 libflux-idset.so.1.0.0
-rwxr-xr-x 1 moussa1 moussa1    1022 Jun 11 14:30 libflux-optparse.la
lrwxrwxrwx 1 moussa1 moussa1      25 Jun 11 14:30 libflux-optparse.so -> libflux-optparse.so.1.0.0
lrwxrwxrwx 1 moussa1 moussa1      25 Jun 11 14:30 libflux-optparse.so.1 -> libflux-optparse.so.1.0.0
-rwxr-xr-x 1 moussa1 moussa1  213376 Jun 11 14:30 libflux-optparse.so.1.0.0
-rwxr-xr-x 1 moussa1 moussa1    1116 Jun 11 14:30 libflux-schedutil.la
lrwxrwxrwx 1 moussa1 moussa1      26 Jun 11 14:30 libflux-schedutil.so -> libflux-schedutil.so.1.0.0
lrwxrwxrwx 1 moussa1 moussa1      26 Jun 11 14:30 libflux-schedutil.so.1 -> libflux-schedutil.so.1.0.0
-rwxr-xr-x 1 moussa1 moussa1   63064 Jun 11 14:30 libflux-schedutil.so.1.0.0
drwx------ 3 moussa1 moussa1    4096 Jun 11 14:31 lua
drwx------ 2 moussa1 moussa1    4096 Jun 11 14:31 pkgconfig
drwx------ 3 moussa1 moussa1    4096 Jun 11 14:33 python3.6
drwx------ 3 moussa1 moussa1    4096 Jun 11 14:31 python3.7

[cmoussa1]

I think we can solve that problem by doing flux python -m pip install ... rather than just pip3 install ....

I think you are right. I was mistakenly using the system installed /usr/bin/flux instead of the ~/flux/accounting-target/bin/flux, which resulted in multiple Python versions. I ran:

$ ~/flux/accounting-target/bin/flux python -m pip install --install-option="--prefix=$FLUX_INSTALL_PREFIX" . --upgrade

And here are the resulting contents of lib/:

[moussa1@quartz1148:flux-accounting]$ ll ~/flux/accounting-target/lib/
total 2344
drwx------ 5 moussa1 moussa1    4096 Jun 12 12:47 flux
-rwxr-xr-x 1 moussa1 moussa1    1029 Jun 12 12:47 libflux-core.la
lrwxrwxrwx 1 moussa1 moussa1      21 Jun 12 12:47 libflux-core.so -> libflux-core.so.2.0.0
lrwxrwxrwx 1 moussa1 moussa1      21 Jun 12 12:47 libflux-core.so.2 -> libflux-core.so.2.0.0
-rwxr-xr-x 1 moussa1 moussa1 1979720 Jun 12 12:47 libflux-core.so.2.0.0
-rwxr-xr-x 1 moussa1 moussa1     974 Jun 12 12:47 libflux-idset.la
lrwxrwxrwx 1 moussa1 moussa1      22 Jun 12 12:47 libflux-idset.so -> libflux-idset.so.1.0.0
lrwxrwxrwx 1 moussa1 moussa1      22 Jun 12 12:47 libflux-idset.so.1 -> libflux-idset.so.1.0.0
-rwxr-xr-x 1 moussa1 moussa1   79136 Jun 12 12:47 libflux-idset.so.1.0.0
-rwxr-xr-x 1 moussa1 moussa1    1022 Jun 12 12:47 libflux-optparse.la
lrwxrwxrwx 1 moussa1 moussa1      25 Jun 12 12:47 libflux-optparse.so -> libflux-optparse.so.1.0.0
lrwxrwxrwx 1 moussa1 moussa1      25 Jun 12 12:47 libflux-optparse.so.1 -> libflux-optparse.so.1.0.0
-rwxr-xr-x 1 moussa1 moussa1  213376 Jun 12 12:47 libflux-optparse.so.1.0.0
-rwxr-xr-x 1 moussa1 moussa1    1116 Jun 12 12:47 libflux-schedutil.la
lrwxrwxrwx 1 moussa1 moussa1      26 Jun 12 12:47 libflux-schedutil.so -> libflux-schedutil.so.1.0.0
lrwxrwxrwx 1 moussa1 moussa1      26 Jun 12 12:47 libflux-schedutil.so.1 -> libflux-schedutil.so.1.0.0
-rwxr-xr-x 1 moussa1 moussa1   63064 Jun 12 12:47 libflux-schedutil.so.1.0.0
drwx------ 3 moussa1 moussa1    4096 Jun 12 12:47 lua
drwx------ 2 moussa1 moussa1    4096 Jun 12 12:47 pkgconfig
drwx------ 3 moussa1 moussa1    4096 Jun 12 12:47 python3.7

However, trying to then use the flux-account command results in the following error:

[moussa1@quartz1148:flux-accounting]$ flux-account -h
Traceback (most recent call last):
  File "/g/g0/moussa1/flux/accounting-target/bin/flux-account", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
    @_call_aside
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
    f(*args, **kwargs)
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'flux-accounting==0.0.1' distribution was not found and is required by the application

I wonder if this has anything to do with the Python version that pip is using? Maybe it is using the system installed python version, which is 2.7?

[moussa1@machine1148:flux-accounting]$ python --version
Python 2.7.16

[SteVwonder]

Yeah, so I think the problem there is that the ~/flux/accounting-target/lib/python3.7/site-packages/ directory is not in the PYTHONPATH when flux-account goes to be run. I would try adding it manually to see if that fixes the issue. If so, then the libexec placement that I mentioned earlier is bumped up a bit in priority I think.

The context behind libexec is that is the directory all of the flux subcommands are placed (e.g., flux-kvs, flux-tree, flux-jobs.py, etc). When you run flux jobs, the flux top-level command searches the libexec directory for any programs named flux-jobs or flux-jobs.py. It'll find flux-jobs.py in this example, and do two things that are helpful in the case of python scripts. 1) it will ensure that the script is run with the python interpreter that flux was configured against 2) it will ensure that the flux installation directory where the python bindings live is included in the PYTHONPATH.

All that said, if we can get flux-account installed into the libexec directory alongside the other flux subcommands (and after typing all this out, i realized we will also need to ensure it ends in the .py extension), then you can run flux account ... and the flux main binary will take care of all the environment changes necessary such that it should "just work":tm:

[SteVwonder]

I guess one question I would have is would you be able to install flux-accounting to libexec/ manually and have Flux recognize it as a command?

Yep! That directory is search at runtime everytime a flux X command is invoked. When we added the flux-tree command to flux-sched, all we had to do was make sure that flux-tree ended up in the $prefix/libexec/ directory, and then flux tree .... worked auto-magically. :)

[cmoussa1]

Awesome! Is there a Python-way to only install the executable in a specific directory? Or did you use a special --install-option? When specifying a --prefix, the bin/ and lib/ directories get installed as well with the pip install . command I have been using thus far. A quick Google search hasn't turned up anything yet, but I'll keep looking.

EDIT = Stephen had already given me the answer, and I just didn't look hard enough (sorry @SteVwonder ). Here is the command I used to install flux-account.py to libexec:

$ ~/flux/accounting-target/bin/flux python -m pip install --install-option="--prefix=$FLUX_INSTALL_PREFIX" --install-option="--install-scripts=~/flux/accounting-target/libexec/flux/cmd/" . --upgrade

And the contents of libexec/flux/cmd/:

[moussa1@machine:cmd]$ ll
total 2644
-rwx------ 1 moussa1 moussa1    455 Jun 16 14:21 flux-account.py
-rwxr-xr-x 1 moussa1 moussa1   1723 Jun 16 09:55 flux-admin.py
-rwxr-xr-x 1 moussa1 moussa1 193880 Jun 16 09:55 flux-aggregate
-rwxr-xr-x 1 moussa1 moussa1 850472 Jun 16 09:55 flux-broker
-rwxr-xr-x 1 moussa1 moussa1  16374 Jun 16 09:55 flux-cron
-rwxr-xr-x 1 moussa1 moussa1  48832 Jun 16 09:55 flux-event
-rwxr-xr-x 1 moussa1 moussa1 118152 Jun 16 09:55 flux-exec
-rwxr-xr-x 1 moussa1 moussa1 336288 Jun 16 09:55 flux-job
-rwxr-xr-x 1 moussa1 moussa1  18643 Jun 16 09:55 flux-jobs.py
-rwxr-xr-x 1 moussa1 moussa1   7954 Jun 16 09:55 flux-jobspec.py
-rwxr-xr-x 1 moussa1 moussa1  57424 Jun 16 09:55 flux-keygen
-rwxr-xr-x 1 moussa1 moussa1 347192 Jun 16 09:55 flux-kvs
-rwxr-xr-x 1 moussa1 moussa1  43424 Jun 16 09:55 flux-logger
-rwxr-xr-x 1 moussa1 moussa1  17710 Jun 16 09:55 flux-mini.py
-rwxr-xr-x 1 moussa1 moussa1  74368 Jun 16 09:55 flux-module
-rwxr-xr-x 1 moussa1 moussa1  53792 Jun 16 09:55 flux-ping
-rwxr-xr-x 1 moussa1 moussa1  47856 Jun 16 09:55 flux-queue
-rwxr-xr-x 1 moussa1 moussa1  10051 Jun 16 09:55 flux-resource.py
-rwxr-xr-x 1 moussa1 moussa1 135608 Jun 16 09:55 flux-start
-rwxr-xr-x 1 moussa1 moussa1 232512 Jun 16 09:55 flux-terminus

This also required changing the name of the executable in setup.py:

    entry_points={
        'console_scripts': [
            'flux-account.py = accounting.accounting_cli:main',
        ]
    }

And voila! I can call flux account:

[moussa1@mach9ine:cmd]$ ~/flux/accounting-target/bin/flux account -h
usage: flux-account.py [-h] {view-user,add-user,delete-user,edit-user} ...

Description: Translate command line arguments into SQLite instructions for the
Flux Accounting Database.

positional arguments:
  {view-user,add-user,delete-user,edit-user}
                        sub-command help
    view-user           view a user's information in the accounting database
    add-user            add a user to the accounting database
    delete-user         remove a user from the accounting database
    edit-user           edit a user's value

optional arguments:
  -h, --help            show this help message and exit

[SteVwonder]

Stephen had already given me the answer, and I just didn't look hard enough (sorry @SteVwonder )

No apologies necessary. I thought I had posted the --install-scripts flag in this issue, but I think we just chatted about it on the phone on Friday. My bad.

Here is the command I used to install flux-account.py to libexec
This also required changing the name of the executable in setup.py:
And voila! I can call flux account:

Excellent!!!!! This looks perfect to me. I think the main question now is whether or not we can rely on pip in the RPM that we build for TOSS, but I think that's a separate packaging issue. For now, IMHO if the command that you listed above gets added to the Makefile under make install, then this issue can successfully be closed! Thanks @cmoussa1 for iterating on this. I think this is a really nice outcome!

[cmoussa1]

thank YOU for helping with the iteration @SteVwonder! 😃 here's where I've gotten:

• clone both repos:

$ git clone https://github.com/flux-framework/flux-core.git
$ git clone https://github.com/flux-framework/flux-accounting.git

• set FLUX_INSTALL_PREFIX:

$ export FLUX_INSTALL_PREFIX=~/flux/accounting-target/

• build flux-core:

[flux-core]$ ./autogen.sh && ./configure --prefix=$FLUX_INSTALL_PREFIX && make -j && make install

The Flux executable gets placed here:

[flux-core]$ ll ~/flux/accounting-target/bin/
total 724
-rwxr-xr-x 1 moussa1 moussa1 736104 Jun 17 10:36 flux

• set the PATH and PYTHONPATH environment variables:

$ export PYTHONPATH=~/flux/accounting-target/lib/python3.7/site-packages/
$ export PATH=$PATH:~/flux/accounting-target/bin/

Here are the contents of flux-accounting's Makefile:

install:
	${FLUX_INSTALL_PREFIX}bin/flux python -m pip install -r requirements.txt --user
	${FLUX_INSTALL_PREFIX}bin/flux python -m pip install --install-option="--prefix=${FLUX_INSTALL_PREFIX}" --install-option="--install-scripts=${FLUX_INSTALL_PREFIX}libexec/flux/cmd/" . --upgrade

check:
	pip3 install -r requirements.txt
	python3 -m unittest discover -b	

• Run make install:

[flux-accounting]$ make install 
/g/g0/moussa1/flux/accounting-target/bin/flux python -m pip install -r requirements.txt --user
Requirement already satisfied: pandas==0.25.3 in /g/g0/moussa1/.local/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (0.25.3)
Requirement already satisfied: numpy>=1.13.3 in /collab/usr/gapps/python/build/spack-toss3.3/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/python-3.7.2-asydydmavj2puklmx5t6cu3ruzmg2b3a/lib/python3.7/site-packages (from pandas==0.25.3->-r requirements.txt (line 1)) (1.16.2)
Requirement already satisfied: python-dateutil>=2.6.1 in /g/g0/moussa1/.local/lib/python3.7/site-packages (from pandas==0.25.3->-r requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: pytz>=2017.2 in /collab/usr/gapps/python/build/spack-toss3.3/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/python-3.7.2-asydydmavj2puklmx5t6cu3ruzmg2b3a/lib/python3.7/site-packages (from pandas==0.25.3->-r requirements.txt (line 1)) (2017.2)
Requirement already satisfied: six>=1.5 in /collab/usr/gapps/python/build/spack-toss3.3/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/python-3.7.2-asydydmavj2puklmx5t6cu3ruzmg2b3a/lib/python3.7/site-packages (from python-dateutil>=2.6.1->pandas==0.25.3->-r requirements.txt (line 1)) (1.12.0)
/g/g0/moussa1/flux/accounting-target/bin/flux python -m pip install --install-option="--prefix=/g/g0/moussa1/flux/accounting-target/" --install-option="--install-scripts=/g/g0/moussa1/flux/accounting-target/libexec/flux/cmd/" . --upgrade
/usr/tce/packages/python/python-3.7.2/lib/python3.7/site-packages/pip/_internal/commands/install.py:207: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Processing /g/g0/moussa1/src/flux-framework/flux-accounting
Skipping bdist_wheel for flux-accounting, due to binaries being disabled for it.
Installing collected packages: flux-accounting
  Running setup.py install for flux-accounting ... done
Successfully installed flux-accounting

• Now I can call flux-account commands using the flux(1) command driver:

$ ~/flux/accounting-target/bin/flux account -h
usage: flux-account.py [-h] {view-user,add-user,delete-user,edit-user} ...

Description: Translate command line arguments into SQLite instructions for the
Flux Accounting Database.

positional arguments:
  {view-user,add-user,delete-user,edit-user}
                        sub-command help
    view-user           view a user's information in the accounting database
    add-user            add a user to the accounting database
    delete-user         remove a user from the accounting database
    edit-user           edit a user's value

optional arguments:
  -h, --help            show this help message and exit

• And from the directory where the database file is located (like before), I can run the flux account commands and interact with the DB:

[accounting]$ ~/flux/accounting-target/bin/flux account add-user --username=fluxuser --admin-level=1 --account=acct --parent-acct=pacct --shares=10 --max-jobs=100 --max-wall-pj=1440

[accounting]$ ~/flux/accounting-target/bin/flux account view-user fluxuser
   id_assoc  creation_time    mod_time  deleted user_name  admin_level account parent_acct  shares  max_jobs  max_wall_pj
0         1     1592415668  1592415668        0  fluxuser            1    acct       pacct      10       100         1440

[accounting]$ ~/flux/accounting-target/bin/flux account edit-user --username=fluxuser --field=max_jobs --new-value=12345

[accounting]$ ~/flux/accounting-target/bin/flux account delete-user fluxuser 

I can go ahead and post a PR with these changes to flux-accounting (probably should update the documentation too to reflect the installation changes now that they can be linked to flux-core).

[dongahn]

The following might be useful.

https://github.com/flux-framework/flux-sched/blob/master/configure.ac#L122

https://github.com/flux-framework/flux-sched/blob/master/src/cmd/Makefile.am#L1

[cmoussa1]

Perhaps you mean to install a utility such that it is picked up by the flux(1) command driver, e.g. flux accounting?

Thanks for making this clearer @grondo - this is what I mean. I'll reflect this edit in the initial post so I don't get confused.

Maybe, working on the initial level integration (installation issues and such) first gives @cmoussa1 a more concrete idea about how flux-accounting should interface w/ the rest of flux and users.

I agree. We can talk in more detail about this probably during our scheduled meeting today!

[cmoussa1]

Thanks @SteVwonder! I ended up using --user for my own install. I've been playing around with it this morning, and I just thought I would document what was required to get this working on one of the Linux clusters:

If I am starting from scratch, I'd pull down the repo:

$ git clone https://github.com/flux-framework/flux-accounting.git

Then, if I try to install the requirements for this package without specifying a --target directory, I get the following error:

$ pip3 install .
Processing /g/g0/moussa1/src/flux-framework/flux-accounting
Installing collected packages: flux-accounting
  Running setup.py install for flux-accounting ... error
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/moussa1/pip-1fsdhkqt-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/tmp/moussa1/pip-plf5nu0c-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/accounting
    copying accounting/__init__.py -> build/lib/accounting
    copying accounting/accounting_cli.py -> build/lib/accounting
    copying accounting/accounting_cli_functions.py -> build/lib/accounting
    copying accounting/create_db.py -> build/lib/accounting
    creating build/lib/test
    copying test/__init__.py -> build/lib/test
    copying test/test_accounting_cli.py -> build/lib/test
    copying test/test_create_db.py -> build/lib/test
    copying test/test_example.py -> build/lib/test
    running install_lib
    creating /usr/local/lib/python3.6
    error: could not create '/usr/local/lib/python3.6': Permission denied
    
    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/moussa1/pip-1fsdhkqt-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/tmp/moussa1/pip-plf5nu0c-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/moussa1/pip-1fsdhkqt-build/

Specifying the --user option to install to ~/.local/bin, however, allows me to install the package successfully:

$ pip3 install . --target ~/.local/bin/
Processing /g/g0/moussa1/src/flux-framework/flux-accounting
Installing collected packages: flux-accounting
  Running setup.py install for flux-accounting ... done
Successfully installed flux-accounting-0.0.1

I have to do the same thing to install the dependencies for the package:

$ pip3 install -r requirements.txt --user
Collecting pandas==0.25.3 (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/52/3f/f6a428599e0d4497e1595030965b5ba455fd8ade6e977e3c819973c4b41d/pandas-0.25.3-cp36-cp36m-manylinux1_x86_64.whl
Collecting python-dateutil>=2.6.1 (from pandas==0.25.3->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl
Collecting numpy>=1.13.3 (from pandas==0.25.3->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/b3/a9/b1bc4c935ed063766bce7d3e8c7b20bd52e515ff1c732b02caacf7918e5a/numpy-1.18.5-cp36-cp36m-manylinux1_x86_64.whl
Collecting pytz>=2017.2 (from pandas==0.25.3->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/4f/a4/879454d49688e2fad93e59d7d4efda580b783c745fd2ec2a3adf87b0808d/pytz-2020.1-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil>=2.6.1->pandas==0.25.3->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, numpy, pytz, pandas
Successfully installed numpy-1.18.5 pandas-0.25.3 python-dateutil-2.8.1 pytz-2020.1 six-1.15.0

~/.local/bin wasn't in my $PATH, so just installing them while specifying the --target directory won't work out of the box. I have to add this to my PATH:

$ export PATH=$PATH:~/.local/bin

Now I can find flux-accounting (along with the other dependencies required):

$ which flux-account
~/.local/bin/flux-account

and use its commands:

$ flux-account -h

usage: flux-account [-h] {view-user,add-user,delete-user,edit-user} ...

Description: Translate command line arguments into SQLite instructions for the Flux Accounting Database.

positional arguments:
  {view-user,add-user,delete-user,edit-user}
                        sub-command help
    view-user           view a user's information in the accounting database
    add-user            add a user to the accounting database
    delete-user         remove a user from the accounting database
    edit-user           edit a user's value

optional arguments:
  -h, --help            show this help message and exit

[SteVwonder]

Thanks @cmoussa1 for investigating that! I wonder if it works to do something along these lines:

export FLUX_INSTALL_PREFIX=$HOME/flux/accounting-target
cd flux-core
./autogen.sh && ./configure --prefix=$FLUX_INSTALL_PREFIX && make -j && make install
cd ../flux-accounting
pip3 install --target=$FLUX_INSTALL_PREFIX

If it does work, one question I have would be, do all of the python dependencies that flux-accounting needs also get installed there? That might not be what we want exactly.

[cmoussa1]

Thanks @SteVwonder! I've started to look at this.

What is the $HOME/flux/accounting-target path? I know the $HOME points to my home directory, but I'm guessing the flux/accounting-target part is not the path to the flux-accounting directory?

For reference, I have the following structure:

[moussa1@machine:~]$ ls -l src/flux-framework/

drwx------ 14 moussa1 moussa1 4096 Jun 11 12:58 flux-accounting
drwx------ 11 moussa1 moussa1 4096 Jun 11 12:56 flux-core
drwx------  4 moussa1 moussa1 4096 Jan 30 15:29 flux-depend
drwx------  4 moussa1 moussa1 4096 Jan 30 15:33 flux-rs
drwx------  9 moussa1 moussa1 4096 Dec 13 14:33 flux-sched
drwx------  5 moussa1 moussa1 4096 Jan 30 15:26 flux-sys-rs
drwx------ 14 moussa1 moussa1 4096 Apr 29 09:26 flux-workflow-examples
drwx------  5 moussa1 moussa1 4096 Jan 30 16:06 hackathon

[cmoussa1]

I think I am going to guess that flux/accounting-target/ is storing the Flux executables? Anyways, I have been messing around with it some more! Here's where I got:

1. clone both repos into src/flux-framework/:

$ ls -l src/flux-framework/

drwx------ 14 moussa1 moussa1 4096 Jun 11 12:58 flux-accounting
drwx------ 11 moussa1 moussa1 4096 Jun 11 12:56 flux-core
drwx------  4 moussa1 moussa1 4096 Jan 30 15:29 flux-depend
drwx------  4 moussa1 moussa1 4096 Jan 30 15:33 flux-rs
drwx------  9 moussa1 moussa1 4096 Dec 13 14:33 flux-sched
drwx------  5 moussa1 moussa1 4096 Jan 30 15:26 flux-sys-rs
drwx------ 14 moussa1 moussa1 4096 Apr 29 09:26 flux-workflow-examples
drwx------  5 moussa1 moussa1 4096 Jan 30 16:06 hackathon

2. set FLUX_INSTALL_PREFIX:

$ export FLUX_INSTALL_PREFIX=$HOME/flux/accounting-target

$ echo $FLUX_INSTALL_PREFIX
/g/g0/moussa1/flux/accounting-target

3. Build flux-core:

$ ./autogen.sh && ./configure --prefix=$FLUX_INSTALL_PREFIX && make -j && make install

Now it looks like the flux executable gets installed here:

$ ls -l ~/flux/accounting-target/bin/
total 724
-rwxr-xr-x 1 moussa1 moussa1 736104 Jun 11 14:31 flux

Then, from the flux-accounting directory, I can install it as a package, but the --target option wasn't giving me any luck. So instead I used --install-option and passed in FLUX_INSTALL_PREFIX:

$ pip3 install --install-option="--prefix=$FLUX_INSTALL_PREFIX" . --upgrade

Now, in flux/accounting-target/:

$ ls -l
total 28
drwx------ 2 moussa1 moussa1 4096 Jun 11 14:33 bin
drwx------ 3 moussa1 moussa1 4096 Jun 11 14:31 etc
drwx------ 3 moussa1 moussa1 4096 Jun 11 14:30 include
drwx------ 7 moussa1 moussa1 4096 Jun 11 14:33 lib
drwx------ 3 moussa1 moussa1 4096 Jun 11 14:31 libexec
drwx------ 5 moussa1 moussa1 4096 Jun 11 14:31 share
drwx------ 3 moussa1 moussa1 4096 Jun 11 14:31 usr

And in bin/:

$ ls -l bin/
total 728
-rwxr-xr-x 1 moussa1 moussa1 736104 Jun 11 14:31 flux
-rwx------ 1 moussa1 moussa1    416 Jun 11 14:33 flux-account

I have to then add this path to my PATH:

$ export PATH=$PATH:~/flux/accounting-target/bin/
$  which flux-account
~/flux/accounting-target/bin/flux-account

And now I can run flux-account commands:

[moussa1@machine:~]$ flux-account -h
usage: flux-account [-h] {view-user,add-user,delete-user,edit-user} ...

Description: Translate command line arguments into SQLite instructions for the
Flux Accounting Database.

positional arguments:
  {view-user,add-user,delete-user,edit-user}
                        sub-command help
    view-user           view a user's information in the accounting database
    add-user            add a user to the accounting database
    delete-user         remove a user from the accounting database
    edit-user           edit a user's value

optional arguments:
  -h, --help            show this help message and exit

[SteVwonder]

Awesome! Thanks @cmoussa1. That looks pretty good. Presumably the flux-accounting python library files are in lib/ somewhere?

Minor note for whoever does the RPM packaging in the future: I think the one hiccup is that we typically put flux-* binaries in libexec. So when distributing flux-accounting via an RPM that is intended to be installed alongside flux-core and flux-sched RPMs, we may need to follow the instructions here: https://dacav.roundhousecode.com/blog/2019-05/06-install-some-python-executables-in-usr-libexec.html

[SteVwonder]

Hmmm. Odd that there are two different python versions in the lib directory. I wonder if flux-core was configured against a different version of python that the python version used for pip3. I think we can solve that problem by doing flux python -m pip install ... rather than just pip3 install ....

[cmoussa1]

I have been playing around with it some more and just wanted to continue documenting:

before installing flux-accounting, here is the output from flux python -m pip list:

$ ~/flux/accounting-target/bin/flux python -m pip list
Package                  Version  
------------------------ ---------  
pandas                   0.24.1 

Installing the dependencies for flux-accounting with the following command updates the version of pandas to the version specified in flux-accounting's requirements.txt:

$ ~/flux/accounting-target/bin/flux python -m pip install -r requirements.txt --user

and the resulting flux python -m pip list:

Package                  Version  
------------------------ ---------
pandas                   0.25.3

Then, installing flux-accounting with flux python adds the package to this package list:

$ ~/flux/accounting-target/bin/flux python -m pip install --install-option="--prefix=$FLUX_INSTALL_PREFIX" . --upgrade

and the resulting flux python -m pip list:

Package                  Version  
------------------------ ---------
flux-accounting          0.0.1
pandas                   0.25.3

and I can call the flux-account commands (even the ones that use pandas, like view-user:

$ flux-account view-user fluxuser
   id_assoc  creation_time    mod_time  deleted user_name  admin_level account parent_acct  shares  max_jobs  max_wall_pj
0         1     1592251394  1592251394        0  fluxuser            1    acct       pacct      10       100         1440

Late last week I was running into issues where flux-accounting was complaining that it couldn't find pandas even after installing requirements.txt, but I haven't run into that issue today. Guess I just needed to clean everything up and start over? 😅

EDIT:

All that said, if we can get flux-account installed into the libexec directory alongside the other flux subcommands (and after typing all this out, i realized we will also need to ensure it ends in the .py extension), then you can run flux account ... and the flux main binary will take care of all the environment changes necessary such that it should "just work"™️

I guess one question I would have is would you be able to install flux-accounting to libexec/ manually and have Flux recognize it as a command? Or would we have to install it at the same time as all of the other subcommands?

[SteVwonder]

Sounds good!

My only suggestion would be to set a default value for FLUX_INSTALL_PREFIX in the Makefile using ?= in case the user doesn't have that specified on the command line or in their environment [1]. In that case, the Makefile will gracefully fallback to some known good directory (e.g., /usr) rather than the empty string (which I think will result in the package being installed in the current directory):

FLUX_INSTALL_PREFIX ?= /usr

install:
	${FLUX_INSTALL_PREFIX}/bin/flux python -m pip install -r requirements.txt --user
	${FLUX_INSTALL_PREFIX}/bin/flux python -m pip install --install-option="--prefix=${FLUX_INSTALL_PREFIX}" --install-option="--install-scripts=${FLUX_INSTALL_PREFIX}/libexec/flux/cmd/" . --upgrade

[SteVwonder]

And from the directory where the database file is located

Out of curiosity, where does the database end up in this installation scheme?

[cmoussa1]

Out of curiosity, where does the database end up in this installation scheme?

The database file FluxAccounting.db will end up in the accounting/ directory inside of flux-accounting. Here is a tree view:

[moussa1@machine:flux-accounting]$ tree
.
|-- AUTHORS
|-- CONTRIBUTING.md
|-- COPYING
|-- LICENSE -> COPYING
|-- Makefile
|-- NOTICE -> NOTICE.LLNS
|-- NOTICE.LLNS
|-- README.md
|-- accounting
|   |-- FluxAccounting.db
|   |-- __init__.py
|   |-- __pycache__
|   |   |-- __init__.cpython-37.pyc
|   |   |-- accounting_cli_functions.cpython-37.pyc
|   |   `-- create_db.cpython-37.pyc
|   |-- accounting_cli.py
|   |-- accounting_cli_functions.py
|   |-- create_db.py
|   |-- db_creation.log
|   `-- test
|       `-- docker
|           |-- centos8
|           |   `-- Dockerfile.centos8
|           `-- ubuntu
|               `-- Dockerfile.ubuntu

The location of the file is set inside of create_db.py:

conn = sqlite3.connect(filepath)

.
.
.

def main():
    create_db("FluxAccounting.db") # will place DB in same directory where create_db.py is located

[cmoussa1]

My only suggestion would be to set a default value for FLUX_INSTALL_PREFIX in the Makefile using ?= in case the user doesn't have that specified on the command line or in their environment.

Thanks for making this suggestion. I'll push a fix!

[cmoussa1]

One thing we need to figure out is how to handle the workaround that flux-core uses to avoid "polluting" PYTHONPATH with system paths when users run flux: flux-framework/flux-core#1999. Ultimately the problem is that the directory that flux-accounting is installed into is correct, but it is not the directory that flux automatically adds to the PYTHONPATH.

I think I'll continue the discussion here about using the egg-links instead of the pull-request thread since there might be some more iteration needed.

I've been playing around with it some more, so I wanted to share where I got. I ended up using egg-links to just create a reference to where the flux-accounting source code lies (or at least, I think so 😅).

With my first iteration, after running make install, the source code for flux-accounting and its .egg-info directory were getting installed into flux/accounting-target/lib/flux/python3.7/. I changed the Makefile so that it instead contains the following install instruction:

${FLUX_INSTALL_PREFIX}/bin/flux python setup.py develop -xd ${FLUX_INSTALL_PREFIX}/lib/flux/python${FLUX_PYTHON_VERSION}/ install_scripts --install-dir=${FLUX_INSTALL_PREFIX}/libexec/flux/cmd/

which results in the following:

running develop
running egg_info
writing flux_accounting.egg-info/PKG-INFO
writing dependency_links to flux_accounting.egg-info/dependency_links.txt
writing entry points to flux_accounting.egg-info/entry_points.txt
writing top-level names to flux_accounting.egg-info/top_level.txt
writing manifest file 'flux_accounting.egg-info/SOURCES.txt'
running build_ext
Creating /g/g0/moussa1/flux/accounting-target/lib/flux/python3.7/site.py
Creating /g/g0/moussa1/flux/accounting-target/lib/flux/python3.7/flux-accounting.egg-link (link to .)
Adding flux-accounting 0.0.1 to easy-install.pth file

Installed /g/g0/moussa1/src/flux-framework/flux-accounting
Processing dependencies for flux-accounting==0.0.1
Finished processing dependencies for flux-accounting==0.0.1
running install_scripts
Installing flux-account.py script to /g/g0/moussa1/flux/accounting-target//libexec/flux/cmd/

So the flux-account.py executable still gets installed where we originally wanted, but the rest of flux-accounting remains in its original location, along with the newly-created .egg-info directory. And the contents of ~/flux/accounting-target/lib/flux/python3.7/ now contain:

total 16
drwx------ 2 moussa1 moussa1 4096 Jun 25 13:42 __pycache__
lrwxrwxrwx 1 moussa1 moussa1   70 Jun 25 13:29 _flux -> /g/g0/moussa1/flux/accounting-target/lib/python3.7/site-packages/_flux
-rw------- 1 moussa1 moussa1   49 Jun 25 13:42 easy-install.pth
lrwxrwxrwx 1 moussa1 moussa1   69 Jun 25 13:29 flux -> /g/g0/moussa1/flux/accounting-target/lib/python3.7/site-packages/flux
-rw------- 1 moussa1 moussa1   50 Jun 25 13:42 flux-accounting.egg-link
-rw------- 1 moussa1 moussa1 2302 Jun 25 13:42 site.py

I am still not too sure if we want easy-install.pth and site.py in this directory though. These get installed after running setup.py. The only contents of easy-install.pth is the path to the flux-accounting directory, and I think site.py appends the flux-accounting/ directory to sys.path. Thoughts @SteVwonder ?