techlib / celus

Celus is a web application for harvesting and visualizing usage statistics of electronic information sources

Celus

Celus (from CzechELib Usage Statistics) is a web application for harvesting and visualization of usage data of electronic information sources, such as scientific journals, etc. It was designed specifically to handle consortia of individual user organizations and separate usage statistics for each member organization.

You can find the documentation at Read the Docs:

• production version
• development version

License

Celus is open-source project licensed under the MIT license. It is offered as-is without warranty of any kind.

This repository contains the code as it is used by the CzechELib consortium. For customization, we recommend creating a fork of the repository and modifying the code there.

Please use the Issues feature of Github to report any problems with the software. Fixes of improvements in form of pull requests are welcome.

celus's Issues

Error during Celus install from playbook using project_branch: master

We attempted to install Celus from project_branch: master onto a RHEL8 server.

When it got to the step of downloading the html-root file from github it ran into this error:

fatal: [celus.lib.auburn.edu]: FAILED! => {"changed": false, "msg": "Failure downloading https://github.com/techlib/celus/releases/download/master/html-root.tar.gz, HTTP Error 404: Not Found"}

So we attempted to install from the latest tag instead (v1.0.38) but ran into other issues (see #16).

Should we be installing from master instead? If so, how should we fix the above error (or is that fixed on your end?)

Thanks!

Feature Request: Building Collections / virtual platforms for pricing

Hi,
we have a few cases (SpringerNature, Highwire) where we don't a price per platform but we have a price of a bundle of journals from one platform, or sometimes also for individual journals. For SpringerNature we receive now the Nature and the Springerlink usage via one download. Would be possible to define bundle of journals from a platform (one to x) which I can use to add a price. It could be for example a virtual platform and the title assignment (platform title) can also be edited not only deleted? This function would allow us to do all the cost per usage calculations via celus.

Django stuff in Apache

Can someone help me make a working connection between Apache and Celus (via module mod_wsgi_python3) ?

So far I succeeded to install Celus without errors via the playbook and I configured /opt/celus/config/settings/* for my setup.

My <VirtualHost *:80> Apache config:
# Django stuff - mod_wsgi
TimeOut 300
WSGIScriptAlias /production /opt/celus/config/production.wsgi
WSGIScriptAlias /api /opt/celus/config/wsgi.py/api
WSGIDaemonProcess celus python-home=/opt/virtualenvs/celus/ python-path=/opt/celus/ processes=8 threads=10
WSGIProcessGroup celus

    <Directory /opt/celus/config>
            <Files production.wsgi>
                    Require all granted
            </Files>
    </Directory>

    # Javascript routing needs the following
    FallbackResource /noindex/index.html

    Alias /media/ /var/www/celus/media/

error.log:
[Tue May 18 10:43:53.814741 2021] [wsgi:error] [pid 13630:tid 139945342514944] [remote 172.17.0.1:48892] Not Found: /production/

Probably some more configuration needs to be done, but I don't know where to start (finish).

Celus not running?

Dear celus team,

after trying out celus via run.lcelus.one, I decided to give it a go and try to install it locally ... I did went through (the really tough I must admit) palybook installation process which I managed to make it end without errors. However when I try to launch celus it doesn't seem to work ... at first I was getting an internal server error showing on the web page with the error log showing something like ...

[Fri Jun 02 11:21:23.988086 2023] [core:error] [pid 1033819:tid 140692659631872] [client 195.134.90.104:63851] AH00125: Request exceeded the limit of 10 subrequest nesting levels due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.

I then saw that the static dir was empty and moreover making a link to itself ... and then I went through the repoerted cases on git (especially #10) and rerun the installation and now statric has content ... but no index.html file as I would expect ... and the insternal server error still coming up but with different error log messages ...

then I copied manually the index.html found in design/ui/public and at least I didn't get the insternal server error on the web page as I was gettign a blank page and no error at the log.

if I hit a /api or /wsEc67YNV2sq the error log would show:
[Fri Jun 02 11:28:33.134461 2023] [wsgi:error] [pid 1033816:tid 140693052458752] [remote 195.134.90.104:63940] mod_wsgi (pid=1033816): Failed to exec Python script file '/opt/celus/config/wsgi.py'.
[Fri Jun 02 11:28:33.134657 2023] [wsgi:error] [pid 1033816:tid 140693052458752] [remote 195.134.90.104:63940] mod_wsgi (pid=1033816): Exception occurred processing WSGI script '/opt/celus/config/wsgi.py'.
[Fri Jun 02 11:28:33.135012 2023] [wsgi:error] [pid 1033816:tid 140693052458752] [remote 195.134.90.104:63940] Traceback (most recent call last):
[Fri Jun 02 11:28:33.135093 2023] [wsgi:error] [pid 1033816:tid 140693052458752] [remote 195.134.90.104:63940] File "/opt/celus/config/wsgi.py", line 12, in
[Fri Jun 02 11:28:33.135121 2023] [wsgi:error] [pid 1033816:tid 140693052458752] [remote 195.134.90.104:63940] from django.core.wsgi import get_wsgi_application
[Fri Jun 02 11:28:33.135201 2023] [wsgi:error] [pid 1033816:tid 140693052458752] [remote 195.134.90.104:63940] ModuleNotFoundError: No module named 'django'
[Fri Jun 02 11:28:33.222661 2023] [authz_core:error] [pid 1033817:tid 140691871078144] [client 195.134.90.104:63941] AH01630: client denied by server configuration: /path, referer: https://hippo.lib.uoa.gr/wsEc67YNV2sq

Looks like something wrong with django and mod_wsgi?

I would really appreciate your help (I'm pretty sure I'm missing something in my configuration but i can't figure out that!!!)

Thanks in advance for your time and effort.

Costas

I have tried the installation on a Rocky 8.7 Linux

Here's the contents of the /static dir:
(celus) [root@hippo config]# ls -l /var/www/celus/static/
total 28
drwxr-xr-x. 6 root root 68 Jun 1 16:11 admin
drwxr-xr-x. 3 root root 42 Jun 1 16:11 colorfield
drwxr-xr-x. 2 root root 29 Jun 1 16:11 css
-rw-r--r--. 1 root root 0 Jun 1 16:11 do_not_remove.txt
-rw-r--r--. 1 root root 9662 Jun 2 09:58 favicon.ico
-rw-r--r--. 1 root root 1705 Jun 2 09:58 favicon.png
drwxr-xr-x. 2 root root 72 Jun 1 16:11 import_export
-rw-r--r--. 1 root root 1226 Jun 2 09:58 index.html
-rw-r--r--. 1 root root 1636 Jun 2 09:58 maintenance.html
drwxr-xr-x. 4 root root 27 Jun 1 16:11 modeltranslation
drwxr-xr-x. 2 root root 4096 Jun 1 16:11 mptt
drwxr-xr-x. 7 root root 63 Jun 1 16:11 rest_framework

And also this is what I have in the celus.conf file

<VirtualHost *:80>
ServerName hippo.lib.uoa.gr
ServerAdmin [email protected]
Redirect / https://hippo.lib.uoa.gr/

#
<VirtualHost *:443>
ServerName hippo.lib.uoa.gr
ServerAdmin [email protected]

DocumentRoot /var/www/celus/html

    DocumentRoot /var/www/celus/static
    DirectoryIndex index.php index.htm index.html
    Alias /icons/ /var/www/icons/
    # ScriptAlias /cgi-bin/ /var/www/sub-domains/this.site/cgi-bin/

CustomLog "/var/log/httpd/celus-access_log" combined
ErrorLog  "/var/log/httpd/celus-error_log"


    SSLCertificateFile /etc/letsencrypt/live/hippo.lib.uoa.gr/fullchain.pem

SSLCertificateFile /etc/letsencrypt/live/hippo.lib.uoa.gr/cert.pem

    SSLCertificateKeyFile /etc/letsencrypt/live/hippo.lib.uoa.gr/privkey.pem

#is the next line needed?
SSLCertificateChainFile /etc/letsencrypt/live/hippo.lib.uoa.gr/fullchain.pem
Include /etc/letsencrypt/options-ssl-apache.conf

<Directory /var/www/celus/html>

     <Directory /var/www/celus/static>
            Options -ExecCGI -Indexes
            AllowOverride None

            Order deny,allow
            Deny from all
            Allow from all

            Satisfy all
    </Directory>

#RewriteEngine on
#RewriteCond %{SERVER_NAME} =hippo.lib.uoa.gr
#RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

this tag was here ....

#

#the following added for celus

Django stuff - mod_wsgi

TimeOut 300
#cv 2-6-2023 added next line
#WSGIScriptAlias / /opt/celus/config/wsgi.py
WSGIScriptAlias /api /opt/celus/config/wsgi.py/api
WSGIScriptAlias /wsEc67YNV2sq /opt/celus/config/wsgi.py/wsEc67YNV2sq
WSGIDaemonProcess celus python-home=/opt/virtualenvs/celus/ python-path=/opt/celus/ processes=8 threads=10
WSGIProcessGroup celus

<Directory /opt/celus/config>

#
Require all granted

<Directory /opt/celus/config>

Require all granted

Javascript routing needs the following

FallbackResource /index.html

Alias /media/ /var/www/celus/media/
#cv 2-6-2023 added next line
Alias /favicon.ico /path/to/mysite.com/static/favicon.ico

# Compress HTML, CSS, JavaScript, Text, XML and fonts AddOutputFilterByType DEFLATE application/javascript AddOutputFilterByType DEFLATE application/json AddOutputFilterByType DEFLATE application/xhtml+xml AddOutputFilterByType DEFLATE application/xml AddOutputFilterByType DEFLATE image/svg+xml AddOutputFilterByType DEFLATE image/x-icon AddOutputFilterByType DEFLATE text/css AddOutputFilterByType DEFLATE text/html AddOutputFilterByType DEFLATE text/javascript AddOutputFilterByType DEFLATE text/plain AddOutputFilterByType DEFLATE text/xml

Remove browser bugs (only needed for really old browsers)

BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
Header append Vary User-Agent

Manual Data uploads not working

Good afternoon,

Thank you for providing this solution. We are currently working to get it implemented at our site, but have run into a couple of issues that may or may not be related.

I have reviewed a similar issue here.

But this appears to have been a different cause, however, we are getting the same errors in the httpd logs and seeing similar problems.

Whenever we try to do a manual data upload we get to step 2 and after selecting the report type and file, it goes to Check before data import and just keeps spinning. Eventually I get an error that preflight failed with error 500. If I try to import this same file in Celus One it works immediately.

Error loading preflight data: Error: Request failed with status code 500

I have verified that celery is running.

Attached is the file that I am trying to import:

Manual_Import_Test.csv

The other issue that we are seeing is, the Interest section of the Dashboard is not populating, even though we do have data imported from SUSHI sources.

I also noticed that Django has several report types listed, but the manual data uploads page only allows us to choose between 2, "Counter 5 - Title report" or "Counter 4 - Journal Report 1"

Error during TASK [celus : install project python dependencies using poetry]

I getting an error because of Pillow=7.2.0

• Installing pillow (7.2.0): Failed

EnvCommandError

Command ['/root/.cache/pypoetry/virtualenvs/celus-5b8WzCEC-py3.9/bin/pip', 'install', '--no-deps', 'file:///root/.cache/pypoetry/artifacts/55/45/72/fe6bf35255ee373358c616d0b77dc220e3cb574e7e8501c3e6d791ab2c/Pillow-7.2.0.tar.gz'] errored with the following return code 1, and output:
Processing /root/.cache/pypoetry/artifacts/55/45/72/fe6bf35255ee373358c616d0b77dc220e3cb574e7e8501c3e6d791ab2c/Pillow-7.2.0.tar.gz
Building wheels for collected packages: Pillow
Building wheel for Pillow (setup.py): started
Building wheel for Pillow (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /root/.cache/pypoetry/virtualenvs/celus-5b8WzCEC-py3.9/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-d1hmq26_/setup.py'"'"'; file='"'"'/tmp/pip-req-build-d1hmq26_/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-l517yf3q
cwd: /tmp/pip-req-build-d1hmq26_/
Complete output (171 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/PIL
copying src/PIL/BdfFontFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/BlpImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/BmpImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/BufrStubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ContainerIO.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/CurImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/DcxImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/DdsImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/EpsImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ExifTags.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/FitsStubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/FliImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/FontFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/FpxImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/FtexImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GbrImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GdImageFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GifImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GimpGradientFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GimpPaletteFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GribStubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/Hdf5StubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/IcnsImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/IcoImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/Image.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageChops.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageCms.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageColor.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageDraw.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageDraw2.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageEnhance.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageFilter.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageFont.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageGrab.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageMath.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageMode.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageMorph.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageOps.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImagePalette.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImagePath.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageQt.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageSequence.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageShow.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageStat.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageTk.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageTransform.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageWin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImtImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/IptcImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/Jpeg2KImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/JpegImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/JpegPresets.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/McIdasImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/MicImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/MpegImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/MpoImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/MspImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PSDraw.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PaletteFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PalmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PcdImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PcfFontFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PcxImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PdfImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PdfParser.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PixarImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PngImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PpmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PsdImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PyAccess.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/SgiImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/SpiderImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/SunImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/TarIO.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/TgaImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/TiffImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/TiffTags.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/WalImageFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/WebPImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/WmfImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/XVThumbImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/XbmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/XpmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/init.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/main.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/_binary.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/_tkinter_finder.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/_util.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/_version.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/features.py -> build/lib.linux-x86_64-3.9/PIL
running egg_info
writing src/Pillow.egg-info/PKG-INFO
writing dependency_links to src/Pillow.egg-info/dependency_links.txt
writing top-level names to src/Pillow.egg-info/top_level.txt
reading manifest file 'src/Pillow.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '.c'
warning: no files found matching '.h'
warning: no files found matching '.sh'
warning: no previously-included files found matching '.appveyor.yml'
warning: no previously-included files found matching '.coveragerc'
warning: no previously-included files found matching '.editorconfig'
warning: no previously-included files found matching '.readthedocs.yml'
warning: no previously-included files found matching 'codecov.yml'
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.pyc' found anywhere in distribution
warning: no previously-included files matching '.so' found anywhere in distribution
no previously-included directories found matching '.ci'
adding license file 'LICENSE'
writing manifest file 'src/Pillow.egg-info/SOURCES.txt'
running build_ext

The headers or library files could not be found for zlib,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
   https://pillow.readthedocs.io/en/latest/installation.html

Traceback (most recent call last):
  File "/tmp/pip-req-build-d1hmq26_/setup.py", line 864, in <module>
    setup(
  File "/root/.cache/pypoetry/virtualenvs/celus-5b8WzCEC-py3.9/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib64/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib64/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/root/.cache/pypoetry/virtualenvs/celus-5b8WzCEC-py3.9/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 299, in run
    self.run_command('build')
  File "/usr/lib64/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib64/python3.9/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib64/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib64/python3.9/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/tmp/pip-req-build-d1hmq26_/setup.py", line 694, in build_extensions
    raise RequiredDependencyException(f)
__main__.RequiredDependencyException: zlib

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-req-build-d1hmq26_/setup.py", line 918, in <module>
    raise RequiredDependencyException(msg)
__main__.RequiredDependencyException:

The headers or library files could not be found for zlib,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
   https://pillow.readthedocs.io/en/latest/installation.html


----------------------------------------
ERROR: Failed building wheel for Pillow
Running setup.py clean for Pillow

Failed to build Pillow
Installing collected packages: Pillow
Running setup.py install for Pillow: started
Running setup.py install for Pillow: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /root/.cache/pypoetry/virtualenvs/celus-5b8WzCEC-py3.9/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-d1hmq26_/setup.py'"'"'; file='"'"'/tmp/pip-req-build-d1hmq26_/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-646_iyv5/install-record.txt --single-version-externally-managed --compile --install-headers /root/.cache/pypoetry/virtualenvs/celus-5b8WzCEC-py3.9/include/site/python3.9/Pillow
cwd: /tmp/pip-req-build-d1hmq26_/
Complete output (173 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/PIL
copying src/PIL/BdfFontFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/BlpImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/BmpImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/BufrStubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ContainerIO.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/CurImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/DcxImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/DdsImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/EpsImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ExifTags.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/FitsStubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/FliImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/FontFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/FpxImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/FtexImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GbrImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GdImageFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GifImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GimpGradientFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GimpPaletteFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/GribStubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/Hdf5StubImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/IcnsImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/IcoImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/Image.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageChops.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageCms.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageColor.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageDraw.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageDraw2.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageEnhance.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageFilter.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageFont.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageGrab.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageMath.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageMode.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageMorph.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageOps.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImagePalette.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImagePath.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageQt.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageSequence.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageShow.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageStat.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageTk.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageTransform.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImageWin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/ImtImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/IptcImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/Jpeg2KImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/JpegImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/JpegPresets.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/McIdasImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/MicImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/MpegImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/MpoImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/MspImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PSDraw.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PaletteFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PalmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PcdImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PcfFontFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PcxImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PdfImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PdfParser.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PixarImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PngImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PpmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PsdImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/PyAccess.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/SgiImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/SpiderImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/SunImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/TarIO.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/TgaImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/TiffImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/TiffTags.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/WalImageFile.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/WebPImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/WmfImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/XVThumbImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/XbmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/XpmImagePlugin.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/init.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/main.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/_binary.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/_tkinter_finder.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/_util.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/_version.py -> build/lib.linux-x86_64-3.9/PIL
copying src/PIL/features.py -> build/lib.linux-x86_64-3.9/PIL
running egg_info
writing src/Pillow.egg-info/PKG-INFO
writing dependency_links to src/Pillow.egg-info/dependency_links.txt
writing top-level names to src/Pillow.egg-info/top_level.txt
reading manifest file 'src/Pillow.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '.c'
warning: no files found matching '.h'
warning: no files found matching '.sh'
warning: no previously-included files found matching '.appveyor.yml'
warning: no previously-included files found matching '.coveragerc'
warning: no previously-included files found matching '.editorconfig'
warning: no previously-included files found matching '.readthedocs.yml'
warning: no previously-included files found matching 'codecov.yml'
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.pyc' found anywhere in distribution
warning: no previously-included files matching '.so' found anywhere in distribution
no previously-included directories found matching '.ci'
adding license file 'LICENSE'
writing manifest file 'src/Pillow.egg-info/SOURCES.txt'
running build_ext

  The headers or library files could not be found for zlib,
  a required dependency when compiling Pillow from source.

  Please see the install instructions at:
     https://pillow.readthedocs.io/en/latest/installation.html

  Traceback (most recent call last):
    File "/tmp/pip-req-build-d1hmq26_/setup.py", line 864, in <module>
      setup(
    File "/root/.cache/pypoetry/virtualenvs/celus-5b8WzCEC-py3.9/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib64/python3.9/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib64/python3.9/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib64/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/root/.cache/pypoetry/virtualenvs/celus-5b8WzCEC-py3.9/lib/python3.9/site-packages/setuptools/command/install.py", line 61, in run
      return orig.install.run(self)
    File "/usr/lib64/python3.9/distutils/command/install.py", line 551, in run
      self.run_command('build')
    File "/usr/lib64/python3.9/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib64/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib64/python3.9/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib64/python3.9/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib64/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib64/python3.9/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/tmp/pip-req-build-d1hmq26_/setup.py", line 694, in build_extensions
      raise RequiredDependencyException(f)
  __main__.RequiredDependencyException: zlib

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-d1hmq26_/setup.py", line 918, in <module>
      raise RequiredDependencyException(msg)
  __main__.RequiredDependencyException:

  The headers or library files could not be found for zlib,
  a required dependency when compiling Pillow from source.

  Please see the install instructions at:
     https://pillow.readthedocs.io/en/latest/installation.html


  ----------------------------------------

ERROR: Command errored out with exit status 1: /root/.cache/pypoetry/virtualenvs/celus-5b8WzCEC-py3.9/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-d1hmq26_/setup.py'"'"'; file='"'"'/tmp/pip-req-build-d1hmq26_/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-646_iyv5/install-record.txt --single-version-externally-managed --compile --install-headers /root/.cache/pypoetry/virtualenvs/celus-5b8WzCEC-py3.9/include/site/python3.9/Pillow Check the logs for full command output.
WARNING: You are using pip version 21.2.3; however, version 22.1.2 is available.
You should consider upgrading via the '/root/.cache/pypoetry/virtualenvs/celus-5b8WzCEC-py3.9/bin/python -m pip install --upgrade pip' command.

at /usr/local/lib/python3.9/site-packages/poetry/utils/env.py:1195 in run
1191│ output = subprocess.check_output(
1192│ cmd, stderr=subprocess.STDOUT, **kwargs
1193│ )
1194│ except CalledProcessError as e:
→ 1195│ raise EnvCommandError(e, input=input)
1196│
1197│ return decode(output)
1198│
1199│ def execute(self, bin, *args, **kwargs):

Any idea how to solve this issue

Error during playbook step "apply django migrations" - Ident authentication failed for user "celus"

Hello!

I'm attempting to use the ansible playbook to install Celus using the latest tagged release (v1.0.38) onto a RHEL8 server. The playbook moves along until the task [celus | apply django migrations]

Here I get this error:
\npsycopg2.OperationalError: FATAL: Ident authentication failed for user "celus"

This is the full error message with stack trace. Any suggestions on how to continue the install? Thank you!

TASK [celus : apply django migrations] *******************************************************************************************************************************
fatal: [celus.lib.auburn.edu]: FAILED! => {"changed": false, "cmd": "./manage.py migrate --noinput --settings=config.settings.production", "msg": "\n:stderr: Traceback (most recent call last):\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 217, in ensure_connection\n    self.connect()\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 195, in connect\n    self.connection = self.get_new_connection(conn_params)\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django_prometheus/db/common.py\", line 44, in get_new_connection\n    return super(DatabaseWrapperMixin, self).get_new_connection(*args, **kwargs)\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/postgresql/base.py\", line 178, in get_new_connection\n    connection = Database.connect(**conn_params)\n  File \"/opt/virtualenvs/celus/lib64/python3.6/site-packages/psycopg2/__init__.py\", line 127, in connect\n    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\npsycopg2.OperationalError: FATAL:  Ident authentication failed for user \"celus\"\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"./manage.py\", line 21, in \n    main()\n  File \"./manage.py\", line 17, in main\n    execute_from_command_line(sys.argv)\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/core/management/__init__.py\", line 381, in execute_from_command_line\n    utility.execute()\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/core/management/__init__.py\", line 357, in execute\n    django.setup()\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/__init__.py\", line 24, in setup\n    apps.populate(settings.INSTALLED_APPS)\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/apps/registry.py\", line 122, in populate\n    app_config.ready()\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django_prometheus/apps.py\", line 23, in ready\n    ExportMigrations()\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django_prometheus/migrations.py\", line 49, in ExportMigrations\n    executor = MigrationExecutor(connections[alias])\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/migrations/executor.py\", line 18, in __init__\n    self.loader = MigrationLoader(self.connection)\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/migrations/loader.py\", line 49, in __init__\n    self.build_graph()\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/migrations/loader.py\", line 212, in build_graph\n    self.applied_migrations = recorder.applied_migrations()\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/migrations/recorder.py\", line 73, in applied_migrations\n    if self.has_table():\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/migrations/recorder.py\", line 56, in has_table\n    return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 256, in cursor\n    return self._cursor()\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 233, in _cursor\n    self.ensure_connection()\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 217, in ensure_connection\n    self.connect()\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 217, in ensure_connection\n    self.connect()\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 195, in connect\n    self.connection = self.get_new_connection(conn_params)\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django_prometheus/db/common.py\", line 44, in get_new_connection\n    return super(DatabaseWrapperMixin, self).get_new_connection(*args, **kwargs)\n  File \"/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/postgresql/base.py\", line 178, in get_new_connection\n    connection = Database.connect(**conn_params)\n  File \"/opt/virtualenvs/celus/lib64/python3.6/site-packages/psycopg2/__init__.py\", line 127, in connect\n    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\ndjango.db.utils.OperationalError: FATAL:  Ident authentication failed for user \"celus\"\n\n", "path": "/opt/virtualenvs/celus/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "syspath": ["/tmp/ansible_django_manage_payload_ppp6woc3/ansible_django_manage_payload.zip", "/usr/lib64/python36.zip", "/usr/lib64/python3.6", "/usr/lib64/python3.6/lib-dynload", "/usr/local/lib/python3.6/site-packages", "/usr/lib64/python3.6/site-packages", "/usr/lib/python3.6/site-packages"]}

Export raw data displays only "initializing export"

Dear developers,

we have some issues when going to a platform and exporting raw data.

When I start the export only "Initializing export" is displayed and there is no progress.

Our installation is from the master branch running on Debian 10.

I haven't found any related errors in the apache logs. Do you know if some of the logs in /var/log/celery could be useful? Many thanks in advance!

After install, internal 500 error when loading pages

We got through the basic install instructions for celus. Also, we're not using Shibboleth so we also completed the suggested instructions in issue #10

In browser, the outline of the celus dashboard starts loading but we get these 500 errors

api/user/:1 Failed to load resource: the server responded with a status of 500 (Internal Server Error)
api/organization/:1 Failed to load resource: the server responded with a status of 500 (Internal Server Error)

In /var/log/httpd/error_log we see these details:

mod_wsgi (pid=3884): Failed to exec Python script file '/opt/celus/config/wsgi.py'.
mod_wsgi (pid=3884): Exception occurred processing WSGI script '/opt/celus/config/wsgi.py'.
Traceback (most recent call last):
File "/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/virtualenvs/celus/lib/python3.6/site-packages/django_prometheus/db/common.py", line 44, in get_new_connection
return super(DatabaseWrapperMixin, self).get_new_connection(*args, **kwargs)
File "/opt/virtualenvs/celus/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/opt/virtualenvs/celus/lib64/python3.6/site-packages/psycopg2/init.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Permission denied
\tIs the server running on host "127.0.0.1" and accepting
\tTCP/IP connections on port 5432?

Is there something we're missing with port 5432? Thanks!

wsgi errors in httpd log after install

Celus team,

We're seeing two errors repeatedly in our Apache log files and are not sure what to make of them. We think it may be blocking some basic functionality in our local install.

Note we aren't using shibboleth, and have enabled user/pass authentication using the suggestion here
#10 (comment)

First error:

[wsgi:error] Identity: None
[wsgi:error] Headers: HTTP_X_USER_ID: None; HTTP_X_FULL_NAME: None; HTTP_X_FIRST_NAME: None; HTTP_X_LAST_NAME: None; HTTP_X_USER_NAME: None; HTTP_X_MAIL: None; HTTP_X_CN: None; HTTP_X_ROLES: None; HTTP_X_IDENTITY: None

(this is frequently written to the logs even when we're logged in. is there another user/auth setting that needs changing?)

Second error:

[wsgi:error] Internal Server Error: /api/report-type/
[wsgi:error] Traceback (most recent call last):
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
[wsgi:error]     response = get_response(request)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
[wsgi:error]     response = self.process_exception_by_middleware(e, request)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
[wsgi:error]     response = wrapped_callback(request, *callback_args, **callback_kwargs)
[wsgi:error]   File "/usr/lib64/python3.6/contextlib.py", line 52, in inner
[wsgi:error]     return func(*args, **kwds)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
[wsgi:error]     return view_func(*args, **kwargs)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/viewsets.py", line 114, in view
[wsgi:error]     return self.dispatch(request, *args, **kwargs)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/views.py", line 505, in dispatch
[wsgi:error]     response = self.handle_exception(exc)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/views.py", line 465, in handle_exception
[wsgi:error]     self.raise_uncaught_exception(exc)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
[wsgi:error]     raise exc
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/views.py", line 502, in dispatch
[wsgi:error]     response = handler(request, *args, **kwargs)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/mixins.py", line 46, in list
[wsgi:error]     return Response(serializer.data)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/serializers.py", line 760, in data
[wsgi:error]     ret = super().data
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/serializers.py", line 260, in data
[wsgi:error]     self._data = self.to_representation(self.instance)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/serializers.py", line 678, in to_representation
[wsgi:error]     self.child.to_representation(item) for item in iterable
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/serializers.py", line 678, in 
[wsgi:error]     self.child.to_representation(item) for item in iterable
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/serializers.py", line 514, in to_representation
[wsgi:error]     for field in fields:
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/serializers.py", line 375, in _readable_fields
[wsgi:error]     for field in self.fields.values():
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
[wsgi:error]     res = instance.__dict__[self.name] = self.func(instance)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/serializers.py", line 363, in fields
[wsgi:error]     for key, value in self.get_fields().items():
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/serializers.py", line 1063, in get_fields
[wsgi:error]     source, info, model, depth
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/serializers.py", line 1208, in build_field
[wsgi:error]     return self.build_unknown_field(field_name, model_class)
[wsgi:error]   File "/opt/virtualenvs/celus/lib/python3.6/site-packages/rest_framework/serializers.py", line 1325, in build_unknown_field
[wsgi:error]     (field_name, model_class.__name__)
[wsgi:error] django.core.exceptions.ImproperlyConfigured: Field name `name_cs` is not valid for model `ReportType`.

(we installed from the v1.0.39 release, is it possible we ended up with some wrong version of a file?)

Thanks!

Manual data uploads not working

Dear developers, many thanks for providing Celus. I have provided a test installation for our library and they like it!

There is an issue with manual data uploads. When trying to upload a .csv file we get these errors in the log.

[Thu Jul 29 16:20:37.102147 2021]  Internal Server Error: /api/manual-data-upload/4/preflight/
[Thu Jul 29 16:20:37.102170 2021]  Traceback (most recent call last):
[Thu Jul 29 16:20:37.102173 2021]    File "/opt/celus/apps/logs/views.py", line 266, in preflight_check
[Thu Jul 29 16:20:37.102183 2021]      stats = custom_import_preflight_check(mdu)
[Thu Jul 29 16:20:37.102185 2021]    File "/opt/celus/apps/logs/logic/custom_import.py", line 137, in custom_import_preflight_check
[Thu Jul 29 16:20:37.102187 2021]      records = list(mdu.data_to_records())  # type: [CounterRecord]
[Thu Jul 29 16:20:37.102189 2021]    File "/opt/celus/apps/nigiri/counter5.py", line 229, in file_to_records
[Thu Jul 29 16:20:37.102191 2021]      for rec in self._fd_to_records(infile):
[Thu Jul 29 16:20:37.102193 2021]    File "/opt/celus/apps/nigiri/counter5.py", line 244, in _fd_to_records
[Thu Jul 29 16:20:37.102195 2021]      header[header_line[0]] = header_line[1]
[Thu Jul 29 16:20:37.102196 2021]  IndexError: list index out of range
[Thu Jul 29 16:20:37.102198 2021]  
[Thu Jul 29 16:20:37.102200 2021]  During handling of the above exception, another exception occurred:
[Thu Jul 29 16:20:37.102202 2021]  
[Thu Jul 29 16:20:37.102203 2021]  Traceback (most recent call last):
[Thu Jul 29 16:20:37.102205 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
[Thu Jul 29 16:20:37.102207 2021]      response = get_response(request)
[Thu Jul 29 16:20:37.102209 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
[Thu Jul 29 16:20:37.102211 2021]      response = self.process_exception_by_middleware(e, request)
[Thu Jul 29 16:20:37.102213 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
[Thu Jul 29 16:20:37.102215 2021]      response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Thu Jul 29 16:20:37.102216 2021]    File "/usr/lib/python3.7/contextlib.py", line 74, in inner
[Thu Jul 29 16:20:37.102218 2021]      return func(*args, **kwds)
[Thu Jul 29 16:20:37.102220 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
[Thu Jul 29 16:20:37.102222 2021]      return view_func(*args, **kwargs)
[Thu Jul 29 16:20:37.102224 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/rest_framework/viewsets.py", line 114, in view
[Thu Jul 29 16:20:37.102226 2021]      return self.dispatch(request, *args, **kwargs)
[Thu Jul 29 16:20:37.102230 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/rest_framework/views.py", line 505, in dispatch
[Thu Jul 29 16:20:37.102232 2021]      response = self.handle_exception(exc)
[Thu Jul 29 16:20:37.102233 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/rest_framework/views.py", line 465, in handle_exception
[Thu Jul 29 16:20:37.102235 2021]      self.raise_uncaught_exception(exc)
[Thu Jul 29 16:20:37.102237 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
[Thu Jul 29 16:20:37.102239 2021]      raise exc
[Thu Jul 29 16:20:37.102241 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/rest_framework/views.py", line 502, in dispatch
[Thu Jul 29 16:20:37.102243 2021]      response = handler(request, *args, **kwargs)
[Thu Jul 29 16:20:37.102244 2021]    File "/opt/celus/apps/logs/views.py", line 270, in preflight_check
[Thu Jul 29 16:20:37.102246 2021]      mail_admins('MDU preflight check error', body)
[Thu Jul 29 16:20:37.102248 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/django/core/mail/__init__.py", line 101, in mail_admins
[Thu Jul 29 16:20:37.102250 2021]      mail.send(fail_silently=fail_silently)
[Thu Jul 29 16:20:37.102252 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/django/core/mail/message.py", line 306, in send
[Thu Jul 29 16:20:37.102254 2021]      return self.get_connection(fail_silently).send_messages([self])
[Thu Jul 29 16:20:37.102256 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/django/core/mail/backends/smtp.py", line 103, in send_messages
[Thu Jul 29 16:20:37.102257 2021]      new_conn_created = self.open()
[Thu Jul 29 16:20:37.102259 2021]    File "/opt/virtualenvs/celus/lib/python3.7/site-packages/django/core/mail/backends/smtp.py", line 63, in open
[Thu Jul 29 16:20:37.102261 2021]      self.connection = self.connection_class(self.host, self.port, **connection_params)
[Thu Jul 29 16:20:37.102263 2021]    File "/usr/lib/python3.7/smtplib.py", line 251, in __init__
[Thu Jul 29 16:20:37.102265 2021]      (code, msg) = self.connect(host, port)
[Thu Jul 29 16:20:37.102267 2021]    File "/usr/lib/python3.7/smtplib.py", line 336, in connect
[Thu Jul 29 16:20:37.102268 2021]      self.sock = self._get_socket(host, port, self.timeout)
[Thu Jul 29 16:20:37.102270 2021]    File "/usr/lib/python3.7/smtplib.py", line 307, in _get_socket
[Thu Jul 29 16:20:37.102274 2021]      self.source_address)
[Thu Jul 29 16:20:37.102276 2021]    File "/usr/lib/python3.7/socket.py", line 727, in create_connection
[Thu Jul 29 16:20:37.102278 2021]      raise err
[Thu Jul 29 16:20:37.102280 2021]    File "/usr/lib/python3.7/socket.py", line 716, in create_connection
[Thu Jul 29 16:20:37.102281 2021]      sock.connect(sa)
[Thu Jul 29 16:20:37.102283 2021]  OSError: [Errno 101] Network is unreachable

Our installation is running on Debian 10, but that should not be an issue, right?

Not sure if there is a problem with the uploaded file itself or the network. It would be great if you could point me in the right direction. Let me know if you need more information.

Prerequisites RHEL/CentOS 8.0

What exactly need to be installed on the server before installing Celus?
(python3-devel , gcc, .....)

I'm trying to install Celus on a Centos-8 server in a container (docker). I'm progressing however now I run into a systemctl permission issue with postgresql:

playbook output :

PLAY [all] **************************************************************************************************

TASK [Gathering Facts] **************************************************************************************
ok: [celus.now.im]

TASK [celus : make sure some basic applications are installed] **********************************************
ok: [celus.now.im]

TASK [celus : start the redis server] ***********************************************************************
ok: [celus.now.im]

TASK [celus : ensure virtualenv and helpers are installed] **************************************************
ok: [celus.now.im]

TASK [celus : activate virtualenvwrapper] *******************************************************************
ok: [celus.now.im]

TASK [celus : get project source files from git] ************************************************************
ok: [celus.now.im]

TASK [celus : install project python dependencies into virtualenv] ******************************************
ok: [celus.now.im]

TASK [celus : ensure PostgreSQL is installed] ***************************************************************
ok: [celus.now.im]

TASK [celus : initialize the PostgreSQL cluster] ************************************************************
skipping: [celus.now.im]

TASK [celus : ensure PostgreSQL contrib package is installed] ***********************************************
changed: [celus.now.im]

TASK [celus : make sure PostgreSQL is running] **************************************************************
fatal: [celus.now.im]: FAILED! => {"changed": false, "msg": "Unable to start service postgresql: Job for postgresql.service failed because the control process exited with error code.\nSee "systemctl status postgresql.service" and "journalctl -xe" for details.\n"}

PLAY RECAP **************************************************************************************************
celus.now.im : ok=9 changed=1 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0

journalctl -xe output

-- Unit postgresql.service has begun starting up.
May 12 14:32:02 c0194e6d0b60 systemd[1]: systemd-journald.service: Failed to add fd to store: Operation not permitted
May 12 14:32:02 c0194e6d0b60 postgresql-check-db-dir[1031]: Directory "/var/lib/pgsql/data" is missing or empty.
May 12 14:32:02 c0194e6d0b60 postgresql-check-db-dir[1031]: Use "/usr/bin/postgresql-setup --initdb"
May 12 14:32:02 c0194e6d0b60 postgresql-check-db-dir[1031]: to initialize the database cluster.
May 12 14:32:02 c0194e6d0b60 postgresql-check-db-dir[1031]: See /usr/share/doc/postgresql/README.rpm-dist for more information.
May 12 14:32:02 c0194e6d0b60 systemd[1]: postgresql.service: Control process exited, code=exited status=1
May 12 14:32:02 c0194e6d0b60 systemd[1]: postgresql.service: Failed with result 'exit-code'.
-- Subject: Unit failed