diracgrid / diracos2 Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 9.0 225 KB

License: GNU General Public License v3.0

Shell 19.49% Python 80.51%

diracos2's People

Contributors

Watchers

Forkers

chrisburr chaen aldbr fstagni atsareg andresailer rupozzi simon-mazenoux iueda

diracos2's Issues

Fix deprecation notices in CI config

The CI has started showing deprecation warnings so we should fix them at some point:

The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: actions/checkout, actions/cache, conda-incubator/setup-miniconda, actions/upload-artifact, conda-incubator/setup-miniconda, actions/cache, actions/checkout

Add authlib, pyjwt, dominate

The following packages are required for #5149:

pytest-mock
authlib >= 1.0.0.a2
pyjwt >= 1.7.1
dominate

Add missing commands required for the server

These commands should be included and tested:

for cmd in glite-ce-job-submit glite-ce-job-status glite-ce-delegate-proxy glite-ce-job-cancel condor_submit condor_history condor_q condor_rm condor_transfer_data ldapsearch; do
  echo $cmd
  which $cmd
done

ARM/CentOS8

Leaving it here, I did not do any follow-up.

Given the availability of 1 ARM host from lxplus (ssh lxplus8-arm) I just tried installing DIRACOS2 and:

[fstagni@oci-lxplus-arm-001 ~]$ $ bash DIRACOS-Linux-$(uname -m).sh
-bash: $: command not found
[fstagni@oci-lxplus-arm-001 ~]$ bash DIRACOS-Linux-$(uname -m).sh
PREFIX=/afs/cern.ch/user/f/fstagni/diracos
Unpacking payload ...
  0%|                                                                                                                                                                                                                                                     | 0/273 [00:00<?, ?it/s]Could not remove or rename /afs/cern.ch/user/f/fstagni/diracos/pkgs/c-ares-1.18.1-hf897c2e_0ohqlqpab/.__afs53C7.  Please remove this file manually (you may need to reboot to free file handles)
Could not remove or rename /afs/cern.ch/user/f/fstagni/diracos/pkgs/lerc-3.0-h01db608_0unc3jdgq/.__afs2537.  Please remove this file manually (you may need to reboot to free file handles)
Could not remove or rename /afs/cern.ch/user/f/fstagni/diracos/pkgs/tzdata-2021e-he74cb21_0_kg9fwos/.__afsB61F.  Please remove this file manually (you may need to reboot to free file handles)
Could not remove or rename /afs/cern.ch/user/f/fstagni/diracos/pkgs/mysql-common-8.0.27-h8af1aa0_3ffz0kwqp/.__afsF642.  Please remove this file manually (you may need to reboot to free file handles)
Could not remove or rename /afs/cern.ch/user/f/fstagni/diracos/pkgs/graphite2-1.3.13-h7fd3ca4_1001njlbb8ea/.__afs4812.  Please remove this file manually (you may need to reboot to free file handles)
Could not remove or rename /afs/cern.ch/user/f/fstagni/diracos/pkgs/jpeg-9e-h3557bc0_0ha4_ytio/.__afs0D19.  Please remove this file manually (you may need to reboot to free file handles)
Could not remove or rename /afs/cern.ch/user/f/fstagni/diracos/pkgs/icu-69.1-h3621d8e_0jtmdj7yu/.__afsE7C.  Please remove this file manually (you may need to reboot to free file handles)
Could not remove or rename /afs/cern.ch/user/f/fstagni/diracos/pkgs/giflib-5.2.1-hb9de7d4_2yhrk0v_w/.__afsCDEF.  Please remove this file manually (you may need to reboot to free file handles)
Could not remove or rename /afs/cern.ch/user/f/fstagni/diracos/pkgs/bzip2-1.0.8-hf897c2e_4g__o3iq8/.__afs7266.  Please remove this file manually (you may need to reboot to free file handles)
concurrent.futures.process._RemoteTraceback:                                                                                                                                                                                                                                      
'''
Traceback (most recent call last):
  File "concurrent/futures/process.py", line 384, in wait_result_broken_or_wakeup
  File "multiprocessing/connection.py", line 256, in recv
TypeError: __init__() missing 1 required positional argument: 'msg'
'''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "entry_point.py", line 69, in <module>
  File "concurrent/futures/process.py", line 559, in _chain_from_iterable_of_lists
  File "concurrent/futures/_base.py", line 608, in result_iterator
  File "concurrent/futures/_base.py", line 445, in result
  File "concurrent/futures/_base.py", line 390, in __get_result
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
[1144884] Failed to execute script 'entry_point' due to unhandled exception!

Add sphinx-rtd-theme, sphinx-panels

Based on the discussion in #5313 raised by #5309 issue, it seems that the RTD theme can be applied in the new release. The following packages are used in it:

sphinx-rtd-theme
sphinx-panels

MAMBA_ROOT_PREFIX: unbound variable

The error is related to micromamba.
When I execute install_site.sh, I get the following error:

/opt/dirac/versions/v10.4.8-1646818775/Linux-x86_64/diracosrc: line 108: MAMBA_ROOT_PREFIX: unbound variable

Seems like the MAMBA_ROOT_PREFIX should be set before calling the following line in diracosrc:

eval "$(${DIRACOS}/bin/micromamba shell hook activate -s bash)"

The following piece of code (weird by the way) seems to trigger the error (this is what we get from the command above):

    if [ -n "${_CE_CONDA+x}" ] && [ -n "${WINDIR+x}" ]; then
        PATH="${MAMBA_ROOT_PREFIX}/condabin:${PATH}"
    else
        PATH="${MAMBA_ROOT_PREFIX}/condabin:${PATH}"
    fi

This is also explained in the documentation at the end of the "Quickstarts" section here: https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html?highlight=MAMBA_ROOT_PREFIX#quickstarts.

micromamba expects to find the root prefix set by $MAMBA_ROOT_PREFIX environment variable. You can also provide it using CLI option -r,--root-prefix.

What should be the value of MAMBA_ROOT_PREFIX?

Stop using fork of constructor

Currently we're using a minor fork of constructor since #19. Assuming conda/constructor#440 is merged, this should be change once there is a suitable release.

How to handle updates?

Yes, I think it's early enough that you can bump the version in DIRACOS2 in the next patch release (I would not do it so > easily if it was production code, but as of now it's still experimental)

Originally posted by @fstagni in DIRACGrid/DIRAC#5000 (comment)

It doesn't need to be solved now but at some point we will have to decide how to handle "riskier" updates as in DIRACOS 1 we didn't have to care too much given how static CentOS 6 was. For example:

Updating the middleware XRootD/ARC/...
Updating other dependencies, gsoap/boost/...
Updating minor Python versions (these should be stable and fairly low risk)
Major updates, I have nothing specific in mind but imagine a siltation like Python 2 to Python 3

My initial thoughts:

We should avoid having active releases which cannot be used with the latest DIRACOS 2 version, especially considering there could be something like an "urgent" OpenSSL bug which requires everyone to update at short notice regardless of the DIRAC version they're running. I think this can be handled relatively by making sure that bug fixes for changes in dependencies target an appropriate branch.
As there aren't many downstream dependencies of things like XRootD (and I'm responsible for most of them) we can fairly easily control the versions we're given and rebuild older ones like XRootD 4.x if needed
Here we basically need to follow the "global pinning" in conda-forge but I don't expect this to be a problem. We might need to be careful with things like OpenSSL 3 when it's release but that migration will be relatively slow with both OpenSSL 1.1.1 and OpenSSL 3 being build for months so we will have time to run certifications with the new version.
Since Python 3.4ish these have been fairly stable. We might want to run a certification with the latest version but I don't think we should be too afraid of updates.
This would become DIRACOS 3 and we would likely need to support both DIRACOS 2 and 3 for a reasonably long time. Conda-forge would also likely support both variants for a long time so it wouldn't be a problem (this already happened for updating the C++ ABI and Python 2 -> 3).

Optimise installation

Currently the installer hasn't been optimised at all however once things are stable it should be adapted to minimise

Compressed installer size
Installation file count (and hence IO load)
Installation time

The following might be helpful for this:

Removing unused packages
Using .conda packages instead of .tar.bz2
Using micromamba instead of conda-standalone
Modifying the packages themselves to split into smaller components or to avoid dependencies

Of course it's important that DIRACOS2 is still easy to build and maintain, even if that comes at a small cost.

Add rucio-clients

As this is in DIRACOS it should probably also be added to DIRACOS2. Some pointers for whoever wants to do this:

It needs to be added to conda-forge via staged-recipes
The recipe itself can be mostly generated by grayskull, there is an online version of it here
Once the PR is ready someone can ping me on it to review/merge
Finally, it can be added to the list of specs in https://github.com/DIRACGrid/DIRACOS2/blob/main/construct.yaml

A couple of useful tutorials:

Official docs: https://conda-forge.org/docs/maintainer/adding_pkgs.html
Very detailed step-by-step blog post: https://gallon.me/3-simple-steps-to-build-a-python-package-for-conda-forge/

Add other owners to the diracgrid conda channel

Currently I'm the only owner of https://anaconda.org/diracgrid/.

@fstagni Can you make an account, or assign someone else to, so that there can be at least two people with access?

Remove GFAL_PLUGIN_DIR and associated variables

Some sites "helpfully" set variables like GFAL_PLUGIN_DIR and cause DIRACOS2 to not work (DIRACOS1 wasn't affected as it was relying on these).

We should unset them in the diracosrc.

Add a preinstall script to check if there is enough disk space remaining

Currently the error message is horrible if you run out of disk space during the install. We could use a pre-install script to make sure there is a minimum amount of space available.

See https://github.com/conda/constructor/blob/master/CONSTRUCT.md#pre_install

Run DIRAC integration tests in CI

Issues in DIRACOS 2.22

During yesterday's hackathon we noticed that there were data management issues with DIRACOS 2.22.

Eventually me and @chaen figured out that it only affects using SRM to communicate with dcache sites and comes from using the latest version of the globus community toolkit.

That led us to this issue gridcf/gct#174 which we verified using:

$ GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=99 GLOBUS_GSSAPI_DEBUG_LEVEL=99 GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=99 timeout 2 gfal-ls -vvv 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/' 2>&1 | grep -i 'Using TLS'; echo Exited with $?
Using TLSv1.3.
Exited with 143

$ GLOBUS_GSSAPI_MAX_TLS_PROTOCOL=TLS1_2_VERSION GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=99 GLOBUS_GSSAPI_DEBUG_LEVEL=99 GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=99 timeout 2 gfal-ls -vvv 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/' 2>&1 | grep -i 'Using TLS'; echo Exited with $?
Using TLSv1.2.
Exited with 0

$ GLOBUS_GSSAPI_MAX_TLS_PROTOCOL=TLS1_3_VERSION GLOBUS_GSI_OPENSSL_ERROR_DEBUG_LEVEL=99 GLOBUS_GSSAPI_DEBUG_LEVEL=99 GLOBUS_GSI_GSS_ASSIST_DEBUG_LEVEL=99 timeout 2 gfal-ls -vvv 'srm://gfe02.grid.hep.ph.ic.ac.uk:8443/srm/managerv2?SFN=/pnfs/hep.ph.ic.ac.uk/' 2>&1 | grep -i 'Using TLS'; echo Exited with $?
Using TLSv1.3.
Exited with 143

Unfortunately we currently can't easily rebuild gct and patch it in conda-forge as the sources are unavailable gridcf/gct#194.

As a temporary workaround I'll make a new release that sets GLOBUS_GSSAPI_MAX_TLS_PROTOCOL=TLS1_2_VERSION in the diracosrc.

Files not found when sourcing diracosrc with latest diracos release

After running the installation script, sourcing the diracosrc gives the following errors:

$ source ~/.local/diracos/diracosrc 
realpath: '': No such file or directory
realpath: /etc/grid-security/certificates: No such file or directory
realpath: '': No such file or directory
realpath: '': No such file or directory

Review of DIRACOS2

As discussed at the BiLD meeting on 8th October 2020, @andresailer will be reviewing DIRACOS2. I'm opening this issue to keep track of the process.

Ready to be reviewed:

Recipes for DIRAC specific conda packages in DIRACGrid/management#23
This repository, though there are still some remaining issues to deal with that are tracked by the v2.0 milestone.

Still to be implemented:

DIRAC packaging improvements (DIRACGrid/DIRAC#4774)
Integration with dirac-install (DIRACGrid/DIRAC#4775)

cc @fstagni @atsareg @chaen @petricm

Test linux-ppc64le and linux-aarch64 installers

It would be possible to test the linux-ppc64le and linux-aarch64 installers using QEMU in CI so we probably should. 😄

Add fts-rest

@chaen Has there been any news on the state of fts-rest and Python 3?

Add automatic changelog generation

Currently the release notes are being filled with a TODO placeholder:

DIRACOS2/scripts/make_release.py

Lines 117 to 119 in 95751df

 "## Changes", 

 "* TODO", 

 "",

This should be replaced with a similar system based on the PR messages, as is used in the other DIRACGrid repositories.

As far as I'm aware the only DIRACOS extension is LHCbDIRACOS which wouldn't require a full DIRAOCS extension to implement and could rely on just the setuptools metadata. I'll list this under the "later" milestone until there is a solid use case.

Add DIRAC 8.0.x to CI

Currently the only tests run for v7.2, v7.3 and integration.

Relax M2Crypto < 0.38 constraint after it's fixed

Revert #43 after https://gitlab.com/m2crypto/m2crypto/-/issues/298 is fixed

make_release scripts needs docstrings

DIRACOS2/scripts/make_release.py

Line 1 in 5a5483b

#!/usr/bin/env python

For #7

For maintainability, the script needs a number of docstrings to explain what it does and what the individual functions do. They can be brief, but at the moment there are none.

https://github.com/DIRACGrid/DIRACOS2#making-a-release could refer to the make_release script

warning and symlink issue with DIRACOS v2.34

Downloading the latest version, and simply executing the bash command, we get for all packages warnings like
warning libmamba Extracted package cache '/tmp/ddd/diracos/pkgs/six-1.16.0-pyh6c4a22f_0' has invalid url

and at the end an error message related to the creation of a symbolic link

Or to execute a single command in this environment, use:

    micromamba run -p /tmp/ddd/diracos mycommand

ln: failed to create symbolic link '/tmp/ddd/diracos/lib/python3.9/site-packages/etc': No such file or directory

DIRACOS has been installed sucessfully in /tmp/ddd/diracos

I don't yet know if it actually impacts the behavior or not, I'll check it asap but we may need to rollback

Generate aarch64 and ppc64le installers for pilot clients

Most of the work required for this has already been done. Possible tasks remaining are (these are all optional at this stage depending on how we decide to proceed):

Re-enable support for cross-platform constructor builds
Find a CI provider for running tests natively on an the new architecture(s). Drone CI and Travis CI are currently used by conda-forge very successfully.

If packages are found to be missing from conda-forge, they should be added to the conda-forge arch-rebuild migration by making a PR to add the package names to this file. The status of the migration can then be tracked using the conda-forge status page.

[discussion] oidc-agent package

I am currently testing the ability to use an oidc-agent for a DIRAC Client installation. I'm wondering if there may be any caveats about adding a this package installation, and how to properly add it to a client installation?
It also has a python library liboidcagent to communicate with the oidc-agent through UNIX domain sockets.

Move to apptainer and improve container testing

Singularity can be a little fiddley with relocation. As part of moving to apptainer we should improve the DIRACOS2 unit tests to better catch issues.

Prompt problems with DIRACOS v2.34

Hi,

We've noticed that the latest DIRACOS seems to have trouble with the default prompt on RHEL. When the diracosrc is sourced, the prompt gets set to just "(base) " with the old prompt going missing:

[user@host dirac_dir]$ source diracos/diracosrc 
(base) # Where did my prompt go?
(base)

The old behaviour was to keep the old prompt, e.g.:

[user@host dirac_dir]$ source diracos/diracosrc 
(base) [user@host dirac_dir]$

On RHEL PS1 is not exported by the system-wide bashrc, for some reason this upsets v2.34 but not earlier versions (Running "export PS1" is a workaround). I imagine something has changed upstream, but it wasn't obvious to me what and I couldn't find any bug reports for this. Any ideas?

Regards,
Simon

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble