rucio / jupyterlab-extension

Rucio Jupyterlab plugin (GSOC Project)

Rucio - Scientific Data Management

Rucio is a software framework that provides functionality to organize, manage, and access large volumes of scientific data using customisable policies. The data can be spread across globally distributed locations and across heterogeneous data centers, uniting different storage and network technologies as a single federated entity. Rucio offers advanced features such as distributed data recovery or adaptive replication, and is highly scalable, modular, and extensible. Rucio has been originally developed to meet the requirements of the high-energy physics experiment ATLAS, and is continuously extended to support LHC experiments and other diverse scientific communities.

Documentation

General information, API/REST description and guides can be found in our documentation or on our webpage.

Try it out

We provide a dockerized environment which serves both as a demo environment and a development environment. It includes all the necessary preconfigured components for multiple storage and transfers developments.

Developers

For information on how to contribute to Rucio, please refer and follow our CONTRIBUTING guidelines. We strongly recommend to use the dockerized environment for development.

Operators

To learn how to deploy and configure Rucio, consult the documentation available online.

Getting Support

If you are looking for support, please contact us via one of our official channels.

jupyterlab-extension's Issues

rucio-client version needs >=1.26.0

We are upgrading rucio versions on our escape instance and have noticed that the setup.py requires rucio-clients~=1.26.0. Could you change it to rucio-clients>=1.26.0, please? @didithilmy do you think this will break anything?

Failing test build

Because of outdated dependencies, the test build of the extension seems to fail.

For example https://github.com/rucio/jupyterlab-extension/actions/runs/6225180068

Once fixed the bdage for both workflows in the readme should be updated, as they are not displayed correctly.

This guide might be needed to migrate to JL v4: https://jupyterlab.readthedocs.io/en/latest/extension/extension_migration.html

Question: Variable failed to attach

Dear All,

Thanks for such a nice and useful job done with this tool!

I have been using the extension since a few days ago, and I have a problem where even when I find and successfully download a dataset, I can't attach a variable in a Python3 kernel notebook.

I am not sure if this is the right forum, but can I ask where to explore the reason for this failure? Or any other advice you can give me

Your help is very much appreciated,

Cheers,
Arturo

Implement testing framework to inspect extension working correctly

Compatibility with Lab3

Hi guys,

Lab 3 was released early this year and brings some changes on how extensions are built and deployed. The current extension is thus incompatible. How can we proceed to update this extension?

The Jupyter team has provided a guide and even some tools to ease the transition: https://jupyterlab.readthedocs.io/en/latest/extension/extension_migration.html

Diogo

Documentation, enable IPython Kernel extension

Enabling the IPyhton Kernel extension is critical for the extension usability, this section should have more visibility in the documentation

JupyterHub OIDC authentication method and process

Motivation

Enhance the JupyterHub documentation to include comprehensive instructions for configuring the Rucio extension and Rucio Command Line Interface (CLI) in a JupyterHub environment, such as Z2JH, with a focus on the OpenID Connect (OIDC) authentication method and process. The documentation should be user-friendly and provide step-by-step guidance to ensure seamless setup and configuration of Rucio in a JupyterHub environment.

Modifications

• rucio extension documentation for JupyterHub with OIDC authentication method, including
  • Docker image for single user
  • config.yaml for JupyterHub with necessary Python functions for OIDC tokens

There is some existing work that showcases the setup of the OIDC authentication method for Rucio in JupyterHub in the linked PRs.

There is also a demo repository hosted on my private GitHub account that could potentially be migrated to the Rucio organization or integrated into the https://github.com/rucio/k8s-tutorial repository.

In the end, the documentation should be hosted on the Rucio website as well.

Expected result

The expected outcome of this effort is comprehensive documentation that serves as a reference guide for setting up Rucio in a JupyterHub environment. This documentation should cover all the necessary configuration files, and Docker images, and provide clear instructions for a successful setup of Rucio with OIDC authentication in a JupyterHub environment.

Problems and tests

Currently, I'm testing the Rucio extension in download mode with rucio version 1.30.3 for local tests and 1.29.6 on JupyterHub. The auth server runs on version 1.29.6.

Token format

The decoded token jwt access_token looks like this:

{'wlcg.ver': '1.0', 'sub': '<hide>', 'aud': 'rucio', 'act': {'sub': '<hide>'}, 'nbf': 1681724949, 'scope': 'email openid profile', 'iss': 'https://iam-escape.cloud.cnaf.infn.it/', 'exp': 1681728549, 'iat': 1681724949, 'jti': '<hide>', 'client_id': '<hide>'}

A simple request to the Rucio whoami API with the token looks like this:

{"account_type": "SERVICE", "email": "[email protected]", "deleted_at": null, "updated_at": "2022-09-16T09:00:13", "status": "ACTIVE", "suspended_at": null, "created_at": "2022-09-16T09:00:13", "account": "dogosein"}

So the token seems to be valid and working.

Rucio Extension with token stored in an environment variable

When the above token is stored in an environment variable and as such specified in the config, the extension gets setup fine and DIDs can be listed, however when downloading a file ("Make Available") the following error is encountered:

Rucio Client did not succeed to get AuthN/Z URL from the Rucio Auth Server.                                    
This could be due to wrongly requested/configured scope, audience or issuer.
Process Process-1:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.8/site-packages/rucio_jupyterlab/rucio/download.py", line 35, in start_download_target
    results = RucioFileDownloader.download(dest_folder, did)
  File "/opt/conda/lib/python3.8/site-packages/rucio_jupyterlab/rucio/download.py", line 62, in download
    client = Client()
  File "/opt/conda/lib/python3.8/site-packages/rucio/client/client.py", line 77, in __init__
    super(Client, self).__init__(**args)
  File "/opt/conda/lib/python3.8/site-packages/rucio/client/baseclient.py", line 282, in __init__
    self.__authenticate()
  File "/opt/conda/lib/python3.8/site-packages/rucio/client/baseclient.py", line 929, in __authenticate
    self.__get_token()
  File "/opt/conda/lib/python3.8/site-packages/rucio/client/baseclient.py", line 827, in __get_token
    raise CannotAuthenticate('OIDC authentication failed for account=%s' % self.account)
rucio.common.exception.CannotAuthenticate: Cannot authenticate.
Details: OIDC authentication failed for account=None

At the moment is it not quite clear how the account needs to be set.

Using the rucio CLI in the notebook

Using the OIDC token in the notebook with the rucio CLI for download does not work and throws the following error:

2023-04-17 08:38:48,473 DEBUG   logging.py      gfal.Default: downloading file from gsiftp://eulakeftp.cern.ch:2811/eos/eulake/tests/rucio_test/eulake_1/elena_test/c5/3a/test-file-rucio-2023-04-17-01.txt to /home/jovyan/elena_test/test-file-rucio-2023-04-17-01.txt.part
2023-04-17 08:38:48,524 DEBUG   downloadclient.py       The requested service is not available at the moment.
Details: An unknown exception occurred.
Details: Could not open source: globus_ftp_control: gss_init_sec_context failed GSS failure:  GSS Major Status: General failure GSS Minor Status Error Chain: globus_gsi_gssapi: Error with gss context globus_gsi_gssapi: Error with gss credential handle globus_credential: Valid credentials could not be found in any of the possible locations specified by the credential search order. Valid credentials could not be found in any of the possible locations specified by the credential search order. Attempt 1 globus_credential: Error reading host

Further tests

Further tests have shown that:

• download of the same file with a different authentification method (e.g. X509) works fine in the extension as well as with the rucio CLI in JupyterLab Notebook
• download on a local machine with a token generated by the rucio CLI works fine

This points to an issue with the token or missing configuration with the extension. Further debugging needs to be done to find the missing pieces.

File download issue using extension "Something went wrong"

Hi,

I already opened this issue in the old repo. When I am trying to download a test file I got the error "Something went wrong". However, since the data lake was down last week I could not do the test and provide the log file. The log file looks like this:

rjlab.log

Thanks in advance,
Maisam

Update extension for v32 (and higher) compatibility

Tasks

• #26
• #27
  • Support to both versions ?
• #29

JupyterLab version (v3 to v4) update