irods / contrib

A pooled collection of community-contributed code that works alongside iRODS

contrib

A pooled collection of community-contributed code that works alongside iRODS

• avu-protect: Protect AVUs from being modified by users
• ec2-irods-demo: Deployment scripts for Amazon EC2 image
• irods-docker: Deployment scripts for iRODS Docker images
• irods-mode: Emacs Major Mode for iRODS Rules
• microservices: Demonstration non-core microservices
• vim-syntax: Syntax detection for iRODS rule language in vim
• generate_targz_with_submodules.sh - Generates tar.gz of git superproject, including submodules
• parallel_recursive_irm.py - Uses GNU Parallel to speed up recursive irm
• prescan_profiler.py - Counts files/dirs/symlinks in a nested tree (local or S3)
• restore_unixstat_to_resource.py - Restores unixstat information from the catalog
• run_scanbuild.sh - Generates clang scan-build report for main iRODS source
• sync_ad_accounts.sh - Syncs LDAP/AD user list into iRODS
• vm-prep-ugm2015.md - Preparation steps for 2015 User Group Meeting VirtualBox VM

contrib's Issues

msisync_to_archive removal from contrib

Hi,

The function msisync_to_archive is now part of iRODS 4.1.8. Will it be removed from the contrib once iRODS 4.1.8 is released? Otherwise we will have a conflict if we install both 4.1.8 and the contrib package. Which one will it use than?

Greetings,

Robert

Investigate swapping JDK for JRE in elk stack container

With #26, we are now using Temurin's JDK instead of the JDK/JRE provided by the distribution or the JDK/JRE bundled with elasticsearch.

The decision not to use elasticsearch's bundled JDK/JRE was made for two reasons:

• To de-bloat the container image. Having multiple JDK/JRE installations uses a lot of space.
• To get everything using the same JDK/JRE

Temurin was chosen over the distro-provided JDK/JRE for a few reasons:

• The Hotspot AdoptOpenJDK flavor of JVM handles memory pressure very well.
• The AdoptOpenJDK flavors of JVM work well in containers.
• I've just generally had better experience with AdoptOpenJDK flavors of JVM than any others.

Given that we are no longer installing plugins in logstash, it is likely that we do not need a full JDK.
However, after the AdoptOpenJDK working group was absorbed into the Eclipse foundation and became Adoptium, JRE-only packages (and OpenJ9 packages) were no longer provided in the apt repositories.
Eclipse provides a focal-based docker image containing JRE-only Temurin 17 _[Dockerfile], but it is not set up to work properly with Ubuntu's java-common system so it cannot be used.
Fortunately, Adoptium has a couple of blog posts that might help us out:

• https://blog.adoptium.net/2021/08/using-jlink-in-dockerfiles/
• https://blog.adoptium.net/2021/10/jlink-to-produce-own-runtime/

Microservice registry

A method is needed to track the set of micro-services that are being used. The iCAT catalog has state variables defined for
MSRVC_ID, MSRVC_NAME, MSRVC_VERSION, MSRVC_LANGUAGE, MSRVC_CREATE_TIME, MSRVC_MODIFY_TIME

Associated micro-services are
msiAdmReadMSrvcsFromFileIntoStruct
msiAdmInsertMSrvcsFromStructIntoDB
msiAdmRetrieveMSrvcsFromDBIntoStruct
msiAdmWriteMSrvcsFromStructIntoFile

Docker image directory structure for volume mounting

Hi there,

I was wondering if there is a suggested way of mounting docker volume(s) to the iRODS docker container (iCat + iDrop) to keep the data persistent.

I checked the dockerfile, but couldn't figure out the directory structure of the docker image. Hoping if there is any suggestion/documentation for this.

Also, is there an up-to-date docker image we can use? From this dockerfile, it seems to be in version 4.1.3, and this image I found on DockerHub is in version 4.0.3. Not sure if the directory structure would change across versions?

Rule registry

Provide support for a registry of rules. The metadata is available in the iCAT catalog for
RULE_BASE_NAME which defines the rule set
RULE_NAME, RULE_ID, RULE_BODY, RULE_RECOVERY, RULE_INPUT_PARAMS, RULE_OUTPUT_PARAMS, RULE_DESCR_1, RULE_DESCR_2, RULE_EVENT, RULE_DOLLAR_VARS, RULE_ICAT_ELEMENTS, RULE_VERSION

Micro-services to manipulate the registry are;
msiAdmReadRulesFromFileIntoStruct,
msiAdmInsertRulesFromStructIntoDB
msiAdmRetrieveRulesFromDBIntoStruct,
msiAdmWriteRulesFromStructIntoFile

Micro-service to add token names

The iadmin command support the addition of token names to a token namespace. An equivalent operation is needed from a micro-service
iadmin
at tokenNamespace Name [Value1] [Value2] [Value3](add token)
rt tokenNamespace Name [Value1](remove token)

iRODS major mode for Emacs

It would be nice to have an Emacs major mode that includes at least color syntax highlighting for the iRODS rules language. I have a first draft of this that I can contribute.

new microservice to rebalance a single data object

From @rwmoore:

Current resources:

demoResc
LTLResc:passthru
└── LTLRepl:replication
    ├── LTLRenci:unixfilesystem
    └── LTLSils:unixfilesystem

I put a file on demoResc, and tried to use msiDataObjRepl() to replicate the file. Using

• "destRescName=LTLRenci" gives an error "DIRECT_CHILD_ACCESS"
• "destRescName=LTLRepl" gives an error "DIRECT_CHILD_ACCESS"
• "destRescName=LTLResc" works. Two files are created, one on LTLRenci and one on LTLSils.

The problem occurs when a file is created on LTLSils through execution of a script for encryption. I want to replicate the encrypted file to LTLRenci, but cannot specify the destination resource.

What is the correct way to replicate a file that already exists on one of the resources specified by a replication node? Previously I invoked a reBalance operation on the replication node. But I only needed to reBalance a single file.

ec2-irods4.0.3-idw2 should set hostname to public hostname before running setup_irods.sh

Integrate SANBI-SA/irods_docker for persistent storage

Hi there

At a recent code jamboree we created https://github.com/SANBI-SA/irods_docker that splits the Docker iRODS catalogue server into multiple containers linked using docker-compose, with a data-only container to facilitate persistent storage of iRODS data and configuration. I'm interested in contributing to this to irods/contrib if you are interested.

Encryption micro-service

Given an encryption key and a choice of encryption algorithm, a micro-service is needed to apply the encryption algorithm to a specified file, and update the file size.

stop using systemd in elk stack container

With #26, our elk container will be updated to ES 8. Starting with ES 7, elasticsearch, kibana, and logstash have switched from init.d scripts to systemd unit files.
In order to get #26 done in a timely manner, we followed ubi8-init's pattern for running systemd in a docker container.
However, systemd is a bit overkill for our purposes, and using it requires passing --privileged when running the container.

OSCS POC

create msifree_microservice_out

This microservice can be used to fix leaks like those in irods/irods#2929.

Currently implemented to free the output of:

• msiExecCmd()

User permissions for updating metadata

An ability to specify an access control on users for permission to manipulate metadata on a file is needed. This would be a lower permission than write, but higher than read.

new api plugin to list operations per plugin type

To automate the documentation of all dynamic PEPs, a list of operations, per plugin type, should be generated and sent to stdout as JSON.

• walk API table
• load all resources
• get array of operations
• dump microservice table
• dump all plugins
  • network
  • auth
  • database
  • resource
  • api
  • microservice
  • rule engine (needs introspection operation (list_rules) )
    • not unique-ified
    • boxed by language / instance

msisync_to_archive does not work for normal rods user anymore.

We did an put of a file using a normal rods user

09:35 irodstest2.storage.sara.nl:/home/robertv
robertv$ iput -R eudat rpmbuild/RPMS/noarch/irods-eudat-b2safe-dpm-client-1.0-0.noarch.rpm tokkie.rpm

09:35 irodstest2.storage.sara.nl:/home/robertv
robertv$ ils -l
/bob/home/robertv:
 ...
  robertv           0 eudat;eudatCache        16372 2015-12-03.09:35 & tokkie.rpm

There is a rule fired which replicates it to archive:

09:35 irodstest2.storage.sara.nl:/home/robertv
robertv$ iqstat -a
id     name
10437
                        #writeLine("serverLog","filePath: $filePath");
                        *CompoundRescName="eudat"
                        *CacheRescName   ="*CompoundRescName;eudatCache";
                        *ArchiveRescName ="*CompoundRescName;eudatPnfs";
                        writeLine("serverLog","Execute command to replicate (in resource *CompoundRescName) $objPath ($filePath) to *ArchiveRescName, because of put");
                        msisync_to_archive("*CacheRescName", $filePath, $objPath );
                |

This now fails if it is a normal rodsuser and NOT the rodsadmin.

Dec  3 09:37:12 pid:17810 NOTICE: writeLine: inString = Execute command to replicate (in resource eudat) /bob/home/robertv/tokkie.rpm (/var/lib/eudatCache/home/robertv/tokkie.rpm) to eudat;eudatPnfs, because of put
XXXX - last: eudatCache
XXXX - prev: eudat
XXXX - resolve
XXXX - success
XXXX - auto_repl :: off
Dec  3 09:37:12 pid:17810 NOTICE: rsDataObjRepl - Failed to replicate data object.
msisync_to_archive - fileModified failed [[-]   iRODS/server/drivers/src/fileDriver.cpp:723:fileModified :  status [CAT_INSUFFICIENT_PRIVILEGE_LEVEL]  errno [] -- message [fileModified - Failed to call modified interface.]
        [-]     libcompound.cpp:496:repl_object :  status [CAT_INSUFFICIENT_PRIVILEGE_LEVEL]  errno [] -- message [Failed to replicate the data object [/bob/home/robertv/tokkie.rpm] for operation [sync_object]]

] - [-830000]
Dec  3 09:37:12 pid:17810 ERROR: executeRuleAction Failed for msisync_to_archive status = -830000 CAT_INSUFFICIENT_PRIVILEGE_LEVEL
Dec  3 09:37:12 pid:17810 NOTICE: executeRuleBody: Microservice or Action msisync_to_archive Failed with status -830000
Dec  3 09:37:12 pid:17810 DEBUG: execMicroService3: error when executing microservice
line 6, col 3
                        msisync_to_archive("*CacheRescName", $filePath, $objPath );
                        ^

Dec  3 09:37:12 pid:17810 NOTICE: postProcRunRuleExec: exec of freq: 1h DOUBLE UNTIL SUCCESS OR 6 TIMES
Dec  3 09:37:12 pid:17810 NOTICE: modExeInfoForRepeat: rulId=10437,opStatus=-830000,nextRepeatStatus=4
Dec  3 09:37:12 pid:17810 NOTICE: Rule id 10437 set to run again at 1449135432 (frequency 2h DOUBLE UNTIL SUCCESS OR 5 TIMES. ORIGINAL TIMES=6 seconds)
Dec  3 09:37:12 pid:17810 NOTICE: Agent exiting with status = -830000

This works for the rodsadmin. Can this be fixed again?

In a previous version this worked.

Greetings,

Robert Verkerk

update 3.x microservices to work with 4.2+

• DATA_EXPIRY modification in msiSysMetaModify (irods/irods#2885)
• modify file size in msiSysMetaModify (irods/irods#2886)
• create microservice to encrypt a file (irods/irods#2887)
• port msiStoreVersionWithTS (irods/irods#2889)
• port msiDataObjAutoMove (irods/irods#2890)
• port msiGetDataObjACL (irods/irods#2891)
• port msiCopyAVUMetadata (irods/irods#2892)
• port msiLoadACLFromDataObj (irods/irods#2893)
• port msiCreateUserAccountsFromDataObj (irods/irods#2894)
• create microservice to modify META_DATA_ATTR_UNITS (irods/irods#2895)
• create microservice to modify META_COLL_ATTR_UNITS (irods/irods#2896)
• port msiGuessDataType (irods/irods#2897)

Remove self-generated certificates. Use http instead of https.

update ec2 instance demo for 4.1.8

hostsname being derived from $HOSTNAME doesn't work well on boot2docker

Boot2docker containers are available through an internal 192.168 network. But $HOSTNAME resolves to the Mac OSX host's name. Container environment variable $hostsname is set to $HOSTNAME, and the .groovy config file for iDrop-web is populated with this variable as a prefix. Thus, attempts to access iDrop Web fail to find the container.

Either update documentation or update how .groovy file is modified.

irods_audit_elk_stack container RabbitMQ test user missing

The irods_audit_elk_stack image has RabbitMQ user named test added to it when the when the image is built. The Dockerfile uses rabbitmqctl to add the user to a temporarily running RabbitMQ broker. Unfortunately, this user doesn't exist (isn't accessible?) from a container instantiated from this image.

RabbitMQ persists its data, like users, keyed to the its server's node name. By default, the node name is rabbitmq@$(hostname). When the test user is created and persisted during build time, the host name is something random, and very likely different from the host name of a container launched from the image. This causes the RabbitMQ broker running in the container to not know about the test user that was created when the image was built.

Fortunately, the node name is configurable using the NODENAME parameter in the file /etc/rabbitmq/rabbitmq-env.conf, If NODENAME is set in this file with the host name being localhost, e.g., NODENAME=rabbitmq@localhost, prior to rabbitmq-server being started during the image build, the node name will be rabbitmq@localhost when test user is created. When the container is started, the node name will be the same, and the broker will know about test user.

Here's a modified version of the RabbitMQ configuration command that sets the node name to localhost.

# Install RabbitMQ plugins and create administrator account
RUN rabbitmq-plugins enable \
        rabbitmq_amqp1_0 \
        rabbitmq_management && \
    echo 'NODENAME=rabbitmq@localhost' > /etc/rabbitmq/rabbitmq-env.conf && \
    chmod 755 /etc/rabbitmq/rabbitmq-env.conf && \
    /etc/init.d/rabbitmq-server start && \
    rabbitmqctl add_user test test && \
    rabbitmqctl set_user_tags test administrator && \
    rabbitmqctl set_permissions -p / test ".*" ".*" ".*" && \
    /etc/init.d/rabbitmq-server stop