nbisweden / aida-data-hub Goto Github PK

Meetings with NSC on Thursday and Friday.

Rough: 10MSEK ≲ 113U.

However: Need updated estimates for correct storage space and gpu model.

Checklist of items that newcomer's accounts should be added to:

• Bigpicture wp2 and task forces
• NBIS GitHub Repo
• AIDA website GitHub Repo
• Neicnordic GitHub Repo
• AIDA wiki
• AIDA Nextcloud
• Zenhub Board
• Slack Channel
• Miro Board
• AIDA Data hub Google Drive
• CMIV Firewall <- Pass test on sec policy first!
• Subscribe to bigpicture email updates <- fill in the consent form and you will be added!
• NBIS Web page
• CMIV IT Teams
• AIDA compute mailing list

Help SCAPIS make imaging data available through the AIDA Data Hub DGX-2 service.

AIDA Data in routine

• Contact the project lead
• Check if the dataset is appropriately anonymized and is legal for sharing.
• Prepare the legal agreements if needed
• Get approval from the Clinical council for prioritizing this data type
  • Contact project lead for suitable data type description.
• Upload the data
• Create a public dataset register page

Tasks related to AIDA Days and 5th Year Celebration

Checklist for this issue:

• Ensure Betul and Emre have access to AIDA Datahub safespring environment
• Deploy Federated Node HA-proxy - repo link
• Deploy Federated Node Main Cluster - repo link
• Deploy Cytomine in AIDA Data Hub Safespring environment

• Schedule service downtime
  • announce to users on #aida-dgx2
  • announce to users on CMIV IT users
• Take system offline, and install ESL7.lod on replacement disk.

Develop the AIDA Data Hub onboarding document.

Include rclone instructions in the Help page of the AIDA Data Hub website

Bigpicture tools to be installed:

• Docker
• Docker Compose
• Minikube
• Helm chart
• Terraform

The main idea of this card is to install the tools in the local system. Each of these services is quite big to know in-depth and it takes a while to know each of them. RULE NUMBER 1 in this project is to NOT GET OVERWHELMED by these big services.

To write a nicer report, we want to run with a provided container on data we've been allowed to use for this. RÖ has provided such data and uploaded into BP Archive.

This needs to be made available for running within safespring, easiest way likely to a VM using grand-challenge or similar.

• Contact Martin Norling and ask for guidance with access to BP Archive to pull data provided by RÖ.
• You will hopefully receive guidance, including additional work e.g. by RÖ to allow you access (might be skipped if deemed better by SDA-dev team)
• Set up tool chain (sda-cli download?)
• Pull data

Follow-up with old AIDA activities

Cf email subject "Running a wp4 AI model against stored wp3 data on wp2 platform infrastructure" for description and collaborators.

• Number of free Ethernet ports in switch

9 and the next step is to discuss it with Hakan

• get clearance to procure more disk
• wait for offer
• coordinate with procurement people
• buy more disk
• schedule visit to compute room to install disk
• install disk
• check with Dell what firmware to install
• upgrade firmware on disk (if needed)

Website repositary and follow the instructions in README.

• Set up compute project LiU internal agreement for liu-mit.
• Get connection details for all relevant project members from Daniel.
• Set up compute project.
• Set up accounts for all relevant members.

SDA team has started migrating end user documentation Quarto, we should check it out.

AIDA Data sharing routine

• Contact the project lead
• Check if the dataset is appropriately anonymized and is legal for sharing.
• Prepare the legal agreements if needed
• Upload the data
• Create a public dataset register page

• Wait for new offer from correct Dell vendor.
• Update purchase order to match
• Get CMIV director signature on order
• Email order to vendor (with offer attachment).
• Register order & offer.
• Make sure service is actually extended.

• Incoming
• Waiting for copyright holder approval to share
• Preparation
• Waiting for recipient signature
• Waiting for countersignature
• Sharing
• Done

• Announce service stop (Monday Nov 28 09:00-)
• Upgrade firmware as per instructions in Slack.

Learn about how cytomine works and give a demo of the application.

• Get the exact same data that Uppsala will use for upload
• Be sure service is running
• Contact Uppsala to ensure the service is available to use at the Demo and not being messed by the pentesters
• Plan a very simple Demo so everything should work as normal

• Incoming
• Waiting for copyright holder approval to share
• Preparation
• Waiting for recipient signature
• Waiting for countersignature
• Sharing
• Done

• Incoming
• Waiting for copyright holder approval to share
• Preparation
• Waiting for recipient signature
• Waiting for countersignature
• Sharing
• Done

• Get time frame for new disk arrival: planned Monday, 11/28/2022.
• Schedule visit to computer room to replace faulty disk (late next week?).
• Return faulty disk (if needed).

AIDA Data sharing routine

• Contact the project lead
• Check if the dataset is appropriately anonymized and is legal for sharing.
• Prepare the legal agreements if needed
• Upload the data.
  • Replicate data to dgx2.
  • Verify transfer.
• Create a public dataset register page

Update the AIDA Data Hub booking sheet Reference

• Check with Caroline about the status of different projects
• Contact the PIs whose projects are expired and check if they can be deleted
• Remove the old project storage from DGX-front

• Incoming
• Waiting for copyright holder approval to share
• Preparation
• Waiting for recipient signature
• Waiting for countersignature
• Sharing
• Done

Deploy grand challenge using docker-compose. Link to the repository and follow the instructions until the deployment of the grand challenge.

Reference

• Wait for rebalancing and integration to finish.
• Grow storage.

SciLifeLab Roadmap for Precision Medicine Capability

AIDA Data in routine

• Contact the project lead
• Check if the dataset is appropriately anonymized and is legal for sharing.
• Prepare the legal agreements if needed
• Upload the data
• Create a public dataset register page

AIDA Data in routine

• Check if the data type is present in the data acquisition priorities
• Contact the project lead
• Check if the dataset is appropriately anonymized and is legal for sharing.
• Prepare the legal agreements if needed
• Upload the data
• Create a public dataset register page

Exact wording is: "MRI and histology of excised meningioma tumor samples", to be appended to the prio list

• Work with contributor and clinical council to get exact wording
• Append data type to prio list.

Work related to establishing CMIV research database 2.0, aka LiU Health tripod, aka LiU part of the LiU-RÖ double data lakes common vision.

Needed components of presentation:

• organogram simple
• projects
• CMIV
• breakdown of AIDA Data Hub components and services
• List of meetings, when and who (or in separate doc?)

• Spin up a VM in safespring
• Install docker-compose in the VM
• Deploy grand challenge inside the VM
• Run an AI algorithm against any data on platform using GC

• Read the DGX-2 Admin Docs to prepare for the discussion on friday

• Figure out training needs
• Get people an office space and equipment
• Get people accounts
• Compile self-study materials

Install cytomine using the helm chart. Link to the repository

AIDA Data in routine

• Contact the project lead
• Check if the dataset is appropriately anonymized and is legal for sharing.
• Prepare the legal agreements if needed
• Upload the data
• Create a public dataset register page

• Incoming
• Waiting for copyright holder approval to share
• Preparation
• Waiting for recipient signature
• Waiting for countersignature
• Sharing
• Done