idr / idr.openmicroscopy.org

Source for the IDR static website.

IDR website

This repository contains the source code for the IDR Website hosted at https://idr.openmicroscopy.org.

Versioning

This website has multiple version strings.

• The repository itself uses a CalVer scheme YYYY.0M.0D.
• The version displayed on the webpages should be the IDR deployment release, not the website release. This is independent of the website release and is not known until deployment time. The IDR version is set to devel in this repository, and should be overridden during deployment by creating a file VERSION containing the version string.

Contributing

Please read our contributing guidelines for ways to offer feedback and contribute.

License

All content is released under Creative Commmons Attribution 4.0.

idr.openmicroscopy.org's Issues

Using CH5 Files in Python

After @pwalczysko's great help with #158, I am able to use the Aspera download client to download well data for idr0013 in the form of a .ch5 file. From what I have read, the CH5(Cellh5) format is quite outdated and does not have much support. I have been able to open the files in ImageJ with the Bio Formats Plugin so the data is readable. However, when I try to install the CellH5 python library I have the same issue described in CellH5/cellh5#14. My attempts to load a CD5 file with python-bioformats result in the error:
OSError: [Errno 22] Could not load the file as an image (see log for details).

What is the best way to either:

1. Open the CH5 files in python and get image data for manipulation
2. Use the Aspera download client to download the idr0013 well data in a different format (e.g. tiff)

Thanks!

CC @gwaygenomics to keep you in the loop

ImJoy on ITR

I just noticed that you have the nice ITR page, and wondering if we can add
ImJoy (https://imjoy.io) to it.

We have already the BIII page: https://biii.eu/imjoy

For domains, I suggest "bio-imaging, medical-imaging, bioinformatics, deep
learning".

cc: @jburel @oeway

Update EuroBioimaging logo

The current EuroBioimaging logo is out of date. The current ones are on:
https://www.eurobioimaging.eu/about-us/for-press-supporting-material

Download of studies with screen files

The page IDR page https://idr.openmicroscopy.org/about/download.html gives an inaccurate workflow when a study such as idr0013 with .screen files download is attempted. See #158

List image magnification in IDR API

Thanks for your helpful pointers in #147, I have another API-related question :)

Is there a way for me to pull microscopy image magnification from the API?

I imagine this info would have been in the image annotations endpoint (see example below), but I am not readily seeing it. I also did not see this mentioned in the API docs. Maybe magnification exists in another API section?

Thanks!
Greg

MAP_URL = f"https://idr.openmicroscopy.org/webclient/api/annotations/?type=map&image={rand_image}"

annotations = session.get(MAP_URL).json()["annotations"]

print(annotations)

[{'id': 6625160,
  'ns': 'openmicroscopy.org/mapr/gene',
  'description': None,
  'owner': {'id': 2},
  'date': '2016-12-13T23:05:19+00:00',
  'permissions': {'canDelete': False,
   'canAnnotate': False,
   'canLink': False,
   'canEdit': False},
  'link': {'id': 22801231,
   'owner': {'id': 2},
   'parent': {'id': 14529,
    'class': 'ImageI',
    'name': 'DTT p1 [Well 77, Field 1 (Spot 229)]'},
   'date': '2016-12-13T23:05:19+00:00',
   'permissions': {'canDelete': False,
    'canAnnotate': False,
    'canLink': False,
    'canEdit': False}},
  'class': 'MapAnnotationI',
  'values': [['Gene Identifier', 'YNL106C'],
   ['Gene Identifier URL', 'https://www.yeastgenome.org/locus/YNL106C'],
   ['Gene Symbol', 'INP52']]},
 {'id': 6625787,
  'ns': 'openmicroscopy.org/mapr/organism',
  'description': None,
  'owner': {'id': 2},
  'date': '2016-12-13T23:06:04+00:00',
  'permissions': {'canDelete': False,
   'canAnnotate': False,
   'canLink': False,
   'canEdit': False},
  'link': {'id': 22834937,
   'owner': {'id': 2},
   'parent': {'id': 14529,
    'class': 'ImageI',
    'name': 'DTT p1 [Well 77, Field 1 (Spot 229)]'},
   'date': '2016-12-13T23:06:35+00:00',
   'permissions': {'canDelete': False,
    'canAnnotate': False,
    'canLink': False,
    'canEdit': False}},
  'class': 'MapAnnotationI',
  'values': [['Organism', 'Saccharomyces cerevisiae']]},
 {'id': 6626243,
  'ns': 'openmicroscopy.org/mapr/phenotype',
  'description': None,
  'owner': {'id': 2},
  'date': '2016-12-13T23:09:03+00:00',
  'permissions': {'canDelete': False,
   'canAnnotate': False,
   'canLink': False,
   'canEdit': False},
  'link': {'id': 22918669,
   'owner': {'id': 2},
   'parent': {'id': 14529,
    'class': 'ImageI',
    'name': 'DTT p1 [Well 77, Field 1 (Spot 229)]'},
   'date': '2016-12-13T23:09:04+00:00',
   'permissions': {'canDelete': False,
    'canAnnotate': False,
    'canLink': False,
    'canEdit': False}},
  'class': 'MapAnnotationI',
  'values': [['Phenotype', 'GFP localization: punctate'],
   ['Phenotype Term Name', 'protein localized in punctate foci phenotype'],
   ['Phenotype Term Accession', 'CMPO_0000400'],
   ['Phenotype Term Accession URL',
    'http://www.ebi.ac.uk/cmpo/CMPO_0000400']]},
 {'id': 6631107,
  'ns': 'openmicroscopy.org/omero/bulk_annotations',
  'description': None,
  'owner': {'id': 2},
  'date': '2016-12-13T23:14:34+00:00',
  'permissions': {'canDelete': False,
   'canAnnotate': False,
   'canLink': False,
   'canEdit': False},
  'link': {'id': 23067151,
   'owner': {'id': 2},
   'parent': {'id': 14529,
    'class': 'ImageI',
    'name': 'DTT p1 [Well 77, Field 1 (Spot 229)]'},
   'date': '2016-12-13T23:14:34+00:00',
   'permissions': {'canDelete': False,
    'canAnnotate': False,
    'canLink': False,
    'canEdit': False}},
  'class': 'MapAnnotationI',
  'values': [['Strain', 'Y6545'],
   ['Environmental Stress', 'dithiothreitol'],
   ['Channels',
    'H2B-mCherry:cytosol;GFP:tagged protein;bright field/transmitted:cell'],
   ['Has Phenotype', 'yes'],
   ['Phenotype Annotation Level', 'experimental condition and gene']]}]

what is the meaning of 5D images, dimension, timepoint for image details?

Hi,
Thanks for your great job, I am instesting in it. Now, I have a problem with the meaning of image details.

what does mean for Z-sections,Timepoints, Dimensions (XY)?

Best

Clarify help desk

"Do you know who to contact about issues with the IDR? They don't seem to have a help desk and their Twitter accounts don't accept DMs. I am trying to download a dataset but the AsperaConnect thingy they use keeps refusing my connection for only this dataset. "

I pointed to image.sc but we should clarify under idr.o.org (or make it more obvious)

Update release stats

After each release the stats have to be updated. Most figures can be acquired via omero fs usage and stats.py script.

Problem 1:

studies.tsv wants:
Study | Container | Introduced | Internal ID | Sets | Wells | Experiments (wells for screens, imaging experiments for non-screens) | Targets (genes, small molecules, geographic locations, or combination of factors (idr0019, 26, 34, 38) | Acquisitions | 5D Images | Planes | Size (TB) | Size | # of Files | avg. size (MB) | Avg. Image Dim (XYZCT)

From stats.py you'll get
Container | ID | Set | Wells | Images | Planes | Bytes
Example:
idr0052-walther-condensinmap/experimentA | 752 | 44 of 54 | 0 | 282 | 699360 | 85.4 GB
What does 44 of 54 sets mean? What is Bytes, does that have to be used for Size (TB) and Size?

omero fs usage give you something like
Total disk usage: 115773571855 bytes in 25 files . What about this size? And is the 25 files the # of Files?

The workflow doc has an hql query how to get the Avg. Image Dim (XYZCT), but only for projects not for screens.

And how to get Targets? As this can be multiple things, can't think of an easy/generic script which can go through any annotation.csv and pull the number of unique 'targets'.

Problem 2

releases.tsv wants:
Date | Data release | Code version | Sets | Wells | Experiments | Images | Planes | Size (TB) | Files (Million) | DB Size (GB)
From stats.py you'll get some of it:
Container | ID | Set | Wells | Images | Planes | Bytes
Total | | 13044 | 1213175 | 9150589 | 65571290 | 334.2 TB
But where to get Files (Million) from? And how to get DB Size (GB)?

/cc @sbesson wasn't really sure where to open the issue, here (stats) or idr-utils (stats.py script).

Expose notebooks

• #140

The points to discuss are about the notebooks sections.

• The First section is for the notebooks/apps (Shiny) specific to a study i.e. explore the results associated to the study and presented by the authors. This could be include in gallery to allow "search for notebooks/apps" similar to what we do for Authors, gene. and an icon could be added to the results found
• The second section is for "generic notebooks" i.e. using the various APIs like idr-py. This is not attached to any specific study or tools. It explores the data in IDR e.g. compounds, genetophenotypes
• The third section is for notebooks using a tools listed in ITR. Some of that could be a review of ITR. and/or could follow the same approach that the notebooks of the first section. In that case we will need to introduce annotation for the tool used to allow a search

Irregular Download Path Usage

I trying to download just the first plate for idr0013-screenA and am struggling to use the aspera download client to complete this partial download. The IDR download tutoral mentions to remove the leading /uod/idr/filesets/<idrNNN>-<author>-<description>/ when submitting a partial download command. However, the download paths for idr0013 do not follow the /uod/idr/filesets/<idrNNN>-<author>-<description>/ format.

For example, plate LT0001_02 lists its relative path as ../screens/LT0001_02--ex2005_11_16--sp2005_02_17--tt17--c3.screen. Attempting to run
ascp -TQ -l40m -P 33001 -i "path/to/asperaweb_id_dsa.openssh" [email protected]:screens/LT0001_02--ex2005_11_16--sp2005_02_17--tt17--c3.screen /tmp/data/ returns Session Stop (Error: Server aborted session: No such file or directory).

How should I use this relative path to complete a partial download?
Thanks!

Unable to Download Data with Aspera

Hello,

Thanks for making this repo and data public. I have been working in the Way Lab for the past year to develop a tool for streaming IDR image data downloading and processing (IDR_stream). This tool was originally developed in 2022, and thus uses Aspera high-speed transfer client to download data from IDR at high speeds.

I attempted to run the command below within IDR_stream to download a video:
sudo /home/roshankern/.aspera/ascli/sdk/ascp -TQ -l500m -P 33001 -i example_files/asperaweb_id_dsa.openssh [email protected]:20150916-mitocheck-analysis/mitocheck/LT0001_02--ex2005_11_16--sp2005_02_17--tt17--c3/hdf5/00049_01.ch5 ../tmp/downloads/LT0001_02

This command had worked for me in the past but this time I got the following output:
Session Stop (Error: Server aborted session: Permission denied)

The IDR download page and #189 both indicate that downloading IDR data with Aspera is no longer possible.

Thus, I have the following questions:

• Is it still possible to download IDR data with Aspera?
• If yes, what is the best way to download this data? Do I need to modify my command or redownload the Aspera public key (I couldn't find a new version online)?
• If no, will FTP or IDR API be significantly slower for downloading image data? Is there any way to approach Aspera-like speeds with these other image downloading methods?

Thanks in advance!
Roshan

Logos on ITR

From oeway:

One suggestion, for the page, I think would nice also to have a column to
have the logo of the tool.

List all screen IDs via API

I am playing with the IDR API (thanks for putting this out there by the way!), and I am struggling to find an endpoint that lists all study IDs.

Do you know of a strategy that I could use other than brute force?

Thanks!

Inconsistencies in JSON inidices

Hello,

Thanks for making this repo and data public. I’m working with the IDR API (https://idr.openmicroscopy.org/about/api.html) to pull image metadata, but I’m running into an issue.

Specifically, in the JSON API output, the grid object indices appear to be inconsistent in some studies. Where grid was obtained by using the following code:

WELLS_IMAGES_URL = f"https://idr.openmicroscopy.org/webgateway/plate/{plate}/"
plate_dict = requests.Session().get(WELLS_IMAGES_URL).json()
grid = plate_dict['grid']

Many of the metadata for plate JSON files begin at grid[0], however for plate 1753 (https://idr.openmicroscopy.org/webgateway/plate/1753) the metadata begin at grid[1]. See screenshot below:

Similarly, if the metadata are located in grid[0], sometimes the metadata won't appear until grid[0][1] instead of grid[0][0]. For example, see specifically https://idr.openmicroscopy.org/webgateway/plate/1760. See screenshot below:

Are plate metadata supposed to begin at grid[0][0]?

This is a potential issue that I can manage, but I’d like to report these observations in case these inconsistencies are errors and can be relatively easily standardized.

Thanks,
Parker