GithubHelp home page GithubHelp logo

brisvag / blik Goto Github PK

View Code? Open in Web Editor NEW
21.0 5.0 6.0 2.49 MB

Python tool for visualising and interacting with cryo-ET and subtomogram averaging data.

Home Page: https://brisvag.github.io/blik/

License: GNU General Public License v3.0

Python 100.00%
cryo-et visualization numpy python teamtomo napari

blik's Introduction

logo

blik

DOI License PyPI Python Version CI codecov

blik showcase

blik is a tool for visualising and interacting with cryo-ET and subtomogram averaging data. It leverages the fast, multi-dimensional napari viewer and the scientific python stack.

DISCLAIMER: this package is in development phase. Expect bugs and crashes. Please, report them on the issue tracker and ask if anything is unclear!

Installation

You can either install blik through the napari plugin system, through pip, or get both napari and blik directly with:

pip install "blik[all]"

The [all] qualifier also installs pyqt5 as the napari GUI backend, and a few additional napari plugins that you might find useful in your workflow:

Nightly build

If you'd like the most up to date blik possible, you can install directly from the main branch on github. This also uses napari main, so expect some instability!

pip install "git+https://github.com/brisvag/blik.git@main#egg=blik[all]"
pip install "git+https://github.com/napari/napari.git@main#egg=napari[all]"

Basic Usage

From the command line:

napari -w blik -- /path/to.star /path/to/mrc/files/*

The -w blik is important for proper initialization of all the layers. Always open the main widget open to ensure nothing goes wrong!

blik is just napari. Particles and images are exposed as simple napari layers, which can be analysed and manipulated with simple python, and most importantly other napari plugins.

Widgets

The main widget has a few functions:

  • experiment: quickly switch to a different experiment id (typically, everything related to an individual tomogram such as volume, particles and segmentations)
  • new: generate a new segmentation, a new manually-picked set of particles, or a new surface picking or filament picking for segmentation, particle generation or volume resampling.
  • add to exp: add a layer to the currently selected experiment (just a shorthand for layer.metadata['experiment_id'] = current_exp_id)
  • slice_thickness: changes the slicing thickness in all dimensions in napari. Images will be averaged over that thickness, and all particles in the slice will be displayed.

There are also widgets for picking of both surfaces and filaments.

  • surface: process a previously picked surface picking layer to generate a surface mesh and distribute particles on it for subtomogram averaging, or resample a tomogram along the surface.
  • filament: process a previously picked filament picking layer to generate a filament and distribute particles on it for subtomogram averaging, or resample a tomogram along the filament.

References

If you use blik, please cite the repo on zenodo and the paper on Plos Biology: https://doi.org/10.1371/journal.pbio.3002447.

blik's People

Contributors

alisterburt avatar brisvag avatar dependabot[bot] avatar haesleinhuepf avatar mchaillet avatar rdrighetto avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

blik's Issues

Import times are SLOW

Something about importing the package is very slow. Some profiling is needed, I'm sure there are improvements to be made, There's probably a lot of messy nested imports that could be prevented.

Base Classes

So far we've been calling eveything a DataBlock, despite the name change. I feel like it's better to just go back to DataBlock being the base classe, and calling the current DataBlock something like Uni/Single/Simple/MonoBlock.

issue with star file reading?

Hi Lorenzo!

I wanted to try Blik as I was curious if there were any nice napari based plugins for opening tomograms with STAR format particle annotations, but I ran into the issue below. Do you have any idea what I am doing wrong? I also pasted the header of my STAR file below (it is a homecooked file), perhaps its not the right format? I did try reading it via cryohub and that seems to work fine.

Command to open tomogram and particle list:

napari -w blik -- output/particles.star output/tomo_filt.mrc

Terminal output:

Traceback (most recent call last):
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/layers/points/points.py", line 806, in size
    self._size = np.broadcast_to(size, len(self.data)).copy()
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/numpy/lib/stride_tricks.py", line 413, in broadcast_to
    return _broadcast_to(array, shape, subok=subok, readonly=True)
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/numpy/lib/stride_tricks.py", line 349, in _broadcast_to
    it = np.nditer(
ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (3,)  and requested shape (144,)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/components/viewer_model.py", line 1202, in _open_or_raise_error
    added = self._add_layers_with_plugins(
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/components/viewer_model.py", line 1318, in _add_layers_with_plugins
    added.extend(self._add_layer_from_data(*_data))
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/components/viewer_model.py", line 1392, in _add_layer_from_data
    layer = add_method(data, **(meta or {}))
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/components/viewer_model.py", line 4, in add_points
    import itertools
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/layers/points/points.py", line 506, in __init__
    self.size = size
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/layers/points/points.py", line 814, in size
    raise ValueError(
ValueError: Size is not compatible for broadcasting

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/_qt/qt_viewer.py", line 953, in _qt_open
    self.viewer.open(
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/components/viewer_model.py", line 1102, in open
    layers = self._open_or_raise_error(
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/components/viewer_model.py", line 1211, in _open_or_raise_error
    raise ReaderPluginError(
napari.errors.reader_errors.ReaderPluginError: Tried opening with blik, but failed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/marten/miniconda3/envs/blik/bin/napari", line 8, in <module>
    sys.exit(main())
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/__main__.py", line 564, in main
    _run()
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/__main__.py", line 340, in _run
    viewer._window._qt_viewer._qt_open(
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/_qt/qt_viewer.py", line 961, in _qt_open
    handle_gui_reading(
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/_qt/dialogs/qt_reader_dialog.py", line 190, in handle_gui_reading
    readers = prepare_remaining_readers(paths, plugin_name, error)
  File "/home/marten/miniconda3/envs/blik/lib/python3.10/site-packages/napari/_qt/dialogs/qt_reader_dialog.py", line 245, in prepare_remaining_readers
    raise ReaderPluginError(
napari.errors.reader_errors.ReaderPluginError: Tried to read [output/particles.star, ...] with plugin blik, because it was associated with that file extension/because it is the only plugin capable of reading that path, but it gave an error. Try associating a different plugin or installing a different plugin for this kind of file.

My starfile looks like this:

# Created by the starfile Python package (version 0.4.12) at 20:17:22 on 25/07/2023

data_

loop_
_rlnCoordinateX #1
_rlnCoordinateY #2
_rlnCoordinateZ #3
_rlnAngleRot #4
_rlnAngleTilt #5
_rlnAnglePsi #6
_ptmLCCmax #7
_rlnDetectorPixelSize #8
_rlnMicrographName #9
_rlnMagnification #10
_rlnGroupNumber #11
181	163	86	-36.460765	-161.427020	160.142076	0.455640	13.790000	tomo200528_110.mrc	10000.000000	0
416	343	126	97.352181	-102.015107	38.385390	0.441689	13.790000	tomo200528_110.mrc	10000.000000	0
152	56	114	-159.478539	-104.188364	121.661090	0.424355	13.790000	tomo200528_110.mrc	10000.000000	0
449	180	104	-141.136705	-95.532863	-153.204503	0.422766	13.790000	tomo200528_110.mrc	10000.000000	0

Alchemist

Alchemist is a base class for turning DataBlocks into other DataBlocks

e.g.

ApplyTransformation(transformationblock=tb, particleblock=pb) -> transformed particles
DeriveMesh(pointblock) -> meshblock

Convert meshblock to a multiblock

Meshblock is currently broken. It should become a Multiblock with points for vertices, and probably a PropertyBlock for the faces.

Read dynamo `em` files

It would be nice to quickly look at these files in napari. I couldn't find the format specification anywhere, so we should probably ask Daniel.

'Alchemist' renaming proposal

I'm not convinced by the name Alchemist for something which takes blocks and makes new blocks... I don't think it's sufficiently descriptive, what're your thoughts?

I think I prefer something which conveys the operation/operator aspect, so something like BlockOperator is my first idea?

sub-peepers and their methods: abstract classes?

I think that with out current idea of sub-Peepers, there would be quite a bit of code duplication, or weird inheritance networks.

What do you think of having a few abstract classes with topic-specific methods, and then our peepers could inherit from any combination of those, while adding whatever connection is needed?

Suggestion: show tomograms in the same orientation as IMOD

(separate topic, separate issue)

Hi again Lorenzo,

If I may suggest, would it be possible to make blik display the tomogram in the same orientation as IMOD and ChimeraX, at least as an option?
Or if this option is already available somewhere in blik or napari, can you please let us know?

image
image
image

It seems there is a mirroring around the X axis going on, presumably because of how Python loads the data.
I know this is probably only a visualization thing, and I don't know if it would violate any internal napari conventions, but would be nice to be able to look at the data in the same way as other visualization tools commonly used in cryo-ET.

(FWIW, Amira does the same mirroring thing...)

Thanks a lot!

Images

A simple issue to upload images to, so we can add them to the repo without slowing down cloning times.

RefinementPeeper

Peeper class to view the results of a refinement

  • should stack into one 4d time series to avoid performance issues with many layers
  • should colour by class info if results are from a classification

Documentation

We should add some basic documentation. We shoul be able to use jupyterbook, since we have some experience with it.

Clean up non-visualization code

As we decided to restrict this repo to vosualization-related code only, we should remove a few things, mainly:

  • the analysis module
  • alchemists

MeshBlock

MeshBlock datablock for representing meshes (can be fished out from old ntp code which was based on the mesh data structures in pytorch3d)

Layer management for DataCrates

Currently, we have basic systems in place for generating multiple DataCrate objects from metadata files which act as containers for multiple objects which exist within the same 3d reference frame across a dataset e.g. tomogram, cc-volume, template matched positions, refinements

We need a system for easily switching between DataCrates from within the napari viewer

@brisvag you mentioned that this is currently made more difficult by a problem on the napari side, I couldn't find the issue, would you mind linking it when you get a sec?

out of slice fading of point and orientation layer

The fading (out of slice) is very useful when going through tomographic slices but is not happening at the same rate for the point and orientation layer. The point size first needs to be increased to size 5 to function as I would expect. @brisvag also mentioned in discussion he needed to fix something about sizes and lengths related to this ;)

@ldjager tagging you here as I thought you might want to follow!

Autoscaling issues

#69 Introduces scaling based on pixel size. Sometimes autoscaling just won't work, and having an interface to override it may be useful. Otherwise, we may want to add some flags to the io stuff to specify what data is being read.

Use case: in CC volumes from Warp, particle positions are scaled to [0, 1], and need 2 transformations to match the "original" image. In short:

positions *= image.shape[::-1]  # to undo the deformation
positions *= image.pixel_size

what's the best approach to solve problems like this?

Transformblock

Transformblock is currently broken by #69 , because it relied heavily on reshaping, which is both impossible on xarray, and against its strengths. We should rewrite it. While we are at it, I'll bring up a conversation we probably had already but haven't concluded as far as I remember:

I think Transformblock should be a simple ParticleBlock subclass that allows for a different depiction, but the transformation function should live in an Alchemist that goes from Particle + Transform to Particle, for example.

failed reading of star file

for the following file (truncated here)


# version 30001

data_optics

loop_ 
_rlnOpticsGroup #1 
_rlnOpticsGroupName #2 
_rlnSphericalAberration #3 
_rlnVoltage #4 
_rlnImagePixelSize #5 
_rlnImageSize #6 
_rlnImageDimensionality #7 
           1 opticsGroup1     1.733000   300.000000     3.300000          168            3 
           2 opticsGroup2     1.342000   300.000000     3.300000          168            3 
           3 opticsGroup3     4.727000   300.000000     3.300000          168            3 
           4 opticsGroup4     -0.71100   300.000000     3.300000          168            3 
           5 opticsGroup5     -0.20100   300.000000     3.300000          168            3 
           6 opticsGroup6     1.365000   300.000000     3.300000          168            3 
           7 opticsGroup7     1.940000   300.000000     3.300000          168            3 
           8 opticsGroup8     0.797000   300.000000     3.300000          168            3 
           9 opticsGroup9     2.208000   300.000000     3.300000          168            3 
          10 opticsGroup10     4.387000   300.000000     3.300000          168            3 
          11 opticsGroup11     1.373000   300.000000     3.300000          168            3 
          12 opticsGroup12     2.334000   300.000000     3.300000          168            3 
          13 opticsGroup13     2.712000   300.000000     3.300000          168            3 
          14 opticsGroup14     4.272000   300.000000     3.300000          168            3 
          15 opticsGroup15     3.564000   300.000000     3.300000          168            3 
          16 opticsGroup16     0.956000   300.000000     3.300000          168            3 
          17 opticsGroup17     -1.03200   300.000000     3.300000          168            3 
          18 opticsGroup18     4.942000   300.000000     3.300000          168            3 
          19 opticsGroup19     3.800000   300.000000     3.300000          168            3 
          20 opticsGroup20     1.674000   300.000000     3.300000          168            3 
          21 opticsGroup21     -1.39300   300.000000     3.300000          168            3 
          22 opticsGroup22     -0.61800   300.000000     3.300000          168            3 
          23 opticsGroup23     2.443000   300.000000     3.300000          168            3 
          24 opticsGroup24     1.801000   300.000000     3.300000          168            3 
          25 opticsGroup25     1.328000   300.000000     3.300000          168            3 
          26 opticsGroup26     1.145000   300.000000     3.300000          168            3 
          27 opticsGroup27     4.383000   300.000000     3.300000          168            3 
          28 opticsGroup28     6.647000   300.000000     3.300000          168            3 
          29 opticsGroup29     4.669000   300.000000     3.300000          168            3 
          30 opticsGroup30     1.476000   300.000000     3.300000          168            3 
          31 opticsGroup31     1.486000   300.000000     3.300000          168            3 
          32 opticsGroup32     1.284000   300.000000     3.300000          168            3 
          33 opticsGroup33     2.313000   300.000000     3.300000          168            3 
          34 opticsGroup34    12.665000   300.000000     3.300000          168            3 
          35 opticsGroup35     -5.61400   300.000000     3.300000          168            3 
          36 opticsGroup36    10.807000   300.000000     3.300000          168            3 
          37 opticsGroup37    12.419000   300.000000     3.300000          168            3 
          38 opticsGroup38    16.443000   300.000000     3.300000          168            3 
          39 opticsGroup39     -6.85400   300.000000     3.300000          168            3 
          40 opticsGroup40     2.700000   300.000000     3.300000          168            3 
          41 opticsGroup41     -2.67000   300.000000     3.300000          168            3 
          42 opticsGroup42    12.496000   300.000000     3.300000          168            3 
          43 opticsGroup43    -10.44100   300.000000     3.300000          168            3 
          44 opticsGroup44     -5.80600   300.000000     3.300000          168            3 
          45 opticsGroup45     5.766000   300.000000     3.300000          168            3 
          46 opticsGroup46    10.354000   300.000000     3.300000          168            3 
          47 opticsGroup47    21.054000   300.000000     3.300000          168            3 
          48 opticsGroup48     -2.27400   300.000000     3.300000          168            3 
          49 opticsGroup49     -5.74600   300.000000     3.300000          168            3 
          50 opticsGroup50     0.405000   300.000000     3.300000          168            3 
          51 opticsGroup51     2.600000   300.000000     3.300000          168            3 
          52 opticsGroup52     9.185000   300.000000     3.300000          168            3 
          53 opticsGroup53     -5.82000   300.000000     3.300000          168            3 
          54 opticsGroup54     -8.68600   300.000000     3.300000          168            3 
          55 opticsGroup55     8.268000   300.000000     3.300000          168            3 
          56 opticsGroup56     -3.18700   300.000000     3.300000          168            3 
          57 opticsGroup57     -2.43500   300.000000     3.300000          168            3 
          58 opticsGroup58     -6.36900   300.000000     3.300000          168            3 
          59 opticsGroup59     -5.69500   300.000000     3.300000          168            3 
          60 opticsGroup60     -4.77600   300.000000     3.300000          168            3 
          61 opticsGroup61     -1.88900   300.000000     3.300000          168            3 
          62 opticsGroup62    -19.60700   300.000000     3.300000          168            3 
          63 opticsGroup63     -6.17500   300.000000     3.300000          168            3 
          64 opticsGroup64     -5.70800   300.000000     3.300000          168            3 
          65 opticsGroup65     -4.86600   300.000000     3.300000          168            3 
          66 opticsGroup66     -3.33300   300.000000     3.300000          168            3 
          67 opticsGroup67     -3.39600   300.000000     3.300000          168            3 
          68 opticsGroup68     -6.18900   300.000000     3.300000          168            3 
          69 opticsGroup69     -6.47900   300.000000     3.300000          168            3 
          70 opticsGroup70     2.955000   300.000000     3.300000          168            3 
          71 opticsGroup71     -8.08200   300.000000     3.300000          168            3 
          72 opticsGroup72     3.985000   300.000000     3.300000          168            3 
          73 opticsGroup73     -6.77400   300.000000     3.300000          168            3 
          74 opticsGroup74    -10.19300   300.000000     3.300000          168            3 
          75 opticsGroup75     -2.01700   300.000000     3.300000          168            3 
          76 opticsGroup76    -11.01700   300.000000     3.300000          168            3 
          77 opticsGroup77     -1.76700   300.000000     3.300000          168            3 
          78 opticsGroup78     -7.98600   300.000000     3.300000          168            3 
          79 opticsGroup79    11.184000   300.000000     3.300000          168            3 
          80 opticsGroup80     -7.52300   300.000000     3.300000          168            3 
          81 opticsGroup81     -3.18100   300.000000     3.300000          168            3 
          82 opticsGroup82     0.817000   300.000000     3.300000          168            3 
          83 opticsGroup83     9.663000   300.000000     3.300000          168            3 
          84 opticsGroup84     -9.84800   300.000000     3.300000          168            3 
          85 opticsGroup85     -6.80100   300.000000     3.300000          168            3 
          86 opticsGroup86     -7.98400   300.000000     3.300000          168            3 
          87 opticsGroup87     -6.08600   300.000000     3.300000          168            3 
          88 opticsGroup88     -4.23700   300.000000     3.300000          168            3 
          89 opticsGroup89     -2.04400   300.000000     3.300000          168            3 
          90 opticsGroup90     -9.28300   300.000000     3.300000          168            3 
          91 opticsGroup91    -13.48900   300.000000     3.300000          168            3 
          92 opticsGroup92    -10.81400   300.000000     3.300000          168            3 
          93 opticsGroup93    12.717000   300.000000     3.300000          168            3 
          94 opticsGroup94     -1.35000   300.000000     3.300000          168            3 
          95 opticsGroup95     -7.84600   300.000000     3.300000          168            3 
          96 opticsGroup96     -7.04200   300.000000     3.300000          168            3 
          97 opticsGroup97    -13.21100   300.000000     3.300000          168            3 
          98 opticsGroup98    18.928000   300.000000     3.300000          168            3 
          99 opticsGroup99     -8.41000   300.000000     3.300000          168            3 
         100 opticsGroup100     -0.28500   300.000000     3.300000          168            3 
         101 opticsGroup101     -0.96000   300.000000     3.300000          168            3 
         102 opticsGroup102     -6.09200   300.000000     3.300000          168            3 
         103 opticsGroup103     -4.89400   300.000000     3.300000          168            3 
         104 opticsGroup104     7.983000   300.000000     3.300000          168            3 
         105 opticsGroup105    -11.26500   300.000000     3.300000          168            3 
         106 opticsGroup106     -5.41400   300.000000     3.300000          168            3 
         107 opticsGroup107     -4.53200   300.000000     3.300000          168            3 
         108 opticsGroup108     2.998000   300.000000     3.300000          168            3 
         109 opticsGroup109     -5.64400   300.000000     3.300000          168            3 
         110 opticsGroup110     -3.21300   300.000000     3.300000          168            3 
         111 opticsGroup111     5.510000   300.000000     3.300000          168            3 
         112 opticsGroup112     -7.84200   300.000000     3.300000          168            3 
         113 opticsGroup113     -9.12300   300.000000     3.300000          168            3 
         114 opticsGroup114     -6.93300   300.000000     3.300000          168            3 
         115 opticsGroup115     -0.18500   300.000000     3.300000          168            3 
         116 opticsGroup116    -10.37800   300.000000     3.300000          168            3 
         117 opticsGroup117     -8.54200   300.000000     3.300000          168            3 
         118 opticsGroup118     -3.12200   300.000000     3.300000          168            3 
         119 opticsGroup119     -2.92600   300.000000     3.300000          168            3 
         120 opticsGroup120     8.847000   300.000000     3.300000          168            3 
         121 opticsGroup121     -9.95900   300.000000     3.300000          168            3 
         122 opticsGroup122     -5.43400   300.000000     3.300000          168            3 
         123 opticsGroup123    10.950000   300.000000     3.300000          168            3 
         124 opticsGroup124    15.787000   300.000000     3.300000          168            3 
         125 opticsGroup125     -2.95300   300.000000     3.300000          168            3 
         126 opticsGroup126     -7.44100   300.000000     3.300000          168            3 
         127 opticsGroup127     1.529000   300.000000     3.300000          168            3 
         128 opticsGroup128     -0.75400   300.000000     3.300000          168            3 
         129 opticsGroup129     1.353000   300.000000     3.300000          168            3 
         130 opticsGroup130     -0.99700   300.000000     3.300000          168            3 
         131 opticsGroup131     0.168000   300.000000     3.300000          168            3 
         132 opticsGroup132     0.539000   300.000000     3.300000          168            3 
         133 opticsGroup133     8.163000   300.000000     3.300000          168            3 
         134 opticsGroup134     -1.10700   300.000000     3.300000          168            3 
         135 opticsGroup135     -0.74500   300.000000     3.300000          168            3 
         136 opticsGroup136     -4.76600   300.000000     3.300000          168            3 
         137 opticsGroup137    -16.87100   300.000000     3.300000          168            3 
         138 opticsGroup138     -2.16600   300.000000     3.300000          168            3 
         139 opticsGroup139     -8.19300   300.000000     3.300000          168            3 
         140 opticsGroup140    15.278000   300.000000     3.300000          168            3 
         141 opticsGroup141    13.201000   300.000000     3.300000          168            3 
         142 opticsGroup142     -3.75200   300.000000     3.300000          168            3 
         143 opticsGroup143     -3.96700   300.000000     3.300000          168            3 
         144 opticsGroup144     2.234000   300.000000     3.300000          168            3 
         145 opticsGroup145     1.923000   300.000000     3.300000          168            3 
         146 opticsGroup146     2.425000   300.000000     3.300000          168            3 
         147 opticsGroup147     1.567000   300.000000     3.300000          168            3 
         148 opticsGroup148     2.173000   300.000000     3.300000          168            3 
         149 opticsGroup149     2.639000   300.000000     3.300000          168            3 
         150 opticsGroup150     2.536000   300.000000     3.300000          168            3 
         151 opticsGroup151     1.949000   300.000000     3.300000          168            3 
         152 opticsGroup152     2.129000   300.000000     3.300000          168            3 
         153 opticsGroup153     3.710000   300.000000     3.300000          168            3 
         154 opticsGroup154     3.353000   300.000000     3.300000          168            3 
         155 opticsGroup155     3.274000   300.000000     3.300000          168            3 
         156 opticsGroup156     2.448000   300.000000     3.300000          168            3 
         157 opticsGroup157     1.160000   300.000000     3.300000          168            3 
         158 opticsGroup158     2.337000   300.000000     3.300000          168            3 
         159 opticsGroup159     2.067000   300.000000     3.300000          168            3 
         160 opticsGroup160     3.200000   300.000000     3.300000          168            3 
         161 opticsGroup161     1.977000   300.000000     3.300000          168            3 
         162 opticsGroup162     3.002000   300.000000     3.300000          168            3 
         163 opticsGroup163     2.435000   300.000000     3.300000          168            3 
         164 opticsGroup164     3.486000   300.000000     3.300000          168            3 
         165 opticsGroup165     2.795000   300.000000     3.300000          168            3 
         166 opticsGroup166     2.396000   300.000000     3.300000          168            3 
         167 opticsGroup167     1.070000   300.000000     3.300000          168            3 
         168 opticsGroup168     2.761000   300.000000     3.300000          168            3 
         169 opticsGroup169     3.698000   300.000000     3.300000          168            3 
         170 opticsGroup170     3.211000   300.000000     3.300000          168            3 
         171 opticsGroup171     3.210000   300.000000     3.300000          168            3 
         172 opticsGroup172     2.857000   300.000000     3.300000          168            3 
         173 opticsGroup173     2.462000   300.000000     3.300000          168            3 
         174 opticsGroup174     1.340000   300.000000     3.300000          168            3 
         175 opticsGroup175     3.417000   300.000000     3.300000          168            3 
         176 opticsGroup176     3.826000   300.000000     3.300000          168            3 
         177 opticsGroup177     1.357000   300.000000     3.300000          168            3 
         178 opticsGroup178     3.101000   300.000000     3.300000          168            3 
         179 opticsGroup179     3.288000   300.000000     3.300000          168            3 
         180 opticsGroup180     2.048000   300.000000     3.300000          168            3 
         181 opticsGroup181     3.676000   300.000000     3.300000          168            3 
         182 opticsGroup182     3.153000   300.000000     3.300000          168            3 
         183 opticsGroup183     0.370000   300.000000     3.300000          168            3 
         184 opticsGroup184     5.088000   300.000000     3.300000          168            3 
         185 opticsGroup185     0.895000   300.000000     3.300000          168            3 
         186 opticsGroup186     2.610000   300.000000     3.300000          168            3 
         187 opticsGroup187     3.714000   300.000000     3.300000          168            3 
         188 opticsGroup188     2.331000   300.000000     3.300000          168            3 
         189 opticsGroup189     3.156000   300.000000     3.300000          168            3 
         190 opticsGroup190     2.799000   300.000000     3.300000          168            3 
         191 opticsGroup191     2.433000   300.000000     3.300000          168            3 
         192 opticsGroup192     2.350000   300.000000     3.300000          168            3 
         193 opticsGroup193     3.083000   300.000000     3.300000          168            3 
         194 opticsGroup194     2.703000   300.000000     3.300000          168            3 
         195 opticsGroup195     5.394000   300.000000     3.300000          168            3 
         196 opticsGroup196     2.343000   300.000000     3.300000          168            3 
         197 opticsGroup197     2.168000   300.000000     3.300000          168            3 
         198 opticsGroup198     3.367000   300.000000     3.300000          168            3 
         199 opticsGroup199     2.311000   300.000000     3.300000          168            3 
         200 opticsGroup200     1.341000   300.000000     3.300000          168            3 
         201 opticsGroup201     2.756000   300.000000     3.300000          168            3 
         202 opticsGroup202     4.371000   300.000000     3.300000          168            3 
         203 opticsGroup203     4.294000   300.000000     3.300000          168            3 
         204 opticsGroup204     3.125000   300.000000     3.300000          168            3 
         205 opticsGroup205     3.881000   300.000000     3.300000          168            3 
         206 opticsGroup206     4.179000   300.000000     3.300000          168            3 
         207 opticsGroup207     2.676000   300.000000     3.300000          168            3 
         208 opticsGroup208     4.577000   300.000000     3.300000          168            3 
         209 opticsGroup209     2.384000   300.000000     3.300000          168            3 
         210 opticsGroup210     1.509000   300.000000     3.300000          168            3 
         211 opticsGroup211     2.582000   300.000000     3.300000          168            3 
         212 opticsGroup212     2.307000   300.000000     3.300000          168            3 
 

# version 30001

data_particles

loop_ 
_rlnCoordinateX #1 
_rlnCoordinateY #2 
_rlnCoordinateZ #3 
_rlnAngleRot #4 
_rlnAngleTilt #5 
_rlnAnglePsi #6 
_rlnImageName #7 
_rlnCtfImage #8 
_rlnRandomSubset #9 
_rlnPixelSize #10 
_rlnMicrographName #11 
_rlnOpticsGroup #12 
_rlnGroupNumber #13 
_rlnOriginXAngst #14 
_rlnOriginYAngst #15 
_rlnOriginZAngst #16 
_rlnClassNumber #17 
_rlnNormCorrection #18 
_rlnLogLikeliContribution #19 
_rlnMaxValueProbDistribution #20 
_rlnNrOfSignificantSamples #21 
  836.294700   862.086200   900.246400   142.071592   103.564906    69.139249 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000000_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000000_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     -1.94698     0.759021     0.165021            1     1.000000 1.859544e+07     0.347848            1 
  804.912400   965.178500   670.322700    -65.15434    74.051902    24.788571 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000001_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000001_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     0.165021     1.353021     -1.02298            1     1.000000 1.858428e+07     0.317947            6 
  752.829500   868.314300   739.809900    57.524627   101.513572    29.101890 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000002_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000002_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     0.165021     -0.42898     1.947021            1     1.000000 1.858530e+07     0.769978            5 
  879.070100   895.073900  1102.871000   131.889120   106.369129    45.962248 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000003_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000003_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     0.165021     -0.42898     -1.02298            1     1.000000 1.858327e+07     0.417911           12 
  891.395000   987.073400   743.099200   177.201641    96.547760    64.345483 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000004_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000004_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     -0.42898     0.165021     1.947021            1     1.000000 1.858951e+07     0.438819            2 
 1103.266000  1079.348000   578.694000    -45.82911    74.412707    38.652809 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000005_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000005_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     -0.42898     2.277021     -1.02298            1     1.000000 1.860413e+07     0.105477            6 
 1025.856000  1020.370000   753.358300   -128.27792    78.319160    56.221480 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000006_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000006_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     -0.42898     0.165021     0.759021            1     1.000000 1.859226e+07     0.406918            2 
  845.024800   882.678100   993.333100    68.964732   105.006347    42.480576 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000007_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000007_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     0.759021     1.353021     -0.42898            1     1.000000 1.858975e+07     0.494501            7 
 1210.012000  1134.229000   782.332500   -167.32558    82.490846    84.452430 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000008_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000008_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     0.165021     1.683021     0.165021            1     1.000000 1.857891e+07     0.472795            3 
 1092.010000  1069.259000  1025.639000   -160.29667    87.290816    76.462462 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000009_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000009_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     -1.02298     0.165021     0.165021            1     1.000000 1.859281e+07     0.154063           15 
 1194.654000  1127.082000  1070.406000    40.283766    98.223473    41.720168 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000010_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000010_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     -0.42898     1.353021     0.759021            1     1.000000 1.858563e+07     0.508501           10 
 1240.532000  1062.131000   587.446700    32.531176    89.258724    38.183493 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000011_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000011_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     0.759021     0.759021     0.165021            1     1.000000 1.857254e+07     0.286359            2 
 1066.733000  1028.368000   868.432300    20.236184    97.652883    26.430236 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000012_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000012_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     0.759021     1.353021     0.165021            1     1.000000 1.857658e+07     0.433513            1 
  828.343900   809.924200  1089.123000     9.857874    92.119779    11.866594 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000013_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000013_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     -0.42898     -0.42898     0.165021            1     1.000000 1.859472e+07     0.656570            2 
  935.039100   923.952800  1030.524000     2.826499    96.611524    34.813151 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000014_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000014_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     0.759021     -0.42898     -1.02298            1     1.000000 1.858572e+07     0.379623            3 
 1010.528000  1001.212000  1009.573000   172.983482    94.150286    71.155722 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000015_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000015_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     -1.02298     -0.42898     -1.02298            1     1.000000 1.859664e+07     0.682908            1 
 1267.484000  1167.001000   718.755600    32.894060    95.832586    71.700566 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000016_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000016_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     0.165021     1.353021     1.947021            1     1.000000 1.858123e+07     0.259793            8 
 1076.106000  1083.085000   694.161800   134.687941   101.439659    65.171771 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000017_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000017_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     0.165021     0.759021     -2.21098            1     1.000000 1.859661e+07     0.270216            9 
  971.170000   959.461900  1117.676000    86.385962   109.483393    43.903872 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000018_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000018_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     -1.35298     -1.94698     -1.02298            1     1.000000 1.860108e+07     0.204165            2 
 1219.039000  1151.760000   978.116700   109.569066   104.417065    66.857291 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000019_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000019_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     -0.42898     0.165021     0.759021            1     1.000000 1.860157e+07     0.323060            6 
 1301.657000  1146.548000   812.080400     -3.67781    87.495162    98.776776 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000020_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000020_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     0.165021     0.165021     -2.21098            1     1.000000 1.859517e+07     0.612479            8 
  951.537900   962.291100   631.813100    10.212995    98.414262    12.102395 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000021_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000021_ctf_3.30A.mrc            1     3.300000 TS.mrc.tomostar          194            1     2.277021     -0.42898     -1.02298            1     1.000000 1.858420e+07     0.268300            6 
 1035.807000  1045.579000  1096.881000   -133.55361    78.306521    70.529614 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000022_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000022_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     -0.42898     -0.42898     0.759021            1     1.000000 1.860067e+07     0.241227            2 
 1176.110000  1137.728000   880.803600   -147.27627    81.008040    77.127323 subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000023_3.30A.mrc subtomo_05Feb21/TS.mrc/TS.mrc_YTURC_0000023_ctf_3.30A.mrc            2     3.300000 TS.mrc.tomostar          194            1     -1.02298     1.353021     0.165021            1     1.000000 1.857888e+07     0.247428            8 
 1189.390000  1360.073000   476.312900   -139.76938   107.130491   118.119352 subtomo_05Feb21/TS10.mrc/TS10.mrc_YTURC_0000000_3.30A.mrc subtomo_05Feb21/TS10.mrc/TS10.mrc_YTURC_0000000_ctf_3.30A.mrc            2     3.300000 TS10.mrc.tomostar          186            2     1.947021     7.359021     -1.02298            1     1.000000 1.858866e+07     0.161733           12 
 1050.070000  1096.027000   567.046900    14.956454    92.564289     5.319966 subtomo_05Feb21/TS10.mrc/TS10.mrc_YTURC_0000001_3.30A.mrc subtomo_05Feb21/TS10.mrc/TS10.mrc_YTURC_0000001_ctf_3.30A.mrc            2     3.300000 TS10.mrc.tomostar          186            2

I get the following error in blik

(py) ➜  gamma-turc-subboxing blik particle_star_file.star -p 3.3
Traceback (most recent call last):
  File "/Users/aburt/mambaforge/envs/py/bin/blik", line 8, in <module>
    sys.exit(cli())
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/blik/__main__.py", line 62, in cli
    dataset = blik.read(*paths,  # noqa: F841
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/blik/utils/generic.py", line 36, in wrapper
    return func(*args, **kwargs)
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/blik/io_/reading/main.py", line 182, in read
    dataset = read_files(*globs, **kwargs)
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/blik/io_/reading/main.py", line 125, in read_files
    datablocks.extend(read_file(file, name_regex=name_regex, pixel_size=pixel_size,
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/blik/io_/reading/main.py", line 46, in read_file
    datablocks = listify(func(file_path, **kwargs))
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/blik/io_/reading/star.py", line 126, in read_star
    particle_blocks = reader_function(raw_data, star_path=star_path, **kwargs)
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/blik/io_/reading/star.py", line 105, in parse_relion31
    return extract_data(df, mode='RELION 3.1', **kwargs)
  File "/Users/aburt/mambaforge/envs/py/lib/python3.9/site-packages/blik/io_/reading/star.py", line 48, in extract_data
    shifts = shifts / px_size
TypeError: ufunc 'true_divide' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''```

will look into it :)

Broken vector depiction.

Stacking particles seems a bit clumsy at the moment, and depiction of vectors when stacked is just wrong.

Examples, preferably with data

ideally we want examples for the following

  • Particles on tomogram
  • Refinement results viewing
  • Template matching results viewing
  • Transformation block applied to chemoreceptor data (hexagon -> c2 axes)

3D interactivity

We need to have a working system for 3d interactivity/annotation eventually, ideally on 2d slices visualised in 3D so we can be precise and remove ambiguity

This is an ideal candidate for something that we should develop directly within the napari ecosystem and then use ourselves

colouring points by data attribute

I think it could be a useful feature to give the points layer a color gradient depending on a data attribute. For example, my template matching package writes out the correlation score (LCCmax) and it would be very useful to colour the points by this data attribute. Or in 3D viewing it could be useful to colour points by the z-coordinate.

From the Napari docs, it seems something like that is possible. Do you think this would be dificult to implement?

TransformBlock

TransformBlock should be a block containing points and orientations (much like a ParticleBlock).

These points and orientations are relative to a point at the origin and should describe the relative positions and orientations of subparticles compared to oriented particles in a ParticleBlock

This will eventually be used to mirror the functionality of dpktbl.subbox.tableOnTable from dynamo for applying - equvalent to the table which is applied on an existing table in that implementation

Rescaling data

So far, we haven't paid any attention to the need to rescale data

A world coordinates system has been implemented in napari/napari#1539 which we can definitely take advantage of but we also need to check about rescaling etc

A safe bet is to use angstroms as our physical units everywhere and, if possible, pixel sizes should be inferred from the data or optionally provided by the user

The circle of life of a DataSet

Our data model went through some stuff, but it feels like it's reaching a state where it could stay for a while. I'm happy with the level of separation between data and depiction in the low-level stuff, and I'm happy with how the high-level DataSet object provides an interface to all of it, without borthering too much with keeping thinkgs separated.

In light of this, I propose to close the circle and go back to the name Peeper, instead of DataSet :)

Minor issue: expected MRC file extension

Hi Lorenzo,

Thanks for releasing blik and the preprint!

Minor issue I noticed, is that it expects files to have the .mrc extension, which is not always the case. Say, we have .rec files from IMOD which is just an MRC file with a different extension. In this case I just trick blik by renaming or linking the file with the .mrc extension and it works fine, but it's kinda annoying.

So, could you make blik pretend that everything is an MRC file no matter how they are called?

Thank you!

RELION shift handling may be incorrect

RELION shifts are currently applied as rlnCoordinateXYZ + rlnOriginXYZ, it's possible that they should be applied as rlnCoordinateXYZ - rlnOriginXYZ, will confirm when I know more

Use dask

The first time we introduced lazy loading we tried with dask.delayed, but ultimately were trying to manually load the images when needed. Napari has pretty good handling of dask delayed now, and we should be able to simpoly load all the images as delayed and let napari handle everything!

Even better, this lets us use do cool thing like load trhough ssh :D

Refactoring `_data_setter` & Co.

We should get rid of the assumption that all datablocks want to use a data attribute, and that they should all use _data_setter for that. It's either resctrictive or useless for how groublocks or simply more complex datablocks work.

I think all datablock where it's obvious and intuitive should have easy access through __setitem__ like now, but without forcing an internal api for how to do that. As for more complex stuff: we should probably implement a "smart" setter/getter; maybe:

Particles[2]

should return something like

[position, orientation, property]

Failed to load mrc

Hi!

Warning: I have a complete lack of understanding of how mrc files are read/written. I also understand that the plugin is in development!

When trying to open an mrc file from a collaborator, I get the following error:

ValueError                                Traceback (most recent call last)
File src/psygnal/_signal.py:912, in psygnal._signal.SignalInstance._run_emit_loop()

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/magicgui/widgets/_function_gui.py:207, in FunctionGui.__init__.<locals>._disable_button_and_call()
    206 try:
--> 207     self.__call__()
        self = <FunctionGui file_reader(files: List[pathlib.Path] = (PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), name_regex: List[str] = [], names: List[str] = [], as_dask_array: bool = True) -> <function NewType.<locals>.new_type at 0x7f376cfd7f70>>
    208 finally:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/magicgui/widgets/_function_gui.py:318, in FunctionGui.__call__(self=<FunctionGui file_reader(files: List[pathlib.Pat... = True) -> <function NewType.<locals>.new_type>>, update_widget=False, *args=(), **kwargs={})
    317 with _function_name_pointing_to_widget(self):
--> 318     value = self._function(*bound.args, **bound.kwargs)
        self = <FunctionGui file_reader(files: List[pathlib.Path] = (PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), name_regex: List[str] = [], names: List[str] = [], as_dask_array: bool = True) -> <function NewType.<locals>.new_type at 0x7f376cfd7f70>>
        bound = <BoundArguments (files=(PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), name_regex=[], names=[], as_dask_array=True)>
        self._function = <function file_reader at 0x7f36fcc02670>
    320 self._call_count += 1

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/blik/widgets/file_reader.py:29, in file_reader(files=(PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), name_regex=[], names=[], as_dask_array=True)
     23 """
     24 Read files with blik.
     25 
     26 name_regex: a regex string. Matching text will be used as name for the piece of data
     27 names: only load data matching this comma separated list of names
     28 """
---> 29 return read_layers(*files, name_regex=name_regex or None, names=names or None, lazy=as_dask_array)
        files = (PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),)
        name_regex or None = None
        names or None = None
        as_dask_array = True
        name_regex = []
        names = []

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/blik/reader.py:73, in read_layers(*paths=(PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), **kwargs={'lazy': True, 'name_regex': None, 'names': None})
     72 def read_layers(*paths, **kwargs):
---> 73     data_list = read(*paths, **kwargs)
        paths = (PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),)
        kwargs = {'name_regex': None, 'names': None, 'lazy': True}
     74     layers = []

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/naaf/reading/main.py:76, in read(name_regex=None, names=None, strict=False, lazy=True, *paths=(PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), **kwargs={})
     75 try:
---> 76     data.extend(read_file(file, name_regex=name_regex, lazy=lazy, **kwargs))
        data = []
        file = PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc')
        name_regex = None
        lazy = True
        kwargs = {}
     77 except ParseError:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/naaf/reading/main.py:35, in read_file(file_path=PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'), **kwargs={'lazy': True, 'name_regex': None})
     34 try:
---> 35     data = listify(func(file_path, **kwargs))
        func = <function read_mrc at 0x7f36fc695790>
        file_path = PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc')
        kwargs = {'name_regex': None, 'lazy': True}
     36     for d in data:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/naaf/reading/mrc.py:16, in read_mrc(image_path=PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'), name_regex=None, lazy=True, **kwargs={})
     14 name = guess_name(image_path, name_regex)
---> 16 with mrcfile.mmap(image_path) as mrc:
        image_path = PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc')
     17     if lazy:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/load_functions.py:208, in mmap(name=PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'), mode='r', permissive=False)
    185 """Open a memory-mapped MRC file.
    186 
    187 This allows much faster opening of large files, because the data is only
   (...)
    206     An :class:`~mrcfile.mrcmemmap.MrcMemmap` object.
    207 """
--> 208 return MrcMemmap(name, mode=mode, permissive=permissive)
        name = PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc')
        mode = 'r'
        permissive = False

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/mrcfile.py:114, in MrcFile.__init__(self=MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r'), name=PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'), mode='r', overwrite=False, permissive=False, header_only=False, **kwargs={})
    113     else:
--> 114         self._read(header_only)
        self = MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r')
        header_only = False
    115 except Exception:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/mrcfile.py:130, in MrcFile._read(self=MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r'), header_only=False)
    129 self._iostream.seek(0)
--> 130 super(MrcFile, self)._read(header_only)
        header_only = False
        self = MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r')
    132 # Check if the file is the expected size.

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/mrcinterpreter.py:173, in MrcInterpreter._read(self=MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r'), header_only=False)
    154 """Read the header, extended header and data from the I/O stream.
    155 
    156 Before calling this method, the stream should be open and positioned at
   (...)
    171          as a valid MRC file and ``permissive`` is :data:`True`.
    172 """
--> 173 self._read_header()
        self = MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r')
    174 self._read_extended_header()

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/mrcinterpreter.py:215, in MrcInterpreter._read_header(self=MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r'))
    214 try:
--> 215     byte_order = utils.byte_order_from_machine_stamp(header.machst)
        header = rec.array((1024, 1024, 512, 2, 0, 0, 0, 1024, 1024, 512, (1024., 1024., 512.), (90., 90., 90.), 1, 2, 3, -995.9688, 480.2282, -2.2351742e-07, 0, 0, b'\x00\x00\x00\x00\x00\x00\x00\x00', b'', 0, b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', (0., 0., 0.), b'MAP ', [68, 32, 32, 32], -1., 0, [b'', b'', b'', b'', b'', b'', b'', b'', b'', b'']),
          dtype=[('nx', '<i4'), ('ny', '<i4'), ('nz', '<i4'), ('mode', '<i4'), ('nxstart', '<i4'), ('nystart', '<i4'), ('nzstart', '<i4'), ('mx', '<i4'), ('my', '<i4'), ('mz', '<i4'), ('cella', [('x', '<f4'), ('y', '<f4'), ('z', '<f4')]), ('cellb', [('alpha', '<f4'), ('beta', '<f4'), ('gamma', '<f4')]), ('mapc', '<i4'), ('mapr', '<i4'), ('maps', '<i4'), ('dmin', '<f4'), ('dmax', '<f4'), ('dmean', '<f4'), ('ispg', '<i4'), ('nsymbt', '<i4'), ('extra1', 'V8'), ('exttyp', 'S4'), ('nversion', '<i4'), ('extra2', 'V84'), ('origin', [('x', '<f4'), ('y', '<f4'), ('z', '<f4')]), ('map', 'S4'), ('machst', 'u1', (4,)), ('rms', '<f4'), ('nlabl', '<i4'), ('label', 'S80', (10,))])
        utils = <module 'mrcfile.utils' from '/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/utils.py'>
    216 except ValueError as err:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/utils.py:199, in byte_order_from_machine_stamp(machst=<class 'numpy.ndarray'> (4,) uint8)
    198 pretty_bytes = pretty_machine_stamp(machst)
--> 199 raise ValueError("Unrecognised machine stamp: " + pretty_bytes)
        pretty_bytes = '0x44 0x20 0x20 0x20'
        "Unrecognised machine stamp: " + pretty_bytes = 'Unrecognised machine stamp: 0x44 0x20 0x20 0x20'

ValueError: Unrecognised machine stamp: 0x44 0x20 0x20 0x20

The above exception was the direct cause of the following exception:

EmitLoopError                             Traceback (most recent call last)
File ~/miniconda3/envs/blik/lib/python3.9/site-packages/magicgui/widgets/_bases/value_widget.py:57, in ValueWidget._on_value_change(self=PushButton(value=False, annotation=None, name='call_button'), value=False)
     55 if value is self.null_value and not self._nullable:
     56     return
---> 57 self.changed.emit(value)
        value = False
        self.changed = <SignalInstance 'changed' on PushButton(value=False, annotation=None, name='call_button')>
        self = PushButton(value=False, annotation=None, name='call_button')

File src/psygnal/_signal.py:849, in psygnal._signal.SignalInstance.emit()

File src/psygnal/_signal.py:891, in psygnal._signal.SignalInstance._run_emit_loop()

File src/psygnal/_signal.py:892, in psygnal._signal.SignalInstance._run_emit_loop()

File src/psygnal/_signal.py:914, in psygnal._signal.SignalInstance._run_emit_loop()

EmitLoopError: calling <function FunctionGui.__init__.<locals>._disable_button_and_call at 0x7f36fc5c7790> with args=() caused ValueError in emit loop.
^CTraceback (most recent call last):
  File "/home/joran.deschamps/miniconda3/envs/blik/bin/napari", line 8, in <module>
    sys.exit(main())
  File "/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/site-packages/napari/__main__.py", line 447, in main
    _run()
  File "/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/site-packages/napari/__main__.py", line 336, in _run
    run(gui_exceptions=True)
  File "/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/site-packages/napari/_qt/qt_event_loop.py", line 402, in run
    app.exec_()
  File "/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/contextlib.py", line 126, in __exit__
    next(self.gen)
  File "/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/site-packages/napari/_qt/utils.py", line 459, in _maybe_allow_interrupt
    old_sigint_handler(*handler_args)

I installed the env following #130, with naaf==0.2.3, which may or may not explain the problem.

Warning raised by dipoleblock

Pytest raised the following warning.

peepingtom/core/datablocks/dipoleblock.py:39
  peepingtom/peepingtom/core/datablocks/dipoleblock.py:39: RuntimeWarning: invalid value encountered in true_divide
    return self.orientation_vectors / norm

-- Docs: https://docs.pytest.org/en/stable/warnings.html

read starfile puts everything in properties

I don't think we should dump all the columns of a starfile into the properties of a particleblock.
At the very least, we should remove positions, euler angles and other already used columns!

Dependency issues

Hi!

A collaborator tried to use blik and had some issues. I could reproduce the errors with their data. I encountered problems with some of the dependencies, both when starting napari and then when starting the reader plugin.

  1. Related to imageio==2.22.1 that prevents napari from starting. i would suggest to pin either imageio!==2.22.1 in setup.cfg or suggesting to pip install "napari[all]==0.4.15 in the Readme.

  2. naaf version 0.3.1 doesn't have naaf.data submodule:

Traceback (most recent call last):
  File "/home/user/miniconda3/envs/blik/lib/python3.9/site-packages/npe2/_command_registry.py", line 33, in resolve
    self.function = utils.import_python_name(self.python_name)
  File "/home/user/miniconda3/envs/blik/lib/python3.9/site-packages/npe2/manifest/utils.py", line 240, in import_python_name
    mod = import_module(module_name)
  File "/home/user/miniconda3/envs/blik/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/user/miniconda3/envs/blik/lib/python3.9/site-packages/blik/reader.py", line 5, in <module>
    from naaf.data import Particles, Image
ModuleNotFoundError: No module named 'naaf.data'

Maybe you could pin naaf>=0.2.2, <=0.2.3 in a new version at least until you merge a branch compatible with naaf>=0.3.0?

problems in refactored io

There are a couple of issues with the way the IO has been refactored, listing them here so I can try to fix them

  • there seems to be a misunderstanding about what a STAR file is/means, it doesn't necessarily contain particles and a dynamo table file is not a star file

  • consequently, euler2matrix should not be a function only related to STAR file handling, but generally for handling euler angles in any file type that may come with euler angles (relion star files, dynamo tables, PEET/EMAN/STOPGAP etc etc)

  • star conventions now contains dynamo table info, wrong for above reasons

  • read_star is a monster of a function, difficult to customise file reading with the current conventions because we don't always know what a star file will contain in advance in terms of properties we might want to use

  • read_star is difficult to refactor into smaller pieces because in the various nested loops it calls data from various other stages, this should be handled in some kind of StarHandler object

Data parsing flow/architecture

In #78 we started discussing some incompatibilities of the current data flow and a delayed interface. I'm opening this issue to discuss how to move forward (and a proposal of what I think might make things easiest)

Current flow

Currently we infer how to read a file from the file extension, pass the file to a function and generate an in memory representation of the data. Sometimes we then dispatch this in memory representation to further function(s) for input checking and extracting the data we need and generating blocks.

# Issues for creating a delayed interface

The current flow lacks separation between inference of how to extract data and DataBlock creation, usually these two things happen in one function (e.g. star file reading)

In practice, this makes it hard to implement a delayed/lazy interface for functions like this one because the parsing logic is directly coupled to the data extraction and block generation

Proposed solution

Having a defined process for how data gets from its source into DataBlock form would simplify the addition of extra features (like a lazy interface, although there may be others we haven't thought of)

my feeling is that this is best implemented as an abstract BlockReader class from which any 'data source' -> DataBlock implementations would inherit.

This class would have a defined, layered structure, what I imagine is...

  1. data source -> in memory representation
  2. parsing logic, defining output block type
  3. in memory representation -> raw data required for generation of final DataBlock
  4. any necessary transformations of raw data to data required for DataBlock creation
  5. creation of DataBlock

This clear definition of the separate parsing 'layers' creates many 'seams' where we can implement any extra logic we might want. A good example of this is lazy loading which would be implemented at step 1. One day we might want the ability to halt after step 3 if transformations involved for going from in raw data to final representation involve expensive calculations? just an idea

When implemented, these classes will only provide references to the simple functions required for each layer. I definitely don't think each BlockReader implementation should contain all of the logic for a given file type

Let me know what you think?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.