reconstrue / brightfield Goto Github PK
View Code? Open in Web Editor NEWReconstruction of biocytin-stained neurons detected in brightfield microscopy image stacks
Home Page: http://reconstrue.com
License: Apache License 2.0
Reconstruction of biocytin-stained neurons detected in brightfield microscopy image stacks
Home Page: http://reconstrue.com
License: Apache License 2.0
This webpage simply reads specimens_manifest.json and generates a list of images. Each neuron is one image. Probably a single colorized, cropped MinIP. Click on any mugshot to visit the cell's solo personal page, which reads build_artifacts_manifest.json to generates its view. Probably a React app.
Some manually curated images are on reconstrue.com: cropped Turbo'd demo images on reconstrue.com
Novelty:
import reconstrue.brigthfield.depth_coder
Algorithm:
Finally, just show the color accumulator image, with or without transparency (dunno).
Bonus, take those images saved to disk (depth coded with inverse intensity as opacity), start with a pure white brightfield then merge that with each colored z-index image. That's what it would look like with light shining through but color filtered to depth. Then animate that as a GIF/movie. Pair that with the depth colored 2D projection mugshot.
See the ShuTu tutorial for multiple examples of datsets being loaded. Datasets generated multiple ways.
This issue describes a Jupyter notebook which is an image gallery of the ten neurons in the test set. The neurons images have SWCs superimposed upon them. There may be one or two SWCs per neuron (gold standards versus candidates). Note, this is an eyeballing tool; there is no programmatic quality evaluator that scores a candidate SWC against the corresponding gold standard.
If both SWCs are provided (gold standard and "other") then they can be juxtaposed by simultaneously rendering them in the same plot. This is how judges can review a submission.
If the gold standard SWC is not provided then this same notebook is essentially how a computer vision researcher can evaluate a models performance on the 10 specimen test set. For those ten specimens, this notebook visually overlays the skeleton atop the MinIP image (inverted TIFF stack).
The challenge is very specific: there are 10 neurons that a contestant needs to generate SWC files for. In its simplest form that could be simply the submission really being just 10 SWC files in a folder on the web. Each of the SWCs will be named according to its neuron ID, e.g. 651806289.swc
and those 10 are:
But the human judges will want more than just 10 SWC files. They will want those submitted SWCs juxtaposed with gold standard SWC, so as to compare.
This notebook is simply told the folder's URL and it renders the 10 neuron stacks and 20 SWCs (10 from human, 10 from the submitted challenge folder).
The goal is to have a Jupyter notebook already run on some set of 10 SWC files in a folder. I.e. it's a pre-run notebook showing what the layout will look like when done properly.
This artifact is the template (a Jupyter notebook) that challengers fill out to submit their work to the challenge. That URL will be forwarded to the judges, one copy the 10 SWC template notebook for each submission.
So, the evaluation notebook simply repeats the same thing for each of the ten TEST_DATA_SET neurons:
Basically, contestant needs to submit a list of 10 URLs to SWC files, and this notebook should be able to layout the results.
This also means that each of the ten SWCs can be generated sequentially on Colab, and then the ten SWCs viewed together finally, in this notebook.
Note: having the 10 neuron hardwired is bad. Want arbitrary array of neurons. This way a comp vision coder can juxtapose a model run against the 105 specimens in the training set. I.e. can be a tool for when training.
In the end, the topmost view of the cells in the dataset would be to plot them together in The Allen's CCF i.e. position them in the brain, to whatever accuracy is possible. 10 test neurons would be ok, 105 training neurons might get a bit crowded/overplotted.
This SVM tech is some simple stuff that should be checked out before going full on deep and convolutional. SVM is what was state-of-the-art a decade ago before deep learning stole the spotlight.
Identification of individual cells from z-stacks of bright-field microscopy images This
Really nice 2018 paper out of France. Simple, fast techniques (SVM classifier) simply looking at "z-pixels" (i.e. array of pixel intensities on the thin Z plane of a microscope slide; the colored curves). Basically, classifying SVM is light absorption around cell.
Somewhere in The Allen's YouTube channel there's bound to be some tech talk showing some background on the data collection pipeline. That would be engaging and easy to add to the intro backgrounder.
So, the Brightfield Challenge dataset is basically just a raw z-stack of 2D images (plus SWC files), each neuron's data in a separate directory. This project seeks to explore "doing neuroscience on Colab." So, crank out some code that does basic image processing on the image stack.
Ideally want:
See also
#182 TinyUnet
University of Freiburg seems to be the originator of U-Net:
This deep neural network is implemented with Keras functional API, which makes it extremely easy to experiment with different interesting architectures... runs seamlessly on CPU and GPU.
pytorch Implementation of U-Net, R2U-Net, Attention U-Net, Attention R2U-Net
https://github.com/LeeJunHyun/Image_Segmentation
This work is being done on Colab, datatset_get_eyes_on_image_stack.ipynb.
Maximal Image Projection (MIP) is the basic 3D z-stack to 2D overview viz for microscopy slides. Technically "Minimal" because brightFIELD. This seems to be a pretty fundamental overview visualization of slide image stacks. The algorithm to make the 2D projection just sifts through the z-stack, and for each (x,y) column keep the lowest (darkest) pixel value for the 2D projection.
Colab can be limiting in terms of what is allowed. Once we move to a Binder deploy, stock JupyterLab can be used for any Jupyter widget.
This was implemented in datatset_get_eyes_on_image_stack.ipynb Just a slider which sets the id of the specific z-index slide from input dataset to show.
So, this project is recapitulating a bunch of basic microscopy software, on Colab. One of the first software utilities that every toolset makes is a Z-stack explorer:
This would be relatively small as far as the animated GIFs already being produced (20 MB to 70 MB) and would be an interesting use of simple web tech to make a visualization.
Might want to have a transition color. This might best be done in JavaScript.
"B.G.-M. and M.d.J.S.M. are inventors on a patent application that describes QRBF."
A full MinIP has to be provided for anyone who might want to wander around the slide. But usually the action all happens around the biocytin stained cell. The MinIP cropped to the interesting part is a good cover image or specimens gallery catalog of mug shots.
Don't need to animate outside the ROI. So show MinIP as background and have animted GIF of just ROI laid out to align with MinIP background PNG.
This is done in brightfield_neuron_swc_by_shutu.ipynb, which essentially translates ShuTu's build.sh
into a Colab code cell, thereby customizing the build for Colab.
On 2019-10-12, it was shown that ShuTu can be installed on Colab. But the stock install instructions raises errors on Colab, harmless yet confusing errors.
It would be better to rewrite the build.sh as Colab code s.t. there are no errors. It's pretty simple: basically calling make
twice. Also no need to install demo data because will be using Allen Institute data.
Latest progress is on Colab: ShuTu process Allen.
To Do:
Work being done on Colab at: Sample the stack.
This will be a good way to demonstrate the nature of the brightfield microscope images. A way for those not familiar with the physics of microscopy to get an understanding of the effects of the data collection method on the data.
Bang out a start page for the project. Looks like it will involve at least 4 separate Jupyter notebooks. There should be one start page for an overview of what's going on.
That page is brightfield_neuron_reconstruction_challenge.ipynb
ShuTu tutorial says:
To get basic statsitics of the neuron after scaling, use Python script analyzeSWC.py. The command is
python analyzeSWC scaledSwcFilename
Might as well, CLI can work on Colab so it's something to do after SWC generation.
Allen Institute's Morphology technical white paper:
Serial images (63X magnification) through biocytin-filled neurons were evaluated for quality, and cells that passed a quality threshold entered a detailed morphological analysis workflow. Reconstructions of cell dendrites and the initial segment (spiny neurons) or complete axon (aspiny neurons) were generated for a subset of neurons using a 3D Visualization-Assisted Analysis (Vaa3D) workflow. The automated 3D reconstruction results were then manually curated using the Mozak extension of Vaa3D.
Whelp, can Vaa3D be spun-up somehow on Colab?
In the end, a submission to the challenge consists of 10 SWC files, one for each of the 10 neurons in TEST_DATA_SET
. That's the result of a lot of work but not that much data.
It would be interesting to see how well Google Forms could be used to actually perform an evaluation. This would dump the evaluation answers (quality score ranging from 1 to 10) directly into a Google Sheet spreadsheet.
This would be a low overhead way to admin a challenge and its evaluation competition.
Each neuron has an image stack.
So, how about "compress" the image stack into an animated GIF file of no more than a few MB, for easy viewing over the web? Looks like ImageJ already does this. That's Java; would be nice to find a Python implementation.
This work is implemented in a Jupyter notebook, challenge_dataset.ipynb.
brightfield-auto-reconstruction-competition
.TEST_DATA_SET
)Google Colab's allocated file system is 50G, so pulling down one neuron at a time will probably work.
As of Saturday 2019-10-12, this is confirmed to be working. See brightfield_neuron_swc_by_shutu_on_colab.ipynb
https://github.com/seung-lab/data-cube-x
Check out the demo: https://seung-lab.github.io/data-cube-x/
This is done: brightfield_neuron_swc_by_shutu.ipynb. One notebook that:
Note that this does not use Allen Institute data.
If there were a simple matplotlib (or fancier) histogram of the grayscale pixels, that might enable some intensity filtering a la brightness/constrast tweaking.
So, The Allen (and the rest of the scientific computing community(ies)) have decided that Python is the main language for users of the Allen stack. Of course Python would not have won out if not for the fact that mathematically intense Python code can be accelerated. Numba is the latest, greatest way to accelerate Python.
Part of the Reconstrue message is that neuroscience can really benefit from deploying to the cloud.
One of the very simplest things to Numba-ify would be intensity projections of 3D image stacks to 2D. This involves marching down the Z-index, each 2D image at a time and keeping the highest/lowest value for each "z-pixel" in a composite image/array.
In dataset_manifest.ipynb, keeping the testing version and dropping the training version (because all specimens are manifested by id so duplicate will not work.
Consider
That's a pretty legitimate use of Google subsidized research credits.
On Colab, you have to explicitly request a GPU. As of late 2019, there are three that have been seen recently
The first two work with Rapids, the K80 does not. Let's make sure we have one that works with Rapids.
In the first pass of getting ShuTu working on Colab, there is no focus on the Nvidia T4 GPU that is made freely available on Colab. ShuTu is not written to leverage GPUs, rather it uses multiple CPU cores, of which there is only one on Colab (cue sad trombone).
Nonetheless, the whole imaging pipeline should run on RAPIDs and CUDA etc. Just chuck the image stack onto the GPU and out comes a SWC. Yup, just that simple :)
The original image stacks are 6GB to ~60GB. That's too big for the web and to cram into laptop GPUs. But if the resolution is taken down by half, that 1/8 original size. Smallest datasets would be under 1 GB already.
Basically, want whole 3D stack at some resolution being manipulated by WebGL.
A much slicker implementation would be to translate the image stack into a Neuroglancer precomputed file set. The precomputed rendering is just flat files for 3D volumetric rendering over the web. That's about as optimized as it's going to get.
The code for generating specimens_manifest.json
is in dataset_manifest.ipynb.
The manifest is a JSON file. The cache is a Python object for downloading files in the manifest and caching them on the Colab local file system. This is handy for re-running notebooks after frequent disconnects.
Specimen features
The cache presents the 115 specimens in a unified way:
These are just objects in the manifest.json.
[These ideas are for later in the game, not early "just make it work" phase of this project.]
Once this is in a module, could check if manifest file exists and if not then crawl the dataset bucket.
The Allen is big on NWB:N, this dataset seems like a perfect use of NWB file. This project uses Python as the main programming language and there is PyNWB for working with NWB files. Additionally, the AllenSDK works with NWB files.
On the other hand, having the files loose in FS does make it clear what's in the challenge dataset.
Actually, the sweet spot might be Exdir: HDF5 structure but just file system and .yaml files.
Perhaps we could reorg the dataset. It would be nice to have sub-directories labeled:
Great 2015 talk:Python Packaging from Init to Deploy (26 min)
A while-deving, pre-module hack/mocking: If "modules are a thin wrapper around dictionaries" then mock a module and hang all the utilities off it.
I guess the name of the module is used like:
from reconstrue.brightfield import colab_display
Since using allensdk
requires a !pip install allensdk
going to be needing to do some pip-ing. So, might as well package up a bunch of the code as a git repo on github and just pip install
from the github URL. This will simplify the notebooks, and should make it easier for others to reimplement.
TReMAP from out of the Allen seems to be somewhat similar to ShuTu in that both project to 2D before tracing and then mapping back to 3D space.
CellTypes_Morph_Overview.pdf
After multi-scale enhancement, an image was then ready for automatic 3D reconstruction. Automatic 3D neuron reconstruction for very large 3D images remains a challenge in neuroscience. For these datasets, an automatic 3D neuron tracing method called TReMAP was developed (Zhou et al., 2016). The key advancement of TReMAP was to utilize 3D Virtual Finger (a reverse-mapping technique) to detect 3D neuron structures based on tracing results on 2D projection planes. TReMAP was used for all data generated after 07/21/2016, while NeuronCrawler was used prior to that date (Zhou et al., 2015).
Tracing the 2D Enhanced Plane
For each labeled group in the 2D enhanced image, 2D projection trees were traced. TReMAP uses All-PathPruning 2 (APP2) as the basic tracing module, as APP2 has been shown to be relatively fast and accurate based on pruning a dense initial reconstruction of a neuron to generate a compact representation of the neuron (Peng et al., 2011; Xiao and Peng, 2013).
Push it to the 12 hours.
This is working in initial_dataset_visualization.ipynb: Part 5: Juxtapose projection and skeleton.
This will simply copy The Allen's style:
Goal is to view SWCs on Colab.
[Viewing on desktop apps is nice but this project has an explicit goal of realizing all UI in Jupyter. But the ShuTu desktop app runs on windows, macOS, Linux – if one wishes to go desktop app old-school.]
Various potential types of tools, simplest first:
Note: The gold standard SWCs distinguish between axon and dendrites. What of somas?
See #62.
Initially, following the style of the paper handout from the first Challenge meeting, the viewer should do render
Various existing skeleton viewer codebases:
This work was done in initial_dataset_vizualization.ipynb.
Note that Twitter says:
What are the size and file type requirements?
Photos can be up to 5MB; animated GIFs can be up to 5MB on mobile, and up to 15MB on web.
We accept GIF, JPEG, and PNG files.
We DO NOT accept BMP, TIFF or other file formats.
Your photo will be automatically scaled for display in your expanded Tweet and in your gallery.
There should be a script that walks a built/ dirtree and manifests all files. This dumps the info to artifacts_manifest.json. A catalog of specimens would have one artifacts_manifest.json per specimen directory. An html page with a specimens_manifest.json could then walk the catalog: just a dropdown to select specimen and all resources from that dir are loaded into the page.
Part of the web catalog is simply the downsampled slides. So there are relative links to those. But the originals could be behind an absolute URL to another site. That site could be Wasabi, but it's not configured for that, currently. That's 2.5 TB of storage.
Demo: datatset_get_eyes_on_image_stack.ipynb
To do
Goal, animated GIF of stack i.e. all images into a single GIF. This is an initial "eyes on data" thing, which does not involve any automated reconstructions, just visualizing the input dataset.
The Brightfield Challenge dataset is basically just a raw z-stack of 2D images, plus SWC files. This project seeks to explore "doing neuroscience on Colab." So, crank out some code that does basic image processing on the image stack
Each neuron has an image stack.
The whole image stack is on the order of 6 to 60 GB
Each stack has a few hundred images
Each image file is on the order of 10s of MB each
So, how about "compress" the image stack into an animated GIF file of no more than a few MB, for easy viewing over the web? Looks like ImageJ already does this. That's Java; would be nice to find a Python implementation.
The simplest and crudest thing to do is simply download the SWC file.
But folks can also set up Google Drive and mirror a folder to localhost to make the process simpler. Having the code write the SWC to Drive will ensure it is persisted across VM reboots (at least every 12 hours, max time allowed on Colab).
Would be cool if this had the stack as a movie embedded in a tweet, simply by tweeting the URL to a neuron's page.
This is all static HTML markup.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.