openworm / openworm

Repository for the main Dockerfile with the OpenWorm software stack and project-wide issues

About OpenWorm

OpenWorm aims to build the first comprehensive computational model of Caenorhabditis elegans (C. elegans), a microscopic roundworm. With only a thousand cells, it solves basic problems such as feeding, mate-finding and predator avoidance. Despite being extremely well-studied in biology, a deep, principled understanding of the biology of this organism remains elusive.

We are using a bottom-up approach, aimed at observing the worm behaviour emerge from a simulation of data derived from scientific experiments carried out over the past decade. To do so, we are incorporating the data available from the scientific community into software models. We are also forging new collaborations with universities and research institutes to collect data that fill in the gaps.

You can earn a badge with us simply by trying out this package! Click on the image below to get started.

Quickstart

We have put together a Docker container that pulls together the major components of our simulation and runs it on your machine. When you get it all running it does the following:

1. Run our nervous system model, known as c302, on your computer.
2. In parallel, run our 3D worm body model, known as Sibernetic, on your computer, using the output of the nervous system model.
3. Produce graphs from the nervous system and body model that demonstrate its behavior on your computer for you to inspect.
4. Produce a movie showing the output of the body model.

Example Output

NOTE: Running the simulation for the full amount of time would produce content like the above. However, in order to run in a reasonable amount of time, the default run time for the simulation is limited. As such, you will see only a partial output, equivalent to about 5% of run time, compared to the examples above. To extend the run time, use the -d argument as described below.

Installation

Pre-requisites:

1. You should have at least 60 GB of free space on your machine and at least 2GB of RAM
2. You should be able to clone git repositories on your machine. Install git, or this GUI may be useful.

To Install:

1. Install Docker on your system.
2. If your system does not have enough free space, you can use an external hard disk. On MacOS X, the location for image storage can be specified in the Advanced Tab in Preferences. See this thread in addition for Linux instructions.

Running

1. Ensure the Docker daemon is running in the background (on MacOS/Windows there should be an icon with the Docker whale logo showing in the menu bar/system tray).
2. Open a terminal and run: git clone http://github.com/openworm/openworm; cd openworm
3. Optional: Run ./build.sh (or build.cmd on Windows). If you skip this step, it will download the latest released Docker image from the OpenWorm Docker hub.
4. Run ./run.sh (or run.cmd on Windows).
5. About 5-10 minutes of output will display on the screen as the steps run.
6. The simulation will end. Run stop.sh (stop.cmd on Windows) on your system to clean up the running container.
7. Inspect the output in the output directory on your local machine.

Advanced

Arguments

• -d [num] : Use to modify the duration of the simulation in milliseconds. Default is 15. Use 5000 to run for time to make the full movie above (i.e. 5 seconds).

Other things to try

• Open a terminal and run ./run-shell-only.sh (or run-shell-only.cmd on Windows). This will let you log into the container before it has run master_openworm.py. From here you can inspect the internals of the various checked out code bases and installed systems and modify things. Afterwards you'll still need to run ./stop.sh to clean up.
• If you wish to modify what gets installed, you should modify Dockerfile. If you want to modify what runs, you should modify master_openworm.py. Either way you will need to run build.sh in order to rebuild the image locally. Afterwards you can run normally.

FAQ

What is the Docker container?

The Docker container is a self-contained environment in which you can run OpenWorm simulations. It's fully set up to get you started by following the steps above. At the moment, it runs simulations and produces visualizations for you, but these visualizations must be viewed outside of the Docker container. While you do not need to know much about Docker to use OpenWorm, if you are planning on working extensively with the platform, you may benefit from understanding some basics. Docker Curriculum is an excellent tutorial for beginners that is straightforward to work through (Sections 1 - 2.5 are plenty sufficient).

Is it possible to modify the simulation without having to run build.sh?

Yes, but it is marginally more complex. The easiest way is to modify anything in the Docker container once you are inside of it - it will work just like a bash shell. If you want to modify any code in the container, you'll need to use an editor that runs in the terminal, like nano. Once you've modified something in the container, you don't need to re-build. However, if you run stop.sh once you exit, those changes will be gone.

How do I access more data than what is already output?

The simulation by default outputs only a few figures and movies to your home system (that is, outside of the Docker container). If you want to access the entire output of the simulation, you will need to copy it from the Docker container.

For example, say you want to extract the worm motion data. This is contained in the file worm_motion_log.txt, which is found in the /home/ow/sibernetic/simulations/[SPECIFIC_TIMESTAMPED_DIRECTORY]/worm_motion_log.txt. The directory [SPECIFIC_TIMESTAMPED_DIRECTORY] will have a name like C2_FW_2018_02-12_18-36-32, and its name can be found by checking the output directory. This is actually the main output directory for the simulation, and contains all output, including cell modelling and worm movement.

Once the simulation ends and you exit the container with exit, but before you run stop.sh, run the following command from the openworm-docker-master folder:

docker cp openworm:/home/ow/sibernetic/simulations/[SPECIFIC_TIMESTAMPED_DIRECTORY]/worm_motion_log.txt ./worm_motion_log.txt

This will copy the file from the Docker container, whose default name is openworm. It is crucial that you do not run stop.sh before trying to get your data out (see below)

What is the difference between exit and stop.sh?

When you are in the Docker Container openworm, and are done interacting with it, you type exit to return to your system's shell. This stops execution of anything in the container, and that container's status is now Exited. If you try to re-start the process using run-shell-only.sh, you will get an error saying that the container already exists. You can choose, at this point, to run stop.sh. Doing so will remove the container and any files associated with it, allowing you to run a new simulation. However, if you don't want to remove that container, you will instead want to re-enter it.

How do I enter a container I just exited?

If you run stop.sh you'll delete your data and reset the container for a new run. If, however, you don't want to do that, you can re-enter the Docker container like this:

docker start openworm                 # Restarts the container
docker exec -it openworm /bin/bash    # Runs bash inside the container

This tells Docker to start the container, to execute commands (exec) with an interactive, tty (-it) bash (bash) shell in the container openworm.

You'll be able to interact with the container as before.

Documentation

to find out more about OpenWorm, please see the documentation at http://docs.openworm.org or join us on Slack.

This repository also contains project-wide tracking via high-level issues and milestones.

openworm's Issues

Find the source and solve the problem of 'non-determinism' of the SPH

Our implementation of PCISPH is expected to be fully deterministic (no any random factors included), but in current version different runs show slightly different behavior of the system. The source of this should be found and removed/fixed to make the code work correctly.

Add special type of elastic matter which can contract in answer to incoming signal

Define the known Neuropeptides for each Neuron

Create an on-going spreadsheet that defines each Neuron and the known peptides that are expressed within that Neuron. This data is handlely available through many sources starting with the WormAtlas.org

Document the Ion Channels within the Neurons

Define how each Neuropeptide is created and stored in the Neuron cell

Peptides and Proteins are created by specific metabolic pathways within a cell. As we define the Neuropeptides and their end physiology, we want to map out their pathways for creation and distribution within the cell.

Create volunteer signup form

Should ask for basic information, including email address. From this we will create a user account on CATMAID and divide up the work.

Wrap JLEMS

Wrap the current version of JLEMS [https://github.com/robertcannon/jLEMS] fo basic NeuroML compatible single-compartmental functionality.

Update docs and wiki

Solve the correct PCISPH integration problem

Two different implementations of boundaries (impermeable planes or walls made of non-moving particles) show noticeably different behavior of the system, including time scale, as it seems to me. In mentioned cases different algorithms of particle posityions integration is used, but observed behavior of the system should be almost similar, difference should show itself in small details. We should find the source of observed significant difference, to make sure that the version in which boundaries are represented with non-moving particles (which should be more stable, as expected) works correctly.

Make a complexity vs understanding figure

Create HivePlot data set from connectome

A HivePlot is a great way to visualize complex networks. To better understand what these are, check out the general introduction slides.

The c. elegans connectome is the connectivity graph of all of the neurons of its simple "brain". Visualizing this graph has yielded some insight into its structure, but it is still hard to pull patterns out of this graph as it is highly complex.

As far as I can find, HivePlots have not yet been applied to connectomes. This is a great opportunity to start.

• Take the connectivity graph data from our spreadsheet (first sheet only to start)
• Download the java HivePlot program
• Open an example under this program to understand the format:

A [attribute1=100 attribute2=1000] # node 'A'
B [attribute1=200 attribute2=2000] # node 'B'
C [attribute1=400 attribute2=4000] # node 'C'
A -> B [atttribute1=100 attribute2=1000 attribute3=positive] # edge from 'A' to 'B'
B -> C [attribute1=200 attribute2=2000 attribute3=negative] # edge from 'B' to 'C'

• Write a python script to convert the connectivity graph data into this format.
• Load the data up, play with different ways of creating the hive plot for this data set, generate images & share.

Name the Simulation Engine

The Simulation Engine needs a name, I created a document to gather proposals, please fill in yours by 18/12/12:

https://docs.google.com/document/d/1kUovm_Qpzlr7RIb0nwn7PtgLQJSfmIyz7DIR_PXnN0w/edit

Review VET section and make edits

Release Fabric script that loads the latest simulation engine packages on AWS EC2

The current deployment instructions still don't make it obvious how to install certain parts like OpenCL drivers. A Fabric script will pick a specific AMI to launch and will be able to control the version of drivers and of the JVM to ensure maximum consistency with the framework.

Create localisation script for the website

Create a Python script to generate alternative versions of the website in other languages.
The script will go through a resource file for every page of the website and replace the string contained in the value of the DOM elements according to the ID specified in the resource file.

Example:

index.it

...
catch_prase_1=Stiamo costruendo un verme digitale. Per davvero.
get_started_lbl=Inizia
get_involved_lbl=Collabora
...

index.html

...
<div class="container">
    <h1>OpenWorm</h1>
    <p id="catch_prase_1">We are building a digital worm. For real.</p>
    <p><a id="get_involved_lbl" href="./getting_started.html" class="btn btn-primary btn-large">Get Started</a></p>
...

The script file will replace "We are building a digital worm. For real." with "Stiamo costruendo un verme digitale. Per davvero." and so on.
The script will write the translated version in a folder named like the extension of the resource file, "it" in this case.
There will be one resource file per page.
If the website has 8 pages and we support 5 languages we will have 40 resource files.

The rationale behind this script is to not have redundant code and still have the website indexed in all the languages.

In order to select the different DOM elements pyquery looks at first glance like a good solution allowing to have selectors like in jQuery inside a Python script.

Update design/architecture docs

Create a reference implementation of the Hines optimizations to Hodgkin Huxley equations

Without implementing key optimizations, creating an efficient multi-compartmental neuronal solver will be very difficult. The paper describing the optimizations is available here. A good reference implementation language would be Python, although Java would be great as well.

Muscle contraction in response to scalar input from an external program.

The external program will later be a neural simulation. This depends on the anticipated implemention of SPH muscle contraction by Andrey and Sergey.

Document Ion Channels: Research Claims

Document how the part that Ion channels play within a cell determine specific behaviors and outcomes.

Ref: http://www.nature.com/emboj/journal/v27/n24/full/emboj2008252a.html
A synaptic DEG/ENaC ion channel mediates learning in C. elegans by facilitating dopamine
signalling
Giannis Voglis and Nektarios Tavernarakis

Translate the text on the site to Spanish

We have the (beginning of a document doing this translation)[https://docs.google.com/document/d/1e0rMQfAfm9UZdLVbW09KAE5CAldqbZFoo1tY8MGJmHY/edit]. We need to add to it as we finalize the text of the project.

Create documentation and package to allow others to play with the model optimization code

Port latest version of PCI-SPH algorithm

Port the latest version of PCI-SPH algorithm [https://github.com/openworm/Smoothed-Particle-Hydrodynamics/tree/IntegratedVersion] to the Java solver for the simulation engine [https://github.com/openworm/org.openworm.simulationengine.solver.sph].

Move wiki from Google Code to GitHub

Working on this with @JohnIdol -- @vellamike was involved in getting this started -- @tarelli may want to keep an eye on this as it relates to the website revamp.

Currently the wiki has been converted but the conversion script messed up a lot. Each page requires us to go through it, and we need to come up with a better high level outline.

Posted a revised outline for the wiki over on the wiki homepage.

Secondary mechanical/electrophysiological muscle cell integration.

Second phase is bi-directional integration (electro-mechanical communication) implemented (muscle cell surface stress affects membrane potential/membrane chemical conditions).

Rename bundles

Bundles need to be renamed from org.openworm.simulationengine.X to org.geppetto.X
Short name is going to be useful also to make the repository easily browsable on GitHub.

Create wiki page that has instructions for how to help with synapse marking

Current guide to the rules of engagement from which to create the tutorial is here

Primary muscle cell mechanical/electrophysiological integration

For the primary phase, mechanics will be integrated with muscle cell electrophysiology in a uni-directional manner (muscle cell membrane potential affects cell mechanics but not vice-versa)

Review all the text

All the text needs to be reviewed before deploying the new website.

Define the Physiology of each Neuropeptide for each given Neuron

Document how the Neuropeptide is used by the Neuron and the effects of changes that the Neuropeptide expresses. Initially we will create this in a spreadsheet but the end result of data storage is to be determined based on the complexity of what we find.

Define the extended reasearch on Ion Channels

Write about open source nature of project

Add data structure and functions responsible for implementation of impermeable surface

Elemental object is a triangle build on 3 adjacent elastic matter particles.
It is necessary to build complex constructions made of such triangles to make worm body, possibly cell membranes as well, etc.

Make a BTT figure

Find ion channels that relate to motor neurons AS01, AS02, DA01, DA02, DB01, DD01, SMDDL, SMDDR

Define the specific types of Ion Channels and how they work within the cell.

Here's a powerpoint presentation that explains what ion channels are and some of the math behind simulating them.

Here's a book explaining what ion channels are, and how they relate to neurons.

This spreadsheet contains a very early documenting of ion channels in muscle cells, but begins to lay out the way in which we should start describing them.

Gene expression patterns that can be used to infer the location of ion channels are in another sheet in that same spreadsheet.

Here's a list of neurotransmitter receptors and what neurotransmitters they receive.

A wealth of neurotransmitter receptors are described on [https://docs.google.com/spreadsheet/ccc?key=0ArNOKkIblIRadE5lVkN4WGMzNUZLdVROUEZVTUlUVGc#gid=0](this spreadsheet), maintained by Z.F. Altun.

Textpresso for c elegans provides full text search of c. elegans journal articles and can be used to find papers that talk about specific ion channel types.

Create a new website

Remove Glutamate_GJ etc in neuroConstruct project

These names have been given to the "neurotransmitter" in the XL spreadsheet for gap junctions, e.g.

AIAL ASIL GapJunction 2 Acetylcholine_GJ

However, this only means: There are 2 gap junction connections between AIAL and ASIL (and AIAL's chemical synapses' neurotransmitter is Acetylcholine)

Thankfully there is an equivalent entry for the other direction too:

ASIL AIAL GapJunction 2 Glutamate_GJ

These should only be added as one gap junction connection in the nC project between AIAL and ASIL with 2 connections.

As a first approximation all GJ connections can have the same strength, set by a new Cell Mechanism GapJunction (as opposed to each connection having a cell mechanism Acetylcholine_GJ etc.)

Consolidate all spreadsheets into one

All excel spreadsheets with data on connectivity, annotations for cells etc. should be consolidated into on central resource.

This can be used for seeding the nC project & generating annotated NeuroML, which can in turn be used for the 3D visualiser. Excel is easier for people to contribute to/annotate/check than XML.

This can be the version under GitHub https://github.com/openworm/CElegansNeuroML/blob/master/CElegansNeuronTables.xls or on google docs: https://docs.google.com/spreadsheet/ccc?key=0Avt3mQaA-HaMdHZuZnFuZmI5Q1VRU0VMekZ5d1QyZVE#gid=0 (probably best option)

Ideally there shouldn't be a copy on Dropbox, but clear pointers to the central version

Document the Ion Channels used in the Neurons

Brief detail about optimization in tech section

First DataViz Experiment using D3JS

We'd like to create a map of the completeness of the model and it involves a few different steps. We've found the following viz example that we'd like to try to build off of:

http://mbostock.github.com/d3/talk/20111116/pack-hierarchy.html

this uses the following library (http://d3js.org/)

The pack-hierarchy example is driven by the following JSON:

http://mbostock.github.com/d3/talk/20111116/flare.json

We figure if we can write JSON out in this manner that comes from the data sets that we have then it will be easy to create this visualization without having to do much d3js programming. So the problem is basically just converting a spreadsheet into this JSON.

For iteration 0, we'd like to have a Python script that can take this sheet that lists the c. elegans neurons and creates JSON like flare.json that can drive a visualization.

The simplest way to do this is to download the sheet as CSV and parse as text. A better way would be to use the GSpread library to access the spreadsheet programmatically on Google Drive.

The top level should be the neuron names in Column B, then as children from that top level should be the neurotransmitter in column D and the Receptor information in Column F. From here there are a lot of bells and whistles to add but this is a good place to start. We are interested in having a Python script rather than a program in another language because we'd like to put it under version control and evolve the visualization as we have additional data to put into it (we already do but merging the other data sets is yet another project).

The neurons send chemical signals using different molecules -- those are the neurotransmitters. The receptors are the "locks" that the "key" of the neurotransmitters fit into. The receptors then have their own dynamics that influences how the neurons work.

Create Geppetto logo

Get numerical data from experimental videos.

Data is to be used in optimization process.

Automated optimization for development of integrated worm

Initial investigation into possibility of using automated optimization to tune model worm behaviour. Worm behaviour will be cost function.

Consolidate edits to Turing test manuscript

The Turing test manuscript has many edits that are needed. Make a pass particularly on the computational architecture before the 12/18 publication meeting.

Complete tutorial video that includes marking synapses in CATMAID

Currently Steve Cook is working on this.

Develop a strict mathematical definition for "goodness" of fit for an experimental data vs model

This is a kind of "cost function for humans". It is necessary to set a benchmark for what we mean by developing a model which replicates the experimental observations.

Decide on how to define the Turing Test

We need to come to some consensus on basic features of the Turing test paper. I have tried to spell it out here in the issue. I have marked with first initial where I think some of you fall, but please correct if this is not accurate.

• Human creativity test (B, G) versus set of specific metrics (P, SL)
  • Human creativity test broadly means we don't set any specific tests other than providing access to the model. Humans can do whatever they want to test.
  • Specific metrics means we list out the tests we want to see done.
  • Possible combination of both? - Testing specific metrics by providing user access to constrained model where they can only interact in limited ways. (MV)
• Black box (B, P, G, MV) vs. glass box (SL)
  • Black box only considers behavior that is visible from outside the body
  • Glass box also considers cellular activity
• General theory for tests (G, B, SL) or very specific test? (P)
  • General theory for tests would propose a broader context into which tests are done
  • Very specific test would avoid a broader context and only focus on a specific battery of tests, for which there could be others that won't be touched on at all by this paper.

G - @JohnIdol T - @Interintel A - @a-palyanov SK - @skhayrulin MV - @vellamike P - @pgleeson SL - @slarson B - @balazs1987 - MC - @tarelli

Highly-simplified proof of concept electro-mechanical worm

Multiple simplified "cells" connected together in a highly simplified worm model (no neural network).

Build simulation which simulates full experimental protocl (hyperpolarizing ramp, current ramp etc)

This will require additions to Pyramidal and possibly libNeuroML

Have a parallel Spanish version of the site

This requires having a link in the upper left hand corner that can switch between english and spanish.