aidenlab / juicebox

Visualization and analysis software for Hi-C data -

---

Breaking News!

This codebase is in the process of being split up to better support long-term maintenance and modular aspects of different tools available. The new repositories will be:

• JuiceboxGUI for visualization of Hi-C maps with Juicebox Desktop and genome assembly correction with Assembly Tools.
• HiCTools for building and writing .hic files (Pre, Addnorm, and Statistics)
• JuicerTools for downstream analysis of .hic files (HiCCUPS, Arrowhead, APA, etc.)
• Java Straw to quickly read and stream data from .hic files into Java, and is used by the above repositories.

We also have new tools:

• Straw to quickly read and stream data from .hic files into C++, python, R, and MATLAB.
• POSSUMM for new C++ code to rapidly calculate A-B compartments (i.e. eigenvectors) for higher resolutions
• EMT for upgrading older .hic files or making smaller .hic files for regions of interest.

---

About Juicebox

Juicebox is visualization software for Hi-C data. This distribution includes the source code for Juicebox, Juicer Tools, and Assembly Tools . Download Juicebox here, or use Juicebox on the web. Detailed documentation is available on the wiki. Instructions below pertain primarily to usage of command line tools and the Juicebox jar files.

Juicebox can now be used to visualize and interactively (re)assemble genomes. Check out the Juicebox Assembly Tools Module website https://aidenlab.org/assembly for more details on how to use Juicebox for assembly.

Juicebox was created by Jim Robinson, Neva C. Durand, and Erez Lieberman Aiden. Past contributors include Ido Machol, Zulkifl Gire, Marie Hoeger, Fanny Huang, Nam Hee Kim, Vi Nguyen, Jay Ryu, Nathaniel T. Musial, and Ragib Mostofa.

Ongoing development work is carried out by Muhammad Saad Shamim, Neva C. Durand, and Olga Dudchenko.

---

Questions?

For FAQs, or for asking new questions, please see our forum: aidenlab.org/forum.html.

---

IntelliJ Setup

Use IntelliJ IDEA (Community edition - free)

To set up in IDEA, have the Java SDK installed then you'll point to it (IntelliJ has lots of documentation on this sort of thing).

• Then go to VCS -> checkout from version control.

• You'll need to do is be sure *.sizes is included as a file to be copied over to the class files. Set this up via IntelliJ Preferences -> Compiler. Add ?*.sizes to the list of Resource Patterns.

• While there, also go to Java Compiler and put this into additional command line options: -Xlint:all -target 1.7 The former turns on all warnings, the latter gives some flexibility since some people haven't updated Java to 1.8 yet.

• Then go to Run -> Edit Configurations.

• With the + sign, add Application.

• You'll create two of these, one for the GUI (call it Juicebox GUI or whatever you want, really) and one for the CLT.

• Set the main class by clicking the little ... button next to the text box for main class

    MainWindow.java is the main method class for the visualization/GUI portion of the software.
    HiCTools.java is the main method class for the analysis/CLT portion.
  

• For the GUI under VM Options:

    -Xmx2000m
    -Djnlp.loadMenu="https://hicfiles.tc4ga.com/juicebox.properties"
  

• For the CLT use

    -Xmx2000m
  

• Note that the Xmx2000m flag sets the maximum memory heap size to 2GB. Depending on your computer you might want more or less. Some tools will break if there's not enough memory and the file is too large, but don't worry about that for development; 2GB should be fine.

• One last note: be sure to Commit and Push when you commit files, it's hidden in the dropdown menu button in the commit window.

---

Hardware and Software Requirements

The minimum software requirement to run Juicebox is a working Java installation (version > 1.6) on Windows, Linux, and Mac OSX. We recommend using the latest Java version available, but please do not use the Java Beta Version. Minimum system requirements for running Java can be found at https://java.com/en/download/help/sysreq.xml. To download and install the latest Java Runtime Environment (JRE), please go to https://www.java.com/download.

We recommend having at least 2GB free RAM for the best user experience with Juicebox.

To launch the Juicebox application from command line, type java -Xms512m -Xmx2048m -jar Juicebox.jar

Note: the -Xms512m flag sets the minimum memory heap size at 512 megabytes, and the -Xmx2048m flag sets the maximum size at 2048 megabytes (2 gigabytes). These values may be adjusted as appropriate for your machine.

---

Documentation

We have extensive documentation for how to use Juicebox at https://github.com/theaidenlab/juicebox/wiki including a video, a Quick Start Guide, and a detailed tutorial.

For using Juicebox to assemble genomes see https://www.aidenlab.org/assembly/.

---

Command Line Tools Usage

See the documentation at https://github.com/theaidenlab/juicer/wiki for information on how to use the Juicer tools.

---

Compiling Jars from Source Files

1. You should have Java 1.8 JDK and Apache Ant installed on your system. See below for more information.
2. Go to the folder containing the Juicebox source files and edit the juicebox.properties file with the proper Java JDK Address.
3. Open the command line, navigate to the folder containing the build.xml file and type ant The process should take no more than a minute to build on most machines.
4. The jars are written to the directory out/. You can change this by editing the build.xml file.

• Installing Java 1.8 JDK

For Windows/Mac/Linux, the Java 1.8 JDK can be installed from here: https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (Alternative) For Ubuntu/LinuxMint https://tecadmin.net/install-oracle-java-8-jdk-8-ubuntu-via-ppa/

• Installing Apache Ant Mac Ant should be installed on most Macs. To verify installation via the command prompt, type ant -version If Ant is not on your Mac, install it via homebrew. At the command prompt, type brew update brew install ant You may need to install Homebrew (https://brew.sh/) on your machine See the following Stackoverflow post for more details: https://stackoverflow.com/questions/3222804/how-can-i-install-apache-ant-on-mac-os-x

Windows Installing Ant requires some minor changes to your system environment. Follow the instructions in this article: https://www.nczonline.net/blog/2012/04/12/how-to-install-apache-ant-on-windows/

Linux In the command prompt, type sudo apt-get install ant or sudo yum install ant depending on your package installer

juicebox's Issues

Have all the expected maps in subtabs of the Expected tab in Dataset metrics

Would be excellent to do this earlier than 1.1.

Color range tool tip incorrect

Load a map. Load a control. Go to some chromosome, some zoom level. Change the slider, e.g. by moving up the minimum. Now switch to control view. The tool tip gives the previous values instead of the correct ones.

Persistent Annotations

Load an annotation (1D and 2D). Now load an invalid annotation (I had to generate one) while unchecking the 2 previous annotations. The previous 2 will persist and cannot be removed while the latest one is not loaded. The latter issue is being worked on, but the persistence is a separate bug. The persistent 2D annotation can be removed if it is loaded and unloaded again, but the 1D annotation is persistent unless manually deleted.

Add waiting cursor / glass pane when loading pearsons

glass pane on loading annotations and loading from Recent

Legacy Code? - TrackLabelPanel

TrackLabelPanel may be legacy code, or may deal with eigenvector calculations, etc. If not used, remove.

Juicebox Crash

Several duplicate errors occurred and froze Juicebox.
Not perfectly reproducable bug, but often occurs at low resolution Chr 19 on NHEK HIC068 map.

After this bug occurs, even zoomed out high res maps appear all white, and the color range also messed up. This bug disappears when chromosome selection is reset.

May relate to Issue #29

java.util.zip.DataFormatException: incorrect header check
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:259)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:80)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:49)
at juicebox.data.DatasetReaderV2.readBlock(DatasetReaderV2.java:544)
at juicebox.data.DatasetReaderV2.readNormalizedBlock(DatasetReaderV2.java:638)
at juicebox.data.MatrixZoomData$1.run(MatrixZoomData.java:191)
at java.lang.Thread.run(Thread.java:745)

Pearson and "auto resolution" problem.

Open map, move to chromosome. switch to Pearson.
Move to smaller chromosome.

Message "Pearson is not available at current resolution"

Reason:
"Auto resolution" figures that the smaller chromosome requires different resolution.

Suggested solution: either cancel the smart resolution change for Pearson or check availability of Pearson in the decision.

Neva: open 2009 HindIII, click on chromosome 1, click on resolution slider for frags, the tick marks are messed up

addNorm is slow

KR portion of addNorm is slow, particularly at very high (fragment) resolution.

Upload all HiC files mentioned in the paper to AWS

Sync/Linked maps not working when maps are from different types/species.

Reason: some of the maps concatenate "chr" to chromosome names. When sync is trying to send the info between maps, it will fail to translate from chr to non chr annotation. e.g. chr12 ->12 or 12->chr12.

Wrong error message when no map is selected in "Open" menu

Use:
File->Open->Cancel

Error: "Can't find properties file for loading list"

Color range slider

The color range slider has a serious bug involving the way it updates. There's no model behind it right now and it leads to very bad behavior

Make sure sync is actually working and potentially document how to use it

Talk to Suhas or Erez, I'm not really sure how either sync or link work. Link is very buggy and should be disabled.

O/E color range

Ability to adjust color range on O/E

LX maps chromosomes are labeled "chr+#" while others are labeled just "#"

Click on x,y pixel, create a track

This is a high priority of Erez's. Go here and click on a point to see what he means:

http://hic.umassmed.edu/welcome/welcome.php

Listed as an enhancement for 1.1, though Erez would prefer it in 1.0.

Documentation

Resizing label panel

If only 1 or 2 annotation tracks are loaded, the panel remains small and the text gets cut off. It would be better if for such cases when the panel size < max labelTextSize for the panel to be properly resized so that the text will fit. However, this will also require resizing of the other panels. Alternative is to only resize the labelPanel with a transparent background and force it to be painted last / on top of everything. That way excess text will overlap on the upper annotation tracks instead.

{ // can use this to find text width
FontMetrics fontMetrics = graphics.getFontMetrics();
Rectangle2D textBounds = fontMetrics.getStringBounds(text, graphics);
int labelTextSize = textBounds.getWidth();
}

Also, change font to match others (not sure which is used).

HIC104 MQ > 30

After opening of HIC104 MQ > 30, and zomming in,
I got endless repeat of:
java.util.zip.DataFormatException: incorrect header check
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:259)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:80)
at org.broad.igv.util.CompressionUtils.decompress(CompressionUtils.java:49)
at juicebox.data.DatasetReaderV2.readBlock(DatasetReaderV2.java:544)
at juicebox.data.DatasetReaderV2.readNormalizedBlock(DatasetReaderV2.java:629)
at juicebox.data.MatrixZoomData$1.run(MatrixZoomData.java:191)
at java.lang.Thread.run(Thread.java:745)

Balanced All vs All

I've opened the K563 map.
Clicked to zoom into chromosome.
Changed normalization to balanced.
Changed chromzomes back to "All", and refreshed...

Result:

HiC file version: 8
java.io.IOException: Normalization missing for: All_All_BP_6197
at juicebox.data.DatasetReaderV2.readNormalizedBlock(DatasetReaderV2.java:634)
at juicebox.data.MatrixZoomData$1.run(MatrixZoomData.java:191)
at java.lang.Thread.run(Thread.java:745)
INFO [2014-11-26 23:11:13,234] [MessageUtils.java:60] [Thread-25] Normalization missing for: All_All_BP_6197

• Empty message box.

IGV tracks when click in HiC

Will require Jim's help, low priority

Update HIC maps that need it

This requires making sure we are happy with the new format for Dataset metrics and then creating a script and having people update a bunch of hic files on the Broad

Take GoTo out of where it is now and put it in the GUI

Have it be like in IGV, with two little editable text boxes with the extents.

Make addNorm automatic

Add norms by default instead of as a separate option

Adding Unit Testing to Juicebox

hitting the "minus" button too many times can make the color range invalid and uneditable

In this case I loaded HIC031 as the control then continuously hit the minus button to change the color range. It became infinity and could not be edited in the dialog.

black box

A black box appears in the corner of the heat map. Eventually it goes away, or if the refresh button is hit it goes away. Not sure this happens on the jnlp or just in development.

Hardcoded Broad Links

there's an HTML or XML file where it gets the track file names. it's hardcoded somewhere. probably if you "Find in Path" broadinstitute you'll find it.

Loading same map

If I load the same map that is already open, Juicebox downloads the map again rather than simply switching to the full view. This causes an unnecessary delay and seems like a simple fix (verify that the names are different or something?). This occurs in both the Open... and Recent methods.

"Normalization missing" exception raised when trying to read two data parts.

Reproduce: go to balanced normalization, switch quickly between resolutions.

BAM support

This would require some thought and work; instead of merged_nodups, we could have a BAM file. Is this really useful?

get rid of command line tools for release

Make it possible to easily change the statistics or graphs in header

Requires a submethod for rewriting an already created hic map.

Dataset metrics overhaul

get rid of hardcoded links/change

change hard coded econpy links to tc4ga. also search code for broadinstitute links and copy over. do not delete before copying what they refer to (otherwise you'll break things)

Linux debugging

Annotation errors should alert

An annotation file (1D or 2D) that is improperly formatted doesn’t return any error message when it is loaded, the viewer just doesn’t display anything.

Color Range tooltip not detailed enough

The tooltip on the color range slider should give both range information and value information

Labels on left for tracks

heat map right side white space

Loop Tracks

Erez (or rather Eric) would like implementations of loops via tracks, in the form of arc type connections. Lower priority, but ideally in 1.0.

mini-hic viewer in IGV

Will require Jim's help

Exceptions thrown when HeLa annotations loaded

Ability to save state and reload

This would be really useful; some way to save exactly where you're at in Juicebox and later reload that saved state

Statistics done in CLT

Instead of using the Perl file, run Juicebox CLT on the merged_nodups file, statistics calculated automatically. This will take some work

This is a test

testing

Too many warnings when norms failed

Sometimes KR norms fail and the norm vector is all NaNs, this just shows as an empty gray map (look at chromosome 19 on a smaller HiSeq map to see)

Return to All vs All

Open a map, go to chromosome, change to O/E. Change chromosome chooser boxes to "All vs All", hit refresh. Map is all red.