kennethsinder / oct-opus Goto Github PK

This includes adding a README.md in appropriate places and making sure our team can work with the Jupyter / port forwarding stuff easily.

https://www.digitalocean.com/community/tutorials/how-to-install-run-connect-to-jupyter-notebook-on-remote-server

I brought this up in the Optics FYDP Messenger chat at 2019-07-16 around 5:55 p.m. Currently the code in the notebook that loads a (post-training) checkpoint for the cGAN and goes through the test set generating the inferred OMAG-like OCTA images is pretty slow. It spawns a Dataset of length one for every B-scan file, to guarantee the order of predicted images and create the file so that it corresponds to that B-scan number. (Just using glob.glob and sticking the file results into a single Dataset of length ~1280 (full length) doesn't give that property.)

It shouldn't take too long, but requires some thinking to figure out how to avoid this inefficiency. This task is to time-optimize the process of generating the inferred OCTA files as much as possible, based on the metric of total running time of that snippet of code (from the start of loading the checkpoint until the end of the loop that generates all of the images.)

Links that may be partially useful:

• https://stackoverflow.com/questions/51162500/how-can-i-access-the-filenames-gathered-by-tf-data-dataset-list-files
• https://stackoverflow.com/questions/48896762/how-to-use-the-tensorflow-dataset-api-to-read-files-with-different-names-without
• Documentation for from_generator: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_generator

As part of this task, we can also look into whether we want to add any functionality to skip over N B-scans. For example, with sedated rat data where we have 4 acquisitions (B-scans) of the same cross section, the code should actually output N/4 (1280/4 = 320) inferred images, one for each different cross-section (every group of consecutive 4 inferred images should look almost exactly the same in this case, since it's the same cross section taken over a short period of time!) But for human data, the point is we'll end up only having 1 acquisition and want to infer blood flow, so there, we'll want all the B-scans run through the for loop. We should ask Zohreh if this should be configurable (with a default of 1) or if this isn't worth spending time on.

Add images with:

• Rotations
• Noise (this is important)

Look for automated ways to generate these as much as possible. We can explore first just applying random jitter to each image (like the way we're currently doing it, but we don't have rotations yet) and mutating it, and also keeping the original image but adding different versions of each one with random jitter applied to each so that we have a richer training set out of the existing one (i.e. more training images).

The place you want to be adding your code is in this function:

We want to do this as soon as possible and then kick off a job doing training with as many epochs as possible. Number of epochs is controlled in the constants file:

If you just want to verify that your code runs, you can set EPOCHS = 1 temporarily to just run one round. In fact, to verify the rotations are working, you could modify the code in:

We should just use num_epochs instead of both a starting_epoch and an ending_epoch in training mode for the cgan.py CLI. It's not clear to me that specifying a starting_epoch other than 1 would even log consistently and work the way you expect it to. If we want that functionality we really should be auto-detecting the last epoch we left off at from the training checkpoints folder passed in under -c.

This should be a fairly straightforward change 🙏🏻 (famous last words?)
Other than updating this with argparse, remember to also update run_job.sh and any other relevant scripts or documentation to account for the fact that the starting epoch is no longer supplied.

Octave?

Let's add a Markdown file to the repo, or documentation that lives somewhere else, that lists submission deadlines and conference dates for upcoming conferences and events we should target. This involves sending a follow-up e-mail to Zohreh and Dida clarifying expectations and possibly double-checking with Patrick (if we don't already know) if it's okay to look at conferences that are after Symposium Day but will let us know if our paper is accepted before then (IIRC this is fine, paper accepted is the bar, not paper presented).

Since the resulting OMAGs get combined into an en-face image, it follows that the capillary cross-sections should smoothly connect, so the position of the capillaries in adjacent B-scans could provide insight into the correct position and scaling of the capillaries in the current B-scan.

Consider looking into techniques like the following paper uses to ensure "temporal smoothing" (ensuring that generated video frames flow smoothly into each other).

https://arxiv.org/pdf/1808.07371.pdf

This issue additionally requires:

• Clean up the code to parameterize the number of input channels
• Figure out #21 to determine whether the results are an improvement

Stretch our current pix2pix cGAN baseline in a few different directions. This will involve:

1. Learning more (and hopefully documenting your learning) about how our current neural net works. (Explaining in terms of some of the block diagrams, other diagrams, etc. included in the pix2pix tutorial may help you structure this documentation.)
2. Changing high-level things like learning rate of the Adam optimizer (e.g. 5e-4 vs. 2e-4) and seeing how that changes training running time and quality of our results. I don't expect any changes deep within the layers at this stage but just getting a feel for some of the things that are affecting the quality of our results at a higher level as much as possible would be great.

Feel free to immediately close this if you feel this is already done to an acceptable level of completion.

For each of the scripts in this repo's /scripts folder, we should:

1. finalize whether they're meant to be run as is or via the GUI

2. document them in the manual, including whether they're meant to be a pre- or post-processing step, if applicable, and whether they need to be run on a GPU machine like Sharcnet

We want to define a common "front-end" interface to our "back-end" so that Zohreh will eventually be able to run our algorithm more easily in her lab. This will also help everyone in the group better understand the high-level inputs, outputs, processing, and storage of our project in case we're not all on the same page. The software should have two modes:

1. Training the neural nets.
2. Load a checkpoint (after training is done) and invoke the entire end-to-end process of generating all the cross-sections for a set of B-scans and combining them (calling out to the MATLAB script @pl3li will be toying around with) to get an en-face result.

Mode 2 will be the most common use case, but 1 will be necessary for us to repeat the training as we tune things.

Setup meeting with Alex Wong to go over ML part (important) and make sure that we have enough substance to earn a publication.

• schedule a meeting (if so, when)?
• degree of contribution (i.e. authorship)?

According to https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shuffle, the buffer needs to be at least equal to the dataset size for perfect shuffling.
Since we have ~63*512 elements in the dataset, the buffer needs to be increased.

We have one human test dataset:

(For some reason, it's in the Dropbox, not the school OneDrive we're trying to converge to).

This task is to:

1. (Ideally): redo training on eceUbuntu4 and get a new set of training checkpoints with more test data (since we now have way more sedated rat images on that machine thanks to @SamiSab)
2. Generate a set of inferred cross-sections from the human retina B-scans.
3. Use @pl3li's documented enface script process to run MATLAB and generate an en-face (C-scan) image from the results of Step 2 above.
4. Visually compare it to the en-face produced from the OMAGs and inspect how similar they are visually.
5. Ask Zohreh what she thinks (e-mail her the side-by-side comparison and give her access to the raw folders that went into the above steps as well)
6. Document the results from Step 4 and Step 5 above.

There are a number [0] [1] [2] of out-of-the-box scripts that do pix2pix cGAN training and can get you results even simpler than what we did (type out the tutorial code). See if you can download some of these packages (say, 1 or 2 of them), write a wrapper script (if necessary) to massage our data to feed into the scripts appropriately, and see what the output looks like. We can just use visual inspection for now to see if the capillaries (and resulting en-face from running the MATLAB post-processing) are as visible, better, or worse than the tutorial code.

It would be super helpful if you could also read through these repos' READMEs to get a sense of whether their neural nets look different (substantially different maybe?) from the ones in the diagram of the pix2pix tutorial [3]. Write this up in a doc somewhere, and take your time (and ask me questions) if what I'm asking you to do is unclear.

[0] https://github.com/affinelayer/pix2pix-tensorflow
[1] https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix
[2] https://github.com/phillipi/pix2pix
[3] https://www.tensorflow.org/beta/tutorials/generative/pix2pix

Dida gave us a lot of information about how abstracts, mini-abstracts, papers, and collaboration works in general. Let's set up templates (without actually filling in any sections) for the image abstract and text abstract that Dida requested. I know this is premature, and I agree some faculty stakeholders might be a tad over-excited about our early progress. However, we'll need to set up these templates regardless unless we hit a brick wall, so let's front-load that work. Dida wants us to use word instead of latex.

Again, this task does not include actually typing up the entire abstract or paper or anything, but it's about being ready to jump into that once we're ready.

Whether it's edge sharpness, signal-to-noise ratio, or something else, crawl around prior art (existing papers that we have and also have not read) to get one or more formulas we can use to objectively score our results (Zohreh, Dida, and Alex should be able to help). This task is complete when we have run the formula/equation you come up with against OMAG and one set of inferred results from our work and compared them to make sure the loss calculation makes sense.

Once that's done please create a subsequent follow-up GitHub Issue to track the work of incorporating that formula into our loss function like other papers have done, to see if that improves our results. Other follow-up tasks can include automating the process of running the scoring formula against an inferred image set so we can quickly throw numbers into a paper.

Wait until closer to Thursday to finish this please <3

At minimum:

• Dropbox data moved -> OneDrive student data folder. 80/20 split of training/test sets.
• Google Docs instead of Dropbox Paper?

This task is to sort out a few details that will affect how we write https://docs.google.com/document/d/1Pv8ePSXc_DDfyJrNQ4N-2b45PCjYM3s1kmEAAX2D644/edit#heading=h.gjdgxs

• Figure out what (the formal name of the) department we're in is. We need this information no matter what paper we write. See https://paper.dropbox.com/doc/Email-Draft-Queue--Ag1jUrhCauU1enitbp~ecJCTAg-64ZTdPQAC4qoDq1fHwIYJ
• @pl3li's words, not mine: "Authorship priority …. need agreement from team then send email to Dida (postpone as late as possible)" (emphasis mine)

Actually read the paper that Zohreh e-mailed us carefully (the one that Céline is using to establish a 2nd baseline) and add a summary/analysis of it to our literature report.

Working with @savoie here might be a good idea since she's reading it too to set up a 2nd baseline (but this doesn't block her, since I'm reading more to get a sense for how it might fit in as a citation and how their work relates to ours rather than homing in on the method specifically, which @savoie will be doing.)

See #35 for context.
Possible approach might involve the CheckpointManager (see https://www.tensorflow.org/api_docs/python/tf/train/CheckpointManager).

We decided it might be useful if each person had their own EC2 instance to play around with. Link is https://docs.google.com/forms/d/e/1FAIpQLSco5DKP1MXL-NM1T3FlZNTFbv_seRpjSZr0aoKfr8K15QFh3g/viewform

Currently, ModelState.reset_weights() resets only the discriminator and generator weights.
We should also be resetting the discriminator optimizer and the generator optimizer.