nki-ai / direct Goto Github PK

Progress issue for implementing a RIM baseline for the FastMRI multicoil challenge. This does require coil sensitivity estimates, as the baseline RIM is too sensitivity if these are not provided. While these sensitivities can typically be measured, they are not provided in the FastMRI dataset.

Several ways to do this:

• Estimate coils in the network
• Estimate coils externally, e.g. with the bart toolkit.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/tools/val_rim.py#L65-L70

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/a966c485d8bc497f3801e6ce91b7ab044e35d591/direct/environment.py#L271-L276

This issue was generated by todo based on a TODO comment in a966c48. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/data/datasets.py#L67-L72

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/ab2ca8db55fae871d9bd0ac39a1e67afd7fa69a7/direct/checkpointer.py#L110-L115

This issue was generated by todo based on a TODO comment in ab2ca8d. It's been assigned to @jonasteuwen because they committed the code.

Normalization should be done on the basis of the ACS signal, as this is the only part We can ensure is there.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/data/mri_transforms.py#L416-L421

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

Learning rate does not seem to decrease at milestones. A few options would be to change the LR scheduler to a configurable object as other options also would be a feasible solution.

Current loss and metrics are hardcoded in rim_engine.py, mainly because it is unclear what would be the proper way of using this: for instance, we might want to have L1 + alpha * SSIM loss, where alpha is configurable, and we would want both the L1 and SSIM loss logged.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/data/datasets.py#L66-L71

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/checkpointer.py#L119-L124

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

When having several validation sets, the validation metrics get logged correct, but the logging of the images fails. Likely only the last dataset is displayed.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/data/mri_transforms.py#L442-L447

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

In v0.1.2 the validation is done in batches per volume name, likely requiring simpler logic in the evaluation function in the engines.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/tools/train_rim.py#L108-L113

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/nn/rim/rim_engine.py#L427-L432

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/tools/train_rim.py#L85-L90

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/a966c485d8bc497f3801e6ce91b7ab044e35d591/direct/environment.py#L118-L121

This issue was generated by todo based on a TODO comment in a966c48. It's been assigned to @jonasteuwen because they committed the code.

This will be handled once Hydra is implemented.

https://github.com/directgroup/direct/blob/d951fcaa291cfe8756b5506d6d96e15fd7044cd8/direct/environment.py#L249-L254

This issue was generated by todo based on a TODO comment in d951fca. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/nn/rim/rim_engine.py#L389-L394

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/nn/rim/rim_engine.py#L390-L395

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

Using hydra rather than OmegaConf will make composition of configurations much more convenient and allow easy access to a lot of hyperparameter search libraries such as Ax Sweeper, nevergrad, and several launcher systems.

For now, this is scheduled for version 1.0 as currently, the configuration system is reasonably stable. Adding more reconstruction algorithms and fixing the open issues is a priority for now.

Additional models are supported, but not logged, a generic method should be setup such that we can set:

tensorboard:
    log_images:
        - sensitivity_map

and these are subsequently logged to tensorboard (or anywhere).

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/nn/rim/rim_engine.py#L388-L393

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/data/mri_transforms.py#L185-L190

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/data/h5_data.py#L64-L68

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

A general strategy for seeds should be devised.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/data/mri_transforms.py#L605-L610

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/data/h5_data.py#L149-L154

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

When a model is initialized from a previous checkpoint it would be useful to immediately have a validation result

Create a serialize_dict function, or build the write_json to serialize tensors.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/nn/rim/rim_engine.py#L380-L383

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/data/mri_transforms.py#L454-L459

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/engine.py#L140-L145

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/tools/train_rim.py#L210-L215

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

When working with a lot of configurations it is convenient to get the complete config in a human-readable format in the project folder, which can be checked against the parameters given by the config itself.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/nn/rim/rim_engine.py#L379-L383

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/nn/rim/rim_engine.py#L122-L127

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

Currently, the training and validation metrics are treated differently and are logged with a median window - this is clearly something we want to avoid.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/nn/rim/rim_engine.py#L508-L513

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

When validating the whole dataset is properly passed, and the sampling indices are properly passed, but nevertheless less volumes end up in the output metrics than what has been passed through the model.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/nn/rim/rim_engine.py#L393-L398

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/nn/rim/rim_engine.py#L422-L427

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

The accelerations argument should be parsed correctly to the build_masking_function

For now, the acc factor will be selected automatically from the cfg file and set to the first option used for validation.

Two ways to fix it:

1. Load it from the cfg file in a proper way and remove the argument
2. Parse the argument option and remove loading from the file

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/tools/train_rim.py#L107-L112

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

When the evaluation fails with an uncaught exception the resuming will happen after the exception, it would be more convenient and consistent if resuming would immediately trigger evaluation.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/engine.py#L321-L326

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/nn/rim/rim_engine.py#L146-L149

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/checkpointer.py#L199-L201

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

Goal of the library should be to also provide a reproducible baseline for the other MRI reconstruction models on public datasets. This should include, but is not limited to:

• Learned Primal Dual
• U-net postprocessing
• VNet

cc @wdika: any additions?

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/data/datasets.py#L326-L331

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.

Current travis build fails, the code however should work.

https://github.com/directgroup/direct/blob/9cb5f43e877b2e9ae3aaa305a7915f4117f976ac/direct/data/datasets.py#L65-L70

This issue was generated by todo based on a TODO comment in 9cb5f43. It's been assigned to @jonasteuwen because they committed the code.