welthungerhilfe / cgm-rg Goto Github PK

Problem: The cgm-rg code is hard to understand

Potential Solution: Type annotations will help to understand more about the objects we pass back and forth

img = tf.image.decode_jpeg(img, channels=3)
Standing-laying classier perform wrong due to problem with the above line

Problem: This results in a docker image of 7.85GB. This is difficult for working on developer laptops limited resources

To reproduce, run cgm-rg steps in https://github.com/Welthungerhilfe/cgm-rg/wiki/Setup-local-development

Solution:

• combine layers
• remove cache after apt-get
• remove cache after pip install
• https://medium.com/@gdiener/how-to-build-a-smaller-docker-image-76779e18d48a

Other problems:

• docker build takes 12-15minutes with fast internet (gist)
• at the moment pytorch gets installed with requirements.txt, then uninstalled and reinstalled when running: RUN pip install torch==1.9.0+cpu torchvision==0.10.0+cpu torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

Tasks:

• Provide documentation for setting up this local environment
• mount code instead of copying it avoid rebuild of container

cgm-rg is currently quite slow when performing the face detection.

When inserting a batch of old scans, it needs weeks to perform result generation.

We can benchmark and profile and also think about alternative architectures for cgm-rg (e.g. scalable AzureML endpoints)

Problem: Environment creation code is present in many Training notebooks.

Solution: Create Utils and reuse the code

Code duplication is present in Pose 3d model code.
Pose prediction is running again for 3d- model code.

• remove code duplication between workflows
• use pathlib instead of os
• make code more readable
• TODOs in the code
• result_gen_with_api.py has very long blocks: split it into fcts

Motivation: Verify (e.g. in the ACI logs) that the scan was processed correctly

Status Quo: Currently there are only logs from the cgm-rg initialization (how workflows got loaded)

Status quo: In preprocessing.py, we set width and height for the whole program.

Problem: The program execution might handle artifacts with different height and width.

Solution: We could therefore store width and height as state in an object of class Artifact.

Here is the idea with some pseudo-code:

class Artifact:
    def __init__(self, filename):
        self.filename = filename
        self.width, self.height, depthmap = load_depth(filename)

file_ = Aritvact('4ed427b5-3fd9-4f4d-8e58-19e39c7d77b6.zip')
file_.width  # instead of using global WIDTH

Motivation: We currently don't plan to put the ensembles into cgm-rg in the near future. If we decide to add them back, this might look very different by then.

Desired changes:

• e.g. ENSEMBLE_RUN_IDS in src/download_model.py
• Check if there are more places