lhackel-tub / configilm Goto Github PK

ILMTypes are currently called Vision... instead of Image... even tho everything else is called Image...
They should be renamed

There are a lot of tests and they are very small. Optimizing the tests could help development speed

The current naming suggests that any Vision-related task is included. However, the Library is specific to Images. Therefore the library should be renamed to ConfigILM - including all references for vision within.

The example written for google colab is not working as the package cannot be installed

If the library implements some benchmark models using the configurations, it would be easier for end users to fast test against the existing methods. In addition these benchmarks would show as examples how the library could be used.

Possible starting points could be:

• Language Transformers for Remote Sensing Visual Question Answering
• LiT-4-RSVQA: Lightweight Transformer-based Visual Question Answering in Remote Sensing

The current Scripts use WandB for logging. However, this requires a WandB account and Internet access to work properly. To showcase how to use the framework, logging is not necessary.

Documentation (docstrings) should be extended/created for

• ConfigILM.get_timm_model

• ConfigILM.ILMType

• ConfigILM.ILMConfiguration

• ConfigILM.ConfigILM.init

• ConfigILM.ConfigILM.init

• ConfigILM.ConfigILM._check_input

• ConfigILM.ConfigILM.forward

• ConfigILM.util.huggingface_tokenize_and_pad

• ConfigILM.extra.BEN_lmdb_utils (all)

• ConfigILM.extra.CustomTorchClasses (all)

• ConfigILM.extra.RSVQAxBEN_DataModule_LMDB_Encoder (all)

• ConfigILM.extra.BEN_DataModule_LMDB_Encoder (all)

Currently, the example scripts are only written in pytorch lightning. Add example scripts (one per use case) in pure pytorch as well.

The RSVQALR dataset cannot be loaded because there is a KeyError in _get_question_answers when loading the answers. The problem here is, that inactive answers are still part of the answer list before they are filtered and therefore the question_id key is accessed for inactive elements which don't have this attribute

The current version uses a pytorch/huggingface (or maybe other dependencies) combination that is very slow, as shown in this and this issue.

The current workaround would be to use version 0.3.0 and manually update the packages (e.g., timm) after installing ConfigILM until a fix is given in the dependencies or otherwise known.

psutil is not a requirement when installing, however it is used in RSVQAxBEN Data module. The import fails when not installed additionally

A documentation of the api including the descriptions of the docstrings should be added in the guide to look up e.g. parameter names for specific parts of the library.

In BENLMDBReader there should be an option that allows to omit specified classes in the returned ground truth label vectors (and reduce the ground truth vectors accordingly). This can be helpful when working with subsets of BigEarthNet that do not contain all the classes.

The transformations in RSVQAxBEN and BEN Datamodules throw user warnings due to changed default behavior in torchvision 0.17. This can be mitigated using anitalias=true in all Compose calls.

The parameters load_hf_if_available and load_timm_if_available of the configilm.ILMConfiguration class are switches for loading a pretrained huggingface or timm model according to this docstring. Their purpose could be made clearer and more explicit by renaming them to something like load_pretrained_hf_if_available and load_pretrained_timm_if_available respectively.

This is actually done in a less user-facing function here.

Supporting only 3.10 and up seems very restrictive. Maybe one could increase this threshold and also test against different versions?

The provided Datamodules do not expose a way of changing the number of classes in the datasets. Therefore datamodules are not usable when a different number of classes is required or wanted

The datamodules contain a lot of redundant code. To make this more flexible and less duplicate, an abstract class (based on Lightning datamodule) should be implemented that all datamodules inherit from.
Also Tests should use this class to test all basic functionality and only the individual additional functionality should be in the individual test classes

Some existing fusion methods could be added for simplicity such that more model combinations can be directly applied without having to re-implement the fusion functions.

Possible candidates:

• MUTAN
• MIDF (implementation)

The current fusion implementation only supports fusion types with the same input and output dimensions. However, there are some fusion types - e.g. MUTAN like in this implementation where the fusion output dimension can be different than the input dimension.

Right now, the library uses pytorch_lightning, which was renamed to lightning. To reflect this change, the dependency should be updated.

Dataset HRVQA was released and could be a nice addition.

The current state of the library cannot be published as there are to many examples in the mock data. Only up to 100MB can be published, therefore the number of examples in the mock data has to be reduces significantly

The datasets as implemented use every answer as its own class. However, in the paper the datasets use quantization (II.B, page 5) so that e.g. are buckets and number buckets are created, which results in drastically fewer answer classes. A flag should be added to the datasets to allow the same quantization approach

The documentation seems to explain features only by example so far. I would like to see a more abstract documentation page which answers questions like:

• What are the main user-facing classes of this library? (Answer as far as I understand: ILMConfiguration and ConfigILM)
• What are the roles of these classes?

The logos in the Documentation are not displayed. Instead, the alt text is shown. This is expected, as relative paths are not trivial in static HTML.
https://pradyunsg.me/furo/customisation/footer/

The folder example_scripts should be called only scripts and the scripts from baselines folder should also be moved to scripts

The current state of dependencies always installs everything. This should not be the case and some dependencies should be optional - therefore the name extra for the subpackage

The current state of the example scripts only works for classification. The scripts should be divided into one example for classification and one for VQA.

Currently the original RSVQA data sets are not included. As they are publicly available, they should be added.

The function should use the libraries function to assert the same functionality.

In File: extra/BEN_lmdb_utils.py:117

For visualization of image retrieval results using BEN, it is necessary to assign patch-names to samples returned from the BENDataSet.__getitem__() method. A convenient extension to this class would therefore be to (optionally) return the patch-names (key) in addition to image and labels.