The current naming suggests that any Vision-related task is included. However, the Library is specific to Images. Therefore the library should be renamed to ConfigILM - including all references for vision within.
If the library implements some benchmark models using the configurations, it would be easier for end users to fast test against the existing methods. In addition these benchmarks would show as examples how the library could be used.
The current Scripts use WandB for logging. However, this requires a WandB account and Internet access to work properly. To showcase how to use the framework, logging is not necessary.
The RSVQALR dataset cannot be loaded because there is a KeyError in _get_question_answers when loading the answers. The problem here is, that inactive answers are still part of the answer list before they are filtered and therefore the question_id key is accessed for inactive elements which don't have this attribute
The current version uses a pytorch/huggingface (or maybe other dependencies) combination that is very slow, as shown in this and this issue.
The current workaround would be to use version 0.3.0 and manually update the packages (e.g., timm) after installing ConfigILM until a fix is given in the dependencies or otherwise known.
A documentation of the api including the descriptions of the docstrings should be added in the guide to look up e.g. parameter names for specific parts of the library.
In BENLMDBReader there should be an option that allows to omit specified classes in the returned ground truth label vectors (and reduce the ground truth vectors accordingly). This can be helpful when working with subsets of BigEarthNet that do not contain all the classes.
The transformations in RSVQAxBEN and BEN Datamodules throw user warnings due to changed default behavior in torchvision 0.17. This can be mitigated using anitalias=true in all Compose calls.
The parameters load_hf_if_available and load_timm_if_available of the configilm.ILMConfiguration class are switches for loading a pretrained huggingface or timm model according to this docstring. Their purpose could be made clearer and more explicit by renaming them to something like load_pretrained_hf_if_available and load_pretrained_timm_if_available respectively.
This is actually done in a less user-facing function here.
The provided Datamodules do not expose a way of changing the number of classes in the datasets. Therefore datamodules are not usable when a different number of classes is required or wanted
The datamodules contain a lot of redundant code. To make this more flexible and less duplicate, an abstract class (based on Lightning datamodule) should be implemented that all datamodules inherit from.
Also Tests should use this class to test all basic functionality and only the individual additional functionality should be in the individual test classes
Some existing fusion methods could be added for simplicity such that more model combinations can be directly applied without having to re-implement the fusion functions.
The current fusion implementation only supports fusion types with the same input and output dimensions. However, there are some fusion types - e.g. MUTAN like in this implementation where the fusion output dimension can be different than the input dimension.
The current state of the library cannot be published as there are to many examples in the mock data. Only up to 100MB can be published, therefore the number of examples in the mock data has to be reduces significantly
The datasets as implemented use every answer as its own class. However, in the paper the datasets use quantization (II.B, page 5) so that e.g. are buckets and number buckets are created, which results in drastically fewer answer classes. A flag should be added to the datasets to allow the same quantization approach
The documentation seems to explain features only by example so far. I would like to see a more abstract documentation page which answers questions like:
What are the main user-facing classes of this library? (Answer as far as I understand: ILMConfiguration and ConfigILM)
The logos in the Documentation are not displayed. Instead, the alt text is shown. This is expected, as relative paths are not trivial in static HTML. https://pradyunsg.me/furo/customisation/footer/
The current state of dependencies always installs everything. This should not be the case and some dependencies should be optional - therefore the name extra for the subpackage
The current state of the example scripts only works for classification. The scripts should be divided into one example for classification and one for VQA.
For visualization of image retrieval results using BEN, it is necessary to assign patch-names to samples returned from the BENDataSet.__getitem__() method. A convenient extension to this class would therefore be to (optionally) return the patch-names (key) in addition to image and labels.