How is trained classification model used in cropping model? about cacnet-pytorch

[cip8]

Hi 👋!
This is not very clear to me either, but from what I understand, best-acc.pth is the checkpoint for the composition model only. You can use this as a stand-alone model for direct inference on your photos.

The cropping model has the composition part already included: CACNet class has a self.composition_module that is called during training. Thus, the pre-trained composition model (best-acc.pth) does not need to be called directly, but its architecture will still be used.

It would be indeed great if @bo-zhang-cs could clarify this for us ❤️ 🖖

[bo-zhang-cs]

Thanks for your interest in my implementation!

You are right @cip8 👍 The best-acc.pth generated by running python train_composition_classification.py is the checkpoint for the composition model only.

Running python train_image_cropping.py means training the cropping and composition branches simultaneously on the corresponding datasets, which is suggested by the paper's author in Section 3.4 of the paper. However, I found the multi-task training strategy leads to significantly inferior results on composition classification compared to training the classification branch alone. So if you're only interested in the cropping task, skip python train_composition_classification.py and execute python train_image_cropping.py directly.

[bo-zhang-cs]

Hi, @cip8 , I have uploaded the pretrained classification model best-acc.pth to here.

[bo-zhang-cs]

Q: So to get the best accuracy (88.4%) you recommend to train them separately and then do the inference directly on the best-acc.pth checkpoint ?

A: Actually, if you are only interest in the cropping results, you could skip training composition classification and directly train the cropping model, which will generate best-FLMS_iou.pth. After that, you can do inference by running python test.py. Correspondingly, if you want the best classification accuracy (88.4%), I suggest to directly train the classifion model and evaluate on the best-acc.pth.

Q: Would it be possible to skip the training of the comp branch here entirely & infer directly on the best-acc model during crop training?

A: I'm afraid that doing so may affect the cropping results. In fact, I have also tried training the cropping branch with fixing the parameters of classification branch to best-acc.pth, yet resulting in an unsatisfactory results. So to achieve the classification and cropping performance at the same time (as in the paper), it's best to consult the paper's author.

I really hope this works for you !

[bo-zhang-cs]

@khushboo-anand0909 I'm afraid you cannot get such score. The CACNet can only generate the prediction of composition category through the classification branch (see their paper).

[cip8]

Thanks for the extra info @bo-zhang-cs, it's really helpful 👍

Indeed, when training both branches at the same time I noticed the composition accuracy peaks at around ±80%.

So to get the best accuracy (88.4%) you recommend to train them separately and then do the inference directly on the best-acc.pth checkpoint?

In this case, would it be possible to skip the training of the comp branch here entirely & infer directly on the best-acc model during crop training?

Would it be possible to also add a download link for the pre-trained composition model @ 88.4%?
That would be greatly appreciated - thanks again!

[khushboo-anand0909]

Thanks for the clarification @bo-zhang-cs . I have one more query, is it possible to get some score of an image as well? I am able to get the crops on test images and on custom single images but I also wish to get some quality score or composition score of an image. Is it possible to get any score via your approach? If yes, please share the steps/script? Thanks.

[cip8]

I have one more query, is it possible to get some score of an image as well? [...] I also wish to get some quality score or composition score of an image.

@khushboo-anand0909, for that you can take a look at the Awesome Image Aesthetic Assessment and Cropping repo.

Unfortunately most papers in that compilation have no code implementation, but you might still find something useful.

For example, you could chain this code with an aesthetic evaluation model (like this one) and have it output a score for how good the auto-cropped image looks.

I have uploaded the pretrained classification model best-acc.pth to here.

Following your code & the corresponding paper has been an outstanding source of learning for a rookie like me.

So, again, a big thank you for maintaining this open-source project & lighting the flame of knowledge in others. ❤️ 🖖