How to generate class embedding files? about zsd-sc-resolver

[sandipan211]

Hi @GuangyuanLiu1999 ,

Thanks for showing interest in our work. The class embedding files can be:

1. taken directly as a reference from previous repositories that provided them, like our repository, PL-ZSD, etc.
2. generated yourself by downloading pretrained word2vec/glove/fasttext (whichever embedding you need) embeddings and obtaining the specific vectors for each object class following this tutorial.

• Note that in both these cases, there might be differences in the vector elements you obtain, as some people apply different vector normalizations before using them as semantic vectors.
• Moreover, to follow our work you need to have semantic vector of a "background" class which can be obtained from our given fasttext.npy. You might obtain that semantic vector even yourself, for example, by averaging the vectors of all object classes[1], or by assigning a fixed vector[2,3]. However, performance might not be the same.

To answer the indexing issue, before the training of our cWGAN starts, in trainer.py, we first load the class embedding matrix, for which utils can be found at generate.py. The mapping of classes with labels/ids can be found at splits.py.

[1] Rahman, Shafin, Salman Khan, and Fatih Porikli. "Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts." Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part I 14. Springer International Publishing, 2019.
[2] Bansal, Ankan, et al. "Zero-shot object detection." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
[3] Zhu, Pengkai, Hanxiao Wang, and Venkatesh Saligrama. "Don't even look once: Synthesizing features for zero-shot detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[GuangyuanLiu1999]

Hi @GuangyuanLiu1999 ,

Thanks for showing interest in our work. The class embedding files can be:

1. taken directly as a reference from previous repositories that provided them, like our repository, PL-ZSD, etc.
2. generated yourself by downloading pretrained word2vec/glove/fasttext (whichever embedding you need) embeddings and obtaining the specific vectors for each object class following this tutorial.

• Note that in both these cases, there might be differences in the vector elements you obtain, as some people apply different vector normalizations before using them as semantic vectors.
• Moreover, to follow our work you need to have semantic vector of a "background" class which can be obtained from our given fasttext.npy. You might obtain that semantic vector even yourself, for example, by averaging the vectors of all object classes[1], or by assigning a fixed vector[2,3]. However, performance might not be the same.

To answer the indexing issue, before the training of our cWGAN starts, in trainer.py, we first load the class embedding matrix, for which utils can be found at generate.py. The mapping of classes with labels/ids can be found at splits.py.

[1] Rahman, Shafin, Salman Khan, and Fatih Porikli. "Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts." Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part I 14. Springer International Publishing, 2019. [2] Bansal, Ankan, et al. "Zero-shot object detection." Proceedings of the European Conference on Computer Vision (ECCV). 2018. [3] Zhu, Pengkai, Hanxiao Wang, and Venkatesh Saligrama. "Don't even look once: Synthesizing features for zero-shot detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

Thank you very much for your serious reply. I will seriously follow your prompts to do experiments, and I would like to express my heartfelt thanks to you again.