ritaramo / smallcap Goto Github PK

Hi,
I have a problem downloading stanford-corenlp file.

In coco-caption/get_stanford_models.sh,

CORENLP=stanford-corenlp-full-2015-12-09
wget http://nlp.stanford.edu/software/$CORENLP.zip

and I also try
wget http://nlp.stanford.edu/software/$CORENLP.zip --no-check-certificate

But I got this in linux
Unable to establish SSL connection.

I couldn't even access the URL and homepage..
https://stanfordnlp.github.io/CoreNLP/download.html

I would appreciate it if you could tell me what the problem is or send me the file.
Thank you

1

there is not such file in feature directory, which just has train.hdf5 and val.hdf5.

How to create my own datastore

I was reproducing the experiment accoding to README, and everything was ok until entering "retrieve captions". After images were encoded, it started to retrieve neighors. But the process seemed like frozen, nothing happened for like 2-3 hours. So I'm wondering if it is expected? How long is "retrieving neighors" gonna be?

Hey! We are doing research following your work. We have reproduced your work over coco dataset which performs very well, even better than your provided data! But when it comes to other datasets you referred in your paper, like Vizwiz, MSR-VTT, we find it very complicated to get the corresponding metrics results. So I wonder if you could provide the code for these experiments?

In addition, is the datastore/coco_index_captions.json the augmented datastore you mentioned in 5.2 Augmenting the datastore?

Error in SmallCap_demo.ipynb

NameError Traceback (most recent call last)
Cell In [6], line 10
7 with torch.no_grad():
8 image_embedding = retrieval_model.encode_image(pixel_values_retrieval.unsqueeze(0)).cpu().numpy()
---> 10 nns = retrieve_caps(image_embedding, retrieval_index)[0]
11 caps = [captions[i] for i in nns][:4]
13 # prepare prompt

NameError: name 'retrieve_caps' is not defined

there is no VisionEncoderDecoderModel in vision_encoder_decoder.py. when I read the file. I change it to "from transformers import VisionEncoderDecoderModel" in train.py . but there is another error.

thanks!

parser.add_argument("--decoder_name", type=str, default="opt", help="Decoder name as found of HuggingFace or stored locally").
error:OSError: opt is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token or log in with huggingface-cli login and pass use_auth_token=True.

sorry if this is caused by stupid reasons, I am new to Java language
I encountered the following problem when commanding 'python /smallcap/caption/dramaEvalCap.py'

tokenization...
PTBTokenizer tokenized 23639 tokens at 411834.08 tokens per second.
PTBTokenizer tokenized 47091 tokens at 630900.25 tokens per second.
setting up scorers...
computing Bleu score...
{'testlen': 44550, 'reflen': 21080, 'guess': [44550, 42008, 39466, 36925], 'correct': [5332, 1242, 258, 99]}
ratio:2.113378
Bleu_1: 11.97
Bleu_2: 5.95
Bleu_3: 2.85
Bleu_4: 1.58
computing METEOR score...
METEOR: 9.98
computing CIDEr score...
CIDEr: 11.78
computing SPICE score...
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Stopwatch
at edu.anu.spice.SpiceScorer.scoreBatch(SpiceScorer.java:69)
at edu.anu.spice.SpiceScorer.main(SpiceScorer.java:60)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Stopwatch
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 2 more
Traceback (most recent call last):
File "/home/lufan/Projects/smallcap/caption/dramaEvalCap.py", line 38, in
dramaEval.evaluate()
File "/home/lufan/Projects/smallcap/caption/pycocoevalcap/eval_drama.py", line 56, in evaluate
score, scores = scorer.compute_score(gts, res)
File "/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/spice.py", line 70, in compute_score
subprocess.check_call(spice_cmd,
File "/home/lufan/miniconda3/envs/smallcap/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/lufan/java/jdk1.8.0_371/bin/java', '-jar', '-Xmx8G', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/spice-1.0.jar', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/tmp/tmpiai6r6_r', '-out', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/tmp/tmpxbfagpuw', '-subset', '-silent']' returned non-zero exit status 1.

It seemed that guava.jar is not recognized.
Would you help, please?

Thanks for great paper!

I am curious how to get hidden_state value?

and why it has more than double dim size up? I thought it would be (b,50, 768), not a (b,140,768)

Could you explain why is it?

Thanks!!!
gpt2.py 87lines

hidden_states: Optional[Tuple[torch.FloatTensor]]

https://github.com/RitaRamo/smallcap/blob/513f4f795950328129014eb37f011d686ab6ed24/src/gpt2.py#L87C13-L87C13

It seems to me that there is only code for training. I want to follow your work to do some research. Now I am trying to reproduce your work. But I didn't find code for evaluation.

line 50
feature_extractor = CLIPFeatureExtractor.from_pretrained(args.encoder_name)

line 144
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
data_collator=default_data_collator,
train_dataset=train_dataset,
tokenizer=feature_extractor,
)

toknizer parameter in Seq2SeqTrainer, you use a feature_extractor which is for image processing(CLIP), not text toknizer like BERT, GPT

can you explain why you use CLIPFeatureExtractor for tokenizer in Seq2SeqTrainer?

Thanks for your excellent work, I just wonder when infering on COCO using COCO captions as datastore to prompt, the prompt may include the ground-truth captions, which may lead the data-leakage question. Have you adopted some strategies to avoid this question?