ritaramo / smallcap Goto Github PK

sorry if this is caused by stupid reasons, I am new to Java language
I encountered the following problem when commanding 'python /smallcap/caption/dramaEvalCap.py'

tokenization...
PTBTokenizer tokenized 23639 tokens at 411834.08 tokens per second.
PTBTokenizer tokenized 47091 tokens at 630900.25 tokens per second.
setting up scorers...
computing Bleu score...
{'testlen': 44550, 'reflen': 21080, 'guess': [44550, 42008, 39466, 36925], 'correct': [5332, 1242, 258, 99]}
ratio:2.113378
Bleu_1: 11.97
Bleu_2: 5.95
Bleu_3: 2.85
Bleu_4: 1.58
computing METEOR score...
METEOR: 9.98
computing CIDEr score...
CIDEr: 11.78
computing SPICE score...
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Stopwatch
at edu.anu.spice.SpiceScorer.scoreBatch(SpiceScorer.java:69)
at edu.anu.spice.SpiceScorer.main(SpiceScorer.java:60)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Stopwatch
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 2 more
Traceback (most recent call last):
File "/home/lufan/Projects/smallcap/caption/dramaEvalCap.py", line 38, in
dramaEval.evaluate()
File "/home/lufan/Projects/smallcap/caption/pycocoevalcap/eval_drama.py", line 56, in evaluate
score, scores = scorer.compute_score(gts, res)
File "/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/spice.py", line 70, in compute_score
subprocess.check_call(spice_cmd,
File "/home/lufan/miniconda3/envs/smallcap/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/lufan/java/jdk1.8.0_371/bin/java', '-jar', '-Xmx8G', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/spice-1.0.jar', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/tmp/tmpiai6r6_r', '-out', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/tmp/tmpxbfagpuw', '-subset', '-silent']' returned non-zero exit status 1.

It seemed that guava.jar is not recognized.
Would you help, please?

How to create my own datastore

1

I was reproducing the experiment accoding to README, and everything was ok until entering "retrieve captions". After images were encoded, it started to retrieve neighors. But the process seemed like frozen, nothing happened for like 2-3 hours. So I'm wondering if it is expected? How long is "retrieving neighors" gonna be?

Hey! We are doing research following your work. We have reproduced your work over coco dataset which performs very well, even better than your provided data! But when it comes to other datasets you referred in your paper, like Vizwiz, MSR-VTT, we find it very complicated to get the corresponding metrics results. So I wonder if you could provide the code for these experiments?

In addition, is the datastore/coco_index_captions.json the augmented datastore you mentioned in 5.2 Augmenting the datastore?

line 50
feature_extractor = CLIPFeatureExtractor.from_pretrained(args.encoder_name)

line 144
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
data_collator=default_data_collator,
train_dataset=train_dataset,
tokenizer=feature_extractor,
)

toknizer parameter in Seq2SeqTrainer, you use a feature_extractor which is for image processing(CLIP), not text toknizer like BERT, GPT

can you explain why you use CLIPFeatureExtractor for tokenizer in Seq2SeqTrainer?

Hi,
I have a problem downloading stanford-corenlp file.

In coco-caption/get_stanford_models.sh,

CORENLP=stanford-corenlp-full-2015-12-09
wget http://nlp.stanford.edu/software/$CORENLP.zip

and I also try
wget http://nlp.stanford.edu/software/$CORENLP.zip --no-check-certificate

But I got this in linux
Unable to establish SSL connection.

I couldn't even access the URL and homepage..
https://stanfordnlp.github.io/CoreNLP/download.html

I would appreciate it if you could tell me what the problem is or send me the file.
Thank you

Thanks for great paper!

I am curious how to get hidden_state value?

and why it has more than double dim size up? I thought it would be (b,50, 768), not a (b,140,768)

Could you explain why is it?

Thanks!!!
gpt2.py 87lines

hidden_states: Optional[Tuple[torch.FloatTensor]]

https://github.com/RitaRamo/smallcap/blob/513f4f795950328129014eb37f011d686ab6ed24/src/gpt2.py#L87C13-L87C13

Hi SmallCap team,

Congrats on this work! Niels here from the open-source team at Hugging Face.

It's great to see the model being available on Hugging Face along with a demo: https://huggingface.co/spaces/RitaParadaRamos/SmallCapDemo.

However, I noticed that you register the model everytime you run the code. This is actually not necessary, you just need to register it once, then push the code to the hub and the model then becomes usable through the Transformers library.

We wrote a guide about it here: https://huggingface.co/docs/transformers/custom_models.

Along with that, it'd be great to update the model card to include a code snippet indicating how to run the model.

Let me know if you need any help!

Cheers,

Niels
ML Engineer @ HF

there is not such file in feature directory, which just has train.hdf5 and val.hdf5.

I love your work. Thank you for sharing the code.

I have a question about the input and label loading in the Class-TrainDataset. When self.rag = False, the outputs from the __getitem__ method were as follows:

prefix = 'This image shows'
prefix_ids = [1212, 2939, 2523, 220]
text = caption = 'people stand facing a person sitting on a horse'
text_ids = [15332, 1302, 6476, 257, 1048, 5586, 319, 257, 8223 ...]

input_ids = [1212, 2939, 2523, 220, 15332, 1302, 6476, 257, 1048, 5586, 319, 257, 8223, 0, ...]
label_ids = [-100, -100, -100, 15332, 1302, 6476, 257, 1048, 5586, 319, 257, 8223, 13, -100, ...]

However, according to modeling_gpt2.py#L1330:

loss = None
if labels is not None:
    # move labels to correct device to enable model parallelism
    labels = labels.to(lm_logits.device)
    # Shift so that tokens < n predict n
    shift_logits = lm_logits[..., :-1, :].contiguous()
    shift_labels = labels[..., 1:].contiguous()
    # Flatten the tokens
    loss_fct = CrossEntropyLoss()
    loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))

The loss is computed with shift_logits and shift_labels.

Therefore, as I understand it, the input_ids and label_ids should be as follows:

input_ids = [1212, 2939, 2523, 220, 15332, 1302, 6476, 257, 1048, 5586, 319, 257, 8223, 0, ...]
label_ids = [-100, -100, -100, -100, 15332, 1302, 6476, 257, 1048, 5586, 319, 257, 8223, -100, ...]

Note that the positions of '15332, 1302, 6476, ....' should be the same in both variables.

Could you please point out where I might have misunderstood?

Thanks for your excellent work, I just wonder when infering on COCO using COCO captions as datastore to prompt, the prompt may include the ground-truth captions, which may lead the data-leakage question. Have you adopted some strategies to avoid this question?

Error in SmallCap_demo.ipynb

NameError Traceback (most recent call last)
Cell In [6], line 10
7 with torch.no_grad():
8 image_embedding = retrieval_model.encode_image(pixel_values_retrieval.unsqueeze(0)).cpu().numpy()
---> 10 nns = retrieve_caps(image_embedding, retrieval_index)[0]
11 caps = [captions[i] for i in nns][:4]
13 # prepare prompt

NameError: name 'retrieve_caps' is not defined

there is no VisionEncoderDecoderModel in vision_encoder_decoder.py. when I read the file. I change it to "from transformers import VisionEncoderDecoderModel" in train.py . but there is another error.

thanks!

As shown in the bold part of the following code：
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
data_collator=default_data_collator,
train_dataset=train_dataset,
tokenizer=feature_extractor,
)

trainer.train()
Perhaps it should be replaced by tokenizer=tokenizer

parser.add_argument("--decoder_name", type=str, default="opt", help="Decoder name as found of HuggingFace or stored locally").
error:OSError: opt is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token or log in with huggingface-cli login and pass use_auth_token=True.

It seems to me that there is only code for training. I want to follow your work to do some research. Now I am trying to reproduce your work. But I didn't find code for evaluation.