ritaramo / smallcap Goto Github PK
View Code? Open in Web Editor NEWSmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation
SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation
sorry if this is caused by stupid reasons, I am new to Java language
I encountered the following problem when commanding 'python /smallcap/caption/dramaEvalCap.py'
tokenization...
PTBTokenizer tokenized 23639 tokens at 411834.08 tokens per second.
PTBTokenizer tokenized 47091 tokens at 630900.25 tokens per second.
setting up scorers...
computing Bleu score...
{'testlen': 44550, 'reflen': 21080, 'guess': [44550, 42008, 39466, 36925], 'correct': [5332, 1242, 258, 99]}
ratio:2.113378
Bleu_1: 11.97
Bleu_2: 5.95
Bleu_3: 2.85
Bleu_4: 1.58
computing METEOR score...
METEOR: 9.98
computing CIDEr score...
CIDEr: 11.78
computing SPICE score...
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Stopwatch
at edu.anu.spice.SpiceScorer.scoreBatch(SpiceScorer.java:69)
at edu.anu.spice.SpiceScorer.main(SpiceScorer.java:60)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Stopwatch
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 2 more
Traceback (most recent call last):
File "/home/lufan/Projects/smallcap/caption/dramaEvalCap.py", line 38, in
dramaEval.evaluate()
File "/home/lufan/Projects/smallcap/caption/pycocoevalcap/eval_drama.py", line 56, in evaluate
score, scores = scorer.compute_score(gts, res)
File "/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/spice.py", line 70, in compute_score
subprocess.check_call(spice_cmd,
File "/home/lufan/miniconda3/envs/smallcap/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/lufan/java/jdk1.8.0_371/bin/java', '-jar', '-Xmx8G', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/spice-1.0.jar', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/tmp/tmpiai6r6_r', '-out', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/tmp/tmpxbfagpuw', '-subset', '-silent']' returned non-zero exit status 1.
It seemed that guava.jar is not recognized.
Would you help, please?
How to create my own datastore
1
I was reproducing the experiment accoding to README, and everything was ok until entering "retrieve captions". After images were encoded, it started to retrieve neighors. But the process seemed like frozen, nothing happened for like 2-3 hours. So I'm wondering if it is expected? How long is "retrieving neighors" gonna be?
Hey! We are doing research following your work. We have reproduced your work over coco dataset which performs very well, even better than your provided data! But when it comes to other datasets you referred in your paper, like Vizwiz, MSR-VTT, we find it very complicated to get the corresponding metrics results. So I wonder if you could provide the code for these experiments?
In addition, is the datastore/coco_index_captions.json
the augmented datastore you mentioned in 5.2 Augmenting the datastore?
line 50
feature_extractor = CLIPFeatureExtractor.from_pretrained(args.encoder_name)
line 144
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
data_collator=default_data_collator,
train_dataset=train_dataset,
tokenizer=feature_extractor,
)
toknizer parameter in Seq2SeqTrainer, you use a feature_extractor which is for image processing(CLIP), not text toknizer like BERT, GPT
can you explain why you use CLIPFeatureExtractor for tokenizer in Seq2SeqTrainer?
Hi,
I have a problem downloading stanford-corenlp file.
In coco-caption/get_stanford_models.sh,
CORENLP=stanford-corenlp-full-2015-12-09
wget http://nlp.stanford.edu/software/$CORENLP.zip
and I also try
wget http://nlp.stanford.edu/software/$CORENLP.zip --no-check-certificate
But I got this in linux
Unable to establish SSL connection.
I couldn't even access the URL and homepage..
https://stanfordnlp.github.io/CoreNLP/download.html
I would appreciate it if you could tell me what the problem is or send me the file.
Thank you
Thanks for great paper!
I am curious how to get hidden_state value?
and why it has more than double dim size up? I thought it would be (b,50, 768), not a (b,140,768)
Could you explain why is it?
Thanks!!!
gpt2.py
87lines
Hi SmallCap team,
Congrats on this work! Niels here from the open-source team at Hugging Face.
It's great to see the model being available on Hugging Face along with a demo: https://huggingface.co/spaces/RitaParadaRamos/SmallCapDemo.
However, I noticed that you register the model everytime you run the code. This is actually not necessary, you just need to register it once, then push the code to the hub and the model then becomes usable through the Transformers library.
We wrote a guide about it here: https://huggingface.co/docs/transformers/custom_models.
Along with that, it'd be great to update the model card to include a code snippet indicating how to run the model.
Let me know if you need any help!
Cheers,
Niels
ML Engineer @ HF
I love your work. Thank you for sharing the code.
I have a question about the input and label loading in the Class-TrainDataset. When self.rag = False
, the outputs from the __getitem__
method were as follows:
prefix = 'This image shows'
prefix_ids = [1212, 2939, 2523, 220]
text = caption = 'people stand facing a person sitting on a horse'
text_ids = [15332, 1302, 6476, 257, 1048, 5586, 319, 257, 8223 ...]
input_ids = [1212, 2939, 2523, 220, 15332, 1302, 6476, 257, 1048, 5586, 319, 257, 8223, 0, ...]
label_ids = [-100, -100, -100, 15332, 1302, 6476, 257, 1048, 5586, 319, 257, 8223, 13, -100, ...]
However, according to modeling_gpt2.py#L1330:
loss = None
if labels is not None:
# move labels to correct device to enable model parallelism
labels = labels.to(lm_logits.device)
# Shift so that tokens < n predict n
shift_logits = lm_logits[..., :-1, :].contiguous()
shift_labels = labels[..., 1:].contiguous()
# Flatten the tokens
loss_fct = CrossEntropyLoss()
loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
The loss is computed with shift_logits
and shift_labels
.
Therefore, as I understand it, the input_ids
and label_ids
should be as follows:
input_ids = [1212, 2939, 2523, 220, 15332, 1302, 6476, 257, 1048, 5586, 319, 257, 8223, 0, ...]
label_ids = [-100, -100, -100, -100, 15332, 1302, 6476, 257, 1048, 5586, 319, 257, 8223, -100, ...]
Note that the positions of '15332, 1302, 6476, ....' should be the same in both variables.
Could you please point out where I might have misunderstood?
Thanks for your excellent work, I just wonder when infering on COCO using COCO captions as datastore to prompt, the prompt may include the ground-truth captions, which may lead the data-leakage question. Have you adopted some strategies to avoid this question?
NameError Traceback (most recent call last)
Cell In [6], line 10
7 with torch.no_grad():
8 image_embedding = retrieval_model.encode_image(pixel_values_retrieval.unsqueeze(0)).cpu().numpy()
---> 10 nns = retrieve_caps(image_embedding, retrieval_index)[0]
11 caps = [captions[i] for i in nns][:4]
13 # prepare prompt
NameError: name 'retrieve_caps' is not defined
As shown in the bold part of the following code:
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
data_collator=default_data_collator,
train_dataset=train_dataset,
tokenizer=feature_extractor,
)
trainer.train()
Perhaps it should be replaced by tokenizer=tokenizer
parser.add_argument("--decoder_name", type=str, default="opt", help="Decoder name as found of HuggingFace or stored locally").
error:OSError: opt is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token
or log in with huggingface-cli login
and pass use_auth_token=True
.
It seems to me that there is only code for training. I want to follow your work to do some research. Now I am trying to reproduce your work. But I didn't find code for evaluation.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.