ritaramo / smallcap Goto Github PK
View Code? Open in Web Editor NEWSmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation
SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation
Hi,
I have a problem downloading stanford-corenlp file.
In coco-caption/get_stanford_models.sh,
CORENLP=stanford-corenlp-full-2015-12-09
wget http://nlp.stanford.edu/software/$CORENLP.zip
and I also try
wget http://nlp.stanford.edu/software/$CORENLP.zip --no-check-certificate
But I got this in linux
Unable to establish SSL connection.
I couldn't even access the URL and homepage..
https://stanfordnlp.github.io/CoreNLP/download.html
I would appreciate it if you could tell me what the problem is or send me the file.
Thank you
1
How to create my own datastore
I was reproducing the experiment accoding to README, and everything was ok until entering "retrieve captions". After images were encoded, it started to retrieve neighors. But the process seemed like frozen, nothing happened for like 2-3 hours. So I'm wondering if it is expected? How long is "retrieving neighors" gonna be?
Hey! We are doing research following your work. We have reproduced your work over coco dataset which performs very well, even better than your provided data! But when it comes to other datasets you referred in your paper, like Vizwiz, MSR-VTT, we find it very complicated to get the corresponding metrics results. So I wonder if you could provide the code for these experiments?
In addition, is the datastore/coco_index_captions.json
the augmented datastore you mentioned in 5.2 Augmenting the datastore?
NameError Traceback (most recent call last)
Cell In [6], line 10
7 with torch.no_grad():
8 image_embedding = retrieval_model.encode_image(pixel_values_retrieval.unsqueeze(0)).cpu().numpy()
---> 10 nns = retrieve_caps(image_embedding, retrieval_index)[0]
11 caps = [captions[i] for i in nns][:4]
13 # prepare prompt
NameError: name 'retrieve_caps' is not defined
parser.add_argument("--decoder_name", type=str, default="opt", help="Decoder name as found of HuggingFace or stored locally").
error:OSError: opt is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token
or log in with huggingface-cli login
and pass use_auth_token=True
.
sorry if this is caused by stupid reasons, I am new to Java language
I encountered the following problem when commanding 'python /smallcap/caption/dramaEvalCap.py'
tokenization...
PTBTokenizer tokenized 23639 tokens at 411834.08 tokens per second.
PTBTokenizer tokenized 47091 tokens at 630900.25 tokens per second.
setting up scorers...
computing Bleu score...
{'testlen': 44550, 'reflen': 21080, 'guess': [44550, 42008, 39466, 36925], 'correct': [5332, 1242, 258, 99]}
ratio:2.113378
Bleu_1: 11.97
Bleu_2: 5.95
Bleu_3: 2.85
Bleu_4: 1.58
computing METEOR score...
METEOR: 9.98
computing CIDEr score...
CIDEr: 11.78
computing SPICE score...
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Stopwatch
at edu.anu.spice.SpiceScorer.scoreBatch(SpiceScorer.java:69)
at edu.anu.spice.SpiceScorer.main(SpiceScorer.java:60)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Stopwatch
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 2 more
Traceback (most recent call last):
File "/home/lufan/Projects/smallcap/caption/dramaEvalCap.py", line 38, in
dramaEval.evaluate()
File "/home/lufan/Projects/smallcap/caption/pycocoevalcap/eval_drama.py", line 56, in evaluate
score, scores = scorer.compute_score(gts, res)
File "/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/spice.py", line 70, in compute_score
subprocess.check_call(spice_cmd,
File "/home/lufan/miniconda3/envs/smallcap/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/lufan/java/jdk1.8.0_371/bin/java', '-jar', '-Xmx8G', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/spice-1.0.jar', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/tmp/tmpiai6r6_r', '-out', '/home/lufan/Projects/smallcap/caption/pycocoevalcap/spice/tmp/tmpxbfagpuw', '-subset', '-silent']' returned non-zero exit status 1.
It seemed that guava.jar is not recognized.
Would you help, please?
Thanks for great paper!
I am curious how to get hidden_state value?
and why it has more than double dim size up? I thought it would be (b,50, 768), not a (b,140,768)
Could you explain why is it?
Thanks!!!
gpt2.py
87lines
It seems to me that there is only code for training. I want to follow your work to do some research. Now I am trying to reproduce your work. But I didn't find code for evaluation.
line 50
feature_extractor = CLIPFeatureExtractor.from_pretrained(args.encoder_name)
line 144
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
data_collator=default_data_collator,
train_dataset=train_dataset,
tokenizer=feature_extractor,
)
toknizer parameter in Seq2SeqTrainer, you use a feature_extractor which is for image processing(CLIP), not text toknizer like BERT, GPT
can you explain why you use CLIPFeatureExtractor for tokenizer in Seq2SeqTrainer?
Thanks for your excellent work, I just wonder when infering on COCO using COCO captions as datastore to prompt, the prompt may include the ground-truth captions, which may lead the data-leakage question. Have you adopted some strategies to avoid this question?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.