shagunsodhani / image-caption-generator

A simple implementation of neural image caption generator

Image-Caption-Generator

A simple implementation of neural image caption generator

Please note that the code in this repo is for use in talks/workshops. There is a lot of room for improvement (in terms of both accuracy and efficiency) so that these aspects can be discussed during the sessions.

You may refer to Tensorflow's im2text Model for a stable and accurate implementation.

Setup

Create Directories

• Run ./scripts/mkdir.sh

Downloading Datasets

• Run ./scripts/download_images.sh
• This downloads Flick8K dataset

Downloading Models

• The VGG16 model would be downloaded automatically when the model is trained for the first time and would be cached on the disk.
• Alternatively, run python3 vgg16.py. It would download the VGG16 model, produce the embeddings for a test image and compare with a pre-computed embedding.

Processing Images

• Update data_dir in code/preprocess.py and set mode_list=["train", "test", "debug"]
• Run python3 preprocess.py

Train

• Run python3 train.py

image-caption-generator's Issues

Gives me same reply for any picture i give it

I've trained the model. And when i run test.py it gives me same output for any different picture no matter what. Please help

Generated caption is: start man on skis looking at artwork for sale displaying paintings in the snow end end skier skier in a blue hat end end end end end end end end end end end end end end end end end end
Score is: 36.81266660988331

Generated caption is: start a man in a hat is displaying pictures next to a skier in a blue hat end end end end end end end end end end end end end end end end end end end end end end end
Score is: 37.113694965839386

Happens for all inputs

Cannot find debug_image_encoding.pkl

Hi where can I locate the debug_image_encoding.pkl which is used in preprocess.py ?

Thanks in advance

When i run test.py i get this error what should i do??? thanks in advance

Traceback (most recent call last):
File "test.py", line 108, in
mode="debug")
File "test.py", line 84, in predict
model.load_weights(data_dir + "model/" + weights_path)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2613, in load_weights
f = h5py.File(filepath, mode='r')
File "/home/ajith/.local/lib/python2.7/site-packages/h5py/_hl/files.py", line 271, in init
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
File "/home/ajith/.local/lib/python2.7/site-packages/h5py/_hl/files.py", line 101, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-nCYoKW-build/h5py/_objects.c:2840)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-nCYoKW-build/h5py/_objects.c:2798)
File "h5py/h5f.pyx", line 78, in h5py.h5f.open (/tmp/pip-nCYoKW-build/h5py/h5f.c:2117)
IOError: Unable to open file (Unable to open file: name = '/home/ajith/image-caption-generator/data/model/weights-00.hdf5', errno = 2, error message = 'no such file or directory', flags = 0, o_flags = 0)

can not .pkl file

Hey shagun i am getting two errors. please help me.
i run python3 preprocess.py
error: os error cant find image file
when i run python3 train.py
error:file not found error, debug_image_encoding.pkl

please contact on [email protected] sir.
please help.
Thanks in advance.
Naveen vadavalasa

Can I know which algorithm or paper is used for this implementation.