nicolafan / image-captioning-cnn-rnn

Tensorflow/Keras implementation of an image captioning neural network, using CNN and RNN

• 👋 Hi, I’m @nicolafan
• 👀 I’m interested in Machine Learning applications with Python, web development and algorithms.
• 💞️ I’m looking to collaborate on coding projects on GitHub, I think this will be a method for deeply understanding what I'm studying.
• 📫 How to reach me: forums or my email address (or send me a DM on Twitter).
• 📚 Currently I'm a Computer Science student in Italy.

image-captioning-cnn-rnn's Issues

Update the cookiecutter project organization in the README.md

Since there are new files in the project structure, the README should be updated with brief explanations.

Add command line option `--beam_width` to model prediction

The file is /src/models/predict_model.py.

Load model weights

Load some model weights so that users can directly use the model without previous training.

Update the Makefile

I create this issue just to remember to update it when a clear pipeline, with CLI arguments etc., is defined.

Long evaluation time

Should think about some way to make evaluation (computation of the BLEU scores over different beam widths) faster, since yesterday it required around 4 hours for an evaluation on a grid search over 3 beam widths.

Implement model prediction

Implement the content of the predict_model.py file.

In the file a stateful model should be loaded to use it for prediction.

The current implementation requires that an image has to be given as an input to the model together with a sequence of length MAX_CAPTION_LENGTH, where the first element of the sequence is 1 *corresponding to the <start>token) and all the others are zeros (for masking).

Then we take the model output at the first timestep, corresponding to the softmax probability distribution over the vocabulary, and sample one word from this probability distribution (remember that here the 0-th neuron corresponds to the token <start> which has index 1 in the vocabulary and so on). Sampling can be performed:

• by taking a random element according to the probability distribution
• by taking the argmax of the probability distribution
• with a beam search.

These three methods should all be implemented.

When we sample a word, we create the input for the next timestep, made of the same image (which will be encoded by the model but not used - adjust this) and a new caption sequence where the first element is the index of the sampled word and all the other elements are 0. The fact that the model is stateful, means that it will keep the state it had at the previous timestep and can continue the prediction of the sequence without problems. We will sample until MAX_CAPTION_LENGTH or until <end> gets produced by the model.

Support mini-batches at prediction time

This issue is related to the performances that we want to improve with #1 .

The current implementation of the beam search only works with batch size equal to 1. There are some slight changes that can be made to support mini-batches with a size greater than 1.

Remove cartesian axes from prediction plot

The prediction plot shows the cartesian axes over all the images (except for the last one). Remove the axes (file is src/models/predict_model.py).

Improve (?) implementation of the BLEU metric

The BLEU metric is a numerical metric used in image captioning.

They need to be implemented inside the src/models/metrics.py file (not sure if it is the correct place by the way).
I think this should not be implemented as a tensor metric that can be used by Tensorflow, but as a metric that has to be applied directly to strings.

Basically, we will provide the ground truth caption string and a string predicted by the model. How the prediction string is produced depends on the implementation to try: sampling, beam search, or max likelihood, but is not of interest for the BLEU Implementation.

Restore model from config and weights for model training, via CLI option

Add a CLI option to model training that provides the possibility to continue training an existing model.

If the option is specified, with the model filename, we should ignore all the parameters that refer to the model architecture or raise an error if they are specified.

"Separate" the encoder from the entire model

Use the Subclassing API to divide the ShowAndTell model into two sub-models: the encoder and the decoder. This is required since at prediction time the model is applied at each step of beam search on the same image over and over again. The image encoding is unique and so it can be calculated just once, at the beginning of the process.