A Dual Prediction Network for Image Captioning

General captioning practice involves a single forward prediction, with the aim of predicting the word in the next timestep given the word in the current timestep. In this paper, we present a novel captioning framework, namely Dual Prediction Network (DPN), which is end-to-end trainable and addresses the captioning problem with dual predictions. Specifically, the dual predictions consist of a forward prediction to generate the next word from the current input word, as well as a backward prediction to reconstruct the input word using the predicted word. DPN has two appealing properties: 1) By introducing an extra supervision signal on the prediction, DPN can better capture the interplay between the input and the target; 2) Utilizing the reconstructed input, DPN can make another new prediction. During the test phase, we average both predictions to formulate the final target sentence. Experimental results on the MS COCO dataset demonstrate that, benefiting from the reconstruction step, both generated predictions in DPN outperform the predictions of methods based on the general captioning practice (single forward prediction), and averaging them can bring a further accuracy boost. Overall, DPN achieves competitive results with state-of-the-art approaches, across multiple evaluation metrics.

Guo Yanming, Liu Yu, de Boer Maaike H.T., Liu Li, Lew Michael S.

Publication type:
A4 Article in conference proceedings

Place of publication:
2018 IEEE International Conference on Multimedia and Expo, ICME 2018

deep supervision, Dual Prediction Network, image captioning, reconstruction


Full citation:
Y. Guo, Y. Liu, M. H. T. De Boer, L. Liu and M. S. Lew, “A Dual Prediction Network for Image Captioning,” 2018 IEEE International Conference on Multimedia and Expo (ICME), San Diego, CA, 2018, pp. 1-6. doi: 10.1109/ICME.2018.8486491


Read the publication here: