Performance Evaluation on the Effect of Different Text Representation Models on the Image Captioning Systems

Authors

  • Jafar Alkheir
  • Samer Sulaiman
  • Rasha Mualla

Keywords:

Deep Learning, Natural Language Processing, Image Representation, Text Representation, FastText Model, GloVe Model.

Abstract

This research deals with one of the most important and recent topics in the field of machine learning in general and deep learning in particular, which is image Captioning systems. In this research, an image-captioning system is built based on the ResNet50 model, which is a deep learning network based on convolution neural networks CNN, through which the features of the image representation are obtained. As for the textual representation, five different models are proposed, based mainly on the GloVe and FastText models provided by Twitter and Facebook, respectively. The effect of different vocabulary dictionaries on the performance of the proposed system is studied. A global MS-COCO dataset is used, from which a subset of 10,000 images is token, 9,000 images from them are chosen for the Training and validation group. While the testing process includes 1000 images varying from the training-set. This test-set is applied to the five designed models.

        To find out the precision of the results used by the five proposed systems as well as how well they match between the original description sentences and the resulting description ones, performance measures are used such Accuracy, Average of Depth Similarity, Top-1, Top-5 and BLEU. The results show the superiority of systems based on FastText models although they take longer time than GloVe models.

 

 

Downloads

Published

2020-10-01

How to Cite

1.
الخير ج, سليمان س, معلا ر. Performance Evaluation on the Effect of Different Text Representation Models on the Image Captioning Systems. Tuj-eng [Internet]. 2020Oct.1 [cited 2024Nov.24];42(4). Available from: https://journal.tishreen.edu.sy/index.php/engscnc/article/view/9915

Issue

Vol. 42 No. 4 (2020): Tishreen University Journal for Research and Scientific Studies - Engineering Sciences Series

Section

Articles

License

Copyright (c) 2020 ttps://creativecommons.org/licenses/by-nc-sa/4.0/

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

The authors retain the copyright and grant the right to publish in the magazine for the first time with the transfer of the commercial right to Tishreen University Journal for Research and Scientific Studies - Engineering Sciences Series

 Under a CC BY- NC-SA 04 license that allows others to share the work with of the work's authorship and initial publication in this journal. Authors can use a copy of their articles in their scientific activity, and on their scientific websites, provided that the place of publication is indicted in  Tishreen University Journal for Research and Scientific Studies - Engineering Sciences Series   .  The Readers have the right to send, print and subscribe to the initial version of the article, and the title of  Tishreen University Journal for Research and Scientific Studies - Engineering Sciences Series     Publisher

journal uses a CC BY-NC-SA license which mean

You are free to:

  • Adapt — remix, transform, and build upon the material
  •  
  • The licensor cannot revoke these freedoms as long as you follow the license terms.

 

  • Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
  • NonCommercial — You may not use the material for commercial purposes.
  • ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
  • No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.