Discussion

Limitations

One thing we found was that there were some issues with the caption generation step in this process. The punctuation is all over the place, which can be addressed by cleaning up the text. However, if you look at the example below, you'll notice that for genres such as science fiction, the caption generator has a hard time recognizing non-human entities. This makes for stories that do not necessarily match the images and genres as intented.

The output of the romance-trained story generator reflects that the caption generator does not recognize that this is a robot, so in order to get science fiction- (or other genre-) themed stories, we would need a caption generator that recognizes images characteristic of those genres like robots and spaceships for science fiction, or historical period-specific items for biographies.

Generated Story using Romance text
My massive fire had nothing to do with the knowledge , and it took me a moment to realize what was going on . A few hours ago , I had no idea how much of it . In fact , I could n't help but watch as the rain beat down heavily on the inside of his skull . She looked like she belonged to him , and she was one of the most beautiful girls in New York . The light bulb that hung from the ceiling , as if it were a safe haven and a trace of guilt . Now that Logan had his magic , I m devoted to it and made up at last .

Future Work

In the future, we would like to develop an evaluation metric to see how closely the image matches the generated story. When modifying the parameters, it is difficult to see what the difference in output really is, and it's hard to tell how different each story is to the corresponding image.

We also want to work on an evaluation metric for determining how creative a story is based on the image. For applications such as generating fun captions for Instagram, we want to make sure this generator is able to output something more than just a technical description of the image.

Another area of future work is related to the limitation of the caption generator. We noticed that it was difficult to generate captions that actually fit well with science fiction images, so we would like to try to perhaps find another caption generator that would perform better in these cases.