Results

Adjusting Knobs

We first adjusted the parameters of the model to see how these parameters affect the stories generated for romance novels. The parameters are the following:

  • k = # of captions to use to generate story: Does increasing or decreasing the number of captions used as input change the quality of the output story?
  • b = Beam search width: breadth of the search for the decoder RNN

We chose to use the following sample image:

k = 100; b = 50 (default)

We were barely able to catch the breeze at the beach , and it felt as if someone stepped out of my mind . She was in love with him for the first time in months , so she had no intention of escaping . The sun had risen from the ocean , making her feel more alive than normal . She 's beautiful , but the truth is that I do n't know what to do . The sun was just starting to fade away , leaving people scattered around the Atlantic Ocean . I d seen the men in his life , who guided me at the beach once more .

k = 200; b = 100

We were barely able to breathe the breeze at the beach , and for the first time in months . The man who had just stepped out of my line of sight , as he watched her walk away from him . The sun was coming down on my face , making her feel more alive than it had ever been before . I do n't know what it would be like to go fishing , but that s the most important thing in the world . The sun was starting to fade , and the ocean waters gave chase . I m certain people who happened on their beach had no desire at the moment .

k = 20; b = 50

He had the ability to catch my breath at the beach , and after another day , I let out a soft sigh as the sun rose up off the horizon . I do n't know why it was so beautiful . The man who had fallen in love with her was gone , and he seemed to be the only person in the world . There was no reason for it , but it felt like a lot of people watching over her as they rode back to San Diego . The sun rose , bathing her body in waves . I had no intention of seeing him at the beach , especially when the sun came into life .

Varying the number of captions used in story generation and the beam width do not seem to change the quality of the ouput - all are very similar in topic and relevance to the image, at least for "romantic" images using the romance-trained storyteller.

The one difference of note is that training on many more captions incorporates less-relevant captions and therefore generates slightly less relevant stories. When k=200, the story includes a mention of a fisherman, not present in the image, that does not show up when trained on fewer captions.

Genre-Based Input Text Results

We were unable to train the model on our intended text genres (science fiction and biography) due to version compatibility issues (primarily involving Python 2.7) and deprecation of numpy and Theano versions for the neural storyteller from Kiros et al. (2015). Instead, we compared two versions of the neural storyteller that were trained by the original research group: the romance novel version and an older version trained on excerpts from adventure novels.

We compared the output of the romance and adventure models on images relevant to romance and adventure themes. We only had access to the input-ouput pairs of the adventure-novel-trained model, not the model itself, so we were limited in the kinds of comparison we could do (i.e. we could only directly compare the output of the two models for the images input to the adventure model, no additional images).

Output

Bus Romance

That bus was much closer to the curb , and for the first time in a week , I boarded the bus . I had no idea what it would be like to be with them . In fact , I could n't help but notice that I had fallen asleep in his arms . The streets of the city were empty , and the bus driver pulled me out of the parking lot . I lifted my head , searching his features until he saw the sign that threatened to fade . That it was the only way I m going to be in New York City , and close by .

Bus Adventure

In the second day , I panicked . I haven t seen him enter the city , so I quickly corrected the bus ride to the city . I was hoping to see that they were on such a bus , and as soon as we got within the city limits , I had no idea where I was . The bus driver was on the verge of tears running down his nose and mouth . I picked up the rental car and headed back towards London . The ferry never faltered in such a state , but it s not .


What we found was that when the content of the images did matter. Images with content relevant to adventures produced adventure-like stories. An image of a bus, which invokes themes of travel and captions related to travel, produces a short story that sounds like it could come from a contemporary adventure novel. It mentions panicking, moving quickly, and being lost - although it does stay very close to the image content, repeatedly mentioning the bus across several sentences.

The romance model, given the same picture, outputs something that you would indeed expect from a romance novel. There are mentions of being with or close someone, looking for emotions in faces, and generally a lot of attention to facial expressions and gestures.

We also tried the romance model on a very adventure-themed picture: a still from an Indiana Jones movie. The output, again, is very much what you might expect from a romance novel. Instead of an impending rollickin battle and fearing for one’s life on a rope bridge, the story is about seeing a man on a bridge and its emotional effect on a woman’s heart.

Output

Indiana Jones Romance

My heart was a train at the same time , and I let go . I had no idea what had happened to me . In fact , I saw him walking around the bridge , holding out his hands so he could be on the other side of the bridge . As much as it hurt , I did n't know what to say . The last thing she wanted was to wander through the streets of St. Giles , as if he were giving her a false sense of satisfaction . She looked up and down the steep hill . My God , it s a man I was accustomed to walking under a bridge that instant .