Sunday, December 4, 2011

Writing a short horror story with Amazon Mechanical Turk

I've been wanting to experiment with Amazon Mechanical Turk ("Artificial Artificial Intelligence") and recently came up with an idea for a fun project from my days as an improv theater teacher and actor. There's an exercise improvisors use, and sometimes perform before an audience, called "word at a time story" where each actor takes a turn advancing a story with one word. If you commit fully to it with no self-censoring, you can often get a pretty coherent story.

After reading Solving Instagram’s Unshredder with Mechanical Turk and $0.50 I wondered if I could ask mechanical turk workers to write a story using one sentence at a time. I setup a manual Human Intelligence Task (HIT) for workers to write the first sentence of a story, just to get a feeling for how the system worked. I got 20 opening lines, paying about $0.20 each. I rejected three of the responses for not being complete sentences or for being plagiarized. I threw away three others for being overly creepy, leaving 14 interesting openers:
  • Jim approached the open door cautiously, unsure of the source of noise he had heard just moments prior.
  • It was midnight, she was driving back home after a long day at work still thinking of the stranger that had spoken to her in the street.
  • Yesterday was the best day of my whole life.
  • Today would be the third day Sarah and her younger brother had been on the road alone after running away from the orphanage.
  • Once upon a time, lived a little elf named snowball, who worked for Santa at the North Pole.
  • Hurriedly, she pushed the email icon on her phone to read the message from Sandy, the drug counselor.
  • There was something off about the way he watched her, his eyes never broke away - even after she had crossed the street.
  • "Well, everything went better than expected", Andy thought as he left the building with the business card still in his hand.
  • A lump rose in Andy's throat and he brushed away tears as he reread the chilling message taped to his dorm room door.
  • There was a funny little house in the town, that had stood vacant for many years.
  • It was a dark rainy night in November as Marcus felt a cold wind blow down Hanging Dog Creek.
  • It was the height of summer, it was warm and sunny and beautiful outside but Cooper didn't feel in any way excited as he sat at his kitchen table, sipping his by now luke warm morning coffee, trying to make sense of what had happened the night before.
  • I was a big, fast, tailback with a nasty streak and a taste for blood.
  • The old woman turned and smiled.
I chose "lump rose in Andy's throat" for the rest of the experiment because I thought it would be good to start with a strong genre. I then wrote a script based on sample code I found in the ruby-aws gem. The script posts one HIT to Amazon displaying the story so far and asking the worker to add one sentence. Here's what each HIT looked like on the worker side:


I reviewed each sentence before creating a new HIT to make sure bad results didn't corrupt the story. Usually you would just upload a whole batch of HITs to be worked on in parallel; if you need to do serial processing with Mechanical Turk my code may help you.

I didn't need to reject any sentences, and in the spirit of the original improv exercise I did no editing. The workers were also in charge of starting new paragraphs. Below is what they came up with. I cut off the story arbitrarily as it seemed we were descending into minutiae and, hey, this was costing me money ($0.10 per sentence).
A lump rose in Andy's throat and he brushed away tears as he reread the chilling message taped to his dorm room door. This can't be happening he thought, How could someone do this to me? Andy carefully took the note off of the door, and entered his room. He sat at the edge of his bed, thoughts racing through his head as he stared at the paper in his hand. He tore it furiously. 
His girlfriend left him. She was never happy in the relationship and decided against continuing it. He was at a loss at what to think or say or do.  She was his first girlfriend, his first love. Andy decided to talk to his girlfriend and see if there was anything that could be done to save their relationship. He ran as fast as he could, almost tripping down the stairs, as he held back tears, to her dorm room three flights below his. When he got there, he stopped for a while and took a deep breath. He entered without knocking the door. Something was amiss - his girlfriend's dorm room looked like it had been ransacked, and she was nowhere to be found. As he backed slowly out of the room, he bumped into his girlfriends roommate Ella, who shoved him and said stearnly "Why are you in here?  Don't you know, Alexis broke up with you!" Andy looked at her with empty eyes, pushed her aside and run away. He was starting to suspect that someone had kidnapped his girlfriend and left him a fake break-up note so that he would stay away and not become suspicious - and maybe her friend Ella knew something. He had to go back. But before that he needed to take a closer look at the note to make sure that it was not Ella's handwriting. He entered his room and took carefully the pieces of the note from the floor. He put them together. He was really confused, it was her handwriting. He felt so devastated that he couldn't even cry. 
A sense of calm came over Andy as he reached for the phone to call the police. "Someone is bleeding to death in the basement" he stated. "Are they still conscious?" Asked the officer. "Barely so.", he replied, "Her breathing is faltering." "What is your address?" the voice asked, calm and collected. Andy, aware of the address belonging to a large dormitory, calmly stated the address, spelling out the street name with perfect enunciation. He was completely numb, unaware of the true circumstances that laid before him. He had finally done it.
Other notes:
  • Some of the workers wrote two sentences instead of the required one.
  • ProPublica has a wonderful guide explaining more practical uses for Mechanical Turk as a way to help their reporting. 
  • I'd like to thank Mechanical Turk workers A13KRCVFOPZ3UX, A141RVMIW610GL, A14543PX95IM4I, A14IQ4GLNWNPOJ, A17PLH2GYHRALL, A1801RSQIIDWHW, A19OCN9KO1NWNL, A1A76U56BNERB9, A1NHNUBZEN1EN8, A1SAOAUD2D8OKR, A1SFABJ4NX5DFY, A1WFA8NWU6RVQ9, A25L94D9I3GJFQ, A26L91YL0GDGD8, A28G4QO0DRY8OZ, A2DTTNGUXTEGHQ, A2EG6T3WW37TGV, A2SEG0I7JX3WJ5, A2SOI6L2A4F34D, A2T4CZ6TC2SY5R, A2VXI416SODAQC, A30EBLWYETOTFG, A3B86TCZRF9AOI, A3D1BHGFDBSSC1, A3FS6G1PEDN3GH, A3HMBHM8HJLKRD, A3IBW6XVOXK3P6, A3NUWV8UAHEDQ1, A3OWHW7XYQU52K, A6SYXKX5P76DL, AB217P6RM0JX4, ACS62XDG6GAV4, AFEWKFA1I80PP, AHFTRTLN0R79E, and AUTSDI5IHO8XN for their help in this project!

3 comments:

aantix said...

Love the experiment Mike! But I didn't see the code committed on Github..?

If you ever make it down to San Francisco, drop me an email (jim.jones1@gmail.com). I'm the creator of the Turkee gem and my friend Mark Percival is the creator of the RTurk gem, and we both love to talk creative uses of Mechanical Turk.

So much so in fact, we gave a presentation on it recently at the San Francisco Ruby meetup. http://www.sfruby.info/events/42829062/

Mike Subelsky said...

oh sorry about that, here's the code:
https://github.com/subelsky/storyteller

Mike Subelsky said...

PS definitely will do, I usually get out there once a year or so, thanks for reading the article!