Twitter’s Clockwork Raven on Heroku

Last week, we were really excited to see Twitter open-source their internal Mechanical Turk tool, Clockwork Raven, which “steps in to do what algorithms cannot”. At first glance, it looks like it’s shaping up to be a great tool for people wanting to get started on MTurk with a lot of great features planned for the future including:

• JSON/REST API
• Option to have multiple Mechanical Turk users complete each task
• In-depth analytics about workers and automate the process of choosing trusted workers.

We won’t go over Clockwork Raven’s existing features since they’ve already done a great job of it on their Getting Started wiki page. So if you’re not familiar yet, definitely check it out before reading on.

Heroku Demo

If you want to give Clockwork a spin without having to install and configure the app yourself, we’ve deployed a demo to Heroku that we’ll keep up for at least the next few weeks: houdini-raven-demo.herokuapp.com

(You can log in with username ‘demo’ and password ‘demo’, which will give you access to create and review tasks in the Mturk Sandbox.)

After logging in you can view the results of an example evaluation or you can create a new evaluation from the evaluation list page.

Deploying Your Own Heroku App

If you do want to spin it up on Heroku for your own use, here’s how:

# Make sure we're using Ruby 1.9.3 and checkout the heroku-demo branch of our fork
rvm use 1.9.3
git clone git://github.com/chrisconley/clockworkraven.git my-raven
cd my-raven/
git checkout heroku-demo
 
# Create the heroku app and add the Redis and Mysql add-ons
heroku create
heroku addons:add redistogo:nano
heroku addons:add cleardb:ignite
# Copy the value of CLEARDB_DATABASE_URL and set DATABASE_URL with it
heroku config | grep CLEARDB_DATABASE_URL 
# Make sure to change the protocol from `mysql://` to `mysql2://`
heroku config:add DATABASE_URL='mysql2://aaaaaaa:aaaaaaa@us-cdbr-east.cleardb.com/heroku_aaaaaa?reconnect=true'
 
# Now we need to set a few environment variables for the app to run
 
# copy the secret key generated from `rake secret`
rake secret
# and set the SECRET_TOKEN with it
heroku config:add SECRET_TOKEN=3688f877842a0e9324c14ea558268290
 
# Set your AWS credentials from  https://portal.aws.amazon.com/gp/aws/securityCredentials#access_credentials
heroku config:add AWS_KEY_ID=[your_key_id]
heroku config:add AWS_KEY=[your_key]
 
# Deploy the app to heroku
git push heroku heroku-demo:master
 
# Scale up our background worker needed to send tasks to Mturk
heroku scale worker=1
 
# Bootstrap the db
heroku run rake db:structure:load
 
# Add our first user
heroku run rake users:add

Future Development

We’re looking forward to seeing where @echen, who’s now maintaining the project at Twitter HQ, takes Raven from here. (Btw, Edwin has an awesome blog at echen.me). It’s really exciting that Twitter open-sourced the software – it’s a great move towards improving the accessiblity of Mturk.

Houdini Beta

I’d also be remiss if I didn’t do a little (shameless) promotion, but if you’re looking for some of the forthcoming features on the Clockwork Raven roadmap (such as API access or the ability to have multiple workers complete each task) and/or want a hosted solution, sign up for our beta – we’re going to start letting users in in the next week or two. (or email me directly at chris at houdiniapi dot com and I’ll get you in this week.)