Mechanical Turk unveils new Categorization App

Amazon Web Services recently released the new “Categorization App” for Mechanical Turk.  The Categorization App is designed to make it easier for requesters to design and run applications that require data to be assigned to one or more categories.  If, for example, you have a list of businesses that need to be filtered by industry, or a list of websites to be organized into different topical or subject areas, each of these tasks involves categorization of data into different classes. Popular crowdsourcing applications like sentiment analysis and image moderation also fall under the “categorization” umbrella.

Example Categorization Tasks

Example Categorization Tasks

One of the complaints we frequently hear from customers is that Mechanical Turk can be difficult to get set up, particularly for those who are less technically inclined. The Categorization App is intended to simplify the setup for tasks involving categorization.  Instead of the usual WYSIWYG interface that Mechanical Turk provides for designing html forms, the Categorization App uses a simple question and answer wizard that walks requesters through each of the steps of setting up a categorization task. Based on the responses provided, the app will populate a pre-designed template and automatically construct the form to be shown to workers, thereby eliminating the need for requesters to build their own html form.

Categorization App Wizard

Categorization App Wizard

The main advantage of using the Categorization App over the standard Mechanical Turk interface is that it allows requesters to create tasks without needing to construct their own HTML by hand (or fiddle with Turk’s native WYSIWYG page builder).  This helps lower the barrier to entry for requesters who might otherwise be intimidated by Mechanical Turk’s standard interface. Besides the easier setup process, the Categorization App also provides a measure of quality assurance by automatically submitting each task to two separate workers before returning a result.  Using multiple workers to cross-check answers is a common technique for improving the data quality of crowdsourced labor, so it is nice to see Amazon supporting this as a native feature.

Improvements we’d like to see

While the Categorization App’s wizard-style interface can make the process of getting started less intimidating for new users of Mechanical Turk, there are a few features we’d like to see in the future versions of the App:

  1. API access.  For the time being, Amazon does not provide API access to the Categorization App, so all tasks must be submitted via manually uploading/downloading a .csv.  If you need to submit tasks via API (e.g. real-time image moderation) you will need to stick with Mechanical Turk’s standard interface for now.
  2. Support for multiple categories.  In cases where you want data to be assigned to two or more “best” categories, or to a primary and a secondary category,  or simply assigned to “all relevant” categories, the existing Categorization App won’t work.
  3. Ability to ask more than two workers to complete a task.  Many applications can benefit from having three or more workers complete the same task.  In particular, tasks with a binary result set (true/false, acceptable/unacceptable) will benefit from using an odd number of workers to avoid resulting in a tie.   This includes popular applications like content moderation. Also, applications that are prone to frequent differences of opinions – such as sentiment analysis – will also benefit from employing a larger number of workers in order to derive a “consensus” answer (this represents “crowdsourcing” in the most literal sense).
  4. Batching of multiple tasks within in a single HIT.  A HIT in Mechanical Turk (short for Human Intelligence Task) represents a single page of work units for the worker to complete.   When using Mechanical Turk to process short, quick tasks, its is often best practice to “batch” 3 or 5 or 10  assignments together into a single HIT.   This makes it easier for the worker to complete several tasks all at once rather than having to page through each assignment one at a time.  In addition to improving worker efficiency, this allows requesters to avoid overpaying for Mechanical Turk fees – because Amazon charges the greater of (a) 10% of the worker fee or (b) half a cent, for every HIT created, it is more cost efficient to batch several tasks together than create many smaller hits worth one or two cents a piece.

In summary, the Categorization App provides an easier to use option for requesters looking to complete basic categorization tasks.  Although the initial version of the App is not perfect, we are excited to see Amazon taking strides toward providing a friendlier interface and opening the platform to a broader range of users.  Requesters who are already familiar with Mechanical Turk will find little here that cannot be accomplished with the standard interface.  But new users will likely find the Categorization App to provide an easier, more streamlined initial experience.