Communicating with Reviewers

Learn about communicating with human reviewers within the super.AI platform.

Communication with the human labelers involved in your project is an essential part of identifying problems and improving your project’s performance. You can use the information you receive from labelers for the following:

  • Improve your project’s instructions and task design
  • Identify edges cases or points of confusion that require more context
  • Surface input data that is problematic (e.g., broken links, poor quality images, etc.)

Any human labelers working on your project—whether from the super.AI crowd or your own labelers—can leave messages on data points that they process. Labelers select a category for their message within the labeling interface and these categories appear as labels on the messages within your project dashboard.

These are the default categories labelers can choose from:

  • Task unclear
  • Input problem
  • Duration problem
  • Tool problem
  • Other

You can also customize these categories and add subcategories to keep things organised: just contact your customer success representative.

How to view all messages in your project

  1. Head to your super.AI dashboard
  2. Open the relevant project
  3. Click on Messages in the left-hand menu
2440

The Messages section of a project dashboard

The messages inbox shows a list of all data points that have messages attached. The most recent message appears at the top by default and each data point only displays its most recent message.

The colored tags indicate the message categories. The number in each tag at the top of the inbox indicates how many messages are in that category. You can click on a tag to show only messages related to that category—the tags at the top of the inbox will also update to show any subcategories that you have defined.

You can open the data point details view—where you can view all the messages related to the data point—by clicking on a message in the inbox. Within the data point details view, you can leave notes for yourself using the text input box in the Messages section. Currently, labelers will not see the notes you leave; they are only for your and your team member’s reference.

How to filter your messages

To filter your messages inbox, you can click on a colored category tag at the top of the inbox. You can also use the Order and Filter buttons at the top right of the page. To learn more about filters, see our How to filter data points page.

1222

An animation showing message filtering and an update to task instructions

You can download all data points with messages of a certain category by clicking on the corresponding tag and choosing Download filtered. This can be useful when performing an analysis on all data points with problematic input data, for example.

How to know when you have messages

When viewing your project in the dashboard, a badge appears against the Messages menu item if any new messages have come in since you last viewed your messages.

How to view messages on individual data points

  1. Open your super.AI dashboard
  2. Click on a data point in the work queue to open the data point details view
  3. Any messages appear in the Messages section on the right alongside a colored tag indicating the message category
1134

The data point details view with a labeler message

You can leave notes for yourself using the text input in the Messages section. Currently, labelers will not see the notes you leave; they are only for your reference.

📘

Filter for data points with messages

You can filter your work queue to only show data points with messages: click on Filter above the work queue and select Messages from the first dropdown. For more information, see our How to filter data points page.