Bringing survey methodology to machine learning: Effects of annotation collection methods on training data and models

Join us for an event in partnership with City, University of London.
  • Event time:
    8th June 2023 13:00 – 14:00
  • Format:


The instruments used to collect training data for machine learning models have many similarities to web surveys, such as the provision of a stimulus and fixed response options.

Survey methodologists know that item and response option wording and ordering, as well as annotator effects, impact survey data.

Our previous research showed that these effects also occur when collecting annotations for model training.

Our new study builds on those results, exploring how instrument structure and annotator composition impact models trained on the resulting annotations.

Using previously annotated Twitter data on hate speech, we collect annotations with five versions of an annotation instrument, randomly assigning annotators to versions.

We then train ML models on each of the five resulting datasets.

By comparing model performance across the instruments, we aim to understand:

  1. whether the way annotations are collected impacts the predictions and errors by the trained models
  2. which instrument version leads to the most efficient model.

In addition, we expand upon our earlier findings that annotators' demographic characteristics impact the annotations they make. Our results emphasize the importance of careful annotation instrument design.

The event is free to attend, simply register your details to receive a unique Zoom Webinar link.

Please note that you will be required to register for the event using an email address linked to a valid Zoom account.

Attendance at City events is subject to terms and conditions


  • Stephanie Eckman
    Researcher University of Maryland
    Stephanie Eckman is a Researcher and Data Scientist at the University of Maryland's Social Data Science Centre. She has a PhD in Statistics and Methodology and has worked in survey research for more than 20 years.