Guidelines for Academic Requesters

From WeAreDynamo Wiki
Revision as of 06:26, 1 August 2014 by Excited iguana (Talk | contribs)

Jump to: navigation, search

Doing Ethical Research with Amazon Mechanical Turk Workers and Communities v0.1

Drafted by: Lilly Irani (UCSD), <add your name here>

Names are listed in no particular order. We can discuss if there is a particular order that makes sense as this develops.

Requests

Ethical engagement with worker forums

Worker forums are sites where workers socialize and collaborate on doing Turk work and living life. These include, for example, Turkopticon, TurkerNation, CloudMeBaby, Reddit's mTurk subreddits, and mTurkGrind. (Please note that not all communities participated in the drafting of these guidelines, so these are starting points but you should always discuss and pilot your research with some of the people who might be affected on it to do right by on the ground and evolving needs.)

Experiments might sow confusion and mistrust among participants in Mechanical Turk forums.

Why this matters (history): For example, one academic experiment simulated requesters with varying ratings in Turkopticon to measure the effects of ratings on worker behavior and outcomes. Turkers found some of the requesters and smelled something fishy but did not know if it was a scam, academic research, vandalism, or something else; through what amounted to at least 50 hours of sleuthing over two days, Turkers across reddit and turkopticon-discuss hypothesized that this was a research project. The researcher wanted to make positivist knowledge claims about ratings, workers, and the economics of Turking but neither he nor the IRB understood that:

  • simulating requesters would break the fragile trust that makes Turkopticon ratings meaningful to workers
  • that worker harm includes not only unpaid wages in AMT, but also the time they spent anxiously trying to track down these mysterious apparations

Principles:

  • Protect not only workers, but protect the quality of interaction in the worker forums that enable workers to make a living and a life through Mechanical Turk.
  • Do not assume that any experimental practice involving forums is legitimate unless expressly forbidden. Worker forum members cannot anticipate all the interventions researchers might imagine.
  • Discuss research protocols involving worker forums with workers from those forums themselves (best) or with forum administrators to get guidance on the risks, needs, and vulnerabilities of those forums.

Our process when academic HITs accidentally (or purposefully) run afoul of these conventions

  • Email a list to report it? A mod from the list can email the requester?
  • Under what circumstances might we contact the academic's institutional review board (IRB) and ask them to intervene?

Question bin <insert other themes here as you think of them...there are no "bad ideas", just material to be sorted through and refined>

  • Fair money / wages for participating in experiments
  • How do people feel about researchers logging, scraping forum data, lurking on forums to write papers unannounced?
  • What guidance to provide IRB about rejections and blocks, the risk that poses to worker livelihood?


History: Why do we need these guidelines?

Please fill in cases of the problems academic research have caused workers and worker communities. We can then see about grouping these cases by common attributes and the guidelines that come out from them. We might move cases from here up to the principles, but this can be a holding pen for "the problems"

Other guidelines as resources

Signatories