Guidelines for Academic Requesters

From WeAreDynamo Wiki
Jump to: navigation, search

Doing Ethical Research with Amazon Mechanical Turk Workers and Communities v0.1

Drafted by: Lilly Irani (UCSD, Tense ringworm (talk) 18:28, 1 August 2014 (UTC)), <add your name here>

Names are listed in no particular order. We can discuss if there is a particular order that makes sense as this develops.

Guidelines

Ethical engagement with worker forums

Worker forums are sites where workers socialize and collaborate on doing Turk work and living life. These include, for example, Turkopticon, TurkerNation, CloudMeBaby, Reddit's mTurk subreddits, and mTurkGrind. (Please note that not all communities participated in the drafting of these guidelines, so these are starting points but you should always discuss and pilot your research with some of the people who might be affected on it to do right by on the ground and evolving needs.)

Experiments might sow confusion and mistrust among participants in Mechanical Turk forums.

Why this matters (history): For example, one academic experiment simulated requesters with varying ratings in Turkopticon to measure the effects of ratings on worker behavior and outcomes. Turkers found some of the requesters and smelled something fishy but did not know if it was a scam, academic research, vandalism, or something else; through what amounted to at least 50 hours of sleuthing over two days, Turkers across reddit and turkopticon-discuss hypothesized that this was a research project. The researcher wanted to make positivist knowledge claims about ratings, workers, and the economics of Turking but neither he nor the IRB understood that:

  • simulating requesters would break the fragile trust that makes Turkopticon ratings meaningful to workers
  • that worker harm includes not only unpaid wages in AMT, but also the time they spent anxiously trying to track down these mysterious apparations

Principles:

  • Protect not only workers, but protect the quality of interaction in the worker forums that enable workers to make a living and a life through Mechanical Turk.
  • Do not assume that any experimental practice involving forums is legitimate unless expressly forbidden. Worker forum members cannot anticipate all the interventions researchers might imagine.
  • Discuss research protocols involving worker forums with workers from those forums themselves (best) or with forum administrators to get guidance on the risks, needs, and vulnerabilities of those forums.

Question bin <insert other themes here as you think of them...there are no "bad ideas", just material to be sorted through and refined>

  • Fair money / wages for participating in experiments
  • How do people feel about researchers logging, scraping forum data, lurking on forums to write papers unannounced?
  • What guidance to provide IRB about rejections and blocks, the risk that poses to worker livelihood?

Our process when academic HITs accidentally (or purposefully) run afoul of these conventions

  • Email a list to report it? A mod from the list can email the requester?
  - Email kills anonymity, Turkers may prefer an anonymous option to provide info, like Crimestoppers
  • Under what circumstances might we contact the academic's institutional review board (IRB) and ask them to intervene?

History: Why do we need these guidelines?

Please fill in cases of the problems academic research have caused workers and worker communities. We can then see about grouping these cases by common attributes and the guidelines that come out from them. We might move cases from here up to the principles, but this can be a holding pen for "the problems"

- Dr. Aaron Sojourner

- David Martin's Being a Turker paper

Other guidelines as resources

Epigraph

"What we need to do is teach requesters about the human side of Mturk. Mturk encourages anybody that uses Mturk to think of us us as little computing units not as people." - Project2501 (a Turker)

"Dehumanization is the result of an unjust order that engenders violence in the oppressors, which in turn dehumanize the oppressed. Because it is a distortion of being more fully human, sooner or later being less human leads the oppressed to struggle against those who made them so. In order for this struggle to have meaning, the oppressed must not, in seeking to regain their humanity (which is a way to create it) become in turn oppressors of the oppressors, but rather restorers of the humanity of both. This, then, is the great humanistic and historical task of the oppressed: liberate themselves and their oppressors as well." - Freire's Pedagogy of the Oppressed

(we can delete the above if it is too corny, but I just found it inspirational! Tense ringworm (talk) 07:00, 1 August 2014 (UTC))

Signatories