Difference between revisions of "Guidelines for Academic Requesters"

From WeAreDynamo Wiki
Jump to: navigation, search
(Ethical engagement with worker forums)
Line 8: Line 8:
 
====Experiments might sow confusion and mistrust among participants in Mechanical Turk forums.====
 
====Experiments might sow confusion and mistrust among participants in Mechanical Turk forums.====
 
'''Why this matters (history)''': For example, one academic experiment simulated requesters with varying ratings in Turkopticon to measure the effects of ratings on worker behavior and outcomes. Turkers found some of the requesters and smelled something fishy but did not know if it was a scam, academic research, vandalism, or something else; through what amounted to at least 50 hours of sleuthing over two days, Turkers across reddit and turkopticon-discuss hypothesized that this was a research project. The researcher wanted to make positivist knowledge claims about ratings, workers, and the economics of Turking but neither he nor the IRB understood that:
 
'''Why this matters (history)''': For example, one academic experiment simulated requesters with varying ratings in Turkopticon to measure the effects of ratings on worker behavior and outcomes. Turkers found some of the requesters and smelled something fishy but did not know if it was a scam, academic research, vandalism, or something else; through what amounted to at least 50 hours of sleuthing over two days, Turkers across reddit and turkopticon-discuss hypothesized that this was a research project. The researcher wanted to make positivist knowledge claims about ratings, workers, and the economics of Turking but neither he nor the IRB understood that:
** simulating requesters would break the fragile trust that makes Turkopticon ratings meaningful to workers
+
* simulating requesters would break the fragile trust that makes Turkopticon ratings meaningful to workers
** that worker harm includes not only unpaid wages in AMT, but also the time they spent anxiously trying to track down these mysterious apparations
+
* that worker harm includes not only unpaid wages in AMT, but also the time they spent anxiously trying to track down these mysterious apparations
  
 
'''Principles''':
 
'''Principles''':
* Protect not only workers, but protect the quality of interaction in the worker forums that enable workers to make a living and a life through Mechanical Turk.  
+
* Protect not only workers, but protect the quality of interaction in the worker forums that enable workers to make a living and a life through Mechanical Turk.  
* Do not assume that any experimental practice involving forums is legitimate unless expressly forbidden. Worker forum members cannot anticipate all the interventions researchers might imagine.  
+
* Do not assume that any experimental practice involving forums is legitimate unless expressly forbidden. Worker forum members cannot anticipate all the interventions researchers might imagine.  
* Discuss research protocols involving worker forums with workers from those forums themselves (best) or with forum administrators to get guidance on the risks, needs, and vulnerabilities of those forums.  
+
* Discuss research protocols involving worker forums with workers from those forums themselves (best) or with forum administrators to get guidance on the risks, needs, and vulnerabilities of those forums.
  
 
=== Question bin <insert other themes here as you think of them...there are no "bad ideas", just material to be sorted through and refined>===
 
=== Question bin <insert other themes here as you think of them...there are no "bad ideas", just material to be sorted through and refined>===

Revision as of 22:34, 31 July 2014

Doing Ethical Research with Amazon Mechanical Turk Workers and Communities v0.1

Drafted by: Lilly Irani (UCSD), <add your name here>

Requests

Ethical engagement with worker forums

Worker forums are sites where workers socialize and collaborate on doing Turk work and living life. These include, for example, Turkopticon, TurkerNation, CloudMeBaby, Reddit's mTurk subreddits, and mTurkGrind. (Please note that not all communities participated in the drafting of these guidelines, so these are starting points but you should always discuss and pilot your research with some of the people who might be affected on it to do right by on the ground and evolving needs.)

Experiments might sow confusion and mistrust among participants in Mechanical Turk forums.

Why this matters (history): For example, one academic experiment simulated requesters with varying ratings in Turkopticon to measure the effects of ratings on worker behavior and outcomes. Turkers found some of the requesters and smelled something fishy but did not know if it was a scam, academic research, vandalism, or something else; through what amounted to at least 50 hours of sleuthing over two days, Turkers across reddit and turkopticon-discuss hypothesized that this was a research project. The researcher wanted to make positivist knowledge claims about ratings, workers, and the economics of Turking but neither he nor the IRB understood that:

  • simulating requesters would break the fragile trust that makes Turkopticon ratings meaningful to workers
  • that worker harm includes not only unpaid wages in AMT, but also the time they spent anxiously trying to track down these mysterious apparations

Principles:

  • Protect not only workers, but protect the quality of interaction in the worker forums that enable workers to make a living and a life through Mechanical Turk.
  • Do not assume that any experimental practice involving forums is legitimate unless expressly forbidden. Worker forum members cannot anticipate all the interventions researchers might imagine.
  • Discuss research protocols involving worker forums with workers from those forums themselves (best) or with forum administrators to get guidance on the risks, needs, and vulnerabilities of those forums.

Question bin <insert other themes here as you think of them...there are no "bad ideas", just material to be sorted through and refined>

  • Fair money / wages for participating in experiments
  • How do people feel about researchers logging, scraping forum data, lurking on forums to write papers unannounced?
  • What guidance to provide IRB about rejections and blocks, the risk that poses to worker livelihood?


History: Why do we need these guidelines?

Please fill in cases of the problems academic research have caused workers and worker communities. We can then see about grouping these cases by common attributes and the guidelines that come out from them. We might move cases from here up to the principles, but this can be a holding pen for "the problems"

Signatories