Guidelines for Academic Requesters

From WeAreDynamo Wiki
Revision as of 17:35, 8 August 2014 by Excited iguana (Talk | contribs)

Jump to: navigation, search

Doing Ethical Research with Amazon Mechanical Turk Workers and Communities v0.1

This guideline is a work in progress. Unfinished sections are highlighted in yellow. (Edits welcome in finished sections as well.) Please see the Timeline for our plan to finish this guideline.

Goal: Guidelines Turkers can share with researchers or the IRB

Lots of academic research happens through AMT or about Turkers, but ethics boards (IRBs) who review and approve research protocols often don't know how workers want to be treated. Let's collectively author guidelines that will educate researchers and let Turkers hold them accountable to a higher standard.

Timeline

  • Fri, Aug 8 - Get a first draft that can be circulated broadly to Turkers for comments.
  • Sat, Aug 9 - Announce the draft's availability on your forums.
  • Aug 9-12 - Get revisions and comments on the wiki through the forum or in the wiki itself.
  • Aug 16 - Freeze guidelines v1.0 and publish it on crowdresearch.org

TODOs

  • Read what others have added and make it clearer or more succinct.
  • How do people feel about researchers logging, scraping forum data, lurking on forums to write papers unannounced? Draft a guideline.
  • What guidance to provide IRB about rejections and blocks, the risk that poses to worker livelihood? Draft a guideline.
  • Check the forum, respond to open questions workgroup members have posted there.
  • What do we do when there is a violation?

Don't worry about adding the wrong thing. The wiki is easily editable. Write more and we can all fix it later together.

What counts as research?

If you are a machine learning researcher gathering training data for your newfangled convolutional neural network, do you need to follow these guidelines? What if you are running a controlled experiment within your HIT? Or a survey? What if you are just trying to better characterize Turker life?

There are many excellent discussions offsite about what is considered human subjects research. In brief, if this is something you would need IRB approval to perform offline, you probably also need IRB approval to perform it online. And if you need IRB approval to do it, then you should follow this guideline. Check with your local IRB. If you are performing human subjects research from industry rather than academia, there are industrial IRBs you could use if you desire.

Even if you do not need IRB, however, these guidelines tell you how Turkers want to be respected. Academic researchers see the value of their research to the world in a way that blinds them to the implications of their research to the people actually participating in it. These guidelines are for them too, IRB or not.

Guidelines

Basics of how to be a good requester

There are many basics to being a good requester and getting good results. Longtime Turker and Turkopticon moderator T has written advice for academics on his blog. Start there. He has advised: "Treat your workers with respect and dignity. Workers are not numbers and statistics. Workers are not lab rats. Workers are people and should be treated with respect." This is where these guidelines pick up.

Ethical engagement with worker forums

Worker forums are sites where workers socialize and collaborate on doing Turk work and living life. These include, for example, Turkopticon, TurkerNation, CloudMeBaby, Reddit's mTurk subreddits, and mTurkGrind. (Please note that not all communities participated in the drafting of these guidelines, so these are starting points but you should always discuss and pilot your research with some of the people who might be affected on it to do right by on the ground and evolving needs.)

Experiments might sow confusion and mistrust among participants in Mechanical Turk forums.

Why this matters (history): For example, one academic experiment simulated requesters with varying ratings in Turkopticon to measure the effects of ratings on worker behavior and outcomes. Turkers found some of the requesters and smelled something fishy but did not know if it was a scam, academic research, vandalism, or something else; through what amounted to at least 50 hours of sleuthing over two days, Turkers across reddit and turkopticon-discuss hypothesized that this was a research project. The researcher wanted to make positivist knowledge claims about ratings, workers, and the economics of Turking but neither he nor the IRB understood that:

  • simulating requesters would break the fragile trust that makes Turkopticon ratings meaningful to workers
  • that worker harm includes not only unpaid wages in AMT, but also the time they spent anxiously trying to track down these mysterious apparations

Principles:

  • Protect not only workers, but protect the quality of interaction in the worker forums that enable workers to make a living and a life through Mechanical Turk.
  • Do not assume that any experimental practice involving forums is legitimate unless expressly forbidden. Worker forum members cannot anticipate all the interventions researchers might imagine.
  • Discuss research protocols involving worker forums with workers from those forums themselves (best) or with forum administrators to get guidance on the risks, needs, and vulnerabilities of those forums.

Fair payment

Crowdsourcing workers are legally considered contractors and therefore are not protected by any minimum wage laws. Unfortunately some requesters may interpret this to mean they are allowed to pay extremely low rates and consider this to be "the norm of the market". This is not acceptable to the majority of the established worker community. These requesters survive because of the constant influx of new workers who have not established themselves in the workplace and the large population of international workers who view these lower payments as acceptable. [add more content about what's wrong].

Consider the estimated duration and difficulty of your task when deciding about payment.

On many turker forums a rate of $0.10 per minute is considered the bare minimum that most workers will work for. Although this is only $6 per hour and far below minimum wage standards in Western countries, it has currently become a guidline to present to requesters when discussing minimum payment for the general population on Mturk. The downside to using this guideline is that most expierenced workers will state that $0.15 - 0.20 per minute is their bare minimum requrement on Mturk and will not work for even a penny less per minute. So using a guidline of 0.10 per minute to grab turker interest will exclude a large portion of the most expierenced and knowledgable workers. No matter what guidline is used, it has always been up to the individual worker to decide how much their time is worth, but when large groups of workers are excluded from research because of poor payment, the results of the research cannot be considered a valid sampling of a population.

Since Mturk is a worldwide website, what may be an acceptable wage in Asia is not acceptable in many North American and European countries. If a requester would like to use the entire range of worldwide users they should pay the same wage in India as they do in Indiana. If a requester would like to use only workers from emerging economies, it would be acceptable to break from the Western payment norms and price work according to fair wages within those countries.

When requesters pay a fair wage and treat workers like people, both sides recieve positive results.

User interface and title of your HIT

Let's use a common format for describing academic HITs. It signals good intentions to the Turkers, it helps spread word to other academic requesters, and it builds positive credibility for the larger community.

Please title your HIT in this format:

[HIT title], [Estimated completion time], [hourly rate=reward/time]

For increased visibility, consider adding "Dynamo" or "Dynamo-compliant" to the end of the HIT title.

So that workers can find these HITs, include "dynamo" in the keyword set for the HIT.

Within the HIT text itself, somewhere please include "This HIT follows the Dynamo guideline for academic requesters." Including a link to the IRB approval is appreciated, so that we know who to contact.

Who should you contact?

If you're not sure if something is ethical research to engage with Turkers on, who should the academic contact? TODO

Here are some resources on ethics of research online:

http://plato.stanford.edu/entries/ethics-internet-research/

http://aoir.org/reports/ethics2.pdf

http://irb.uconn.edu/internet_research.html

Our process when academic HITs accidentally (or purposefully) run afoul of these conventions

  • Email a list to report it? A mod from the list can email the requester? TODO
  • Under what circumstances might we contact the academic's institutional review board (IRB) and ask them to intervene? TODO

History: Why do we need these guidelines?

Please fill in cases of the problems academic research have caused workers and worker communities. We can then see about grouping these cases by common attributes and the guidelines that come out from them. We might move cases from here up to the principles, but this can be a holding pen for "the problems"

- A positive example: researchers add a bonus to their task to reach $9.32/h, the highest state-level minimum wage in the United States (link on Reddit).

- Dr. Aaron Sojourner (Added above by Tense ringworm (talk) 02:25, 6 August 2014 (UTC))

- David Martin's Being a Turker paper [what from that paper? could you take a stab at quickly writing what seems like the key points of relevance here? another one of us could clean it up once we have a sense of what you mean. Tense ringworm (talk) 06:29, 4 August 2014 (UTC))

- Some IRBs seem to have policies that encourage - or even demand - low payments. I have been told by Requesters "I agree, $.05 is a ridiculously small amount of money, but I unfortunately needed to match the "standard" amount to get IRB approval." and by another "The amount of pay that I am offering was reviewed at length by the Institutional Review Board at my university ... there was concern that offering a large reimbursement for the HiT would be coercive (in that participants would feel that they have to comply in order to obtain a large reward.)"

Other guidelines as resources

Epigraph

"What we need to do is teach requesters about the human side of Mturk. Mturk encourages anybody that uses Mturk to think of us us as little computing units not as people." - Project2501 (a Turker)

"Dehumanization is the result of an unjust order that engenders violence in the oppressors, which in turn dehumanize the oppressed. Because it is a distortion of being more fully human, sooner or later being less human leads the oppressed to struggle against those who made them so. In order for this struggle to have meaning, the oppressed must not, in seeking to regain their humanity (which is a way to create it) become in turn oppressors of the oppressors, but rather restorers of the humanity of both. This, then, is the great humanistic and historical task of the oppressed: liberate themselves and their oppressors as well." - Freire's Pedagogy of the Oppressed

(we can delete the above if it is too corny, but I just found it inspirational! Tense ringworm (talk) 07:00, 1 August 2014 (UTC))

Signatories