Academic guidelines brainstorm and staging
This space is dedicated to brainstorming and staging future changes to the Academic Guidelines. In order for these changes to be added to the frozen guidelines, mutual agreement must be reached, the version number will increase, and all those who have signed the guideline will receive email notification.
IRBs to contact about the guidelines
Not at all exhaustive. Just a place to start tracking very obvious candidates
- https://uwaterloo.ca/research/sites/ca.research/files/uploads/files/crowdsourcing_guidelines_access_check_done.pdf - U Waterloo's guidelines include advice to offer a token payment but not a very high payment to avoid workers working strictly for remuneration. (!) Clearly violates the guidelines.
- http://couhes.mit.edu/guidelines/couhes-policy-using-amazons-mechanical-turk - suggests paying "comparable to that of other tasks offered" in a range of $0.02-$0.08/min; suggests requesters check emails once a week; inclusion of outdated TN link near end indicates likely last updated ~3-5 years ago
Potential ideas to consider in future guidelines revisions
- I composed the following to email an academic requester I thought would be interested in the guidelines after I read a piece that he wrote. I'm copying the part of that email I feel others could reuse to let academic requesters know about the guidelines here:
I thought you’d be interested in a set of guidelines for academic requester composed by AMT workers last summer. The guidelines explain top principles for how to treat workers well, with detailed explanations of why. Our hope is that IRBs will find this document and hold researchers accountable. We are asking ethical researchers like yourself to view the guidelines, join the signatories who have pledged to work by them, and use the guidelines to spread the message and educate colleagues. You can view the guidelines here:
- DoesNotExist needs to be added to Avoiding Duplicates. Also, we might want to make mention of the fact that a Turker takes time to find the HIT, accept it and then return it. This means that time is unpaid since they can't submit the HIT. Any of the methods other than DNE are subject to this waste of time, but using DNE means they can't accept the HIT and therefore don't waste as much time on it. --Dark bird of paradise (talk) 23:27, 30 August 2014 (UTC)
- Jeff Bigham pointed out that the emails to communicate with requesters shouldn't be used willy-nilly. In an earlier draft there were time recommendations to wait between emails. Can we re-add those? --Excited iguana (talk) 1 September 2014
- Dr David Brake, Professor of Journalism, at Humber College has signed the guidelines and made the following suggestion: I suggest modifying the guidelines to increase the minimum suggested payment level. If (in the US) you said you were paying f2f research assistants that wouldn’t you be told it was too low? It is below even the US minimum wage! --Light dragonfly (talk) 1 September 2014
- Unfortunately, many HITs, even from academic sources, currently don't come close to even that modest minimum. Fair_payment#What_is_ethical_pay_for_Turkers_in_studies.3F , which is referenced from the main page's section about pay, explains where the $6/hr minimum came from (it is, and has been for years, the most commonly stated target minimum number among turkers, some of whom have very mixed feelings about trying to ask for a higher minimum), and lists many reasons why researchers should strongly consider paying more than that minimum in many situations. The idea is for requesters to read that page and make a more-informed decision on what they would feel ethical to pay for their particular HITs, without setting firm demands for a higher minimum than the turker community would necessarily agree with at this time. If a researcher decides the ethical thing for themselves to do is to pay turkers at rates equivalent to their local research assistants, turkers would likely be pleased. --Gorgeous monarch butterfly (talk) 06:16, 2 September 2014 (UTC)
- Ken Arnold has created a template for academic HITs, maybe we can suggest using it in the guidelines: http://i.snag.gy/4FGoV.jpg, html code: https://gist.github.com/kcarnold/471352dba8391ae54d42
- Add a template for reporting the use of MTurk in academic articles. This could instruct researchers to report the number of HITs, the number of Turkers, how long their tasks took, how much they paid, etc.
- Edit the template for contacting researchers and invite them to sign the guidelines at the end.
- The AMT Policies page gives a list of examples of types of HITs that violate the Terms of Service. One of these examples was “HITs that require Workers to download software”. At some point after the last Archive.org capture on March 17, 2015, Amazon changed this sentence to say “HITs that require Workers to download software that contains any malware, spyware, viruses, or other harmful code.”. This change now means some parts of the Dynamo Guidelines for Academic Requesters are outdated and need changing. Read more: http://www.wearedynamo.org/forum/actions/topics/call-for-comments-turker-authored-guidelines-for-research-on-amt?page=5#post-527
Archived: Draft blog post for CrowdResearch
Update: The resulting post, 1 September 2014: http://crowdresearch.org/blog/?p=9039
Discussion / context: Excited_iguana suggests that a good blog post needs to have one message only because people read them fast. The goal of this blog post is to announce the guidelines to the turk research community. tense_ringworm's goal for this is to have it be highly googleable. What are relevant keywords we might include to make sure interested parties land on this page? (Search Engine Optimization activism!)
Turkers' guidelines for academic requesters on Amazon Mechanical Turk
If you've spent time talking with Turkers, you probably know that academic requesters have been a continuous source of strain. Research surveys with horrendous pay and arbitrary rejections are common. Despite Mechanical Turk's attractive availability, a large number of researchers make innocent missteps and cause serious stress. Recently, the tension came to a head on Turkopticon. An IRB-approved researcher experimented on the platform unannounced. The result was Turker confusion, strife, and wasted time, in a system where time is what it takes to make ends meet.
Before the guidelines, Turkers would deal with research problems on a case-by-case basis through e-mail or by calling human subjects review boards (e.g. IRBs, HRPPs) for help.
These guideline make Turkers' expectations and rights available in advance to mitigate these tensions from the start. They address how to be a good requester, how to pay fairly, and what Turkers can do if HITs are questionable. They apply to Turkers both as experimental subjects or data processing workers who fuel academic research.
We'll publicly maintain these guidelines so IRBs and researchers can easily find them, and Turkers can easily point to them in advocating for themselves.
Read the guidelines: http://guidelines.wearedynamo.org
They were developed over several weeks, and have been circulated and debated by workers. Lots of Turkers have been signing it to show their support.
As a requester, you are part of a very powerful group on AMT. Your signature in support of this document will help give Turkers a sense of cooperation and goodwill, and make Mechanical Turk a better place to work.
Today is Labor Day, a US holiday to honor the achievements of worker organizations. Honor Turkers by signing the guidelines as a researcher, and treating Turkers with the respect they deserve.
If you have any questions, you can email them to firstname.lastname@example.org or submit a reply to this post.
- The Dynamo Collective