Guidelines for Academic Requesters

From WeAreDynamo Wiki
Revision as of 06:14, 21 August 2014 by Lazy urchin (Talk | contribs)

Jump to: navigation, search

About the project

(Current version 0.3)

"Treat your workers with respect and dignity. Workers are not numbers and statistics. Workers are not lab rats. Workers are people and should be treated with respect." - turker 'T', a Turkopticon moderator

This document consists of this main Guidelines for Academic Requesters page, and several subpages with important additional details, which are referenced with "Read more" links at relevant points throughout this main page:

Use http://guidelines.wearedynamo.org/ as a shortcut to quickly reach this document.

Goal: Guidelines that IRB will use to approve responsible AMT research

Plenty of academic research passes through AMT or is about Turkers, but ethics boards (IRBs) who review and approve research protocols often don't know how workers want to be treated. Turkers have collectively authored these guidelines to help educate researchers and let Turkers hold them accountable to a higher standard.

For Turkers: what can you do when these guidelines are violated?

If an academic requester is not being a positive member of the Mechanical Turk marketplace and community, Turkers may want to reach out to fix the situation. If a Turker wants to report concerns about a HIT that may be in violation, they can post details in a private thread (only readable by registered Dynamo members) in the Dynamo forum.

Dynamo members may then:

  • Encourage the worker to email the requester, and to CC ethics@wearedynamo.org if the worker would like copies to be posted in a private Dynamo thread for ongoing feedback. Dynamo can help provide a template email to make sure it's framed well, and provide feedback.
  • If there is no timely response or inadequate response from the requester, the Dynamo admins can help the worker adapt a template email to contact the requester's IRB.
  • If there is no timely response or inadequate response from the IRB, and if it's a serious problem, the Dynamo community helps the worker adapt a template email to colleagues of the researcher, or to other administrators at the university, as appropriate. (Use this option sparingly!)

Resources and template emails for these communications are available here: Resources for communicating with requesters

These are suggestions and resources available for MTurk workers to utilize if and when they choose, not an obligatory plan of action.

Guidelines

Clearly identify yourself to give workers a sense that you are accountable and responsible

Your HIT should include a consent or intro page with the following information:

  • the full name/s of the researcher/s responsible for the HIT's project;
  • the university/organization/s they're affiliated with and its state/country;
  • their department name, lab, project group, etc;
  • a direct line of communication, including an email address, phone number, and contact person within the IRB.

Also, convey as much information as you can in your:

  • requester display name
  • HIT description
  • HIT preview

Why? Workers generally are more willing to take a chance on a requester they're not familiar with (particularly one who hasn't yet been reviewed by any workers on Turkopticon). Academic requesters seem legitimate by virtue of their position. Also, academic requesters are part of a university 'chain of command' with IRB oversight and a means of redressing worker grievances should something go wrong.

What about my privacy? Turkers who want to know (for the above reasons) can often figure out much of this information for an academic requester who doesn't provide it; however, this takes workers' time and effort, and burns their good will.

Example: When a large batch of HITs was posted by a new requester with no Turkopticon reviews and whose only visible identification was their first-name-only requester display name, some Turkers hesitated, trying to decide if it was too risky to do more than a few. When a Turker was able to identify the requester's full name and affiliation with a major university, the Turkers felt more confident to do a larger quantity of those HITs.

Example: Researchers working on spam algorithms did not identify themselves in HITs. Turkers grew concerned that the HITs were coming from spammers trying to bypass filters. Turkers avoided doing the HITs and posted negative reviews and discussion comments.

Provide reasonable time estimates

State up front how long the task is likely to take for a careful person unfamiliar with the task. Know that task experts always underestimate how long it takes for novices to complete a task [Hinds 1999]. Err on the side of overestimation to avoid disappointment and frustration.

Why? Turkers calculate estimated earnings based on time estimates, and their target earnings inform their choice of HITs. If a HIT takes longer than estimated, Turkers may speed through it to keep it to the requester-provided estimate, hurting quality and damaging requester reputation. Read more

Approve work as soon as possible

Set your auto-approval time as short as reasonably possible. 7 days should generally be sufficient. Many requesters approve work in less than 3 days, and some in less than 24 hours. Many workers rely on MTurk to pay bills and manage their cash flow, so timely pay makes a big difference in their lives. Read more

Maintain worker privacy

Don't require workers to provide personally identifying information to complete your HITs. This includes:

  • email address
  • birth dates
  • real names
  • Facebook logins

Don't require workers to register on sites that require this kind of personal information to complete your HITs, or similarly require a Facebook login.

If you don't follow the Terms of Service, particularly in the aforementioned ways that pose potential threats directly to workers, some workers will give your requester account negative Turkopticon reviews with flags for ToS violations, and report your HITs to Amazon. Read more

IRB Considerations

Many times when running a study on MTurk researchers will use the Exempt form with IRB which allows a quicker response to the PI, these requests as per 45 CFR 46, Subpart A (Common Rule) require that no identifiable information be collection. Collecting data such as email addresses would violate this rule and would require the Expedited Review process. One way to do this and maintain worker confidentiality would be to use the message worker feature in AMT. Messaging though AMT is not a simple process but there are applications such as MTurkR which make this process much easier, for an example of how to do this please visit the [MTurkR wiki on Git Hub]. If there are any problems feel free to message someone on the MTurkR Git Hub or you may contact William Kyle Hamilton at whamilton@ucmerced.edu and he will assist you in setting everything up.

Abide by AMT Terms of Service

When you established a requester account with Amazon Mechanical Turk, you promised to adhere to Amazon's MTurk Terms of Service (ToS). To conform with these guidelines, AMT academic researchers shall provide their IRB with a copy of the ToS, as a requisite part of submitting their application for IRB approval.

The MTurk Terms of Service include some protections for Turker privacy and systems. See a list of prohibited uses of Amazon Mechanical Turk in the MTurk 'General Policies' FAQ page or in the 'General Policies' section of the MTurk Requesters FAQ page. Note that requiring users to download software is against AMT's Terms of Service. Some workers are willing to download software, but others will refuse as it can be a security risk to their systems. Read more

Ensure conditions for rejecting work are clear and fair

Rejections leave workers with a mark counting against them on their 'permanent record' at MTurk that may take them below a qualification threshold necessary for certain other HITs. Before deciding a rejection is justified, be sure you've considered several factors:

  • State any reasons for which you plan to automatically reject submissions.
  • Test your instructions and attention checks with compensated workers to ensure they are not ambiguous or unclear.
  • Make sure your survey will actually provide the promised completion code to workers who complete it, and that the code is correctly saved in your database. Learn how to do completion codes well
  • Keep lines of communication with workers open through email and forums. Workers run into 'edge cases', particularly in large batch HITs.
  • Don't reject workers solely based on majority rules, even if you use majority internally for your analyses.
  • Reject work only as a last resort. Know how to undo a rejection before you do. After thirty days, a rejection can never be reversed. Don't be in a hurry to pull the trigger.

Example: There have been several situations where requesters wrongly rejected large amounts of workers for 'incorrect completion codes'. The requester was randomly generating the codes and they were not being correctly stored in their database for matching.

Read more about ensuring rejections are fair

Do not block workers to avoid duplicate subjects

Blocks should only be used for bad-faith workers, as they can result in workers being suspended by Amazon. Suspensions of this type are equivalent to a permanent ban in most cases; this simple mistake can cost livelihoods.

Say up-front if you do not want duplicates. However, recognize that workers cannot easily remember whether they participated in your survey several months ago. There are several tools requesters can use when setting up their HITs to make this easier, rather than expecting workers to keep your records. Learn how to avoid duplicate subjects (retakes) fairly

Maintain a responsive line of communication with Turkers

Check the email account associated with your MTurk requester account frequently. Respond to messages from workers as quickly as possible, preferably in less than 24 hours. Visit worker forums to seek advice and find knowledgeable Turkers to vet your HIT.

Read more about how to communicate well with workers

Pay Turkers fairly. They are a workforce, not a volunteer study population

Crowdsourcing workers are a labor force. Many depend on income from crowdsourcing as critical income. Crowdsourcing workers are legally considered contractors and therefore are not protected by any minimum wage laws. When requesters pay a fair wage and treat workers like people, both sides receive positive results.

Pay (at least) community norms of minimum Turking wage

Many workers consider $0.10 a minute to the minimum to be considered ethical, though many studies pay more and there are excellent arguments to pay more (Read more). Tasks paying less than $0.10 a minute are likely to tap into a highly vulnerable work pool and constitutes coercion.

Since Turkers work independently, they are responsible for their own computers, electricity, taxes, health care, etc. Different workers consider fair pay anywhere from $6 an hour to $22 an hour. Learn why

If your task takes longer than you predicted, you can send workers bonuses to bring the wages up to ethical levels after the fact. In July 2014, a requester did this unexpectedly for workers who took one of their surveys, basing their target pay rate on Washington state's $9.32/hr minimum wage.

Clearly communicate possible bonuses

Explain what the potential amount will be and how to earn it, and how soon workers should expect it to be paid. Pay promptly. Read more

Compensate for qualifier/screener surveys

If you are using qualifier surveys, compensate all those who correctly complete the survey. Read more

Do not experiment with forum relationships for research

Forums only work because of delicate relationships of trust and mutual aid among participants. Sociological experiments such as breaching experiments can sow discord and destroy relationships. Positivist research that attempts to control and measure a forums effects can confuse workers, create anxieties in the community, and drain community energy as members try to make sense of the unusual intervention. To learn how a forum works, talk to administrators about your project, goals, and a plan for creating mutually beneficial research with workers.

Example: One academic experiment simulated requesters with varying ratings in Turkopticon to measure the effects of ratings on worker behavior and outcomes. Turkers found some of the requesters and smelled something fishy but did not know if it was a scam, academic research, vandalism, or something else; through what amounted to at least 50 hours of sleuthing over two days, Turkers across reddit and turkopticon-discuss hypothesized that this was a research project. The researcher wanted to make positivist knowledge claims about ratings, workers, and the economics of Turking, but neither he nor the IRB understood that:

  • simulating fabricated requesters and reviews broke the fragile trust that makes Turkopticon ratings meaningful to workers
  • that worker harm includes not only unpaid wages in AMT, but also the time they spent anxiously trying to track down these mysterious apparitions

Links to other resources on AMT and online research ethics

Epigraph

"Turking is work, even if it is for science, and academic researchers shouldn't assume that people are happy to do it for fun. They should pay and respect people's time." - Dr. Lilly Irani (of Turkopticon), Department of Communication, University of California at San Diego

"What we need to do is teach requesters about the human side of Mturk. Mturk encourages anybody that uses Mturk to think of us as little computing units, not as people." - Project2501 (a Turker)

"Dehumanization is the result of an unjust order that engenders violence in the oppressors, which in turn dehumanize the oppressed. Because it is a distortion of being more fully human, sooner or later being less human leads the oppressed to struggle against those who made them so. In order for this struggle to have meaning, the oppressed must not, in seeking to regain their humanity (which is a way to create it) become in turn oppressors of the oppressors, but rather restorers of the humanity of both. This, then, is the great humanistic and historical task of the oppressed: liberate themselves and their oppressors as well." - Freire's Pedagogy of the Oppressed

Signatories: Ratify these guidelines

Workers, to lend the strength of your support to these guidelines, please click on the "Sign" button in the Dynamo signing campaign, to sign with a pseudonym. (You will need to submit a HIT on MTurk to receive a code to register for a Dynamo account, ensuring there is one Dynamo account per worker.)

Researchers, to show your agreement to follow these guidelines in your future research, please send us an email at info@wearedynamo.org so we can add your signature. Please email us from your academic email account to help verify your identity.

Signatures from MTurk Workers

  • Gorgeous monarch butterfly (14 August 2014)
  • Courageous cockroach (14 August 2014)
  • Fancy cod (15 August 2014): I support this
  • Faithful fly (15 August 2014)
  • Dark bird of paradise (15 August 2014)
  • Elated sea urchin (15 August 2014)
  • Lonely wombat (15 August 2014)
  • Amused hedgehog (15 August 2014)
  • Jolly otter (16 August 2014)
  • Terrible cat (16 August 2014)
  • Funny giant panda (16 August 2014)
  • Obedient otter (16 August 2014)
  • Determined firefly (16 August 2014)
  • Glamorous mollusks (16 August 2014)

Signatures from Academic Researchers/Requesters

  • Niloufar Salehi, Stanford University (Light dragonfly) (13 August 2014)
  • Lilly Irani, University of California at San Diego; Turkopticon co-founder (Tense ringworm) (13 August 2014)
  • Michael Bernstein, Stanford University (Excited iguana) (14 August 2014)