Guidelines for Academic Requesters

From WeAreDynamo Wiki
Jump to: navigation, search

About the project

(Current version 0.3)

"Treat your workers with respect and dignity. Workers are not numbers and statistics. Workers are not lab rats. Workers are people and should be treated with respect." - turker 'T', a Turkopticon moderator

This document consists of this main Guideline for Academic Requesters page, and several subpages with important additional details, which are referenced with "Read more" links at relevant points throughout this main page:

Goal: Guidelines that IRB will use to approve responsible AMT research

Plenty of academic research passes through AMT or is about Turkers, but ethics boards (IRBs) who review and approve research protocols often don't know how workers want to be treated. Let's collectively author guidelines that will educate researchers and let Turkers hold them accountable to a higher standard.

We need your input

Please tell us about any feedback or issues on these guidelines on the forums that go along with this wiki:
Discuss these guidelines

Dynamo is the platform that has hosted the drafting process so far. Dynamo is a place where Turkers can join together to reach a mutual agreement on matters that affect them all, and act collectively to make a change. Turkers from all forums and lone wolves are all welcome here.

The legitimacy of actions on Dynamo lies on the fact that each Dynamo user is actually a Turker with a minimum of 100 HITs under their belt. To log in to Dynamo, workers complete a HIT on AMT; upon completing it a registration code is received. That registration code is the key to obtaining an anonymized username on the Dynamo sign up page. So, there are no spammers or lurkers, and each Turker receives one voice to make change. Register on Dynamo.

Once a worker signs into Dynamo, the "Activate wiki account"" in the wiki authoring campaign button is pressed to receive login authorization to edit this Wiki.

TODO: How do people feel about researchers logging, scraping forum data, and lurking on forums to write papers unannounced? Draft a guideline.

Guidelines

Clearly identify yourself to give workers a sense that you are accountable and responsible

Your HIT should include a consent or intro page with the following information:

  • the full name/s of the researcher/s responsible for the HIT's project;
  • the university/organization/s they're affiliated with and its state/country;
  • their department name, lab, project group, etc;
  • a direct line of communication, including an email address to contact the IRB (phone calls cost money)

Also, convey as much information as you can in your:

  • requester display name
  • HIT description
  • HIT preview

Why? Workers generally are more willing to take a chance on a requester they're not familiar with (particularly one who hasn't yet been reviewed by any workers on Turkopticon). Academic requesters seem legitimate by virtue of their position. Also, academic requesters are part of a university 'chain of command' with IRB oversight and a means of redressing worker grievances should something go wrong.

What about my privacy? Turkers who want to know (for the above reasons) can often figure out much of this information for an academic requester who doesn't provide it; however, this takes workers' time and effort, and burns their good will.

Example: When a large batch of HITs was posted by a new requester with no Turkopticon reviews and whose only visible identification was their first-name-only requester display name, some Turkers hesitated, trying to decide if it was too risky to do more than a few. When a Turker was able to identify the requester's full name and affiliation with a major university, the Turkers felt more confident to do a larger quantity of those HITs.

Example: Researchers working on spam algorithms did not identify themselves in HITs. Turkers grew concerned that the HITs were coming from spammers trying to bypass filters. Turkers avoided doing the HITs and posted negative reviews and discussion comments.

Provide reasonable time estimates

State up front how long the task is likely to take for a careful person unfamiliar with the task. Know that task experts always underestimate how long it takes for novices to complete a task [Hinds 1999]. Err on the side of overestimation to avoid disappointment and frustration.

Why? Turkers calculate estimated earnings based on time estimates, and their target earnings inform their choice of HITs. If a HIT takes longer than estimated, Turkers may speed through it to keep it to the requester-provided estimate, hurting quality and damaging requester reputation. Read more

Approve work as soon as possible

Set your auto-approval time as short as reasonably possible. 7 days should generally be sufficient. Many requesters approve work in less than 3 days, and some in less than 24 hours. Many workers rely on MTurk to pay bills and manage their cash flow, so timely pay makes a big difference in their lives. Read more

Maintain worker privacy

Don't require workers to provide personally identifying information to complete your HITs. This includes:

  • email address
  • birth dates
  • real names
  • Facebook logins

Don't require workers to register on sites that require this kind of personal information to complete your HITs, or similarly require a Facebook login.

If you don't follow the Terms of Service, particularly in the aforementioned ways that pose potential threats directly to workers, some workers will give your requester account negative Turkopticon reviews with flags for ToS violations, and report your HITs to Amazon. Read more

Abide by AMT Terms of Service

When you established an account with Amazon, you promised to adhere to Amazon's Terms of Service (TOS). The Amazon Terms of Service include some protections for Turker privacy and systems. See a list of prohibited requirements at mturk.com or Amazon's TOS summary. Note that requiring users to download software is against AMT's Terms of Service. Some workers are willing to download software, but others will refuse as it can be a security risk to their systems. Read more

Ensure conditions for rejecting work are clear and fair

Rejections leave workers with a mark counting against them on their 'permanent record' at MTurk that may take them below a qualification threshold necessary for certain other HITs. Before deciding a rejection is justified, be sure you've considered several factors:

  • State any reasons for which you plan to automatically reject submissions.
  • Test your instructions and attention checks with compensated workers to ensure they are not ambiguous or unclear.
  • Make sure your survey will actually provide the promised completion code to workers who complete it, and that the code is correctly saved in your database. Learn how to do completion codes well
  • Keep lines of communication with workers open through email and forums. Workers run into 'edge cases', particularly in large batch HITs.
  • Don't reject workers solely based on majority rules, even if you use majority internally for your analyses.
  • Reject work only as a last resort. Know how to undo a rejection before you do. After thirty days, a rejection can never be reversed. Don't be in a hurry to pull the trigger.

Example: There have been several situations where requesters wrongly rejected large amounts of workers for 'incorrect completion codes'. The requester was randomly generating the codes and they were not being correctly stored in their database for matching.

Read more about ensuring rejections are fair

Do not block workers to avoid duplicate subjects

Blocks should only be used for bad-faith workers, as they can result in workers being suspended by Amazon. Suspensions of this type are equivalent to a permanent ban in most cases; this simple mistake can cost livelihoods. Learn how to avoid duplicate subjects fairly

Maintain a responsive line of communication with Turkers

Check the email account associated with your MTurk requester account frequently. Respond to messages from workers as quickly as possible, preferably in less than 24 hours. Visit worker forums to seek advice and find knowledgeable Turkers to vet your HIT.

Read more about how to communicate well with workers

Pay Turkers fairly. They are a workforce, not a volunteer study population

Crowdsourcing workers are a labor force. Many depend on income from crowdsourcing as critical income. Crowdsourcing workers are legally considered contractors and therefore are not protected by any minimum wage laws. When requesters pay a fair wage and treat workers like people, both sides receive positive results.

Pay (at least) community norms of minimum Turking wage

Many workers consider $0.10 a minute to the minimum to be considered ethical, though many studies pay more and there are excellent arguments to pay more (Read more). Tasks paying less than $0.10 a minute are likely to tap into a highly vulnerable work pool and constitutes coercion.

Since Turkers work independently, they are responsible for their own computers, electricity, taxes, health care, etc. Different workers consider fair pay anywhere from $6 an hour to $22 an hour. Learn why

If your task takes longer than you predicted, you can send workers bonuses to bring the wages up to ethical levels after the fact. In July 2014, a requester did this unexpectedly for workers who took one of their surveys, basing their target pay rate on Washington state's $9.32/hr minimum wage.

Clearly communicate possible bonuses

Explain what the potential amount will be and how to earn it, and how soon workers should expect it to be paid. Pay promptly. Read more

Compensate for qualifier/screener surveys

If you are using qualifier surveys, compensate all those who correctly complete the survey. Read more

Do not experiment with forum relationships for research

Forums only work because of delicate relationships of trust and mutual aid among participants. Sociological experiments such as breaching experiments can sow discord and destroy relationships. Positivist research that attempts to control and measure a forums effects can confuse workers, create anxieties in the community, and drain community energy as members try to make sense of the unusual intervention. To learn how a forum works, talk to administrators about your project, goals, and a plan for creating mutually beneficial research with workers.

Example: One academic experiment simulated requesters with varying ratings in Turkopticon to measure the effects of ratings on worker behavior and outcomes. Turkers found some of the requesters and smelled something fishy but did not know if it was a scam, academic research, vandalism, or something else; through what amounted to at least 50 hours of sleuthing over two days, Turkers across reddit and turkopticon-discuss hypothesized that this was a research project. The researcher wanted to make positivist knowledge claims about ratings, workers, and the economics of Turking, but neither he nor the IRB understood that:

  • simulating fabricated requesters and reviews broke the fragile trust that makes Turkopticon ratings meaningful to workers
  • that worker harm includes not only unpaid wages in AMT, but also the time they spent anxiously trying to track down these mysterious apparitions

Links to other resources on AMT and online research ethics

For Turkers: what can you do when this guideline is violated?

If an academic requester is not being a positive member of the marketplace and community, Turkers may want to reach out to fix the situation. There will be a private thread on Dynamo for reporting concerns and violations. If a HIT is in violation, Turkers can post details here: (Link to come)

Dynamo members will then:

  • Encourage the worker to email the requester politely and cc the list. We can help provide a template email to make sure it’s framed well, and provide feedback.
  • If no response or inadequate response from the requester, the Dynamo admins help the worker adapt a template email to contact the requester’s IRB.
  • If no response or inadequate response from the IRB, and if it’s a serious problem, the Dynamo admins help the worker adapt a template email to colleagues of the researcher, or to other administrators at the university, as appropriate. (Use this option sparingly!)

Resources and template emails for these communications will be authored here: Resources for communicating with requesters

Epigraph

"Turking is work, even if it is for science, and academic researchers shouldn't assume that people are happy to do it for fun. They should pay and respect people's time." - Dr. Lilly Irani (of Turkopticon), Department of Communication, University of California at San Diego

"What we need to do is teach requesters about the human side of Mturk. Mturk encourages anybody that uses Mturk to think of us as little computing units, not as people." - Project2501 (a Turker)

"Dehumanization is the result of an unjust order that engenders violence in the oppressors, which in turn dehumanize the oppressed. Because it is a distortion of being more fully human, sooner or later being less human leads the oppressed to struggle against those who made them so. In order for this struggle to have meaning, the oppressed must not, in seeking to regain their humanity (which is a way to create it) become in turn oppressors of the oppressors, but rather restorers of the humanity of both. This, then, is the great humanistic and historical task of the oppressed: liberate themselves and their oppressors as well." - Freire's Pedagogy of the Oppressed

Signatories: Ratify these guidelines

The legitimacy of this guideline with requesters and IRB boards lies in Turkers reaching mutual agreement on it. After everyone has had a chance to comment, we will have a signing campaign through Dynamo. To lend the strength of your support to this guideline, these are your options:

  • click on "sign" in the Dynamo signing campaign
  • or add your chosen moniker to the last section of this wiki (you'll need a Dynamo login): here.
  • or send us an email at: info@wearedynamo.org so we can add your signature.

(You validate your Dynamo login by doing a HIT. This lets us make sure one Dynamo screen name is one person while keeping you anonymous in the system.)

  • Light dragonfly (Niloufar Salehi, Stanford; Light dragonfly (talk) 17:57, 15 August 2014 (UTC) )
  • Tense ringworm (Lilly Irani, UCSD and Turkopticon; Tense ringworm (talk) 17:52, 13 August 2014 (UTC))
  • Excited iguana (Michael Bernstein, Stanford; Excited iguana (talk) 18:33, 14 August 2014 (PDT))
  • Gorgeous monarch butterfly (talk) 01:47, 15 August 2014 (UTC)
  • Courageous cockroach (14 August 2014)
  • Fancy cod (15 August 2014): I support this
  • Faithful fly (15 August 2014)
  • Dark bird of paradise (15 August 2014)
  • Elated sea urchin (15 August 2014)
  • Lonely wombat (15 August 2014)
  • Amused hedgehog (15 August 2014)
  • Jolly otter (16 August 2014)
  • Terrible cat (16 August 2014)
  • Funny giant panda (16 August 2014)
  • Obedient otter (16 August 2014)
  • Determined firefly (16 August 2014)
  • Glamorous mollusks (16 August 2014)