- 1 Fair payment
- 1.1 Posting to Turk is not just like collecting a survey
- 1.2 What is ethical pay for Turkers in studies?
- 1.3 Why not pay a small, token amount?
- 1.4 What about my research sample?
- 1.5 What if I want Turkers from different countries?
- 1.6 What if my task takes longer than I thought, so the wages sink below what is fair?
Crowdsourcing workers are a labor force. While we cannot speak for all crowd workers, many depend on income from crowdsourcing as a supplementary or primary income. Crowdsourcing workers are legally considered contractors and therefore are not protected by any minimum wage laws. When requesters pay a fair wage and treat workers like people, both sides receive positive results.
Many workers consider $0.10 a minute to the minimum to be considered ethical, though many studies pay more and there are excellent arguments to pay more (see below). Tasks paying less than $0.10 a minute are likely to tap into a highly vulnerable work pool and constitutes coercion.
Posting to Turk is not just like collecting a survey
Posting an academic survey on Amazon Mechanical Turk is different than traditional forms of survey collection. Workers presume that they will be paid a fair wage and do not respect requesters who offer an extremely low rate, though what that means for different crowdworkers (India vs. US) may be different. Unfortunately some requesters may interpret this to mean they are allowed to pay extremely low rates and consider this to be "the norm of the market". This is not acceptable to the majority of the established worker community. These requesters survive because of the constant influx of new workers who have not established themselves in the workplace and the large population of international workers who view these lower payments as acceptable.
What is ethical pay for Turkers in studies?
Consider the estimated duration and difficulty of your task when deciding about payment. On many turker forums a rate of $0.10 per minute is considered the bare minimum that most workers will work for. Although this is only $6 per hour and far below minimum wage standards in Western countries, it is the current guideline presented to requesters across a number of forums. Many more experienced and knowledgeable Turkers, by contrast, refuse to work for less than $0.15-$0.20 per minute or more.
To date, many academic requesters' published papers, and many of the few university IRB websites that have any specific guidelines about MTurk, have stated that they paid or recommend paying rates equivalent to $2-$3 or less per hour ($0.03-$0.05 or less per minute) on MTurk, because it's slightly better than or similar to reported average rates (perpetuating the status quo), and sometimes mention reasoning such as that they think the workers all either don't really need or care about the money or are in low-income countries where this seems like a lot of money. There is a large US university that has routinely posted survey HITs for less than $0.01 per expected minute. There has been a lot of debate in turker forums about what a fair, or even acceptable, rate of pay is or should be. So far, the most commonly mentioned rate among workers as a suggested minimum has been $6/hr ($0.10/min), largely not because this is what most workers really prefer and are satisfied with, but as an easy-to-remember figure and a relatively realistic target in light of the preponderance to date of both academic and non-academic HITs paying much less so far. There isn't one answer appropriate for all situations, but here are some points to consider when trying to decide what a fair and ethical rate of pay on your HITs would be for US-based workers:
- Honest US-based turkers will generally be paying taxes on their MTurk earnings as self-employment.
- There is a lot of unpaid overhead time involved in turking, including: looking for the next suitable HITs to do, taking uncompensated qualification tests to hopefully qualify for certain HITs, checking reviews for unfamiliar requesters to decide whether to work on their HITs, writing reviews of requesters, communicating with other workers on forums, dealing with some of Amazon's security measures such as periodic Captchas and forced logouts that can interrupt workflow, dealing with occasional malfunctions of the worker's ISP/browser/computer, communicating with requesters (or in about half the cases, apparently futilely sending messages into a void) about questions/problems/suggestions, keeping track of the work they've done and the payments and bonuses they have or haven't received so far, checking their records of work they've done to see if it's safe to take a survey that threatens rejections if you take it more than once, and more. All break time (even going to the restroom) is also uncompensated.
- The more specialized knowledge/skills/characteristics, and/or the more stringent the qualifications (such as higher number of HITs approved, higher approval rate, scores on requesters' custom quals, and/or Masters) that your HITs will expect or require, the higher the pay rate for it generally should be if you want to continue to be fair (a fair minimum pay rate logically would only be considered as fair for HITs with minimum requirements, just like more-qualified/experienced workers in the traditional workforce generally expect to receive higher pay than less-qualified/experienced workers).
- Although self-employment work is not legally obligated to comply with minimum wage laws, they are commonly used as a benchmark in evaluating what pay would be fair and ethical. Points to consider regarding the minimum wage include:
- As of Aug 2014, the current national minimum wage has been $7.25/hr (~$0.12/min) since July 2009, due to the final of three gradual tiers of increases that were passed in May 2007.
- As of Aug 2014, there is currently a movement trying to raise the national minimum wage to $10.10/hr (~$0.17/min), but a bill that would've done that by late 2016 is stalled in Congress due to the political situation. The President was still able to set $10.10/hr as a minimum wage for employees of companies contracting on federal government projects, which will take effect for contracts that are new or renegotiated after Jan 1, 2015. - Wage and Hour Defense Blog
- An increasing number of states/territories, and even some cities, have stepped in to raise their own minimum wages higher than the national one. The current highest state/territory minimum wages as of Aug 2014 are $9.00/hr ($0.15/min) in California (increasing to $10.00/hr (~$0.17/min) on Jan 1, 2016), $9.32/hr (~$0.16/min) in Washington state, and $9.50/hr (~$0.16/min) in DC effective July 1, 2014 (increasing to $10.50 (~$0.18/min) on July 1, 2015, to $11.50 (~$0.19/min) on July 1, 2016, and annual inflation-indexed increases thereafter). - National Conference of State Legislatures
- If increases in the national minimum wage had kept pace with basic inflation of consumer prices since 1968, it should be $10.86 (~$0.18/min) as of 2013. - National Employment Law Project
- If increases in the national minimum wage had kept pace with nationwide productivity growth in all industries since the 1940s, it should be $16.54 (~$0.28/min) as of 2012. - Center for Economic and Policy Research
- If only considering 'non-farm' productivity growth (i.e. excluding agricultural workers from the calculation), the minimum wage should be $21.75 (~$0.36/min) as of 2012. - Center for Economic and Policy Research
- "If minimum-wage workers received only half of the productivity gains over the period, the federal minimum would be $15.34 [~$0.26/min]. Even if the minimum wage only grew at one-fourth the rate of productivity, in 2012 it would be set at $12.25 [~$0.20/min]." - Center for Economic and Policy Research
- The per-subject costs for other non-MTurk ways researchers can recruit survey participants reportedly tend to be much higher than you would be paying MTurk workers even at much more fair and ethical pay rates than is currently prevalent. Researchers who say they 'don't have the funding' to pay better rates on MTurk should consider that the alternatives are often to pay even more for a participant pool that may be less diverse and in some cases less attentive than MTurk workers. Even in the case of unfunded student projects, please try to consider that the total difference between fair and unfair pay will usually be less than you might think, particularly compared to the other costs you've committed to in pursuing your education; even just the textbooks.
- The availability of non-US workers on MTurk has apparently been gradually decreasing since Amazon stopped accepting registrations of new international worker accounts in late 2012. And with the exception of India (the only country besides the US that has ever been able to receive direct monetary payment), there were never a large percentage of workers from any other particular country. So even if you don't specifically require workers to be in the US to accept your HITs, a large and re-growing proportion of them will be in the US, unless you specifically exclude US workers from your HITs.
Learning more about the demographics and other statistics of the turker workforce can help requesters make more informed decisions about how to structure and compensate their HITs. Many turkers are indeed casual participants performing a small number of HITs, but studies indicate the vast majority, perhaps 80%, of the HITs completed on MTurk are performed by turkers who are in the top 10% or so of productivity among active turkers, each completing hundreds or thousands of HITs per week; and many of those put forth that much effort because the money is very important to them to make ends meet, whether they have other significant sources of income or not.
Several dozen academic papers and blog posts, covering much of the above information and other related topics (many demographics, as well as work consistency, work distribution, etc), are listed with links and quotes of the relevant portions, at 'Demographics of Mechanical Turk' by turker 'clickhapper' at mTurk Grind.
Why not pay a small, token amount?
Many Turkers won't work for less than $0.10 / minute. That means if you pay $.02 / minute, you are getting workers who are too desperate to boycott. This constitutes coercion. If you really cannot pay the minimum, then it is better to pay nothing because at least then you get true volunteers.
What about my research sample?
No matter what guideline is used, it has always been up to the individual worker to decide how much their time is worth, but when large groups of workers are excluded from research because of poor payment, the results of the research cannot be considered a valid sampling of a population.
What if I want Turkers from different countries?
Since MTurk is a worldwide website, what may be an acceptable wage in Asia is not acceptable in many North American and European countries. If a requester would like to use the entire range of worldwide users, they should pay the same wage in India as they do in Indiana. If a requester would like to use only workers from emerging economies, it would be acceptable to break from the Western payment norms and price work according to fair wages within those countries.
What if my task takes longer than I thought, so the wages sink below what is fair?
If an amount of pay you expected to be a fair rate turns out not to be because you accidentally underestimated how long your survey would take for reasonably-efficient workers to complete, consider adjusting for this situation by sending bonuses to the workers to make up the difference (could base the bonus amount on the mean or median time to complete, in case a few workers are unusually inefficient). In July 2014, a requester did this unexpectedly for workers who took one of their surveys, basing their target pay rate on Washington state's $9.32/hr minimum wage.