Crowdsourcing

There are many terms for what we're talking about here: clickworking, collaboration, digital piecework, microlabor, and more.

Booksprints

Booksprinted books

Surveys and research

Regulation

Amazon Mechanical Turk

Went online in 2005. Name comes from the Turk, a chess-playing automaton of the 18th century, which was made by Wolfgang von Kempelen. Beat Napoleon and Benjamin Franklin.

Exploring the mturk.com website

  • The tagline is still “artificial artificial intelligence”
    • Small updates and adjustments are occasionally being reported.
  • DARPA case study - “Mechanical Turk Workers translated 1.5 million words of Arabic, which allowed DARPA to build their social media translation database in 8 weeks.” Also, US Army Research Lab
    • “Workers in India have the option of receiving bank checks denominated in Indian Rupees or disbursing their earnings to an Amazon.com gift certificate. All other international Workers can only disburse to an Amazon.com gift certificate. Please note that all disbursements are made in whole cents only.”

Communities and activism

      • “Plenty of academic research passes through AMT or is about Turkers, but ethics boards (IRBs) who review and approve research protocols often don't know how workers want to be treated. Turkers have collectively authored these guidelines to help educate researchers and let Turkers hold them accountable to a higher standard.”
      • Recommends an escalation in case of problem: contact requester (cc-ing Dynamo) > IRB > colleagues.
      • A list of key recommendations. See doc.
      • “Many workers consider $0.10 a minute to the minimum to be considered ethical”
      • “Forums only work because of delicate relationships of trust and mutual aid among participants. Sociological experiments such as breaching experiments can sow discord and destroy relationships. Positivist research that attempts to control and measure a forums effects can confuse workers, create anxieties in the community, and drain community energy as members try to make sense of the unusual intervention.” — and tells a horror story of this being a problem.
      • comment on making researchers aware that there is a lot of unpaid work involved with MTurk life. Also, taxes.
        • CUNY recommends not doing data collection on MTurk to protect user privacy.
        • Pomona appears to allow lower rates for MTurk than other research contexts.
        • Lots more interesting examples.
      • “University IRBs review and approve proposals for scientific research, much of which gets done on AMT. Several times, researchers (sometimes unwittingly) caused damage and pain in Turkland, even with IRB approval. After a recent research mishap on Turkopticon, a group of Turkers got together to draft guidelines for academic/research requesters working with Turkers.”
      • “I’m actually in a bit of a unique position where I do work on mturk and also have been a requester (hooray grad school)“—raises the question of exemption from IRB review
      • Collecting emails is a violation of the terms of service, but “AMT workers are highly individualistic. Most of them, at least occasionally, work HITs which violate ToS .”
      • “I’d like to see some comprehensive requirements for tasks that are posted.”
      • “I would like to see this happening and would volunteer time and effort to help make it happen. I have been extremely disappointed in the Mturk platform and the opaque and careless way that it is run. It didn’t take long for me to realize that it could be run much, much better to everyone’s benefit.”
      • “if we’re the ones doing it [building the platform], we need to be compensated for our time”

Data

Readings

  • Amazon.com. ”Amazon Mechanical Turk Participation Agreement.” December 2, 2014.
    • “Amazon Mechanical Turk and its Affiliates are not involved in the transactions between Requesters and Providers. As a result, we have no control over the quality, safety or legality of the Services, the ability of Providers to provide the Services to Requesters' satisfaction, or the ability of Requesters to pay for Services. We are not responsible for the actions of any Requester or Provider. We do not conduct any screening or other verification with respect to Requesters or Providers, nor do we provide any recommendations. As a Requester or a Provider, you use the Site at your own risk.”
    • “Upon completion of Services to Requesters' reasonable satisfaction, Requesters must pay Providers for their Services.” — “If a Requester is not reasonably satisfied with the Services, the Requester may reject the Services. As a Requester, you will be charged a fee for your use of Amazon Mechanical Turk in connection with each request for Services.”
    • “You recognize and agree that Amazon Mechanical Turk will implement mechanisms allowing us and others to track your requests for, or your performance of, Services and rate your performance as a Requester or Provider, and Amazon Mechanical Turk reserves the right to collect feedback regarding your performance and to post such feedback on the Site. You may not take any actions that may undermine the integrity of the feedback system.”
    • “Your use of the Site is at your own risk. Because Amazon Mechanical Turk is not involved in the actual transaction between Providers and Requesters, Amazon Mechanical Turk will not be involved in resolving any disputes between participants related to or arising out of the Services or any transaction.”
    • “We (and our Affiliates) act only in the capacity of a payment processor in facilitating the transactions between Requesters and Providers, and are not otherwise involved in the actual transactions.”
    • “You may use information or other data acquired from your use of the Site solely to the extent necessary for you to use the Site and for no other purpose, including but not limited to, for purposes of solicitation, advertising, marketing, unsolicited e-mail or spamming, harassment, invasion of privacy, or otherwise objectionable conduct.”
    • “THE SITE, THE PAYMENT SERVICE AND THE SITE SERVICES ARE PROVIDED ON AN “AS IS” BASIS.”
  • Bigham, Jeffrey P. “My MTurk (half) Workday.” July 8, 2014.
    • “Last Tuesday, I spent 4 hours, half a work day, as a Mechanical Turk worker.”
    • After trying some turking, “students are less likely to refer to turkers as 'lazy' and more likely to think about the usability and transparency of their tasks.”
    • “At hour three, I started sorting by least number of HITs. My goal here was to find academic studies because I thought maybe they would pay reasonably, and probably wouldn’t check too carefully whether the HIT was done perfectly.”—“Most of what came up were surveys, though, and almost all of them paid extremely poorly.”
    • “In the end, I submitted HITs totaling $17.29, and received $2.17 in bonuses. This works out to an effective hourly wage of $4.87. As of one week later, none of my HITs have been rejected, but only about half have been accepted. I’ve only received $8.50 of the $17.29 (~49%).”
    • “My pay was pretty low, but that wasn’t the most frustrating part of it. For me, it was the lack of transparency regarding how well I was doing, and poorly designed HITs that were the most frustrating. I had no idea which jobs would be good jobs, which meant that I wasted a lot of time on duds.”
    • “I’ll also admit to occasionally being the so-called lazy turker. When I saw my pay rate dipping below $6/hour, I just tried to finish the task as quickly as I could.”
    • “It seems like Mechanical Turk could reasonably introduce changes that could help with these problems. For instance, it would be great if there was a better flagging mechanism, and if Mechanical Turk took it more seriously. Perhaps they could consider charging requesters who post bad HITs more on subsequent HITs.”
    • “Amazon doesn’t seem to like to update Mechanical Turk very often, however, so I’m not holding out too much hope. As I’ve heard Rob Miller say, 'The great thing about Mechanical Turk is that they so rarely update things that I can still use my slides about them from 2008.'”
    • “Most of the problems I experienced seem fairly trivial to fix. I think we need to make certain going forward with research that we’re not fixing issues with the site that Amazon or requesters could fix overnight if they were incentivized to do so.”
  • Buhrmester, Michael. “Amazon Mechanical Turk Guide for Social Scientists.” January 18, 2012.
  • Buhrmester, Michael, Tracy Kwang and Samuel D. Gosling. “Amazon's Mechanical Turk : A New Source of Inexpensive, Yet High-Quality, Data?.” Perspectives on Psychological Science 6, no. 3 (2011).
    • “MTurk participants are slightly more demographically diverse than are standard Internet samples and are significantly more diverse than typical American college samples”
    • “participation is affected by compensation rate and task length, but participants can still be recruited rapidly and inexpensively”
    • “realistic compensation rates do not affect data quality”
    • “the data obtained are at least as reliable as those obtained via traditional methods”
    • “Our analyses…of worker motivation suggest that they are internally motivated (e.g., for enjoyment)”
  • Chandler, Jesse, Pam Mueller, and Gabriele Paolacci. “Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers.” Behavior Research Methods 46, no. 1 (2013): 112-130.
    • “We show that crowdsourced workers are likely to participate across multiple related experiments and that researchers are overzealous in the exclusion of research participants. We describe how both of these problems can be avoided using advanced interface features that also allow prescreening and longitudinal data collection.”
    • “We demonstrate that although cross talk is a minimal concern (in that workers are more interested in instrumental features such as payment and length of a HIT, rather than its content), duplicate workers are more common than researchers may assume. Repeated participation by workers is associated with increased self-reports of exposure to common experimental paradigms, as well as changes in responses to a paradigm likely to be vulnerable to practice effects”
    • “there are practical reasons why the research community should avoid overusing shared participant pools such as MTurk. For more commonly used methods and measures, the pool of MTurk workers presents a “commons dilemma” for researchers: It should not be assumed that respondents are naïve, and groups of researchers would be better off if they could coordinate their recruitment efforts.”
    • “we recommend that researchers who care about participant naïveté avoid commonly used paradigms and, at minimum, make an effort to measure whether participants have participated in similar experiments before. Furthermore, researchers using unique paradigms should track who has previously participated in their experiments”
    • “we believe that the single most important advantage of MTurk over other online recruitment tools is often ignored. Although the identity of workers is typically not known, their responses across different HITs can be monitored, integrated, and managed”
  • Cushing, Ellen. “Dawn of the Digital Sweatshop.” East Bay Express. August 1, 2012.
  • Emanuel, Gabrielle. “Post A Survey On Mechanical Turk And Watch The Results Roll In.” National Public Radio. March 5, 2014.
    • “Not all research is suited to Mechanical Turk. If your study involves children, that's a no go. Need to observe your subject's facial expressions? Can't do that on MTurk. Those researchers can, of course, just do their research the old-fashioned way. But Phillips worries some researchers might be shying away from tough topics.”
  • Ipeirotis, Panos. “The New Demographics of Mechanical Turk .” A Computer Scientist in a Business School. March 9, 2010.
    • “Since the last survey, a few things have changed. First, Amazon allows now workers in India to get paid in cash in rupees, essentially encouraging many people from India to start using Mechanical Turk as workers. Second, the recession has affected many households, leaving many people at home looking for cash to cover their needs.”
    • “in February 2010, I conducted a new survey on Mechanical Turk, paying the workers 10 cents for participating.”
    • “In the past 70%-80% of the workers were coming from the US, but now the percentage is closer to 50% and it may decrease even more. India is now a major contributor of workers, with almost 35% of the workers coming from the subcontinent”
    • “Across US-based workers, there are significantly more females than males, while the situation is reversed for Indian workers.”
    • “In general, the (self-declared) educational level of the workers is higher than the general US and Indian population.”
    • “In the US, the shape of the distribution roughly matches the income distribution in the general US population. However, it is noticeable that the income level of US workers on Mechanical Turk is shifted towards lower income levels.”
    • “Very few Indian workers participate on MTurk for “killing time”, and significantly more Indians treat MTurk as a primary source of income.”
    • Also this: Mechanical Turk: Now with 40.92% spam. - “With the amount of spam in the worker side, and the amount of spam in the requester side, then Mechanical Turk would slowly turn into a market where spammers requester talk to spammer workers… Ah yes, and academics running experiments…”
  • Irani, Lilly. “Difference and Dependence Among Digital Workers: The Case of Amazon Mechanical Turk.” The South Atlantic Quarterly 114, no. 1 (Winter 2015).
    • “Poorly compensated data work has been the foundation of this information economy, from telegraph messenger boys to online chatroom moderators”
    • Microlabor allows employers do ditch the identity of employers, and to focus on being whatever they see themselves as: researchers, entrepreneurs, programmers, etc.
    • “Once a worker submits completed work, the employer can choose whether to pay for it. This discretion allows employers to reject work that does not meet their needs, but also enables wage theft.”
    • “In recent years, however, Amazon appears to have cut off international workers, instead focusing on US workers, who are understood to generate less 'spam' work.” — “Though the crowd seems unstructured, AMT employers rely on invisible social work and cultural bonds to smooth and simplify their operations.”
    • “Workers dissatisfied with a requester’s work rejection can contact the requester through AMT’s web interface. Amazon does not require requesters to respond and many do not; several requesters have noted that a thousand-to-one worker-to-requester ratio makes responding cost prohibitive”
    • Quoting a large-scale requester: ““You can work in a different way, you can work much faster, you can try things. To me, the try things thing is a wonderful thing about crowdsourcing on Mechanical Turk. You don’t have to get your questions perfect. . . . When I was wrong, it really didn’t matter. I spent a few bucks. The loss was minimal. It inspires the willingness to try a lot of things” — Irani: “Microwork, then, enabled this engineer to tinker with human workers.”
    • “Hiding the labor is key to how these startups are valued by investors, and thus key to the speculative but real winnings of entrepreneurs. Microwork companies attract more generous investment terms when investors perceive them as technology companies rather than labor companies.”
    • “The characterization of Turk work as menial and mindless serves the project of attributing innovation and agency to the software engineers and entrepreneurs that employ Turkers”
    • “First, the more visible the workers in human computation become, the less the “software” companies look like software—there go the valuations.”
    • “This division of labor enables the capture of value or reduction in cost while producing at the same time highly stratified labor subjectivities. Programmers, innovators, lean startups, and IT managers reinforce their claim as the celebrated actors of knowledge-economy projects—the brains that drain, circulate, and congregate in centers of capital”
    • “Turkers need programmers to survive; programmers need Turkers to sustain the magic of their technologies and the fun of their work. Within these relations of exploitation, where among the multitudes is liberation to be found?”
  • Irani, Lilly and M. Six Silberman. “Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk.” Proceedings of CHI (2013).
    • “We argue that human computation currently relies on worker invisibility.”
    • “Rather than conducting HCI research to reveal and represent values and positions, and then building systems to resolve those political differences, we built a system to make worker-employer relations visible and to provoke ethicaland political debate”
    • “Turkopticon grew out of a tactical media art project intended to raise questions about the ethics of human computation. Tactical media, one tradition within activist art, emphasizes developing urgent, culturally provocative interruptions and resistance through the design of media”
    • “(We began this research in 2008, prior to the growth of popular online worker forums turkernation.com and mturkforum.com.)”
    • “We were biased – decidedly so.” “Our biases were validated by some workers and challenged by others.”
    • “Amazon legally defines the workers as contractors subject to laws designed for freelancers and consultants; this framing attempts to strip workers of minimum wage requirements in their countries.”
    • “When Jeff Bezos launched AMT to an MIT audience in 2006, he announced: 'You’ve heard of software-as-a-service. Now this is human-as-a-service'”
    • “Because AMT’s participation agreement grants employers full intellectual property rights over submissions regardless of rejection, workers have no legal recourse against employers who reject work and then go on to use it.” — complaints from workers are normally used by employers not to correct the transaction but to correct the algorithm to avoid the problem next time“
    • “Turkopticon’s sustained dissent over the last four years has qualities of adversarial design [15]; the system stands as a visible reminder of the microlabors that sustain crowd platforms. … However, Turkopticon’s existence sustains and legitimizes AMT by helping safeguard its workers. AMT relies on an ecosystem of third party developers to provide functional enhancements to AMT … Ideally, however, we hoped that Amazon wouldchange its systems design to include worker safeguards.This has not happened.”
  • Irani, Lilly and M. Six Silberman. ”[From Critical Design to Critical Infrastructure: Lessons from Turkopticon].“ Interactions (July-August 2014).
    • “Amazon Mechanical Turk (AMT) works by keeping worlds apart.”
    • “Unable to replace low-status workers with machines, Amazon simulated machines with hidden, globally distributed, contingent, low-status workers.”
    • Turkopticon project began out of surveys trying to cull a Turkers' bill of rights
    • “Unfavorable reviews on Turkopticon have prompted more than a few employers to wonder why their tasks are not being completed—and, eventually, to engage with workers through Turkopticon and other online venues.”
    • “We have maintained Turkopticon for five years. It has become a staple worker tool, with more than 28,000 registered users, 110,000 reviews of 24,000 employers, and 400,000 visits per month.”
    • “Building solidarities is one way of countering HCI’s tendency, as Dourish has argued [6], to take market framings for granted”
    • “Amazon uses the Turk platform for its iPhone app where potential shoppers take a photo of a product they see and want and receive a link to that product on Amazon”
    • “The average wage is $1.40 per hour (of course, you’ll get better work the more you pay).”
    • “the service, so far, is only available in English and to make job requests you have to have a U.S. address.”
    • “Evidence suggests the rate of failing attention on Turk is no higher than other formats (e.g., lab, other internet survey).”
    • “Rand (2011) used IP address logging to verify subjects’ self-reported country of residence, and found that 97% of responses are accurate.”
  • Marvit, Moshe Z. ”How Crowdworkers Became the Ghosts in the Digital Machine.“ The Nation. February 4, 2014.
    • “In this virtual world, the disparities of power in employment relationships are magnified many times over, and the New Deal may as well have never happened.”
    • “Even in its early stages, Mechanical Turk was a “Jeff project,” meaning that Amazon CEO Jeff Bezos took a special interest in it and worked closely with the project managers”
    • “Requesters can engage in such broad exclusions because the Civil Rights Act does not touch upon these workers. In fact, most of the hard-won worker protections of the twentieth century do not apply to Turkers.”
    • “As a result, it is estimated that the average wage of Turkers is approximately $2 an hour. For access to this unregulated labor pool, Amazon charges a 10 percent commission from the Requesters.”—“Turkers are categorized as independent contractors, meaning that they are not legally entitled to minimum wage, overtime pay, workers’ compensation, unemployment insurance or the various other statutory protections that cover employees. The multi-page participation agreement that all Turkers must consent to before entering the site specifies that the Turker is neither an employee of the Requester nor of Mechanical Turk.”
    • Cites a case of the New York Attorney General failing to act on a complaint.
  • Mason, Winter and Siddharth Suri. ”Conducting behavioral research on Amazon’s Mechanical Turk.“ Behavior Research Methods 44, no. 1 (March 2012).
    • “The central purpose of this paper is to demonstrate how to use this website for conducting behavioral research and lower the barrier to entry for researchers who could benefit from this platform.”
    • “The single most important feature that Mechanical Turk provides is access to a large, stable pool of people willing to participate in experiments for relatively low pay.”
    • “experiments can be built and put on Mechanical Turk easily and rapidly, which further reduces the time to iterate the cycle of theory development and experimental execution.”
    • An example of replicating a Norenzayan study that showed the impact of religious priming
    • “as of this writing the majority of workers come from the United States and India because Amazon only allows cash payment in U.S. dollars and Indian Rupees—although workers from any country can spend their earnings on Amazon.com”
    • Studies suggest that “most workers are not trying to scrape together a living using Mechanical Turk (fewer than 8% report earning more than $50/week on the site), but rather are caregivers, college students, and the “bored-at-work” crowd spending free time to earn some extra spending money”
    • The rest is a lengthy primer on how to use the platform for research.
  • Mason, Winter and Duncan J. Watts. ”Financial incentives and the 'performance of crowds.'.“ ACM SIGKDD Explorations Newsletter 11, no. 2 (December 2009).
    • “Here we investigate the effect of compensation on performance in the context of two experiments, conducted on Amazon's Mechanical Turk (AMT). We find that increased financial incentives increase the quantity, but not the quality, of work performed by participants, where the difference appears to be due to an “anchoring” effect: workers who were paid more also perceived the value of their work to be greater, and thus were no more motivated than workers paid less. In contrast with compensation levels, we find the details of the compensation scheme do matter–specifically, a “quota” system results in better work for less pay than an equivalent “piece rate” system. Although counterintuitive, these findings are consistent with previous laboratory studies, and may have real-world analogs as well.”
  • Martin, David et. al. ”Being a Turker.“ Presented at Performing Crowd Work conference in Baltimore, Maryland, February 15-19, 2014.
    • A study of TurkerNation. “we analyse Turking as a form of invisible work”
    • “Some key findings are that they treat their activities as work where pay is the most important factor and that they understand and orient to AMT as a labour marketplace. Their biggest concerns are having enough information to make good decisions on selectingjobs, having good relationships with requesters, and how to act collectively. We find that the key function of Turker Nation is to help reduce the information deficit and promote better collective action”
  • Paolacci, G., Chandler, J. (2014). “Inside the Turk: Understanding Mechanical Turk as a Participant Pool.” Current Directions in Psychological Science, 23(3), 184-188.
      • “Demographically speaking, the MTurk workforce is made up of over 500,000 people from 190 countries, with about 75% of workers living in the United States and India. Paolacci and Chandler report that MTurk offers researchers a participant population that is more diverse than the typical college student population, but still not representative of the population as a whole.”
      • Quoting from paper: ”'Workers tend to be younger (about 30 years old), overeducated, underemployed, less religious, and more liberal than the general population.' Furthermore, within the US’s MTurk workforce, Asians are overrepresented, while Blacks and Hispanics are underrepresented.“
      • “increased experience completing research tasks, particularly economic games and problems, may lead to a practice effect impacting worker responses. The authors also caution that arbitrary factors in experimental design could impact participant selection, and emphasize the need for researchers to take steps to understand and report the make-up of their participant population.”
    • ” review numerous replication studies indicating that AMT data is reliable. I also present two new experiments on the reliability of self-reported demographics. In the first, I use IP address logging to verify AMT subjects’ self-reported country of residence, and find that 97% of responses are accurate. In the second, I compare the consistency of a range of demographic variables reported by the same subjects across two different studies, and find between 81% and 98% agreement, depending on the variable. Finally, I discuss limitations of AMT and point out potential pitfalls.“
    • Lists kinds of experiments that can't be done reliably: “experiments that correlate behavior with hormone levels,” “designs that require complete control of subjects’ attention (such as those using cognitive load manipulations) are also not practical on AMT.” Also: “non-random attrition.”
    • Concerned about the possibility of false positives: “researchers may (most likely subconsciously) continue to re-run an experiment until a falsely positive significant result is obtained”
  • The Reasons Why Amazon Mechanical Turk No Longer Accepts International Turkers,“ Tips for Requesters on Mechanical Turk. January 17, 2013.
    • Includes a letter from Amazon explaining how international accounts are not being accepted.
    • “They have also essentially banned international requesters on Amazon Mechanical Turk by requiring a “verified” Amazon Payments account. You cannot have a “verified” Amazon Payments account without having a Tax ID or a Social Security number.”
  • Salehi, Niloufar, Lilly Irani, Ali Al Khatib, Michael Bernstein. ”Dynamo: Designing Interactive Technology to Support Social Movements in Digital Labor.“ Future of Work Project position paper. 2014.
    • “Dynamo’s biggest effort was aimed at academic research practices. In the weeks prior to Dynamo’s launch, an academic researcher began experimenting on Turkopticon, an independent Turker rating system. This experimentation included the injection of fabricated data into Turkopticon, causing disarray among Turkers who rely on the accuracy of this information for their jobs. After several days of investigation, the Turkopticon community discovered that this was the work of an academic researching the dynamics of requester reputation on AMT. This incident took up much time and energy and caused much frustration, prompting questions about the ethics of research on AMT. While this specific research project was approved by IRB, Turkers agreed that IRB committees lacked adequate exposure to issues and vulnerabilities they faced as Turkers, both individually and as a community. / A group of Turkopticon maintainers suggested that the community draft publicly available ethics guidelines. The guidelines were intended both to guide behavior, and also to back Turker claims about normative standards should Turkers take issue with a job’s research ethics. Like in many social movements, emotionally charged incidents became a trigger for collective action [10]. This campaign became Dynamo’s largest and most active. Using the Dynamo Wiki, members collaboratively generated a sprawling online guide on matters ranging from how to pay fairly, how to respect Turker privacy, and how to respect Turkers’ communities online. The guidelines are also available online as a 22 page write-up that has been viewed 13’000 times.”
  • Tasse, Dan. ”Some things I learned from running a big Mechanical Turk study.“ Tales 'n' Ideas. August 1, 2014.
    • Recommends $8/hour. “Why $8? People want to make about US minimum wage. It's a nice round benchmark, at least. It's still way cheaper than you could get it done any other way.”
    • “Spell out exactly what will make you reject people.”—“Turkers will get really mad if you reject them for something you didn't warn them about. I think that's a feature of mturk, not a bug.”
    • “Because most HITs are accepted, and because the difference between being a 95% turker and a 98% turker is so big, every rejection really hurts workers.”
  • Williams, George. ”The Ethics of Amazon’s Mechanical Turk.“ The Chronicle of Higher Education. March 1, 2010.
    • “For the sake of argument, I’d like to ask why actually paying someone a nominal hourly wage to complete simple tasks is somehow a shady practice but getting someone to contribute to your project for free is perfectly acceptable.”
  • Zittrain, Jonathan. ”Minds for Sale.“ BerkmanCenter YouTube account. November 18, 2009.
    • “A tour of the Ubiquitous Human Computing pyramid”—various levels of crowdsourcing, from the most skilled to the least. MTurk is kind of in the middle.
    • Emphasizes that MTurk isn't always transparent about the reasons for doing something. “What if you are laboring for a cause, or an employer, that you actually would not want to help?” (29:00)
    • Example about crowdsourcing US border webcams. And then another about images of protesters in Iran. What if it were farmed out to MTurk? (33:25) Estimates $17,000 will identify a given protester in Iran. And participants would not necessarily know the purpose their labor is being used for.
    • Recommends some regulations, especially against deception

Internal corporate systems

Other crowd-labor platforms

Further readings