More For Less: Adaptive Labeling Payments in Online Labor Markets
by
Maytal Saar-Tsechansky
Associate Professor
McCombs School of Business, University of Texas at Austin
Friday, October 19, 2018
10:30 AM – noon
Speakman Hall Suite 200
Abstract
In many predictive tasks where human intelligence is needed to label training instances, online crowdsourcing markets have emerged as promising platforms for large-scale, cost-effective labeling. However, these platforms also introduce challenges that must be addressed for these opportunities to materialize. In particular, it has been shown that different trade-offs between payment offered to labelers and the quality of labeling arise at different times, possibly as a result of different market conditions and even the nature of the tasks themselves. Because the underlying mechanism giving rise to different trade-offs is not well understood, for any given labeling task and at any given time, it is not known which labeling payments to offer in the market so as to produce accurate models cost-effectively. Effective and robust methods for dealing with these challenges are essential to enable a growing reliance on these promising and increasingly popular labor markets for large-scale labeling. In this talk I will first present a new data science problem, Adaptive Labeling Payment (ALP): how to learn and sequentially adapt the payment offered to crowd labelers before they undertake a labeling task, so as to produce a given (machine learning) predictive model performance cost-effectively. I will then present our approach to address the problem and a rich set of results that demonstrate its performance under a variety of market settings. We also show that the method is highly versatile and can acquire more labels of lower quality (and cost) under some market conditions, while pursuing fewer and higher quality labels in other settings. Overall, our method yields significant cost savings and robust performance; as such, it can be used as a benchmark for future mechanisms to determine cost-effective payments.
Bio
Maytal Saar-Tsechansky is an Associate Professor of Information, Risk and Operations Management at the McCombs School of Business, The University of Texas at Austin, and a co-founder of Sweetch — a mobile health startup firm. Her research focuses on developing machine learning (ML) and artificial intelligence (AI) methods to improve decision-making and to benefit people, organizations, and society. Most of her work aims to augment ML & AI by bringing to bear the problems that machine learning and AI inform in practice and the context in which learning itself occurs, with the goal of effectively dealing with the constraints and taking advantage of the opportunities presented in these environments. Her research integrates business, machine learning and artificial intelligence, and she has addressed challenges in different domains, including health care, smart electricity grid, fraud detection, finance, and emerging forms of work, such as online labor markets. Maytal received her Ph.D. from New York University’s Stern School of Business. Her research has been published in the Journal of Finance, Management Science, Information Systems Research, Journal of Machine Learning Research, and Machine Learning Journal, among other venues. Maytal’s research has been supported by both government and industry, including the National Science Foundation, SAP, and the Israeli Science Ministry. In recent years she has served on the editorial boards of the Machine Learning Journal, the Information Systems Research (ISR) journal, the INFORMS Journal on Computing, Decision Sciences, and she is a frequent Program Committee member in the premier machine learning, data mining, artificial intelligence, and Information Systems conferences. At McCombs, Maytal has developed and taught popular applied machine learning and data mining courses tailored to business students.