Abstract:This talk describes a research effort on the peer grading problem, running the gamut from theory to practice. First, I'll describe a theoretical investigation of different approaches to incentivizing students to invest the effort to perform thoughtful and honest peer evaluations. One promising approach is to draw on the large literature on so-called "peer prediction". The mechanisms proposed in this literature can incentivize truth telling in equilibrium, but also give rise to equilibria in which agents do not pay the costs required to evaluate accurately, and hence fail to elicit useful information. We show that this problem is unavoidable whenever agents are able to coordinate using low-cost signals about the items being evaluated (e.g., superficially glancing at the text to be graded, focusing on grammar and formatting). We then consider ways of circumventing this problem by comparing agents' reports to ground truth, which is available in practice when there exist trusted evaluators---such as teaching assistants in the peer grading scenario---who can perform a limited number of unbiased (but noisy) evaluations. Of course, when such ground truth is available, a simpler approach is also possible: rewarding each agent based on agreement with ground truth with some probability, and unconditionally rewarding the agent otherwise. Surprisingly, we show that the simpler mechanism achieves stronger incentive guarantees given less access to ground truth than virtually all previously-proposed peer-prediction mechanisms.
The second part of the talk will describe our experiences with using this simpler mechanism in practice. No existing peer review software made it possible to leverage TA grades, so we built our own, which we called "Mechanical TA" and published as open-source software. In this system, human TAs both evaluate the peer reviews of students who have not yet demonstrated reviewing proficiency and spot check the reviews of students who have. Mechanical TA also features "calibration" reviews, allowing students to quickly gain experience with the peer-review process. We used Mechanical TA for three years to run weekly essay assignments in a class of about 70 students, a course design that would have been impossible if every assignment had had to be graded by a TA. We show evidence that Mechanical TA helped to support student learning, leading us to believe that it may also be useful to others.

Biography: Kevin Leyton-Brown is a professor of computer science at the University of British Columbia. He holds a PhD and M.Sc. from Stanford University (2003; 2001) and a B.Sc. from McMaster University (1998). He studies the intersection of computer science and microeconomics, addressing computational problems in economic contexts and incentive issues in multiagent systems. He also applies machine learning to the automated design and analysis of algorithms for solving hard computational problems.
He has co-written two books, "Multiagent Systems" and "Essentials of Game Theory," and over 100 peer-refereed technical articles; his work has received over 7,000 citations and an h-index of 35. He is the recipient of UBC's 2015 Charles A. McDowell Award for Excellence in Research, a 2014 NSERC E.W.R. Steacie Memorial Fellowship—previously given to a computer scientist only 10 times since its establishment in 1965—and a 2013 Outstanding Young Computer Science Researcher Prize from the Canadian Association of Computer Science. He and his coauthors have received paper awards from JAIR, ACM-EC, AAMAS and LION, and numerous medals for the portfolio-based SAT solver SATzilla at international SAT solver competitions (2003–15).
He is chair of the ACM Special Interest Group on Electronic Commerce, which runs the annual Economics & Computation conference. He serves as an associate editor for the Artificial Intelligence Journal (AIJ), ACM Transactions on Economics and Computation (ACM-TEAC), and AI Access; serves as an advisory board member for the Journal of Artificial Intelligence Research (JAIR, after serving as associate editor for two 4-year terms), and was program chair for the ACM Conference on Electronic Commerce (ACM-EC) in 2012. He has co-taught two Coursera courses on "Game Theory" to over half a million students, and has received awards for his teaching at UBC—notably, a 2013/14 Killam Teaching Prize. He is currently a visiting faculty member at the Simons Institute for the Theory of Computing; previously, he split his 2010–11 sabbatical between Makerere University in Kampala, Uganda, and the Institute for Advanced Studies at Hebrew University of Jerusalem, Israel. He currently advises Auctionomics, Inc. (and through them, the Federal Communications Commission), Zynga, Inc., and Qudos, Inc. He is a co-founder of Kudu.ug and a new UBC spinoff, Meta-Algorithmic Technologies. In the past, he served as a consultant for Trading Dynamics Inc., Ariba Inc., Cariocas Inc., and was scientific advisor to UBC spinoff Zite Inc. until it was acquired by CNN in 2011.

Abstract: For decades, physical behavioral labs have been a primary, yet limited, method for controlled experimental studies of human behavior. Now, software-based "virtual labs" on the Internet allow for studies of increasing complexity, size, and scope. In this talk, I highlight the potential of virtual lab experiments for studying social interaction and coordination. First, we explore collective intelligence and digital teamwork in "crisis mapping", where digital volunteers organize to assess and pinpoint damage in the aftermath of humanitarian crises. By simulating a crisis mapping scenario to study self-organization in teams of varying size, and find a tradeoff between individual effort in small groups and collective coordination in larger teams. We also conduct a study of cooperation in a social dilemma over a month of real time, using crowdsourcing participants to overcome the time constraints of behavioral labs. Our study of about 100 participants over 20 consecutive weekdays finds that a group of resilient altruists sustain a high level of cooperation across the entire population. Together, our work motivates the potential of controlled, highly instrumented studies of social interaction; the importance of behavioral experiments on longer timescales; and how open-source software both can speed up the iteration and improve the reproducibility of experimental work.

* based on joint work with Lili Dworkin, Winter Mason, Siddharth Suri, and Duncan Watts.

Biography: Andrew Mao is currently a postdoctoral researcher in Computational Social Science at Microsoft Research in NYC. He received his PhD from Harvard University in 2015. His research focuses on studying collective intelligence and social interaction on the Internet, such as teamwork in online communities and coordination in crowdsourcing systems. Andrew specializes in designing and gathering data from real-time, interactive, web-based behavioral experiments, and he is the designer of TurkServer (http://turkserver.readthedocs.io/), an open-source platform for building such experiments. He received the Yahoo! Key Scientific Challenges award in 2011.

Abstract: In recent years, interactive digital games are starting to move from mainly relying on the mouse and keyboard for user inputs towards using body movements as inputs. Examples of such game platforms include the Nintendo Wii, the Microsoft Kinect, and the Leap Motion device. These devices have changed the nature of computer games to make it possible for people who are not technically savvy, such as senior citizens, to enjoy the game play experience. Researchers are starting to leverage body gesture-based games to unobtrusively study the progression of medical conditions and long term individual wellness trends. Many interesting results have emerged.

In this presentation, I will share with the audience on some of the methods, results and lessons learnt from our research in wellness games and aging-in-place IoT technologies for senior citizens. I will draw upon the research experience from the Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY), Singapore, to illustrate how technologies could help to advance the field of personalized healthcare at a population scale.

LILY focuses on applying research advances in Artificial Intelligence, Human Computation, Data Analytics and Digital Media for helping senior citizens. An impactful example of LILY’s innovative research which exploits the combined power of AI, human wisdom and data analytics is an engaging game for health and wellness – Pumpkin Garden. It is a population level screening and rehabilitation game for neurodegenerative diseases such as Parkinson and dementia. In this game, data are continuously collected and analyzed to provide health professionals with insights into the progress of their patients. I will also introduce one of our latest works in using motor patterns for stroke detection and rehabilitations.

Biography: Dr Chunyan Miao is an Associate Professor School of Computer Engineering Nanyang Technological University and Director of Joint NTU-UBC Research Centre of Excellence in Active Living for Elderly (LILY) co-initiated by IDMPO/NRF, MDA, NTU and UBC Canada. Over the last ten years, she has successfully led variously national research projects in AI, Intelligent interactive digital media, smart health, data analytics, aging in place, ageless aging design & technologies from both government agencies and industry.