Sendhil Mullainathan

Sendhil Mullainathan

Chicago Booth

                                “Do Large Language Models Understand? How Would We Know if They Did?”

NYU Department of Economics
19 W 4th Street, Room 517

Abstract

Large language models do impressive things. Large Language models do strange things. They present a conundrum: how are we to evaluate their capabilities and know where/when they can be used? I will present some frameworks and empirical results that speak to the following three interrelated questions: how can we assess if they have world models; how can we assess if they have learned the facts in their corpus; and finally, does it matter if people can understand their capacities?

Speaker Bio 

Sendhil Mullainathan is the Roman Family University Professor of Computation and Behavioral Science at Chicago Booth. His current research uses machine learning to understand complex problems in human behavior, social policy, and especially medicine, where computational techniques have the potential to uncover biomedical insights from large-scale health data. In addition to being a co-PI at the joint Berkeley-UChicago Laboratory for Systems Medicine, Sendhil is the cofounder of the computational medicine initiative, Nightingale.  He’s also a co-founder of Pique, a an app that changes how people read books and learn; and Dandelion, a company that catalyzes AI in healthcare.

In past work he has combined insights from economics and behavioral science with causal inference tools—lab, field, and natural experiments—to study social problems such as discrimination and poverty. Papers include: the impact of poverty on mental bandwidth; how algorithms can improve on judicial decision-making; whether CEO pay is excessive; using fictitious resumes to measure discrimination; showing that higher cigarette taxes makes smokers happier; and modeling how competition affects media bias.