Talk: Grounding Language by Seeing, Hearing, and Interacting

Rowan Zellers: PhD Candidate, University of Washington in Computer Science & Engineering

As humans, our understanding of language is grounded in a rich mental model about “how the world works” – that we learn through perception and interaction. We use this understanding to reason beyond what is literally said, imagining how situations might unfold in the world. Machines today struggle at making such connections, which limits how they can be safely used. This talk will discuss three lines of work to bridge this gap between machines and humans. These directions suggest a path forward for building machines that learn language rooted in the world.