This Soc(AI)ety Seminars session is presented by Lucy Family Institute for Data and Society & Human-centered Analytics Lab (HAL).
The Soc(AI)ety Seminars, hosted by the Lucy Family Institute for Data & Society, are a collection of talks with a vision for AI’s present and future impact on society. Each session is meant to inspire a dialogue on ethical and socially responsible Data & AI innovation. For more information, and to view previous Soc(AI)ety Seminars sessions, please visit the Soc(AI)ety Seminars webpage.
Description:
AI operationalization is lagging behind the rapid growth in AI capability. In this talk, I will build on a multi-disciplinary framework I have introduced to guide the operationalization of AI in order to reduce the documented AI capability-deployment gap. The talk is in equal parts an overview of ongoing work in AI measurement and evaluation and in AI operations. Using LLM model consistency under adversarial prompting as context, I will introduce measurement and evaluation issues related to understanding LLM models and their behaviors at different levels of granularity (e.g., black box and chains of thought). Shifting from evaluation to its use in operational deployment, I will move up a level of abstraction, and present a mathematical model and computational pipeline that determines how task and workflow level AI deployment changes workforce roles, the tasks they bundle, and the skills they demand from human workers. The talk draws on the following papers ([2510.02712] Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attacks (ICLR,26),[2504.04717] Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Models (TMLR, 2026) and https://arxiv.org/abs/2605.29087 and work being done as part of a Laude Moonshot award ( https://www.laude.org/moonshots).
Guest Speaker Bio:
Ramayya Krishnan is the Dean Emeritus and W. W. Cooper and Ruth F. Cooper Professor of Management Science and Information Systems at the Heinz College Information Systems and Public Policy at Carnegie Mellon University. He is an expert in data and decision analytics and digital transformation. He served as President of INFORMS in 2019 and helped lead the creation of its AI strategy.
He is an AAAS Fellow (section T), an INFORMS Fellow, and an elected member of the National Academy of Public Administration. He chaired the AI futures Committee of the National AI Advisory Committee to the President and the White House office of AI Initiatives and was chair of the DOD’s RAI academic council. He directs the CMU-NIST cooperative research center on AI measurement science and engineering (AIMSEC). Please see https://tinyurl.com/k8kbej25 for additional information.