91��Ƭ��

William R. Penuel is a professor of learning sciences and human development in the School of Education at the University of Colorado Boulder. His current research examines conditions needed to implement rigorous, responsive, and equitable teaching practices in STEM education. At iSAT, he is a Co-Principal Investigator and Co-Lead of Strand 3 - which focuses on inclusive co-design processes to empower stakeholders with diverse identities to envision, co-create, critique, and apply AI learning technologies for their schools and com­munities.

As school and district leaders, you are used to building planes while flying them. But the advent of AI—specifically Generative AI—in classrooms has caught many of us off guard and not sure what airspace we’ve entered. Generative AI is the technology behind popular tools like ChatGPT, as well as tools today that use AI to help teachers build lesson plans and assessments for use in their classrooms. It’s a specific kind of AI that learns from the data it’s been fed (such as text, video, or images) to create new content. If you’ve tried it out, you may be impressed both by its capabilities to simulate human interaction, as well as its limitations.

As an education leader, Generative AI presents many interrelated challenges to you, to teachers, to parents, and to students pertaining to safety, transparency, and ethics. In this blog post, we want to focus on two other central issues that Chief Academic Officers, district technology leaders, principals, and instructional coaches should keep in the foreground when evaluating the potential integration of AI into schools: learning and evidence of impact. Learning has to do with both our goals for learning and how we support them. Evidence of impact has to do with the power and limits of tools to achieve those learning goals. Good evidence also involves evidence of what’s required of teachers to implement tools well, to achieve benefits for students. Both these considerations are important in evaluating Generative AI and other tools, but often they live in the background of discussions about Generative AI.

Take the discussion of the potential of Generative AI for personalization and differentiation of learning. This is chief among the advantages that advocates of AI tout. The questions to consider are: What kinds of learning goals can Generative AI support? What do we know about the potential of Generative AI for supporting these goals?

Intelligent Tutors Help Personalize Individuals’ Mastery of Discrete Knowledge and Skills

There is more than 50 years of research on intelligent tutoring systems (ITSs) that we can draw on to give us a sense of what learning goals AI for personalization can support. ITSs are trained when their developers subdivide knowledge to be taught into smaller components—skills, abilities, and concepts—allowing ITSs to recommend tasks based on a student’s mastery level. There’s a large body of evidence of impact that suggests that for the kinds of problems ITSs are used to help students with, they do as least as well as human tutors do in supporting learning.

However, while AI excels at guiding students toward specific, well-defined learning goals (like solving a math problem), it struggles with more open-ended tasks where multiple solutions exist, or where collaboration and dialogue are essential. Further, it may limit deeper engagement and valuable experiences like productive struggle or peer collaboration. The evidence base applies only to well-designed ITSs, as well. Many of the Generative AI tools today can’t achieve the results of the best ITSs. While they are good at handling requests in everyday language, many of these tools still give  students encounter in schools.

This is not to say that Generative AI won’t become more capable of solving math problems or helping support critical thinking, teamwork, and real-world problem solving in the future, but there is not strong evidence of impact for achieving these learning goals. There is even less evidence related to what’s needed to prepare teachers to use these tools well. There’s reason to be skeptical, then, about claims that the current class of tools of Generative AI can support these goals. 

AI Can Support Collaborative Problem Solving in Inquiry-Rich Environments

There’s an equally rich body of evidence of impact for a set of AI tools that support collaborative learning. For more than two decades, the field of computer-supported collaborative learning has created and tested different tools focused on fostering group awareness and giving students feedback on small groups’ cognitive and social dynamics. A  of these kinds of group awareness tools show improvements to students’ knowledge and skill, as well as group task performance and social interaction in collaborative learning. The relevance of these findings for K-12 schools, though, is not as clear, because many of these tools were designed for online environments in higher education. 

Here’s where emerging research comes in – the kind designed to build evidence of impact grounded in a robust vision for teaching and learning. The Institute of Student AI-Teaming is developing AI partners—the Community Builder (CoBi) and the Jigsaw Interactive Agent (JIA)—that perform the key functions of group awareness tools. These tools are intended to be integrated with rich curricula that focus on collaborative problem solving in STEM. These tools do something very different from what Generative AI tools as currently used to plan instruction or support personalization do: they help students learn to collaborate more effectively and equitably. They support a different kind of learning, too, one that is focused on students figuring out ideas and solving problems together, using disciplinary practices from STEM that are targeted in today’s standards. And while we are still gathering evidence of impact, we already know that students are using some collaborative solving skills more when they are using an AI partner to support their learning. We aim to make these partners—and the instructional materials to teach about AI—available to schools for free in the coming year.

Questions to Ask 91��Ƭ�� Learning and Impact

AI is here to stay, and as a leader, you know you have an obligation to approach how to use AI responsibly and ethically to achieve your vision for teaching and learning. No doubt, AI may now or in the future be useful for increasing efficiency in how teachers plan and how students develop discrete knowledge and skill. As vendors continue to rush to offer generative AI products to schools and districts, it’s important to ask three questions:

What kind of learning does this tool support?

What kind of preparation do teachers need to use the tool well?

What evidence of impact is there for the claims being made about Generative AI?

Integrating AI into classrooms is likely to lead to changes in how teachers teach and how students learn. Teachers will need support in learning how the AI works, and how to use AI tools to support teaching and learning that is consistent with what we know about how students learn. A generative AI chat bot doesn’t understand how people learn, no matter how skillful its interactions seem. That leaves it as your responsibility as a critical consumer of AI tools to ask tough questions of vendors about their ideas about teaching and learning and to demand they present evidence of bold claims about the power of AI.

Now is a moment when we are all particularly open and keen to learn about AI, and it is as imperative as ever to create opportunities where educators and leaders can learn together about the potential and limits of Generative AI and other tools that support learning goals for collaborative problem solving. We not only have to be “in the loop”: as decision makers about teaching and learning, we need to stay at the center, working at a pace that protects both our children and takes care of our visions for teaching and learning and that follows evidence more than hype.