Victor Lee: Has AI ’broken’ our schools?

I mean automation is sort of a shortcut in the interest of whatever worker is getting a little bit less of the repetitive activity or for the company that’s going to have a better bottom line. So yeah, I mean we we tend to appreciate shortcuts. With the academic stuff where it’s been hard now is we’re heading towards almost like the symbiosis of our lives with technology and we’ve been very used to oh no. So I said at the very beginning the big shock was AI can now do these things that we thought oh no that’s only a people thing.

Only people can do that. People must be trained to do that. They’re the only ones who can do those things.

And that has been you know turned a bit on its head. Victor Lee. Thank you for joining.

Oh thanks for having me. We’ve we’ve shared at least one I think more than one conference stage. This is a bit self-indulgent because I I actually haven’t had the chance to sit down with you and really dive in at length and so this is really opportunity for me and hopefully our audience to hear more about your research but I think more importantly just your perspective on this big question of AI and education the future of learning we have an hour to solve all of that no is that all easy we’ll be done in 30 minutes and then you can mention the advertisement or anything like that yeah advertisement standard.

Yeah. So, so just tell tell us about who you are and you know you you’re doing a lot of research. Maybe you can share there’s one piece of research that has really I think made a lot of waves in the space.

You know what I’m referring to and you can sort of like talk a little bit about that piece but also your work more broadly at Stanford. Sure. I am a associate professor of education at Stanford University.

My training is in learning sciences which means I think a lot about what people learn, how people learn it, what kinds of environments, settings and tools facilitate that. And then I also have as one of my responsibilities at Stanford as the faculty lead for AI and education at the Stanford Accelerator for Learning. The accelerator is a unit on campus that serves to be a bridge entity for real world impact on questions or problems that involve things related to learning across the lifespan.

It has some key focus areas but AI’s definitely taken a large chunk of that. And in that role, you know, mobilizing folks from like computer science, from music, from humanities, to collaborate with one another to connect out to the community and to try to move research along faster in ways that are more legible and consequential. By trying to get interesting questions from the challenges people are seeing or the wonderings that they really have and you know creating kind of a virtuous cycle.

Of research. So those are two of the main hats that I wear at Stanford. I also maintain my own research lab, but because I’m from the learning sciences and we have this combination of research and design work, there’s a number of projects ranging from craft which is a AI curriculum resource repository that we have been exploring how teachers think about design of AI literacy experiences.

The study that has made some waves is the cheating study. And that was looking at academic integrity. The last publication on that was looking at academic integrity before chatbd came out and then at the end of that first academic year.

And you know, the headline was that despite the news that kept popping up in the very scary stories that the overall level was staying the same. But what a lot of people didn’t know is that the overall level was already really high and has been high for decades. And there’s some interesting complications that are in there.

But you know in addition to doing research in that have collaborations some domestically some internationally having a really fun collaboration right now with Google research on thinking through digital game-like experiences for students. That that’s been a lot of fun just to operationalize what we know about learning into some of the game mechanics in there. And I think everybody’s learning a lot from from that collaboration.

But I could go on, but the hour would go and I would be like, and I have a project like this and a project like that and, you know, turn the table on to you. Yes, you have a lot of projects. We’re going to talk about the cheating study.

I I do want to just zoom out. I mean, you’re you’re one of the foremost voices right now sort of like providing some guideposts as we think about AI education and AI literacy. Oh boy.

Then we’re in trouble, aren’t we? We AI literacy is sort of it’s a term that kind of has kind of hit the become mainstream. And yet I myself don’t really feel like I fully understand.

When someone says aiteracy, they could be saying one thing, they could mean something else. Like what does a literacy mean to you? And then I can I’ll we can go sort of unpack that.

But just sort of like broad strokes when you think about a honest answer is I don’t think we know and I think we collectively being research or any of the leadership or education. So I have a chapter coming out in a handbook with Dury Long. Duri is a professor over at Northwestern University and did one of the first big studies on synthesizing AI literacy in the research literature.

And there is a bit of talking past one another that’s happening with people coming up with these are the most important ideas to learn or if someone doesn’t learn about how bias is a product of society that gets embed in the data they didn’t learn anything and we have a a huge number of interesting topics some are looking at the mechanics of AI some are looking at social impact of AI some are saying people aren’t going to be dealing with the mechanics of it and just how do you use it as our jobs and daily lives change with AI in there. So in this forthcoming chapter we’ve put forward characterization of how all the different voices that are talking to each other and saying no this is AI literacy, no that’s AI literacy and have said that there’s basically three perspectives that are getting emphasized. One is the user perspective and it’s really just focused on how do I use this to help me write?

How do I use this to help me think through problems? How do I use it to analyze data? You have the developer perspective which is not because it’s an AI literacy angle.

It’s not having you jump straight to developing the AI, but it’s actually understanding like what’s a decision tree, what’s a neural network, training data and testing data. So it gets to more of those foundations to go further into the technical side if you want to build it or critique one learning algorithm versus another. And then there’s the critic perspective which speaks to both the societal impacts but also the healthy skepticism that people should have about what gets generated especially when it’s using something like an LLM or other technologies that are based probabilistically.

So those there’s no right or wrong. There’s some value in all of those. But what what Dury and I say in this particular writing is that the different stakeholders who are pushing for AI literacy land in different spots in that triangle.

Some may really emphasize developer and critic. Some might really be user focused only. And it’s not a right or wrong.

It’s just what is the entire space that folks are playing out. And I’ve even been playing with a little bit of a side analysis to see how different people’s documents or frameworks land and it so far is looking like yeah we’re we’re capturing u some of the different polls that that are being put around that term. So you asked me for a definition and I completely avoided giving you one.

I mean, this is a common thread that the the experts in the space, the true experts are the the ones least willing to make really certain predictions about the future. And I think there’s a humility in that and saying there’s there’s a lot that we don’t know. More you know, the more you know, you don’t know.

Yes. So, and this is an opportunity for you, speaking of critics, I’d be curious for your your critiques if if any. So aiEDU we I don’t I don’t know exactly where in the triangle that we fit.

I think there’s little pieces of all of them, but I think broadly speaking, we think of AI literacy as like how do you understand what AI is? How do you use it? And but we when we do work in schools and what I believe is really the most like the really big thing that we have to solve, it’s not not AI literacy, it’s AI readiness.

And my definition of AI readiness would be our definition is the the skills and knowledge you need to thrive in a world where AI is everywhere. And and the the analogy that I use sometimes is the if if you go back to, you know, 2007, you’d be correct. You’d be correct in saying that everybody is going to be using their phones.

You cannot do a job certainly not a job in knowledge work without the ability to use your phone. And yet, I think we would probably agree that the focus back in 2007 should not necessarily have been mobile phone literacy. It probably should have been a focus.

It’s not something that you should talk about in school. And I kind of feel the same way about AI literacy. I feel like the idea that teachers are going to teach students how to use AI seems perhaps quite silly.

Like I think students are going to be teaching the teachers getting it on TikTok already. I mean, they’re picking up lots from their friends and whatever influencers are on there. So there’s a lot of interesting points that you had raised.

For one, and people may not know this about me, is I actually don’t really like the term AI literacy. It’s the one that’s taken hold. If you look at and unpack literacy, it gets very complex very quickly.

However, it’s just a lot easier to say it when I’m having a quick conversation with somebody, or if they’re doing sort of just the information tagging as to what is this subject about. So, I don’t love the term because I don’t think we have even an agreement on like textual literacy. There’s different notions of what it means to be literate with text and that gets complicated because then we have these new forms of text, graphic novels or you know arguably you’re seeing film as as a form of text and a broadly conceived notion.

So I I I never really loved the term AI literacy. I think readiness is apt. You know, I also hear AI education being talked about, not AI in education, but education about AI, and some folks are positioning that as like the more technical stuff, the, you know, the like here’s the math of AI.

What I do tend to think a lot about though is that as we settle on kind of our literate standard as a country, as a society, it’s really what do we want every person by the time they finish 12th grade in our country to know? Every person if even if they’re not going to go work in an AI profession. And making sure that you know the public education experience they’ve got has made sure that they’ve been equipped with that based knowledge and that means not everybody’s going to be building AI not everybody’s need to distinguish between supervised and unsupervised learning in there and there’s going to be different resonances or cases that are going to be important for understanding why we have justified concerns about societal biases or other kinds of harmful effects that AI could have.

So I would tend to think about literacy as like what is it we want everybody and I don’t think we all agree on what is our model of everybody to have. Now we go back to 2007. Yeah, we wouldn’t necessarily have a mobile phone literacy, but we do probably kick ourselves right now for some of the information literacy, especially with misinformation and social media and responsible social media use.

And you know hindsight’s 2020. However, you know for AI right now things have moved very quickly in a very dramatic way and it is the case that not everybody has a common base sense of AI like even the idea is like is AI sentient? Can AI solve all the problems?

Is AI really going to take my job? And there’s so much hype that’s going around that I do think in the near term we have a bit of an imperative to get everybody up to speed whether you know 50 years down the road there’s AI literacy standards because it’s already so stitched into the fabric of our lives you know whether it needs to be specifically a school subject that’s who knows you know we’ll be on to different things one of the activities I am working on however is the nationalmies of science engineer ing and medicine is has a consensus group where we’ve been meeting over the past year and a half, two years because what’s happened in so many of these new tech developments and data advances is things have changed. You know, we’re starting to talk about computational thinking and computer science and all the schools and that movement’s not very old.

We’ve been talking about data science and then people are saying we need machine learning education and now it’s like AI education and then there’s like talk about quantum and then you know there’s probably going to be something that I like bioinformatics education or whatever is the next thing or you know robotics and it’s just become this pileon. So what we’re focusing on are what are the core competencies that despite whatever development is going to be that thing to hang on to and kind of connect back to rather than let’s teach you a specialized quantum thing and fit that into the negative hours that you have in the school day. But rather like what’s the core idea there and what have we been kind of missing that all of these things are trying to speak to.

So I mean you’re you’re honing in on exactly what I was getting at which is so I I think you’re right that that the and let’s just use a literacy because we we also talk about a literacy in part because it is sort of now common vernacular critical I mean there’s the the the the capabil the potential for AI to present itself as authoritative the fact that like my understanding is I think the single the one thing that AI can do better than people at the very top of the list is persuasion. Well recently I saw is really good at sucking up to people. Yeah.

Which is I mean these are all sort of persuasion. Yeah. So it’s absolutely critical that people have the awareness of the fact that they need to ask these questions.

You need to be aware by like so I I’m not really arguing against a literacy as something that should be happening in schools. My curiosity is around there’s something there sometimes a bit of a twostep where we we’re talking about AI literacy and then suddenly we’re what we’re actually talking about is workforce readiness and I think that is actually where I’m really I’m I’m not so sure. I think in fact I have a our our relatively strong opinion is we don’t know what the jobs of the future are.

Using AI is not going to be a differentiated skill. And so if you think about like how do we make sure that you’re able to thrive in the workforce, the real what you’re really asking is how do we equip you with skills and expertise and knowledge that will command economic value. So using AI isn’t really going to be necessary but not sufficient.

And every all the like studies that I’ve seen and I haven’t gone super deep in this maybe you have. I’d be curious. We’re in the midst of doing our own quantitative research right now with Burning Glass Institute.

Basically, you know, my my lived experience and my understanding at least so far is the the most effective users of AI tools are people that have domain expertise in the domain that they’re using those tools for. And if you sort of abstract that, it it’s it sort of sounds like, you know, the the the what are the skills and knowledge that you need to thrive in the workforce is like math, reading, writing, science, social studies, like the stuff that is computer science, data science, the stuff they’re already teaching in schools. Like I I I worry sometimes about this AI as the the shiny object that distracts us and that makes us feel like we don’t we can sort of pivot to this thing when actually we we actually we need to double down and now can you use AI to help you teach math better?

Sure. That like that’s that but it’s not going to solve the problem. It’s it’s a tool.

But I mean I feel feel free to pay poke holes in that. Waiting they were trying to wait out the whole AI thing because was it the new shiny object? Would everybody get distracted you know 6 months after ChatGPT and the naysayers are saying like yeah just this is a fad we’ve seen this come and go you know it was blockchain you know 2 years ago and who talks about that now or crypto although apparently we’re talking about crypto again but that was that was what people were waiting on I believe the realization with AI is that it is on the sort of influence type we’d say social media IA or the internet or television airwave television had where it just becomes so stitched into everything that we do that this is not one of those flash in the pans even though it may be that you know we stop talking about ChatGPT the same way that we don’t talk about friendster anymore or MySpace but we do have social media in there but I mean I I hear you it’s it’s really tough and I have the utmost empathy and respect for educators in the classroom because they’re just being dumped on.

And some of it is, well, teach this new content, teach that new content. Your standards just changed for this. And we’re not really giving them the space to do the jobs and work that they’re really good at.

And that that’s frustrating for sure. You know you’re sitting with one of the foremost experts on this topic and even you are have the humility to admit that we don’t know answers to a lot of these questions and teachers are you know being are expected to sort of navigate and retool their assessments and their assign well their assignments I don’t know about assessments but certainly their homework assignments and it’s I mean I feel like they’re in over their heads. Yeah, it’s taken for granted and there’s a lot of pressures that are going on with you know teacher workforce and retention right now, but that’s kind of a growing focus in research right now is how teachers learn and adapt with AI and how is AI sort of integrating into teaching practice in line with all the things that we know that help make really effective teachers.

But I’m going to throw something at you. You know, Alex, you’re a smart good-looking guy. The good-looking part doesn’t matter, but the smart does and you know I’ll just throw the sort of PhD student 101 thing that you get when you start doing education for a PhD.

Why do we send our kids to school? Why do we have school? Well, it was I mean I never had to answer that question in the abstract like that.

I mean to prepare them to live and work in society and to do so in a way that where they in a way where they have dignity. I don’t think it’s just careers. I think career is a really big part of it.

We we think about career a lot, but I think school is also a place it’s a formative part of your development where you learn social skills and sort of develop this awareness of your place in the world. Yeah. I mean, schools are a relatively recent invention historically.

I’m not going to go all lectury in in the history of schooling, but you know, it’s kind of complicated and I I just pressed on it because you’re talking about, you know, thriving and economic value in there. And I do think especially in the world that we live, there’s importance to that as a parent. You know, I’m thinking about that.

I think about that for just young people I work with in general. But, you know, this is a a debate that plays out in a lot of places like why do you go to college? And and some folks will say it’s to get a job and others is it’s just to get the piece of paper.

You’re really there to like network. And then still other people will say no, it’s to learn how to think, you know, and just be prepared for whatever kinds of things change and think. And then similar sorts of questions go down to K-12 schooling.

Is it like what’s the relationship of the state to the schools? Is it to help develop citizens, intellectuals? Is it strictly a workforce thing?

And depending on which answer we prefer, that has pretty different implications for what we do in schools and what we do for and with schools. And I would just also say there’s some folks whose answer is child care. You know, that’s child care for several hours a day.

As a parent and I know other parents as soon as your kid starts going to K-12 like your life changes. They’re not on beck and call and that’s that’s a really underappreciated and I don’t think the the primary calling for all teachers who go into working education but it is one of the functions of school and you know during COVID remote work with kids crashing into our calls and seeing all that it was rough because we didn’t have that so it it’s tough I do think thriving is the right direction to go in But thriving in what kind of world and in what role do these people play in that world? You know, how much are they joining existing companies versus taking on a social cause?

But that’s why we sort of do it as like the 101 PhD question because we want to then start to get at the roots of why and how we’ve designed educational systems. What did we do even before schooling the apprenticeship model? Looking at how schooling was something just for the religious and then you know wealthy and elite.

Why do we have the school day structure that is modeled after Harvard from 100 years ago with the committee of 10 and we still stick with that to this day. So it’s I think education’s incredibly fascinating and complicated. And so when we think about well AI in schools I’m not one to push for any policy of this is the new algebra by any means.

And we we all take our very best informed guesses on what’s the important stuff to know for a generation or population at a time or place. Right. But it’s I I want to get we’re not we’re close to Stanford.

We’re not literally in Stanford. And so in an effort to get out of the ivory tower a bit and speak just directly to like I think most people if you were to ask like like you know Billy on the street style. I think I think most people will will give you some variation of like learning about the world and how to live in the world.

But the vast majority like the the certainly the mode is going to be you know getting a career. And so and the reason I say that is because as as as we are doing the work and we’re actually sort of intersect I think quite a lot in in sort of our combined efforts to create clarity or at least to orient people towards a curiosity that is going to be required to eventually get clarity even if it doesn’t perfectly exist today. The most accessible way to actually meet people where they are to like address what will eventually very I think very soon become their absolute top concern is is going to be about sort of like jobs because at the end of the day if we can’t solve if we can’t ensure that a student is on a path to being gainfully employed however that looks.

The other stuff is like it’s nice but you but you know if you’re unemployed like your mental health is definitely like it doesn’t matter if you have like you know self-regulation and all this other stuff. I mean you know you you and this is a world that doesn’t have universal basic income right. That’s right.

Yeah. We’re we’re not we’re not ready we we’re sort of operating within sort of the political environment that we’re in. And I I think it’s actually interesting though because some of when I hear about people talk about AGI and if you sort of like depending on how far down you look and it might some people say we’re five or 10 years away or 3 years I think Dario said two or one but even 10 years it’s like not that far and so there is a and when I actually when I founded AIED this is like 2019 I started really in the space 2018 the AI ethics AI I the the conversation about like social impact in AI was really driven by the effect of altruism movement and it was focused on AI safety and essentially the argument would go AI is going to potentially destroy humanity and so all of our effort around artificial intelligence should be focused on just like how what do we do to mitigate that risk.

But the challenge is it becomes kind of like a it’s it shuts down the conversation because it’s like it’s it’s a whole separate it’s like a complete about face. It’s like okay if we’re actually talking about AGI then what we’re talking about is literally restructuring the social contract and our relationship to capitalism and it’s like you’re you’re almost moving out of the school of education the school of philosophy and and maybe that the argument is the school of education needs that’s what the PhD is for. I am a doctor of philosophy.

It just happens to be around education which then gets to messy things about society and the world. And so I guess taking for granted that we have to it’s not that we we can’t address those things but over you know you and I are often at conferences you know we’ve been speaking and people are sort of like looking for answers and they’re orienting right now I think very heavily towards like using AI in education as sort of like the primary thing to be solved for and which brings us to sort of your this question of cheating and your research and I think this the cheating question is I think part of the The reason your research has been making waves is because it’s not just a topic that people are really interested in. It’s not just something that teachers are obviously concerned about and so it’s the natural thing for everybody to get obsessed with.

It actually gets the crux of why AI is I think quite scary for education because you know teachers are I think correctly intuiting that their students are and I like using the word shortcut instead of cheating because it’s just it removes the judgment out. It’s like it seems unequivocal that like students are using shortcuts and they’re more advanced than most of their teachers. And so there’s this sort of like arbitrage where if your teacher doesn’t really know how to use a language model, you know, they’re you’re always going to be able to outsmart them because like the the solutions I’ve seen like, oh, you know, I have the students use AI, but then I ask them to critique the AI and it’s like, no, you can ask the AI to critique itself.

Like that’s that doesn’t work. Or the Trojan horse where you you add some like white text with some like silly think I saw it’s put in the amperand NBSP and because that’s just a a character that you wouldn’t use but it produces the space and u but like and and maybe that does work but at some point your students are going to catch on and they have a very effective communication system. I mean like like this will this will get out there very quickly.

And so it so the question of cheating is really I think comes down to how do we address the concern that people have about the potential for these shortcuts or the use of AI as a shortcut to undermine productive struggle. I think one of the things that made the cheating work really intriguing is that the fear that we have with AI. For one, being able to do a bunch of things that we were always in cultured to believe are just not within the range of what computers can do.

Was the sense that the kids would be corrupted. It’s it’s just too tempting and they will not do anything. It it has these almost these puritanical kinds of things that it’s that evil temptation and once it’s made available it is going to turn them into lazy thoughtless cheating slackers.

And and I think the cheating study itself said well the folks that you were concerned about they were already doing versions of that. You know, there’s a lot of use of CHEG. There’s a lot of use of copying from Wikipedia and restating the words or passing old versions of tests down from from the same class or having your friend help you out on something that’s supposed to be a solo effort.

So what I have seen actually just a the top lines of the study what did you mind just giving the the TLDDR? Okay. Okay, the TLDDR for the the study was we happened to have data available on some schools before chat GPD came out and it was data that was included amongst it what was happening in terms of cheating in all different activities whether it was plagiarism whether it was you know forging you know excuses from parents whether it was you know copying from an outside source and forgetting to give credit, whether it’s looking over someone’s shoulder at their test.

So yeah, all of those I’ I’ve heard some really amazing and clever ones in there. But then because that was already there, Chat TPT came out, there was the moral panic that everybody was going to start cheating and whatever research came out, it would have been like shocking, just shocking how many people are using it. But we actually saw what was happening with the cheating beforehand and what was happening in that first academic year after it had come out and it was the same.

You know the numbers stayed the same. No, in fact there were some things that had some weird decreases in there. So then the thing is well it’s cheating.

You know cheating on a test by looking some over someone’s shoulder is still different from using AI to write a paper. But we then looked at the different items that would point to plagiarism or copying. And it’s it’s subtle.

There’s a line where they don’t want to use it for their entire paper. They do want it to help them get started or to polish off some of their ideas if they’re having a hard time with it. What they really like to use it for is to have it explain concepts to them they don’t feel like they understand from class.

And they’re coming up with a lot of really creative uses for it. So there’s this interesting trajectory of you know this gray zone of use and so that was you know really there and we talked a lot about in that original study but the top line was oh it didn’t increase. That has been misinterpreted at times to mean eh don’t do anything about it.

It’s like no if you look at the numbers it was already high to start with. So, if you really are concerned about cheating, we need to like go to cheating and think about like what’s going on. And I’d channel my colleague Denise Pope and she’s been working in the space for a long time.

A lot of it is school engagement. The kids are stressed out. They don’t feel like the assignments are worth it.

It’s not relevant to things that they’re doing. And if you really want to address that, then okay, let’s address that. If it’s, you know, just this idea that they’re very corruptible beings, well, we’re in a tough place, in terms of how to respond to these super corruptible little people who are going to be inheriting the world in the next decade.

Yeah. I think I mean this is why I think we just need to get rid of like cheating as the moniker for what’s happening. It’s it’s like well, it’s like the AI literacy.

I don’t love the term, but it’s also like the term everybody knows and you go with it and it’s just sort of, you know, the sociology of language and how it works. Yeah. Oh, that’d be an interesting if you have your recommendations for someone who I know some sociologists, but you know, that’s a that’s fascinating.

After this, I’m pretty sure you’re never going to invite another college professor again because it’s like I’m getting, you know, my speaking turn. I talk for eight minutes, Alex gets one minute, I go on for another 13 minutes, Alex gets one minute. Well, I I have plenty I the point of this is actually not for me to talk.

I have plenty of opportunities to talk. This is really I mean this is a chance for me to learn and then people just sort of are along for the ride. Okay.

I’m going to push a little bit and don’t I’m not actually debating it for the sake of debating it. This is truly just something I’m trying to unpack and careful. You’re going to want to get a PhD one of these days.

You’re starting to exhibit those qualities to get a PhD. And truly, if I could reinvent myself, I would go get a PhD in anthropology. And maybe sociology and like linguistics.

That could be really fun. Yeah. But let’s replace the word cheating with shortcut though for the purpose of this discussion.

Shortcuts are not universally bad. In fact, it in the workplace we are we use shortcuts all the time. You’d be foolish.

Automation is sort of a shortcut in the interest of whatever worker is getting a little bit less of the repetitive activity or for the company that’s going to have a better bottom line. So yeah, I mean we we tend to appreciate shortcuts. With the academic stuff where it’s been hard now is we’re heading towards almost like the symbiosis of our lives with technology and we’ve been very used to oh no. So I said at the very beginning the big shock was AI can now do these things that we thought oh no that’s only a people thing.

Only people can do that. People must be trained to do that. They’re the only ones who can do those things.

And that has been you know turned a bit on its head. There’s a bunch of things that we now have the capabilities to do that we thought were off limits. And we’ve set up an education system that aside from the child care and the citizenship and other things was to really train those things in people in a world where that was not automated.

But the world changes all the time and it’s pretty hard for the education system to you know turn on a dime. But doesn’t doesn’t need to I like my I really I really am convinced that what’s what’s different about this new flavor of automation like if you think like like spreadsheets or just computerization in general it’s it was shortcuts but shortcuts that generally speaking they they replaced routine work they didn’t necessarily replace thinking. Now there’s like some sort of like, you know, there’s certain models that actually could replace certain decision-m with decision trees, but like it most of the, you know, if you want to get picky on words, I’ll pick on replace.

Replace is very loaded by itself there too. And that’s the fear. Whereas a lot of times it just changes things.

The nature of the work changes, but it replaces tasks. So I think the argument sometimes goes you know technology and computers the ATM machine didn’t actually replace it replaced bank tellers but the total employment in the financial sector increased significantly because now banks are doing all this other stuff. So yeah, I guess I guess there’s a bit of a semantic about there’s some semantics about are if if a basket of your tasks are replaced with different tasks, is the job replaced?

Yeah, this is a I don’t know. I have to think about when I go to college and they have the selfch checkout lines and you know back when I was young and people were first seeing the selfch checkouts like there was this movement to reject them. We’re not going to let this displace, you know, our folks who are working the register.

We choose to not use that. Anyway, those lines are just as packed as the lines with the people. And there’s also one employee who’s there hopping around with the folks doing the selfch checkckout.

And then there’s still employees who are like helping to direct folks to lines whether it’s going to be the selfch checkckout or or one of the ones with people there. So, you know, it might have been that they didn’t need to hire an additional five checkers in there. They have one instead of five to accommodate that station of six selfch checkckout kiosks.

And maybe like the number of employees had changed, but I do not see a successful case. I know there’s been attempts like with Amazon Go or other sorts of things where we do not have employees in the stores. It even if you buy stuff online there are the help desks and call centers and moderators and things that so it’s not a peopleless business it’s just the work has changed sometimes in ways that we like and sometimes in ways that we don’t but it just we change you know society changes press away no I I think there might be something materially different about using a shortcut to to scan items in a grocery store to fill out to to make bookkeeping more efficient to you know searching for information like there’s it I completely understand the idea that there were probably people and in fact I was talking to somebody I forget who it was you know the push back against putting computers into libraries and like well students need to learn the Dewey decimal system and it’s just like no there are there are examples of things like this is the way we do things.

I don’t think you can really make an argument that the Dewey decimal system is like a fundamental part of let’s say being able to think. I think writing is a little bit different. I think writing is actually Peter Gold at quill.org he really keeps pushing me.

He’s like, "Writing is thinking." And I was just I just had Tony Juan who co-ounded Ed Surge and he does a lot of writing and and he was talking about sort of does he use these AI tools and he’s like yeah you know some like he’s played with them but like at the end of the day the the really hard thing about writing a newsletter or a book or whatever it is is that you’re actually trying to express some some idea that you have And if writing is thinking, then what does it mean to to is isn’t there a risk that a student will develop the instinct to whenever they run into a challenge rather than push through that and really challen like and really force themselves to like do the hard part which is actually like getting the words and ideas in your head onto the piece of paper which in many cases requires you to really understand what it it is you’re writing about. If your first instinct is to go to your AI tutor, say, "Hey, I need help with this." And they’re like, "Oh, I’m not going to give you the I’m not going to give you the essay." Like even setting aside the instance where they get the essay. But even just like getting help with the outline, it’s like, is that is that like reflexive is is there a risk of over sort of like an overreiance?

Yeah, I think that’s the current fear everybody has right now. And they’re pushing that we need to save critical thinking. We need to make sure critical thinking is what everybody’s learning to do, especially now because there’s AI.

And I think there’s an argument to be made that knowing knowing when to use AI is actually part of AI literacy. Like this is now at the point I will go and use this tool of knowing what it can do for me. Here’s the skepticism that I’d put in is you know I obviously I’m not that old.

I wasn’t alive at the time. But in the scholarship on the history of literacy, there was the move from oral cultures before we actually had developed writing systems in there. And there was real fear and anxiety.

Not everybody learned to go read and write. I mean that was something for the privileged few. It actually was excluded intentionally to populations, you know, history United States with enslavement.

So it wasn’t a universal thing. But as that started to become more integrated in things the naysayers were saying this is terrible. It is going to make us forget.

We’ve learned how to remember things to tell stories and and rely on that. Now people are just going to write it down on a piece of paper and no one’s going to remember anything anymore. And so that was sort of the the freak out of that.

And granted, you know, there’s things that change where I rely on maps in a very different way as I navigate through the world or I there’s a lot of people’s phone numbers I literally don’t know cuz it’s stored in my phone and I don’t have to remember them and dial them in in the same way when I went to a pay phone or did our our home phone in there. So, I mean, I think it’s it’s really hard to predict. The other example I’ll put out there is you know in the 80s there is a song called video killed the radio star and it was yeah the 80s were great.

It’s the idea that oh video is so much more superior. We’re never going to do like these audio recordings. People aren’t going to sit around by the radio after dinner as a family to like listen to the story hour or the news.

But now it’s the year 2025 and we’re sitting here recording a podcast. You know, it it’s still there. We look better on video, so I think it’s better for people to enjoy it on video with us, but there’s times in the car that you listen to it or when you’re working out.

And so, we just have these different ways that this has fit into different corners of our lives. So, I wouldn’t say radio is dead, but I don’t think radio has the influence that it did in those times in ways. And people are saying print journalism is dead or broadcast television is dead and we haven’t seen it go dead yet but it is certainly being affected in terms of its level of influence.

So, you know, when we see AI, I do agree that and just as a parent, as a person working in education, yeah, there’s times when I probably would not want the kids to be using AI and learning some other skills first and foremost, just as much as you wouldn’t want to stick screens in a 2-year-old’s face all the time, because that’s just not really what our biology was suited for. So similarly knowing what the appropriate times for things are but you know you could say like oh the automobile people will just drive everywhere and get fat. Well maybe that did happen.

I think that did happen also. I think we I think we I think that that did happen and I think we also remember a lot less. I mean I was I think it was Guns, Germs and Steel which I know has had some criticism.

But I read other books on anthropology and if you go back to sort of like you are going to get your PhD, aren’t you? I it’s I I yes I find anthropology fascinating. I just wish I had the time to indulge in it.

But I think you know the social intersection of AI is like close enough. It is a form ofology. Well, it really is and I think it’s it’s what’s been missing and why I’m so drawn to your scholarship and your voice in the space because there is there is this techies basically have like completely have dominated the conversation about artificial intelligence and and I think that they’re actually a relatively narrow part of what has to happen like what you’re talking I mean you’re even like every every time I try to get specific you sort of zoom out to this sort of like societal lens And and that’s because we are dealing with something that is it’s truly crosscutting.

But to go back to your I mean early humans that when they were in huntergatherer tribes, they they I think it was like they literally would memorize like hundreds if not thousands of plants and they knew exactly which plants were edible and in some cases they were like through oral tradition, you know, memorizing like really complicated you know, whatever the the the the knowledge that you needed to be able to survive. And and so that’s changed. And I think the example is really good because I think the point that I’m drawing from that is so we we lost a skill, but but writing unlocked so many other opportunities for us to to do more thinking to to advance our yeah, our ability to to think and to to to tackle problems.

And I think that brings to that bring that brings us to like what is the result of technologies that make it easier to do stuff. So like you know before computers and you know students had literally had to like write pieces of paper and I think they would write like half a page per week on average in high school and today it’s like three to five pages per week. And presumably if you work in a accounting firm, you probably are able to get through a lot more spreadsheets than you used to if you were doing them by hand.

I guess there’s this this question of how do we ensure that teachers are striking the right balance of like increasing the the goalposts like if we assume that okay we can’t it’s like you go back to the beginning of the internet. No, the internet has caused a lot of bad stuff to happen. It displaced lots of jobs.

There was no answer other than this is happening. But on the flip side, it started a lot of businesses and a lot of jobs and it’s expanded reach for lots of things. I mean, the fact that people will listen to us talking right now, thanks internet.

Yeah. And and you’re right, like YouTube radically disrupted the entertainment industry. And so so just because change is happening doesn’t mean that there’s we it doesn’t make sense to spend too much time trying to resist.

We’re going to zoom out again. My colleague Roy P turned me to this really great study of the history of housework and there’s a period of time cuz like you know imagine before we have all our Bosch dishwashers and things I will disclose I do have research funding from the Bosch Foundation but I also do think it’s a great dishwasher and refrigeration and and whatever and so like just the labor that we would spending on that. Well, as what happened was that was domestic work and these conveniences started coming out and instead of creating more time to just do nothing, it became something everybody had to have such that most houses have a dishwasher.

Most houses have refrigerators and stoves and things of this sort. And it morphed into not a luxury but a necessity to just simply keep up. And you could see that even with food like you know we still prepare food.

We may have more packaged foods or other sorts of things but we still do that but we also have restaurants that are helping out and contributing in that way as well. So a lot of these things it’s hard to look at it as a sort of we just lose stuff. We we stop doing the things and in some ways like we do so many other things now with and because of all of these kitchen gadgets.

And it is possible if we don’t socialize and reaffirm the importance of what we do as humans and how we as a culture work and operate and and feel some agency in that that we could go through some really rough patches would be my guess. We don’t know what the future is actually going to hold, but you know, I I think it would be wise to make sure everyone feels wise about AI. So that when they use it, they use it responsibly.

But I would say the same thing goes for be wise about how you search for things or be wise about how you respond to advertisements or things that are going viral and understand like where does this sit and everything. And you know, for all the examples of folks who could forage and identify plants and do other navigation, there’s things that we value socially at certain times. Back then, it was going to be eating.

Now, you know, like I look at my kids and there’s stuff that they know about social media and like peer interactions and things. We were talking about the ’ 80s. We know the great music of the ’8s, the great hair of the ’8s.

There’s just so much stuff we know. So, if we actually had the ability to like count out items of things that we know or stuff that we could do, it would look pretty comparable. May even look a little bit better now just because longer lifespan and the ability to get even more stuff in there.

So, it’s hard to tell. I mean, I I think just as you were hinting out earlier, change happens. Some of it is going to be really bad stuff.

Some of it’s going to be really good stuff. We’re always just trying to reduce and prevent as much of the bad stuff and make sure that we have equitable access to the good stuff and then something else changes and that’s what it means to be human. Zoom back in to computer science.

Yeah, point taken. You’re right. We probably actually consume just to just to evaluate what you’re saying.

I think we probably we probably do deal with and manage way more information than a hunter gatherer did. I mean, I’d argue that’s part of why people have so much stress and anxiety right now and are like feeling so, you know, distracted and and running around all over the place is we’re bombarded and our systems were not really evolved to do that if it wasn’t involving running and chasing things down. Computer science strikes me as something that is generally heavily anchored on jobs.

It’s like you study computer science because it’s going to give you these these skills to go and work in these fast growing technology fields. And there seems to be a lot of like dissonance where you have, you know, people here in Silicon Valley and there’s this exuberance about AI is going to just replace software engineers. I mean, this is going to augment and massively billboards that Yeah.

Advertise companies that will replace your for you. It’s pretty dystopian. Yeah.

And and the VCs, you know, this is qualitative, but you know, VC batches are that the founding teams are getting smaller. They’re getting to, you know, their series 8 with like much smaller teams. And it’s partially vibe coding.

It’s not just vibe coding actually. There’s also just like more general downward pressure on cost, what what have you. But the so you have like Darede anthropic who is saying basically you’re not going to need to learn computer science.

Jensen Hong basically said the same thing or he said I don’t think if if I was talking to my kids I don’t think that they need to learn computer science. Everybody’s going to be able to like use just English to code. And and yet you dig in a little bit and you what people also say is that oh well you know the the most effective vibe coders people are using AI as a co-pilot to write code are people that have classical training in computer science and in this world where there are let’s say it is more competitive to get the really good paying computer science jobs surely companies are going to place a premium on someone there.

Again, everybody’s going to know how to use AI. That’s going to be a baseline assumption. So, they’re going to want someone that knows how to use AI and has that classical training in computer science.

And sure, maybe computer science class classes change to be less focused on just learning lang coding languages and more building apps and getting experience with like product management and you know like it it’s probably a bit more holistic but it but anyways that’s that’s my that’s my current and latest interpretation of how computer science changes which is it changes but not as much as some people are claiming. And I think you can extrapolate to other domains of expertise, whether that’s like communications and marketing or design or art. I don’t think everybody’s just going to use AI to create art.

I think people are going to want to pay human artists to get something that doesn’t just look like what everybody else can have. But well, let me put it science bid. Does that conflict with No, no. I mean I this is where you know in wearing my learning scientist hat we really puzzle over what does it mean to learn and what does it mean to know something.

So like to learn computer science well what are you learning when you learn computer science and I think a lot of folks by default say oh you learn to write code okay but what people are doing is they’re learning to structure problems in a certain sort of a way to think about certain processes and potential automations and practices for what to do. So when you study computer science that’s really what you’re learning. It’s not the specifics of this programming language which will be out of date in 10 years.

You know a lot of coders have the ability to learn and code in other languages. So the question is what is it that we need people to learn that is represented by those success cases within computer science or the things that have tended to have the most use and value for people who’ve benefited the most from that learning experience. And you know it’s hard to say oh you don’t need to learn computer science but I think also just with the writing can we think about writing beyond like the five paragraph essay you know and most of the most interesting non-fiction that we read or writers like op-eds or you know really interesting publications or long form that are not the five paragraph essay but we got very used to no that’s how you learn to write is you do this and seeing what does it mean to actually write and to do the thinking that the writing feels like it is.

And I’m a big proponent of using writing as a major thinking tool. For myself, that’s what we want to preserve and emphasize and keep our eye on that ball. However, you know, we learn things based on what we currently know.

We know what we’re participating in or engaged in or surrounded by. So, we have to speak something to the now. But we just can’t lose focus and like you know the gaze all is only on the now.

There are some things that have some enduring usefulness and seem to have some robustness. You know, math tends to bring out some of that logic kinds of things. Unpacking the human experience by reading literature or enjoying different sorts of media, computer science with the different kinds of creations of of logic structures in there.

So it may change over the generations which ones are the ones that we must do. But we also tend to fall into these little traps of conflating the thing that we tend to do superficially with what’s the actual thinking and insight and expertise and how that develops for those. So you know I would be supportive of people going on to computer science.

Now, if it’s computer science, just so they can make x amount of dollars, I would encourage them to think about what they’re looking for in life and if that’s how they want to pursue things because if they want to just focus strictly on dollars, then you know there’s many choices that they can make. Yeah. So I think we’re actually converging now because I’m I’m understanding your your point your push back on, you know, should we be worried about AI automating writing?

What I’m hearing from you is the sure the five paragraph essay like we may not want to die on that hill. And the question is that like well if you don’t have to sit and write a boring five paragraph essay you’re not going to have productive struggle. It’s like also it’s it’s a struggle in part because it’s boring and it’s not really relevant and nobody I haven’t written if I written the subset kind of accounts but even that it’s like it’s different vernacular.

So what I’m hearing from you is a an English teacher needs to to abstract like what is what is the point of being able to write a five paragraph essay? And so if the point is being able to to think and convey ideas about what you learned in this novel, there may be other formats for getting students to convey that knowledge. And oh by the way, that’s probably a good way to address the potential for shortcuts or cheating is if you know they’re having to come in and you know reenact you know play on play a persona from the story and maybe you use AI to create sort of a role playing game where they’re vision use AI to actually you know record your own 10-minute movie interpretation or something with a different context and like reflecting the complicated human condition in there.

And in some ways that could be a real benefit for some students. Say you have a you’re dyslexic or this could be a way to show how keenly insightful you are and not get hung up on the thing that has never quite worked as well for you. So yeah, I I think partly the teachers, but also it’s the curriculum developers, it’s the parents, it’s the policy makers.

We all need to make this shift and be willing to embrace it, but also acknowledge that it’s not the kind of shift that we feel comfortable with. Change is not things that people tend to like. It’s laborious.

It’s scary. It’s uncertain. There’s a little bit of unpredictability to it.

But that’s that’s where I think everyone’s seeing it in the water. We need to change assessment. We need to change what are the kinds of activities that we value and what actually causes certain outcomes in the end.

Is it you went to college or you spent a lot of time thinking through very complicated problems? And we tend to conflate a lot of those things. I am in favor of people going to college.

I do work at one, but the same thing I think goes for writing. You know, I think there is some very deep thinking going on for families that didn’t have writing. And I think there’s going to be some deep thinking for folks who, you know, go many different life paths in there.

And we want to try to keep that eye on the deep thinking part. It’s important to have some folks that are zooming out because we we’re basically going to be chasing our tail in perpetuity. By the time the education system is able to shift, the hockey puck is now somewhere else.

And and if you’re not if you don’t have some perspective on that that sort of like these like deeper questions you know we’re always going to be sort of like we’re we’re always going to be reactive but what like what advice would you have for organizations like aiEDU CSTA you know innovate edu all the Stanford Accelerator for Learning I guess maybe don’t advise your own organization but civil society organizations that are trying to figure okay how do we how do we help this along? How do we what what is the what is the like the critical path to getting the best possible chance? This is going to be very hard like it like making a mistake, but we have to try.

What should the focus be? I mean there’s obviously you’re advancing the research and sort of like the deeper thinking, but what are the other sort of critical components? This is where I’m going to say I am not a policy expert and I think that that is very much its own expertise.

But we do know policy can be a very big carrot and stick for certain changes to happen. Of course they could be done very poorly and other times more thoughtfully. So I would first off say the community should collaborate.

There is unfortunately some friction where different groups are trying to have more dominance on the thing and you know we’re all in for the same outcome at the end. So if we can all work together and build on whatever insider assets or connections one another has I think we’ll get to where we want to go faster. And in our current system we’re going to need to help steer policy.

So like part of that I would say is if we are going to look at big shifts to what’s happening in schools we need to pave the way for that in terms of the policy system the support for teachers the new kinds of curricula or learning experiences and it may end up looking pretty different than what we have now and there’s also the risk for backlash to that and so I think we should you know do as much as we can to pave the way so it’s easier. Actually be curious what you’re reading nowadays, but one of my more recent books that I probably talked about the most in the past few weeks is abundance Ezra Klein and Derek Thompson’s book. And you know, there’s a lot of interesting points in there, but a lot of it is we’ve kind of regulated ourselves into corners.

They’ve gotten really really good at creating these like structures that tie us up in knots and they end up raising prices and preventing things from actually happening. We ought to, you know, their thesis is we ought to actually start banding together and like trying to make some forward progress towards common purpose and not necessarily end all regulatory structures but be more empowered to take responsible risks in in our decision-m and so I think whatever we start to recommend and I believe that with the great folks who are working in the space different people are trying to contribute different pieces to that. And we should all collaborate on that.

You know, there’s some folks who are trying to build out frameworks, some who are trying to build out curriculum, some that are providing professional development, some that are focusing on system leaders, some that are just helping to rebrand what AI and education is in in the mass public. And the more that we can do it together, I think would help. And yeah, right now policies tend to be a big thing, but there’s also hitting that sweet spot of the right resources at the right time, at the low cost and low burden, which will be in there.

So, you know, this is still going to be a bit of a gamble, but those would be my broad directions. How about you? I think the same thing.

It’s it’s we are really nitpicking as you as you go and look at all the frameworks like the AI literacy frameworks. I mean there are different weighted like sort of different things weighted. Some are a little bit more technical.

Some are more well that’s the challenge too and that’s where I say like there’s this inadvertent rivalry of like no we came up with this one. No, this is the right one. And it’s like you know that we don’t have to necessarily like zero sum game this.

Yeah. I mean, it’s there there’s far more in common than there is different. I mean, I think I think there are and I I don’t even know if they’re nonprofits, but I think there are some forprofit actors that are really pushing like use training to use AI.

And I think I I I do disagree with that. I think that the the push should not just be teachers to use AI tools, teach students to use AI tools. There has to be more to it than that, although that is a part of it.

But all the nonprofits that I’m sure you’re alluding to that we work with, there’s, you know, I think it’d be interesting. It’d be it’d be actually more fun if there was actually like meaningful material disagreement and this was a a battle of, you know, a battle of ideas, for the real AI literacy. But I think actually we’re like, you know, you’re sort of tripping over yourself getting in con semantics.

And then there’s also the like who’s framework is the who is here first, who’s going to get the funding when there’s only a certain amount of funding to support that. Who’s going to be able to go and work with this state or work with that state. And it’s just part of the the system where I I wish we didn’t have to be in in these scarce resource situations, but you know, we’re especially not not seeing the investment to do this except for in certain private sector things now given recent changes.

I think that’s right. I think that’s a it’s a good call out to when people ask this question of like how can we ensure there’s more collaboration in the space? The answer is we need more funding in the space.

Like the the sum total budgets of the organizations doing this work is you know probably less than a 100 million maybe maybe even less depending on like how you how you we could ask AI maybe it will tell us that I I would not bet on 03 getting that right although we should just do it for the scope of the challenge is is so large that I I think it is it is inevitable that there will be more more funding and philanthropy and investment in this and I I worry that it will take a crisis as as is often the case we will wait until it becomes a crisis and then there will be sort of this lagging you know follow on investment I think you know the work that we can do now is just how can we get as much forward progress as possible so that when there is a crisis or whatever however it unfolds whether it’s about economics or societal impacts or mental health I mean there’s lots of you know doom and gloom that you could you rabbit holes of doing you go down. But we’re clearly better serviced, right, from having people who like fundamentally are curious about the technology and I’d be very happy with I think the vast majority of the AI literacy frameworks. I’d be really curious for your book that you mentioned with this the triangle.

Oh, yeah. The handbook. Yeah, that’s that’d be really Yeah.

And it’s funny because everyone’s like, "Oh, we want to see your definition of AI literacy." It’s like, well, it’s actually a framework. It’s a meta framework for all the frameworks. That’s that’s like the checkmate is I’m going to create a framework about your frameworks and then someone can do a meta analysis of the frameworks about frameworks.

There’s like turtles all the way down. Victor Dr. Lee actually Victor’s fine. I’m most people who know me know I will pepper tough questions at conferences, but I’m also highly casual and and nice and like genuinely doing this for the right reasons.

As I think almost everybody in education is nobody really goes into the education space if they have blind raw ambition. I think they’re ambitious people which is great but raw ambition you’re wrong sector. Yeah.

Well, having having me as a guest in this conversation, I think is, you know, a a good honest gesture of like having the cross talk, sharing audiences, and just continuing to forge those relationships. So that way when we’re on stage at the next conference, it’s, just another like it’s a hug rather than a high five. Great.