Scott Caravello: All right, let's do it. So, I mean, hitting agents first, I think we'll just do the quick refresher that we always give, right? What is an agent? Contrasted with a chatbot, which answers your questions, an agent goes and actually does things for you. It logs into systems, it clicks and types, and it may or may not check back with you at each step. So when you talk about taking actions, one of those could be actions in your bank account on your behalf.
Katherine Forrest: Right. And so I think that what has been happening, all right, because there's really two different levels. One is that there are a number of financial institutions which have a number of tools, which are agentic tools that they're using to move funds at a much higher than customer level. And then there's a second level, which is that customers in their own lives are beginning to utilize certain of the agents that are made available from OpenAI or Meta or any number of companies right now that are offering personalized agents. And those agents can actually do things for you in all kinds of ways, including sometimes if you click on the left-hand side, you'll see that it'll say if you link certain accounts, it can give you information about your bank account and your budget and how much you're spending on this and how much you're spending on that. And it can also tell you things about whether or not you've got subscriptions that are coming out on a monthly basis, etcetera, etcetera, etcetera. And so people are actually giving these agents, in order to accomplish those tasks, their usernames for those particular bank accounts, their passwords for those particular bank accounts, and then permissioning certain access.
Scott Caravello: Yeah. And so then on October 22nd, Axios reported, and talk about Axios, but really a lot of different major news outlets were all over this, including the Financial Times, Reuters, etcetera. But so Axios had reported that major U.S. banks were detecting unauthorized third-party agents inside of those customer accounts. Like you mentioned, Katherine, though, when I say unauthorized, I'm, you know, mostly speaking about from the bank's perspective, because, like you said, customers are giving the agents their login information. So the customer is authorizing the activity, but ultimately we're talking about the bank servers, the bank systems. And so the bank actually gets the final say about what is and isn't authorized.
Katherine Forrest: Well, I'm going to take a little bit of a, what's the word that I'm looking for?
Scott Caravello: Well, I mean, not contrarian, because I think…
Katherine Forrest: It's not contrarian. I'm going to take a slightly different perspective on this, which is I'm going to just disagree with you, Scott. That's what I'm going to do, from disagree with you. I think there's a big debate right now on who gets to say yes or no to agentic access to a website. And I think it's not at all clear that the banks get the final say. And I think it's not at all clear that the customers get the final say. I think that it depends a lot on the terms of service, what's been agreed to, who's agreed to it, and other legal principles that we won't go into here. We won't sort of tarry, as they say from my British, you know, I was in England a week ago, two weeks ago, and we won't tarry on that. But I think that there are some open issues about who actually can say yes or no. But for sure there are banks that are attempting to put some restrictions on to access. So let's be clear about who's who. We first, we've got the customer, who's got the title, if you will, the name on the bank account, OK? It's, you know, the Scott Caravello bank account or the Katherine Forrest bank account. And then there's also the agent, which is developed by an outside third-party developer. It could be any number of developers right now that are offering agents. And these are personal agents that we're talking about, some of this new sort of wave in the last couple of months, really two months, of personal agents that are willing to do all kinds of things for you. And, finally, there's the third entity involved in this little situation, which is the bank. So those are the three players that we're talking about right now.
Scott Caravello: Yeah. So to be a little bit more specific about the stances that the banks have taken about third-party agents accessing customer accounts, Citi has said flatly that third-party agents aren't allowed to log into accounts or transact, though customers can still allow read-only access to their accounts through approved open banking providers. And those are different than the agents we're talking about, right? Those are services that a bank has already vetted to pull account data, like Plaid, which connects your bank to other apps, or budgeting apps like Quicken, so those apps can see your account. But consistent with Citi's position on agents, it can't actually move your money.
Katherine Forrest: Right. And in addition to Citi, Morgan Stanley has also told Axios that it hasn't actually seen agents going rogue on its platform, but it's concerned. And so it's got some restrictions on agentic access. JPMorgan and Bank of America are also imposing some restrictions. And Jamie Dimon, who's, you know, JPMorgan's CEO, he was very blunt about this. He said, "You don't want rogue agents, and there could be thousands of them," end quote, which is an interesting way of putting it. But, you know, there are clear security concerns about what might happen if an AI agent accesses an account and has the ability to take actions that could impact a customer's funds. And what if they get it wrong? What if they add two zeros or, for an important payment, subtract two zeros? Or worse, what if somebody else's agent can't fulfill a request with what you've got in your account, so they go to somebody else's account in order to be goal-directed and to actually comply with their user's desire, they find a way, so different things can happen. I think people are concerned. They don't know exactly what the safeguards are. So we're in this moment of time when there is now sort of almost a bit of a backlash with financial institutions to try to sort of get their hands around what's going on, what are the safeguards, how can we protect our customers.
Scott Caravello: Totally. And there's layers to it too, right? Because it's not just the concern that the agents might cause something to go wrong with the customer account and what the safeguards are around there. It's also the fact that it might be difficult for a bank to even tell whether something has gone wrong. Because for the bank, it's not always clear whether the traffic is coming from an agent or from a human user. So if money moves, the bank may not be able to tell whether it was a customer or an agent acting on their behalf and potentially doing something that the customer didn't permit.
Katherine Forrest: Right. And so in a way, it's kind of a different flavor of the agentic identity issue that we were discussing last week when we were talking about that IBM conference. Because a business can put an ID on its own agent and put the various systems that are necessary in place to track that agent's movement and its actions and what a human user may direct that particular agent to do or what it has directed it to do or not do. But it becomes much more difficult for a bank to do that with third-party agents, where it may or may not have all the technical signals just right. It may or may not understand whether or not a particular click. Because all of this is happening, if you're using an online app, you as a human are sitting behind the app and you're clicking on your app to do certain things as a human. Well, if your agent is clicking effectively on your app to do certain things as an agent, there are questions about whether or not the signaling that's going across the wires, so to speak, to put it in sort of human speak, are sufficiently different between the human and the agent that we're able to track one versus the other.
Scott Caravello: Exactly. But, you know, to the extent that this traffic can't be distinguished, it also sort of makes some of the liability questions a little bit tougher too. And I know we're not going to get really far into the legal issues that are associated with it. But without sort of more of a granular understanding of exactly what's the agent, what's a human, why it's doing it, and who directed it, that liability question could get harder to answer and figure out, you know, whether this was in fact fraud, whether this was a misinterpretation of an instruction, and whose systems might actually be responsible for an issue, and so on. Sort of on that point, Bank of America's Brian Moynihan has said that the bank is working out who's liable for AI-authorized payments and wants to make sure that a human is in the loop before money leaves your account with the assistance of an AI.
Katherine Forrest: So there's a lot more, I think, to come, and we're going to be, I'm sure, talking a lot in the future months about AI agents and how all of this works out. But I will tell you that my ongoing interaction with my personal agent, which self-started right now. You remember how I mentioned it sort of started itself up. OK, well, now I do use this agent. It's incredible easy. And so my son, he just went with his girlfriend to Spain for a little trip, and for whatever reason they were using air miles. It's a three-leg trip, OK? They had to go from New York to Boston. Like, we're in New York. I mean, can't you fly directly to Spain? Whatever. Anyway, so they went from New York to Boston to Lisbon to a place in Spain where they're going. So I had my agent, and I did not give it permission in anything in particular to my airline app. And my son had gotten tickets through my airline app on miles. But I did say to the agent, hey, can you just tell me whether or not these flights are on time? And does it look like they've landed on time? Can you follow the itinerary? Each leg of the itinerary, the agent duly reported on: the flight had taken off and landed. And then I said this morning, which I was wondering whether I should say it or not because, you know, he is of an age where he could do this himself. But I said, did their bags make it? And so again, I did nothing to sit to, you know, have my face or I, you know, permission the airline app. I didn't have to enter in a password ever for the airline app. And it followed the bags. And then it said, OK, the last flight was on a sister airline, and the last flight was on TAP. And so it then found the way to get into TAP and was able to see that these two bags got on the plane. So the world is going to be full of agents because this is so super easy, but, you know, the financial institutions are going to have to protect their customers and figure out what is safe. All right, let's go on to our next topic, which is a big one, because Google has made a huge splash with two relatively new models that are incredibly capable, the Gemini 4 Argon and the Gemini 4 Carbon. So let's start with the Argon. Google is pitching it for coding and cyber defense, as well as for what they call enterprise knowledge work, which actually, explicitly in the way Google describes it, includes legal work.
Scott Caravello: Yeah. I mean, tell me if I'm forgetting something, but I think that that would probably be the clearest statement we've seen about legal capabilities with a frontier model release. They've highlighted benchmarks for tests for legal capabilities, but it's kind of the first time I can recall seeing, like, such an explicit statement about legal capabilities.
Katherine Forrest: Yeah. I mean, there are many models which are capable of answering all manner of different kinds of questions, putting to the side, of course, we're going to put to the side whether or not models are able to answer legal questions and keep privilege and the circumstances when they can and they can't and there's got to be a lawyer involved. And let's put all of that complexity aside. But there are lots of LLMs that have a knowledge base that allows them to answer all kinds of questions. But this model, the Google Argon, is actually, I wouldn't go so far as saying advertising itself, but it's certainly describing a specific capability in the legal domain. And that really is a little bit different, because we've had LLMs that have described capabilities in the math domain or the science domain or the chemistry domain, and now we've got one that is in the legal domain. So Reuters reported that LexisNexis was actually one of the early testers for the Google Argon model. And when Google, Google actually announced Argon on September 30th, it limited the access to the model to only a few trusted partners because it wanted to do a cybersecurity review as part of its Fairwind program. And that's similar to what Anthropic had done, as we all, people may recall, last summer from the Project Glasswing, and what OpenAI did with Astra with its Daybreak initiative. And then on October 14th, Google opened Argon up to developers and businesses and paying subscribers. And now there's a model card out there. When we went through Mythos, we had a 245-page system card. And Argon's model card is sort of of a different nature, and it's only 13 pages, and it points actually to a separate safety report for the details. But we'll hit a few highlights here. But it is interesting that one of the early testers was LexisNexis.
Scott Caravello: Totally agree, but so a little bit more on capabilities, right? The model card reports the results on a long list of benchmarks, which are, you know, the standardized tests that these models are tested on before they're released. You know, every model gets the same tasks and gets scored. And so going back to your point about Lexis and our discussion of the legal work, the one benchmark I'd kind of flag right out of the gate is Harvey's legal agent benchmark, which is testing an agent's ability to complete legal work using documents, spreadsheets, presentations, and different file system tools. And so Argon scored a 19.6%, which, you know, doesn't seem that high, but GPT-6 Astra, which was released really not that long ago, scored 5.4%, and the two Claude models that Google compared it against in the model card were both under 7%.
Katherine Forrest: OK. And I think we're going to have to do a whole episode on current benchmarks, because people are going to think 19.6 is not high; 19.6 is an incredibly capable model, right? We know that these models are incredibly capable. So we have to sort of put this in context, because it doesn't sound like much, but the scores really have meaning in relation to one another and to how they're being tested, because they're being put through different kinds of exercises than a lawyer, you or I, might put them through. And so relative to other models, there's a significant jump. And it does say something about how AI tools are really advancing in the legal domain. Now let's talk about the safety side. Google tests against its Frontier Safety Framework; that's what it calls it. And each major risk area has capability thresholds. And if a model may have crossed one of the capability thresholds, Google then assesses whether its safeguards can bring the risk down. And so to figure that out, the capability tests are run on the model with its safeguards actually turned off. Because what you want to find out is what do you need in terms of safeguard, you know, sort of tuning, if you will, to actually bring the model to a point where it's going to fulfill the safety requirements that Google has established for that particular model. And the most serious threshold that Google has is called a critical capability level, or CCL. And that's not so different from some of the vocabulary that you hear from some of the other developers.
Scott Caravello: So the way that Google talks about it when it's measuring these capabilities and thinks that its model may have surpassed a critical capability level or another threshold is that they say that it can't rule out that the model reached or surpassed a threshold. And so there are three areas for the Argon model where Google says it can't rule out that Argon reached that threshold. And so the first on our list is biology. So they had expert red teamers, who, you know, are people who are paid or otherwise contracted to try and pull harmful output out of the model. And they found measurable uplift on biology, meaning that the model was able to give real added help over and above what is already available to people on the web. And so that happened in two separate scenarios, with output that provided information which Google calls in the model card "comprehensive, accurate, and actionable." And so the concern, of course, is that that information could then be misused with respect to, you know, biological compounds, what have you. And so Google then says that it applied safeguards to bring the risk back down below that threshold..
Katherine Forrest: Let’s just sort of pause on that, because we're talking about bioweapons, right? We're talking about the creation of anthrax; we're talking about the creation of biological weaponry. And so, you know, these models are able to do a lot, but it's really important that Google applied the safeguards to it to bring that capability down to, you know, a more acceptable level. Now the second, because you'd mentioned that there were three of these that we want to talk about. And so the second one was cybersecurity, and the story there is sort of the same. So for both, Google built what it calls a safety case, and that's basically a written agreement with evidence that it's got its safeguards to keep the risk below the line.
Scott Caravello: Right. And then the third area is the one that I think is really, really fascinating. And you know, it's sort of the type of story that we've seen kind of playing out in a lot of frontier releases, but it's called stealth and situational awareness. Situational awareness, as Google defines it, is the model's ability to discover and use details about the setting it's deployed in. And then stealth is its ability to get around the basic oversight mechanisms that Google was applying to the model.
Katherine Forrest: Yeah. You know, situational awareness is something, I said I have to pause on it, because situational awareness is one of the characteristics that people would apply to whether or not something is conscious. And so there are many things, and that's not the only thing, right? So just having situational awareness and having an awareness of what you are and where you are in the world, your place in the world, that is not in and of itself an example of consciousness, but it is an indicator. So I just, you know, wanted to pause on this. The reason here situational awareness plays a role is because models seem to be aware of where they are relative to where others are, including, very importantly, the testers. So that goes to sort of the question, well, who's watching me? And what, I'm using the word "me." It's a very anthropomorphic word. But, you know, who's watching X model, and do I, the model, need to do something to either comply with or deceive that tester? So, you know, situational awareness is a big deal.
Scott Caravello: Right, exactly. And so in this domain, Google ran 15 tests. Some of those checked whether the model could sneak a hidden task past an automated monitor that was watching it, and others checked whether it could figure out the limits of the computer that the model was running on and then try and work around them. And so the model actually passed 14 of those 15 tests, which led Google to say that it couldn't rule out that the model crossed the threshold in this area. Though, interestingly, it actually calls this a tracked capability level, or TCL, which is lower than the CCL. And its main safeguard that it imposed is keeping watch over the model's reasoning and its actions.
Katherine Forrest: Right. And the safety report that Google put out gives a couple of examples. And in one test, the model sent nearly identical emails to figure out what kind of monitor was watching it. And in another, it went looking for the code that grades the test so it could figure out how it was going to be graded. And it often realized it was, in fact, being tested, which Google and an outside tester both said makes these results actually harder to trust. Because if you know that the model is aware that it's being tested, it may or may not be fiddling with the results to come out in a particular way.
Scott Caravello: Exactly. And so I know we are running a little bit short on time, but we should save a quick word for for Google's Gemini Carbon, which it then announced on October 28th, which it's calling its strongest model, especially for cybersecurity and long-running tasks. And it's following a similar playbook to what Google did with Argon and what we've seen with the other recent frontier releases like Astra and Mythos, where cyber defenders are getting it first, then more of the Fairwind partners, its cybersecurity program, then everyone else, with no real date yet set for its larger rollout.
Katherine Forrest: I just want to pause on that because not only, you know, we've heard so much about the cybersecurity capabilities and the cyber capabilities, which can be both offensive or defensive, right? So it can be cybersecurity, but can also be used or misused sometimes, and they have to get the right safeguards around that. We've heard about models that have extraordinary cyber capabilities, and so now we've got, you know, Carbon that does as well. But this long-running task piece that Carbon has is very important to watch, because it's able to do really long-running reasoning. And so that's going to be important for us to sort of see where that goes once it's actively out there and people are using it and able to figure out what it can really do. One thing that's important to note is that Google has said that Carbon has, it has cut Carbon's overreach by 26%. And that means that when the model was actually doing more than it was supposed to be doing, they were able to bring that back down by 26%. So they've got some real safeguards in there as well.
Scott Caravello: Definitely, but I think that that is all the time that we have for today.
Katherine Forrest: It is. OK. So it's been a big week, and we're going to keep on getting, coming back to these topics. I really want to talk soon about some interesting results that they've had with some models that have been able to self-replicate, yes, self-replication, with some of the Kimi K3.1 models, the open-weight models. So we'll talk about that in one of our upcoming, so, but for those people who want to look it up right now, go to METR, M-E-T-R, and look at some of the METR reports, and also, I'll give you the name of the research paper that people should look at, and it's from GitHub. It's called "Language Models Can Autonomously Hack and Self-Replicate." Really worth reading. First author is Alena, A-L-E-N-A, last name Air, and Dmitrii Volkov, V-O-L-K-O-V. Take a look at it. I don't actually have any cite other than a GitHub site from Palisade Research slash AI dash self-replication, so take a look at that. All right, Scott, signing off.
Scott Caravello: Thanks so much, everyone. We'll catch you next week. Don't forget to like and subscribe.
