Devin podcast mentions

Quote

And IDEs, ID tools with embedded AI assistance. to for example, do uh tasks asynchronously. Like for example, you can give it a a job that will take even an AI maybe like ours to complete And it will think for some time, go off asynchronously, ask you follow up questions. and and whatnot. And then you have another part of the space which is about autonomous development. There are tools in this category. There's uh I believe uh Devin from Cognition. falls into that category. We operate in that category. And the idea here is that you Hit build and out comes the PR, right? But the PR that comes out is um already tested, validated, everything works. And it's exactly how you intended it to be. Right? There's there's no errors. The code is Right. So the biggest challenge uh that we have on both sides of the spectrum is code access. acceptance, right? You can write a lot of code uh and code is a commodity now, like getting AI um code that is ready for production is a completely different story, right? Because you have uh Uh on one hand you have these greenfield builds or like new products.

Quote

Nelly. maintainer of the CLI, but I won't take claim for the anonymous upload. That's actually the origin story of Netlify. your a zip file or folder from your desktop onto a website, it would have a live URL with no sign in. And so that was the origin story of Netify. And it just persists to today. And it's just like uh it's really nice, interesting that uh both Boltz and Cognition Devin And then a bunch of other sort of agent type startups, they all use Nullify to deploy because of this. They don't really care about the other features on that way. Like if you build an interface for computers specifically that It's easy for them to navigate, then they will be used in agents. And I think that's a learning that a lot of developers That's my bolt launch story and not if I say all that stuff. Uh and yeah, I just wanted to back to like the web containers things, right? Like uh I think you put a lot of weight on the technical Uh I think you also are just like very good at product. So you've you've like built a better agent than a lot of people who the rest of us, including myself, who have tried to build these things and we didn't get Don't shortchange yourself on products. But I think specifically on on infra, on like the Alessio has backs E to B which we'll have on at some point talking

Quote

and they hypothesized that it should be open source and community driven. This is the creation story of Open Hands. So we're working on an open source platform for AI driven development. So every software engineer in the world right now is aware that AI is changing how we do our jobs. And we saw that change starting to happen almost exactly two years ago. I think big event there was a cognition release 13. demo video of Devin back in March of 2024. And that was really the first time we saw an agent that was working on code. It wasn't just a chat GPT inside of the chat window. spitting out a function or something like that. We'd seen AI that could write code, but we'd never seen AI that could actually run the code that it wrote and then add debug statements to to see what was was going on, uh Google an error message that came out and basically do that whole inner loop of development until it got to a solution. That was really exciting to see, right? It it solidified and my imagination what the future of software development was gonna look like. But it was also a little scary that change was being pushed by this stealthy company coming out of a black box. And it was very obvious to me and my co-founders that you know if the job of the software developer is going to be changing, the development community needs a say in how that change happens. So we started a project. Project at…

Quote

It actually is. Yeah. So uh so my background's in a mixture of uh dev tooling and natural language. language processing and in particular I've been working in open source dev tools for over a decade now. And uh, you know, there was there was this moment, you know, maybe a few years ago when Everybody realized Chat GPT was really good at writing code, uh, where a lot of people started getting really excited about. the changes that could be possible for software development uh as LLMs improve. And then there was this moment in March of 2024 when Cognition released their demo video of Devin. Which was the first demonstration of a of a true agent. could run the code that it wrote, could, you know, Google an error message that came out of it, could add debug statements. So basically do that entire inner loop of development until the tests were passing, the code compiles and it's reached some kind of solution to the problem. That video went viral. I think pretty much every software engineer in the world saw that video. And I think I and and all those other software engineers got very excited about what the future of software development is going to look like, but also a little scared that that was happening in a black box. Again, I've been working in open source dev tools for a very long time and immediately on seeing that video it was very So my co-founders and I started a project…

Quote

automate more and more. And I I think that you're right. So um Shoshank is referring to uh this project that I made. I think it was probably like like two or three years ago now, um, that I call GPT engineer. So um I built flawed code like probably two or three years before Claude Code actually even existed. Um before, you know, before cursor, uh before Devin, uh before all of this, like I had GPT engineer which actually um worked uh as well. So So uh yeah, but I I gave it away for free and didn't really market it much. So Um I that's why I'm not a billionaire. Because I was I was actually first. I I I built Uh like the pretty much like what they had for uh something like CloudCome. And uh, you know, looking at like the Claude Code uh source code from the leak. GPT Engineer, which I built like three years ago, was actually not that far. far off um from what I had, but they just waited for the models to get better. It's not that different. Yeah. No, it like the the the the logic was was the same. Um but yeah I I think that uh

Quote

Right. And uh again we don't know too much about it. It could be uh built on uh flux but uh they do claim that this is uh trained And seemingly on their own. So yeah, impressive. Another impressive thing to have shared from Xai uh given that they're sort of in catch up mode. Next up, Cognition Labs, a startup, uh they kind of made a Splash with a demo of Devin, a software engineer. Well a many a few months at least since they uh initially previewed it. And uh you can use it if you're a subscriber. So you have to Pay five hundred dollars per month uh for individuals and engineering teams there is an integrated development environment ext Extension, an API, also an onboarding session and various things like That uh so yeah, another piece of the Agency story. I think they have been integrated pretty deeply into a lot of programmers. workflows, I know for me that's definitely the case. And uh now there is the race

Quote

Le gag. Parlons du fiasco d'Anthropic with their model Mythos. Exactly. Fleuron de la sécurité. Sauf que le jour même de son lancement, l'environnement compromis. Non, non, un simple groupe de petits malins sur un forum privé Discord a piraté l'accès. Comment? Bah ils ont juste deviné l'URL du modèle en Voilà, parce que c'est là qu'on la met dans l'ancien bâtiment. It's ridiculous. Oh wait, l'agence de renseignement American. Nessa intègre ce map modèle Mythos, which is encore in preview d'ailleurs, in his proper flux de travail optionnel. Donc on a les agences de renseignement les plus critiques de la planète qui balance des données sensibles dans une infrastructure dont l'URL peut être devinée par des ados sur Discord.

Quote

are similar version. Uh David Suzuki did a TV show on C B C uh does political stuff because the David Suzuki Foundation is a big nonprofit fighting for the environment and everything. Uh Actually uh Venc uh B C local. Um but yeah, similar thing. But He just yeah, he retired and just slowly did less and less and just kinda s stopped making public appearances. And that's how he did it. Devin Abber though seems to just keep doing it. He's like, I just love talking to a microphone about animals and fine uh flora and fauna and I'm just gonna keep doing it until I can't do it. is uncanny. Oh, it's insane. Now here's a here's a question of future morals and ethics. So here's a guy whose voice we I know where this is going. Well you guys are thinking it too. I'm just the one that's gonna say it. Well and we can tie this go ahead and I know where this is going and I have a Star Wars. Okay, cool. Yeah, yeah, there I I know where you're going with that. So we have out Hours and hours of just like lossless, amazing quality recordings of the guy's voice. And we can deploy those into the future at better and better usages to the point. point where right now you can still tell, but ten years from now it's gonna be indiscernible, I assume. From his his actual person.'Cause you've never seen his I mean. But you don't have to see his face, you just hear his voice. I can hear his voice in my hea…

Quote

But the validation can only work in some use cases, right? Mm-hmm. You can't validate everything. True. And that's probably one of the hardest pieces of Yeah. Yeah. That I think that's absolutely right. And and it's it's a good nuance that you bring up because uh Which is essentially their answer to this question of long running autonomous agents. We've seen some blog posts from Cursor, we've seen some blog posts from Anthropic, I think probably even one from Devin as well. Uh and so you know, they're there there's a lot of experimentation happening in how long can we have these agents run, can I launch them for an hour, multiple hours, multiple days, multiple weeks, and what is possible If you remove the constraint of time. Um what I think was just interesting. was as we were talking about missions, he he was acknowledging that there is something where there reaching almost like a the frontier of what is verifiable or what is like easily validated And and and you know w and we're trying to do that. like interact with some to get API keys from somewhere, you know, like then yeah you probably have to be having I mean that's like tricky, right? Because you're talking about how how

Quote

market tweet that it's an ad. And I was like, it's not an ad. I don't sell my tweets. And he's like, okay, fair enough. And I'm like, that was a weird interaction. Like, oh, I get it people are ratting people out. Like I might be a shareholder in a company, but that doesn't mean I'm getting paid to tweet about the company. So if I tweet I love Uber That's not a paid ad, but I am a shareholder. This gets murky because I get a lot of credits. So like and I disclosed it at the beginning of the show, didn't I? I have a lot of open AI API credits. Fine. Um I have a lot of Devin credits now. Fine. Like I actually think you you probably do need to disclose at a deeper level now, you know, because I'm not being paid by anybody. But I do get crazy. Yeah, I mean if so. Yeah, I think it's like there's like multiple layers. There's I'm explicitly being paid to do this. Clearly the FTC requires you to disclose Then there's I have an affiliation or I'm a shareholder in or I benefit some way if this company wins. That is like an optional thing, I think, or you could put it on your landing play page of disclosures like I'm an investor in six hundred companies. If you look through this list. you're gonna find a gazillion cross-referenced conflicts, right? I'm an LP in 21 venture firms. Like, okay, like I don't know. what they've invested in. So you start getting to conflicts for somebody like…

Quote

What are the all the layers up to the actual agent itself that the Human interacts with, right? So there you have like frameworks and knowledge engines, which are basically databases. is specifically for agents. Frameworks are these frameworks that help you build out the agents as a human. And then on top of that, we have the rest of the layers. And as you get up, you get up. Get all the way to the actual agents that consume it. So the agents that consume it are like open hands and Devin and you know even Cursor to an extent because it actually uses all the layers underneath to be able to serve the human what they want. Yeah. Some of these layers, like the engineering layer, I don't think I recognize a single thing. I don't think I recommend source graph. Okay, source graph. There's a bit there. I mean Langgraph is there, Langfuse is there, g uh source graph um you've Like the CNCF in that sense, let's this a GitHub repository anyone can contribute. Um, so if anyone finds any So the idea is to have a living sort of visualization of these layers of what is needed. to actually build an agent from the hardware layer all the way to the agent that's consuming it. Before the human interacts with the agent, right? Yeah. Like the OCI layers of agent.

Quote

Shared in a blog post how they use textbook rag augmented with more complex retrieval logic, crawling imports and directory structures and taking user intent like like past files you've opened as context. Being able to use this granular detail and retrieval can be a significant moat for startups. Open question two. two, how do we get AI agents to work better for end-to-end coding tasks? Well we still have a way to go to fully functioning AI engineers and A handful of companies and projects like Cognition, Factory, CodeGen, Suie Agent, Oben Devin, Autocode Rover, and Trunk are making meaningful progress. SWEBENC evaluations have revealed that most base models can only fix up to 4% of issues. SWE agent can achieve 12% Cognition reportedly 14%, and OpenDevon up to 21%. An interesting idea, reiterated by Andre Karpathi, is is around the concept of flow engineering, which goes past single prompt or chain of thought prompt and focuses on iterative generation and testing of code. It's true that prompting can be a great way to increase performance without needing to train a model, although it's unclear to me how much of a moat that can be for a company in the long run. Note that there are some limitations to the Sui agent form of measurement. For context, Suibench consists of GitHub pairings of issues and pull requests.