NVIDIA podcast mentions

Quote

MD had a growing up um and a journey to get to where we are in terms of like our suffer, uh but I think we are there And and um you know um it's great to see That software is just tokens now. Absolutely. I I think we are Very close to a feature of heterogeneous compute where instead of focusing on the computer. being, I mean, locked into NVIDIA's tech stack, as many folks are. uh you are enabled to leverage compute across devices, obviously something You know, we're very excited to be partnering on the modular front with AMD around much of this. And uh though this this episode's not about that, so I'm not gonna dive into it too deep. Uh but I I We love modular by the road. Yes. We got a good time. Um uh Notion, I can't tease too much of that though. So I I think there's a Very exciting things ahead of us. Um and the fact that AMD has lean so hard into open source just provides this incredible acceleration opportunity. to, you know, have this speed as a moat as you've talked about and as you've been I think hammering home. for for years now. And the testing opportunities that provides. acceleration opportunities that provides the simple you know awareness for developers and

Quote

Thanks, Sam. Glad to be here. I'm a longtime listener. I've been listening since twenty nineteen. That's amazing. and it is so great to hear. I am excited to meet you And I'm really looking forward to digging into your experiences Let's uh let's dig right in, but start by talking a little bit. bit about your background. You were at NVIDIA before you started, let's see? Yeah, I was at NVIDIA. Yeah, yeah, yeah since twenty sixteen, january twenty sixteen and back then the day I joined uh Nvidia's stock was worth thirty two billion. That was Nvidia's market cap. Thirty two billion dollars. And I think anthropic's revenue. It was quite quite experience, you know, being at NVIDIA at that time and um NVIDIA. was structured. Uh I don't know if they still are, but it it it functioned very much. much like a startup for the entire time that I was there, right? From twenty sixteen to uh twenty twenty two and um I you know the when the attention is all you need paper dropped. I was right there. You know, I was inventing things for NVIDIA. Uh In the generative AI space, I was deep into Gyans or Genitive Adversarial Network.

Quote

of doing otherwise malware can get injected or s something that can happen and so that's a lot of what we've been thinking about is like, you know, how do we both enable this because it's clearly the future, but then Also, you know, what what are these enforcement points that we can start to like protect? All right, welcome to the Lane Space. We're back with our guest host, Vu. Welcome. Good to have you back. And our friends uh Netter and Kyle from NVIDIA. Yeah, thanks for having us. Yeah, thank you. Actually I don't even know your titles. Uh I know you're like Architect something of Dynamo. Yeah, I I'm one of the engineering leaders. And arch architects of Dynamo. And you're director of something developer tech. And we're we're kind of recording this ahead of Invaded GTC, which is Coming to town uh again, uh taking over town. Uh which uh Yeah, yeah, we're super excited for it. favorite memories for Nader, like you always do like marketing stunts. And like While you were brev, you like had this surfboard that you like went down to G T C with like And like Nat Nvidia apparently liked it so much that they bought you.

Quote

if you're in the space. Um, you know, we've had a lot of major like technological advancements like big step functions where it's like industrial revolution or you know when we have had like calculators and then we had computers and then like the internet happened and then mobile happened. And this one feels like it's going to create the biggest, you know Say we stay on this trajectory and we do it in a thoughtful way, which I think the people that are involved in building it whether it's you know the GPU side of NVIDIA, whether it's the cloud Whether it's more like infrastructurally or application layer, like there's a lot at stake here. Also from a government perspective, you know, governments want this to be successful too because economics. booms generate more taxes and good for the citizens and good for quality of life. So I think a lot of the the techno optimists are excited about the potential to just do a lot more with a lot less. You know, and there's gonna be a lot of there's gonna be a lot of shifting in terms of disruption around like what jobs look like and what tasks look like and the ability to And I have no idea. What that's gonna look like in five years.

Quote

Yeah, that's definitely something that I have The vague sense of as well that all the cloud providers are trying to push their specific I haven't seen any real noise about people actually using them and I think that they're largely just there. for being the substrate for those cloud providers dedicated services like AWS NVIDIA has invested so much in the software ecosystem. Like if you look at the stuff that I don't know exactly who's working on it, but I know Andy Terrell is. If you look at like Like the stuff that they're doing around Food of Python, like that's really exciting, right? To be able to program all li all layers the CUDA stack with directly in Python, like that's that's that's pretty amazing. And it's gonna take a lot of work for anyone. else to catch up to that kind of stuff, right? I think A and B's not gonna catch up completely, right? But I'm hope I'm optimistic that they'll get their stack to be good enough such that you know you can at least you know be successful with PyTorch TensorFlow Jack's work. on their chips. But I but I think yeah, I just I just need a level investment. Like I see NVIDIA investing most AMD's close second, I don't see the hyperscalers investing nearly as much in this kind of stuff. But maybe maybe I'm just not seeing it because I don't pay attention too much to the TPU inferentia, like that kind of market. The name of the project is…

Quote

C R That's Claude.ai slash T C R and check out Claud.com. Claude. Well uh at GTC um I think you You revealed that you're using the Nematron uh three nano. Uh NVIDIA's uh LM model. I think it's very it's a very fast small model. And is that is that the model that is being used to kind of do the supervised generation kind of iterative process of search where you search from your index and then after that whatever is found. then it's processed by the Numotron 3 nano and then a new set of queries are created. And then that that continues the query process. different models one is the model that writes at you beautifully that one is uh front tier model. I think at GTC we uh were using cloud science. on it. We often also show it with the GLM model. And so that one, you know, writes to the user. And then the small model. What it does is it says, okay, yeah, um here's some search results. results coming back. Is there anything interesting and additive here?

Quote

how to actually write the code in a style that your company wants. There is the second issue. which is how do they get the data context? Well that usually comes from Google Drive and Confluence and Figma. and other areas where you're like, okay, what is the thing we're trying to build? That's a different part of the graph. Context turns out to be really, really important. Um, a lot of these things use each other's models. So I think we're in the era of AI where everyone's talking about chips. NVIDIA is doing pretty well. And then it's talking about models. And we're at the application layer. So we're like way above all these layers trying to div deliver customer value, but Our AI gateway has I've lost count now, it's north of seventy-five models running in it. in production and rovo dev to do a general task will probably use two or three models on most like most turns of the conversation. from different people, a lot of times it's using clawed code as a model, for example. It's using Gemini because our job is actually to pick the best model for the first time. for the best task based on the data rather than just kind of giving you a a task. Um, I'll tell you another thing that's maybe interesting that I think is underappreciated, and again. I run an organization with a massive amount of technology. A massive amount of what would be deemed legacy technology.

Quote

Where you look for patterns that the user is trying to inject or use against your app. And try to do some cleanup like the static cleanup. And then you also have And you can say hey d depending on both at the system prom level which is hit or miss and then with actual gut real got rails like uh an email from Nvidia right you can specify in their own language, but you can specify hey If these are the patterns of behavior I want to see from the user, anything that deviates from that another block report, yada yada yada. So there's a couple of layers that that complicates the equation a little bit more, right? So now you have to make sure that Your model is safe enough that users can put data and it's not gonna leak any private information or sensitive data out of the training. And you have to look for injections and the static data that's been So it's just to me it's just a wild new world of the first time. possibilities in terms of what you can attack. Because now your attack surface, maybe you can y leverage Some of the guardrails, some of the cleanups that are happening statically before they hit the model in So you talk about jail

Quote

Like yeah, even if people use Bing, they still say Google it, right? And And I I think that Chat G BT has come into the name. So I think that is worth a lot of money. And then like if we can compare it, the uh potential of where openai is going uh to some of these other bigger companies like I I think the the valuation isn't necessarily completely off. So for reference, right, NVIDIA and Microsoft, I believe, just both hit four. trillion dollars. Uh Apple is like a three trillion dollar company. Amazon is I think Like two trillion, something like that. So um a lot of these companies are in the multi trillion millions of dollars. So being worth only, you know, like eight eight percent of NVIDIA for being the predominant LLM whereas Nvidia with all the AI hype got up to a four trillion dollar market cap. Um openai, which is actually like the company that's like really Let's look at building the hype, right? I mean like NVIDIA is kind of selling the shovels metaphorically. for the the AI companies, right? Nvidia is designing the hardware. uh which um they don't even build right like nvidia only designs the hardware which T SMC makes and then uh once they have that hardware then

Quote

Trading starts uh the next day after that. That in December OpenAI actually loaned Sarah Brass a billion dollars. They secured it by So that's basically a multi billion dollar stake that comes in. So opening I just became on paper one of the largest single share shareholders of this publicly traded AI chip company that is going to compete head to head with in the U.S. Nvidia, right? Pretty interesting because NVIDIA is one of opening eye's probably biggest expenses. And now they're funding one of their you know biggest rivals. So I think there is a lot of of strategy at play here. OpenAI has been signaling for a couple years that they want to diversify off of NVIDIA. I mean this just makes a lot of economic sense and business sense having one supplier that you're really relying on. on especially because open air NVIDIA has a lot of export restrictions or yeah or not even just export restrictions but like when they come up with a new chip they gotta allocate it. Everyone in the industry wants to buy basically Every chip NVIDIA has, and NVIDIA has to decide who to give it to. So you get like chip allocated. And I just think that doesn't feel good for a company like OpenAI that wants to scale super aggressively. They would love more suppliers. They don't want to feel like, you know, at some point NVIDIA is gonna lower their allocation for one reason or another, it also helps thei…

Quote

yet about labour market disruption is I still believe deeply that humans are the source of deep understanding and creativity in the world. If I didn't believe that Uh I would be very worried. So I think it's gonna be an amplifier of sort of these these latent Transformers architecture or something massive is discovered by AI and we're all NVIDIA GTC starts Monday in San Jose and it's free. to attend virtually online. There's already been a leak this week of something called Nemo. Claw which is an open source agent platform. the bigger announcements this year. So it's definitely worth watching Jensen's keynote for that alone. I'm giving away a DGX Spock. NVIDIA just hiked the price seven. You probably heard about these memory shortages, right? So yeah, it's now forty seven hundred dollars. which is very, very expensive. And uh Merv from Hugging Face, by the way. She got one for her birthday and she said she literally cried. So it's a really cool bit of kit. Um if you register through my link in the description and do you attend at least one session? then you are in the draw. This is a massive conference, physical, AI and robotics. are going to be the breakout theme. And Jensen does the keynote Monday at eleven AM Pacific.

Quote

ha you know brings the larger the larger good into into picture. And are there anything I know you just named selling maybe to the U.S. potentially hostile uh uh countries and such, but I'm I'm just curious how you see that Well I think it it depends on if we're talking about the relationship between the government and AI companies or the government and chip companies. The chip companies, companies like NVIDIA and the like, our posture was We thought their technology was so important, so fundamental, um, and also so scarce. that we did not want it going to countries like China because of the ways in which it would modernize Chinese military and the like. And because China really struggles in part powerful technology. Uh China will not make a chip as powerful as the one Trump has agreed to sell. Chinese propaganda, they won't get there till twenty twenty eight. So they're they're just as an exp As I said before, an extraordinary advantage. And I would we we took put a policy that I would defend. bend of export controls to say we we want these this technology, especially given how scarce it is. Every chip that gets made will get sold. We want this ch technology to go to um