Artificial Intelligence Livestream - AI, Games, and DevOps – Build Smarter, Ship Faster

Build Smarter: Welcome back everybody. You are watching the Artificial Intelligence Live

Jon Radoff: Stream. This is the stream where we talk about all kinds of artificial intelligence subjects, everything from LMS to vibe coding to generative art, everything in between how you can apply AI. But so many of you are game developers. We always talk about games and game dev and that is going to be four-shore one of the topics that we cover today. So we're going to go in deep but we're going to start high before we jump into what my friends from Devstream Labs do. Just want to let you know that I am Jon Radoff. I am one of the founders in the CEO of a company called Beamable. We build infrastructure for game developers. So if you need game servers, online economies, social systems, chat, real-time multiplayer. Like we do all of that. We are a tech stack to support all of the back-end needs of your game. So you can think front-end, unity, unreal engine and HTML5 JavaScript games. We have SDKs for all of the above. That sits on top of our back-end. That's all you need. You start just building your game and building a great one. So that's what we do. I love to talk to people from AI, from game development, game developers in the trenches, people in decentralized technology. But this is the AI topic. So I'm going to introduce my friends from Devstream Labs now. So before we get into the problem you guys solve, can we start with just your own personal story. What led you to this point in time and space on this stream today? Ready to talk about what we're going to cover? Do you want to

Build Smarter: who wants to start? We got two of you. I can go first.

Build Smarter: It's super exciting to be here. I really appreciate you guys inviting us to join and participate. So my name is Brent. I'm the CEO of Devstream Labs. My background is pretty extensive in the games and entertainment industry. I actually started my career out at Apple working on early 3G hardware technology. I've worked at Bungie. I've worked at Abbott on Pro Tools and the Abbott video workstation. I've worked at DreamWorks Animation. I started a company with Alex Ropian and Tim Harris called Industrial Toys. We started at 2012 and we got acquired by Electronic Arts in 2018. Then after that I started

Build Smarter: another company with Stephen McDonald called Magic Pushing Games. Then from

Build Smarter: there I saw the business need for what we're doing at Devstream Labs. So I started that. That's my background. Awesome. And Bruce. Yes. Super excited to be here. Very grateful for the opportunity, John. I think what Bima both does is fabulous. I think we have a synergistic point of view of the game depth space. My background is I spent almost a decade and a

Build Smarter: half doing automation and optimization for hardware software systems at Shell.

Build Smarter: And then saw the the emergence of machine learning and artificial intelligence as the natural evolution of control systems and optimization of automation. So went back to school, got a master's degree in data science and was lucky to get onto Brent's team as an intern starting out at Industrial Toys. Electronic

Build Smarter: Arts we were built in Battle Toys Mobile. It was such a great time. It was an

Build Smarter: amazing team, amazing product. And from there after Industrial Toys was sold I went on to have built FC and FIFA Mobile. It was also amazing and now I'm

Build Smarter: super excited to be here running AI systems for Devstream Labs. Awesome.

Build Smarter: Also, I also have the privilege of teaching applied AI to executives at UC Berkeley. So if anybody has any questions about other industries in that space

Build Smarter: happy to talk about that as well. Awesome. So you guys have tremendous

Jon Radoff: experience, lots of credibility. It's what we love to cover on this program. Like people actually building real stuff because there's just so much bullshit out there frankly. So love to talk to other people, helping other people build real stuff. So let's talk about that. So let's take it through the problem that you solve and start there and then we'll kind of dive into like the AI aspects.

Build Smarter: Yeah, sure. Thanks John. Yeah. So I'll start out pretty stupid. You can chime in whenever you feel like it. Yes, I'm going to add. But really I started this company based off of a problem that I kind of hit over and over again in the development space, which is how you approach building your product. Like a battle field mobile at EA, we have over 30 engineers. And throughout

Build Smarter: the day they would just be doing multiple check-ins and there would be like one

Build Smarter: night in the build. And then most of the time, maybe 50% of the time that night in the build didn't work. And then it would take away from our lead engineers time because they would have to go spend time figuring out why they did the night in the build. Not compile, which the check-ins didn't cause the broken build. And then even once we got a good build, like did it work. Like, could you actually do a play test on it? And that also was like very challenging. So we found that we only hit our play test goals for a fraction of what our goals were. And so I started kind of seeing this pattern just throughout being in my career at so many different places. I thought, how people build their product? That's not something that's like really differentiating. No one plays Call of Duty. No one plays Battlefield Mobile because they have a great infrastructure, a great build system, right? That's totally not what game developers want to do. No one becomes a game developer because they want to work on build systems. It's something that I think people kind of do, you know, reluctantly or maybe if you're lucky, you have

Build Smarter: someone who is actually interested in it, that's kind of rare. And so I really

Build Smarter: saw the broken build as a productivity killer, as a morale killer, as a big pain point. And when you have a broken build, you know, people sitting around that can't check in because the build's broken, that is costing the game developers and publishers, you know, millions of dollars and you have a team that can't be productive. So I really wanted to turn that around. You know, I wanted to essentially take the problem of having to do builds away from you know, developers that should want to make games and we just provide that service. So we can work with an existing source control system or we host our own source control system to GitHub or our live again or per course, we also

Build Smarter: can connect to GitHub. And then it's super easy to set up. You just create an

Build Smarter: organization on our website and you can create a project, you know, an

Build Smarter: organization to have multiple projects. And then from there you can create if

Build Smarter: you want to have us host your source control, it's like one click. Like one click you can create per-for-series. There's no need that it needs to be anything more complicated than that. Same thing with GitHub or just, you know, connect to GitHub. And then another click and you've got a build farm. And then we set it up so that anytime someone does a check-in, we do a build. So you get one build per check-in. And so you know immediately if something's broken the build so that you can fix it right away. Then also we have technology that looks at what changed and creates a series of test steps that can be performed specific to that check-in for that build. So you can build and you can test every check-in. So you know that your source code is always stable and that is a priority of the team to make sure that the source code is always stable. And when that happens a lot of good things happen. First, developers have a lot more confidence that the current set of sources are stable. Sometimes that's a big incentive that developers don't want to update their code because they're worried about it being unstable. The second thing is you can obviously verify the code is always working because you're doing test steps. You can create, you have a bug bug. So you know if one bug was introduced for which which is definitely going to be. And then when you think of a polishing and finishing stages, you now have a fairly accurate backlog of bugs that you know you need to go through. And if you if you need to go back in time, fix a bug and maybe was introduced five months ago because of one of the prior event, but it's a priority for shipping because you know exactly what check-in caused a bug. You can go look at those sources and that makes it 10 times easier to fix the bug. So you now have a much more predictable ship time because you have much more predictable you know polishing and bug fixing stage which is traditionally the most unpredictable part of any project. So that's just a little bit about us and what we do, we have like a whole AI component for helping people diagnose broken builds and I want to let Bruce talk about that

Jon Radoff: because he created that system. We're here on the AI livestream. So but before we just before we get to the AI component just to get maybe net it out a little bit. So the typical way teams of that size work together is everyone's checking in their piece. They might be doing local builds to test it on their own computer. I mean we used to not be very happy back at Disruptor Beam, the Studio I ran if people like checked in something that didn't even build on their local machine. Everyone would get annoyed by that. But let's say that you don't have too big of a problem with that. Like people are doing a local build at least built, they check in their change. You're kind of aggregating across the course of the day all those check-ins that lots of developers are putting in and I think for most of what we were doing we were doing continuous integration but we would end up with kind of the nightly build because to do a real full build we've had to also bring in a lot of assets and all kinds of stuff was going on. So we did the nightly build and the problem there is because there's a lot of different contributions happening at the same time. You can still break the build because people are in conflict with each other for example there might be a merger sheer that didn't merge clean.

Build Smarter: Exactly. In addition to that when a developer is not confident that the current set of sources are stable they're reluctant to update. So someone may make a change on set of sources from like three days ago and it works in compiles on their machine but then they go checking it in thinking oh it's going to be fine and the source control system will let them merge their changes in and they may see that it was a successful checking book because the three days behind from the latest update then breaks the build in an unexpected way because they're afraid that if they update they're going to get a bad set of sources and then they won't be able to check their stuff in and then they're going to be blocked like they're not being able to do anything further from that. So there is kind of a disincentive for people to do the right procedure when they don't have confidence that the current set of active sources is

Jon Radoff: stable in working. Right because most teams these days don't do what we used to do in the old days like exclusively check out a file and you're the only one working on it like you work on the file and then we merge. Exactly. Yeah. So then I'm checking in every single check in that comes in a build is kicked out I know if

Build Smarter: it's going to be broken right away I can test it. Yep exactly. Cool. Let me ask

Build Smarter: your questions since you have so much experience in this space because you're resonating with a problem we're trying to solve exactly. When that build broke at this ruptured who was it who had to go figure that out was it was it your most senior guy was it your most junior person you know how does that dynamic

Jon Radoff: work out in a studio. Well the so it changed over time but what it almost always could involve like one of the most senior engineers it because like that's the person with the most perspective on all these pieces are fitting together. What

Build Smarter: we tried to do is have a fairly senior person who supervised it overall and

Jon Radoff: kind of established what the build process and the integration process was but then we would have we didn't really I don't even want to call them junior engineer we didn't really have any junior engineers at disruptor being like everyone was fairly veteran but a less lead engineer not quite a lead engineer would would kind of be more the day-to-day person they would be the first line of defense and often they could resolve problems but it could get come back to a to like lead engineer for the whole project if things were really breaking down could but it could come land back at the VP of engineering right ahead of Dave Cham our VP of engineering for a number of years super engineer can be hands-on in high level like he would get pulled into stuff if things were breaking so like it's whatever happened like you got you're right though like if the build is broken people can still build on their machine but like a lot of things grind to a halt if you're not able to get code running

Build Smarter: with everybody's recent changes yeah that's what we're seeing we're trying to

Build Smarter: figure out what where we fit in these studios and we've heard this again and again it's but you're most one of your most senior guys and you know instead of working on that build issue they could be building a new feature that actually helps the game you know get to market or something that the players love and enjoy right so that's also a problem space that we're trying to solve okay so

Jon Radoff: so how do we do this magic of the every check-in creates a build and then how

Build Smarter: are you using AI to help diagnose the issues yeah so I'll talk about the

Build Smarter: every check-in aspect of it so we actually we have a large build farm and you know we hook up to the source control so that even if the build takes a long time to build which is pretty you know the issue as long as a build comes out

Build Smarter: even if it maybe takes an hour or two as long as the build comes out per build

Build Smarter: you can you can you still know what what check-in caused the problem so in terms of having long build times you know we just handled that by having a fairly large

Jon Radoff: farm and you you're offering the essentially a cloud build service yeah we

Build Smarter: actually I mean we do some there's some similarities between us and what you guys provide we have our own private cloud you know we we have that that knowledge and ability to create and manage that so we have a data center here in Austin to tear three data center and we have one of the processes getting

Build Smarter: our sought to type to security certification so you know security is

Build Smarter: hugely important to us as it needs to be if we're handling IP for our customers but in that data center is where we would do our our builds for a lot of our customers our system can also work on premise so for a larger enterprise customers we could actually just take our system you know install it on whatever servers the enterprise customer like us to be on and we're off to the

Build Smarter: races so for the AI component which I think is what is really amazing I'll hand

Build Smarter: that over to Bruce talk about yeah so AI lives and dies based on data quality you know the question that I that we've been asking at UC Berkeley to our executives is your data AI ready right it's not just is yours is your company AI ready so what we found is that especially in the build process when you get your logs they can be massive super overwhelming files right some of these logs for these companies some teams that we're partnering with for our beta are like over a hundred megabytes 300 thousand lines of log files long it takes a senior person an expert person to even dig through that figure what's going on so how do we democratize that and so we we started where probably everyone starts like how how far can we get with chat GPT or Gemini or quad can we just

Build Smarter: put the file in there and start prompting it and say hey what's going on here

Build Smarter: and what we find there based on how the systems are architected is the probabilistic functions rather not deterministic so they are looking for predicting the answer that you want to hear and when you have that much

Build Smarter: context they experience this it's almost a form of hallucination called

Build Smarter: context rot right where it does actually know which piece of context is the right piece to to predict to give you your answer and the way that we overcame that is actually going back to fundamentals first principles a lot of the work that I did at Shell I was actually one of some of my favorite work that I did was actually a root cause analysis root cause investigation so bringing back the fundamentals of that when you get your data in what's the first thing that you do but you find the data points in chronological order that you know so for example something that happened after the top domain events the breakage something that happened after that couldn't be a cause you put in these heuristic these rules into the system then you just start cleaning the data you start doing your standard no data removal data clustering trying to figure what the patterns are into that data before you start deploying these amazing technologies that we have like that are called large language models that'll

Build Smarter: help you actually make sense of all that data that you've actually

Build Smarter: architected and put together into it into a nice package that it can then use to say hey this is the root cause of your problem and here are secondary causes and so both fix these things in this way and you should back back off to

Jon Radoff: the races so I want to introduce a whole other layer of complexity because I'm curious how you think about it so we've been talking about build process so far but I'm assuming that we're talking about a singular piece of code where I make changes to it like the game client code essentially there's this other aspect of what happens with servers and I want to just maybe talk a little bit about what we found it beable because this is a big part of our solution that that we found became important for developers so in creating like a most modern games now like 95% of the revenue in this industry comes from games that are online in one way or another whether you call that games as a service or not it's games that require content updates or have some kind of even just an asynchronous online component or it could be full-blown call of duty battlefield real-time multiplayer game experiences but virtually all games have an online component so that means you now have a mixed code base which is client code for the game experience layer graphics etc and then you've got server components and the server components have to work harmoniously with all of the builds so for example just individual developers workstation for example you have to run your build not only with code that works in terms of like your local code repository you have to make sure that all of your connections out to various servers and services that are part of your game are also synchronized and you're using all the latest stuff there so what we discovered was really important was developers want to essentially instantiate the entire server environment on their local machine so that they could test it locally there's other needs to also test against staging servers or even production servers and QA does different things but just for that individual developer we found that if you make them connect to like the shared dev server like that doesn't work that great because they have to make changes to dev server components at the same time so everything gets out of six so they have to be able to work on things holistically on their machine so part of the solution we found is first of all get people on one language structure so if you're programming in Unity for example just you see sharp top to bottom including your server components and then use Docker to containerize all the server components make sure you pull down local containers that are the authoritative versions of the server code that you're going to depend on and recreate that whole environment on your machine and then not only are you testing it locally you you're testing it locally you can also debug locally so you can set like a breakpoint on server code and initiate it from client code and trace it all the way up and down the stack and you can only really do that effectively if you're on one language with all the code actually runable on local okay so that's just the preamble that's kind of part of the magic of beamable and why some developers really like us is because we integrate the whole development process

Build Smarter: including debugging and stuff like that but now I'm thinking about what you do

Jon Radoff: and thinking about the build collectively because you still have to ship all these pieces off to the staging server where QA is gonna sign off on it and other developers might get the current version of the server components or it may also have like more complete like a common thing is like the more complete databases there like one thing we did back at Disruptor Beam is we'd

Build Smarter: actually back propagate the production user database back to staging so that you

Jon Radoff: could test code against because we didn't want to break production data on production databases we wanted to test the environment against a mirror essentially of what had been recently on production so like so much complexity when you start having databases and server components all of that so now I'll take a step back I've talked about kind of the piece that beamable thinks about how do you think about all of that interdependent complexity that happens with server code when you add that yeah 100% yeah and I would say that you know the

Build Smarter: most common client people are gonna be developing for certainly they're all over the map most common I think is Windows and a lot of the server components

Build Smarter: they'll only run on Linux machines or they're all run effectively on Linux

Build Smarter: machines and so now if you want to run everything locally you have to have you know a multi-machine kind of local environment which can get very complicated we handle that a couple ways first of all we can for every client built we

Build Smarter: can produce a server build and we can we can specify like basically what

Build Smarter: what components what versions of what components need to be part of that that server build the way we handle this at Magic Potion Games is we had you know we had a development environment obviously a staging environment and production environment and anytime we did a build on the DevSources that produce a client and we're produce a server build and then our system would automatically basically load that you know that that C-sharp executable that

Build Smarter: like that's a Unity produce you know loading to make it available to all our

Build Smarter: servers so that when our client launched we could guarantee that your client was gonna attach to a matching build that so that you didn't have any sink issues and so the Magic Potion Games initially we were using dots and so you know we used new dicks and dots it's it's basically required that you build a client then a server and have those those match and so we actually have like a

Build Smarter: meta server that the client would connect to and then that meta server would

Build Smarter: make sure that there was a server instance running that matched the client version that got built and so then any client of that version would connect to

Build Smarter: a server of that version of multiple people were running the same client the same

Build Smarter: version they would be essentially on the same server but different people can be running different different versions and they would get get their own server environment I think in practice it definitely is very very useful to

Build Smarter: be able to set break points and server code and see that happen and see the

Build Smarter: interaction and that that's definitely the preferred way of doing it that can

Build Smarter: get very problematic particularly when you know like you are basically having

Build Smarter: set break points on a Linux machine because your code there have a Linux machine also all these systems are real time systems and so when you you know setting break points in a real-time system that affects how real-time system operates so what we found is the most effective way to do bugging in that kind of

Build Smarter: environment is actually through logs and that can be you know that could be

Build Smarter: difficult to commit your eyes bleed when you have a super long log file but that is going to be you know the most effective way to to bug you know a real time system where you know you just can't put into a break point and so we actually you know this is something we you know we haven't announced yet but you know we are working on systems to basically make you know that that process you know much much more friendly I think actually analyzing log files is very similar to analyzing the results of builds you know like a build log file can be 100 megabytes long it details everything and you know our AXS system learns from that you can actually apply very similar technology to analyzing log files where it looks for something that it hasn't seen before it looks for a pattern of logs that you know it thinks you know hey this is not the usual pattern of behavior that I see so it can really be super amazingly useful for helping you you know find anomalous problems anomalous you know work paths that you know

Build Smarter: that lead bugs or lead unpredictable behavior so you know that's something

Build Smarter: that you know we're working on as well that that complete picture so that you know everything works from the client to the server and you know we make sure

Jon Radoff: it builds static I want to understand again the AI component a little bit more so Bruce you were explaining a moment ago context rot all all these things that can sort of go sideways I guess when the source state it sounds like context rot is a is a consequence of supplying a very large amount of inputs did I get

Build Smarter: that right so 100 percent throw the hundred million lines of log code at it

Jon Radoff: and it no law it it it probablyistically finds issues that are not the issue it's it's just pattern matching it doesn't have enough superverns to learning around that particular data set to actually or any kind of reinforcement learning I guess to to gear it towards the more specific solutions that would be appropriate in that situation you nailed it John absolutely

Build Smarter: I really really well said and part of that is because if it's probabilistic it's looking for the things that are happening most often and then oftentimes these

Build Smarter: these built breaking errors are like one in 10,000 lines right you're asking if you look for a needle in a haystack okay so take me through that a little bit

Jon Radoff: more though so how do I find the one out of 10,000 lines well yeah so I mean

Build Smarter: humans humans are good at this kind of thing or we've been trained ourselves on this kind of thing that's why we have investigators and detectives it's using that really methodical approach using our discipline and there's a lot of there's a lot of creativity to it too that I think that you know like the

Build Smarter: machines are ready to to do yet which is you know just taking things apart line by line chronologically using some rules very specifically is this particular

Build Smarter: potential cause both sufficient and necessary to result in the error that broke

Build Smarter: the entire build checking you know everything at least twice right we have you

Build Smarter: know a double verification system but checking every input every precursor to a cause to make sure that they're logically reasonably connected to cause the

Build Smarter: upstream issue or downstream issue however you want to say it.

Jon Radoff: Bruce one of the things I'm curious about so the you know context size and this context rock problem I get it the I guess the lesson from AI generally is like okay you run into this problem you scale up to a bigger LLM and problem goes away I'm curious if you've tracked between major LLM versions it sounds like you work with chat GPT for example or open AI's interface to to underlying LLM have you seen that problem not really resolve relative to say size of the

Build Smarter: model as they've scaled out is it a problem unrelated to scale. John I think

Build Smarter: that's a brilliant point of view and so we're actually one of one of the systems that I tried on early in prototyping days was Gemini which is one of the largest context windows of all the major providers and even in that you know even if you played with Gemini and you were having a continuous conversation after about 30 back and forth or so it starts forgetting you say you know you could say hey I told you know we we would specifically discuss that I don't want to talk about that and I'll bring it back up so if this is a

Build Smarter: symptom of that that context rot so yeah I don't think that this is a problem that can necessarily be solved by scale because you want any set of

Jon Radoff: results I'll give a I'll give a funny anecdote and maybe it's the same kind of thing so chat GPT in recent versions has gotten pretty good with its memory like and sort of giving results based on its memory of all your interactions and so I don't drink or do it extremely rarely and this isn't a bragging point at all it's just more of a chat GPT point so I've told it that and but without fail every time I tell it help me organize a trip with some real highlights in this area I'll get hey go to this brewery like or go this place is known for like it's wine selection I'll be like I told you this before that is not a thing that I'm looking for like revise and so I have to go through invariably a second process from like okay revise it with that out by the way remember this please it's important like I just don't want that so you know that's just kind of one example and I'm sure if you if anybody out there has used chat GPT you've seen like all these versions of the same thing where it forgets a key memory and just it doesn't stick even when flagged is important job that you nailed it

Build Smarter: you're absolutely nailed it and so I guess one way to get around that is through through training or fine tuning right so you could take you know your own model and essentially eliminate those nodes within it that so that it doesn't need to know that John doesn't drink it just won't talk about drinking at all so that's one way to get around that problem and that's called fine tuning yeah for us we are very much building a system that is a specialist and you know a root cause analysis specialist whereas I think the chat GPT's the Gemini's the Cods the Grocery will they are amazing generalists right they know a lot about

Build Smarter: everything we want this system to be to know a lot about builds how a log

Build Smarter: files how they break and how to fix them so that's another different way of

Jon Radoff: thinking about it yeah at the same time we really see language models doing so well at coding generally like I didn't expect that code assistance would be as I mean two years ago I knew code assistance would start becoming amazing but five years ago I what it wouldn't have been my guess that we now have vibe coding and like computers through AI models would be so good at like generating code what's the fundamental difference between generative code writing that we see LM's are getting better at and actually that's an assumption so maybe I'm begging the question a little bit happy to have you take any any varying view on that or maybe point to the limitations that are possible and at least current generative code what's so different about that versus pouring over a log

Build Smarter: file your example well I think it's got a lot of trading data you know I when I

Build Smarter: was when I was looking for trading data for log files you know looking at stack overflow and such I think and people can check my numbers here if I wrote my queries right was only able to find like 24,000 community log files on stack overflow that had both a question and an answer to it and if that's the

Build Smarter: data that was used for you know you know for whatever large language model then

Build Smarter: it doesn't just have it doesn't have a lot of data in that space but it has a lot of code examples from GitHub from stack overflow right that's one

Jon Radoff: possibility parallels in some ways what we're seeing in game development which is 2D art generation is really really strong because there's I don't know how

Build Smarter: many 2D images on the internet like there's close to infinity 2D images on

Jon Radoff: the the internet or something and then there's lots of video as well so huge amount of training data but with 3D modeling it's actually proven to be a harder problem there are people actually doing a really good job at it like there's a company out here in Cambridge common sense machines who's building a new foundation model just for 3D so like there's examples where people are

Build Smarter: actually doing pretty good at it and you can see that they'll eventually get

Jon Radoff: there but part of what has made the problem so challenging a 3D is just that lack of training data there aren't tons and tons and tons of 3D models present on the internet that you can just train everything up and then just prompt everything into existence sounds so that problem I get so people are just not posting their log files to the internet with their problem description and the code solution so I don't know if people start doing that and we start having I know probably millions of log files and I think you said 20 24,000 24,000 24,000 still a lot but not enough right we probably need like 24 million or something right and then we can start just LLMing away so again but it does speak to a kind of a scale problem sometimes it's the scale of the model sometimes it's the scale of how much training data is available so you have to build a specialized approach to this market based on expertise I guess and the ability to take take I don't know to the extent that you can talk about it because maybe I'm starting to get into secret sauce territory for you guys but I'm just curious like how do you go about that are you effectively building a

Build Smarter: new foundation model that's specific to this problem or is it like what what are

Build Smarter: we building here yeah so I mean we definitely entertain small language models you know fine tuning models so that they're very they're very

Build Smarter: good at a certain task but you know actually so this is not an industry

Build Smarter: secret or anything even when I was back at Shell that's like 20 years ago now we we had lots of automation lots of optimization but we found that when we first went let's try to optimize how this reactor runs the first step that we take is go talk to our most senior operators the guys who've been running this thing for 10 15 years they could do it in their sleep as we say hey so what

Build Smarter: would you do in this situation and then start you know having our systems

Build Smarter: learn those rules right and so then we're operating these things based on

Build Smarter: trying to mimic what humans do and a lot of the machine learning models that

Build Smarter: we've developed like neural networks we're trying to mimic artificial neurons which are trying to do the things that our natural neurons do right you know any any sort of regression models these are all just mimicking how humans perceive

Build Smarter: you know and project extrapolations right so yeah I think that that's the

Build Smarter: secret sauce it's we just took a very systematic step by step approach like you said an expert methodology to say hey how how would a human approach this and then how would how would an expert senior engineer approach this a dev engineer yeah we got some we got some pretty so pretty good feedback from some of our senior engineers saying you know yeah this is cute but you know this is not so this is not the right not the right root cause and so how do you make your system a little more oh by the way John to your point earlier a lot of our LLM

Build Smarter: systems are programs to want to please right so don't don't find like the first answer and they go hey thumbs up then here you go so we had to you know we had to

Build Smarter: um to our systems to say hey we don't want the first answer you want the best

Build Smarter: answer right interesting so you are working with game developers who are going

Jon Radoff: to use this technology to solve the problem that we talked about at the outset and if you're tuning in late it's people break builds and so the solution is basically kick out a build every single check-in of your code and couple that with some I'll call them diagnostic tools I don't know if that's how you describe it but tools that give you insight into why that problem happened so that you can then fix the build more quickly get your build process on track it into QA ship you know ship faster increased development philosophies you're

Build Smarter: not paused by these problems so it's you so I'm guessing that in talking to a

Jon Radoff: lot of this market you get exposed to how a lot of game developers are thinking about things like AI and where AI is applied within their business and you have a specific solution that is using AI to solve a specific problem I think often in the industry there are conversations that are kind of speculative aspirational hey someday we're gonna vibe code call of duty press one button call of duty is now downloaded to my computer I can start selling it I actually do think that's possible someday because there's no law physics as we can but might require a lot of energy might be more expensive than people expect to do that but I think we eventually get there but in the here and now like the reality that we exist in this moment where do you see people really applying AI in game development like where are the actual wins right now versus and

Build Smarter: it's not really something that what is anything beyond like sounds cool looks cool but can't be used in production yeah you know maybe I'll take the first

Build Smarter: crack at that yeah so I mean I actually do a lot of engineering work my my background is intact in software engineering and I think you know the current state of the art it's relatively limited I mean I I'll just like digress for a minute about how I view AI and how I approach using AI like I've actually been around long enough to remember when like using calculators in the classroom was controversial that's how old I am calculators you know all of a sudden became affordable to use in a classroom and it was a huge controversy everyone all the teachers had you know other arms you know all twisted like oh my gosh how can these students all use calculators in the classroom and obviously now that's like you know a no-brainer so I would say that was like one like kind of generation like revolutionary tool that that kind of came about obviously calculators super simple but it was pretty revolutionary at the time and then personal computers came out and then everyone was typing the term papers on personal computers and editing the term papers and printing them out on that matrix pictures and that was usually controversial you know like how can

Build Smarter: you not be using a typewriter how are you know how can you be using you know this

Build Smarter: personal computer like you know and that was kind of like a second major you know revolution in terms of like how how it works yeah I remember when like

Jon Radoff: grammar checking and grammar checking was terrible yes exactly I remember I was actually really 90's grammar checking was actually controversial like OGs now you get a free pass on grammar and your and your English paper because you could just grammar check it in your word

Build Smarter: process yeah exactly I think AI today is just another iteration obviously much more sophisticated much more complicated and maybe you know the a much bigger impact than those but I think at the end of the day my point

Build Smarter: really is like AI is just a tool you know there's there's no AI system out there

Build Smarter: like acting like a human or thinking like a human or

Build Smarter: initiating things that need to be done I don't know ask you're greasantly but that's fine you know like I think what people

Build Smarter: lose sight of is you know they hear things like oh Microsoft laid out 30% of their team because of AI and actually like like there's not like there's like you know AI directly replacing what the 30% that got laid out for doing you know like they're not taking an issue that AI's not you know like deciding hey you know what here's a great new game idea that I think we should all be building you know but what what AI is doing is it's making engineers in my opinion is making engineers 30% more productive and it's doing that through like what you mentioned earlier with like biocoding I mean I found AI extremely helpful to ask it for very narrow things like hey I need a function that does this like I'm unfamiliar with these win32 API calls how do I use this particular win32 API call to get done what I want to do and then it will it will give me you know a function or give me you know that you know that answer and it doesn't amazingly good job of doing that but I wouldn't say it's turnkey either like I would say like usually the AI will make some mistake or there'll be some context this like it gets wrong but it gave me it gave me a starting point and now I can make that do what I really wanted to do and it takes me maybe five minutes instead of you know taking you know potentially multiple hours and so it's just maybe more productive it hasn't like you know it has a replaced you know the you know the inspiration of the creativity part of what humans do so I think you know when you look at how AI has applied to gaming I think people that find a way to embrace how to use AI to make them more productive though those skills are going to be more more valuable to employers if and I think it's human nature to be scared of something that they see that they think could replace them and I think it's a traditional like if you if you're scared something and you shun it you know you you actually you're not moving for you know you're not you're not moving ahead with your skill set improving your craft whereas I think if you uh looks like maybe we lost John but I'm sure he'll be back he'll be back in a second but let's continue maybe maybe maybe

Build Smarter: I what I'm saying is making so angry he had to rage quit um but I think you know my high

Build Smarter: level piece of advice is I think people that figure out how to use AI in in their jobs to make them more productive um that's just going to be you know I think how how the future is and so I yeah you know it's it's totally human nature normal to have a knee jerk reaction and be afraid of something that you feel is as a threat but if you can figure out how to flip it around and actually make that work for you you you now have new skills and those new skills are going to be you know hugely valuable in that any game tell the two yeah I find it um important to acknowledge that people's fears are based in an existential few like people are scared that their function their role is going to be replaced and there's a lot of people who are just absolutely rejecting the technology or the workflow or the tooling you know on its space just because of that

Build Smarter: philosophical um contradiction and it this is not an isolated incident you could look at the same

Build Smarter: thing that happened when um not that outsourcing is like a brand new thing but like we've we saw like it an exploitation of work to places where things can be done cheaper and you know people give very protectionist about where they are what's going on with them what they built their entire identity around and like their purpose for lack of better word and the scary thing I think with AI

Build Smarter: is that I can make a very they can say in the next

Build Smarter: thousand five hundred days what we've been seeing with vibe coding is going to scale up to a point where maybe it'll be really tight integration between unity as an ecosystem and you know built-in agents I think they actually just rolled in um built-in agents of some sort in unity to answer questions is to provide you know like you know so I remember specifically between the time that unity was introduced and unity became the most hated thing on steam because of all the shovelware that everyone was like really you know ticked off they're like oh you're not real coders you're not real developers you guys are a bunch of you just asset flip this and bub bub bub bub bub bub bub bub and if you did not pay to get rid of unity from your um you know from uh you're long opening

Build Smarter: you're you're you're your flash screen yeah right then that was a problem that could actually affect your success um on steam for example so right now we are going to see a big flood of games coming into the marketplace as a result of AI facilitating people not having as huge a lift um and that's

Build Smarter: going to be the exact same thing with like maybe a fivefold maybe a tenfold increase of like

Build Smarter: titles on steam over the next couple of years the difference will be eventually is a technology

Build Smarter: going to reach the point of fidelity where I can say in the next five years I want to make a call of duty and like iteratively it like you get close enough to be like oh that's not call of duty but it's not you know the worst piece of shit that ever happened and good enough is a hell of a litmus test for entertainment you know I don't know how many kittens singing like that Billy Eilish song with like AI you know domestic squabbles but that became very popular and that is entertaining and engaging content so I can see that same level of automated quick turn around high to the new cycle hey I've got a game it's out there in a week I sold 10,000 that's all I needed to you know to cover my nut but that's a whole separate conversation from I don't think that we're going to see a massive like die off of like game developers you know in the industry but I think that there is going to be a situation where there is a conflict where people who are not rejecting the tooling people who are using AI in their workflow are going to receive preferential treatment when it comes to hiring when it comes to production when it comes to leadership and there's going to be people on the other side of that equation like the the indies people who they want to make their point and click adventure they want to make their visual novel they don't have to wait for art they don't have to budget for you know they don't have to do like crappy janky pixel not that I'm knocking crappy janky pixel art um vibe coding in some ways it's the future and the counter argument is you guys are stupid I've been to develop for 20 years you guys don't have a debug everything sucks you're horrible human beings but but but but but but but but I mean I spent too much time I mean I think that gets back to the human nature you know reaction to being threatened and you know I think I think it's totally understandable and I have a lot of you know I mean you know a lot of sympathy for what what they're going through because it is a bit

Build Smarter: of a struggle but I think at the end of the day you know what what matters is is what what people

Build Smarter: are going to want to consume you know like as a game user like most game users probably don't really care how it was made it like all they know is there here's a piece of content whether it's a game whether it's a movie or a short clip you know whatever it is like you know tick-tock video like they're going to care hey good you know do does this affect me does this resonate with me does this enrich my life in some way that I enjoy and then they're not going to care so much about how that kind you know created and I think what developers I think the trap they get into

Build Smarter: is oh people care about this because I made it or people care about this because I humiliated and

Build Smarter: I think I think that's not you know that that's not at the end of the day going to be a mindset that helps people you know stay deployed doing the things I love to do I think the mindset that keeps people employed is how can I make myself the most relevant how can I make myself the most useful to produce to create the content that it's going to connect with people and I think where when people embrace the state of the art technology that makes them more productive that allows them to do more things you know they're the ones that are at the end of the day it can be the

Build Smarter: one's most viable because they'll be able to produce the content that people want

Build Smarter: yeah I think you know I think also you know what's happening with AIG tool and AIG that personally I think it's super cool is it's it's enabling a lot of creative people to do things that they couldn't do before you know they like a creative person may not have had the skills to

Build Smarter: you know write all the complicated engineering code that is required to make the content that they

Build Smarter: want to create but AI can help them do that then you know in many ways that's that's a pretty cool thing you know you've now empowered someone to do something to affect other people in a way that they couldn't before and I think what what people is doing what we're trying to do I think it's is really you know a fantastic service to people so they can just focus on doing what they love and they can let the the commoditized problems be solved in in a general way and then that can be shared amongst you know all development teams out there because the end of the day that's just not differentiating you know the creativity of their product and then you know the mechanics of their gameplay and how they engage with users fast that that's where the differentiation should be and that's right people should be able to find the different kinds of content that they want absolutely well we've come up we just passed the hour brute's if you want to you know wrap up on your end if there's anything you'd like to share before we close out the episode I'd love to hear what you got to say I mean but no this has been an amazing conversation thank you both so much now we we exist for all we know we're part of the dev community part of the game community so we exist to help this community so thank you so much for having us on it can be more grateful to Oscar to John everybody at Beema Bull looking forward to doing more together thank you guys so much no and yeah just everyone in the audience John asked me to express his you know his apologies the internet is a fickle mistress at times and he caught the short end of it this time but we'll be back soon and we look forward to more conversations with you our lovely audience this has been the Artificial Intelligence live stream and yeah you everybody do good things and be well thank you