Scaling Game Technology and Engineering Teams - with Keith Adams, Ali El Rhermoul and Jon Radoff

Jon Radoff: Getting slowed down in the process of development is sort of the worst thing you can do because the game development is all about trying things because so many. You often just don't even know what's going to work. I mean, it is a bit of a black art.

Keith Adams: Yeah, that's right. And that's why getting to that finish line is super important. Not just from an application world. This is product market fitable. Exactly. Exactly. You don't know if you built something valuable yet and so you just need to be able to optimize for speed. In this episode of Billing the Metaverse with Jon Radoff, John sits down with Keith Adams and Ali L. Ramoul. Keith was a chief architect at Slack and was accountable for Slack's overall technical strategy during a period of rapid growth in users, load and engineering team size. Ali is the co-founder and CTO at Beemable. Let's jump into this fireside chat.

Jon Radoff: All right, everyone. Welcome back to building the Metaverse. This is a first for building the Metaverse because we are going to have a fireside chat not with one other person, but two other people, two people who I really respect, one who I've worked with for the better part of a decade and one who I've worked with for part of a year. I'm learning stuff all the time and we're going to talk about what's involved in scaling up really big technology applications and also games. Let's start there because we live in a day now where a lot has changed. Apps, games, they can get really big, really fast. It's always kind of been the case, but the rate at which things broke has been actually accelerating. And also what I see is the customer has changed as well. It used to be that you could kind of ship your rinky dink beta and if it broke a whole bunch, people kind of tolerated. If it was cool enough, they'd give you that runway to come back and fix things. Today, that could be like the death knell of your company ship and it doesn't break like people never come back. So a lot has changed that's really driving us to think more about how can things be reliable, perform it, how can they scale and what does it mean for, you know, exponential leaps and data users traffic API calls all of that stuff. So just to now dive into this topic, let's maybe start with apps versus games because they're a little bit different, but Keith, you know, I thought it would be really cool to start with you because Slack started as a game company, right? And then they pivoted to what we all know of slack as now and you had to deal with all kinds of interesting issues there. So like, what was the whole background of that?

Keith Adams: Yeah, and I should say, you know, I'm not a founder of Slack. I was cheap architect there from 2016 forward, but to 2020. But, you know, they'd have product market fit and stuff and they'd figured out what slack was by then. But, you know, the shaggy dog story runs like I think there's actually some this is usually told is just a funky anecdote or whatever, but that the founders stir butterfield and his is Mary band of pranksters want to create a massive multiplayer online game. This is sort of in the golden age of social gaming or this is sort of in the middle of when those were getting really well funded. And they did a very interesting game. It was called glitch was before glitch was associated with a bunch of other projects and the company's called tiny spec. They got to sort of tens of thousands of users and they're just for signs that it wasn't going to happen. It wasn't going to work. They started laying people off sort of giving investors their money back and in the process sort of looked at assets they had the hell of a sort, the hell of this music, maybe they could sell that. They also built this internal sort of duct tape and bailing wire version of a persistence layer on top of IRC. Right. So I see it was this way of interacting in real time the technical people been using for a long time. It had this kind of terrible shortcoming that it's not persistent right that you don't happen to be online when you see something. It didn't happen. Sometimes people approach that by like having little bot listeners that record and stuff like that and build search indices and so on. But there's a kind of conarn about slack that it's actually based on IRC technology.

Keith Adams: There's no kind of the protocol IRC isn't involved in product slack and never was.

Keith Adams: So the system that they built was part of an early generation of better real time web tools that were coming out at the time partly in my web sockets. Sockets are sure new at the time there wasn't a really good way to do push in a web app before them. And anyway, so people usually tell this is like a funny anecdote or if it's in a VC context, it'll be like an example of a successful hit it. Or you know, why you back founders not ideas or something like that, but I actually think there's some technical content there that's worth considering. So the pieces go that was carried over from which the game to slack the product was actually a founder of certain more jobs jobless server, the game server. It was the thing that serves on the other end of the socket and it gives you the sense of the other web socket. It gives you a sense of real time presence. It gives you green dots for the people in your organization that are there. It terminates SSL for short round trip things. And it's also the thing that's pushing stuff out to you. And I think both slack and most games are trying to be virtual places one way or another is before we talk about metaverses. But you do, even though slack is textual and doesn't sort of geometrically represent the world, you want it to feel like the people in your slack channel are kind of there with you wherever there is. And that comes out of a bunch of sort of considerations around just latency and trade off to make around.

Keith Adams: You know, doing some ugly things up front to make sure that the real time interaction is sort of nice and smooth.

Keith Adams: One of the really interesting ways you could see the game DNA in early slack was the classic slack basically how to loading screen. It kind of was, you know, we would download a big image of the world, which was everything that everybody's doing and all the new files that have been posted and all the messages you missed and all that everything else. And kind of the way I made a cute little screen and well, it did that. And eventually people realized that that's not what people expect from enterprise app. So that kind of game like element. You know, it wasn't all tolerated in that environment.

Jon Radoff: Interesting, of course, discord has tried to retain more of that as sort of the non corporate communications platform that although interesting that it has been adopted not just by games, but a lot of crypto projects and things like that that I deserve.

Keith Adams: Yeah, I mean, one of the arenas in my life is that I stay in touch with my 15 year old son with a block club on right with this work. Yeah, it's funny because one of the one of the things you mentioned key, the sort of virtual place, one of the things that we do at beamable is we use discord quite a bit for like being able to pop into pop in and out of meetings. So like there's the voice channels, which you can pop in and out. I think slack has created a version of this now, but there was this really sort of powerful notion of like, oh, people are here and there's a co working space voice channel where you know, you can it just felt more real time. It felt more like, you know, a game that was very appealing to us as game developers and business business people. Yeah, and I think there's a chance to talk about that as being sort of fun or being a partner or team of products, but there's also lots of other keys that people rely on will latency for communicating effectively. I think one of the things there's lots of reasons that's infotainable, such a thing they're in pandemic, but one of those that stop and start pattern is like negotiating what I'm been talking about when you're ready to talk or vice versa. We don't do that in your life because like speed of lights fast and speed of sound is fast and we don't have those problems as much. The extent that you can kind of circumvent that even in a textual medium, I think it ends up mattering.

Jon Radoff: I'm just curious, maybe this is a question for anybody watching post in the comments or let me know, but like, it seems strange to me that we straddle multiple messaging platforms. We use Slack and discord and we don't even use one of them for one use case or another like literally Ali will slack me a text message, which is, hey, I'm in discord. I'm going to go into a video chat session with me and like that, that seems like a very bizarre use case. So someone tell me if we're the only weirdos doing that. And I don't why is that, Ali? Why do we, why do we strap up with things? It's because it's so hard to switch from one to another once you adopt it.

Keith Adams: Totally, totally, totally, and then you talk a lot about the decentralized metaverse. There it is. We get what to jump between the different apps. But I think I think he hit the nail on the head there, which is the low latency aspect of like funny enough. It's like faster for me to say, John, I'm on discord and Johnny click on the voice channel and be there. Then for me to click call John and get that pop up. That's like you're calling John, you're calling John, you're calling John or John answered or John didn't answer. So like, I don't know, something about that feels more human about just saying, hey, I'm in the meeting room. Join me when you're ready. You've become forget by the way that that voice, I'm sorry, you know, I'm going to repeat the job with that voice mechanic of the call, right? You call somebody, you're up to them, they pick up and so this is partly is literally mediated by, you know, the hardware that we implemented telephone networks with. Right. That was you were physically was you're being a loop of copper that connected your telephone to the other person's telephone for most of the 20th century. And that was like a negotiation right you and you were using a precious resource of the phone company didn't have enough of the students you did that right. And that's just sort of not really true anymore.

Jon Radoff: Well, let's maybe go back to the scaling question. I'm really curious. In the case of Slack, what were the big problems if there were problems, what were the challenges or whatever in in keeping up with the incredible growth that slack had is seemed like there's this period of time where everybody was moving off Skype and stuff like that. We were a Skype user before that and we moved to Slack because there's so much better. Our daily use cases and then suddenly the whole world is using Slack. So what kind of stuff came up in the context of that?

Keith Adams: Yeah, this is an interesting point. By the way, there were kind of competing products right there was Skype, there was hip chat, there was this world of other stuff that filled the superficially some learning. I think there were kind of two, whatever you talk about scaling, there's a tendency to pretend that scalability is sort of a single topic or that. You know that there's that you get these guys of graybearers were overweight and you write C++ and now you can scale things because we got the scale people right. And I think there's like a, you always have to talk about which dimension you're scaling and I think with Slack there's a really clear kind of discontinuity around early Slack and sort of. I'd say adolescent, you know, kind of still growing into enterprise version of Slack where very early Slack was scaling and this dimension of there are more teams right there are more and more teams every day people starting up more and more companies are about the house. Early Slack market itself is being for small teams is being for people who are 50 or fewer. You can still find like decks out there were they sold it that way or pitch decks that describe this way to investors to. And in the early days that my character has to grow through the well is business small companies and small teams within large companies adopting Slack and that was a kind of horizontal growth in the number of teams where the early Slack team and I think most of the hard part of this by the way was sold before I started to. So when I'm talking about Slack like it's never just my story right like even if I'm there at the time like it's a big group effort but this kind of creates me. So super early Slack we're adding tons of the teams every week. We're basically using kind of horizontal scaling you know playbook from from web applications right so we're adding more web servers there are staple services involved to most important the database. At the time I started at Slack had been in my sequel shop level time it was there so it was application level starting for my sequel. There was a kind of hash out table and that a data table essentially that told you which sharp each team was on right so the idea here is that you had some horizontal number of my sequel servers and basically I can balance allocation of which teams wrong which my sequel servers and they're become you find hot spots right because there's too many teams on server 17. And so an operator would you know get an alert take a look at that and have a bunch of strips the kind of migrants you over to some less popular to share. There so let me pause there for one second right that was kind of early Slack is modestly effective there were a bunch of outages related to that sharp out table that was literally one my sequel master master pair. We use my sequel in a way that it's really intended to be used by the way where you can write both have some of the right replicating back but the application knows to write the odd rows on the left side and even rows on the right side roughly. And so so these were all pairs of machines these logical shards but that one that was the hash out that tells you which best was on which was actually one machine. So this is a single point of failure it came to sort of dominate the causes of outages you know in the year before I started out to say. And around that time the challenge started to shift a little bit there were still lots and lots of new teams being formed but we were good at forming new small teams now. Now the hard thing was that there were these teams that are large after teams that are 100 people or 1000 people or 10,000 people were in the fullest time 100,000 people maybe they've gotten 2 million since I left. And also it's not going to make sense to just have all of their data on one pair of my sequel machines.

Keith Adams: And you can kind of you know there's a playbook of kind of half measures that they get you along way there of just sort of putting more cash in first things and stuff like that.

Keith Adams: We did a lot of stuff but the kind of strategic play there was a migration to the test which is a application kind of aggregator, Charlotte, you know leaf aggregator pattern layer for my sequel that lets my sequel act kind of like distributed database the way people use cockroach or something like that.

Keith Adams: And there's a lot of asterisk caveats like you know just sort of you know stand up a new instance of the tests and migrate everything in one fell swoop.

Keith Adams: It was an incredibly complex project. There were hundreds of tables by the time you're undertaking it each table kind of required a strategy and to some extent required some software for. And I think during the pandemic it finally happened. I think they found it sort of decommissioning a few remaining tables on my sequel. But it was a really complicated major project to get to the point where we could handle vertically scary customers. There was also an assumption there by the way that the customer data doesn't stays in separate streams, right, which is true for a lot of enterprise SaaS applications is never true for games essentially right. If you're an online game, the whole point of it is that you're going to interact with other users and came to not be true of Slack either or it's like sort of having multiplayer channels or what we called network channels in last few years. And you know those channels belong to more than one team and one customer. Now that's one of my favorite features on Slack, the shared channels absolutely between multiple organizations makes so many things easier. Actually we use that to manage a lot of our customer communication. It's so yeah, makes sense that you that the company eventually got there. And you can imagine by the time that came along, there have been I'm sorry John, there have been years and years of people just assuming, you know, that. And that's where data doesn't interact and so there are unbelievable, you know, except online in the application code and data layer. You know, it's a really heroic effort that team is extremely proud of sorry.

Jon Radoff: Oh, I was just going to make an interesting anecdote where I recently spoke to one of the designers from World of Warcraft and what he explained to me was, hey, you know, World of Warcraft is really primarily a chat room. We give people things to do inside the chat room to talk about that's kind of like. No, it's a great. Very different scaling issue because they didn't have like, you know, thousands and thousands of people in a room. You kind of have like a big big server like Orgmar probably had hundreds, maybe, you know, a thousandish kind of players at a time because they had this easier way to segment out the groups across different servers and chat groups. Dungeon instances and things like that, but all we have scaled up some games you and I in the past in which we had things like chat rooms that actually did have to have like millions of concurrent users or at least millions of users in the same shared space. Interacting with each other, we didn't compartmentalize there is like world chat game for uns a cent and start trip timelines and stuff, but anyway, I'm not trying to get you to talk about our chat implementation so much is what what did we learn about scaling up a game and how is it different than some of the things that.

Keith Adams: Keith was just explaining yeah it's it's a decent isn't really different well, I think in some ways it's the same and in other ways it's different I think in the ways that it's the same is. If you can get away with a fleet of stateless you know application servers that will you send requests to and respond and they respond and that's great like you can achieve massive scale a lot of the top biggest tech companies out there do that. But you get to a certain point, especially in games, especially in multiplayer domains, especially in areas where you know there is this kind of the data. I guess the way that I would put it is the shape that the data takes in the database which you know Keith put it called called the database essentially a state state full service. The the way that it takes that it takes shape in my sequel or mango is not necessarily the shape that you want to deliver to the end user and so this disconnection creates a lot of you know if you sort of let leave the data in that shape then you have to do a lot of transformation oftentimes a lot of computation sometimes you abuse the database by writing really, really complex queries that put on you pressure on it. So you end up in this place where you kind of have to decouple the data representation in quote unquote cold storage in the database versus the version that you that you you delivered to players. One example of this that we read into with leaderboards so leaderboards the optimal access pattern for leaderboards is essentially a red and black tree. Typically what you're doing is you're accessing a sorted set of users and you're selecting slices of that data and that data may be arbitrarily deep and so being able to sort of say I want help all players from score you know 2500 and 67 to 5500,000 whatever you know that's going to be different data access pattern than what you're used to in the database. One of the things that we've had to do is to essentially have the data live in one format in the database and then when it gets loaded we had these stateful actor like services that would organize the data into a red and black tree that way you get optimal read and write access pattern so the reads will read from a red and black tree and the rights will basically in the database it'll store a single very small document which represents your player ID your score in which leaderboard is attached to right so the right. So the right is very small and very efficient and the read is very efficient because you've already gone through the trouble of organizing the data and its optimal pattern so those are the kinds of things you start to run into.

Jon Radoff: Let's describe that a little bit more because that leaderboard example is a super interesting use case which actually also shows you how the fundamental data store technology can actually vary a lot in what it's actually what its optimal use cases are so like if you tried to do.

Keith Adams: So I think I'm curious about like I'm not trying to be a jerk here but like beetroot works pretty well right like if you build an index on that score problem like my sequel slap a beetry on it which is you know the block structured cousin of your red, black tree. I mean it's probably more to the story that I don't appreciate but I'd be really curious and also not a mango expert by the way. Yeah, different story of mango.

Keith Adams: But yeah, yeah, no totally and that's a good point. You know you're you're not going to here's the thing like you can get away with it and you'll get something that looks okay right like especially with my sequel.

Keith Adams: My sequel is the ultimate very versatile general purpose you know database in many ways and so you put an index on it you can do that there is a part of the story that I didn't I didn't mention one of the things that we found in games is that oftentimes players. Developers want to attach more than just than just the score to that particular entry they might want to attach additional metadata so there's also a certain amount of filtering that may want to happen on the basis of that metadata. So now you end up having to have either compound indexes or multiple different kinds of indexes. If you if you're trying to do something like that so that that's where things starts fall apart so like you can get a an okay solution and further record like there are databases that are really good at this red is is an example of a database that's really good at sorted set leader board like access patterns and that will work. That'll work that'll work quite well the problem is now you've expanded your maintenance surface area and you've added a sort of a special purpose database to your stack unless of course you're using red is for other things as well which today one of the popular things you can use it for service discovery so. There are situations where people really love red is and will use that but we we actually wanted to reduce our maintenance surface area and we wanted to sort we wanted accommodate not just sort of the basic use case but some of the more advanced use cases that some of the games have like the sort of multiple sorting on the basis of metadata the attachment of arbitrary key value Paris that the leader board score. Idle games are actually a great example of situations where not only do you have leader boards being an important part of them because the sort of basic premise of idle games as you sort of take certain actions and values accumulate whether you're playing or not those values kind of you have to extrapolate where they will be in the future and so you're forced to store like velocity data alongside you know the leader board score to say not only is the score you know 2000 at the last nap shot but it's growing. But it's growing at a rate of 100 per second and the last time it was updated was this particular time stamp so at time you know a day from now it will be not 2000 but it'll be some number right so like that kind of metadata can also be attached to leader boards and this is kind of like the more advanced use cases that we've seen. For leader board utilization that's really interesting one or two kind of know in some ways you know when it's going to change yep. You know like like you know when some range of it. I don't want to consider like questions along there too which is this pattern of having like a in memory view of something that is optimized around some some application level pattern right and whether whether liquidable leader board makes sense or not like there's going to be some version of something somewhere where you need a memory view or that's a lot right.

Keith Adams: Search services are like this right like an informational treat with something where you're not going to just sort but off the database every time some of these types of query right.

Keith Adams: And then you have what you know systems people call a cash variance problem or the view maintenance problem I guess from the database point of view and you have to do something to make sure that your view stays coherent when the underlying data changes and this being blood like a government philosophy there or anything like how do you do that. Yeah so that's a really good question. I'll try to keep this light on in the in sort of the beam of all specific material but. But it the short answer is it kind of depends on the service and the use case like certain services we have have these materialized views that are you know in memory and loaded at a specific application we have essentially. Homegrown actor system where that object lives in at most one place in the cluster and request or deterministically routed to that one place and if the cluster rebalances because you've added a few more than the object may remain or it may migrate to another to another instance that all happens transparently and very quickly and you never really have to notice so there are situations where. Where you know you're going to you're going to have something a little bit different so to to to point to sort of step away from the leaderboard example chat is another example of this and one example that that's a little bit interesting is you don't just global chat is actually not a very good idea generally speaking not just in terms of from a technology perspective but from a human perspective you've got 10,000. That's a bad problem. Yeah exactly you're in a room with 10,000 strangers you don't know who these people are they may start cursing it's a content nightmare it's a community. He's flying by you can't read it anyway. Yeah exactly. 90% of the spam messages exactly a gold farming site or something like that exactly exactly one thing that is actually super valuable and one of the things that we see in a lot of games is group chat also known as guilt chat right so like this is an example of a situation where you know. You you're in a guild and the chat service needs to know that you're in a guild and it needs to essentially automatically manage whether you have access to the guild channel or not so there is a way on the bimbo side where it can subscribe to state updates from guilds and this these objects will essentially when they get loaded up into memory they'll know not only will they won't have to necessarily poll they'll get an event an event update that's published by the by the specific guild that you belong to that says hey by the way Keith has just joined. Are you know are our guild and you need to now update the kids permission so like this this sort of subscription paradigm can be useful for situations where you've got a lot of of these of these sort of materialized states that may be interested in each other they can subscribe to sort of updates from.

Keith Adams: So, Holly just to kind of bring it back to.

Jon Radoff: The broader picture of games like I think would be helpful for everybody to understand a little bit more of like what is the life cycle of a game look like when you start building a game and then you start having users and then you go into like a soft launch with an expanded beta group and then eventually you ship a game and there's a Brazilian people that show up just because there's so many. Game studios out there that are getting funded right now some of them are trying to do this stuff for the first time. You don't always have the resources of you know we talked about world of work after it's hundreds of millions of dollars into fundamental infrastructure to do some of this stuff but like take us through like without like turning the whole conversation into a dissection of like product life cycle stuff in games. Like encapsulators for us like what what how to things begin and what are the big shifts that occur over the course of a game that kind of affect scalability either from like in user standpoint or from team dynamics.

Keith Adams: Yeah, totally store all that stuff. Yeah, totally so I'll start by saying like scalability tends to be heavily anchored as a term in technology but it's not just about technology it's about people to it's about how can your your organization scale effectively right and certain text acts are really great at scaling you know for or will suited to human beings and others are not well suited to human beings are actually terribly suited to human beings their brittle problems are introduced easily introduced and so forth and so on so I'll start by saying like one of the things. Just sort of rewind the first thing that you're doing is you're building a game and the second thing that you're doing is you're operating your game right like those are the two the two major things that you do and when you're in phase one which is your building your game your goal is to try to get across the finish line and ship your product as quickly as possible because anybody who's you know built products either professional air and a hobbyist capacity knows that you know it's super hard to ship something right so just being able to get across that finish line is super important. So that you know that is subdivided in a lot of things in games today generally that means you're not going to be building all of the technology yourself right you're going to be grabbing from off the shelf components. Today it's fairly rare for people to build their own game engine components to build their own physics layers to be do they they'll take usually take a game engine like unity or unreal off the shelf and they'll start building on it. And so once they the sort of John you I'm sure can can speak to the sort of early game design and game development phases right like there's all the prototyping and the sort of ideation and finding the fun and all of that stuff and then there is the exercise of you know actually making a viable business product right because when we're talking about games right now we're not just talking about something that is an artistic exercise we're talking about something that can be sustainable and actually game game game developers you know ability to do that.

Jon Radoff: So ability to make a living so anyway fast forward that finding the fun phase that finding the fun phase the real important thing at that stage is rapid iteration right so I call it shots on goal I game development to me is all about shots on goal it's trying things trying game systems settling on what the core mechanic is that's going to make it fun and then within the game. So it's just a very iterating at a feature level very quickly and then ultimately if your studio is successful and you've got multiple games you know it's being able to have a portfolio of products and repeat that pattern across multiple games so getting slowed down in the process of development is sort of the worst thing you can do because the game development is all about trying things because so many you often just don't even know what's going to work I mean it is a bit of a black art.

Keith Adams: That's right and that's why getting to that finish line is super super important not just from an application world this is this is product market fit or yeah exactly exactly. You don't know if you built something valuable yet so you just need to be able to stop the minus for speed up that's right and the thing that you think you'll see to know whether it works or not. It's okay it's okay the thing to keep in mind is that like today when you're launching a game like the beautiful thing about games is that it's a it's a huge market it's it collapses the other media segments it's growing by double digits but also it's extremely competitive and you're not just competing with your contemporaries right like in the old days you know when I was a kid I'd go to buy a video game I'd go to a physical location and the games that they'll be on the shelf would be competing essentially with games of the same year right there are the same sort of general time. Period today you're competing with games that are in some cases 10 years old right and games that are still going strong right like that the super sells of the world that they've invested you know millions of dollars they have talent forces you know in the hundreds they've been able to sort of build these really competitive these really successful games and who that have been able to stand the test of time so like that's your competition and so when you're entering the market the the player base is less forgiving from the get go right like whereas I'm not going to get the game. Right like whereas 10 years ago you know handful of mobile games you'd get a feature from Apple and you'd be all of a sudden that huge success today Apple is not going to feature you as willingly where if they do it goes less far and and who you and the companies you're competing with are obviously well established so you want to you want to get out there.

Jon Radoff: Let's keep going with the with the product life cycle a little bit more though because so the earlier early stages is like the find the fun rapid iteration you don't want anything that can slow you down but then something happens in the life cycle of almost every game so today almost all games are a live game in some way it doesn't necessarily mean you've got some deep in app economy or something with virtual goods sometimes that's the case it might be as simple as having social elements were happening around your game or content updates every game is live and sort of built around cloud based infrastructure in some way so what that typically means though as a couple of things are happening in in the process of game development that I've seen myself that I think we've seen we've seen you know practically all of our customers a beam of both base which is okay you go from this find the fun stage to suddenly your team sizes growing a little bit and now you have more non-engineers entering entering the process more game designers more artists more content creators play testers Q a so the team is kind of growing out like that but at the same time you're having to like keep up with like tooling content management you might be opening it up to end users who are coming in for the first time and so many teams like grind to a halt at that stage like it goes from this super rapid find the fun process to like a small percentage of the speed that you had before what what do you think in why is that and how can people mitigate that yeah

Keith Adams: yeah I mean listen in a nutshell it comes down to it comes down to the workflow I mean you know when you're when you're iterating when you're trying to make rapid changes when you're trying to scale your team and now there's multiple people making rapid changes you have to make sure that your games architecture can stand up to that right and that's that's really where things start to sort of either force you in a direction where you're refactoring and that doesn't feel good because the last thing you want to be doing is refactoring the game when you're trying to you know ship the game

Keith Adams: or or you're just kind of eating this constant pain you know where there's sort of you know conflict with other team members too many people sort of in the kitchen so I think I think arriving at a decent architecture where your content is separate from sort of your application logic is not just important in terms of allowing designers to be to design and allowing engineers to engineer it also is going to sort of set you up for the next phase of live off of the game and then you can just face a live operations where you can sort of deploy content over the wire and all of that kind of stuff so that architecture component oftentimes you know season the game developers will know what to do right off the bat and people who are you know just getting started they might have to eat that pain quite you know quite painfully for a long time in that first game that they ship before they arrive at something that feels right at the right size team. I can go I'm sorry I said I already click on a problem something of this area but I'm curious about because I think this is actually one of the interesting differences in practice between what I experience as a game consumer and what I expect as a consumer what applications. So something like 2004 2005 or so there became this consensus for sort of what became called web to that like you should never be down that there was no such thing in downtown planned or otherwise no matter what you want to ship new features great. You turn them off in the client you shift the client you wait until every client in the whole world really to use the new feature you start to keep keeping it on the back end.

Keith Adams: So all right.

Keith Adams: And yeah that's the sort of seems like a slow way to develop the feature but it gets you to feedback quicker and keeps the site up right. But I experience as a game consumer all the time there still are sort of planned and unplanned interruptions in the service of you know pretty popular pretty visible games. And I had the glue answer here which is like well that's an entertainment product and so people tolerate that but like Facebook's kind of entertainment product and like Instagram's kind of entertainment product right and they still don't have the attitude that's trackable there were the competitors are going to come eat their lunch if they're down right. Why is that technical is that cultural is it half dependent is the market like what's going on here. It's really good question I'll speak to a to a narrow example for the longest time for the last you know let's say it's gotten better over the years but for the last 10 years which is really forage the beginnings of mobile application development and mobile games development. If you're launching a mobile game in particular you're going to be submitting your app to Apple and Apple is going to take days or weeks to review your app and they might deny it right at that moment and say go back home you've missed this little thing and then you have to repeat that submission process all over again. So that that has forged certain attitudes and it has forced games to sort of adopt a certain mindset in terms of how they manage application life cycles as well as as well as how they manage content delivery so on the one hand as far as content delivery. What you're describing is not absent from games like there's plenty of games out there that slow role features like actually having some kind of an over the wire content update also commonly known as remote config in the games you know sort of ecosystem allows you to say well you know what I'm going to create a segment and only 5% everybody was in this segment is going to get this new feature and that's only 5% of the of the ecosystem so that does exist but equally sort of challenging is if you have a multiplayer game. And let's say you've got some rules where like a character has a thousand health right and you and you and John are all playing that same game and we all have the same character that has a thousand health now the now the developer has to you know they have to sort of change that health from a thousand to 2000 well now they're forced with a conundrum of like either I fragment my community where you know a portion of players will now be playing with a subset of other players or or I just kind of bite the bullet and I say everybody's got to update. And now we don't have sort of a heterogeneous gameplay experience everybody's kind of playing the same kind of people so and you take that that example and you compound it times that that was in because it's not just one character it's like dozens of different changes so so some people still try to achieve this but but but sometimes that this sort of easier path is just to say you know what I'm in a force update my app and everybody has to take the new app and I'm going to accept that and that's okay and there's going to be some churn and that's fine so.

Jon Radoff: So yeah, I knew all the year also talking about I think one of the really important distinction between a web based application and a game as well and we can talk about mobile apps which sort of straddle both worlds in a lot of ways but. Web based apps it's a very very thin client essentially the client is your web browser and everything is essentially up in the cloud to begin with which has a lot of advantages over say a game which is typically shipping a binary and not only one binary but multiple binaries for different kinds of users if you cross platforms you might have an Android binary and iOS binary a PC binary all interacting with this different back end. So dealing with all those content changes and updates and data synchronization all of that stuff just enters into this whole other world of complexity that which is actually already a pretty hard problem or a complex problem for web apps but then you kind of add this whole other dimension on top of it with games.

Keith Adams: I mean in practice nobody delivers an app just over the web anymore like all of his apps also have an iOS clients at the very least and a bunch of desktop clients are to. Yeah, yeah, I'm just curious like like I think there's a lot of it makes sense that the client delivery dynamics are really different. I'm super curious about like what seems to me to be back and down time right like the you know or who's what I seem to experience is back in down time and maybe it's just associated with kind of the kinds of. You know globally changing the rules of play or something kinds of updates you're described that's a big part of it looks a little bit like the way we used to do. Client server apps where you just you know the IT department that your company would say hey be sure not to access the payroll thing over the weekend because we're upgrading it. And that's kind of we have better ways to do back ends now and it seems like that's unevenly adopted in the game community and maybe I'm wrong about this. No, you're not wrong in fact there's a lot of games out there that don't have a backend at all and those that do have a backend you know they they they want to make certain trade offs that that are a little bit different but I think what one of interesting things you said was the uneven player experiences. There's also community backlash to keep in mind which is I think a lot more severe from the games industry like because especially when people are purchasing things in the app and in free to play games that's often you know a significant percentage of the player population. Being you know we when we operated games like we've seen situations where like literally a person one segment of the population will get like a special offer and there'll be a lot of community backlash why did those people get it and I didn't get it right so like that's also part of the equation there's a community management exercise that that can be that can be very detrimental to a game if you get it wrong. Yeah, I mean I think we haven't talked much about Facebook when I was a Facebook for seven years also and in some ways that straddles kind of some of these worlds right you bring back the old Facebook was kind of an internal right which like every feature you ship you get that response essentially but and it literally is information free right like there's no way to interpret that response because everything you do is the response bring that field Facebook. So you get you and that's sort of more of the genesis of the very like data driven culture there which is like well yeah they're saying bring back the old Facebook but they're saying it on Facebook and 20% more engaged than they were last month so let's not bring back the old Facebook or whatever right. And that's a you know probably an anecdote for another time but it reminds me a lot of what you're saying at the community that stuff.

Keith Adams: Yeah.

Keith Adams: Yeah and I guess different from bring back the old Facebook this isn't just who you know who my cheese is actually players turning so it sounds like that's worth worrying about.

Jon Radoff: I think though that the point I was getting at with the multiple application the multiple binaries. You're right like these days like mobile applications sort of wrap around the same web application and then need to be synchronized with it but my observation anyway correct me if I'm wrong either of you is a lot of mobile application development has sort of has been the natural outgrowth of a lot of web stuff so a lot of sort of the synchronization between data and various asynchronous updates of the web. So there's a lot of differentness updates of content and use even user interface patterns that can be brought to life across the different things that has developed kind of consistently in that ecosystem as opposed to game development where you've got this phenomenon of the different binaries but it's in 3D graphics engines you've got this very different mix of asset types like 3D models you've got real time interactivity you've got user interfaces that you've got. So there's a lot of different interfaces that more often they're not are actually built within the 3D environment and so everybody has created this very very heterogeneous mix of UI technology asset updating technology back in technology that all kind of fits that together in the game industry because there hasn't been a lot of consistency in the patterns and the systems around it. Like almost every live game company today has some big mix of like authoring tools and content updates deployment processes scripts etc.

Keith Adams: Right right right.

Keith Adams: Yeah, I mean that's interesting perspective and there's before it's worth like GraphQL was a was a technology that's basically a change in the pattern of running back in communication. It sort of makes the back end a little more flexible and makes it possible to kind of present more at least to iterate a lot more on possible new front of ideas without back in changes if you do it right. That kind of is a technical solution to sort of an organizational problem right it's partly is caused by the fact that you know you can't ship a native binary more often than say every week if you really really push it right at the same time that you're

Keith Adams: back in really quickly.

Keith Adams: The rather than boring of a bunch of details about like Slack or you know Instagram or something like that I think you know every domain is really like has its thing right every domain kind of like has its it's crazy thing that seems impossible about it or that seems impossible. Like when described outsideers I think that you're absolutely right that a lot of these things are that there's a set of fixed set of patterns that are emerge for like the applications for the things that you can use on red, but you can also use them on your phone or whatever. There's a cookbook of ways that you make those patterns work and it's like you know, WebSockets this and GraphQL that and you know, rest this other thing and whatever. That said I think there are also like we're living through time when a lot of these things actually happen on mobile first like Instagram's kind of a super obviously. Yeah, some huge it on phones and really new phones to be a huge hit. Same thing with like the new audio like the clubhouse post clubhouse rules. You kind of needed ubiquitous good microphones and headsets which kind of reparded air pods weirdly like even that was what air pods were for like that's application class that kind of was engendered by new hardware. So those are always coming and I think they keep challenging coming back cookbook approach you know and if you're stuck to the cookbook you're you know you're going to be limited. It's it's a really good it's a really good question Keith. I mean like I think about examples I remember World of Workraft had like the Tuesday maintenance window and that's like a really big company with more revenue than virtually any other game company and yet they still took this you know Tuesday maintenance period and it just kind of it's interesting. I think there's one last thing which is like and I think you brought this up just just a second ago it's like what's the difference between an app that starts out predominantly web and then evolves into mobile where it's like binary heavy versus something that starts off mobile native or binary native. And what I mean by that is like a lot of games start off like with the binary that's where they start not with the backend API that that sort of supports it they start with the binary and they sort of expand from there so that might also be a key difference in terms of where you starting point inevitably sort of colors your evolution.

Keith Adams: So I wanted to maybe talk about a different dimension of this which is like all the things a game studios are having to do that are very similar to what a lot of other web technology and mobile application technology companies have had to do to grow successfully over time which is add capacity for DevOps and server engineering and deployment and AI and all this stuff.

Jon Radoff: So I'm reminded that like in this recent acquisition that Microsoft is making of Activision one of the reasons and I think there's some other reasons that we don't need to talk about that they were getting acquired but one of the principal reasons given was. You know a company even the size of Activision was having a really hard time keeping up with all the live services personnel like that AI the DevOps the backend technology the whole networking piece because today essentially every game company is competing against Amazon Microsoft Facebook every for that same talent pool for scaling up all of that backend complexity that exists in development team complexity. And that's a really tough place to be in an industry that's now by far the biggest media industry right you want to be able to have people get as big as a world of warcraft Activision product but you also want to be able to have smaller teams be able to have a game that can scale up rapidly and I'm reminded also like the office of the office ship done us very very recently like this is maybe a slight difference in the way. Game ship and then engage an audience then a lot of web stuff a lot of web stuff sort of builds in public to some extent and kind of weaves its way to product market fit and then when they start connecting they build around that nucleus whereas in games the product market fit is established more in the earlier stages and that we're a little more secretive with alpha testers and then beta testers. And then you start hyping the game because you want a ton of people and you go to Apple and you say please feature me and you know you suddenly have massive numbers of people on the first day so like the office like big IP like day one all the office super fans of the world are showing up so they're hitting like you know 10,000 API calls a second or something on the backend which. You know very easy to fall down in that situation they didn't get the benefit of like a kind of a organic growth. What is that introduce. I'll leave it.

Keith Adams: And Keith yeah one quick thing which is a little bit interesting is that the bigger game companies or the ones with large license IPs are more likely to be in this phenomenon of everybody shows up on day one then the smaller indie games which actually oftentimes launch their games and kind of what. We try to weave their way up the the sort of rankings among us is actually an example of a game that was around for years before they hit it big right like we're sort of the office and the super sell games like they literally put those games out there they they start spending millions of dollars in paid advertisement and if the game doesn't show up if the audience doesn't show up in the first few months like they acts the game right like so so that that's also sort of an interesting difference but I'll yield to Keith let him have a word. No that's a that's an interesting point I think there's there's lots of different adoption curves and kind of patterns of of growth and there I think the exogenous shock or at this sort of the pattern that is an exogenous kind of some event the world changes or you should feature you change the client works or whatever. And so suddenly there's a step function up in the kinds of stuff to the backend is hitting is a pattern that happens. I'm an advisor at Coinbase for instance and like they're kind of like exogenous macro economic things that happen sometimes that all of a sudden cause a flood of traffic to them and especially like in the early days of this like in 2017 there was an interesting induced demand for them which is like the more they could stay up the more trading the more appetite for trading that was basically because the more they stayed up the more the price move because they were the only thing that was really driving the price. So and that induced demand phenomenon is interesting that you like you have that with sort of network effect games possibly. And the area I'm like most familiar with is very very different actually which is just data analytics systems inside of companies right if you kind of have a presto or a system lets you like gain insights into what's going on with your backend. And sorry that pattern of like there's just a big step up is one of those things that happens it's kind of an emergency whenever it happens even in the context of kind of a going concern. You want to understand why it's happening and want to sort of economically respond to it usually it can be good news and like in the case of a game like this is a nice problem. This is the problem you have of having hit on your hands I hope.

Keith Adams: Yeah well let's talk about some of the technology patterns or or approaches that people have been adopting to try to mitigate either these rapid demand curves in advance or in the case of like that Activision comment about like not being able to hire enough of the expertise to keep up with live services demand.

Jon Radoff: So I'm just going to throw out a couple of terms serverless is one and microservices is another that's just a couple of approaches people are thinking about let's just start with serverless I want first of all let's define that I feel like in the web world like a lot of people are very familiar with this and it's become a pretty significant approach to solving like DevOps scalability issues and things like that and it seems still new for me. I'm going to talk about it for a lot of game developers for whatever reason but let's maybe just start by defining it Keith you want to just take a stab at like what does that mean.

Keith Adams: Yeah I'll talk about it the way that I understand I think of the word serverless is literally a white label staff on AWS lambda which is the product that produces this pattern in a way that you care about and it means that the way that you specify services as a literally a function to be called. Behind the scenes there are of course servers but if you tell a yes where the entry point of this function is they manage the servers for you and they you know they're little hooks for warming them up and starting them up and trying to down and so on and it's evolved past Python it's still pretty much more broadly adopted and Python that I know of. And the the glorious part of this is that it lets you if you're doing it right it lets you kind of forget about growing your applications over fleet right the kind of promise of this is that people who specialize in worrying about traffic patterns and economically addressing spikes and load and all these things are going to be able to respond to this instead of operators that you page and pay.

Jon Radoff: Yeah super weird term that whoever coined this and got away with it called it serverless because there's totally there's probably more servers than ever before in the background. Allie get how do you feel about the definition would you add or very anything or even just looking at it through the lens of like games is there a variation.

Keith Adams: I'll just I think kids definition is is excellent and I'll add just one sort of specific thing which is it's a usage based model right like whereas we're as server you know dedicated infrastructure you're paying by them by the minute or by the second while that infrastructure is up and running. Whether it's being used or not right whether it's seeing you know hundreds of thousands of requests per second or whether it's seeing nothing you're paying for that infrastructure for serverless you're paying often not only art is it. It's sort of a fleet that you don't manage and that AWS takes care of scaling but you're also the business model is different you're charged by the request typically and then there is some additional additional sort of dials there like how much memory you allocate and you know how long the request runs for. So like some calculation of those three variables ends up being your bill as opposed to you know hey i've got some infrastructure running so that that has some important ramifications important implications like. If you're if you don't have a lot of traffic you pay nothing right which is which is great right that's wonderful for somebody who wants cost elasticity in addition to be freed from the burden of DevOps.

Jon Radoff: And you're also do it if we take take a step back and think of it in terms of like capital requirements teams things like that server full for lack of a better term means you're signing up for a lot of DevOps specifically you're going to be doing 20. You're going to be doing 24 by 7 monitoring someone's going to get up in the middle of the night if there's a problem fix it you're going to you're going to provision servers you have to scale them there's just a lot a lot of like dealing with the iron of servers from an economic standpoint from a capital team standpoint what are the advantages of a serverless approach to this versus.

Keith Adams: Yeah server full yeah server full or dedicated infrastructure approach means that number one you've got to go and configure a lot of stuff right there's a lot of stuff you have to configure including load balancing auto scaling rules subnet security groups like there's just a whole bunch of stuff that has to do with. Managing infrastructure cloud or otherwise that you have to be mindful of and and somebody who understands how to tune those things in a way that is aligned with best practices not only in terms of cost efficiency but also performance and also and also you know security so those are the kinds of things that you know serverless helps with no that's it's not a panacea right like you're not going to be entirely like you still have to author the code and you still have to figure out how to take that code from living on your computer. To living on in the cloud and you still need to figure out how you're going to segment environments how you're going to allow your sort of you know your internal folks to do testing and validation and unit tests and still going to have to figure out debugging so like it's not it's not going to free you from the burden of deployment builds and and sort of engineering just just generally speaking but it will help you out when it comes to specifically the act of dev ops of managing a virtual. Private cluster or a physical cluster of machines provisioning servers making sure you can scale out to demand all of that stuff yeah and I will say in particular there's one I think this has been the serverless revolution has been sort of appropriately sold mostly I actually think like it does what it says and gets out of the box. But one thing that I think sometimes people do get tricked into being maybe a little over promised on is that they won't have to think about resource usage again right so the reality is that like while yes that that here that is working serverless right that sort of whatever is in that operating mode you have to do less manual intervention unless sort of pencil and paper planning around how big it is and when to grow it and how much to grow it. He still has these spikes in traffic and the reason that application is useful is because it interacts with other things is because it like reads data from databases and generates logs that go into logging systems and so on and so gigantic weird spikes in traffic are still going to affect your runtime operational environment ways that you need to kind of understand and think about and even cause problems that can then be arguably trickier to go back and be above because there's sort of more magic surrounding. The events that cost it so I don't think this is negative like on the net I think the serverless is the right operational model for you it's just so much cheaper that you. Be a fool not to try to do it it's just that it doesn't completely liberate you from understanding how your thing works at runtime yeah and in particular.

Jon Radoff: It also doesn't just give you license to not care about making performant code and stuff like that anymore because unless you don't care about getting a monster bill from Amazon or something in which case your Amazon account exec is going to be calling all the time and buying you you know sports stadium tickets and dinners and stuff like that but other than that like. You know you still got to pay attention to how things are actually implemented on the back end as well right. You know another part of that to the like just before we move on to microservices that other word that I raised was I think one of the people one of the things people don't pay enough attention to in game development and it relates back to this comment. I made earlier about agility and shots on goal and stuff is so as you're building out a game infrastructure over time and you start to add in all these auto scaling scripts and content management automation pipelines and all that stuff like. People really underestimate the extent of the compounding technical debt that that ends up presenting to your organization and I've rarely seen a company that can keep up with that effectively and when that technical debt starts competing against the things players care about like a new feature or content in the game very bad place to be as a game developer and and often those choices are made very early on and it's so easy like I guarantee there's someone. Watching right now who heard someone in their sit tell them hey just just make this little script like it'll take you an hour and like then we're then we solve the problem and then that one hour becomes a day and eventually it's like months and months and.

Keith Adams: Yeah at the time yeah as it doesn't keep up with various aspects there's the magical can't you just phrase that every engineer dreads.

Jon Radoff: I think I've said that plenty we've all said it everybody's guilty but video. So serverless the way I hear you describing it it's sort of like a means of. Automating your deployment your provisioning things like that in a way that mitigates a certain amount of DevOps doesn't completely eliminated and it doesn't eliminate thinking about what lives out in the cloud and your servers doesn't completely eliminate that but it can cut down on a lot of the labor and the continuous monitoring things that you might ordinarily need to do so I mentioned the word micro services as well which is different than that like that's an actual way of building applications. Who wants to take a stab at that key if you want to comment on micro services and what that means and why people should care about that word.

Keith Adams: Right so and I should sort of say upfront that I partly dine out on being a microservices contrarian right so I'm I am your local monolith apologist if you want to be talked out of your microservice transition I am your guy to talk to you but. The the microservice revolution is sort of the extraordinary claim that the complicated computer program you have trouble developing an understanding needs to be a complicated distributed system instead. And that that will make it easier to understand and deploy and change. It's demonstrably right sometime but it's an extraordinary claim and I actually put it in those terms because I do think it's an extraordinary claim and I think. You should go and do a transition into it with the eyes wide open. I think the one of the reasons it's sort of rare to see a super textbook adoption of microservices that is. Entirely successful and that everybody's entirely proud of and entirely happy about is that it doesn't make sense for small teams to start with it just tends not to. Right if there's four people they're going to write a computer program they're not going to write a bunch of services to talk to other people. And then by induction it's rare to take a working computer program and make it easier to use them by applying a microservice discipline. The times where it does I actually think a lot of what you're doing is. It is actually relying on the rpc gap to impose a type system. If you sort of identify some down there inside of your application and say we could carve a service out of that and you make methods and you said that this one will take this and it'll return that. A lot of the folks that have the most powerful persuasive microservice stories there are in untyped languages with a multi-million line code base. And for them like that discipline of just like this takes this and returns that actually really really helps a lot. But we can do that the language level now and even if you are in in Ruby or JavaScript or PHP there is some gradual typing. You know on ramp to doing that without doing a service decomposition. So I actually think microservices are an absurd look along the answer to like. I actually think microservices like mostly as organizational technology in a way like more so that a computer program technology like as a raw bag of technical claims. I'm not sure it makes sense. But it fits the way that corporations that need to deliver big complicated things with relatively big engineering teams. That only have finite capacity to understand what each other are up to and find a capacity to organize and synchronize. I think for that it's an effective technology. But it's tricky to sort of recommend it because it's sort of very contingent and depends on who's doing what. So there's the most lukewarm micro service.

Jon Radoff: But I think let's just be clear on defining it for everybody as well. But yes, it injects a certain amount of complexity, particularly for a small team to start organizing their application around it. But what's what's a microservice?

Keith Adams: To me, a microservice is some of the world defined API, well, the five inputs outputs that is short of a self contained application.

Keith Adams: Not useful by itself.

Keith Adams: The idea is that we're going to build up a larger and useful application out of these things. And it depends on how micro, you know, like computer people are like to use the phrase micro when it's like they mean senti or like that. I see really like. I don't know if it wants to break a few credits into a million services, right? Lambas are more at least I'd like to let you fund us through the phone. And this deserves the name more in some ways. I said, lambdas deserve the name more in some ways. Like the sort of individual function is the unit of work is it feels more micro than sort of the. Yeah, but. Yeah, you might really have one. Yeah, maybe right now.

Jon Radoff: I'll be build on top of what Keith was just explaining to us though. I don't like the complexity, the appropriateness, but you know talk about games now. Yeah, totally. Why, first of all, is this appropriate for games and why would game developers want to consider it?

Keith Adams: Yeah, so, you know, I honestly would say that games in general tend to be at the cutting edge of technology across kind of almost all vectors, whether it's back end or. Or on the front end game engine sort of extracting every ounce of resources out of the device. So so I would say yes, it is appropriate and to the same extent that it's appropriate, you know elsewhere and I would say. I would say definitely agree with Keith on a particular point, which is like. You know, you can write a well architected application that is a monolith with separation of concerns. The correct sort of type separations at the application level that doesn't break them out into separate services like it's entirely doable. But humans are humans and oftentimes what's there's some expression out there. I think there's some like you know software inherits the sort of corporate organization or something like that your application on the booking like your organization something along those lines. But I think that there is there's value in creating you know that creating lines right in creating sort of nice encapsulated separations so I would say can you build a monolithic application that is as effective from a performance standpoint from from in every standpoint is as a microservice I would say you can probably build a monolithic application that is more performance and shapes off less more of the overhead than microservices but. In terms of of of having nice clean separations they can be extraordinarily powerful layering it on top of the organizational layer but also in terms of just having nice semantic separations where you can scale services individually but also the bug services individually easier to know when you have a very large code base which services misbehaving if they are it's separate applications. But I would say that that that it's a hundred percent true that development teams need not start there right they need not start with 20 different microservices and you know a bunch of overhead to build them into separate containers and deploy them and do all of that stuff now I'll add another thing which is I think you know monoliths and microservices you know I would say microservices were also dramatically enabled by containerization. So whereas before you know the deployment of applications as standalone binaries with a bunch of dependencies was kind of a nightmare before containers right like you had to have machines with pre installed dependencies and if you were to layer on top of that a bunch of different microservices each with their own dependencies potentially depending on different library versions potentially different programming languages. You know in the era before containers that was almost prohibitive in the era after containers it becomes a lot more feasible and it becomes not only feasible but convenient and very with very little additional cost from a from a sort of deployment standpoint. So I would say I would say the era of containers kind of is in some ways synonymous with the era of microservices and that the big win for for for for for microservices was this this container deployment. Now I add one thing which I didn't mention before because we weren't talking about microservices but like serverless is a mode of deployment of applications microservices can be deployed in a serverless manner in many ways the lines between lambdas or serverless serverless quote as it's used in a practical sense and microservices is increasingly blurred in the sense that lambdas are a great example of a bunch of applications that are designed to be you know deployed in a very different way. So I would say that the way you would deploy in a very small atomic unit and an additionally microservices today with services like AWS fargate can be deployed in a way that you think of the container as the atomic unit in the same way that you would think as the function as the atomic unit for lambdas right like you don't manage the underlying infrastructure. So I would say that the container and the scheduling of that infrastructure is handled by by a serverless stack so that's also there is there is sort of a rough approach mom between that they sort of a they're coming closer and closer together right yeah so between between these two stacks I would say.

Keith Adams: I just want to say like I really like the clarity around saying like serverless is a way of operating and that's a wonderful way of describing there is more thogunality here and you're right that's our discussions of this too often or sort of focused on either or I also kind of want to point out I mean that you know I have been fortunate working a bunch of software companies that had some success and so like a lot of times people are want to know anecdotally right like okay what practices did these places use that made them successful.

Keith Adams: I really I am something that actually really brings things di ast how and the sort of the orders as ARM is sometimes to be a welcome in other things but we really want to introduce some of them more popular whether or not to offer or whatever and I get separated be like well these this practice of that practice and there are such things as good and passed is to be clear like I think they're are practices that objectively make your job partner. It's a good sort of really caring, but making awesome stuff and really being curious about whether it's working the way you think it is and really wanting to deliver something amazing. And that's, there's sort of good news and bad news, right? The bad news there is that if your team lacks that, it doesn't matter what process your tools you adopt, right? Like, I think test drift development is good, right? So it's a process that helps people deliver code networks. If you turn TVB into a game that your team doesn't understand and we're trying to hit 100% line coverage on unit tests this quarter and nobody actually cares, you're going to get a bunch of unit tests that are traces of the way your application happens to work that don't actually give you any more confidence that it's staying correct as you change it. Because you approach this process without the right ethos, right? Like you approach this process or tool without the right intention, right? And the good news of this is that it's actually catchy, right? Like the reason people got into computers is that they sort of have some like ability to be excited about doing awesome things and having just a few people that that, when I manifest that and that demand that other is can sort of rapidly spread through an organization. But I think that's enormously more important than, you know, whether you do continuous integration and whether you do TDD and what you, you know, what monitoring stack you use or whatever. There's a lot of religious wars in the engineering world, right? And it's what event and frankly in some ways that's what kind of advances each separate paradigm in a meaningful way, people that care and believe deeply and want to do the right thing and 100% agree on the TDD. That's a great anecdote. I mean, certainly you can end up by you can end up in some ways following the letter of the law without following the spirit of the law, right? It's like, you know, if you're if your intentions are true and you're you're actually seeking the best possible topology for your application, then more often than not you'll end up in a good place, no matter the paradigm. And then diving back to the anecdote mode for a hot second, like the seven years, I was at Facebook. There were two big code repos, both mono repos, but I do or repos if you like, right? One big repo was sort of the web front end application. This is the thing that those what users are and posts are and what likes are and so like that. And that thing was an enormous model. So they've talked about the number 100 million lines of going public before. It's bigger than windows, right? It's bigger than office. There's really a lot of code in that model. And the other mono repo sort of comparable size was a bunch of SQLs plus we'd call them microchairs now, but we weren't calling them microchairs as that.

Keith Adams: It was kind of how we were doing things.

Keith Adams: And by we had to invent containers contemporaneously with Docker, there's internal system called a proper word that was a Docker or a book like that they built. And both of those systems and teams were effective, right? It's not as if Facebook successes due to its adoption of microservices in this part or due to its useful model. I hit that part to extend its success in those years was driven by technology at all. It was driven by the fact that everybody really wanted to understand this thing. Really wanted to deliver something new and exciting and useful, right? And when we failed to do so, we were curious about why. And so I think that has a lot more to do with teams that are effective. The language they use, the mono-reboes versus little repos, microservices, 2DD, CIO, this container of the orchestration framework, that container orchestration framework, and so on.

Keith Adams: And I think that's a, that's not quite right tool for the job, right?

Keith Adams: That's not just being entirely there. No, I know totally, which is the best first. Yeah, it was first, yeah. Awesome.

Jon Radoff: I love ending on this topic of just curiosity and passion for the problem being solved, though, because the thing that I keep telling game studios that I end up advising is, first of all, you have to be really passionate about not just the game, because the game can change a little bit, but the audience that you're trying to solve for, and really find the path to fun and engaging. So like, really get curious about that audience and understand what they care about, and be a little less married, actually, to the game. Because if you're super interested in that audience and what they care about and what kind of joy they want to get from the experience, that's what will lead you to make some really intelligent decisions in the path of the game itself. Just like with Star Trek, we did, we did basically five different games until we ended up on the fifth, really connecting with the storytelling experience that both we wanted and the customers of the game did. But that curiosity, I think, you know, is something that has to go through the whole organization. It's not just the game designers, it's the engineers, it's everybody who participates in that process. And I guess what I want to really encourage people to think about particularly on game development teams today is start thinking about like planning for the success of your game a little bit earlier as well in terms of making some great architectural decisions. Think about like, what are the amazing things that can happen to you when a million players show up in your first month, and also the horrifying things that can happen if that occurs to you. Because you should kind of explore both of that. And some of the words we've used in this, like containerization and serverless of microservices and like the whole deployment process, authoring, provisioning, all the stuff that goes into making a live services game. I really want encourage people to start thinking about that a lot earlier in their game development process so that it's less of that compounding technical debt that you can get shots on goal. You can try a lot of things and ultimately delight an audience, not just at launch, but for hopefully the many years. That you're going to operate the game.

Keith Adams: Well, well, so totally. Yeah. Well, Oli,

Jon Radoff: Keith, thanks for joining in this conversation today. It got super technical relative to a lot of the conversations that I have on building the metaverse. I, but I have a feeling it'll be super useful to people. And if you stuck with us through this hour and 20 minute conversation, and you know a CTO that would benefit from some of this, definitely pass it along. And by the way, if you enjoy these conversations, subscribe down below here. We love having these conversations. I try to bring in people who are total experts in their field once or twice a month, whether it's technology like we've just done or design or the business aspects of all this stuff. I super enjoy these conversations and hope you do as well. So Oli, Keith, thank you so much.

Keith Adams: Thank you. Like us.