Rebroadcast: Artificial Intelligence Livestream: Opening the Hidden Door with Hilary Mason

Hilary Mason: Welcome back everybody. This is the Artificial Intelligence Livestream where we talk about

Jon Radoff: everything related to AI, Gen-AI, LMs, machine learning. It's all good here. Today we've got an amazing guest who I have spoken with a whole bunch of times in the past and that is Hillary Mason. I'm going to give her a chance to introduce herself in just a moment. But if you look back in my archives two years ago, I think it was April of two years ago, I had my first conversation with Hillary back when I was doing YouTube videos. And by the way, there's a reason I'm not doing YouTube videos anymore and we do this live. It's because it allows you to ask questions live. So you're just trickling in. We got 27 people already. But if you're watching at this in replay, take this is your prompt to come back to us on one of the live episodes because you get to ask questions and actually be part of the community. And I'm also accompanied here by Oscar who keeps us on track and does cool things like that opening video animation. I think Oscar that was a call back to Tolkien and the door to Moria. Am I right? Or I don't know why you would ever get that by looking at that hidden door. Hidden door goes with it's all it's all it's yeah it's all part of one continuum. But I immediately thought of that what was the secret password, mail on or something to a friend speak friend and yeah, okay. In any case, we're here to talk about AI and less about Tolkien and more about Hillary. And I'm Jon Radoff. I'm your host for this program. I'm the CEO of a company called Beamable. We make infrastructure for live games. So if you need servers and multiplayer and social systems and online economies, that's what we do. We help you get online with your game. But let's talk about Hillary. Hillary, let's start with you. Tell us a little bit about how you got into all this. What is your background? Probably most people didn't watch our video from two years ago yet, but recommend that you maybe go back, look at it for context. Hillary, introduce yourself.

Hilary Mason: Well first, thank you for having me. I'm really excited to be here in chat for an hour. It's going to be a blast. I'm Hillary Mason and the CEO and co-founder of a company called Hidden Door

Hilary Mason: and Oscar, I love the door. Thank you. That was rad. And I am by background a computer scientist.

Hilary Mason: I studied machine learning, became a professor of computer science, realized I was extremely mediocre at that, but really liked breaking and playing video games. And then started a company, moved back to New York, which is where I grew up and where I am right now. And I've been working around startups ever since. So I was on the first team for a company that became Bitly, so like short links across social media back in 2009, was the chief scientist there. We had a lot of, let's say, companies startup adventures, like I have stories about that. But I became very, very interested in studying and essentially building products in that space where there's some new technical capability. And it opens up the surface area of products and businesses you can now build that

Hilary Mason: just weren't feasible or weren't possible before. So after Bitly had a bunch of other adventures.

Hilary Mason: In 2014, I started a company called Fast-Bored Labs, which was an applied AI research and product development company. And what we did was our own program of machine learning research had a whole bunch of researchers on the team. And then we sold access to that research to other companies, and then we would build products with them. And we'd advise them on questions of strategy. I felt like basically like CTO or CEO therapy was really the core that was the pain we were solving with

Hilary Mason: that business, helping people really reason through. How do I think about my business and the context

Hilary Mason: of data and machine learning AI? What's the possibility space? What makes sense? How do you do this? Well, how do I get the right people on staff? That company was acquired by Claude Erath. I don't know if anyone in this audience will be familiar with them, but they are a data platform vendor. So like all your data in the data center, lots of Fortune 500 customers. And I ran their machine learning AI business unit for some years. And then my co-founder and I started Hit Endor. And this company really came, we started five and a half years ago if you can believe it. And this came out of the, we had built several products with language generation, with embeddings at that point. And they hallucinate, they go wrong, they go weird. And Hit Endor really came out of this personal love of role-playing games of like Open World RPGs, which like Fallout 4 remains one of my favorite games of all time. I was a DM mostly through grad school, like running a variety of different systems. And the fact that this tech, like I thought we were just at the beginning of the window five and a half years ago, where it was possible to start to build some of those like really magical improvisational experiences you have around a table with friends to do it in an online. And so we started Hit Endor to explore that. And here we are today. And should I

Jon Radoff: tell you what we do? Or yeah? Let's dive into it. And by the way, Hillary has some really great content online. I think before I even met you, I saw this video you did with Wired about the five levels machine learning. Pretty fun video. I won't ask you the five levels of Hit Endor because I'm not going to steal their stick, but could we start through the lens of what is the player going to care about? Right? Like we have a tendency to talk about AI as a cool gadget. We can use game development, maybe some neat things. Yeah, developers can do to speed up development. But you're using AI in the loop of the experience. It's actually inherent to how people are going to play the game. Why am I as a player going to care? And what kind of new experiences am I going to have?

Hilary Mason: Yeah. So what you can do on Hit Endor is you've just finished a book. You just watched a movie. It was awesome. Now you can go make your own character in that world. And then role play your own stories that are true to the rules of the world. You can even meet some of the Canon characters from the original movie or the original show along the way and have your own interactions with them. You can try to date them. You can try to kill them. You can take it wherever you want to take it within the rules that are set. And so what we're doing is like trying to create the experience. The guiding star is that if your favorite author or favorite showrunner could come sit with you around a table and run a session of a game for you, where you're role playing and they're making up a whole new set of scenarios and everything in the world is responding the way it should. And it's got the right balance of tension and challenge and comedy and sort of like build up and resolution. We're creating that experience for fans. And we do it in partnership with the authors and creators. So, you know, we're doing it in a way that is officially blessed.

Jon Radoff: Very cool. So I've had the great fortune and just luck of working with a bunch of great authors of my career. So I worked with George R. Martin to launch a game of Thrones game back in the day. And it was always really important for us to bring across the authenticity of his world. It's sort of how we differentiate ourselves. We didn't use any AI, like literally no AI or of any kind in this game. But I remember in my first meeting with George where I demoed the game. I showed him the game and he liked, I think he liked it. He was like very interesting, but are you killing

Hilary Mason: people off enough in this game? I mean, I was really happy to be sort of George's role.

Hilary Mason: And then also seen the show like I can imagine he would ask that.

Jon Radoff: So Hillary, it sounds like this idea of working with authors is really central to this though. So you're going after the fans of certain IP. I'm just curious if there's anything you can share on that would love to know what's coming or hint at it or whatever. But just curious, like what do you look for in authors as well? Like, is it just big audiences? Is it certain kinds of worlds? Is it

Hilary Mason: anything? Yeah, now those are great questions. Because for us, the most interesting worlds are the ones where you can imagine more than one character story in that world. And the world itself is almost a character. The world is really interesting. It has some twists on the tropes or the rules. And it can be a real world setting, but something has to be a little different or fun. The books that don't work well for us are ones in which most of the narrative transformation happens in one character's head. Or where the world itself is deliberately bland so that the character and the changes in relationships really shine. So we really love worlds that are distinct. And like you said, you worked on this Game of Thrones game. And you would know within, you should know within 10 seconds that you're in the Game of Thrones world. And we spend a lot of time thinking about what are the words that someone should say to really feel like you're in this world, even if you're sort of in the shoe store of the world, you're not in the main battlefield or wherever the main action is happening. And then we don't, the size of the fandom, like sure everybody wants to work with huge fandoms, but we've actually found that the most interesting fandoms are the ones that are alive. And it can be very relatively pretty small, like maybe there are a few thousand people in a fan-fit community. Maybe it's the kind of thing where you get a lot of fan art on Tumblr and Instagram or Reddit, but it doesn't have to be huge. And the reason is that there are some fandoms that are large, but kind of bland. Like I would almost give the example of the movie Avatar in the world of Pandora being a super interesting world, but the fandom is not, like I don't see people walking around New York with their Pandora shirts on. Whereas I do for Star Wars, or I do for like even some, you know, some books that are like, oh, you know, you've read that thing that came out 10 years ago, and now you're still wearing that t-shirt, like that's exciting.

Jon Radoff: Or even Game of Thrones, everybody loves, right? I mean, we looked for similar things in our worlds, and now if you look at the like Game of Thrones, you can imagine that you could be virtually any person in that world, and they would have an interest in. Story to tell Star Trek, there's an infinity of people that exist in that universe going on adventures. Walking dead, like every person who survived that long and was facing that has a story to tell. So we were looking for that big world. The difference though, which is interesting about what you're talking about, is you are not necessarily looking for the 10 or so things in the world with that kind of scale. Like Game of Thrones

Hilary Mason: was 25 million viewers per episode, and it's huge. This is really interesting. If you can use AI

Jon Radoff: essentially as a mechanism to allow more worlds to be made accessible through gaming to the fandom that really cares about that universe, because normally we'd have to spend millions and millions of

Hilary Mason: dollars like taking Game of Thrones and making it any version of that game. This is different here.

Jon Radoff: Does this open up new markets for games and maybe authors who have content that normally wouldn't

Hilary Mason: be at the top of the options list for studios? Yeah, I think you've figured it out. This is one of

Hilary Mason: our tricks is that because of the nature of the way the platform works, our development timelines are really small. And the cost to bring something out to market is relatively very modest.

Hilary Mason: And by that, I mean that, and I'm happy to talk in more depth. I can geek out about this stuff

Hilary Mason: forever, so I don't want to monopolize sort of our whole space here. But the way our platform works at a very high level is that we have representations of sort of narrative arcs and tropes and what beats ought to happen. And these are structures that we're all familiar with like a heist, right? So

Hilary Mason: this is one of my favorites to play, where it's like cool, you need to have a thing you want to go

Hilary Mason: after and a motivation for it. Usually it's personal. You need to get your team together, you need to have a plan. There's usually a montage of you practicing to defeat whatever those security measures are. You actually have to go to do the heist, something has to go wrong, but it actually ends up being the best interest of the plot, but either the reader, the player isn't quite sure in that moment how it's going to play out. And then it has to get resolved in the end in a way that's really satisfying either successfully, and you have your fast and the furious style family barbecue, or you're now like on the run, and it's setting up for the next story, which is where you're going to like get your revenge or whatever it may be, right? But the trick is that these tropes are very universal, so a heist in a sci-fi world or a heist in a more romance-oriented world still use the same narrative beats of the heist. And what we do is combine tropes dynamically with the world we're in, and then the player's characters and their choices. And so it's very, very possible with the pit in door to be like, hey, I'm in Pride and Prejudice, and I'm playing by the way with this vampire twist. So it's supernatural Pride and Prejudice and Mr. Darcy's a vampire, and we are in love, and now we're going on a heist, and now we're in the car chase scene, so it's going to be like a handsome cab race where Darcy's getting burned by the sun, and we're trying to get, you know, you can imagine how all these pieces come together, coming back around to the the timelines, because we have pre-written our narrative team has written out these tropes,

Hilary Mason: what we do when we bring a new work to the platform is sort of say, okay, is there anything in this

Hilary Mason: that we don't represent well or that we really want to focus on making better, and then how do we represent this world to make it the best that it can possibly be? So when can the player enter the world? What kind of character can they play? Do we have rules about that? What kinds of scenarios should they encounter by default? Like, what makes, we always write down like, what is a excellent experience look like? And then let's say we have a player who's like, honestly, like, I just want to flirt with everybody and see how far I can go. How do we make that experience great within the boundaries that this world is going to enforce? Or what if they go the other way where they're like, I'm just going to come into every scenario and try to mess things up. And so we sort of lean really hard into the combinatorial and probabilistic nature of the underlying tech to let our players bring their ideas and their direction. The system sort of provide the scaffolding, and then the original creator who has the vision to sort of set both the setting and sort of ground it, and then also set the limits for where they want people to be able to take it, which characters you can meet, how those characters must behave, and all that stuff. And then we do, I should say, we don't use AI art. So the game is primarily a text and sort of card-based experience. And then we also, if we're bringing a new world online, we will usually commission art for that world. So that is one thing that takes a little bit of time

Jon Radoff: to get right. A sense or you know. Yeah. Okay. Well, we've touched on the technology a little bit. I, since this is the artificial intelligence livestream and not only game development, I'd love to dig in to the tech. So when we talked a couple of years ago, there were a couple of things that stood out about how you were thinking about the technology at the time. And one was just, you were actually not using, if I recall correctly, a large language model. You were creating a much more optimized language system around AI for allowing players to interact. And I think a lot of it may have been

Hilary Mason: economic and performance optimization and things like that. The other part that stood out was linking

Jon Radoff: the language interface with the data persistence engine. So in issue with chat GPT, for example, though its memories have gotten more extensive, if you continue to use it today, it's not really like a data persistence system. It doesn't really like associate where were you in the world? What was your inventory and bring that into like a multiplayer space? And you were thinking a lot about that. So the persistent world elements plus the optimization record, which makes it much more practical to actually deliver. So that's the context. The questions really are, how are you still thinking about two years later? Has any of that changed? Have you continued to double down on that? And how does that sit relative to like some of the LMs that people may now be familiar with? Like chat GPT with the O3 model or 4.0? And some of these things that have gotten really, really quite sophisticated

Hilary Mason: in terms of what they're capable of doing. Yeah, now this is a great question and I'm really

Hilary Mason: impressed by your memory, honestly. So we do, we do use LLMs. We try to use specific ones fine tuned and optimized for specific tasks. So I will say we can do everything we are doing today without them as well. And we, I know how to do that. We can talk about that too. Let's go around to your other question. Like why do you something like head endurance? I'm just opening up chat GPT or whatever your favorite model is and playing directly. And I think there is definitely a lot of fun you can have doing that. However, it is not a game. And specifically, it lets you, even if you are very clear about these are the boundaries of the kinds of experience I want or the kinds of the kind of game I want to be playing with you. It still lets you push them in ways that are, you know, it has this sort of like both slightly obsequious but slightly condescending way of managing a lot of

Jon Radoff: its communication. Not a real thing, master. It's not going to enforce boundaries and real rules. And you can kind of get clever and jailbreak it and get what you want. Yes. And also,

Hilary Mason: it doesn't like an LLM is fundamentally pulling all of this data in, compressing the representation of relationships between bits of it. And then when you ask for something,

Hilary Mason: it is prompted in a way that it sort of pulls a new representation out. And so it is like

Hilary Mason: aspirationally going to spew mediocrity at you. It is not going to surprise you too much. And it can be used to make fun things, but I think it's more on the side of being a writing tool and a toy

Hilary Mason: than itself a game. And our players want a game. They want there to be rules. They want to be immersed.

Hilary Mason: They want to be told no sometimes or that wink of like I see what you're doing. And I'm going to pull you back onto the path of where I want you to go or be like, okay, cool. I soapen up a whole new side story. The way of really great GM would. And so I think there is, there are no like right or wrong decisions here, but there are a spectrum of product design choices one can make from it is a video game or a game of some kind or a game with rules and restrictions and constraints. All the way over to actually we're making a word processor. We're making a writing tool. And none of these are good or bad, but we want them for different reasons. I will also say that

Hilary Mason: because I think we're going to get there anyway. There's a lot of people walking around being like,

Hilary Mason: hey, I can give us limitless stories, infinite stories. And I always feel like, you know, we don't need more stories if you have ever been on the fanfic parts of the internet. Like there are so many stories. More is not good. What is good is personal, fun, meaningful, things I can share with you that we have a shared experience with. So the social element of the game too is really important. So these are the product design and game design decisions we've made it head and door to really think about how do we give you the player, the world building game where you're in our case, it is a card based metaphor where you are creating, collecting and achieving access to cards that represent

Hilary Mason: the people, places and things in the world. And you're collecting them in a deck and you can share

Hilary Mason: things out of that deck. And then you also have the role playing component where you're changing the things in the deck in really meaningful ways. And you're also like deep in a story from one character's point of view. And so for us, it is really thinking about if that's our vision, just sort of going to an LM is not going to get you there. So what do we do to get you there instead? I'll come back around to your question about architecture, right? We have an architecture that relies very much on these pre-written narrative beats and tropes. And then we break up every turn of the game into somewhere around 16 to 20 different machine learning tasks, some of which we use homemade classifiers for. So again, I am super old. So these are the kinds of things that take, you can type in, like I, you know, drink from this glass of water or I throw this glass of water at the wall or at the guy I don't like. And we'll say cool for this character in this setting, in this moment, is that a, you know, impossible action, hard medium easy or an automatic. And that's just a little classifier that sits in RAM and it's super fast. And then we say, okay, now what happens? Well, where are we in the course of the narrative beats we're playing out? Does it make sense to progress any of them? How do we combine all the different tropes in this room, this moment of story together into something really compelling? How do we make our NPCs behave in a way that's compelling and exciting and moves the plot along? And one of the choices we've made, by the way, is that we don't have a chatbot in the game at all. Like you cannot have endless conversations with an NPC in a hidden door game. There are plenty of other places you can go if you want to just chat with a chatbot loosely skinned as a character. But we decided against that because it very quickly loses the quality of experience we want to have. And it doesn't let you move the plot along. It's like everything pauses. We're going to have a little interaction here. I'm going to start getting bored with you and asking you questions that are not really about this game or this moment just to test your knowledge or see what I can get it to do. And it's out of the action of the story. So instead with hidden door, you can say like, oh, I ask about, you know, where the the silver slippers are or if anyone's seen the witch. But it's not a chatbot experience. The characters will speak to you. You can ask them other things, but it's more written in the style of a narrative that is always moving things forward. They might say like, I'm not answering that and then attack you. You know, it's always moving the plot ahead. And this architecture we have where every turn we're breaking it down into a bunch of tasks handled with different models,

Hilary Mason: algorithms, approaches, much of it. For each of them, we're saying like, what is the best way to get

Hilary Mason: the quality we want out the other end that is sufficient to run and that it's interpretable. So we always know at every turn of our game why something has happened. And then at the end of every turn, we store it in a Postgres database, which is our game engine layer, which lets us also do things like if we want to, we could run a physics simulation or we could actually do some like mathematical calculations or we could simulate a space here in or an action. And then we use that data structure to sort of generate the text and whatever art updates would happen in that turn of the game. There's a really long way to say that we like to decompose everything and solve every problem in parallel, but individually and well, and then bring it all back together

Hilary Mason: into what our players end up experiencing. So I'm really curious about the social elements that

Jon Radoff: you described. I guess it's a multiplayer component. So people are playing together when they're

Hilary Mason: playing these adventures or are they playing individually? When you're in a story, you're playing

Hilary Mason: individually, but you can share your version of the world broadly. So I could send you a link to my version of the Pride and Prejudice with the Vampire Darcy. And we've really leaned into a model where people build off of other people's creations. You could take that Vampire Darcy and be like, here's how I took him on a date or like, here's how I got him to, you know, go to this event with me. And then I can take something from your version of that world and pull it back into mine. And the idea is really that we don't want another player to be able to mess up your experience, but we want to give you the ability to show off everything you're creating and get credit for it and have people build on it. So it is almost like forking experience and people can take over work

Hilary Mason: and see how they experience it from there. Something I'm just curious about is how emergent

Jon Radoff: are these stories. So we might need to even define emergence for the audience, but the idea that a lot of the storylines are the things that happen in gameplay are not extremely predictable from the original state, but maybe in retrospect, you can kind of see how you got there, but you don't

Hilary Mason: know you would have known you could get there in advance. Is this a emergent game?

Hilary Mason: I think so in the sense that nothing is on rails, no story, even if you try, even if you made the same character choices and through the same

Jon Radoff: actions in, nothing happens the same way twice. And we are often by no means. Inputs, I'm going to end up with a different story.

Hilary Mason: Yes, and because there is also, there are some very simple mechanics, which are, you know, I have this character, you know, let's say in the game, I'm like really good at athletics and really bad at cooking. And then I try to do something that I'm really not that great at. The die rolls going to be harder. There is a role I succeed or fail and the outcome is a direct consequence of my character and my choice and that die roll. And even though it's simple, it really leads to different experiences of the narrative and the overall story in a way that is entirely unpredictable and also very hard to QA sometimes.

Jon Radoff: So that's interesting. I want to keep coming back to the enabling technology again, just because I think a lot of people who tune into this are just curious about artificial

Hilary Mason: intelligence and which technologies are used when. So I understand what you're describing where

Jon Radoff: you need to be able to put the experience on rails and have enforceable constraints and the game master so to speak that makes sure that the experience remains fun and that it's really a game. Yeah. Is that something that you can't do really with an LLM like a chat GPT or a clawed if you give it enough structure around the experience or even have like an intermediate game layer that invokes LLM's. I just want to get at the heart of like the advantages of the technical underpinnings of what you're doing. Yeah. I'd comment on that if I remember as like you're also creating the AI that runs language on your system is also more organized around CPUs, right? It's not a GPU

Hilary Mason: specific technology. Yeah. Yeah. So let me think about how to talk through that without

Hilary Mason: going on too long. I think it is what we use the tech for. We really use, I'd say first let's talk

Hilary Mason: about like what do we mean when we say AI here? There's LLMs, there's we use embeddings extensively so we do a lot of similarity. We cheat outrageously so we have a database of tens of thousands of words and phrases with metadata and every time someone types you know, I stab him in the eye with a fork. We take every word in that sentence and throw it against that database. We say like cool that's an attack. We know that unambiguously an attack and therefore we have a module that comes in that's like cool. We're going to run like the attack module on what ought to happen, what the output should be and the kinds of success or failure outcomes we expect. And then that goes into a system that says given here's the sort of like component action, here's the outcome, figure out what happens next, like how people react to that in the world and what the consequence should be and by the way we're in this overall narrative arc, the tension should be rising. We have maybe we have sometimes we do, sometimes we don't like a secret goal for where the character should end up for for some information that should be revealed so we do have that in the context. And then by the way again we have these tropes and play, this is what has just happened, these characters have their own beliefs and motivations, make it make sense. So we are using different bits, different algorithms, model systems for different bits of this and we cheat wherever we can because if I know a priori that you are trying to attack or flirt or any of these like approximately half of the actions people type in map to one sort of unambiguous like bubble of stuff and if it doesn't map unambiguously then we go to an embedding and say like cool does this similarity is it sufficient to be like okay it's probably one or two of these and if it can't do that then we'll fall back to an LLM and be like okay of this option palette what do you think this player is trying to do here? And so I'm trying to answer your question away that's interesting for the audience which I assume are people really interested in the guts of

Hilary Mason: these systems like how would you build something like this? Double click on the embedding thing before

Jon Radoff: we continue that so I feel like a third of the audience here has a really great handle on what that is maybe we return to the five levels of AI that right had you on like between child and teen like explain embeddings like for our audience here and what how that gets applied in a product click yours?

Hilary Mason: Yeah so an embedding model is really one that quantifies the semantic relationship between tokens or words or phrases and in our case we use language embeddings extensively but it lets us put in a sentence and then say in the embedding space so we're in this end-dimensional space what is this closest to and it will actually spit back we can compare it to a sentence like I attack the enemy I flirt with the enemy and it will give us a number as to which one it's more similar to you and then we can decide if it meets whatever arbitrary threshold for how to use it embeddings are completely underappreciated because they don't generate words they're just ways to take words or other forms of media if you're using different embeddings models and figure out how similar they are we also use this to pull our art together so I mentioned we don't do generated art but we will have artists who will create pieces of art that get tagged with words and we use the language embeddings to say these are the correct art assets based on what we've got to show for this character this location and it is a very clever trick also the embeddings models are much much smaller and can run on your server and not your computer and not they don't require some sort of expensive API call and I think it is tremendously underutilized we also use it in things like pre-generating

Hilary Mason: generic forms of content and mapping into that pre-generated content and then taking maybe an LLM

Hilary Mason: and saying okay now translate this generic sentence into a specific sentence for this character this story in this moment and it is a fantastic trick so okay so let me check my own knowledge of it

Jon Radoff: so in embedding I could think of it as a mathematical representation of some language and a simple way to think of it is that each of these things each of these tokens the words are a number representing a vector which for simplicity sake we'll think of it as like array right and if it's really different then the those two vectors are far apart if they get closer and closer together they're semantically close together in meaning so I could say I punch the enemy and it knows that I meant

Hilary Mason: attack the enemy right instead of fruit punch or whatever it may be the other thing I'll say if the classic example if you want to look it up is look for word to back and you'll see people using this example of king plus woman equals queen so you can use embeddings also to have these analogies that are also really useful when you think about language-based games okay so let's

Jon Radoff: elevate back up a little bit now so we're we're using a language model it's more CPU based you're using embeddings you kind of classifying the inputs that the user is providing how does that come

Hilary Mason: back at me as gameplay and and language oh yeah so you will get a couple sentences of text that'll

Hilary Mason: be like hey um actually I was just playing a game right before joining the session where I have created our Brooklyn office as a test location and in this particular scenario it said you're

Hilary Mason: trapped in a time share presentation you have to escape um and so then I get I know who my

Hilary Mason: character is I already made my character I have her card her character she and then I get to type in like okay I excuse myself to go to the bathroom and then the game engine updates and says cool now you're in the bathroom the bathroom has its own card the salesperson follows you in and she says what are you doing and you notice there's you know a window that you might be able to reach and then I get to decide do I yell at her do I attack her do I try to climb out the window do I go back to the main sales room I set up this particular um scene as a essentially an escape room where there's some something they're trying to sell you there's somebody who want who is motivated to sell it to you at all costs and will not let you leave they will interpret anything you say as an interest in whatever it is they're selling figured out um and so from the gameplay perspective you sort of type in whatever you want your character to try it says oh you succeeded or failed or it was impossible or automatic and here's what happens next and we have behind the scenes also a level of tension that'll be rising the longer you go on the salesperson will be getting more desperate the any like let's say you light something on fire it will start burning and growing because you know this is a this is fiction like we always talk about check out gun and it hit

Hilary Mason: indoor we talk a lot about check out armory like it's check of salesperson with a gun to your head

Hilary Mason: and the buildings on fire and you know they really want you to buy that time share whatever it is um I got one this morning with a vicious venture capitalist who you know who wanted to murder me there's another random scenario yeah right um so uh so from a gameplay point of view all none of this stuff a player is aware of or should be thinking about it all they're really in the mode of like hey I'm playing this character in this story and I have my goal which is you know you have an objective your objective updates as you play depending on what you do like if you punch her it'll say like win the fight if you try to escape it'll say like escape from her like there's your little sub objective um you can try to do it however you want and the system knows uh here's the scenario here's your objective here's the level of tension like I said it has its own secret information which it may want to or be have be waiting to tell the player about um and you can sort of take it wherever you want to go and eventually uh the scene ends and then you get a choice of like what should happen next what kind of action should you go into after this one so one of the earlier things

Jon Radoff: you talked about was really making it feel like the worlds of the authors that you're working with I'd like to kind of return to that a little bit what does that mean does that make it mean making

Hilary Mason: it sound like the author's writing style is it no absolutely not that well let's talk about in the gameplay yeah right no it's a wonderful question and uh you know we have a out of teaching I've cried in prejudice Jane Austin is a amazing writer but make trying to make the more like her style we made it the less anyone wanted to read it um and and this has been true with other authors we've worked with too even when they're modern authors to write in a very you know concise and clear style um we want it to feel true to that style like it can't be discordant but we're not trying to mimic it necessarily um and tuning that as something our narrative team does a great job of this is sort of figuring out like what's the right vibe for this is it and it's really about more is it should it be spooky should it be like tacky should it feel like fantasy like magical should it

Hilary Mason: feel um like an like an action movie um and we do whether we think about it or not we have all of

Hilary Mason: these preconceived notions expectations we've been trained by the books we've read in the movies we've seen on how those the writing should feel for any of those scenarios um and so we try to create that but but even more than the style which actually isn't um isn't the most important thing and we do by the way since we've been talking about the technical underpinnings we do all of this processing what should happen what are the consequences uh are there any like physics we should know about did i just try to put a ferris wheel in my pocket which was a real bug we had to solve where the metadata was wrong because it's too large obviously you should not fit in my pocket um but uh like after all of that we have the generic language version of the what happens and we call those microbeats in our internal lexicon and then we do a pass to translate it into the style of the world with the guidance um and that is an lm call um we're just saying like here's everything that we've decided happens right the two paragraphs out of these like generic english simple beats and this is also where we can do language translation at that endpoint to say uh here's what happens make it make sense or we could have um like simplifying lash if we wanted to for so we could even let players choose the style they receive it in it doesn't change anything of the underlying process except that last bit but what makes you feel immersed in a world and this is one thing i've learned from working on this project every fictional world has a vibe and then some detail that you have to mention in the first like five minutes and then you're there and so for worlds we're all gonna know like Star Wars somebody might say the Force or the Jedi just as an offhand mention one time it's not

Hilary Mason: even the point of the scene but it has to be there um you know for Game of Thrones they're gonna

Hilary Mason: mention winter or they're gonna mention one of the houses and then you're like you've sort of checked that off the list so we spend a lot of time thinking about like what makes you feel like you're there like this incredible romance novel we've been looking at it's like takes place in modern America what makes you feel like you're in this city what makes you feel like you're open to

Hilary Mason: like new love and excitement like the vibes and the details of it um and every book or every show has there are a couple of those little details that matter okay so i i want to talk a little bit more

Jon Radoff: about the tech before i do i just also want to restate what i said at the beginning of our conversation for our audience because we've got over 400 people watching this live right now and i see a lot of like fun hello kind of comments in the chat hello to all of you glad you're all here there's a really great opportunity here for you to ask Hillary who is an expert in AI machine learning and bring that together with gaming but she really understands language models and a lot of these systems that we've just been talking about and the beddings and all that and everything that's sort of in that universe so do use it as an opportunity to post your questions on youtube or linkedin or x or twitch or facebook any of these places that we are because we're watching this and we'll put it live and on the air and we can ask Hillary those questions and in fact if you have a camera we'll even bring you on live if you want oscar can dm you the stream yard link we're crazy enough to to push those edges as well so if you want to be part of the live conversation this is a great opportunity like i said at the beginning though if you're watching this and replay which i know that that's probably 99% of you that's totally great we're really grateful for the time you're taking to watch this but think about coming to one of our live shows sometime in the future because you get to participate live so the tech question that i've got for you Hillary those we've been kind of talking about a lot of pieces you said you make an ln paul a moment ago in creating the couple paragraphs we've talked about your more like cpu driven i kind of optimized i don't know what the correct words are for that but a language classifier you've embeddings that you use it sounds and i think this is maybe the interesting thing for people

Hilary Mason: that are building products and trying to learn from this first of all i heard you say you are

Jon Radoff: using lm's for part of it but you're not using lm's for big chunks of it is that mostly is that an optimization decision is it because that lm's are not actually the right thing to use for

Hilary Mason: particular use cases that are important for you it's a controllability decision so we value quality of experience and controllability over i guess ease of prototyping and given that we've been working on this for five years like no joke it took a long time but um the the if you want again it's like mediocre plots spit back at you with no control like use any off the shelf lm that you're gonna spend endless hours in prompt tweaking and i don't know about you but like i hate writing prompts it is not engineering it is extremely frustrating um and it is not controllable and so we do a ton of work to manage context so that the few lm calls we do have exactly the right context a very clear task and it is a task that an lm is good at and i really believe that they're excellent for translating information from one style to another style you know one format to another format um or taking sort of three discordant bits of information and pulling them together but not in making up great writing like they're not creative they are a distillation of the most tropy you know mediocre stuff by design and so i think a lot of yeah and so if you want the same dribble everywhere like an lm is fine and i also like i don't want to like crap on them as much as i am because like we will often use lm's to prototype a feature and then decide okay this is great for like we understand what we want now now let's do it right and by do it right i mean uh there are a wide variety of ways to approach any task i think one of the things i'm actually good at is if you give me a problem i will give you a solution space of all the the machine learning tools and technical approaches you could take and then we can pick one depending on what you're optimizing for so if anyone has any of those problems like find me on the internet or bring them here we can uh we can do like tech therapy um but what will do is say great like like i'll give you an example like we have these uh like narrative beats for a scene um we pre-filter them by understanding their requirements programmatically like do you have a crush if you don't have a crush you can't go on a date like so we'll just take all the dates like right out of the pool of content that might happen because you're not in a state there's nothing in our game engine that says you're you're eligible for this pool of content yet um and if you do have a crush like it how might that change you're gonna have a breakup scene or you're gonna have a like let's go on a date scene um and just that layer of like actual programmatic filtering and then uh throwing only good options with correct context into another model to make it choice whatever it may be boost the quality and the controllability so high that we know what it is another thing i'll mention and i think i said this earlier in one example we do use lulms as a fallback if our other systems don't have the right context or metadata or can't come to a conclusion is they are an excellent the very they're

Hilary Mason: excellent for that they're sort of like a catch all um in the lake well you know it turns out we

Hilary Mason: really don't know uh what should happen next or there are zero options in our content that will raise a flag to our team and then it'll try to figure something out uh as a placeholder will then come around and actually write the beat that should have been there and off we go the whole system is better um so i love them i think they're incredible but they are not everything for every problem all the time and i don't know about you but often when i am interviewing people uh they are unaware

Hilary Mason: that there are a variety of other tools one might use um and that's you know that leads to like i've

Hilary Mason: got one hammer and i'm going to use it on everything and i see honestly a lot of products that probably could have been built more efficiently and with higher quality experiences right now um because of that but again i'm old and i've been building this stuff a long time but not you're saying

Jon Radoff: not everything needs to be an open AI API call to run your your language based system i'm curious about one of the things that you were just mentioning though which is you talked about using LLM's for prototyping a lot of the time when people talk about prototyping using an LLM what they're talking about is things like coding co-pilots or what has now become known as vibe coding like just using a coding assistant to help you get to executable code i'm actually thinking different system that i use so our my other co-founder Ali one of my other co-founders Ali is our CTO we went to an a16z hackathon a couple of years ago and we used an LLM not to accelerate the coding of this little dnd game that we were making we used it as a replacement for coding at all so we actually just said hey generate like these these modular outputs and give us a package of XML back telling us how the game state was changed but otherwise you figure out what a fireball spell is going to mean we don't feel like coding that and later if we need to code it we will replace that with actual code for performance reason. It's faculty. So that's us what is that what you're doing as well like i'm let's talk about your your use of LLM's for prototyping just to give people like how they can use these technologies

Hilary Mason: and i love i love programming with with assistance like i use cursor every day i'm a big fan of cloud code i don't let it like i still need to understand it but it like just being very serious for a minute on a startup project we would not have a comprehensive test coverage at this stage of our product but these tools make it much easier to generate the tests therefore

Hilary Mason: we have much greater test coverage than we otherwise would this is not in any way like replacing

Hilary Mason: any of the brilliant engineers on our team but it means that i'm not submitting PRs without any test and getting like start from my team for it anymore i'm like hey i can like get the test generated

Hilary Mason: read over them be like okay this one needs to be fixed we're good let's go um we will often use

Hilary Mason: an LLM to test that like let's consider we break a task out it's a function it's a black box you know what comes in you know what you want to come out you can now prototype that with an LLM call gonna be slow and your quality may be all over the map but at least you can get your inputs and

Hilary Mason: your outputs once that's working we can then say okay what's the right approach to this do we want to

Hilary Mason: use any of the other context or metadata we have at this moment of the flow to be like we can do this better do we want to use a different kind of model to get a different kind of result do we want to pre-generate content do we want to handwrite some of that content um and then fill in variables for what's going to happen uh we have many options there are lots of tools in the toolbox then it's saying like okay what are we optimizing for here quality interpretability speed reliability um surprise like we have to consider all that stuff and then we can do that like we can write modules for different actions or whatever it may be a bunch of templates we can write a bunch of tropes and

Hilary Mason: pull them in or things that might go wrong in those tropes um and then we can have an LLM as the

Hilary Mason: fallback like I said if we're missing some of our handwritten content or the content wasn't right for whatever reason we can use that um and then we always log it and catch it later to see uh you know

Hilary Mason: okay this is something someone was trying to do we should cover that in our system or not we

Hilary Mason: should block it we should you know whatever it may be we get a lot of stuff we're going to block um but all of that said uh it's super useful as a prototyping technique and I love uh like AI assisted programming because the code is not the hard part it is the modeling of the complex system in my head in the context and then being able to say like okay you know write this function for me or fill in the like lines of code that are going to pull some context from over there and then they're going to pull it from here combine that data structure and then process it like that's great um I also happen to be somewhat dyslexic and dysgraphic and so and seriously I can stare at parentheses for hours and not see that one doesn't match the other so having essentially like a smart auto complete that gets it right is like I feel the difference it is great I like it

Jon Radoff: okay we're coming up towards the the end of our hour before we do the close close and I give you a chance to send people off to all the things they should look at to learn about hidden or for the people who are developers here who have been learning I think we've already dispensed all kinds of wisdom there we're like don't look just at like LMAPIs there's a lot of efficient ways to do things that also potentially give you a lot more control we've talked about the use of coding assistance now big picture I'm starting a startup today clearly AI has to be something I'm thinking about or I'm potentially out competed by a lot of people who are what's your startup entrepreneur advice about how to be thinking about AI whether it's a game business or any

Hilary Mason: kind of business that you can create so I'm going to say two different things the first one is that

Hilary Mason: whatever the business is that you're interested in building um the same things matter so who is

Hilary Mason: the customer what is the business model and what is your go-to-market and like figuring that out and then the AI bit is that what this tech does is actually change the risk surface area of every decision you're going to make because now you're competing against people who are using the tech everywhere or you know it's going to change the cost functions of what you need to actually bring your first product to market and it's going to do it because it's new ish in some ways it's going to do it in a way that means that even if you built a startup two years ago it's different now so you have to really be critically thinking about what makes sense and what doesn't and what is the tech actually do where does it actually make sense to apply and think about it in the sense of like all you do at a startup is like reduce risk and so what is the next biggest decision for us that is risky how do I reduce that risk is this tech important or not it often isn't but sometimes that it's so it's worth thinking about the other thing I'd say is that uh it is so important to think about the principles behind your product and your business model and to make sure that they're in alignment and this is an AI question because the way the tech is often marketed and used especially right now it's positioned around replacing people's jobs and employment

Hilary Mason: and it doesn't work that well so you're you have to think critically for yourself

Hilary Mason: about the design of your product experience whether it's a game or something else how people are going to find you that's the go-to-market piece what your distribution is what that's going to cost you and then design your business and your revenue model to be

Hilary Mason: something you're going to be proud of like that's something that is really about

Hilary Mason: respecting the people you respect and sort of bringing them along with you in that business model and it is really easy for technologists to sort of forget about the business piece and I think really easy for folks on the business side I want to optimize for cash and not sort of think about the downstream long-term consequences so this is my fourth company so I feel like I I can have an opinion on this like it's all one problem and if you're founding something you have to think through it all and AI just makes it harder not easier uh so if I have fun

Jon Radoff: it's great it's what we're all here for just that's why we do startups right so that was absolutely sage advice um how to do a startup in 2025 so I think we're going to save that clip and and and make sure that one gets out for sure about of all this content uh Hilary where should people go to learn more about Hidden Door you've gone a little dark recently right like what's

Hilary Mason: we have so Hidden Door.co sign up for the wait list and if you're in our discord you will get into play as early as next week um and a whole big thing is coming this summer so uh yeah come play

Jon Radoff: with us so that's like find your discord get in there and pay attention and then then you can get in

Hilary Mason: okay yeah we're sign up for our wait list Hidden Door.co and you'll you'll be towards the beginning

Jon Radoff: awesome Hilary thank you so much for joining me for the last hour and talking about AI and LM's and gaming and entrepreneurship and startups we packed a hell of a lot into this hour so and so grateful for all of you who tuned in especially the live audience you've taken the last hour of your life we're peaking here at about 500 people live watching this so we love all of you and we also love you watching it on replay as well please keep tuning into these if you enjoy this and make sure you follow me follow Hilary where can I follow you as well if they want to just track what you're doing and go go stalker mode on on Hidden Door. It probably on LinkedIn we're

Hilary Mason: sharing a ton of stuff about Hidden Door and also on Blue Sky and H. Mason. Outstanding okay Hilary

Jon Radoff: this was great until next time really enjoyed it we'll have you on again in the future and learn how everything's gone thanks everybody for showing up today this has been great until next time folks.

Hilary Mason: Thank you