Text-to-world: Generative AI for world creation with Kayla Comalli, Lovelace Studios | Nyric

Kayla Comalli: There's no bounds as in we just continue to grow out the biomes, we continue to grow out the geologies and and Add and create maybe even You know imagine up new definitions of what what a world can consist of there's there's really no no ceiling there

Jon Radoff: I'm with Kayla Kamali who is co-founder and CEO of lovely studios, which just released this really amazing demo on Twitter and LinkedIn and all these other places online showing a world building system where you can enter text prompt and It turns it into a world that you can start exploring. I thought it was really great other people have attempted this This is the first one I've seen that actually looked real so welcome Kayla. Thanks for joining me to talk about the work you're doing

Kayla Comalli: Yeah, John. Thanks for having me. Super exciting

Jon Radoff: Since it's such a compelling demo. I want to do what I don't usually do and actually like look at the demo a little bit Just because I want to share it with people so they see the magic that's going on here So when I look at this it starts with this create winter alien forest and and suddenly I'm walking around in a world here And it goes through these other biomes and environments that I can that I can go through Really interesting here. So I just encourage people to look at the full-length version But we're looking at it here. Tell me a little bit about what we're looking at and first of all is this a world building tool for Game developers is this a tool that you see consumers using talk about high level what what you're accomplishing with the product and it's called NIRIC, right?

Kayla Comalli: Lovelace studio, but the product the platform is NIRIC named after oh NIRIC, which is the the adjective for the surreal quality of dream like states and you know Pretty representative to how this is kind of operating is that it's consumer facing we want to make it for gamers and players first You know in a space that there's a lot of agency potential in terms of 3d interaction And that's something that was really captivating to me as well It's just something I wanted to see more of and see You know what what the potential could be if you could prove that out and so yeah it uses AI players can give a prompt basically break down The world into constituent components and this can vary you know You can give the same prompt and get a different world So it's kind of exciting and like you know rolling the dice every time you get to as a as a third person and eventually You know with VR incorporations you can explore this world and You know craft within it will be you know some some later features We're trying to Create it as a an aggregate platform was what we like to say we want to you know make it Versatile in such a way that just about everything is kind of generated on the fly all of the content from you know From the meshes to the to the textures and Takes a lot of partnerships to do that for sure Definitely want to to shout out some some of our partnerships You know sloy day eye and amazing Norwegian company. That's that's doing these real-time meshes You know unique mesh objects and poly hive Another an incredible texture startup great fake folks over there Big big thumbs up. They can do some impressive like 4k 8k textures within five minutes and They're out of this world You know super super smooth facing like no matter what the mesh they can make it they can wrap it in a in a very cool way worth checking out So yeah, the idea is that we want to build kind of everything as an aggregate. So imagine you can Have these worlds built on the fly that they could persist that you could have your own realm Then you could have Communities like a multiplayer community that you grow There's social achievements one can make there's crafting achievements You want if you want to promote your space out if you want to you know build and and trade Within you like unique resources these unique textures that you can modify you can then explore other other spaces The ideas is you know eventually if we can make it all generative and Build it out in such a in a what we like to call a ubiquitous rule system Which kind of keeps each world competitively aligned You know limited maybe some amount of randomness but mostly competitive in terms of you know the resources that are spawned so stuff like that people Can find some some some drive and intent that that builds out these communities long term that you've seen in other MMOs You know the world of Warcraft even line that you know that that kind of experience

Jon Radoff: Which is yeah, well, we're we're kind of going for so you just laid out a whole template for a lot of really interesting things So there's there's the game itself and the experience of playing in a virtual world There's the technology let's talk about the technology a little bit first because I think that it's important to appreciate the complexity in the complication of Building out entire worlds because it sounds like we start with the text prompt and that brings something into existence And then we've got to start pulling in assets, right? So it's a world. It's a landscape but it's everything from trees to objects to you name it and then those actually have to be textured and look like something I I've talked about this in in article I wrote online on my sub stack about You know this very big map of the graphics pipeline that exists in games and how How many steps there are and and the complication of that? Let's maybe start at the beginning So I start with a text prompt like what is happening there is that using AI or is that a search?

Kayla Comalli: Yeah, so the text prompt is using AI The game the game can can be connecting to an API in real-time live and within 10 seconds It's it's using chat you BT to parse down breakdown those components the world you know We can we can try to define a world and and probably accurately do so in many you know different ways through through biomes and Geologies and this can apply for you know underwater world this can apply for for alien worlds You can add post processing effects atmosphere skies lighting fog. There's so many Toggleable modifiable components. That's just very exciting and you know This is just you the the product is still early But the we wanted to prove out the potential behind that and so using chat you BT we can you know we can have it Give us those components. So that's what happens with the text it comes in and then we know what we what range of things to build Sometimes continuous sometimes you know we want a lot of variability in the size of things and and such And then those spawn Real-time in the world we within yeah 10 seconds which is you know Has been tough to do that's a designing designing the the generation system has been has been an interesting challenge We're doing it in in unreal 5.1 but 5.2 came out today Which is amazing because the procedural content network Generation works is gonna be very good It's I it's probably gonna help out with landscapes a lot Yeah, so these assets you know static meshes textures most games the world is set somewhat as it is You can manipulate the elements within there, but it's it's like what you see is what you get Whereas what we're trying to do is is Very fluid and to build things out into like packed level instances. We have to use data layers There's like streaming proxy adjustments and dynamic materials that we have to modify It's uh, it's been an interesting challenge super surprisingly harder than than we expected it to be but you know Doable and so that's been fun just figuring it out puzzling through well

Jon Radoff: I want to dive down into those pieces too because I think people think it's interesting from the standpoint of the world building kit basically provided but also Just even within their own startups different components that they might be able to use in the game like some of the vendors that you were mentioning the back to the language model piece so You allow the user to to provide a prompt It sounds like you're sending that to chat GPT So the use that that's interesting because we're now taking advantage of full natural language processing We're using chat GPT to provide the interpretation and it sounds like the if I understood what you said Response that you're getting from chat GPT is Something around the data description or the properties of the biomes that you can then invoke so I'm using Its ability to have this infinite English or I guess any language expressiveness where I say what I want And to the extent that these biomes and properties are available within your system It'll kind of combine those together into the world that's being built and am I getting kind of step one of the process

Kayla Comalli: That's how it works essentially And we have pretty like the prompt gets parsed Just just a Python script and we'll break that down into the components and Make the the prompt that we will ask chat GPT as a series of I I think we have ten or so that we that we do right now um and we're pretty Forbose with our prompts and we're restricting it to you know give us back very specific answers And we're limiting those we're sometimes making it an array where sometimes you know Sometimes it's it's kind of forced to make certain decisions But and so this is what we why we call it white listed is that you know we're making sure that whatever happens a world will be created And it's just going to be a matter of you know how wild is it

Jon Radoff: That's sort of yeah, it sounds almost like code generation when I go to chat GPT and I'm like generates some code on this Or if I say give me this JSON object populated with a bunch of data You know if you're using JSON, but it's giving you back a set of properties. Okay, so now we've got these properties You'll have some of them some of you don't AI Systems are very data hungry that that is I guess the next part like data access to content and objects Because to make this have Like I guess infinite or near infinite combinatorial results that's where some of these other vendors you mentioned come in so sloid for example tell us about that and how do you use them?

Kayla Comalli: sloid is a Fantastic alternative when we're not running on like a Tesla V100 when we don't have to do stable diffusion I know there's a lot of progress within video and nerf and then in November you know Magic 3d which runs off stable diffusion came out as a as a Twice's fast option to the Dreamfusion Google stream fusion and I think April or so of last year, which was kind of one of the first We're like true truly impressive examples of text to 3d dream fusion takes an hour and a half Magic 3d and video is magic 3d is 40 minutes So it's half the time just about and it's it's getting faster. Yes, but it's exceptionally expensive and There are but there's ways to do this More performantly and we and we also have like need to but by necessity if we're expecting people to to work with Mashes that are dynamic, you know, they can't we can't be weighing down people's hard drives so That's where sloid is is phenomenal because they use they use a parametric base engine. They just Algorithmically modify the meshes and like kind of a low poly like a table for example Just consists of of a different series of like cubes and rectangles and components you know and manipulating those adding just like small components and pieces Intelligent lays is a really fantastic system that happens in less than 33 milliseconds So really cool for the turnaround time for us as well because we don't want players to just hang around for 40 minutes

Jon Radoff: I'm kind of hearing though based on your design that You could start aggregating together lots of these systems So it sounds like sloid could be a generator These currently slow 3d model generations are are another resource you can even use like pre labeled meshes that exist in a library and bring them in I'm kind of making things up here But it sounds like the magic is happening in the biome description in the way it attaches this label data to the various components and I my hearing that right or correct me if there's a better way to think about the architecture

Kayla Comalli: You know having having this system where it's it's kind of There's no there's no bounds as then we just continue to grow out the biomes we continue to grow out the geologies and and Add and create maybe even You know imagine up new definitions of what what a world can consist of there's there's really no no ceiling there And you know for another example We we we're working with Convey we're using their SDK to to have a little companion like just a just a particle companion quite popular You know, you see like the little robot circle or sphere thingies and halo and Portal and borderlands all over the place very very good to have just using background characters and and and PC building and you can imagine you know once once somebody locks down on a world theme Then from there they could just you know have characters spawn in that space and build up you know have persistent data You know as ways as Large language models get better with long-term persistence. It's just gonna get cooler and cooler and we're starting to finally see games that are doing that But the crazy thing is it hasn't been really happening up until now. So it's um

Jon Radoff: It's it's a it's a no man's land for sure. Okay. So you've got virtual beings AI conversational AI power and characters and there as well Which is which is really interesting just before we move on to that and some of the other aspects of this so So it sounds like there actually is this very combinatorial aspect already in that you've got these parameterized Objects that you can get from sloyed and bring into the system you hit earlier mentioned poly hive who as I understand they focus on Textramaps that can be applied to 3d objects. So Just right there by just combining those two things together It sounds like there's practical for practical purposes and infinite possibility space are I'm not super familiar with with them. Can you tell me a little bit about them? How are you how you're using the texture mapping within yourself?

Kayla Comalli: Yeah, so this is a this is actually a pretty new partnership Fortunately, they have they have an unreal SDK which is sometimes a bit rare because a lot of games are developed in unity So how they work is they Use a use a model that takes takes an existing mesh Players can either upload them on the website or you can just submit it as a as an fbx or obj file and It will generate an image in Typically around five minutes or so up to five minutes, I think and you can like if you wanted a grenade is one of their examples Like with all of the ridges and like details it can it can actually just make that so you have a grenade mesh which you know Depending it could be the couple hundred or couple hundred thousand Like a vertices it but the the magic is what they wrap on top of it how they they can emulate roughness They can emulate you know sheen and different shading and damaging and like it's it's kind of creating the whole thing And so that's where we think sleep there's there's a lot of power and if we You know what the the monetization component of of what we're building is is We're still working that out because we're trying to be as non or well in as possible about it But you know, we don't want people to definitely not have paid a win, but like Maybe people if they wanted to pay a little extra they could get a fancy or like more Photo realistic world that that can take advantage of Unreels nanite and and lumine Uh, tech which you know does a real time like level of detail based off where the player is its performance and looks amazing So that's for the stuff

Jon Radoff: I feel like every game studio that started building a virtual world more than a year ago and is in the middle of it It's got to be super jealous of you right now because you're taking advantage of Too really interesting convergent trends so all of these generative AI technologies from chat GPT to the image synthesis He's got and something like polyhive to These vast libraries parameterized 3d objects and sloid But also at the same time all of this stuff happening with software based ray tracing Nanite and lumine like it used to be that we had to bake lights for a whole environment And now you don't have like talk a little bit about that because I think that's also a really interesting convergent aspect of this thing where You're really having you're being able to focus on the creativity of the experience now instead of like go and do something super boring

Kayla Comalli: Yeah, no, we we are very lucky in that regard um We we don't have to build roller coaster tycoon to in assembly uh Though that guy is one of my heroes um and no, it's amazing uh I Think you know the timing of it couldn't be better uh officially I founded the company in February 2021 so about Two years and a couple months ago originally as a as a virtual reality company in in unity so very different Trying to build a multiplayer kind of avatar Elemental combat style thing But that has obviously shifted in fact, you know, it was a year and a half of Development in unity and then when when unreal five was released last last April we saw the potential with how Performant this rendering could be how How easy it would be to integrate these assets, you know when you want Uh Unity is very performant as well of course, but um, you know it compiles in c-sharp whereas compiling down into c Can really maximize maximize the opportunity for for runtime capabilities um So like that that photo realistic threshold is kind of there for in terms of visuals and then The blueprint system with unreal is i think fantastic um, you know previously I'd been a coder for eight years writing in c++ type script uh python lots of ascripting I you know it had been doing it for a long time. I worked for robotics company and did perception and computer vision, but um I was It was able to jump into this in a totally you know a different system where you're not typing stuff out You're just connecting nodes and founded incredible. I won't go into the details like I won't go into the weeds about it But um, yeah, big big fan of of that system and it makes production Very quick very fun visually it makes it makes the uh the kind of design process in your head a lot smoother

Jon Radoff: I find yeah, this is the direct from imagination era that we're entering into it's idea of We're using the physics of space and using that within the rendering system So light is happening the way light actually happens versus what happens in game and jins and past which is All these that's a bag of shader graph tricks that people figured out how to do shader programming tricks Which was great experientially, but it means that there's a lot more weird work to do now Now we're basically basically modeling over real objects which for do for combining that with what you're doing Which is all of these vast libraries of 3d objects and new texture maps and and actually bringing those forth into existence You kind of needed that innovation to happen or else you would have been back to the world of like setting up scenes pre-baking them Or something box-alized like a minecraft that this gives you the expressiveness because of the physics that's taking place in engine

Kayla Comalli: Yeah, absolutely Yeah, and there's and it actually can't all be done still You know even now we're we're in the midst of it for sure There there are certain capabilities like engine level code that you can't change You know when it comes to how landscapes and how the the foliage is generated at runtime They just you know There's too many editor functions that are linked to it that you can't you can't touch you can't modify So you have to hack it and really strange in odd ways for you know data layers and level instances come in There's ways to do it now So that's the kind of a high level developers had to face previously It's just like a new a new level of challenges, you know, what can this do? Okay, what can it not do like let's let's and let's figure that out and let's try to break it

Jon Radoff: The oldest paradigm in game development is You know me the game developer the designer. I make something and then I ship it to you and and now you get to play it Along the way we started to have sandbox environments. We've had On user generated content. We've had things like roadblocks and then even Minecraft worlds and things like that Seems like more and more of this is going to happen while you're playing and more Immediately, I guess the question I'm getting at is Talk to me a little bit about The player experience around this how is this going to shape the way The players the users the citizens of the metaverse whoever they are here in the future How does this really shape the world for everybody a lot of the motivation for me and my co-founder Alex

Kayla Comalli: is just phenomenal Then in the game development space for 18 years and just total total powerhouse We both had this shared vision of kind of coming back to the original theme of the metaverse like how it was written out And in terms of you know creative potential in community building It was like one of the first you know quote-unquote MMOs people you know the metaverse term has gone through some Some roller coasters in the past couple years But and some people would say metaverse is a dystopian term even you know when when Stevenson wrote it as well No, the world was dystopian, but the the actual metaverse itself was quite amazing and you know limitless and I think the spaces that we could You know achieve and then we recreate for people span social agency In particular, you know having having assemblons of control and in a crazy time. Yeah, tell tell me what brought you to this So it's you know kind of tinfoil hadi but thinking of where we are now, which is in the AI revolution like how did we get here? What does it mean for people in the future and you know the best thing to do is locally into what happened previously So you know the the industrial revolution on the agricultural revolution So in the late 18th early 19th century there was steam power And you know that the steam power revolution destroyed our artisanal crafting industries and agricultural industries, but it created textiles and iron steel railroads and mechanical engineers Then in the late 19th early 20th century there was the electronic revolution the combustible engine So again further craftsmen were impacted agricultural jobs were destroyed textiles and Kind of a lot of labor intensive jobs were were destroyed But what came out was more manufacturing a lot more engineering transportation development for cars and railroads and you know the whole Assembly line system to the mid late 20th century there was the digital revolution a lot of the manufacturing jobs were destroyed Primarily and then obviously we created it silicon valley software engineering, you know everything every revolution previously had a you know Something got destroyed but something came in to take the place and and oftentimes it was easier It was you know, maybe still entry-level jobs, but more higher technicality more white collar ease So people aren't breaking their backs, you know, but here it's a little different because you know We're in the midst of the the age of autonomy where you know front end McDonald's Ordering systems are replacing face-to-face over the counter People and such but AI is now replacing jobs like solutions and marketing cinema like the arts as well It's kind of fanning everything R&D and like customer support. It's massive scale. It's not clear where where people will go now

Jon Radoff: historically most automation was sort of Replacing for lap what we might call for lack of better term cannot unskilled labor jobs and then and then some skilled labor jobs as well But This is now also absorbing a lot of knowledge work and a lot of relationship oriented work That's different and potentially doing it at a much larger scale. Yeah in the midst of it. We still don't fully know

Kayla Comalli: It's hard to predict exactly what's going to happen, but what we Can probably see already is that there's you know a big Social existential fear like instilling like it's it's a lack of agency if you lose your job to AI You don't have anywhere else to go. It's a lot of you know control-related fear We're building up the world, you know, of course. There's going to there's a lot of advances That's that's where I think crypto and and with more stable like maybe maybe like real coins or stable coins and back Securities things are going to start transforming once You know certain components get settled You know let the dust settle on everything lately, but Yeah, I think that's where the metaverse has a lot of power and so you know, there's it's it's it's straightforward where AI has powers and has Value in these in these fields and offices, right? You know, what what does it do? It gives us these heuristic baselines. It can quickly give us you know Not common sense, but like high level answers to a lot of things I think there's a lot of power there and it just comes down to How can we make it how do we have the to does AI give us the tools to escape? You know some not even escape but build a new digital identity in a new space with your friends with you know Maybe maybe ways to to find education entertainment Eventually, you know fitness and just personal growth and you know, we want it to be more than just a game That's where the world interoperability is really important to us You know, I think Roblox is amazing and and minecraft and their game mechanics are incredible and you know definitely stuff where we're we're fascinated and interested in building out in some forms but um, we want it to kind of matter in a in a way that's that's Interoperable more persistent like not a fleeting kind of short-term experience ideally

Jon Radoff: Well, you you said escape and then you moved on to some other things that are that are applications that move beyond escape is it possible that this is A realm where people will even be able to find meaning like meaning through social connection meaning through You know, whatever they define meaning to be for themselves. What do you think about?

Kayla Comalli: I mean for I'm I'm pretty biased personally. I think I think 100 percent um, you know I've loved games since I was a kid. I played you know roller coaster tycoon twisted metal The Tomb Raider resident evil too. Well, I watched my stepdad play resident evil too. I could Spyro final fantasy tactics like so many games on different consoles and um, it was amazing for me because As a kid, you know My mom passed away when I was 13 from breast cancer. She had a decade like a decade of dealing with breast cancer You as it you don't have any any space to to have Be have your identity to kind of have a sense of control and and in that space. I was able to You know have something there and and it was amazing and powerful In this in that space people built those worlds, but in you know one of my favorite games was the sims and the sims is you know You can build it out and you can do these things and I think um, you know, there's Just it's hard there's there's a hard like Human emotion like that's like that is a component of human emotion that is consistent with many people the vast majority of people I think and you know Games and television has shown that we're capable of Take going outside of our minds. The R has shown that you can put on a headset and literally your brains like Okay, I'm somewhere else now like it's it's totally doable. I believe yeah

Jon Radoff: When you talk about the sims what I'm hearing is you had agency there you were creating things You had this experience of creation Is that part of what you want to bring to life within

Kayla Comalli: Niner? Yeah, I think so and um, you know creation in terms of building your own world having The idea of initially is that you're you kind of your own character, but the character can evolve you know have different facets and and Occupations with when we you know have our early access product hopefully mid july or so we'll be focusing on on talking to people in the role-playing space and the Dungeons and Dragons and like general hobbyists We think that they would have a lot of fantastic creative input To help you know figure out exactly what that what that means we we don't want to we can't assume that we know and we don't We wouldn't want to I think we're trying that's why we're consuming our facing as well We want to build it for people definitely an interesting challenge and then just we have I have Alex Fortunately as my co-founder to just help with all these hyper complicated gaming mechanics

Jon Radoff: Well, see you mentioned web 3 so and I love I love the bravery to use all the terms that that have gotten kind of downtrodden over the last Year or so web 3 metaverse all of that and and kind of going back to the core of it What you mentioned the things with the assets and speculation and whatnot? What's kind of maybe push that aside? What is the core of web 3 if I'm hearing you more about the interoperability Aspects that you were touching on earlier in our competition or is there a bigger picture there that I'm

Kayla Comalli: With our you know focus on you know being maybe like a steam product that you would download as opposed to web based We haven't spent too much time thinking about how that would look I think you know, there's also when people say Interoperable and in reference to web 3 typically it's in terms of Collective like collective other companies coming together in like transferable avatars transferable like presence and Kind of like a decentralized sort of way I think but I also think that there's you know local like there's a lot of power with When we start transferring assets that are larger if if people are local and like can kind of host these services and they pseudo blockchain like sense Then you know we can get we can start sort of leveraging a community server to make the world kind of living and breathing and growing in a new way um and you know of course as we add these game elements and components We will we want to and would have to monetize at some point so Different systems like you know auto-battlers or you know mechanics that you know hopefully aren't like Gambling but more just like you know community experiences are our best fit for best fit for this format Then it's just a matter of you know ease of it ease of access to like how how quickly can people jump on and get into this and we want The idea is is you know for us we want to abstract it all as much as possible get their foot in the doors and just See you see what the minimum requirements are to get that to happen. So Yeah, it's definitely in the pipeline for sure Just you know exactly how we integrate as is up in the air and I think I will leave to leave to the people who are Super super invested in this space for sure when you talk about

Jon Radoff: Community Community servers it almost sounds like a decentralization kind of Approach to this sir. Is it part of your vision for people to own their own worlds and races? I know that's maybe not where you start with steam today

Kayla Comalli: Where does this come? I think like in terms of digital real estate and land claim and ownership we're Reaction like our first reaction is is that we don't we don't want to do that It's it's kind of like counterintuitive to leveraging what would be what is truly a vast, you know Digital space and there's other Mechanics and in my head the idea could be you know if people want a competitive system they can like have their character Function and exist in a like capitalist sphere and then there's a socialist sphere where that does not happen And it's different and you can like it be kind of you know just like even online which which Alex spent a bunch of time working on as well Another shout out Has been working on is there had been working on is um these like incredible like Studies that that you know PhDs have done on social behaviors and and you know how communities form and adapt According to these these parameters that they're built in with these with these with these worlds these these social worlds and Yeah, I think Definitely like allowing kind of anything to happen in a in a constraint and a constrained way, but like you know What kind of person are you? What is your digital person? What do they? Are they interested in and what spheres would they you know want to kind of grow or or help build or explore and and you know Maybe we can make that make that a possibility um, and you know MMOs are very enticing as well Metagravity is a company. I met a gdc awesome brilliant people I think they're based in the UK and they're just They're they're working on like you know managing a single server with Like hundreds of thousands of people which is you know, we're getting to that point and then they're they're actually making it happen So you know once we get there then could be pretty wild I think well

Jon Radoff: What should people know about Naira can and love lace that that we haven't talked about so far?

Kayla Comalli: I think I mentioned briefly on some of the components, but obviously it's it sounds you know pretty pretty large and and still maybe a little nebulous But the idea is We're gonna be building a crafting system. We're proving out, you know Some some some early game components some some core game elements that that people can play and and want to you know Continue coming back for we want we want authentic organic, you know We want to the virality network effect k factor whatever we want to call it We want people to bring in more people and want to do that and so you know and and kind of differentiate ourselves as as a b2c Not not a b2b. We want it to be people focused and So so things like you know the companion system then PC system will be coming out We've been we're working on both the world building as hard the world crafting and the companion system sort of simultaneously And I think you know in the next month or two you'll see some more videos coming out that are that use generative components in a different way And I think that will be really exciting because it's gonna it's gonna start taking more form and differentiating as a as something that that is you know Enticing for people and and what we're hoping for is that you know july mid july We'll have we'll have an early access release Maybe with you know some of the targeted role-playing communities like I mentioned but You know also by the end of the year hopefully I like a like a full on like a multiplayer experience for players to check it out and experience and build out these persistent worlds

Jon Radoff: Baylor that thanks so much for joining us to talk about this at the video you posted recently Has gotten so much attention for a lot of good reasons it looks really cool and We want to come back and talk about it again and I'll be watching it online So thanks for coming here sharing what you've done so far and also your vision absolutely john yeah, thank you so much

Kayla Comalli: You