MAS.S61 presents Jon Radoff, Game Pioneer, Entrepreneur

Joseph Paradiso: Hi everybody online, I'm the interest is me and physically in person this is going to be a really special lecture today we waited a while to get this exciting speakers together and it's so good that it's all happening. So first I'm totally pleased to introduce Jon Radoff. I met Jon on local VR AR panels so evaluating future ideas here on Boston. He's one of the real BoAQ guys within probably a couple hundred miles a year at least if not longer for VR AR stuff. He's got a reputation just as massive in the space. He's a very early player actually both of the speakers today, Jon, R&A and Jon or early players who need a whole idea for sure reality. Jon started and actually gave him this one but he began his career I guess in that institution down the street that we have known about a rather than a situation Harvard and that was someone who come from Harvard actually a good number of you do. I should say a lecture that VR hackathon about the best science in the world. Jon like so many famous Harvard and long as did it be stayed. He heard the siren all to start company I guess right so he stopped his Harvard education like other famous entrepreneurs did very well as a result. He started knowingly, an early internet service provider but then while he was there he created legends of future paths one of the first commercial you can pronounce that name I should know it because it's his standard do I give the letters because that word. Oh MMORPG, an asset-wing multiplayer online role test. That's what I thought about. Good. And Jon has thought deeply about again the roles of virtual worlds and games and also social roles people play in these environments and how that plays out to other things besides games. But since that Jon founded and advised many game companies he's currently the CEO and co-founder of Beamable, a live game services platform that enables the creation of online games based on Unity. So Jon, great thank you.

Jon Radoff: All right thank you for having me here at the media lab.

So Joe kind of told what I've done but the common thread through my career has all been about creativity and people getting to do cool stuff online. So I wanted to talk a little bit about the metaverse what that means whether it's even so good term but through the lens of humanity and culture and also how that's rapidly accelerating with some of the tools we have available now which includes stuff like platforms for user generated content sandbox worlds and the very very quickly accelerating universe of of generative AI technologies all of these things are going to come together to my talk.

Well let's start with the idea of what the metaverse is anyway because we were talking today about where that we've been here before and you know the whole word meta goes way back to ancient Greece. It didn't really get used in the current meaning which is sort of information about information until around 1920 when it became the field of meta-mathematics then we had meta programming meta programming came out of list programming with that medicy on the on the list keyboard. The book in 1967 was one of Timothy Leary's favorites because it was all about frankly taking acid and and figuring out how you could reprogram your brain so it was very much meta programming of the human and then we've got you know meta everything suddenly meta can appear and everything and then metaverse appears. Now I think metaverse when Neil Stevenson was talking about it he was thinking of it almost like a digital plane of existence so that you could kind of live and alternate life in but I really like the idea of applying the other meaning of meta this idea of meta programming and information about information back to it as well because I think as you'll hear in my presentation today the idea of shaping worlds and exposing ourselves to them and allowing us to shape experiences that then affect us as well such as creating an avatar online wearing it in real life we're starting to blur the distinction of like who we are with our digital person on his online.

Now in the current era of metaverse there've been a lot of people who have tried to co-opt this term and that's why with Fortnite creator mode and Roblox and a company that we all know who actually renamed themselves to meta to go after this term as well as some of these web 3 blockchain things like decentralized and everybody has been going after this term and the problem is that all those things are not the same and that has led to a lot of market confusion so I wanted to spend a little bit of time on you know what people mean by those terms first usually they mean one of three things they're either talking about an ARVR a body experience that's that's kind of the Facebook meta version of the world sometimes they mean a virtual world platform and that could be like Roblox Fortnite second life all of these things or sometimes they they have something about blockchain and mind where it's currencies and virtual asset ownership and stuff and all of those are totally fine those are those are perfectly capable product definitions product manifestations of what the metaverse can be but I don't really think they tell us anything really important about what the metaverse is

you could look at it through the lens instead not of like this product definitions but what do you do with the metaverse so a big part of what I find people do is it's a stage for storytelling for imagination you kind of explore other planes of existence that I think goes back to what Neil Stevenson is getting at when he thought about a digital plane of existence that we've been living on it's about real-time activities so most of the internet and the web thus far has been more transactional in nature you go to a page you look at information you pull information off of it this is activities and by its very nature you're then in social experiences with people and sometimes it's actually about transforming the reality we live in in the sense of you take this digital plane of existence and you layer it back on the physical space that's augmented reality okay but I still find that the things you do with the metaverse still doesn't capture the heart of what actually what's important to be about this

I want to think about you know why do we even have a metaverse why is this a thing at this time so let's let's roll back way in time here and think about what a proto-metaverse would have been and I would argue that all games are essentially the proto-metaverse games are abstractions of reality right they have elements of storytelling there's some kind of shared imaginary space that has to take place to play a game so we look here at that ancient 20-sided die it's not a Dungeons and Dragons die for those of you were familiar with the 20s that's over a thousand year old object that was used in games and in the past chess has been around for a long time if you played chess is purely a mathematical abstraction you could do that you could actually create some kind of mathematical set of rules it says here's how you play chess but it's a lot more engaging to think of it as a military conflict with kings and queens and the fact that we've characters allows you to kind of visualize and think about what chess represents and the fact that you can sit there with another player and enter this imaginary experience is what makes chess interesting this is true of pretty much all games I'm sure because it's games there will be some exception that someone will tell me about later but in general that's what games are

all games had these sort of constrained experience until really Gary Gaigax had this amazing vision around letting people tell stories together but also combining it with a set of rules Dungeons and Dragons is so interesting that I often think of it as the first move away from those proto-meta versus to maybe a real metaverse and its instantiation originally had nothing to do with computers and online networks and whatnot but it is a shared imaginative space it's a place for creativity you get to take on different roles and there's enormous emergent play and emergence is something I'm going to spend a lot of time on in this talk as well because it has a it has a big role to play in all these sandbox worlds and where we go next with AI but it's emergent because you can't really fully predict to the outcome even the dungeon master who may have an idea about where he wants to take the story of the game has to accommodate the various inputs from the other players and that means that it all gets interwoven to the other into a story which is entirely emergent in its properties

so a few decades ago we started taking all of these game experiences and we started turning them into digital experiences that's what I think the metaverse now needs it's about taking these imaginary spaces and just digitizing them, dematerializing them, allowing us to cross time and spatial barriers and give us a place where we can go through similar imaginative experiences together online without having to necessarily meet in person.

So the thing I'm getting at here though is because of the digitization that's happening around this and because of these kind of proto worlds where we have this inherent desire as humans to create, to share, socialize, to connect through storytelling, we're entering a new phase, think about how much life has changed in the last 20 years. We have online stuff more than 20 years, they're important words and stuff but it wasn't as widespread as it is now compared 20 years ago to what we have today in terms of how important the digital realm is for all aspects of your life. For most people you are participating in online games, you're in social media, maybe you're in esports, maybe you've done online dating, maybe you participated as a viewer or even a streamer yourself in live streaming platforms, maybe you've done some of the cryptocurrency stuff, maybe like me you're capturing the biometrics 24 hours a day and uploading it to the internet where AI figures out how to tell me how to live a better life. So all of this digital realm is part of what we do now and I think that it's digital identity and it's core and I'm going to elaborate on digital identity and where that really goes next but the key idea here is our identities who we are is now very much comprised not only of who we are physically but who we are digitally and that's changing a lot of the trajectory of human civilization.

Now a way that we express our identity online is through avatars and this is the idea that I want to represent myself online and appear how I want to appear that appearance could be completely different than I want to be in real life you could be you could look like and be any person you want to be. That said that identity through an avatar I think is the very first baby steps on a path it's the core of the path but where we go next is all about extending ourselves further into the digital realm. So if avatars and Twitter accounts and Facebook accounts and all of these things are the basics of identity the way we initially present ourselves online

the next step beyond that is our expression what do we make online what do we put out online as digital beings and there's a stage in and beyond that which is taking our will and directing it to go and do things online that we that are important to us and that's a big part of we are artificial intelligence technology is going to take us.

Okay so I've been talking about ourselves and our our kind of digital identity online and how we project ourselves as humans into space that's not the only thing we're projecting into space we're actually projecting physical space into meta space as well so we're taking our objects with us I'm going to start going into you know some more specific examples now of how some of these things are happening getting into technologies and stuff but just as we're projecting our personas in digital space we're starting to take things with us in this space

this really interesting field of reality capture allows you to take objects from the real world with you so in the past it might have been model something with a 3d system and design something and put it into into digital space that no doubt will continue in some respects although generative AI is going to displace a significant chunk of it we'll get to that as well but

this is a little demo I actually put together a couple months back when there's still snow on the ground here in Massachusetts so this looks like a film it's not a film this is this is all digitally generated the gargoyles reel I scanned it you can see the 3d model that it pulled in there and this is by taking a sparse number of photos on my phone using a technology called neural radiance fields or nerve for short what's cool about nerve is that it's the opposite of ray tracing so ray tracing what is happening is you're essentially simulating the physics of light and how light reflects off objects and then how that light makes it into the camera and that's how we play a lot of 3d graphics games today although there's a whole set of hacks and shader graph programming stuff we're not going to get into today but for at least for cinematic stuff not so much games quite yet it's still ray tracing and nerve is the opposite of that so if you take a bunch of photos see how it sees how the light landed and you essentially back into what the light bounced off of that gives you geometry that's one way we can take objects with us into digital space which is going to lead to even more creativity I'm going to get to next

you could think of the 3d graphic importation of reality individual space is the very most basic aspect of a digital twin digital twins were originally really pioneered by NASA when they thought of ways that they could create a spaceship representation back on earth for things that were in space and synthesize it into all of the data about that particular object all its physical properties what it's doing and whatnot we're going to have more and more digital twins so just like that gargoyle we're going to have lots of objects in the real world that do that you can scale that up so you can scale that up to not just individual objects but to entire factories and if you can do it in a factory you could do an entire city so smart cities are really about having a digital twin at the scale of the entire city knowing all the physical properties but also the social properties that are going on there why stop at a start smart city you can do a twin earth so in video recently talked about earth two which is a super computer application a supercomputer and an application that they've built to simulate the entire world and do things like climate modeling on it

we can take that data that we've captured and projected back into the real world when I was in Nepal a few months back I could there's a lot of mountains there I couldn't identify all the mountains maybe you didn't realize that that's Mount Everest over there on the side but I was looking at Mount Everest and you could look at all this information using AI to interpret the environment using geolocation putting it all together and telling you actually what's around you in the real world so

just as I'm wearing the avatar of myself right now digital realm is also going to affect the real world right it's everything from when you go and go to a Starbucks now and there's a mobile ordering line it's everything about footing information and augmented reality back into real space the smart cities who were talking the refactoring of all of our highway systems to accommodate autonomous vehicles the fact that we didn't have media rooms in our home so there's a there's a loop that's going on here as more and more things come online it'll affect the real world as the real world is affected by that it will be a positive feedback we've been to digitize and learn more of it okay

so that was us projecting ourselves into digital space us projecting the physical space into digital space now I want to talk about emergence and creativity and the next stage that we're going to for identity so the the idea of emergence is just a simple one at its core which is when you have systems with a certain initial set it's really really hard to predict exactly where it's going to go and it can have lots and lots of different forms

we see the synflox of birds we see it in penrose snowflakes is the example is here the classic example is game of life game of life is interesting because there's a creative aspect to it you choose which cells you want to activate at the beginning and then you get to see what the emergent properties of those cells are a somewhat lesser known property of game of life is that this is actually a touring complete computing platform so you can actually create and in more gates on game of life and build computers that can process in theory any computer program so really interesting to think that with this very very simple system with really only like three basic rules of behavior you could actually build computers in it

so I spend a lot of time thinking about what creativity is there's this old book from about 1937 by Olaf Stable then called star maker and I'm not going to get into the book itself right now but he had the notion that some super advanced civilizations that could exist could actually simulate all the possibility spaces that would actually exist it's an interesting way to think of creativity in that the universe is structured around an infinite number well I don't know if it's infinite or not but a heck of a lot of parameters and variables that you can adjust creativity then could be thought of as kind of finding solutions within this vast parameter space humans tend to be pretty good at it

the founder of harmonics who teaches here at MIT was asked you know how many pieces of music are there and the answer is a lot and if you look at one example which is back sweet number one and you give it 40 bars of 16th notes and three octaves then you get 36 to the power of 640 possible melodies think that's a pretty really very large number and we haven't even accounted for all the different instruments textures songs voices sounds combinations compositions all the things that could possibly turn into music I don't I don't know how many more exponential things would be on that but it's a pretty vast search space for creativity

it was an interesting paper recently in the field of artificial intelligence says it's quite to astrophysics actually where he figured out that you could make some of these symbolic regression algorithms a lot more efficient by identifying which of the paths you can't go down right so if you thought of that as like music well there's probably a lot of music that just is not going to sound good based on some basic properties that we know about music and you could maybe exclude that from the search and they were able to do things in this paper like rediscover relativity all over again by feeding it a bunch of data eliminating a lot of the paths that it could go on and getting down to that constrained search space so I think of creativity sometimes as an efficient process for finding the solutions that are really interesting for people but let's talk about how this is played out inside games

so world of warcraft is a really great key tree dish for a lot of emergent behavior years ago there was this thing called the corrupted blood incident where they accidentally unleashed actually a simulated virus in the world because you catch a disease from this creature in it and then you could spread it to other people they didn't realize that people would teleport back to town really fast spread the disease and basically cause widespread doom what what you're seeing there on the screen is all the skeletons of people in the in the town that died because they got it so you had a really interesting emergent properties there like some people wanted to be super helpful they were like the the doctors telling people you know avoid the zone the pandemic is spreading there but of course you had the players you thought it was really hilarious to see a whole city get destroyed that's just an example of emergence but the important takeaway in this is not just the complexity of something like world of warcraft but all of those variables that you also introduce when you create a social system that's present and you suddenly are throwing humans and all the varied human behaviors that people can have

a lot of where we're going now is allowing humans to enter creative spaces and not just play a progression kind of game with some social features like world of warcraft but express their own creativity which really is about taking who we are as people our identity and extending it into the realm of our self expression what is it that we want to talk about what you see there is one of the most sophisticated cities that has built in Minecraft and that's mid-Journey art where the art itself I'd say is fairly emergent the creativity of it comes from what is the prompts that you want to put into it today we have lots of these worlds that can do various forms of creative emergence it runs from second life which was a real pioneer in this but now fortnight creator mode where people that I hear are already making millions of dollars because they're sharing their revenue to build additional worlds and levels and islands off of the fortnight game you've got road blocks which actually is a programming system and then you've got Minecraft which can be programmed in

so speaking of Minecraft I mentioned earlier that game of life is touring complete well Minecraft is touring complete as well even without getting into the Java programming of the game servers so if you're familiar with Minecraft you probably run into this thing called redstone in it so redstone lets you build things like circuits so as you can build circuits you can build anything so people have built computers in Minecraft they built computers that you can play Tetris on for example and this super interesting one from the last year is a guy took a few months and he built a convolutional moral network inside Minecraft so he actually made it so you could do digit recognition inside Minecraft using simply the features that are built into Minecraft no magical programming in Java underneath using APIs to sort computer information just using redstone structuring things and building circuits allowed this person to create a neural network so he draws on it he kind of has an input pad and you can see where it's receiving the input that he's putting in and on the right the neural network has to do a digit recognition task where which is a very well-worn set of weights and biases that you can just download off the internet now so he's able to construct that moral net and do actual digit recognition so that's a lot of emergence there so

user-generated contents is one of the most interesting and growing areas of games in general this is a this is from a company called Overworld that I don't know that they would describe it this way but in my mind it's turning every game or game world that can exist into its own app store meaning all that creativity that emergence they were just seeing there is it possible to package it up in us in such a way that people can almost buy and sell content that that is built off of the core game system of something else all right

now when you add to that generative AI you're going to have an even further acceleration of the creativity that people are going to be capable of having

one of the folks who announced this recently was Roblox so Roblox is an incredibly popular platform it's 200 million plus people per month are in Roblox they're building things they're building experiences it's everything from professional game development studios now to hobbyists who want to build things and by combining it with generative AI they're going to allow much more rapid creation of game objects so just as I pulled objects from the real world in my reality capture demo you'll be able to do that but also you'll be able to generate things into existence because by the way the more and more reality capture we've done all these generative AI's that are needed for 3D graphics are going to feed off of that data sets that we can create more and more of these objects even without walking around with your phone taking pictures you can come buying user generated content together I guess I wasn't showing the yeah so this is an example of cogeneration inside Roblox happening there as in the way it actually changes the game through text plots so maybe call it fly instead of sit on the ground

so user generated content within a more constrained rule set is affecting games alongside of generative AI so

one problem that emerges when you have people building all kinds of content kind of overwolf style off of a core game system as well maybe stuff doesn't match it doesn't have consistency this is a game that was recently announced just a few that had announced a few days ago that they teamed up with a company called scenario scenario does 2d image generation inspires to do more than 2d in the future but they do 2d image generation specifically for game assets and a big part of it is establishing aesthetic consistency in the in the assets that are being created so when you put that power in the hands of the individual people who are creating content in the game now they can start adding to the core the core game and you're almost crowdsourcing the extension of that game with stuff that actually fits together and isn't completely discontinuous sometimes discontinuous is good you want to have a completely different experience just like when you go to Roblox there are things that look completely different from each other there are other times where you want to have a consistent world and that's where you GC and generative AI are fusing together

I mentioned earlier how we are able to import the real world into digital space well alongside of generative AI another really interesting area of creativity is actually re-skimming reality so this is an example by a digital creator named Balavl Seidou and what he did is he went to his parents' house he took photos using photo grammatory recorded this living room and then what he did is he used a combination control map which is a generative AI along with another piece of software called the Ebsynth you worked with all these things together and what you're actually seeing here is as you go through the room re-styling the rooms the paintings are changing the furniture styles are changing so imagine what happens as this gets faster and faster and it's augmented reality glasses you'll get to the point where maybe you can just re-skim reality around you in real time

emergence is also a property of language models so all this innovation all this excitement around chat gpt right now is kind of fed by two things one is it's a user interface improvement so they did a lot to make it easier to use and the results are better but also the scale up in the number of the number of parameters it's got the size of the GPUs the supercomputing clusters it started to express far more emergent properties in particular you're able to pose questions to it where it kind of has to make decisions and that's super interesting it's gone beyond language so if you think of the most basic language model is an auto complete that just looks at spell correction and then we went to sentence completion now we're starting to probe something about the deep fabric of intelligence and reality itself through these tools or creating these incredibly large scale-doubt intelligent artifacts and intelligence is a word by the way that i'm not exactly sure what it means either amongst one's feathers but i think of it again maybe it's simpler to think of as intelligence is when you find the efficient paths through the search space of possibility now

because a lot of these language models have gotten so much more powerful they're getting applied back to virtual worlds in games as well so this is from a company called hidden door they want to be able to take things like a book a favorite movie series take all of the stuff that makes that world unique put it in a system train up language models on it and then be able to play in those worlds but without kind of going off the rails into all the stuff that happens in a chat GPT so instead of completely unconstrained it's more like the dungeons and dragons experience you're playing with a dungeon master who kind of keeps it a little bit in the box but still gives you a lot of creativity to express yourself there

the experiment from meta recently was could you play diplomacy against a computer and could the computer win so diplomacy is really complicated because you have to actually talk and negotiate with other players in the course of diplomacy but what they so what they needed to do was make the AI trustable able to converse and able to defeat humans and they found that they could actually train a computer that could be humans in the game of diplomacy they think there's other applications like this like use it to help negotiate things in the real world this is that can beat people in a game maybe can do that as well

super interesting example of these AI characters is in a technology called video pre-training so what they did here at OpenAI is they trained a character to go into Minecraft travel around and actually play the game on its own trained on YouTube videos so they fed it all kinds of YouTube videos of here's people playing Minecraft and they taught me I had actually go into Minecraft and be able to confidently play the game and do things like those little shelters for themselves

what about when you combine that back with humans so there's a person who goes by Kodniko online they're a live streamer on Twitch they usually talk about games and all kinds of stuff in their life she presents herself as a Vtuber that means she presents herself normally as a 3D avatar representation of herself so that's her in the upper life that's Kodniko which is a human below her is a virtual being called Lucy which is a character playing Minecraft with her so

we can start to think of these virtual beings as being even virtual friends virtual societies which is precisely this field of generative agents which is now currently the very hot topic if you go online and look up things like auto gpt baby agi god mode things like that but Stanford a few weeks ago published a report called generative agents what they did is they created a virtual world with buildings and stores and homes and things like that they populated it with characters and the characters have to interact with each other using language so they're all trained with language they use gpt as the language system and where they found that these had really interesting emergent properties as these characters are talked to each other for example you could ask one character a question and that might affect the entire society people as they talked with each other about what was going on so that's a lot of what's happening in terms of virtual beings and virtual characters

i want to kind of go think forward into the future now so we talked about virtual identity who we are online we talked about these worlds in which we can create things of minecraft to neural network computers to places that we just go to for fun and when we start thinking about what AI could do for us it leads us in an additional direction though which is can we actually carry out our will online autonomously

and that's where these things like baby agi come in or auto gpt this is an example of god mode so god mode is a web browser based system where using language models you can tell it what your goals are and then it'll formulate a plan and it'll go after on it you kind of approve steps as it goes through process and it'll do things like if you want to order a type of coffee from Starbucks it will find Starbucks near you enter the mobile order and then i guess you'd have to go pick it up because we don't have humanoid androids that can do that part quite yet but maybe maybe someone will link it to uberids and they'll go pick it up and bring it for you so this is an idea that these virtual beings will be able to maybe do things in the world but they could also carry out your will in online spaces as well and i think that is the ultimate extension of the self it's the idea that who we are becomes what we can create but then we have these agents that can go out there to do the things that we want

i've called this the director of imagination era of creativity where we're going to get to the point where you just speak things into existence on computers we can join up with our friends and go through these stories and experiences we'll have digital agents that carry out our will in digital space and we'll be exploring new claims of existence with them

the thing that i kind of want to wrap up with here though is the way this might sort out in the future because there's centralized approaches to this and decentralized sometimes you'll use these terms with respect to blockchain technologies but i'm thinking much bigger than blockchain i'm thinking like the basics of the internet the internet was built as a decentralized network the domain name system is decentralized which means that nobody really owns it and i think there's risks associated with purely centralized AI systems where you have to go through APIs pay someone on a model that was trained the way they want to build it it's good in this current state we're in which is that the massive scale up in compute has been required and i think the applications are wonderful so i don't want to take anything away from these things either but to truly carry out our own will for our own digital identities to carry out what we want to do online i think it's going to require taking back some of these AI models on to our own devices

very fast moving field but you know literally two days ago stability AI the company that did stevele diffusion which is mostly known for their image models today really so language model and it's open source there's the dolly model that was released also fairly recently from data books these are models that you could run on your own so these are a lot of language models that because they're open source you could fine tune it the way you want but you could retrain it you could do anything you have with it and you have you have the ability to experiment on it i mentioned how these are vast computational put intelligent objects we don't have a lot of insight into that object when it's a centralized resource that we can't look at but we'll be able to experiment on it as we gain access to it ourselves

i think that this is going to be the next most important internet battleground it comes down to who owns artificial intelligence will it only be entirely by organizations or will we have access to AI models to direct them as we wish to do so that's the some total of my talk thank you for listening i'm available online you can plug the QR code in or here's my links and i'm happy to take questions

thanks so much Sean we have certainly kind of questions i'll start with you two straightforward one oh first of all it's very word alien um you're indeed a great to see you again another another chat over coffee here except for the classroom um Jon I have great great discussions um you uh you've been in gaming for a long time really really in ARBR games uh if you looked at your first slide there we kind of what is midiverse you saw somebody in the head saying so that's one of the things you think about it very good of what i saw you show requires that what do you think the future is for the actual public immersion i mean how how much of the stake is it now and is that going to be a big part of going forward when you think that really looked like there um

well i don't i don't think headsets are required really amazing experiences that the bottom your imagination and allow you to express creativity sometimes their hindrances and i'll stay away i think the magical number is 60 grams or something so they weigh more than 60 grams people don't want to wear them all the time so that's a little bit of a problem the technology is going to prove down to the point where we can wear something roughly like sunglasses so i think that when that happens that'll change a lot that's going to open up a lot of applications we don't have right now because it's been a become the ability for where it continuously and interpret the world around us visually through that lens whereas right now you'd have to hold up your phone for a lot of augmented reality applications so it's really constrained in terms of what you can actually do with like the mountain climbing out that i showed in the course of that that's augmented reality but you have to think to use it and put your phone up and use it because you're like what is that mountain but if the other version of it would be like inform me about something really interesting in the environments i'm walking through show me that super rare mushroom that's on the ground that i wouldn't even take a note of at all unless the lens was picking it up so that's an application of it but i don't think ARVR hardware is required to go and have experiences online and i think it's a little bit of a mistake to think of the metaverse as purely like an embodied ARVR experience i think just having this shared source telling space and being able to bring up the imagination in the mind is what's actually really important

i love the way you read it by meta or properly defined it to be a cover really great of that as i'm hoping mine will take off yeah i mean we support you in that time um next one is going to AI and then you know we can certainly set a lot of time in the middle talking about or sure to be learned when we talk about it whereas we're gonna bring us here but

sam on was at MIT was like any last week and there's big headlines now from one of the things that said which is they hitting a roadblock large-natured models yeah actually i think you remember when Chechi B.T. everyone was asking all these questions or first came out the thing i wanted to do is give this singularity probe what's going to make you more intelligent and it said oh i'm not intelligent i'm just a chef so it's okay what's going to make you perform back more data and i kept trying to say you know what else no more date and if you talk to an expert in AI then we're going to feel this is often considered the same we're dead now they're saying no more date is not this ass and powder so that's great helping what do you think is going to break the login uh when we eventually do assume you miss a log in um

there's a lot that we don't know about what's going on in open AI it's my earlier point yeah i mean i have cognitive dissonance on on open AI i love everything that they're doing i wish i could see more things what they're doing the um the my understanding is that a lot of the improvements going from gpt 3.5 to four were deemed not simply from pure scale up of compute but through a lot of tiny innovations a lot of hyper parameter tuning we're all we know it's a smaller model than gpt 2.5 and i think that's interesting to think about because to make it run on you know a phone you know we've got a decent amount of compute on this device now an impressive amount of neural compute that pretty much nobody is using at the moment you'll still need a much smaller model to do inference on that device that i think is going to be a super interesting world so there's kind of the two parts of it you got to train a model and you got to and then you want to infer from model there's probably quite a few applications where we could get inference right down to the device so we already do that with these diffusion models for example you can download stable diffusion onto a decent computer whether with not even the very best gpu a reasonable gpu and you can do inference and generate images on your machine training it is a lot more like training the stable diffusion model costs at least into the six figures to provision the gq compute power currently and it depends on computers that have shared memory and very high speed interconnect which means they're basically super computers that are co-located i think a field that i'm very interested in which doesn't seem like it's gotten enough progress yet as well is there a way to distribute the training computation so that you can do effective training with the lower latency across the internet for example i haven't seen anything yet that shows that it can be done but i don't know of any law of physics that says that it counts so i think that someone will probably figure it out and that can be interesting but if we could start to train and big networks that would be super cool in terms of pushing out the ability to train up models because it's a lot easier economically speaking to ask a bunch of iPhone users for example to pay the electric bill for the gpu super computer every time they plug in their phone to recharge which is basically the way that will work as opposed to the very high cost of operating super computers not to mention the capital expenditure that's involved

but from an inference level you're going to get this capability on devices really soon

i mentioned in the talk a company called hidden or Hillary Mason's the CEO of that company her whole thing and i've spoken to her you can find a conversation i had online with her recently her whole thing is actually getting the models much smaller like smaller models can be far more efficient so rather than train them train very expensive models up in the cloud why not get them to a level of simplicity and and that opens up the option to either run it in the cloud if that's the way you want to operate the business model or push it onto device so long answer if your question is eventually a lot of the everyday uses of AI maybe not modeling the the planet scale climate but for a lot of the things that we're going to do like interact with virtual beings and characters and be part of simulated worlds it's going to come get pushed a lot more down to devices probably perpetually economic issues i think that's the frontier as much more track life who knows really where the next robots are in the machine's more intelligence it's deep right it's long the structure is on the right but it is obviously an objective for many it into smaller platforms i think you're at close natural places already we talked about since remember it's pretty actually we're very close to some of the runs on the tiny batter you think about the handbrains combined with the most brains combined with the handbrains eventually and that leads to your other point because one issue there is you've got road loss in every point so if you have a vastly connected server that can send high bandwidth measures all over the place and the brain is it does that much slower speed than higher density helps to everything nearby if you have a thin pipe even though in your day it's still a pipe and it's still combined pretty good probably this is mess in the parallel machine you're going to start to have these road blocks that are going to you know the effects how much you can get layers to layers or it's second to second remesh here is doing cybersecurity with that to really make it secure in good bonuses and you train it up you actually have some privacy over there because it was better rated learning yes yeah but yeah a training framework we actually optimizing all four of those narrowed and it's kind of intriguing so we break the brain up and optimize the whole thing so maybe the way the radio works universal the way we do that other questions I don't think so thank you for the right inspired and I'm sorry about by saying that as we are

so if you are in here you see if there will be this any retention on the floor where you're something else out to the digital realm and I was wondering how social commensiveness was the region of the space building was a social connection is emerging

so if you look at games like World of Warcraft they're inherently social and in fact when I've spoken to some of the designers who were early at World of Warcraft some of them would tell you they were really building a chat room with a lot of cool things you could do around the chat so it is the chat the social connection that people are forming with each other that's the whole core of that and just like you can suddenly get a convolutional neural network by building blocks of redstone and Minecraft and you can get the corrupted blood thing and work in World of Warcraft it's essentially what are the applications of chat right so the applications of chat are essentially you know near infinite so there's there's tons of emergent things and emergent spides nature is that I can't predict all the all the versions of that emergence

if you leave a road blocks that's another interesting case though because sometimes there are so there are people out there that don't fully understand road blocks let me just explain that for a moment so road blocks isn't a game World of Warcraft is a game World Roblox isn't a game it's essentially a fusion of a kind of social media with individual worlds that people can create the closest analogy would be like YouTube for game creators so it's got a platform in which you can discover a world that you want to play in you can create worlds and it forms them in network that allows people to find each other but what's really interesting there is now that you've got a social group there's a lot of social glue within the overall environment of the world blocks so you might love a particular game or world inside Roblox but the fact that you have a bunch of friends that can now come with you it creates this really interesting dynamic which we don't really have in in many other aspects of the internet which is you might be doing something for in one moment with your circle of friends you're three four five ten people that you're hanging out with they're like we're done with this let's jump into this other experience and do that and said so we don't do that with websites we're not like hey let's everybody look at this web page and then go to this other web page next the fact that you've got this activity that you're actually experiencing together is what enables that so there's lots of really interesting emerging social structures that are happening from that

to draw upon the blockchain world this idea of decentralized autonomous organizations a DAO is an IS manifestation of that which is is there a way to create new government systems in their case around the financialization of of governing structures and we could debate the pros and cons of whether that makes sense at all or in particular cases but nevertheless it's a social system a social structure which I think is interesting to look at so as people do more and more of these things it's interesting to think about what happens with the agents that represent us online and then how do we form governance around that and how will our intelligent agents working for us play with each other right so that thing that I showed you of Lucy earlier Lucy is one virtual being but the founder of that company is a guy named Edward Sachi, Thable Simulation his idea is to get 10 million virtual beings participating with each other living their own life they're doing things when you're not watching them you can interact with any of them and kind of learn what they've been up to you can play games with them they can kind of be friend you see you can you could start to think about all the really interesting social structures that are going to emerge from that but again it's going to be emergent we don't know yet we're going to discover it soon

so I'm a busy I'm a 30-time fellow Cornell TAP project at the center of the universe at the moment I'm just doing what you can hear in this state so I don't have any other like Pokemon Govon of governance reality I'm curious and I'm just about where AR might intersect with the physical infrastructures and externalities and how security is incorporated into that tool because I've obviously already had a review of everything AI builds and your conditions are the one right with the universe how do I make sure we really need your externalities because I think there's a famous paper that looks like 30,000 people better true the first level that's called the bundle and the strategy for I think not even players of all so I don't hear some thoughts on this but what are the other potential yeah the usage of engineering consequences you hailed some of that ARF will stand to being in the river here which they are they talk about it this way etc but I'm thinking about most use of these is wouldn't that get a burden because I can be again a pure rare person that had sense it is a game mechanic we'll use our smart phones or and we'll just look there at very large versions as well lots of questions too yeah um

well I'm a little bit terrified of regulation in the sense that I'm I'm concerned that people don't have the ability to think out all the implications and things like regulatory capture I think there are of course real safety issues that are concerned but I also don't want to throw out the potential for all the society including civilizational improvement effects that will that will be a huge net benefit to us so I worry about any rush to regulation than any form um

so when you think about the capabilities of augmented reality I just want to go back to sort of Joe and I's earlier brief conversation around that and maybe double click on it so when I was giving a mushroom example of like detecting the object in the world right now AR is a pull mechanism but it can become a push mechanism what I mean by that is when you play Pokemon Go and you have to pull out your phone and tell it you have to check it and basically so it's it's essentially a check in mechanics so you use it when you think of it which is fine for a lot of applications pull as great like pull is basically most applications that we used to do push applications are really interesting as well because it allows you to consume a massive amount of contextual information data about your environment things fed to you by IoT things fed to you through the cameras around your devices you can have an AI layer sitting there that can make inferences about the environment that you're in filter that up to notification mechanisms that are that you choose right like I don't I don't think anybody wants to be continuously augmented reality spam to bet everything that you're dealing with but if I can show it my choices of that that's going to be really interesting because that that opens up a whole new class of applications just the fact that the device is something you can integrate into your life and use it continuously makes available applications that otherwise you wouldn't be able to get at

but when I talk about projecting our will onto onto the online world through intelligent agents like an auto like the next generation of God mode and auto GPT I'm also thinking about our own agency about interpreting the online world so right now in the centralized version of the world it's really governed by algorithms whose objective function is revenue and EBITF from an organization it's totally fine I'm a capitalist like I get it but I personally want to live in a world where it's up where my online experience is optimized around the objective function that I set so I'm going to set an objective function which is maximize my well being through lunch or something like that and that's all that I care about and if my intelligent agent wants to let me know that it discovered a product or service that I ought to consider paying for because it's looked at all the information available and it's patterned match that based on my criteria to what I want that's an example of projecting our will out into the world and that's going to come back into the world of AR and how we interpret smart cities in this huge huge firehose of data that will be upon us it's got to be something we curate for our own intelligent layer that we've talked about well in the back there first because he freem did you sorry

I remember someone else in love and said that's great I was there maybe it was about 2005 or since I was about two years old and five and I was playing not only about my joker and when I go into the city people are dying and then I started dying and this time what was happening at the time some of us are just players and I put it here I left the game where that I'm that I this game's too hard that you just die where do you go to sit but if people go out of control and people go out of the escape and it was interesting to have a little sample of what might have been looked like as you get but then after playing well for about many years I left the really sourness and it was a very beneficial experience and I learned a lot about life coming from the country I learned about you know I global scale about the right people I don't know about the people's psychology but the game essentially graces the vibe and it's a few months as well between you know having two passions to call it a lion's sport and when that's not enough that it's GPP players and environments to PPP players against each other and you know it's it's really quite dangerous well spits my opinion to authenticate that and it's really wrong and I always looked at my flat and I really think like that I've actually brought something educational something that actually made you feel and you know I love everything but I feel like I'm playing with PC mode and currently all that was really intentional and like always trapped about like if there's a urge between something as cool and it's been addictive as well before but as educational as scientific maybe as Minecraft that that's something that I think like this missing the moon must be today would it be possible to create your cat model and share them on in that game or a great ECBs or snippets of code where other people could engage with the art of creation and learn from that that's something that I'm mainly a robot is a template to do that but yeah I'm not quite sure it's being these both worlds environment my final real question to you know is about you know the rise of the ice is for the agents and checking the tea and gets wonderful I can usually get almost every day for various things and astonishing by every time I get with the desk through the conversation you feel like there's some sentient being that sounds interesting but then it shuts down and it gives me errors and it breaks and then it gets down again and it was starting to move on again so I have feeling that they have these algorithms to like actually bring it back down and I just want to ask you where these are going to come back and society like companies are banning it countries are banning it people are just sitting there I to like you know all the police and you malicious actors are you doing they are working to get the point of society

well I think you you hit the nail on the head and you said malicious actors right so there's a lot of focus on this idea of ex-risk ex-essential risk it's probably not a zero percent chance but the real risk is what humans do with these things so I think it's totally valid to be concerned about that so there's going to be everything from industrial accidents to criminals that that misuse it and do bad things with it so we'll have to first defensive technologies right so you know there's knowing the authenticity of content and who it came from it's going to be very important we already have a deep fake problem so you know it's possible that we'll solve that through cryptographic signing and even blockchain technologies as they are good at showing the provenance of information and content that's an example of something that we're going to need to sort out because you're going to need you're going to need to know yesterday if the person you're communicating with is actually that person so that's an example of technologically speaking how we're going to get to it

I think there's huge huge risks for any country that you know just sort of bans it I think they'll be very disabled when the economy we're going into the reality is these technologies will be available and the way that I think you can actually mitigate some of these risks is open it up open source these things be transparent let people experiment on the weights look at look at software generally speaking so the best software in the world that we have from the security standpoint is open source software because it's continuously exposed to a crowd sourced community that goes and determines what the flaws are so things that are closed up we're relying on them to do all the fixes and then if that's all we have then ultimately you depend on governments to come up with rules that they have to live by then you've got regulatory capture then you've got only the largest of companies can participate in it at all and who knows what the heck they're going to do is they're going to optimize on an objective function that is not what you know you or I would do so I think it does come down to exposing these things to light expose them to the potential for experimentation and allow individuals to work with them of course some individuals are going to misuse it there's there's no question individuals misuse email they misuse cryptocurrency they go in games and have toxic behavior like you were describing so you know humans can suck a lot of the time unfortunately and that's nothing we're going to do is going to change that about people AI fluid and power some of those people to do even more malicious things the net benefit is going to be put the power in the hands of individuals I think that on balance the the good of society will prevail over that so they're running a little bit out of it

I think a lot of the question that are in the chat more or less were answered throughout this conversation but I think one point that adds to the conversation of who to just bring one question about what do you think about that how they either they're going through media that we should all in state book and continually oh should we pause six months I mean that would require a long talk that yeah

the answer is no I think that makes no sense I don't even know how you want to part of that what is what is more powerful than GPT for even me so I don't I think the better would be like let's let's open source some of these things and we're seeing that happening with dolly model etc I don't I don't think there's any putting that genie back in the model like we've got GPUs GPUs are going to enable this and if people can't do it in one place they'll go to the place where they can do it and then you're just shifting the innovation off the board so like I I think we have to invest in the engineering I don't think we should be in the business of telling individual companies or researchers what they can or can work on but I think we do have to hold the people accountable to you know harm that they cause just like any product like we have product liability if someone is damaged by something we hope people accountable to that so that's something that makes sense to me but I just I think I I have been in a lot of this conversations my personal feeling is

regulation is going to come and I'm just because I just think talking to lots of the quote unquote man on the street around this there is a lot of fear that question gets asked existential risk questions get asked and and and I think the fear is real and palpable and it will drive politicians towards reacting to that potentially in a way that isn't productive for innovation and maybe even be a net worsening of safety you know so that worries me a lot the rush the rush regulation the rush to say hey push a button and pause it worries me a lot so the answer is no don't trust the pause button because fear comes from the closeness of the organizations that put this forward right now this not just not just now no I think people see the power of these technologies and we as humans evolve to find agency and everything around us so you know we comes from animals right people find agency in a rock they find agency in a cloud so we're wired for that why because it's a survival advantage it's better to guess that something has agency than not because it might be trying to kill you that's exactly what's happening right now we're we're evolved for this right

I think I'll make one comment and we'll have this answer question but in some sense maybe our intelligent agent what part of it about that I think too could be a protective layer for individuals right but how can we trust our intelligent agent and that could be a great significant jargon it's about that too so maybe it's interesting synergy there great so thank you John thank you thank you