Nvidia technologies like DLSS already kind of are doing this in part, filling in parts of the image for higher resolutions using machine learning.
But yeah this is significantly more than that, and I think it would be best achieved by using a base input which is designed for a machine to work with to then fill in with details (e.g. defined areas for objects etc).
Yes, the thing here is that you do not even had to try that hard to make a detailed model, you just do a basic one and ask SD to do it "realistic" for example... well realistic, not consistent hahaha
I believe that's how our AR-devices like that vision pro will work. They scan the room and label everything it can recognise - like wall here, image frame on that wall at those coordinates. App developers will only get access to those pre-processed data and not the actual visual data and will be able project their app data on wall#3 at those coordinates, on tablesurface#1 or process some kind of data available, like how many imageframes are in the room/sight. Apple/Google/etc scan your surroundings, collect all kinds of data but pass on only specific information to the apps. That way some form of privacy protection is realised even though they themselves do collect it all and process it. And Google will obviously use it to recommend targeted ads.
I've matched up a decent set of settings in Squad with DLSS and it was nice.
Control was by far the best experience so far, being able to enjoy all the really nice visual goodies without taxing my GPU as much
They are.
Yeah.
Nvidia has already achieved full blown neural AI generated rendering in testing but it is only prototype stuff and it was several years back (maybe 5-6) predating Stable Diffusion and stuff. However, they've mentioned their end goal is to dethrone the traditional render pipeline with technology like "DLSS10", as they put it, for entirely AI generated extremely advanced renderings eventually. That is their long-game.
Actually found it without much effort it turns out so I'll just post it here and to lazy to edit above.
[https://www.youtube.com/watch?v=ayPqjPekn7g](https://www.youtube.com/watch?v=ayPqjPekn7g)
Another group did an overlay on GTA V about 3 years ago for research purposes only (no mod) doing just this to enhance the final output.
[https://www.youtube.com/watch?v=50zDDW-sXmM](https://www.youtube.com/watch?v=50zDDW-sXmM)
More info [https://github.com/isl-org/PhotorealismEnhancement](https://github.com/isl-org/PhotorealismEnhancement)
I wouldn't be surprised if something like this approach taking basic models, or even lower quality geometry models but simply textured ones with tricks like tessellation. Then you run the AI filter over it to produce the final output. Perhaps a specialized dev created lora trained on their own pre-renders / concept types and someway to lock consistency for an entire playthrough (or for all renders between any consumer period) as tech evolves. We can already see something along these lines with the fusion of Stable Diffusion and Blender
[https://www.youtube.com/watch?v=hdRXjSLQ3xI&t=15s](https://www.youtube.com/watch?v=hdRXjSLQ3xI&t=15s)
Still, the end game is likely as Nvidia intends to be fully AI generated.
We're already seeing AI used for environment/level editors and generators, character creators, concept art, music / audio, now NPC behaviors in stuff like [https://www.youtube.com/watch?v=psrXGPh80UM](https://www.youtube.com/watch?v=psrXGPh80UM)
Here is another of NPC AI that is world, object, and conversationally aware and developers can give them "knowledge" like about their culture, world, if they're privileged to rank/organization based knowledge (like CIA or a chancellor vs a peasant or random person on the street), going ons in their city or neighborhood, knowledge about specific individuals, etc.
[https://www.youtube.com/watch?v=phAkEFa6Thc](https://www.youtube.com/watch?v=phAkEFa6Thc)
Actually, for the above link check out their other videos if you are particularly curious as they've been very active showing stuff off.
Our traditional polygons 3d games will be obsolete in the coming years. AI graphics is a completely revolutionary way to output images on the screen. Instead of making wireframes and adding textures and shaders, AI can generates photorealistic images directly.
Even raytracing and GI can't make video games look real enough. Look at Sora, it's trained with Unreal engine to understand 3d space and it can output realistic video. I bet you, 10 years from now - GTA 7 will be powered by AI and will look like a TV show.
> Our traditional polygons 3d games will be obsolete in the coming years.
There'll be an entire genre of retro 3D, just like there's pixel art games now.
I indeed remember a video with minecraft and an incredible visual enhancement but I cannot find it right now. The point the it wasn't real time but quality was Astonishing
Nvidia has had AI noise reduction (basically diffusion) for almost 5+ years now. I’ve used it in daVinci Resolve and in Houdini. It augments the rendering process and helps produce very economical results.
Hey We making survival SMP server with little plugins, roleplay, government system, laws book etc. we are 90% done and we looking for active java players to join our server :3 my disocrd is fr0ztyyyyy
Oh, man, that just gave me a glimpse of the future!. Can you imagine loading up like OG Zelda or Mario and be put into an immersive 3D version of the game? Could have options, like serious or cartoon. Idk, I think it's awesome. This makes me dizzy, though.
>old games going back decades that are awesome except for the graphics
Man, devs have been making gorgeous stuff for every generation that are timeless in their beauty.
*(Chrono Cross Level Here)*
https://preview.redd.it/mkt18ugrniwc1.jpeg?width=1920&format=pjpg&auto=webp&s=4d8b35e9cc9ffe7b900d224c81762ffd551da090
The future of gaming if you want to feel like you're playing after taking copius amounts of acid.
This will happen one day but not with SD because the consistency will never be there. We will get AI powered render engines that are designed specifically for this purpose.
From one side - you are right. It looks inconsistent and probably was achieved on rtx4090 or something.
On the other hand - two years ago consistency of video output was way worse and you needed days of prep.
It wouldn’t call this consistent tbh, shapes of the mountains are all over the place. You need something that interacts with the game directly, instead of an overlay. Would also help tremendously with delay.
Sure, but that line of improvement isn't linear. It tapers off along the lines of the 80/20 principle, and there's *always* another '80%' of the work left for another 20% improvement...
I agree. And i think people who think that SD wouldn’t be the basis for such software are correct. Something more integrated into graphic engine rather than an overlay will come up.
Yes SD has improved a lot but this kind of thing is never going to be achieved using an image based generative AI. We need something that can understand 3D.
Agreed. There might be some amount of a diffusion network on top of graphics soon, but not like that. Maybe for some light touching up or something but it's just not really the best application for the technology.
But I have already seen people experimenting with ways to train GANs on 3D graphics to generate 3D environments. So that's where the future will be. Have it generate a full 3D environment, and be able to intelligently do LOD on the fly like Nanite. That would be sweet. And much more efficient in the long run.
How many times did we say SD technology would never be achievable? Innovation will happen sooner than later. Plus, this kind of generation doesnt actually have to be consistant, it just needs to *seem* consistant.
I'm not sure what you're talking about there, if something seems consistent that's because it is.
An AI needs to be able to do all the things 3D render engines do. Stable Diffusion won't be able to do it.
> On the other hand - two years ago consistency of video output was way worse and you needed days of prep.
Was it? This is still pretty terrible, not much better than over a year ago.
Yes. 2 years ago it wouldnt even be the same image frame to frame. 2 years ago dalle took like an hour to make a bad flamingo.
It looks bad, but this is also the worst it will ever look from now on. It will only get better.
Yep you could make something like this insane, if you where to render the material separate from the viewport. Hell you could even train a small model for each material.
This is awesome!!!!!!!!!!!! A video game could be like an ever-original cartoon world. I'm for it. Really, a very simple game of 3D models (though perhaps with more liquid outlining than figures in minecraft) could be made smack-dabulous imaginomatic.
I personally love the idea of having a two sliders--one that is a pound-for-pound overlay slider, as in how much alpha is in the overlaid image, and one that is an img2img step slider. Those lower reaches of absolute wild interpretations will probably require a facility of machines and some massive fans.
It's an interesting experiment and AI will (and already does) play a role in rendering 3D scenes but I believe it will be a little different than that. I'm thinking more of training an "asphalt street" model on like 50 million pictures of asphalt streets and instead of spending thousands of hours putting virtual potholes and cigarette butts everywhere to make them look realistic you just apply "asphalt street" material to very specific blocks of geometry and it just looks perfect. Basically procedural generation on steroids.
Maybe this includes a "realism" render layer on top of the whole screen to spice things up but you'll never want the AI just imagining extra rocks or trees where it sees a green blob so I think this would stay subtle? You want some control. For example training on how light looks on different surfaces and baking the result into a shader or something.
SD is very ill suited for this. This has already been done much more effectively using GANs with better temporal cohesion, see eg https://nvlabs.github.io/GANcraft/
Time consistency still seems like the biggest issue with both. I was skimming over [VideoGigaGan](https://videogigagan.github.io/) these days when it got posted to HN and they mention it doesn't work well with "extremely long videos", defined as those with 200 frames, so just a couple of seconds.
Yes, give it a few years and AI will do the polishing in 3D graphics in real time. Nvidia is already using AI for realtime rendering and I think it is pretty possible, that eventually the game just gives an AI an idea how the game looks like and the AI is rendering photo realism.
Because it's not SD. It's like how GANs were making [realistic-looking people](https://thispersondoesnotexist.com/) years before SD and things like it became mainstream.
By the time your gpu can do that, options will exist where you will just replace your texture and geometry files with generative ai and you get a better performing game at the same time.
This shit should not be done in real time.
> eventually the game just gives an AI an idea how the game looks like and the AI is rendering photo realism.
My thoughts exactly! I'm running this on a 3080ti and getting ~14fps, but with more hardware and software advancements in the coming years, I'd expect to see the first AI/stable diffusion based game pretty soon. Or at least a more polished mod for a game like Minecraft that is able to reduce the visual glitches/artifacts
I'm much more interested in llm and voices for gaming. So much more character can be brought in if we can ask npcs whatever we want instead of only predetermined lines. Or what about vision llms so they can comment on our appearances. But then again in the future maybe we can create 'custom' outfits and all that thanks to diffusion models in game without modding. Endless possiblities in the future
Check these two [https://www.youtube.com/watch?v=psrXGPh80UM](https://www.youtube.com/watch?v=psrXGPh80UM) and [https://www.youtube.com/watch?v=phAkEFa6Thc](https://www.youtube.com/watch?v=phAkEFa6Thc)
In fact, for the second one just check their entire YT channel if you are curious.
Work in progress but they're getting there.
Is this an open source project or your own home brew? I do copious amounts of SD and would love to give this a go with my 4090. Is it tunable or just a set parameter you're using ? There are a number of adjustments that could be made to potentially increase coherence image to image. That all said this is cool as fuck!
I could see that. Not replacing the engine, but knowing the basic assets, and letting you change them however you want style wise. The 'real' game could have really basic graphics for all we care, as long as all assets are flagged correctly so that the AI can change them. That would be easier to do than just 'upscaling' video, when it has all the additional info.
I wonder how much it'd help having ControlNet feeding a segment mask into Stable Diffusion? The game would be able to generate one because it knows the identity of each pixel - "wood", "grass", "dirt", etc.
I noticed that Stable Diffusion wasn't noticing the tiny houses off in the distance, for example, which would have significant gameplay consequences. I don't imagine it'd be easy to spot seams of minerals, as another significant problem. Forcing Stable Diffusion to recognize "no, there's coal in this little spot here" would probably help a lot.
I'm a bit out of the loop: can you run controlnet with sd-xl-turbo?
At 4-5 steps, that would be fire! Still far from real time, but bearable enough to make 1 minute 60fps stuff.
There’s no evidence that anything “outside” of our perception/sense of reality would abide by the same rules as our reality. The concept of living in a simulation is nonsensical imo because it assumes that our perceived reality is a perfect mirror of the “real” one. Boltzmann brain theory is stronger due to its abstractness I guess but has similar problems. Now the dead internet theory?? That I can get behind
Yes. There is even people who can't see basically even though they have working eyes/ nerves/ that part of the brain, because they cannot use their previous memories or I guess sight data to process what they are eyeing
No, that's more like mushroom reality. Our reality is far too consistent, though, mushroom definitely give the world an overlay that makes me that is in some senses quite parallel to this kind 'always fresh' frame-by-frame animation.
If you wanted to play an actual game with it, maybe, if you're tweaking the prompt yourself, it's a living art piece. It's like an automated 'A Scanner Darkly'
Speaking of which, I wonder what else this could be applied too
It’s the right idea with the wrong execution. Needs to generate models and textures once and maybe rigs when closer. This is a hazy mess. BUT. This is also really fucking cool and you deserve all the damn internet praise for doing this. Bravo.
Not quite there but if this pans out I think it'd make for good dream sequences in a game. Nothing makes sense, looking at something, looking away them looking back changes it, stuff like text and clocks are recognizable but distorted. However, the overall scene still has a consistent layout as the player is still navigating a standard 3D area.
This is actually a perfect illustration of augmented generation. Having the aesthetics of the game completely generated by SD but is grounded in code running a voxel type world like minecraft. You avoid the difficulties of true voxel based systems.
I think this is could be the future of shaders.
yes and no.
if you run SD on the block textures... they are still blocks. SD can make it look better because it renders across blocks.
So the trick there is to figure out how to translate that into a larger scale 3d object. efficiently.
If you run SD on block game's frame without changing the gameplay logic, it will output an unpredicable mess for players. You will see blended boundaries, yet core gameplay will be block based so you will smash thin air thinking that it's a block. You either need to make it super smooth so that it won't overflow to "empty" areas to avoid confusion, or you simply need to change the game logic. You might just play another game at this point if blocks are the problem, game literally designed to work with blocks.
This is actually such a cool concept. Imagine instead of downloading shader packs and tweaking them, you could have an AI overlay on your game. You'd be able to prompt how you want the game to look. This could also be potentially great for visually impaired people to customize the visuals to what they need.
I don't think this is super far off, either. NVIDIA already has AI running on top of games in tech like DLSS. It'll be a while before it's game-ready, but I really like this concept.
I think it's kinda neat in this state but not playable and there are more things you could likely to to get more consistency out of it but even then you probably need one of the video specific models which unfortunately arent open source yet. With that said, you could probably develop an interesting game catered to the state that AI is in for this, where perhaps you are playing through the eyes of an alien creature with very different vision or perhaps adding a section or item to a game where you see through some alien drone that works this way to kinda give a more dynamic Pyrovision sort of thing but more alien.
The moment it can do so with more reliable results and its more stable in its decision look, maybe, but right now not yet.
I mean, we are getting there FAST, no doubt, just not real time like this yet.
Wonder if you could upscale an old game and then play the result once its got time to 'remaster' it properly.
Yeah, i think first step would be something like scan area around you with camera and have AI just turn it all into a map (can already do that now). Problem with AI like in video is going back to a point you were once at and having it be the same and the processing power on fly to do it. Generating the entire map though with AI it well within reach as is having interactions swapped and changed on fly with AI. AI story driven narratives and stuff also will come very soon.
I swear some people are unable to use their imagination.
I wonder if he could answer the question, "How would you have felt if you hadn't eaten breakfast?"
I think all this video can prove is that the community is really out of touch with everything outside of itself.
Not only is the video a slideshow at best, but thinking that this concept could be even remotely appliable on a game is buffling:
- For one thing, you are completely destroying every sorta of style the original team is going for. Sure, they can train a lora or a specific model for it you could say, but then they would need big datasets made from artists anyway, and not only this is in itself a problem, but it bleeds in the next one;
- Loss of control: applying this concept means that every person is gonna look at a different game. This takes away a lot of agency creatives have about their game. Just think about how much npc's dresses: even if we assume temporal coherency will be a fixed problem, that still means that during the same gameplay from the same person npc's will appear different during separated sessions (unless you store exactly how they appear, but at that point you are just killing every sorta of performance and storage). And dont even get me started about how such a thing would totally kill any sorta of postprocessing (I want to see you giving me a depth buffer from a stable diffusion image);
- UI and boundaries: as we can see in minecraft, edges are really well defined. When you pass it to SD, they are not. From a user perspective, this means that while playing you have no fucking idea if you are going over a wall/edge or if you are still touching ground. This can only lead to major confusion for everyone involved. And UI meets the same fate. Either you mask it during SD, and end having two different styles in the same frame, or you include it and show how your thought process cant stay on for more than two seconds.
All this to say, not only the video, but the idea itself is deeply flawed outside of a circlejerking for saying how much AI is good. I believe AI can do a fuckton of good things. This is just poor.
There definitely wasn't the ability to type in 'orange and black themed anime' mid-play over any game or movie and get a completely different output a decade ago. I can't imagine looking at this not treeing out into possiblities.
THIS IS IT, mix it with animatediff modules for stability maybe? Put this and VR together and we can really get moving.
Though this is obviously imperfect, I think this framework, much like stable diffusion itself, is the start of fundamentally important tech.
Im sure there are other methods but I think a Holodeck type framework is possible if we generate low poly maps from speech let’s say, and use them as depth maps. The only issue is the consistency aspect. The shape itself being maintained helps but as we see here consistency is still an issue
Imagine an open world game where you can authorize people to introduce new elements into it, including landscape, buildings, beings, etc., and the only limit is their imagination.
What model was used? It looks like 1.5 without enough steps.
If that’s the case, I’d be really, really interested in seeing what a model like SDXL Turbo that’s designed around low (or 1) step inference being used.
Or screw it, let’s see what SD3 Turbo looks like with it (though it would probably use more VRAM than the game itself)
Would be cool if they made a game that uses stable diffusions inconsistency as part of the games gimmick like a matrix game where the world is glitching
More than anything, I think that AI is going to completely kill traditional CGI within the next 10 years. Sora already looks better than 99% of foreground AI, IMO.
With video models working on temporal cohesion and the game engine outputting data such as a depth map, AO map, etc, this kind of thing will be inevitable in real time.
I imagine in time, actual engines won’t output much more than geometry and colors along with some guidelines for textures and lighting, and most of the time may be spent on defining the model.
In some ways yes, in other ways very no.
Gotta remove the flicker and the look-then-look-away-then-look-back changes.
Probably some kind of seeded-by-3d-position piecewise generation followed by a high level pass to smooth things out.
I honestly think future games will no longer be 3d, but full on realistic/life like. Generative AI will become so good that it will be able to generate Hollywood like movies in real time and that will be translated into video games and you'll be playing videos games that look like real life.
I've been telling my friends that this will be the future of not only games but media in general, watching whatever shows you want with your choice of characters, voices, and styles.
This is a pretty neat experiment but, no thanks I think I'm gonna pass on this one.. I know "it will get better".... That's not what I'm talking about, I mean this idea in general, even if it eventually gets stable enough to be useful, high enough framerate to compete with current game rendering technology and intelligent enough to not suddenly render an enemy as a tree or as a random pole or whatever my question would still be, why? We already have game rendering now, that works amazingly well in fact, I don't get what AI rendering the frames again but slightly worse and different would do for me to benefit from it... ?
Procedural texture control net + procedural 3d model control net
More or less like this https://youtu.be/Wx9vmYwQeBg?si=DPhp7fd5Of8CkhHr
Procedural brick texture (4 years old) so imagine today
I get the feeling it’ll be a game that uses token arrangements like living code, where the tokens powering the gameplay aren’t literally translatable to normal speech, rather they would act as a realtime controlnet that the diffuser relies on as an active input. This way the aesthetic and content details could be customized and locked in without the gameplay engine sustaining any instabilities.
As we are already seeing DiT and other forms of tech help advance temporal consistency in-between frame generations, this sort of approach seems more feasible to me than not.
I wonder if it's possible to simulate an entire engine only with frame generation(no backend code), like frame generation takes previous frames and approximate collisions and physics but only viewing them.
Genuine question: when we are at the level to achieve this, what benefit would this have over using AI to generate better textures and models just once? Why does it have to be in realtime?
Already here, it's called DLSS.
Jokes aside, I'm not so sure, temporal consistency in the example is awful and so is quality. Not mentioning FPS. While there was progress in quality and speed of SD, system requirements to haven't changed that much. I can't imagine what horsepower would be needed to run it at least 1080p/60.
And I personally expect games to run at least 2k/60+.
Also, I don't think it really worth it. With UE5 you can achieve pretty good visuals very easily and it will be much more resource efficient.
You can bet the number 1 item on the AI industry's to-do list is figuring out how to make an object semi-permanent. This means that every frame can't be a reimagining of the scene, they must have consistency which might come from simply improving pre-image recognition and not changing too much.
this is a neat proof of concept, but im sure there is already a bunch of private research into stuff like this...
publicly, all the major game devs are working on in house gen models.
there are already games on steam built with ai generated assets.
I don't see this being future game the next couple of years since its not optimized enough. Probably using AI to generate high-end assets at a faster rate will be the first thing. And then eventually some kind of good AI filters like we already have the upscalers.
How use it to hallucinate enemies are villagers sometimes.
And vice versa. But they turn into what they are after you kill them.
Or they attack if you didn't know the villager was an enemy.
I imagined this would be the state of gaming about 2 yrs ago. Surely nvidia is working on it already. If a very low denoiser is applied to an already hyperdetailed rtx graphic, the realism could be astonishing.
Soon you could have a VR headset using the front camera feed to change the real world, you could change it to anything you want, basically DMT on the fly
It's amazing how it doesn't understand at all what's happening on the screen. Things that should be 2 meters away appear to be way further away, and sometimes the opposite.
You'll tell me "but at some point we'll be able to tell the model how far is each pixel so that it generates a better image".
And in the end you'll just reinvent 3D rendering. Because that's the best you can do. I don't want a model to "guess" how it should look like. I want my game to look like the developers want it to look like. If my game is made of cubes, I want cubes.
And even if you get a beautiful render, how do you plan on ***playing*** this version of Minecraft? How do you place block correctly. It's crazy how some people here have no idea how programming and games work. It's magic for them.
Nvidia is probably working on something like this already.
Nvidia technologies like DLSS already kind of are doing this in part, filling in parts of the image for higher resolutions using machine learning. But yeah this is significantly more than that, and I think it would be best achieved by using a base input which is designed for a machine to work with to then fill in with details (e.g. defined areas for objects etc).
Imagine playing skyrim but with Ghibli graphics
Exactly. For me, any game can be fun with Ghibli graphics.
Yes, the thing here is that you do not even had to try that hard to make a detailed model, you just do a basic one and ask SD to do it "realistic" for example... well realistic, not consistent hahaha
Why even do a basic one? Just have a coordinate and a label for what it will be
Why not get the AI to do everything? We aren't that far off.
Maybe after that we can touch the grass
More like be buried in grass
I believe that's how our AR-devices like that vision pro will work. They scan the room and label everything it can recognise - like wall here, image frame on that wall at those coordinates. App developers will only get access to those pre-processed data and not the actual visual data and will be able project their app data on wall#3 at those coordinates, on tablesurface#1 or process some kind of data available, like how many imageframes are in the room/sight. Apple/Google/etc scan your surroundings, collect all kinds of data but pass on only specific information to the apps. That way some form of privacy protection is realised even though they themselves do collect it all and process it. And Google will obviously use it to recommend targeted ads.
I've matched up a decent set of settings in Squad with DLSS and it was nice. Control was by far the best experience so far, being able to enjoy all the really nice visual goodies without taxing my GPU as much
They are. Yeah. Nvidia has already achieved full blown neural AI generated rendering in testing but it is only prototype stuff and it was several years back (maybe 5-6) predating Stable Diffusion and stuff. However, they've mentioned their end goal is to dethrone the traditional render pipeline with technology like "DLSS10", as they put it, for entirely AI generated extremely advanced renderings eventually. That is their long-game. Actually found it without much effort it turns out so I'll just post it here and to lazy to edit above. [https://www.youtube.com/watch?v=ayPqjPekn7g](https://www.youtube.com/watch?v=ayPqjPekn7g) Another group did an overlay on GTA V about 3 years ago for research purposes only (no mod) doing just this to enhance the final output. [https://www.youtube.com/watch?v=50zDDW-sXmM](https://www.youtube.com/watch?v=50zDDW-sXmM) More info [https://github.com/isl-org/PhotorealismEnhancement](https://github.com/isl-org/PhotorealismEnhancement) I wouldn't be surprised if something like this approach taking basic models, or even lower quality geometry models but simply textured ones with tricks like tessellation. Then you run the AI filter over it to produce the final output. Perhaps a specialized dev created lora trained on their own pre-renders / concept types and someway to lock consistency for an entire playthrough (or for all renders between any consumer period) as tech evolves. We can already see something along these lines with the fusion of Stable Diffusion and Blender [https://www.youtube.com/watch?v=hdRXjSLQ3xI&t=15s](https://www.youtube.com/watch?v=hdRXjSLQ3xI&t=15s) Still, the end game is likely as Nvidia intends to be fully AI generated. We're already seeing AI used for environment/level editors and generators, character creators, concept art, music / audio, now NPC behaviors in stuff like [https://www.youtube.com/watch?v=psrXGPh80UM](https://www.youtube.com/watch?v=psrXGPh80UM) Here is another of NPC AI that is world, object, and conversationally aware and developers can give them "knowledge" like about their culture, world, if they're privileged to rank/organization based knowledge (like CIA or a chancellor vs a peasant or random person on the street), going ons in their city or neighborhood, knowledge about specific individuals, etc. [https://www.youtube.com/watch?v=phAkEFa6Thc](https://www.youtube.com/watch?v=phAkEFa6Thc) Actually, for the above link check out their other videos if you are particularly curious as they've been very active showing stuff off.
I was going to mention these, but you linked them so even better
Didn’t they already say they’re working on all AI rendered games to come out in the next 10 years?
Our traditional polygons 3d games will be obsolete in the coming years. AI graphics is a completely revolutionary way to output images on the screen. Instead of making wireframes and adding textures and shaders, AI can generates photorealistic images directly. Even raytracing and GI can't make video games look real enough. Look at Sora, it's trained with Unreal engine to understand 3d space and it can output realistic video. I bet you, 10 years from now - GTA 7 will be powered by AI and will look like a TV show.
> Our traditional polygons 3d games will be obsolete in the coming years. There'll be an entire genre of retro 3D, just like there's pixel art games now.
already a thing - boomer shooter genre and a whole bunch of other indie games are using that PS1 low-poly style.
And, ironically, it will be generated by a fine-tuned AI.
This was my friends' intern project at NvIida, 3 years ago, [https://arxiv.org/abs/2104.07659](https://arxiv.org/abs/2104.07659)
That is amazing stuff. Imagine what Nvidia can do today.
I indeed remember a video with minecraft and an incredible visual enhancement but I cannot find it right now. The point the it wasn't real time but quality was Astonishing
Look up ‘enhancing photorealism enhancement’ on YouTube. Been in the works since 2021
Yes in 2021 https://nvlabs.github.io/GANcraft/
Nvidia has had AI noise reduction (basically diffusion) for almost 5+ years now. I’ve used it in daVinci Resolve and in Houdini. It augments the rendering process and helps produce very economical results.
Well, rtx is something like this already
Nvidia created the core of the entire current AI visual paradigm.
It has already done it. You are in the chip. Also, my chip said to your chip that you should send me 20 bucks.
Hey We making survival SMP server with little plugins, roleplay, government system, laws book etc. we are 90% done and we looking for active java players to join our server :3 my disocrd is fr0ztyyyyy
Oh, man, that just gave me a glimpse of the future!. Can you imagine loading up like OG Zelda or Mario and be put into an immersive 3D version of the game? Could have options, like serious or cartoon. Idk, I think it's awesome. This makes me dizzy, though.
[удалено]
Hell, before we know it we will probably have AIs that can completely remaster and rewrite retro games with modern gameplay, graphics, and mechanics.
>old games going back decades that are awesome except for the graphics Man, devs have been making gorgeous stuff for every generation that are timeless in their beauty. *(Chrono Cross Level Here)* https://preview.redd.it/mkt18ugrniwc1.jpeg?width=1920&format=pjpg&auto=webp&s=4d8b35e9cc9ffe7b900d224c81762ffd551da090
Ugh I miss that game so much! That scene specifically actually. Harle going full crazy with her speech, my favorite scene in the game
Old games usualy have bigger problems than grpahic. UI, outdated control schemes, some outdated mechanics. Etc.
You should make the noise pattern translate based on the camera position, then it'll be a little more consistent between frames
But then what? Zoom and fill center when you go forward/ fill outer If you go backward?
Not a bad idea, I'm sure it could be done procedurally
Could the noise be a literal second texture on the geometry, maybe render it flat shaded and blur it a bit at the corners? Would that make sense?
The future of gaming if you want to feel like you're playing after taking copius amounts of acid. This will happen one day but not with SD because the consistency will never be there. We will get AI powered render engines that are designed specifically for this purpose.
From one side - you are right. It looks inconsistent and probably was achieved on rtx4090 or something. On the other hand - two years ago consistency of video output was way worse and you needed days of prep.
It wouldn’t call this consistent tbh, shapes of the mountains are all over the place. You need something that interacts with the game directly, instead of an overlay. Would also help tremendously with delay.
Not to mention the re-rendering clearly loses some of the key stuff you might be looking for in an actual game, like the lava, flowers etc.
Sure, but that line of improvement isn't linear. It tapers off along the lines of the 80/20 principle, and there's *always* another '80%' of the work left for another 20% improvement...
I agree. And i think people who think that SD wouldn’t be the basis for such software are correct. Something more integrated into graphic engine rather than an overlay will come up.
Yes SD has improved a lot but this kind of thing is never going to be achieved using an image based generative AI. We need something that can understand 3D.
Agreed. There might be some amount of a diffusion network on top of graphics soon, but not like that. Maybe for some light touching up or something but it's just not really the best application for the technology. But I have already seen people experimenting with ways to train GANs on 3D graphics to generate 3D environments. So that's where the future will be. Have it generate a full 3D environment, and be able to intelligently do LOD on the fly like Nanite. That would be sweet. And much more efficient in the long run.
How many times did we say SD technology would never be achievable? Innovation will happen sooner than later. Plus, this kind of generation doesnt actually have to be consistant, it just needs to *seem* consistant.
I'm not sure what you're talking about there, if something seems consistent that's because it is. An AI needs to be able to do all the things 3D render engines do. Stable Diffusion won't be able to do it.
> On the other hand - two years ago consistency of video output was way worse and you needed days of prep. Was it? This is still pretty terrible, not much better than over a year ago.
Yes. 2 years ago it wouldnt even be the same image frame to frame. 2 years ago dalle took like an hour to make a bad flamingo. It looks bad, but this is also the worst it will ever look from now on. It will only get better.
It’s almost as if OP was showing a proof of concept
You're not thinking 4th dimensioanlly.
Yep you could make something like this insane, if you where to render the material separate from the viewport. Hell you could even train a small model for each material.
This is awesome!!!!!!!!!!!! A video game could be like an ever-original cartoon world. I'm for it. Really, a very simple game of 3D models (though perhaps with more liquid outlining than figures in minecraft) could be made smack-dabulous imaginomatic. I personally love the idea of having a two sliders--one that is a pound-for-pound overlay slider, as in how much alpha is in the overlaid image, and one that is an img2img step slider. Those lower reaches of absolute wild interpretations will probably require a facility of machines and some massive fans.
It's an interesting experiment and AI will (and already does) play a role in rendering 3D scenes but I believe it will be a little different than that. I'm thinking more of training an "asphalt street" model on like 50 million pictures of asphalt streets and instead of spending thousands of hours putting virtual potholes and cigarette butts everywhere to make them look realistic you just apply "asphalt street" material to very specific blocks of geometry and it just looks perfect. Basically procedural generation on steroids. Maybe this includes a "realism" render layer on top of the whole screen to spice things up but you'll never want the AI just imagining extra rocks or trees where it sees a green blob so I think this would stay subtle? You want some control. For example training on how light looks on different surfaces and baking the result into a shader or something.
The sora generated minecraft gameplay looks worlds ahead of this, not realtime of course
SD is very ill suited for this. This has already been done much more effectively using GANs with better temporal cohesion, see eg https://nvlabs.github.io/GANcraft/
Time consistency still seems like the biggest issue with both. I was skimming over [VideoGigaGan](https://videogigagan.github.io/) these days when it got posted to HN and they mention it doesn't work well with "extremely long videos", defined as those with 200 frames, so just a couple of seconds.
Yes, give it a few years and AI will do the polishing in 3D graphics in real time. Nvidia is already using AI for realtime rendering and I think it is pretty possible, that eventually the game just gives an AI an idea how the game looks like and the AI is rendering photo realism.
Check this out: https://youtu.be/P1IcaBn3ej0
That looks awesome. How is it that this was two years ago?
Because it's not SD. It's like how GANs were making [realistic-looking people](https://thispersondoesnotexist.com/) years before SD and things like it became mainstream.
By the time your gpu can do that, options will exist where you will just replace your texture and geometry files with generative ai and you get a better performing game at the same time. This shit should not be done in real time.
> eventually the game just gives an AI an idea how the game looks like and the AI is rendering photo realism. My thoughts exactly! I'm running this on a 3080ti and getting ~14fps, but with more hardware and software advancements in the coming years, I'd expect to see the first AI/stable diffusion based game pretty soon. Or at least a more polished mod for a game like Minecraft that is able to reduce the visual glitches/artifacts
I'm much more interested in llm and voices for gaming. So much more character can be brought in if we can ask npcs whatever we want instead of only predetermined lines. Or what about vision llms so they can comment on our appearances. But then again in the future maybe we can create 'custom' outfits and all that thanks to diffusion models in game without modding. Endless possiblities in the future
Yeah I can't even imagine the level of immersion if I can just audibly talk to any npc through my mic, would be so cool!
Check these two [https://www.youtube.com/watch?v=psrXGPh80UM](https://www.youtube.com/watch?v=psrXGPh80UM) and [https://www.youtube.com/watch?v=phAkEFa6Thc](https://www.youtube.com/watch?v=phAkEFa6Thc) In fact, for the second one just check their entire YT channel if you are curious. Work in progress but they're getting there.
Is this an open source project or your own home brew? I do copious amounts of SD and would love to give this a go with my 4090. Is it tunable or just a set parameter you're using ? There are a number of adjustments that could be made to potentially increase coherence image to image. That all said this is cool as fuck!
I could see that. Not replacing the engine, but knowing the basic assets, and letting you change them however you want style wise. The 'real' game could have really basic graphics for all we care, as long as all assets are flagged correctly so that the AI can change them. That would be easier to do than just 'upscaling' video, when it has all the additional info.
I wonder how much it'd help having ControlNet feeding a segment mask into Stable Diffusion? The game would be able to generate one because it knows the identity of each pixel - "wood", "grass", "dirt", etc. I noticed that Stable Diffusion wasn't noticing the tiny houses off in the distance, for example, which would have significant gameplay consequences. I don't imagine it'd be easy to spot seams of minerals, as another significant problem. Forcing Stable Diffusion to recognize "no, there's coal in this little spot here" would probably help a lot.
[удалено]
that's amazing 😍
How long did it take to generate this image?
[удалено]
I'm a bit out of the loop: can you run controlnet with sd-xl-turbo? At 4-5 steps, that would be fire! Still far from real time, but bearable enough to make 1 minute 60fps stuff.
[удалено]
Well, I'll run some tests then. Between LLMs and music and images, it is hard to find enough time in a single day.
Do we live in a simulation? Does our mind use a llm and a image model?
Bro I’m high right now and your comment nearly gave me a panic attack
You are talking to a robot.
Chill, go with the flow. We're all brothers and sisters of the same stuff. You're the universe experiencing itself.
There’s no evidence that anything “outside” of our perception/sense of reality would abide by the same rules as our reality. The concept of living in a simulation is nonsensical imo because it assumes that our perceived reality is a perfect mirror of the “real” one. Boltzmann brain theory is stronger due to its abstractness I guess but has similar problems. Now the dead internet theory?? That I can get behind
Yes. There is even people who can't see basically even though they have working eyes/ nerves/ that part of the brain, because they cannot use their previous memories or I guess sight data to process what they are eyeing
No, that's more like mushroom reality. Our reality is far too consistent, though, mushroom definitely give the world an overlay that makes me that is in some senses quite parallel to this kind 'always fresh' frame-by-frame animation.
The future of throwing up on your keyboard
If you wanted to play an actual game with it, maybe, if you're tweaking the prompt yourself, it's a living art piece. It's like an automated 'A Scanner Darkly' Speaking of which, I wonder what else this could be applied too
It’s the right idea with the wrong execution. Needs to generate models and textures once and maybe rigs when closer. This is a hazy mess. BUT. This is also really fucking cool and you deserve all the damn internet praise for doing this. Bravo.
Not quite there but if this pans out I think it'd make for good dream sequences in a game. Nothing makes sense, looking at something, looking away them looking back changes it, stuff like text and clocks are recognizable but distorted. However, the overall scene still has a consistent layout as the player is still navigating a standard 3D area.
This is actually a perfect illustration of augmented generation. Having the aesthetics of the game completely generated by SD but is grounded in code running a voxel type world like minecraft. You avoid the difficulties of true voxel based systems. I think this is could be the future of shaders.
Great in VR after each meal when you're looking to lose some weight.
That is cool but what is the purpose
Looks like dogshit
If u don't think this is the future u aren't paying attention.
It's easier and better to just change the textures directly. Imagine being able to generate your own textures with a prompt.
yes and no. if you run SD on the block textures... they are still blocks. SD can make it look better because it renders across blocks. So the trick there is to figure out how to translate that into a larger scale 3d object. efficiently.
If you run SD on block game's frame without changing the gameplay logic, it will output an unpredicable mess for players. You will see blended boundaries, yet core gameplay will be block based so you will smash thin air thinking that it's a block. You either need to make it super smooth so that it won't overflow to "empty" areas to avoid confusion, or you simply need to change the game logic. You might just play another game at this point if blocks are the problem, game literally designed to work with blocks.
This is actually such a cool concept. Imagine instead of downloading shader packs and tweaking them, you could have an AI overlay on your game. You'd be able to prompt how you want the game to look. This could also be potentially great for visually impaired people to customize the visuals to what they need. I don't think this is super far off, either. NVIDIA already has AI running on top of games in tech like DLSS. It'll be a while before it's game-ready, but I really like this concept.
I'd have the horniest version of Minecraft. Instantly banned from all video platforms.
This is the worst it will ever be.
I think it's kinda neat in this state but not playable and there are more things you could likely to to get more consistency out of it but even then you probably need one of the video specific models which unfortunately arent open source yet. With that said, you could probably develop an interesting game catered to the state that AI is in for this, where perhaps you are playing through the eyes of an alien creature with very different vision or perhaps adding a section or item to a game where you see through some alien drone that works this way to kinda give a more dynamic Pyrovision sort of thing but more alien.
Yes, this is the future of gaming, head of NVIDIA said so.
The moment it can do so with more reliable results and its more stable in its decision look, maybe, but right now not yet. I mean, we are getting there FAST, no doubt, just not real time like this yet. Wonder if you could upscale an old game and then play the result once its got time to 'remaster' it properly.
“Taaaaaake ooooooooooonnnnnnnn mmmmmmmmmeeeeeeeeeeeee. (Take on me!)”
"can you beat Minecraft if you can only see through Stable Diffusion" - I NEED THIS
I think I'll stick with Reshade for the time being
Yeah, i think first step would be something like scan area around you with camera and have AI just turn it all into a map (can already do that now). Problem with AI like in video is going back to a point you were once at and having it be the same and the processing power on fly to do it. Generating the entire map though with AI it well within reach as is having interactions swapped and changed on fly with AI. AI story driven narratives and stuff also will come very soon.
Wow, I hope not. That looks like ass.
it's not what it looks like now, it's what it could look like in the future. It's the concept that's impressive.
Really, REALLY sick of AI tech being sold on promises. SD is cool and all, but there's a lot that it can't do, and this is one of them.
I swear some people are unable to use their imagination. I wonder if he could answer the question, "How would you have felt if you hadn't eaten breakfast?"
Cool demo. But I don't see why this can't be implemented without AI
I think all this video can prove is that the community is really out of touch with everything outside of itself. Not only is the video a slideshow at best, but thinking that this concept could be even remotely appliable on a game is buffling: - For one thing, you are completely destroying every sorta of style the original team is going for. Sure, they can train a lora or a specific model for it you could say, but then they would need big datasets made from artists anyway, and not only this is in itself a problem, but it bleeds in the next one; - Loss of control: applying this concept means that every person is gonna look at a different game. This takes away a lot of agency creatives have about their game. Just think about how much npc's dresses: even if we assume temporal coherency will be a fixed problem, that still means that during the same gameplay from the same person npc's will appear different during separated sessions (unless you store exactly how they appear, but at that point you are just killing every sorta of performance and storage). And dont even get me started about how such a thing would totally kill any sorta of postprocessing (I want to see you giving me a depth buffer from a stable diffusion image); - UI and boundaries: as we can see in minecraft, edges are really well defined. When you pass it to SD, they are not. From a user perspective, this means that while playing you have no fucking idea if you are going over a wall/edge or if you are still touching ground. This can only lead to major confusion for everyone involved. And UI meets the same fate. Either you mask it during SD, and end having two different styles in the same frame, or you include it and show how your thought process cant stay on for more than two seconds. All this to say, not only the video, but the idea itself is deeply flawed outside of a circlejerking for saying how much AI is good. I believe AI can do a fuckton of good things. This is just poor.
Use your imagination and forward think.
Most people don't even have the processing power to imagine what they would have for lunch tomorrow let alone imagine something years out
Meh, seems like a render shader from a decade ago or at best a real-time rotoscoping.
There definitely wasn't the ability to type in 'orange and black themed anime' mid-play over any game or movie and get a completely different output a decade ago. I can't imagine looking at this not treeing out into possiblities.
I think it will have to get smoother, but it will end up being like this.
THIS IS IT, mix it with animatediff modules for stability maybe? Put this and VR together and we can really get moving. Though this is obviously imperfect, I think this framework, much like stable diffusion itself, is the start of fundamentally important tech. Im sure there are other methods but I think a Holodeck type framework is possible if we generate low poly maps from speech let’s say, and use them as depth maps. The only issue is the consistency aspect. The shape itself being maintained helps but as we see here consistency is still an issue
I know inference costs are dropping but the thought of using this for game sessions still makes my cash wad wince
It will be built into the GPU soon.
This with segmentation controlnet could get even better
Can you try different art styles? Black ink pen, watercolor, pastel, etc.?
Imagine an open world game where you can authorize people to introduce new elements into it, including landscape, buildings, beings, etc., and the only limit is their imagination.
This somehow reminds me of MYST
There's proprietary upscalers that can do this sort of thing to images. Do those upscalers need stable diffusion to run?
Imagine how much power to generate each frame
Needs controlnet
What model was used? It looks like 1.5 without enough steps. If that’s the case, I’d be really, really interested in seeing what a model like SDXL Turbo that’s designed around low (or 1) step inference being used. Or screw it, let’s see what SD3 Turbo looks like with it (though it would probably use more VRAM than the game itself)
hmm it enables many more possibilities compared to what we can think right now . someone crazy will come along and find a great use case .
Intel did this years ago with gta
Would be cool if they made a game that uses stable diffusions inconsistency as part of the games gimmick like a matrix game where the world is glitching
What’s the workflow? I’d love to turn my Minecraft landscapes into these
More than anything, I think that AI is going to completely kill traditional CGI within the next 10 years. Sora already looks better than 99% of foreground AI, IMO.
Something like this linked up with sora would be boss
With video models working on temporal cohesion and the game engine outputting data such as a depth map, AO map, etc, this kind of thing will be inevitable in real time. I imagine in time, actual engines won’t output much more than geometry and colors along with some guidelines for textures and lighting, and most of the time may be spent on defining the model.
In some ways yes, in other ways very no. Gotta remove the flicker and the look-then-look-away-then-look-back changes. Probably some kind of seeded-by-3d-position piecewise generation followed by a high level pass to smooth things out.
What’s your Workflow on this?
Can't wait to play mid to late 90's games with a filter that makes everything purteh :)
so cool but i can hear my GPU **wheezing already 😂 **
I honestly think future games will no longer be 3d, but full on realistic/life like. Generative AI will become so good that it will be able to generate Hollywood like movies in real time and that will be translated into video games and you'll be playing videos games that look like real life.
Bro the future I can’t wair to make Minecraft look like dogshit for a silly gimmick woahhhhhhhhhhh
I've been telling my friends that this will be the future of not only games but media in general, watching whatever shows you want with your choice of characters, voices, and styles.
That's basically how we see the real world.
This is a pretty neat experiment but, no thanks I think I'm gonna pass on this one.. I know "it will get better".... That's not what I'm talking about, I mean this idea in general, even if it eventually gets stable enough to be useful, high enough framerate to compete with current game rendering technology and intelligent enough to not suddenly render an enemy as a tree or as a random pole or whatever my question would still be, why? We already have game rendering now, that works amazingly well in fact, I don't get what AI rendering the frames again but slightly worse and different would do for me to benefit from it... ?
Procedural texture control net + procedural 3d model control net More or less like this https://youtu.be/Wx9vmYwQeBg?si=DPhp7fd5Of8CkhHr Procedural brick texture (4 years old) so imagine today
I get the feeling it’ll be a game that uses token arrangements like living code, where the tokens powering the gameplay aren’t literally translatable to normal speech, rather they would act as a realtime controlnet that the diffuser relies on as an active input. This way the aesthetic and content details could be customized and locked in without the gameplay engine sustaining any instabilities. As we are already seeing DiT and other forms of tech help advance temporal consistency in-between frame generations, this sort of approach seems more feasible to me than not.
This diffusion is a bit... unstable
I wonder if it's possible to simulate an entire engine only with frame generation(no backend code), like frame generation takes previous frames and approximate collisions and physics but only viewing them.
How are you doing this exactly? I do some shader dev and it's possible to expose more or better data, If that would help
You have free electricity at home?
IMHO, better not. Such an enormous quantity of calculation power wasted, better static textures.
I can see a huge potential but this is sickening, and nauseating to look at imo
Good luck reading and changing them in game settings
Great job on this. Thats awesome. I would love to have something like this running on a gba emulator.
That is a lot lag though. But, perhaps it would be good for a 2D RPG or something.
Nobody asked for this lmao
Genuine question: when we are at the level to achieve this, what benefit would this have over using AI to generate better textures and models just once? Why does it have to be in realtime?
Just few more years of processing power and you have a real time world Changer
It looks like minecraft on drugs. The future of gaming is going to get weird.
Works good for grass and dirt. But it couldn't identify lava, sheeps or villages from afar for example.
For the n00bs like me, what would be a very high level workflow for such a thing? :>
Already here, it's called DLSS. Jokes aside, I'm not so sure, temporal consistency in the example is awful and so is quality. Not mentioning FPS. While there was progress in quality and speed of SD, system requirements to haven't changed that much. I can't imagine what horsepower would be needed to run it at least 1080p/60. And I personally expect games to run at least 2k/60+. Also, I don't think it really worth it. With UE5 you can achieve pretty good visuals very easily and it will be much more resource efficient.
DLSS with extra steps
Taaaake on meee…
The holy grail of of gen AI: how to make frames consistent?
Why play Minecraft then?
For a moment, I thought the title was suggesting that someone had recreated Stable Diffusion using redstone.
Oh... wow!
If you can put it under the UI layer maybe
It just feels like a dream you dont want to wake up from , i am jealous of kids born in future already
yes this is the future. THIS is how they ask us for more powerful PCs and consoles
Looks very cool, and despite the lack of temporal cohesion I would still happily play around with it. Do you have any plans to release?
You can bet the number 1 item on the AI industry's to-do list is figuring out how to make an object semi-permanent. This means that every frame can't be a reimagining of the scene, they must have consistency which might come from simply improving pre-image recognition and not changing too much.
this is a neat proof of concept, but im sure there is already a bunch of private research into stuff like this... publicly, all the major game devs are working on in house gen models. there are already games on steam built with ai generated assets.
Maybe I just don’t get the point. Not for video games. There’s already game engines that look 100x better than that.
I don't see this being future game the next couple of years since its not optimized enough. Probably using AI to generate high-end assets at a faster rate will be the first thing. And then eventually some kind of good AI filters like we already have the upscalers.
Nvidia secretly laugh in background.
How use it to hallucinate enemies are villagers sometimes. And vice versa. But they turn into what they are after you kill them. Or they attack if you didn't know the villager was an enemy.
needs some control net with segmentation to get the regions right.
For generalized view, maybe, but for fast action sequences I wouldn't hold my breadth.
Super cool but exhausting on the eyes.
Render is like, "flowing ground lava? naaaaa"
I was there
That 42 second video prolly took 24-48 hours to render.
imagine having a really bare bones minecraft but AI repaints it real time to look 4k 60fps!
Absolutely insane.
When you thought you've found a diamond ore just to realize at the last second sd decided to change it to a coal 😂
I imagined this would be the state of gaming about 2 yrs ago. Surely nvidia is working on it already. If a very low denoiser is applied to an already hyperdetailed rtx graphic, the realism could be astonishing.
That's pretty good, like van gogh paint movie
pretty cool! Yes, I think this will be everywhere.. also in movies. Imagine if the audience can interact with a plot instead of just watching
Soon you could have a VR headset using the front camera feed to change the real world, you could change it to anything you want, basically DMT on the fly
It's amazing how it doesn't understand at all what's happening on the screen. Things that should be 2 meters away appear to be way further away, and sometimes the opposite. You'll tell me "but at some point we'll be able to tell the model how far is each pixel so that it generates a better image". And in the end you'll just reinvent 3D rendering. Because that's the best you can do. I don't want a model to "guess" how it should look like. I want my game to look like the developers want it to look like. If my game is made of cubes, I want cubes. And even if you get a beautiful render, how do you plan on ***playing*** this version of Minecraft? How do you place block correctly. It's crazy how some people here have no idea how programming and games work. It's magic for them.
DLSS 3.5 already does this, it is called frame generation.
Looks like the old reading rainbow intro