Jump to content

Leadwerks Software to Assist NASA Building VR Applications

Josh

2,522 views

TLDR: I made a long-term bet on VR and it's paying off. I haven't been able to talk much about the details until now.

Here's what happened:

Leadwerks 3.0 was released during GDC 2013. I gave a talk on graphics optimization and also had a booth at the expo. Something else significant happened that week.  After the expo closed I walked over to the Oculus booth and they let me try out the first Rift prototype.

This was a pivotal time both for us and for the entire game industry. Mobile was on the downswing but there were new technologies emerging that I wanted to take advantage of. Our Kickstarter campaign for Linux support was very successful, reaching over 200% of its goal. This coincided with a successful Greenlight campaign to bring Leadwerks Game Engine to Steam in the newly-launched software section. The following month Valve announced the development of SteamOS, a Linux-based operating system for the Steam Machine game consoles. Because of our work in Linux and our placement in Steam, I was fortunate enough to be in close contact with much of the staff at Valve Software.

The Early Days of VR

It was during one of my visits to Valve HQ that I was able to try out a prototype of the technology that would go on to become the HTC Vive. In September of 2014 I bought an Oculus Rift DK2 and first started working with VR in Leadwerks. So VR has been something I have worked on in the background for a long time, but I was looking for the right opportunity to really put it to work. In 2016 I felt it was time for a technology refresh, so I wrote a blog about the general direction I wanted to take Leadwerks in. A lot of it centered around VR and performance. I didn't really know exactly how things would work out, but I knew I wanted to do a lot of work with VR.

A month later I received a message on this forum that went something like this (as I recall):

Quote

Hey man!!! I love Leadwerks, and I think you are really cool! I work for NASA, want to come and build spaceships with me?

I thought "Okay, some stupid teenager, where is my ban button?", but when I started getting emails with nasa.gov return addresses I took notice.

Now, Leadwerks Software has a long history of use in the defense and simulation industries, with orders for software from Northrop Grumman, Lockheed Martin, the British Royal Navy, and probably some others I don't know about. So NASA making an inquiry about software isn't too strange. What was strange was that they were very interested in meeting in person.

Mr. Josh Goes to Washington

I took my first trip to Goddard Space Center in January 2017 where I got a tour of the facility. I saw robots, giant satellites, rockets, and some crazy laser rooms that looked like a Half-Life level. It was my eleven year old self's dream come true. I was also shown some of the virtual reality work they are using Leadwerks Game Engine for. Basically, they were taking high-poly engineering models from CAD software and putting them into a real-time visualization in VR. There are some good reasons for this. VR gives you a stereoscopic view of objects that is far superior to a flat 2D screen. This makes a huge difference when you are viewing complex mechanical objects and planning robotic movements. You just can't see things on a flat screen the same way you can see them in VR. It's like the difference between looking at a photograph of an object versus holding it in your hands.

Mars-2020-rover.thumb.jpg.6dd183781f8afe8504f7466551bb6b0f.jpg
What is even going on here???

CAD models are procedural, meaning they have a precise mathematical formula that describes their shape. In order to render them in real-time, they have to be converted to polygonal models, but these objects can be tens of millions of polygons, with details down to threading on individual screws, and they were trying to view them in VR at 90 frames per second! Now with virtual reality, if there is a discrepancy between what your visual system and your vestibular system perceives, you will get sick to your stomach. That's why it's critical to maintain a steady 90 Hz frame rate. The engineers at NASA told me they first tried to use Unity3D but it was too slow, which is why they came to me. Leadwerks was giving them better performance, but it still was not fast enough for what they wanted to do next. I thought "these guys are crazy, it cannot be done".

Then I remembered something else people said could never be done.

000f18d5-800.jpg.990e1a219313c9100d0d81c1f9309928.jpg

So I started to think "if it were possible, how would I do it?" They had also expressed interest in an inverse kinematics simulation, so I put together this robotic arm control demo in a few days, just to show what could be easily be done with our physics system.

 

Turbo Game Engine is Born

With the extreme performance demands of VR and my experience writing optimized rendering systems, I saw an opportunity to focus our development on something people can't live without: speed. I started building a new renderer designed specifically around the way modern PC hardware works. At first I expected to see performance increases of 2-3x. Instead what we are seeing are 10-40x performance increases under heavy loads. Once I saw this I was very encouraged, so I decided to name the new engine "Turbo Game Engine" (the point is absolutely unmissable) and bought the domain name turboengine.com. During this time I stayed in contact with people at NASA and kept them up to date on the capabilities of the new technology.

At this point there was still nothing concrete to show for my efforts. NASA purchased some licenses for the Enterprise edition of Leadwerks Game Engine, but the demos I made were free of charge and I was paying my own travel expenses. The cost of plane tickets and hotels adds up quickly, and there was no guarantee any of this would work out. I did not want to talk about what I was doing on this site because it would be embarrassing if I made a lot of big plans and nothing came of it. But I saw a need for the technology I created and I figured something would work out, so I kept working away at it.

Call to Duty

Today I am pleased to announce I have signed a contract to put our virtual reality expertise to work for NASA. As we speak, I am preparing to travel to Washington D.C. to begin the project. In the future I plan to provide support for aerospace, defense, manufacturing, and serious games, using our new technology to help users deliver VR simulations with performance and realism beyond anything that has been possible until now.

My software company and relationship with my customers (you) is unaffected. Development of the new engine will continue, with a strong emphasis on hyper-realism and performance. I think this is a direction everyone here will be happy with. I am going to continue to invest in the development of groundbreaking new features that will help in the aerospace and defense industries (now you understand why I have been talking about 64-bit worlds) and I think a great many people will be happy to come along for the ride in this direction.

Leadwerks is still a game company, but our core focus is on enabling and creating hyper-realistic VR simulations. Thank you for your support and all the suggestions and ideas you have provided over the years that have helped me create great software for you. Things are about to get very interesting. I can't wait to see what you all create with the new technology we are building.

541922main_atlasvcloseup.thumb.jpg.d769aab53db0683ace3fa249e09fd994.jpg

  • Like 13


18 Comments


Recommended Comments

05.gif
I wish you the best of luck in the project, and don't forget the users of LE 4, we are always waiting for an update or two. :)

  • Like 1

Share this comment


Link to comment

Yes, 4.6 will resume development as soon as I get back to California at the end of the week.

  • Thanks 2

Share this comment


Link to comment

Let’s plan on doing a Google Hangout this Saturday so I can answer any questions you have.

  • Like 2
  • Thanks 1

Share this comment


Link to comment
4 hours ago, Josh said:

Let’s plan on doing a Google Hangout this Saturday so I can answer any questions you have.

I'd be up for that.  Depends on time in the land of AUS of course :P

Share this comment


Link to comment

Ground control to Major Josh
Ground control to Major Josh
Get ready to program
Laptop on.

Ground control to Major Josh
Commencing countdown
Leadwerks on
Check ignition
And may Turbo be with you!

  • Like 2

Share this comment


Link to comment

Very cool. Glad you're back in the States. Although Turbo is still the priority, I hope this NASA contact benefits LE4 while Turbo is being worked on. Would be nice to see bugs fixed and more VR options soon.

Also hopefully this means you're not tied to that Gigabyte box so I also hope to see AMD and Linux support with the new engine. (And continued support for LE4.)

Congratulations, and continue making cool stuff!

  • Like 1

Share this comment


Link to comment

If you guys can share this blog entry on Facebook and Twitter it would help a lot. Thanks!

Share this comment


Link to comment

Congratulations Josh! That's some exciting news, I hope this finally puts Leadwerks/Turbo on the map in a big way. I can't wait to see the first AAA media production (game or otherwise) that's powered in Turbo.

Considering your affiliation with NASA perhaps rename Turbo to the ''ROCKET ENGINE'" ? It encapsulates visions of speed, power, performance and the ability to take you to new worlds :)

I wish I would've got my tournament posters signed now...

 

the-space-force-construction-already-underway.thumb.jpg.45bea21467b2fdc8ff9dc9fb1acada94.jpg

Share this comment


Link to comment

Congrats Josh. As stated before in your blogs: unlike the mobile direction, VR is actually bringing back the power to the engine. So leadwerks 5 will contain the Turbo changes, minus the new editor. 

Do you think you will expand your team, now that you have a contract?

Share this comment


Link to comment
8 minutes ago, AggrorJorn said:

Congrats Josh. As stated before in your blogs: unlike the mobile direction, VR is actually bringing back the power to the engine. So that plan is still unchanged right? So leadwerks 5 will contain the Turbo changes, minus the editor. 

Do you think you will expend your team, now that you have a contract?

I don't think Leadwerks 5 like I was suggesting might happen will happen. We will jump straight to Turbo and use the Leadwerks 4 editor until the new editor is available.

Yes, this gives me more predictable cash flow and this is how I plan to add new hires.

  • Like 4
  • Thanks 1
  • Confused 1

Share this comment


Link to comment

 Really proud of you Josh, you have quite literally moved mountains to get where you are. You skipped the low steps on the ladder and went straight to the top. 

gsfc

  • Like 1

Share this comment


Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Blog Entries

    • By Josh in Josh's Dev Blog 4
      Previously I described how multiple cameras can be combined in the new renderer to create an unlimited depth buffer. That discussion lead into multi-world rendering and 2D drawing. Surprisingly, there is a lot of overlap in these features, and it makes sense to solve all of it at one time.
      Old 2D rendering systems are designed around the idea of storing a hierarchy of state changes. The renderer would crawl through the hierarchy and perform commands as it went along, rendering all 2D elements in the order they should appear. It made sense for the design of the first graphics cards, but this style of rendering is really inefficient on modern graphics hardware. Today's hardware works best with batches of objects, using the depth buffer to handle which object appears on top. We don't sort 3D objects back-to-front because it would be monstrously inefficient, so why should 2D graphics be any different?
      We can get much better results if we use the same fast rendering techniques we use for 3D graphics and apply it to 2D shapes. After all, the only difference between 3D and 2D rendering is the shape of the camera projection matrix. For this reason, Turbo Engine will use 2D-in-3D rendering for all 2D drawing. You can render a pure 2D scene by setting the camera projection mode to orthographic, or you can create a second orthographic camera and render it on top of your 3D scene. This has two big implications:
      Performance will be incredibly fast. I predict 100,000 uniquely textured sprites will render pretty much instantaneously. In fact anyone making a 2D PC game who is having trouble with performance will be interested in using Turbo Engine. Advanced 3D effects will be possible that we aren't used to seeing in 2D. For example, lighting works with 2D rendering with no problems, as you can see below. Mixing of 3D and 2D elements will be possible to make inventory systems and other UI items. Particles and other objects can be incorporated into the 2D display.
      The big difference you will need to adjust to is there are no 2D drawing commands. Instead you have persistent objects that use the same system as the 3D rendering.
      Sprites
      The primary 2D element you will work with is the Sprite entity, which works the same as the 3D sprites in Leadwerks 4. Instead of drawing rectangles in the order you want them to appear, you will use the Z position of each entity and let the depth buffer take care of the rest, just like we do with 3D rendering. I also am adding support for animation frames and other features, and these can be used with 2D or 3D rendering.

      Rotation and scaling of sprites is of course trivial. You could even use effects like distance fog! Add a vector joint to each entity to lock the Z axis in the same direction and Newton will transform into a nice 2D physics system.
      Camera Setup
      By default, with a zoom value of 1.0 an orthographic camera maps so that one meter in the world equals one screen pixel. We can position the camera so that world coordinates match screen coordinates, as shown in the image below.
      auto camera = CreateCamera(world); camera->SetProjectionMode(PROJECTION_ORTHOGRAPHIC); camera->SetRange(-1,1); iVec2 screensize = framebuffer->GetSize(); camera->SetPosition(screensize.x * 0.5, -screensize.y * 0.5); Note that unlike screen coordinates in Leadwerks 4, world coordinates point up in the positive direction.

      We can create a sprite and reset its center point to the upper left hand corner of the square like so:
      auto sprite = CreateSprite(world); sprite->mesh->Translate(0.5,-0.5,0); sprite->mesh->Finalize(); sprite->UpdateBounds(); And then we can position the sprite in the upper left-hand corner of the screen and scale it:
      sprite->SetColor(1,0,0); sprite->SetScale(200,50); sprite->SetPosition(10,-10,0);
      This would result in an image something like this, with precise alignment of screen pixels:

      Here's an idea: Remember the opening sequence in Super Metroid on SNES, when the entire world starts tilting back and forth? You could easily do that just by rotating the camera a bit.
      Displaying Text
      Instead of drawing text with a command, you will create a text model. This is a series of rectangles of the correct size with their texture coordinates set to display a letter for each rectangle. You can include a line return character in the text, and it will create a block of multiple lines of text in one object. (I may add support for text made out of polygons at a later time, but it's not a priority right now.)
      shared_ptr<Model> CreateText(shared_ptr<World> world, shared_ptr<Font> font, const std::wstring& text, const int size) The resulting model will have a material with the rasterized text applied to it, shown below with alpha blending disabled so you can see the mesh background. Texture coordinates are used to select each letter, so the font only has to be rasterized once for each size it is used at:

      Every piece of text you display needs to have a model created for it. If you are displaying the framerate or something else that changes frequently, then it makes sense to create a cache of models you use so your game isn't constantly creating new objects. If you wanted, you could modify the vertex colors of a text model to highlight a single word.

      And of course all kinds of spatial transformations are easily achieved.

      Because the text is just a single textured mesh, it will render very fast. This is a big improvement over the DrawText() command in Leadwerks 4, which performs one draw call for each character.
      The font loading command no longer accepts a size. You load the font once and a new image will be rasterized for each text size the engine requests internally:
      auto font = LoadFont("arial.ttf"); auto text = CreateText(foreground, font, "Hello, how are you today?", 18); Combining 2D and 3D
      By using two separate worlds we can control which items the 3D camera draws and which item 2D camera draws: (The foreground camera will be rendered on top of the perspective camera, since it is created after it.) We need to use a second camera so that 2D elements are rendered in a second pass with a fresh new depth buffer.
      //Create main world and camera auto world = CreateWorld(); auto camera = CreateCamera(world); auto scene = LoadScene(world,"start.map"); //Create world for 2D rendering auto foreground = CreateWorld() auto fgcam = CreateCamera(foreground); fgcam->SetProjection(PROJECTION_ORTHOGRAPHIC); fgcam->SetClearMode(CLEAR_DEPTH); fgcam->SetRange(-1,1); auto UI = LoadScene(foreground,"UI.map"); //Combine rendering world->Combine(foreground); while (true) { world->Update(); world->Render(framebuffer); } Overall, this will take more work to set up and get started with than the simple 2D drawing in Leadwerks 4, but the performance and additional control you get are well worth it. This whole approach makes so much sense to me, and I think it will lead to some really cool possibilities.
      As I have explained elsewhere, performance has replaced ease of use as my primary design goal. I like the results I get with this approach because I feel the design decisions are less subjective.
    • By Josh in Josh's Dev Blog 25
      Current generation graphics hardware only supports up to a 32-bit floating point depth buffer, and that isn't adequate for large-scale rendering because there isn't enough precision to make objects appear in the correct order and prevent z-fighting.

      After trying out a few different approaches I found that the best way to support large-scale rendering is to allow the user to create several cameras. The first camera should have a range of 0.1-1000 meters, the second would use the same near / far ratio and start where the first one left off, with a depth range of 1000-10,000 meters. Because the ratio of near to far ranges is what matters, not the actual distance, the numbers can get very big very fast. A third camera could be added with a range out to 100,000 kilometers!
      The trick is to set the new Camera::SetClearMode() command to make it so only the furthest-range camera clears the color buffer. Additional cameras clear the depth buffer and then render on top of the previous draw. You can use the new Camera::SetOrder() command to ensure that they are drawn in the order you want.
      auto camera1 = CreateCamera(world); camera1->SetRange(0.1,1000); camera1->SetClearMode(CLEAR_DEPTH); camera1->SetOrder(1); auto camera2 = CreateCamera(world); camera2->SetRange(1000,10000); camera2->SetClearMode(CLEAR_DEPTH); camera2->SetOrder(2); auto camera3 = CreateCamera(world); camera3->SetRange(10000,100000000); camera3->SetClearMode(CLEAR_COLOR | CLEAR_DEPTH); camera3->SetOrder(3); Using this technique I was able to render the Earth, sun, and moon to-scale. The three objects are actually sized correctly, at the correct distance. You can see that from Earth orbit the sun and moon appear roughly the same size. The sun is much bigger, but also much further away, so this is exactly what we would expect.

      You can also use these features to render several cameras in one pass to show different views. For example, we can create a rear-view mirror easily with a second camera:
      auto mirrorcam = CreateCamera(world); mirrorcam->SetParent(maincamera); mirrorcam->SetRotation(0,180,0); mirrorcam=>SetClearMode(CLEAR_COLOR | CLEAR_DEPTH); //Set the camera viewport to only render to a small rectangle at the top of the screen: mirrorcam->SetViewport(framebuffer->GetSize().x/2-200,10,400,50); This creates a "picture-in-picture" effect like what is shown in the image below:

      Want to render some 3D HUD elements on top of your scene? This can be done with an orthographic camera:
      auto uicam = CreateCamera(world); uicam=>SetClearMode(CLEAR_DEPTH); uicam->SetProjectionMode(PROJECTION_ORTHOGRAPHIC); This will make 3D elements appear on top of your scene without clearing the previous render result. You would probably want to move the UI camera far away from the scene so only your HUD elements appear in the last pass.
    • By Marcousik in Marcousik's Creations Blog 5
      I updated the moto I was working on and now the physics let having fun with this.
       
×
×
  • Create New...