Jump to content

Leadwerks Software to Assist NASA Building VR Applications

Josh

2,774 views

TLDR: I made a long-term bet on VR and it's paying off. I haven't been able to talk much about the details until now.

Here's what happened:

Leadwerks 3.0 was released during GDC 2013. I gave a talk on graphics optimization and also had a booth at the expo. Something else significant happened that week.  After the expo closed I walked over to the Oculus booth and they let me try out the first Rift prototype.

This was a pivotal time both for us and for the entire game industry. Mobile was on the downswing but there were new technologies emerging that I wanted to take advantage of. Our Kickstarter campaign for Linux support was very successful, reaching over 200% of its goal. This coincided with a successful Greenlight campaign to bring Leadwerks Game Engine to Steam in the newly-launched software section. The following month Valve announced the development of SteamOS, a Linux-based operating system for the Steam Machine game consoles. Because of our work in Linux and our placement in Steam, I was fortunate enough to be in close contact with much of the staff at Valve Software.

The Early Days of VR

It was during one of my visits to Valve HQ that I was able to try out a prototype of the technology that would go on to become the HTC Vive. In September of 2014 I bought an Oculus Rift DK2 and first started working with VR in Leadwerks. So VR has been something I have worked on in the background for a long time, but I was looking for the right opportunity to really put it to work. In 2016 I felt it was time for a technology refresh, so I wrote a blog about the general direction I wanted to take Leadwerks in. A lot of it centered around VR and performance. I didn't really know exactly how things would work out, but I knew I wanted to do a lot of work with VR.

A month later I received a message on this forum that went something like this (as I recall):

Quote

Hey man!!! I love Leadwerks, and I think you are really cool! I work for NASA, want to come and build spaceships with me?

I thought "Okay, some stupid teenager, where is my ban button?", but when I started getting emails with nasa.gov return addresses I took notice.

Now, Leadwerks Software has a long history of use in the defense and simulation industries, with orders for software from Northrop Grumman, Lockheed Martin, the British Royal Navy, and probably some others I don't know about. So NASA making an inquiry about software isn't too strange. What was strange was that they were very interested in meeting in person.

Mr. Josh Goes to Washington

I took my first trip to Goddard Space Center in January 2017 where I got a tour of the facility. I saw robots, giant satellites, rockets, and some crazy laser rooms that looked like a Half-Life level. It was my eleven year old self's dream come true. I was also shown some of the virtual reality work they are using Leadwerks Game Engine for. Basically, they were taking high-poly engineering models from CAD software and putting them into a real-time visualization in VR. There are some good reasons for this. VR gives you a stereoscopic view of objects that is far superior to a flat 2D screen. This makes a huge difference when you are viewing complex mechanical objects and planning robotic movements. You just can't see things on a flat screen the same way you can see them in VR. It's like the difference between looking at a photograph of an object versus holding it in your hands.

Mars-2020-rover.thumb.jpg.6dd183781f8afe8504f7466551bb6b0f.jpg
What is even going on here???

CAD models are procedural, meaning they have a precise mathematical formula that describes their shape. In order to render them in real-time, they have to be converted to polygonal models, but these objects can be tens of millions of polygons, with details down to threading on individual screws, and they were trying to view them in VR at 90 frames per second! Now with virtual reality, if there is a discrepancy between what your visual system and your vestibular system perceives, you will get sick to your stomach. That's why it's critical to maintain a steady 90 Hz frame rate. The engineers at NASA told me they first tried to use Unity3D but it was too slow, which is why they came to me. Leadwerks was giving them better performance, but it still was not fast enough for what they wanted to do next. I thought "these guys are crazy, it cannot be done".

Then I remembered something else people said could never be done.

000f18d5-800.jpg.990e1a219313c9100d0d81c1f9309928.jpg

So I started to think "if it were possible, how would I do it?" They had also expressed interest in an inverse kinematics simulation, so I put together this robotic arm control demo in a few days, just to show what could be easily be done with our physics system.

 

Turbo Game Engine is Born

With the extreme performance demands of VR and my experience writing optimized rendering systems, I saw an opportunity to focus our development on something people can't live without: speed. I started building a new renderer designed specifically around the way modern PC hardware works. At first I expected to see performance increases of 2-3x. Instead what we are seeing are 10-40x performance increases under heavy loads. Once I saw this I was very encouraged, so I decided to name the new engine "Turbo Game Engine" (the point is absolutely unmissable) and bought the domain name turboengine.com. During this time I stayed in contact with people at NASA and kept them up to date on the capabilities of the new technology.

At this point there was still nothing concrete to show for my efforts. NASA purchased some licenses for the Enterprise edition of Leadwerks Game Engine, but the demos I made were free of charge and I was paying my own travel expenses. The cost of plane tickets and hotels adds up quickly, and there was no guarantee any of this would work out. I did not want to talk about what I was doing on this site because it would be embarrassing if I made a lot of big plans and nothing came of it. But I saw a need for the technology I created and I figured something would work out, so I kept working away at it.

Call to Duty

Today I am pleased to announce I have signed a contract to put our virtual reality expertise to work for NASA. As we speak, I am preparing to travel to Washington D.C. to begin the project. In the future I plan to provide support for aerospace, defense, manufacturing, and serious games, using our new technology to help users deliver VR simulations with performance and realism beyond anything that has been possible until now.

My software company and relationship with my customers (you) is unaffected. Development of the new engine will continue, with a strong emphasis on hyper-realism and performance. I think this is a direction everyone here will be happy with. I am going to continue to invest in the development of groundbreaking new features that will help in the aerospace and defense industries (now you understand why I have been talking about 64-bit worlds) and I think a great many people will be happy to come along for the ride in this direction.

Leadwerks is still a game company, but our core focus is on enabling and creating hyper-realistic VR simulations. Thank you for your support and all the suggestions and ideas you have provided over the years that have helped me create great software for you. Things are about to get very interesting. I can't wait to see what you all create with the new technology we are building.

541922main_atlasvcloseup.thumb.jpg.d769aab53db0683ace3fa249e09fd994.jpg

  • Like 13


18 Comments


Recommended Comments

05.gif
I wish you the best of luck in the project, and don't forget the users of LE 4, we are always waiting for an update or two. :)

  • Like 1

Share this comment


Link to comment

Yes, 4.6 will resume development as soon as I get back to California at the end of the week.

  • Thanks 2

Share this comment


Link to comment

Let’s plan on doing a Google Hangout this Saturday so I can answer any questions you have.

  • Like 2
  • Thanks 1

Share this comment


Link to comment
4 hours ago, Josh said:

Let’s plan on doing a Google Hangout this Saturday so I can answer any questions you have.

I'd be up for that.  Depends on time in the land of AUS of course :P

Share this comment


Link to comment

Ground control to Major Josh
Ground control to Major Josh
Get ready to program
Laptop on.

Ground control to Major Josh
Commencing countdown
Leadwerks on
Check ignition
And may Turbo be with you!

  • Like 2

Share this comment


Link to comment

Very cool. Glad you're back in the States. Although Turbo is still the priority, I hope this NASA contact benefits LE4 while Turbo is being worked on. Would be nice to see bugs fixed and more VR options soon.

Also hopefully this means you're not tied to that Gigabyte box so I also hope to see AMD and Linux support with the new engine. (And continued support for LE4.)

Congratulations, and continue making cool stuff!

  • Like 1

Share this comment


Link to comment

If you guys can share this blog entry on Facebook and Twitter it would help a lot. Thanks!

Share this comment


Link to comment

Congratulations Josh! That's some exciting news, I hope this finally puts Leadwerks/Turbo on the map in a big way. I can't wait to see the first AAA media production (game or otherwise) that's powered in Turbo.

Considering your affiliation with NASA perhaps rename Turbo to the ''ROCKET ENGINE'" ? It encapsulates visions of speed, power, performance and the ability to take you to new worlds :)

I wish I would've got my tournament posters signed now...

 

the-space-force-construction-already-underway.thumb.jpg.45bea21467b2fdc8ff9dc9fb1acada94.jpg

Share this comment


Link to comment

Congrats Josh. As stated before in your blogs: unlike the mobile direction, VR is actually bringing back the power to the engine. So leadwerks 5 will contain the Turbo changes, minus the new editor. 

Do you think you will expand your team, now that you have a contract?

Share this comment


Link to comment
8 minutes ago, AggrorJorn said:

Congrats Josh. As stated before in your blogs: unlike the mobile direction, VR is actually bringing back the power to the engine. So that plan is still unchanged right? So leadwerks 5 will contain the Turbo changes, minus the editor. 

Do you think you will expend your team, now that you have a contract?

I don't think Leadwerks 5 like I was suggesting might happen will happen. We will jump straight to Turbo and use the Leadwerks 4 editor until the new editor is available.

Yes, this gives me more predictable cash flow and this is how I plan to add new hires.

  • Like 4
  • Thanks 1
  • Confused 1

Share this comment


Link to comment

 Really proud of you Josh, you have quite literally moved mountains to get where you are. You skipped the low steps on the ladder and went straight to the top. 

gsfc

  • Like 1

Share this comment


Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Blog Entries

    • By Josh in Josh's Dev Blog 1
      I started to implement quads for tessellation, and at that point the shader system reached the point of being unmanageable. Rendering an object to a shadow map and to a color buffer are two different processes that require two different shaders. Turbo introduces an early Z-pass which can use another shader, and if variance shadow maps are not in use this can be a different shader from the shadow shader. Rendering with tessellation requires another set of shaders, with one different set for each primitive type (isolines, triangles, and quads). And then each one of these needs a masked and opaque option, if alpha discard is enabled.
      All in all, there are currently 48 different shaders a material could use based on what is currently being drawn. This is unmanageable.
      To handle this I am introducing the concept of a "shader family". This is a JSON file that lists all possible permutations of a shader. Instead of setting lots of different shaders in a material, you just set the shader family one:
      shaderFamily: "PBR.json" Or in code:
      material->SetShaderFamily(LoadShaderFamily("PBR.json")); The shader family file is a big JSON structure that contains all the different shader modules for each different rendering configuration: Here are the partial contents of my PBR.json file:
      { "turboShaderFamily" : { "OPAQUE": { "default": { "base": { "vertex": "Shaders/PBR.vert.spv", "fragment": "Shaders/PBR.frag.spv" }, "depthPass": { "vertex": "Shaders/Depthpass.vert.spv" }, "shadow": { "vertex": "Shaders/Shadow.vert.spv" } }, "isolines": { "base": { "vertex": "Shaders/PBR_Tess.vert.spv", "tessellationControl": "Shaders/Isolines.tesc.spv", "tessellationEvaluation": "Shaders/Isolines.tese.spv", "fragment": "Shaders/PBR_Tess.frag.spv" }, "shadow": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "Shaders/DepthPass_Isolines.tesc.spv", "tessellationEvaluation": "Shaders/DepthPass_Isolines.tese.spv" }, "depthPass": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "DepthPass_Isolines.tesc.spv", "tessellationEvaluation": "DepthPass_Isolines.tese.spv" } }, "triangles": { "base": { "vertex": "Shaders/PBR_Tess.vert.spv", "tessellationControl": "Shaders/Triangles.tesc.spv", "tessellationEvaluation": "Shaders/Triangles.tese.spv", "fragment": "Shaders/PBR_Tess.frag.spv" }, "shadow": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "Shaders/DepthPass_Triangles.tesc.spv", "tessellationEvaluation": "Shaders/DepthPass_Triangles.tese.spv" }, "depthPass": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "DepthPass_Triangles.tesc.spv", "tessellationEvaluation": "DepthPass_Triangles.tese.spv" } }, "quads": { "base": { "vertex": "Shaders/PBR_Tess.vert.spv", "tessellationControl": "Shaders/Quads.tesc.spv", "tessellationEvaluation": "Shaders/Quads.tese.spv", "fragment": "Shaders/PBR_Tess.frag.spv" }, "shadow": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "Shaders/DepthPass_Quads.tesc.spv", "tessellationEvaluation": "Shaders/DepthPass_Quads.tese.spv" }, "depthPass": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "DepthPass_Quads.tesc.spv", "tessellationEvaluation": "DepthPass_Quads.tese.spv" } } } } } A shader family file can indicate a root to inherit values from. The Blinn-Phong shader family pulls settings from the PBR file and just switches some of the fragment shader values.
      { "turboShaderFamily" : { "root": "PBR.json", "OPAQUE": { "default": { "base": { "fragment": "Shaders/Blinn-Phong.frag.spv" } }, "isolines": { "base": { "fragment": "Shaders/Blinn-Phong_Tess.frag.spv" } }, "triangles": { "base": { "fragment": "Shaders/Blinn-Phong_Tess.frag.spv" } }, "quads": { "base": { "fragment": "Shaders/Blinn-Phong_Tess.frag.spv" } } } } } If you want to implement a custom shader, this is more work because you have to define all your changes for each possible shader variation. But once that is done, you have a new shader that will work with all of these different settings, which in the end is easier. I considered making a more complex inheritance / cascading schema but I think eliminating ambiguity is the most important goal in this and that should override any concern about the verbosity of these files. After all, I only plan on providing a couple of these files and you aren't going to need any more unless you are doing a lot of custom shaders. And if you are, this is the best solution for you anyways.
      Consequently, the baseShader, depthShader, etc. values in the material file definition are going away. Leadwerks .mat files will always use the Blinn-Phong shader family, and there is no way to change this without creating a material file in the new JSON material format.
      The shader class is no longer derived from the Asset class because it doesn't correspond to a single file. Instead, it is just a dumb container. A ShaderModule class derived from the Asset class has been added, and this does correspond with a single .spv file. But you, the user, won't really need to deal with any of this.
      The result of this is that one material will work with tessellation enabled or disabled, quad, triangle, or line meshes, and animated meshes. I also added an optional parameter in the CreatePlane(), CreateBox(), and CreateQuadSphere() commands that will create these primitives out of quads instead of triangles. The main reason for supporting quad meshes is that the tessellation is cleaner when quads are used. (Note that Vulkan still displays quads in wireframe mode as if they are triangles. I think the renderer probably converts them to normal triangles after the tessellation stage.)


      I also was able to implement PN Quads, which is a quad version of the Bezier curve that PN Triangles add to tessellation.



      Basically all the complexity is being packed into the shader family file so that these decisions only have to be made once instead of thousands of times for each different material.
    • By Josh in Josh's Dev Blog 0
      I'm back from I/ITSEC. This conference is basically like the military's version of GDC. VR applications built with Leadwerks took up about half of Northrop Grumman's booth. There were many interesting discussions about new technology and I received a very warm reception. I feel very positive about our new technology going forward.

      I am currently reworking the text field widget script to work with our persistent 2D objects. This is long and boring but needs to be done. Not much else to say right now.
    • By Josh in Josh's Dev Blog 4
      Here are some screenshots showing more complex interface items scaled at different resolutions. First, here is the interface at 100% scaling:

      And here is the same interface at the same screen resolution, with the DPI scaling turned up to 150%:

      The code to control this is sort of complex, and I don't care. GUI resolution independence is a complicated thing, so the goal should be to create a system that does what it is supposed to do reliably, not to make complicated things simpler at the expense of functionality.
      function widget:Draw(x,y,width,height) local scale = self.gui:GetScale() self.primitives[1].size = iVec2(self.size.x, self.size.y - self.tabsize.y * scale) self.primitives[2].size = iVec2(self.size.x, self.size.y - self.tabsize.y * scale) --Tabs local n local tabpos = 0 for n = 1, #self.items do local tw = self:TabWidth(n) * scale if n * 3 > #self.primitives - 2 then self:AddRect(iVec2(tabpos,0), iVec2(tw, self.tabsize.y * scale), self.bordercolor, false, self.itemcornerradius * scale) self:AddRect(iVec2(tabpos+1,1), iVec2(tw, self.tabsize.y * scale) - iVec2(2 * scale,-1 * scale), self.backgroundcolor, false, self.itemcornerradius * scale) self:AddTextRect(self.items[n].text, iVec2(tabpos,0), iVec2(tw, self.tabsize.y*scale), self.textcolor, TEXT_CENTER + TEXT_MIDDLE) end if self:SelectedItem() == n then self.primitives[2 + (n - 1) * 3 + 1].position = iVec2(tabpos, 0) self.primitives[2 + (n - 1) * 3 + 1].size = iVec2(tw, self.tabsize.y * scale) + iVec2(0,2) self.primitives[2 + (n - 1) * 3 + 2].position = iVec2(tabpos + 1, 1) self.primitives[2 + (n - 1) * 3 + 2].color = self.selectedtabcolor self.primitives[2 + (n - 1) * 3 + 2].size = iVec2(tw, self.tabsize.y * scale) - iVec2(2,-1) self.primitives[2 + (n - 1) * 3 + 3].color = self.hoveredtextcolor self.primitives[2 + (n - 1) * 3 + 1].position = iVec2(tabpos,0) self.primitives[2 + (n - 1) * 3 + 2].position = iVec2(tabpos + 1, 1) self.primitives[2 + (n - 1) * 3 + 3].position = iVec2(tabpos,0) else self.primitives[2 + (n - 1) * 3 + 1].size = iVec2(tw, self.tabsize.y * scale) self.primitives[2 + (n - 1) * 3 + 2].color = self.tabcolor self.primitives[2 + (n - 1) * 3 + 2].size = iVec2(tw, self.tabsize.y * scale) - iVec2(2,2) if n == self.hovereditem then self.primitives[2 + (n - 1) * 3 + 3].color = self.hoveredtextcolor else self.primitives[2 + (n - 1) * 3 + 3].color = self.textcolor end self.primitives[2 + (n - 1) * 3 + 1].position = iVec2(tabpos,2) self.primitives[2 + (n - 1) * 3 + 2].position = iVec2(tabpos + 1, 3) self.primitives[2 + (n - 1) * 3 + 3].position = iVec2(tabpos,2) end self.primitives[2 + (n - 1) * 3 + 3].text = self.items[n].text tabpos = tabpos + tw - 2 end end  
×
×
  • Create New...