Jump to content

Josh

Staff
  • Posts

    22,901
  • Joined

  • Last visited

Blog Entries posted by Josh

  1. Josh

    Articles
    At long last, the engine that felt like it would never be done is here. This is all the result of an idea I had and wanted to put to the test. Of course I never anticipated it was possible to deliver a 10x improvement in rendering performance, but here we are with the benchmarks to prove it.
    Research & Development
    The last four years of research and development were an opportunity to rethink and redesign what a game engine should be. Along the way there were a few ideas I had that turned out to be not that great of an idea in practice:
    Variance shadow maps: Although I love the look of these under very controlled settings, the technique has extremely bad artifacts that can never be fixed. Depth shadow maps really are the best approach to texture-based shadows. They are also extremely slow to render shadow updates. I would like to see some better hardware filtering options in the future, though.
    Voxel-based indirect lighting: Compelling in theory but extremely messy in practice. I learned a lot and this prompted me to integrate compute shaders into the engine, but ultimately the results I got weren't compelling enough to turn into a finished product. Given the problems everyone else has with this approach, I think it makes sense to focus on hardware raytracing as the next step up in lighting.
    Novel entity component system: My idea was to make it so when you called a component method, all the components that had that method would get called. I had some complicated Lua code set up that handled this, and early builds of Ultra used a C++ preprocessor that analyzed header files and created some complicated wrappers to make C++ work in roughly the same way. Ultimately I decided the confusion wasn't worth the questionable benefit and I implemented a conventional multi-component system like everyone had asked for, without any preprocessor.
    Tabbed Asset Editor: Another thing that sounded good in theory but was not great in practice was to use a single asset editor window with tabs for the different opened files. All it took was a day of using this to realize how much better it was to open each item in its own window. Hunting and clicking file names in tabs is just not fun. I get claustrophobic just looking at this:

    The Results
    Besides the performance, here are the things that I think turned out better than I imagined:
    glTF Integration: Khronos glTF has proven to be a wonderful workflow, and eliminates any "black box" file formats for 3D content. Your game-ready files can always be pulled back into your modeling program, edited, and saved, with no need for secretive conversion pipelines. A wide variety of game-ready content is available for you to use with Ultra, so I am expecting to see no more programmer art!

    Physically-based Rendering: This aligns closely with glTF integration because the file format includes a well-defined PBR materials system. The secret to good PBR is to have good imagery for reflections. To achieve this, I put a lot of work into the volumetric environment probes system that first appeared in Leadwerks. Probe volumes can now be drawn like a brush object for easier creation and precise sizing in the editor. A configurable fade distance for each edge lets you control how probes blend together. Probes also incorporate the sky lighting so you can have seamless transitions between bright outdoor areas and dark enclosed spaces.

    Lua Integration: The VSCode Lua debugger turns Lua into a first-class game programming language and is a dream to work with. The editor scripting integration is in my opinion the best scripting integration of any 3D program I've ever seen. You can access the entire engine API from within editor scripts to add new functionality or modify the program. Additional documentation for the editor's innards will arrive in the coming days.
    Ultra GUI: This took a lot of time but having control over every pixel and event made it worthwhile. Taking control of the user experience by writing my own GUI was one of the best decisions I made. The interface doubles up as the editor's own UI drawn using GDI+ and the in-game UI rendered with Vulkan.

    C++ API: The most significant change is the user of shared pointers, which forever eliminate the problems with uninitialized and invalid pointers games were prone to as they grew more complex. Additionally, the API is generally better thought out and consistent than what we had in Leadwerks.
    Terrain: You wanted more terrain material layers, so you got them. Up to 256, in fact, with fast performance. You can create multiple terrains, position, rotate, and scale them, and even create non-square terrains to fill in any area.

     
    Player Physics: Yes, you can crouch now.
    Pathfinding: It's dynamic and you can create multiple navmeshes, for different areas or differently sized characters. Pathfinding and physics are now decoupled so you can have armies of characters that only use the pathfinding system for movement.
    3D Brush Editing: I was not planning on this feature until the last minute, but you can can create, move, scale, rotate, and shear brushes in any viewport, including the 3D view. the long-asked-for vertex tool is included as well as a face tool for adjusting texture mapping and geometry.
    Last but not least, the engine's multithreaded design is crazy advanced! Your game code runs on its own thread, giving you a full 16 milliseconds to operate without slowing down the renderer. Physics, pathfinding, animation, rendering, and culling all operate on separate threads. It's an amazing feat in engineering that was only possible because I had seen all the bottlenecks people could create in Leadwerks, so I could plan a way around them when designing Ultra.
    Early Access
    The 1.0 release will be marked "Early Access" because there are still a few extra features to add, and we will have a period where changes to the workflow can still be considered based on user feedback. Once decals, particle emitters, VR support, flowgraphs, and brush smooth groups are added, that will be considered version 1.1 (no "early access"). So there will still be a few months of time where I can revise any parts of the workflow you think can be improved. There are also still some compatibility issues with AMD cards that are being resolved, so please be aware of that.
    A big thanks goes out to all the testers who helped me smooth things out before the release. The good times are finally here!
  2. Josh
    In this blog I am going to explain my overarching plan to bring Leadwerks into the future and build our community.
    First, regarding technology, I believe VR is the future. VR right now is basically where mobile was in 2006. The big publishers won't touch it and indies have a small but enthusiastic market they can target without fear of competition from EA. I have an HTC Vive and it is AWESOME! After using it, I don't really want to play games on a 2D monitor anymore. I want to be in the game, but there is still a serious lack of decent content out there.

    At the same time, I think Vulkan is eventually going to be the graphics API we want to migrate to. Although Apple still isn't supporting it, Vulkan provides the best chance at a real cross-platform graphics API in the future.

    We eventually want to move everything over to 64-bit and drop all support for 32-bit Windows (which almost no one has nowadays).
    None of these issues are pressing. In reality, a switch over to Vulkan graphics, if done right, would result in no apparent change in the user experience for you guys. This is more about future-proofing and performance gains.
    There is one last big thing I can do that could have a huge increase in performance. Right now Leadwerks runs your game code, which calls rendering and physics calls when you choose. A concurrent architecture would have four threads running your game logic, physics, rendering, and AI in separate threads, all the time, constantly. The data exchange between these threads is a complicated matter and would likely involve some restrictions on your code. It would also break backwards compatibility with our existing API examples. So this is probably something to discuss with Leadwerks 5, and may be quite a ways off in the future.
    However, it's important that we start developing now with a clear idea of where we want to go. If information changes between now and then I can always change our course in the right direction.
    Where we are today:
    OpenGL 4.0 Windows API / GTK / Cocoa 64/32 bit Sequential architecture Where we are going:
    Vulkan 1.0 Leadwerks GUI on Windows, Mac, Linux 64-bit Concurrent architecture What if VR Flops?
    It doesn't matter. Developing for VR simply means adding in the OpenVR SDK and focusing on performance gains. This isn't like mobile where there's a completely different API and texture formats. VR will not hold back Leadwerks on the PC. It will help it. VR needs a solid 90 FPS, (although it's perfectly fine to lower your quality settings to achieve that). So developing for VR means focusing on performance gains over flexibility. We have a good idea now of how people use Leadwerks and I think we can make those decisions wisely now.
    How to Begin?
    I just outlined a plan to replace some of our low-level APIs and a major architectural change. However, I'm not willing to go back and do that until I feel like we have really polished Leadwerks Game Engine and given the users what they need. There's always a temptation to go back to the low-level details, but that just makes you an eternal tinkerer. Leadwerks is good because it's the easiest way to build your own games. That is the whole reason for Leadwerks to exist and we mustn't forget that.
    If you read our reviews on Steam, it is pretty clear what the user needs. We are sitting right now with a solid 75% ("positive") approval rating, and a community that is actively churning out games. We are almost all the way there, but Leadwerks needs to take one last step to really connect with what the user needs. In my opinion, Leadwerks Game Engine is only as good as the default templates that come with it. If we can't provide polished game examples that look and feel like a real commercial game, it isn't fair to expect the end user to come up with one. I plan to add the following game templates to Leadwerks:
    Shoot-em-up space side-scroller Contra-like side scroller Simple racing game against AI opponents (like 4x4 Evo) RPG / dungeon crawler Next, I plan to add built-in offline help documentation and video tutorials. By default, the editor is going to open the new help system automatically when the program starts. Too many people come on the forum asking questions about YouTube videos from years ago and don't go to the official tutorials still.
    If we are going to make video tutorials, and I know that at some point we are going to revise the GUI, then it makes sense to revise the GUI first. Otherwise we will end up with a lot of video content that looks obsolete once we switch over to a new GUI. This is why I am working on Leadwerks GUI now. (Leadwerks GUI will also make it easier for us to switch the editor over the BlitzMaxNG, so that it can be compiled in 64-bit mode as pure C++.)

    The revised GUI, built-in help system, and additional game templates will raise our approval rating on Steam up to 80%, which is considered "very positive". Once I have a game development system with an 80% positive rating and (by then) 20,000 users on Steam, that gives us quite a bit of leverage. Success breeds success, and I can use those figures to our advantage when negotiating with third parties.
    Here are the steps we will take, in order:
    Leadwerks GUI Add Mac support Built-in offline help with videos New game templates Achieve 80% approval rating on Steam Once we hit the 80% mark, that's when I know Leadwerks is really giving the end user what they need. I can then go back into some of the internals and work on our next iteration of technology:
    Move completely over to 64-bit Vulkan graphics Concurrent multithreaded architecture This plan gives us a path forward into emerging technologies, in balance with the needs of the end user and the realities of the business. We won't have a major interruption of development and the game tournaments will continue as this is happening.
    As always, this plan is subject to change and I may decide to do things differently depending on how circumstances develop.
  3. Josh
    HDR skyboxes are important in PBR rendering because sky reflections get dampened by surface color. It's not really for the sky itself, but rather we don't want bright reflections to get clamped and washed out, so we need colors that go beyond the visible range of 0-1.
    Polyhaven has a large collection of free photo-based HDR environments, but they are all stored in EXR format as sphere maps. What we want are cubemaps stored in a single DDS file, preferably using texture compression.
    We're going to use this online tool to convert the sphere maps to cubemaps. There are other ways to do this, and some may be better, but this is what I was able to find right away. By default, the page shows a background we can use for testing:

    Let's process and save the image. Since we are just testing, it's fine to use a low resolution for now:

    The layout is very important. We will select the last option, which exports each face as a separate image:

    Processing and saving the image will result in a zip file downloaded to your computer, with six files:
    px.hdr nx.hdr py.hdr ny.hdr pz.hdr nz.hdr These images correspond to the positive and negative directions on the X, Y, and Z axes.
    We will start by just converting a single image, px.hdr. We will convert this pixmap to the format TEXTURE_BC6H, which corresponds to VK_FORMAT_BC6H_UFLOAT_BLOCK in Vulkan, or DXGI_FORMAT_BC6H_UF16 in the DDS file format. We need to load the FreeImage plugin for HDR file import, and the ISPC plugin for fast BC6H compression. The R16G16B16A16_SFLOAT is an intermediate format we need to convert the 32-bit floating point HDR file to, because the ISPC compressor expects this as the input format for BC6H compression:
    auto plg = LoadPlugin("Plugins/FITextureLoader"); auto plg2 = LoadPlugin("Plugins/ISPCTexComp"); auto pixmap = LoadPixmap(GetPath(PATH_DESKTOP) + "/px.hdr"); pixmap = pixmap->Convert(TextureFormat(VK_FORMAT_R16G16B16A16_SFLOAT)); pixmap = pixmap->Convert(TEXTURE_BC6H); pixmap->Save(GetPath(PATH_DESKTOP) + "/px.dds"); When we open the resulting DDS file in Visual Studio it appear quite dark, but I think this is due to how the HDR image is stored. I'm guessing colors are being scaled to a range between zero and one in the HDR file:

    Now that we know how to convert a save a single 2D texture, let's try saving a more complex cube map texture with mipmaps. to do this, we need to pass an an array of pixmaps to the SaveTexture function, containing all the mipmaps in the DDS file, in the correct order. Microsoft's documentation on the DDS file format is very helpful here.
    std::vector<std::shared_ptr<Pixmap> > mipchain; WString files[6] = { "px.hdr", "nx.hdr", "py.hdr", "ny.hdr", "pz.hdr", "nz.hdr" }; for (int n = 0; n < 6; ++n) { auto pixmap = LoadPixmap(GetPath(PATH_DESKTOP) + "/" + files[n]); pixmap = pixmap->Convert(TextureFormat(VK_FORMAT_R16G16B16A16_SFLOAT)); Assert(pixmap); mipchain.push_back(pixmap->Convert(TEXTURE_BC6H)); while (true) { auto size = pixmap->size; size /= 2; pixmap = pixmap->Resize(size.x, size.y); Assert(pixmap); mipchain.push_back(pixmap->Convert(TEXTURE_BC6H)); if (size.x == 4 and size.y == 4) break; } } SaveTexture(GetPath(PATH_DESKTOP) + "/skybox.dds", TEXTURE_CUBE, mipchain, 6); when we open the resulting DDS file in Visual Studio we can select different cube map faces and mipmap levels, and see that things generally look like we expect them to:

    Now is a good time to load our cube map up in the engine and make sure it does what we think it does:
    auto skybox = LoadTexture(GetPath(PATH_DESKTOP) + "/skybox.dds"); Assert(skybox); world->SetSkybox(skybox); When we run the program we can see the skybox appearing. It still appears very dark in the engine:

    We can change the sky color to lighten it up, and the image comes alive. We don't have any problems with the dark colors getting washed out, because this is an HDR image:
    world->SetSkyColor(4.0);
    Now that we have determined that our pipeline works, let's try converting a sky image at high resolution. I will use this image from Polyhaven because it has a nice interesting sky. However, there's one problem. Polyhaven stores the image in EXR format, and the cube map generator I am using only loads HDR files. I used this online converter to convert my EXR to HDR format, but there are probably lots of other tools that can do the job.
    Once we have the new skybox loaded in the cubemap generator, we will again process and save it, but this time at full resolution:

    Save the file like before and run the DDS creation code. If you are running in debug mode, it will take a lot longer to process the image this time, but when it's finished the results will be worthwhile!:

    This will be very interesting to view in VR. Our final DDS file is 120 MB, much smaller than the 768 MB an uncompressed R16G16B16A16 image would require. This gives us better performance and requires a lot less hard drive space. When we open the file in Visual Studio we can browser through the different faces and mipmap levels, and everything seems to be in order:

    Undoubtedly this process will get easier and more automated with visual tools, but it's important to get the technical basis laid down first. With complete and in-depth support for the DDS file format, we can rely on a widely supported format for textures, instead of black-box file formats that lock your game content away.
  4. Josh
    This tutorial demonstrates how to create a high-quality skybox for Leadwerks Game Engine using Vue.
    Download
    Cloudy Blue Skies.zip FixVueCubemap.zip Required Third-Party Programs
    Vue Esprit Exporter Module Loading the Example
    Run Vue and select the File > Open menu item.  Extract the zip file above and open the file "Cloudy Blue Skies.vue".

    Atmosphere and Clouds
    You can modify the appearance of the sky with the Atmosphere Editor. Select the Atmosphere > Atmosphere Editor menu item to open this dialog.

    The clouds tab lets you adjust various properties of the cloud layers and add new ones. Skyboxes look best with multiple layers of different kinds of clouds, so don't expect to get the perfect look with just one layer.

    The load button to the right side of the cloud layer list will let you select from a wide range of different cloud types.

    Experiment with different cloud layers to get the look you want. The "Detail amount" setting in particular will really enhance the image, but don't overdo it. You can right-click and drag the mouse to look around in the main panel, so be sure to take a look around to see how the clouds affect the entire sky.
    Lighting
    To edit the sunlight properties in Vue, select the sunlight object in the World Browser on the right side of the main window.

    You can match the exact rotation of the default sunlight angle in Leadwerks to make your skybox line up exactly to the scene lighting. The default sunlight angle in Leadwerks is (55,-35,0). In Vue this corresponds to the values (145,0,215). To get these values we add 90 degrees to the pitch and subtract the yaw from 180. Note in Vue the order of the yaw and roll are switched.

    The sun color is very important for the overall composition of our image. In real life we're used to seeing a very high range of light levels in the sky. Computer monitors cannot represent the same range of colors, so images can easily become washed out and lose details. We want to adjust the sun color so we can get the most detail within the spectrum of a 32-bit color display. Like the rotation, the sun color can be modified in the sun properties.

    If the sunlight color is too bright, the image will be overexposed and the cloud shape will become washed out.

    If the sunlight is too dark, it will look gray and desaturated.

    The right sun brightness will give a balanced look between bright spots and shadows. This is the part in the process that requires the most artistic sense. It's a good idea to look at some screenshots or photos for comparison as you adjust your settings.

    You will get quite a lot of variation in brightness across the sky, so be sure to take a look around the whole scene when you are adjusting lighting. You can also adjust the main camera's exposure value to vary the brightness of the rendered image.
    If you want to hide the sun from view you can do this by setting the "Size of the sun" and "Size of the corona" settings both to zero under the "Sun" tab in the Atmosphere Editor. Exporting
    To export our skybox the exporter module must be installed. Select the File > Export Sky menu item and the export dialog will appear.

    The "Supporting geometry" setting should be set to "Cube". Set the X value to 1536 and the Y value to 2048. This controls the width and height of the saved image. When we press the OK button, the sky will be rendered out into a vertical cube cross with those dimensions. Each face of the cubemap will be 512x512 pixels.

    By default, your skybox will be exported to the file "Documents\e-on software\Vue 2015\Objects\Atmosphere.bmp". The exported cube cross is a nonstandard orientation. To convert this into a cubemap strip ready to load in Leadwerks, use the FixVueCubemap.exe utility posted above.  Drag your exported image file onto the executable, and it will save out a cube strip in PNG format that is ready to load in Leadwerks.

    Importing
    To import your skybox into Leadwerks, just drag the cubemap strip PNG file onto the Leadwerks main window. Open the converted texture from the Leadwerks Asset Browser. Set the texture mode to "Cubemap", uncheck the "Generate Mipmaps" checkbox, and check the clamp mode for the X, Y, and Z axes. Press the Save button to reconvert the texture and it will appear in a 3D view.

    You can use the skybox in the current map by selecting it in the scene settings.

    Disabling mipmap generation will reduce the size of a 1024x1024 cubemap from 32 to 24 mb. Due to the way the image is displayed, mipmaps aren't needed anyways. Final Render
    For the final render, we want each cubemap face to be 1024x1024 pixels. However, we can get a better quality image if we render at a larger resolution and then downsample the image. In Vue, select the File > Export menu item again to open the export dialog. This time enter 6144 for the X value and 8192 for the Y value. Don't press the OK button until you are ready to take a long break, because the image will take a long time to render. When you're done you will have a huge image file of your skybox with a 2048x2048 area for each cubemap face.
    If we resize the image file in a regular paint program, it will create seams along the edges of the cubemap faces. Instead, we're going to pass a parameter to the conversion utility to tell it to downsample the image by a factor of 50%. The "downsample.bat" file is set up to do this, so just double-click on this to launch the executable with the correct parameters. The resulting cubemap strip will be 6144x1024 pixels, with a 1024x1024 area for each face. However, due to the original rendering resolution this will appear less grainy then if we had rendered directly to this resolution.
    Import this texture into Leadwerks as before and enjoy your finished high-quality skybox. Always do a low-resolution pass before rendering the final image, as it can take a long time to process.
  5. Josh

    Articles
    I haven't been blogging much because I am making so much progress that there would be too much to cover. In this article I will talk about one little system I spent the last week on and show the improvements I have made to the workflow over Leadwerks.
    In the asset editor, when the root of a model is selected, a "Collider" field appears in the physics properties. You can use this to quickly calculate and apply a collider to the model.

    The convex hull collider includes an option for tolerance. You can adjust this to get a more or less precise convex hull around the model:

    There is also an "optimize" option for the mesh collider. When this is activated it will attempt to merge adjacent coplanar polygons into a single face, and can result in fewer seams on colliding faces.
    Two new shapes are included, the capsule and the chamfer cylinder, so now all the shapes supported by Newton Dynamics are available. (The chamfer cylinder is good for vehicle tires.)
     
    Unlike Leadwerks, you no longer need to specify the axis cylindrical shapes are oriented around. The editor will calculate which bounding box axes are most like each other, and use the third axis to orient the shape. This works for long skinny objects:

    And it also works for short fat objects:

    It will even detect which direction a cone should be facing, by calculating the total distance from the shape the model vertices are with each direction, and selecting the orientation with the smallest error value. (A single cone is probably not the best choice of shape for this model, but the point is that it correctly guesses which direction the cone should point based on the model vertices.)

    For more advanced geometry, the convex decomposition tool is available in the Utilities panel. This exposes all the parameters available in V-HACD 4.0 and processes the task on a separate thread so you can continue to use the editor as the task runs in the background. It usually only takes a few seconds to compete, but it is nice to not have your workflow interrupted:

    You can export colliders as model files, so it's also possible to just use this to generate convex hulls and export them to .obj. (I could get a better fitting shape if I spent more time adjusting the settings, but I just quickly made this shape with very low settings in order to get this screenshot.)

    The new .collider file format, like everything else in Ultra, is JSON-based. Simple shapes can be stored in pure ASCII text, so they can be edited by hand if needed:
    { "collider": { "parts": [ { "offset": [ 0.0024815797805786133, -0.18715500831604004, -1.055002212524414e-05 ], "rotation": [ 0.0, 0.0, 0.0 ], "shape": "BOX", "size": [ 1.8899539709091187, 2.0, 1.8019688129425049 ] } ] } } More complex shapes like convex hulls and meshes use binary data appended to the end of the file to store their contents, so that they load quickly in the engine:
    { "collider": { "parts": [ { "data": 0, "shape": "CONVEXHULL", "vertices": 61 } ] } }�n�2��<�5�j�k��#�I3��ch�������=b[��I<��~d�I�D�` �[�\{4���A���?XO3���>��վ�v!���f��� ?]����I�!�����V� >�?���mdG�t9�a��@���X?W��0o��e�-�D��h�P?>����>��׾���@��=T�U?��ž�{����w�� ���G��3��ޑ?>V?c�����;���6�s�/?&�=����?q�e?j��N���[#b��bݽ(��>��Ǿ�ڽ�E��MV�ַؽ��8<�G������D?BYT?I���@"N?6?�";-�$?��콜�";��t>8�"�t�";�z��3�0���";����ݐ�)�%=}P?��0?���=/�E?BYT?'>�=�E��MV����={; ?"����>�־��c� �>��?q�e?�$>�ӽ��P��@>��,?�j|=�2�>[o/���P�C��>��?>V?Z�>�������Қ�>��h���@�>��E�CO2��.�>T��[A�y]�>L��=T�U?w4�>���>�쾛?#o��e�-�r?�����U?�"?.�v>��!?�R?�zv����>�@?(�0�S �>.(?�n_�C�?�|-?d�[��}?��4?��>��վ��6?2M4��*?�>?P��=_�s��F?V ���ྲAm?�#�I3�Lr?����� Leadwerks .phy collider files can also be loaded.
    Notice in the shots above, in the drop-down menu there is also a "Browse" option, so you can create a collider file with a low-resolution model and then load the resulting collider to use with a high-resolution model.
    I think this design strikes the right balance to let you do simple things quickly, but still allow for more advanced options when you need them. In the future I would like to add an additional in-depth collider editing utility that would allow you to select each sub-object, select model polygons and generate shapes just from those polys, and other advanced features.
    The pick mode feature has been revised. The original design was based on Blitz3D, which did not support collision primitives at all. Instead of specifying a sphere or other shape for picking, you can now just tell the entity to use the collider for picking. (PICK_SPHERE is removed.)
    On an unrelated note, I had to adjust the camera range based on the distance from the model being viewed, in order to make very small objects visible. This allows extreme close-up shots like in the blog header image that you probably aren't used to seeing except in VR. It's kind of interesting.
  6. Josh
    Not long ago, I wrote about my experiments with AI-generated textures for games. I think the general consensus at the time was that the technology was interesting but not very useful in its form at the time. Recently, I had reason to look into the OpenAI development SDK, because I wanted to see if it was possible to automatically convert our C++ documentation into documentation for Lua. While looking at that, I started poking around with the image generation API, which is now using DALL-E 2. Step by step, I was able to implement AI texture generation in the new editor and game engine, using only a Lua script file and an external DLL. This is available in version 1.0.2 right now:

    Let's take a deep dive into how this works...
    Extension Script
    The extension is run by placing a file called "OpenAI.lua" in the "Scripts/Start/Extensions" folder. Everything in the Start folder gets run automatically when the editor starts up, in no particular order. At the top of the script we create a Lua table and load a DLL module that contains a few functions we need:
    local extension = {} extension.openai = require "openai" Next we declare a function that is used to process events. We can skip its inner workings for now:
    function extension.hook(event, extension) end We need a way for the user to activate our extension, so we will add a menu item to the "Scripting" submenu. The ListenEvent call will cause our hook function to get called whenever the user selects the menu item for this extension.  Note that we are passing the extension table itself in the event listener's extra parameter. There is no need for us to use a global variable for the extension table, and it's better that we don't.
    local menu = program.menu:FindChild("Scripting", false) if menu ~= nil then local submenu = menu:FindChild("OpenAI", false) if submenu == nil then submenu = CreateMenu("", menu)-- divider submenu = CreateMenu("OpenAI", menu) end extension.menuitem = CreateMenu("Text to Image", submenu) end ListenEvent(EVENT_WIDGETACTION, extension.menuitem, extension.hook, extension) This gives us a menu item we can use to bring up our extension's window.

    The next section creates a window, creates a user interface, and adds some widgets to it. I won't paste the whole thing here, but you can look at the script to see the rest:
    extension.window = CreateWindow("Text to Image", 0, 0, winx, winy, program.window, WINDOW_HIDDEN | WINDOW_CENTER | WINDOW_TITLEBAR) Note the window is using the WINDOW_HIDDEN style flag so it is not visible when the program starts. We're also going to add event listeners to detect when the window is closed, and when a button is pressed:
    ListenEvent(EVENT_WINDOWCLOSE, extension.window, extension.hook, extension) ListenEvent(EVENT_WIDGETACTION, extension.button, extension.hook, extension) The resulting tool window will look something like this:

    Now let's take a look at that hook function. We made three calls to ListenEvent, so that means we have three things the function needs to evaluate. Selecting the menu item for this extension will cause our hidden window to become visible and be activated:
    elseif event.id == EVENT_WIDGETACTION then if event.source == extension.menuitem then extension.window:SetHidden(false) extension.window:Activate() When the user closes the close button on the tool window, the window gets hidden and the main program window is activated:
    if event.id == EVENT_WINDOWCLOSE then if event.source == extension.window then extension.window:SetHidden(true) program.window:Activate() end Finally, we get to the real point of this extension, and write the code that should be executed when the Generate button is pressed. First we get the API key from the text field, passing it to the Lua module DLL by calling openal.setapikey.
    elseif event.source == extension.button then local apikey = extension.apikeyfield:GetText() if apikey == "" then Notify("API key is missing", "Error", true) return false end extension.openai.setapikey(apikey) Next we get the user's description of the image they want, and figure out what size it should be generated at. Smaller images generate faster and cost a little bit less, if you are using a paid OpenAL plan, so they can be good for testing ideas. The maximum size for images is currently 1021x1024.
    local prompt = extension.promptfield:GetText() local sz = 512 local i = extension.sizefield:GetSelectedItem() if i == 1 then sz = 256 elseif i == 3 then sz = 1024 end The next step is to copy the user's settings into the program settings so they will get saved when the program closes. Since the main program is using a C++ table for the settings, both Lua and the main program can easily share the same information:
    --Save settings if type(program.settings.extensions) ~= "userdata" then program.settings.extensions = {} end if type(program.settings.extensions.openai) ~= "userdata" then program.settings.extensions.openai = {} end program.settings.extensions.openai.apikey = apikey program.settings.extensions.openai.prompt = prompt program.settings.extensions.openai.size = {} program.settings.extensions.openai.size[1] = sz program.settings.extensions.openai.size[2] = sz Extensions should save their settings in a sub-table in the "extensions" table, so keep data separate from the main program and other extensions. When these settings are saved in the settings.json file, they will look like this. Although generated images must be square, I opted to save both width and height in the settings, for possible future compatibility.
    "extensions": { "openai": { "apikey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "prompt": "tiling seamless texture warehouse painted concrete wall abandoned dirty", "size": [ 1024, 1024 ] } }, Finally, we call the module function to generate the image, which may take a couple of minutes. If its successful we load the resulting image as a pixmap, create a texture from it, and then open that texture in a new asset editor window. This is done to eliminate the asset path, so the asset editor doesn't know where the file was loaded from. We also make a call to AssetEditor:Modify, which will cause the window to display a prompt if it is closed without saving. This prevents the user's project folder from filling up with a lot of garbage images they don't want to keep.
    if extension.openai.newimage(sz, sz, prompt, savepath) then local pixmap = LoadPixmap(savepath) if pixmap ~= nil then local tex = CreateTexture(TEXTURE_2D, pixmap.size.x, pixmap.size.y, pixmap.format, {pixmap}) tex.name = "New texture" local asseteditor = program:OpenAsset(tex) if asseteditor ~= nil then asseteditor:Modify() else Print("Error: Failed to open texture in asset editor."); end end else Print("Error: Image generation failed.") end The resulting extension provides an interface we can use to generate a variety of interesting textures. I think you will agree, these are quite a lot better than what we had just a few months ago.





    Of course the Lua debugger in Visual Studio Code came in very handy while developing this.

    That's pretty much all there is to the Lua side of things. Now let's take a closer look at the module code.
    The Module
    Lua modules provide a mechanism whereby Lua can execute C++ code packed into a dynamic linked library. The DLL needs to contain one retired function, which is luaopen_ plus the name of the module, without any extension. The module file is "openai.dll" so we will declare a function called luaopen_openai:
    extern "C" { __declspec(dllexport) int luaopen_openai(lua_State* L) { lua_newtable(L); int sz = lua_gettop(L); lua_pushcfunction(L, openai_newimage); lua_setfield(L, -2, "newimage"); lua_pushcfunction(L, openai_setapikey); lua_setfield(L, -2, "setapikey"); lua_pushcfunction(L, openai_getlog); lua_setfield(L, -2, "getlog"); lua_settop(L, sz); return 1; } } This function creates a new table and adds some function pointers to it, and returns the table. (This is the table we will store in extension.openai). The functions are setapikey(), getlog() and newimage().
    The first function is very simple, and just provides a way for the script to send the user's API key to the module:
    int openai_setapikey(lua_State* L) { APIKEY.clear(); if (lua_isstring(L, -1)) APIKEY = lua_tostring(L, -1); return 0; } The getlog function just returns any printed text, for extra debugging:
    int openai_getlog(lua_State* L) { lua_pushstring(L, logtext.c_str()); logtext.clear(); return 1; } The newimage function is where the action is at, but there's actually two overloads of it. The first one is the "real" function, and the second one is a wrapper that extracts the right function arguments from Lua, and then calls the real function. I'd say the hardest part of all this is interfacing with the Lua stack, but if you just go carefully you can follow the right pattern.
    bool openai_newimage(const int width, const int height, const std::string& prompt, const std::string& path) int openai_newimage(lua_State* L) This is done so the module can be easily compiled and tested as an executable.
    The real newimage function is where all the action is. It sets up a curl instance and communicates with a web server. There's quite a lot of error checking in the response, so don't let that confused you. If the call is successful, then a second curl object is created in order to download the resulting image. This must be done before the curl connection is closed, as the server will not allow access after that happens:
    bool openai_newimage(const int width, const int height, const std::string& prompt, const std::string& path) { bool success = false; if (width != height or (width != 256 and width != 512 and width != 1024)) { Print("Error: Image dimensions must be 256x256, 512x512, or 1024x1024."); return false; } std::string imageurl; if (APIKEY.empty()) return 0; std::string url = "https://api.openai.com/v1/images/generations"; std::string readBuffer; std::string bearerTokenHeader = "Authorization: Bearer " + APIKEY; std::string contentType = "Content-Type: application/json"; auto curl = curl_easy_init(); struct curl_slist* headers = NULL; headers = curl_slist_append(headers, bearerTokenHeader.c_str()); headers = curl_slist_append(headers, contentType.c_str()); nlohmann::json j3; j3["prompt"] = prompt; j3["n"] = 1; switch (width) { case 256: j3["size"] = "256x256"; break; case 512: j3["size"] = "512x512"; break; case 1024: j3["size"] = "1024x1024"; break; } std::string postfields = j3.dump(); curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers); curl_easy_setopt(curl, CURLOPT_URL, url.c_str()); curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postfields.c_str()); curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback); curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer); auto errcode = curl_easy_perform(curl); if (errcode == CURLE_OK) { //OutputDebugStringA(readBuffer.c_str()); trim(readBuffer); if (readBuffer.size() > 1 and readBuffer[0] == '{' and readBuffer[readBuffer.size() - 1] == '}') { j3 = nlohmann::json::parse(readBuffer); if (j3.is_object()) { if (j3["error"].is_object()) { if (j3["error"]["message"].is_string()) { std::string msg = j3["error"]["message"]; msg = "Error: " + msg; Print(msg.c_str()); } else { Print("Error: Unknown error."); } } else { if (j3["data"].is_array() or j3["data"].size() == 0) { if (j3["data"][0]["url"].is_string()) { std::string s = j3["data"][0]["url"]; imageurl = s; // I don't know why the extra string is needed here... readBuffer.clear(); // Download the image file auto curl2 = curl_easy_init(); curl_easy_setopt(curl2, CURLOPT_URL, imageurl.c_str()); curl_easy_setopt(curl2, CURLOPT_WRITEFUNCTION, WriteCallback); curl_easy_setopt(curl2, CURLOPT_WRITEDATA, &readBuffer); auto errcode = curl_easy_perform(curl2); if (errcode == CURLE_OK) { FILE* file = fopen(path.c_str(), "wb"); if (file == NULL) { Print("Error: Failed to write file."); } else { auto w = fwrite(readBuffer.c_str(), 1, readBuffer.size(), file); if (w == readBuffer.size()) { success = true; } else { Print("Error: Failed to write file data."); } fclose(file); } } else { Print("Error: Failed to download image."); } curl_easy_cleanup(curl2); } else { Print("Error: Image URL missing."); } } else { Print("Error: Data is not an array, or data is empty."); } } } else { Print("Error: Response is not a valid JSON object."); Print(readBuffer); } } else { Print("Error: Response is not a valid JSON object."); Print(readBuffer); } } else { Print("Error: Request failed."); } curl_easy_cleanup(curl); return success; } My first attempt at this used a third-party C++ library for OpenAI, but I actually found it was easier to just make the low-level CURL calls myself.
    Now here's the kicker: Any web API in the world will work with almost the exact same code. You now know how to build extensions that communicate with any website with a web interface, including SketchFab, CGTrader, itch.io, and easily interact with them in the Ultra Engine editor. The full source code for this and other Ultra Engine modules is available here:
    https://github.com/UltraEngine/Lua
    This integration of text-to-image tends to do well with base textures that have a uniform distribution of detail. Rock, concrete, ground, plaster, and other materials look great with the DALL-E 2 generator. It doesn't do as well with geometry details and complex structures, since the AI has no real understanding of the purpose of things. The point of this integration was not to make the end-all-be-all AI texture generation. The technology is changing rapidly and will undoubtedly continue to advance. Rather, the point of this exercise was to demonstrate a complex feature added in a Lua extension, without being built into the editor's code. By releasing the source for this I hope to help people start developing their own extensions to add new peripheral features they want to see in the editor and game engine.
     
  7. Josh
    Along with Leadwerks GUI, Leadwerks 4.4 adds an in-game menu that is available with the default Lua scripted game.  You can use this to allow your users to adjust settings in the game, or provide a more sophisticated way to quit the game than simply pressing the escape key.

    The default window size has been changed to 1280x720 when run from the editor.  Your game will now run in fullscreen mode by default when it is launched outside the editor.
    All of these changes are contained in the Main.lua and Menu.lua scripts, and can all be modified or removed to your heart's content.
    A full build is available now on the beta branch.
  8. Josh
    Package plugins are now supported in Ultra Engine 1.0.2. This allows the engine to handle other package formats besides just ZIP. In order to test this out, I created the Quake Loader plugin, which currently supports the following formats used in the original Quake game:
    PAK WAD BSP (textures) SPR LMP Why Quake? Well, the original Quake game is what got me into game development through modding, but Quake is also great for testing because it's so weird. This game was written before hardware accelerated graphics existed, so it has a lot of conventions that don't match modern 3D standards:
    All animation is vertex morphing (no skeletal animation) Texture coordinates are in pixels / integers instead of floating point values Three different types of "packages" (PAK, WAD, BSP), the latter two of which can be stored inside the first (packages-in-packages) Image files are stored seven different ways: LMP image files SPR sprite files Color pallette ('@') Mipmapped texture in WAD ('D') Console picture ('E') Status bar pictures ('B') Mipmapped textures stored in BSP Each of the image storage methods store data in slightly different ways. Sometimes the width and height are packed into the image data, sometimes they are found elsewhere in a separate structure, sometimes they are hard-coded, and sometimes you have to guess the image dimensions based on the data size. It's a mess, and it seems like this was made up on-the-fly by several people, but this was cutting-edge technology at the time and I don't think usability outside the specified game they were working on was a big concern.
    There are some interesting challenges there, but this forms the basis of several dated but influential games. As time went by, the design became more and more sane. For example, Quake 3 just stores textures as TGA images in a ZIP file with the extension changed to PK3. So I figure if I can make the plugin system flexible enough to work with Quake then it will be able to handle anything.
    Originally I tried implementing this using HLLib from Nem's Tools. This library supports Quake formats and more, but I had a lot of problems with it, and got better results when I just wrote my own PAK loading code. WAD files were an interesting challenge. There is no real Quake texture format, so loading the raw texture data from the package made no sense. The most interesting breakthrough in this process was how I handled WAD textures. I finally figured out I can make the WAD loader return a block of data that forms a DDS file when a texture is loaded, even through the texture name has no extension. This tricks the engine into thinking the WAD package contains a DDS file which can easily be loaded, when the reality is quite different. This introduces an important concept, that package plugins don't necessarily have to return the exact data they contain, but instead can return files in a ready-to-use format, and the game engine will never know the difference.
    The resulting plugin allows you to easily extract and load textures from the Quake game files. This code will extract textures from a single WAD file.
    #include "UltraEngine.h" using namespace UltraEngine; void main(int argc, const char* argv[]) { // Load Quake file loader plugin auto qplg = LoadPlugin("Plugins/QuakeLoader"); // Load FreeImage texture plugin auto fiplg = LoadPlugin("Plugins/FITextureLoader"); // WAD to download WString wadfile = "bigcastle"; // Download a WAD package DownloadFile("https://www.quaketastic.com/files/texture_wads/" + wadfile + ".wad", wadfile + ".wad"); // Load the package auto wad = LoadPackage(wadfile + ".wad"); // Create a subdirectory to save images in CreateDir(wadfile); // Read all files in the package auto dir = wad->LoadDir(""); for (auto file : dir) { // Load the image auto pm = LoadPixmap(file); // Save as a PNG file if (pm) pm->Save(wadfile + "/" + file + ".png"); } OpenDir(wadfile); } All the files found in the WAD package are saved as PNG images. BSP files will work exactly the same way.

    This code will load a Quake package, extract all the textures from all the maps in the game, and save them to a folder on the desktop. This is done by detecting packages-in-packages (WAD and BSP), which return the file type '3', indicating that they can be treated both as a file and as a folder. Since none of these package formats use any compression, pixmaps can be easily loaded straight out of the file without extracting the entire BSP. Since Quake PAK files don't use compression, the whole system just turns into a very complicated blob of data with pointers that store data all over the place:
    #include "UltraEngine.h" using namespace UltraEngine; void main(int argc, const char* argv[]) { // Load Quake file loader plugin auto qplg = LoadPlugin("Plugins/QuakeLoader"); // Load FreeImage texture plugin auto fiplg = LoadPlugin("Plugins/FITextureLoader"); // Path to Quake game WString qpath = "C:/Program Files (x86)/Steam/steamapps/common/Quake"; //WString qpath = "D:/SteamLibrary/steamapps/common/Quake"; // Load game package auto pak = LoadPackage(qpath + "/id1/PAK1.PAK"); //Change the current directory to load files with relative paths ChangeDir(qpath + "/id1"); // Create a folder to save images in WString savedir = GetPath(PATH_DESKTOP) + "/Quake"; CreateDir(savedir); // Load the main package directory auto dir = pak->LoadDir("maps"); for (auto file : dir) { // Print the file name Print(file); // Get the file type int type = FileType("maps/" + file); // If a package-in-a-package is found load its contents (BSP and WAD) if (type == 3) { auto subdir = pak->LoadDir("maps/" + file); for (int n = 0; n < subdir.size(); ++n) { // Load the texture (plugin will return DDS files) auto pm = LoadPixmap("maps/" + file + "/" + subdir[n]); // Save to PNG if (pm) pm->Save(savedir + "/" + subdir[n] + ".png"); } } } OpenDir(savedir); } Here is the result:

    Since all of these images can be loaded as pixmaps, does that mean they can also be loaded as a texture? I had to know the answer, so I tried this code:
    #include "UltraEngine.h" using namespace UltraEngine; int main(int argc, const char* argv[]) { //Get the displays auto displays = GetDisplays(); //Create a window auto window = CreateWindow("Ultra Engine", 0, 0, 1280, 720, displays[0], WINDOW_CENTER | WINDOW_TITLEBAR); //Create a world auto world = CreateWorld(); //Create a framebuffer auto framebuffer = CreateFramebuffer(window); //Create a camera auto camera = CreateCamera(world); camera->SetClearColor(0.125); camera->SetPosition(0, 0, -2); camera->SetFov(70); //Create a light auto light = CreateBoxLight(world); light->SetRotation(45, 35, 0); light->SetRange(-10, 10); light->SetColor(2); //Create a model auto model = CreateBox(world); // Load Quake file loader plugin auto qplg = LoadPlugin("Plugins/QuakeLoader"); // WAD to download WString wadfile = "bigcastle"; // Download a WAD package DownloadFile("https://www.quaketastic.com/files/texture_wads/" + wadfile + ".wad", wadfile + ".wad"); // Load the package auto wad = LoadPackage(wadfile + ".wad"); // Load a texture from the WAD package auto tex = LoadTexture("sclock"); // Create a material auto mtl = CreateMaterial(); mtl->SetTexture(tex); // Apply the material to the box model->SetMaterial(mtl); //Main loop while (window->Closed() == false and window->KeyDown(KEY_ESCAPE) == false) { model->Turn(0, 1, 0); world->Update(); world->Render(framebuffer); } return 0; } Here is the result, a texture loaded straight out of a Quake WAD file!

    To reiterate, testing with Quake files helped me to come up with two important design features:
    Packages-in-packages, indicated with the file type '3'. Packages do not necessarily have to return the same exact data they contain, and can instead prepare the data in a ready-to-use format, when that is desirable. The decision to base file type detection on the contents of the file instead of the file name extension worked well here, and allowed me to load the extension-less WAD texture names as DDS files. Some of these images came from "bigcastle.wad". I don't know who the original author is.
    If you are interested, you can read more about the weird world of Quake file formats here, although I must warn you that specification is not complete! 
  9. Josh

    Articles
    Midjourney is an AI art generator you can interact with on Discord to make content for your game engine. To use it, first join the Discord channel and enter one of the "newbie" rooms. To generate a new image, just type "/imagine" followed by the keywords you want to use. The more descriptive you are, the better. After a few moments four different images will be shown. You can upsample or create new variations of any of the images the algorithm creates.

    And then the magic begins:

    Here are some of the images I "created" in a few minutes using the tool:

    I'm really surprised by the results. I didn't think it was possible for AI to demonstrate this level of spatial reasoning. You can clearly see that it has some kind of understanding of 3D perspective and lighting. Small errors like the misspelling of "Quake" as "Quke" only make it creepier, because it means the AI has a deep level of understanding and isn't just copying and pasting parts of images.
    What do you think about AI-generated artwork? Do you have any of your own images you would like to show off? Let me know in the comments below.
  10. Josh

    Articles
    As I have explained before, I plan for Ultra Engine to use glTF for our main 3D model file format, so that your final game models can be easily loaded back into a modeling program for editing whenever you need. glTF supports a lot of useful features and is widely supported, but there are a few missing pieces of information I need to add into it. Fortunately, this JSON-based file format has a mechanism for extensions that add new features and data to the format. In this article I will describe the custom extensions I am adding for Ultra Engine.
    ULTRA_triangle_quads
    All glTF models are triangle meshes, but we want to support quad meshes primarily because its better for tessellation. This extension gets added to the primitives block. If the "quads" value is set to true, this indicates that the triangle indices are stored in a manner such that the first four indices of every six indices form a quad:
    "extensions": { "ULTRA_triangle_quads": { "quads": true } } There is no other glTF extension for quads, and so there is no way to export a glTF quad mesh from any modeling program. To get quad meshes into Ultra Engine you can load an OBJ file and then resave it as glTF. Here is a glTF file using quads that was created this way. You can see the tessellation creates an even distribution of polygons:

    For comparison, here is the same mesh saved as triangles and tessellated. The long thin triangles result in a very uneven distribution of polygons. Not good!

    The mesh still stores triangle data so the file can be loaded back into a 3D modeling program without any issues.
    Here is another comparison that shows how triangle (on the left) and quads (on the right) tessellate:

    ULTRA_material_displacement
    This extension adds displacement maps to glTF materials, in a manner that is consistent with how other textures are stored:
    "ULTRA_material_displacement": { "displacementTexture": { "index": 3, "offset": -0.035, "strength": 0.05 } } The extension indicates a texture index, a maximum displacement value in meters, and a uniform offset, also in meters. This can be used to store material displacement data for tessellation or parallax mapping. Here is a model loaded straight from a glTF file with displacement info and tessellation:

     
    If the file is loaded in other programs, the displacement info will just be skipped.
    ULTRA_vertex_displacement
    Our game engine uses a per-vertex displacement factor to control how displacement maps affect geometry. This extension adds an extra attribute into the primitives structure to store these values:
    "primitives": [ { "attributes": { "NORMAL": 1, "POSITION": 0, "TEXCOORD_0": 2 }, "indices": 3, "material": 0, "mode": 4, "extensions": { "ULTRA_vertex_displacement": { "DISPLACEMENT": 7 } } } } This can be used to prevent cracks from appearing  at texcoord seams.

    Here you can see the displacement value being loaded back from a glTF file it has been saved into. I'm using the vertex color to visually verify that it's working right:

    ULTRA_extended_material
    This extension adds other custom parameters that Ultra Engine uses. glTF handles almost everything we want to do, and there are just a few settings to add. Since the Ultra Engine material format is JSON-based, it's easy to just insert the extra parameters into the glTF like so:
    "ULTRA_extended_material": { "shaderFamily": "PBR", "shadow": true, "tessellation": true } In reality I do not feel that this extension is very well-defined and do not expect it to see any adoption outside of Ultra Engine. I made the displacement parameters a separate extension because they are well-defined, and there might be an opportunity to work with other application developers using that extension.
    Here we can see the per-material shadow property is disabled when it is loaded from the glTF:

    For comparison, here is what the default setting looks like:

    These extensions are simply meant to add special information that Ultra Engine uses into the glTF format. I do not currently have plans to try to get other developers to adopt my standards. I just want to add the extra information that our game engine needs, while also ensuring compatibility with the rest of the glTF ecosystem. If you are writing a glTF importer or exporter and would like to work together to improve the compatibility of our applications, let me know!
    I used the Rock Moss Set 02 model pack from Polyhaven in this article.
  11. Josh

    Articles
    I have two goals for the art pipeline in the new game engine.
    Eliminate one-way 3D model conversions. Use the same game asset files on all platforms. In Leadwerks, 3D models get converted into the proprietary MDL format. Although the format is pretty straightforward and simple, there is no widespread support for loading it back into 3D modeling programs. This means you need to keep a copy of your 3D model in FBX and MDL format. You may possibly want to keep an additional file for the modeling program it was made in, such as 3D Studio Max MAX files. I wanted to simplify this by relying on the widely-support glTF file format, which can be used for final game-ready models, but can still easily loaded back into Blender or another program.
    Texture Compression
    Texture compression is where this all gets a lot more tricky. Texture compression is an important technique that can reduce video memory usage to about 25% what it would be otherwise. When texture compression is used games run faster, load faster, and look better because they can use higher resolution images. There are two problems with texture compression.
    Supported texture compression methods vary wildly across PC and mobile platforms. Many 3D modeling programs do not support modern texture compression formats. On the PC, BC7 is the best compression format to use for most images. BC5 is a two-channel format appropriate for normal maps, with Z reconstruction in the shader. BC6H can be used to compress HDR RGB images (mostly skyboxes). BC4 is a single-channel compressed format that is rarely used but could be useful for some special cases. The older DXTC formats should no longer be used, as they have worse artifacts with the same data size.
    Here are a few programs that do not support the newer BC formats, even though they are the standard for modern games:
    Windows Explorer Blender Microsoft 3D Object Viewer Now, the glTF model format does not actually support DDS format textures at all. The MSFT_texture_dds extension adds support for this by adding an extra image source into the file. glTF loaders that support this extension can load the DDS files, while programs that do not recognize this extension will ignore it and load the original PNG or JPEG files:
    "textures": [ { "source": 0, "extensions": { "MSFT_texture_dds": { "source": 1 } } } ], "images": [ { "uri": "defaultTexture.png" }, { "uri": "DDSTexture.dds" } ] I don't know any software that supports this extension except Ultra Engine. Even Microsoft's own 3D Object Viewer app does not support DDS files.
    The Ultra Engine editor has a feature that allows you to convert a model's textures to DDS. This will resave all textures in DDS format, and change the model materials to point to the DDS files instead of the original format the model uses (probably PNG):

    When a glTF file is saved after this step, the resulting glTF will keep copies of both the original PNG and the saved DDS files. The resulting glTF file can be loaded in Ultra Engine ready to use in your game, with DDS files applied, but it can still be loaded in programs that do not support DDS files, and the original PNG files will be used instead:

    The same model can be loaded back into Blender in case any changes need to be made. If you resave it, the references to the DDS images will be lost, so just repeat the DDS conversion step in the Ultra Engine to finalize the updated version of your model.

    To cut down on the size of your game files, you can just skip PNG files in the final packaging step so only DDS files are included.
    Basis Universal
    We still have to deal with the issue of hardware support for different codecs. This table from Unity's documentation shows the problem clearly:

    We do want to support some mobile-based VR platforms, so this is something that needs to be considered now. Basis Universal is a library from the author of Crunch that solves this problem by introducing a platform-agnostic intermediate compressed format that can be quickly transcoded into various texture compression formats, without going through a slow decompression and recompression step. We can generate Basis files in the Ultra Engine editor just like we did for DDS: There is a glTF extension that supports basis textures, so glTF models can be saved that reference the Basis files.

    The basis textures are stored in the glTF file the same way the DDS extension works:
    "textures": [ { "source": 0, "extensions": { "KHR_texture_basisu": { "source": 1 } } } ], "images": [ { "mimeType": "image/png", "bufferView": 1 }, { "mimeType": "image/ktx2", "bufferView": 2 } ] The resulting glTF files can be loaded as game-ready models with compressed textures on PC or mobile platforms, and can still be loaded back into Blender or another 3D modeling program. We don't have to store different copies of our textures in different compression formats or go through a slow conversion step when publishing the game to other platforms. If your game is only intended to run on PC platforms, then DDS is the simplest path to take, but if you plan to support mobile-based VR devices in the future then Basis solves that problem nicely.
    Here is our revised table:

    I was also working with the KTX2 format for a while, but I ended up only using it for its support for Basis compression. DDS and Basis cover all your needs and are more widely supported.
    Basis also supports a very lossy but efficient ETC1S method which is similar to Crunch, as well as uncompressed texture data, but in my opinion the one purpose for the Basis format is its UASTC format.
    This article features the lantern sample glTF model from Microsoft.
  12. Josh

    Articles
    I wanted to take some time to investigate geospatial features and see if it was possible to use GIS systems with Ultra Engine. My investigations were a success and resulted in some beautiful lunar landscapes in a relatively short period of time in the new game engine.

    I have plans for this, but nothing I can say anything specific about right now, so now I am working on the editor again.
    Leadwerks had a very well-defined workflow, which made it simple to use but also somewhat limited. Ultra goes in the opposite direction, with extremely flexible support for plugins and extensions. It's a challenge to keep things together because if you make things to flexible it just turns into indecisiveness, and nothing will ever get finished that way. I think I am finding a good balance or things that should be hard-coded and things that should be configurable.
    Instead of separate windows for viewing models, textures, and materials, we just have one asset editor window, with tabs for separate items. The texture interface is the simplest and just shows some information about the asset on the left-side panel.

    When you select the Save as menu item, the save file dialog is automatically populated with information loaded from all the plugins the engine has loaded. You can add support for new image formats at any time just by dropping a plugin in the right folder. In fact, the only hard-coded format in the list is DDS.

    You can save to an image format, or to the more advanced DDS and KTX2 formats, which include mipmaps and other features. Even more impressive is the ability to load Leadwerks texture files (TEX) and save them directly into DDS format in their original pixel format, with no loss of image quality. Here we have a Leadwerks texture that uses DXT1 compression, converted to DDS without recompressing the pixels:

    There are several tricks to make the new editor start up as fast as possible. One of these is that the various windows the editor uses don't actually get created until the first time they are used. This saves a little bit of time to give the application a snappier feel at startup.
    I hope the end result will provide a very intuitive workflow like the Leadwerks editor has, with enough power and flexibility to make the editor into a living growing platform with many people adding new capabilities for you to use. I think this is actually my favorite stuff to work on because this is where all the technology is exposed to you, the end user, and I get to carefully hand-craft an interface that makes you feel happy when you use it.
  13. Josh
    As I have stated before, my goa for this game enginel is not to build a marketplace of 3D models, but instead to just make sure our model loading code reliably loads all 3D models that are compliant with the glTF specification. I started testing more 3D models from Sketchfab, and found that many of them are using specular/gloss materials. At first I thought I could just fudge the result, but I wasn't getting very good results, and the Windows 10 3D Object Viewer was showing them perfectly. This made me very upset because I feel that the software I make for you should be the absolute best thing possible. So I went ahead and implemented the actual KHR_materials_pbrSpecularGlossiness extension. Here are the results:

    It looks quite good without breaking the specular/gloss approach. I think this would be an ideal way to upgrade the Leadwerks renderer without making changes that are too big.
    To implement this, I used the glTF sample viewer source code, which is a bit of a reference renderer for glTF materials. It has changed quite a lot since I first used it as a guide to implementing PBR materials. I decided I might as well integrated the latest code into our renderer. The shader code is very well organized, so it was pretty simple to integrate into my latest code. The glTF materials system is great because it provides a standardized materials system with a lot of advanced effects, along with a reference renderer that shows exactly how the materials are supposed to appear. This means that any engine or tool that is conforms to the glTF standard will always display the same materials the same way, while still allowing additional more specialized features like shadows and post-processing effects to be added.
    In addition to revising our metal-roughness code, I went ahead and added support for some more features. The KHR_materials_clearcoat extension basically makes cars look like cars:
    It adds an extra layer of reflections that gives a dark surface a big boost, and can even use a separate normal map for the clearcoat layer:

    The KHR_materials_sheen extension makes cloth look like cloth. You can see for yourself in the image below. The appearance kind of makes my skin crawl because I can't stand the feeling of velvet / velour. Apparently this is some kind of common tactile hypersensitivity and I actually get a feeling like electricity shooting through my spine just looking at this image, so I think the effect is working well:

    Another creepy example. Just the thought of running my hand over that fuzzy orange fabric is making me shiver:

    The KHR_materials_transmission extensions provides a physically accurate model for transparent surfaces, with support for refraction. Here's a shot of refraction bending the background image:

    This also supports roughness, for making the refracted background appear blurry like frosted glass:

    I feel the background is too pixellated and should be improved. This problem occurs in the reference renderer, and I have several ideas on how to solve it.
    The transparency system is very advanced, and introduces a lot of interesting new commands to the engine:
    Material::SetIndexOfRfraction(const float ior) Material::SetThickness(const float thickness) Material::SetTransmission(const float transmission) These features will give you reliable support for loading glTF models from any source, as well as the best image quality possible for your games. I think realistic materials in VR will also make a big difference in bridging the uncanny valley. And all of this is done with the fastest possible performance for VR.
    And yes, of course I used a gamepad to do all of this.
     
     
  14. Josh

    Articles
    I've been working hard getting all the rendering features to work together in one unified system. Ultra Engine, more than any renderer I have worked on, takes a lot of different features and integrates them into one physically-based graphics system. A lot of this is due to the excellent PBR materials system that Khronos glTF provides, and then there are my own features that are worked into this, like combined screen-space and voxel ray traced reflections.
    Anyways, it's a lot of difficult work, and I decided to take a "break" and focus something else for a few days.
    Before Leadwerks was on Steam, it had a web installer that would fetch a list of files from our server and any files that were missing, or were newer than the locally stored ones. There was no system for detecting updates, you just pressed the update button and the updater ran. The backend for this system was designed by @klepto2 and it functioned well for what it needed to do.

    With Ultra App Kit, I created a simple tool to generate projects. This introduced the account authentication / signin system, which was sort of superfluous for this application, but it was a good way to test it out:

    With Ultra Engine, I wanted some characteristics of both these applications. I wrote a new backend in about a day that handles updates. A PHP script authenticates the user and verifies product ownership, fetches a list of files, and retrieves files. Since C++ library files tend to be huge, I found it was necessary to add a compression system, so the script returns a zip compressed file. Of course, you don't want the server to be constantly creating zip files, so it caches the file and updates the zip only when I upload a new copy of the file. There's also a cache for the info retrieval, which is returned in JSON format, so it's easy to read in C++.
    For the front end, I took inspiration from the Github settings page, which I thought looked nice:

    And here's what I came up with. Projects will show when they are outdated and need to be updated (if a file in the template the project came from was changed). Each of the sections contains info and links to various topics. There's a lot there, but none of it feels extraneous to me. This is all made with the built-in GUI system. No HTML is used at all:

    The Invision Power REST API is extremely interesting. It allows authentication of accounts and purchases, but it can be made to do a lot of other things.
    Post a forum topic: https://invisioncommunity.com/developers/rest-api?endpoint=forums/topics/POSTindex Upload an image to the gallery: https://invisioncommunity.com/developers/rest-api?endpoint=gallery/images/GETindex Download a file: https://invisioncommunity.com/developers/rest-api?endpoint=downloads/files/GETitem Unlock achievements: https://invisioncommunity.com/developers/rest-api?endpoint=core/members/POSTitem_achievements_awardbadge None of that is very important right now, but it does provide some interesting ideas for future development of the game engine.
  15. Josh

    Articles
    Autodesk 3ds Max now supports export of glTF models, as well as a new glTF material type. The process of setting up and exporting glTF models is pretty straightforward, but there are a couple of little details I wanted to point out to help prevent you from getting stuck. For this article, I will be working with the moss rocks 1 model pack from Polyhaven.
    Getting geometry into 3ds Max is simple enough. I imported the model as an FBX file.

    To set up the material, I opened the compact material editor and set the first slot to be a glTF material.

    Press the button for the base color map, and very importantly choose the General > Bitmap map type. Do not choose OSL > Bitmap Lookup or your textures won't export at all.

    Select your base color texture, then do the same thing with the normal and roughness maps, if you have them. 3ds Max treats metal / roughness as two separate textures, although you might be able to use the same texture both if it grabs the data from the green (roughness) and blue (metal) channels. This is something I don't know yet.

    Select the File > Export menu item to bring up the glTF export dialog. Uncheck the "Export glTF binary" option because we don't want to pack our model and textures into a single file: I don't know what the baked / original material option does because I don't see any difference when I use it.

    At this point you should have a glTF file that is visible in any glTF model viewer.

    Now something slightly weird max does is it generates some new textures for some of the maps. This is probably because it is combining different channels to produce final images. In this case, none of our textures need to be combined, so it is just a small annoyance. A .log file will be saved as well, but these can be safely deleted.

    You can leave the images as-is, or you can open up the glTF file in a text editor and manually change the image file names back to the original files:
    "images": [ { "uri": "rock_moss_set_01_nor_gl_4k.jpg" }, { "uri": "M_01___Defaultbasecolortexture.jpeg" }, { "uri": "M_01___Defaultmetallicroughnesstex.jpeg" } ], Finally, we're going to add LODs using the Mesh Editor > ProOptimizer modifier. I like these settings, but the most important thing is to make sure "Keep textures" is checked. You can press the F3 key at any time to toggle wireframe view and get a better view of what the optimizer does to your mesh.

    Export the file with the same name as the full-resolution model, and add "_lod1" to the end of the file name (before the extension). Then repeat this process saving lod2 and lod3 using 25% and 12.5% for the vertex reduction value in the ProOptimizer modifier.
    Here is my example model you can inspect:
    mossrock1a.zip
    Now it is very easy to get 3D models from 3ds max to your game engine.
  16. Josh
    As the first release of Ultra Engine approaches, it seems clear that the best way to maximize its usefulness is to make it as compatible as possible with the Leadwerks game engine. To that end, I have implemented the following features.
    Native Loading of Leadwerks File Formats
    Ultra Engine loads and saves DDS, glTF, and OBJ files. Other formats are supported by plugins, both first and potentially third-party, for PNG, JPG, BMP, TGA, TIFF, GIF, HDR, KTX2, and other files. Additionally, all Leadwerks file formats are natively supported without requiring any plugins, so you can load TEX, MDL, PHY, MAT files, as well as full Leadwerks maps. You also have the option to save Leadwerks game engine formats in more up-to-date file formats such as glTF or DDS files with BC5/BC7 compression.

    Testing a scene from the excellent Cyclone game (available on Steam!)
    Classic Specular Shader Family
    PBR materials are wonderful but act quite differently from conventional specular/gloss game materials. I've added a "Classic" shader family which provides an additive specular effect, which may not be realistic but it makes materials expect the way you would expect them to, coming from Leadwerks game engine. When Leadwerks MAT files are loaded, this shader family is automatically applied to them, to get a closer match to their appearance in Leadwerks.
    Leadwerks Translation Layer
    I'm experimenting with a Leadwerks header for Ultra Engine. You can drop this into your Ultra Engine project folder and then start running code with the Leadwerks API. Internally, the Leadwerks game engine commands will be translated into the Ultra Engine API to execute your existing code in the new engine. I don't think this will be 100% perfect due to some differences in the way the two engines work, but I do think it will give you an easy way to get started and provide a gentler transition to Ultra.
    This code will actually work with both engines:
    #include "Leadwerks.h" using namespace Leadwerks; int main(int argc, const char* argv[]) { Window* window = Window::Create(); Context* context = Context::Create(window); World* world = World::Create(); Camera* camera = Camera::Create(); camera->Move(0, 0, -3); Light* light = DirectionalLight::Create(); light->SetRotation(35, 35, 0); Model* model = Model::Box(); model->SetColor(0.0, 0.0, 1.0); while (true) { if (window->Closed() || window->KeyDown(Key::Escape)) return false; model->Turn(0, Time::GetSpeed(), 0); Time::Update(); world->Update(); world->Render(); context->Sync(false); } return 0; } I hope these enhancements give you a more enjoyable experience using Ultra together with the Leadwerks Editor, or as a standalone programming SDK.
  17. Josh
    In Vulkan all shader uniforms are packed into a single structure declared in a GLSL shader like this:
    layout(push_constant) uniform pushBlock { vec4 color; } pushConstantsBlock; You can add more values, but the shaders all need to use the same structure, and it needs to be declared exactly the same inside the program.
    Like everything else in Vulkan, shaders are set inside a command buffer. But these shader values are likely to be constantly changing each frame, so how do you handle this? The answer is to have a pool of command buffers and retrieve an available one when needed to perform this operation.
    void Vk::SetShaderGlobals(const VkShaderGlobals& shaderglobals) { VkCommandBuffer commandbuffer; VkFence fence; commandbuffermanager->GetManagedCommandBuffer(commandbuffer,fence); VkCommandBufferBeginInfo beginInfo = {}; beginInfo.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO; beginInfo.flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT; vkBeginCommandBuffer(commandbuffer, &beginInfo); vkCmdPushConstants(commandbuffer, pipelineLayout, VK_SHADER_STAGE_ALL, 0, sizeof(shaderglobals), &shaderglobals); vkEndCommandBuffer(commandbuffer); VkSubmitInfo submitInfo = {}; submitInfo.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO; submitInfo.commandBufferCount = 1; submitInfo.pCommandBuffers = &commandbuffer; vkQueueSubmit(devicequeue[0], 1, &submitInfo, fence); } I now have a rectangle that flashes on and off based on the current time, which is fed in through a shader uniform structure. Now at 1500 lines of code.
    You can download my command buffer manager code at the Leadwerks Github page:
  18. Josh

    Articles
    Happy Friday! I am taking a break from global illumination to take care of some various remaining odds and ends in Ultra Engine.
    Variance shadow maps are a type of shadowmap filter technique that use a statistical sample of the depth at each pixel to do some funky math stuff. GPU Gems 3 has a nice chapter on the technique. The end result is softer shadows that run faster. I was wondering where my variance shadow map code went, until I realized this is something I only prototyped in OpenGL and never implemented in Vulkan until now. Here's my first pass at variance shadow maps in Vulkan:

    There are a few small issues but they are no problem to work out. The blurring is taking place before the scene render, in the shadow map itself, which is a floating point color texture instead of a depth texture. (This makes VSMs faster than normal shadow maps.) The seams you see on the edges in the shot above are caused by that blurring, but there's a way we can fix that. If we store one sharp and one blurred image in the variance shadow map, we can interpolate between those based on distance from the shadow caster. Not only does this get rid of the ugly artifacts (say goodbye to shadow acne forever), but it also creates a realistic penumbra, as you can see in the shot of my original OpenGL implementation. Close to the shadow caster, the shadow is well-defined and sharp, but it gets much blurrier the further away it gets from the object:

    Instead of blurring the near image after rendering, we can use MSAA to give it a fine-but-smooth edge. There is no such thing as an MSAA depth shadow sampler in GLSL, although I think there should be, and I have lobbied on behalf of this idea.

    Finally, in my Vulkan implementation I used a compute shader instead of a fragment shader to perform the blurring. The advantage is that a compute shader can gather a bunch of samples and store them in memory, then access them to process a group of pixels at once. Instead of reading 9x9 pixels for each fragment, it can read a block of pixels and process them all at once, performing the same number of image writes, but much fewer reads:
    // Read all required pixel samples x = int(gl_WorkGroupID.x) * BATCHSIZE; y = int(gl_WorkGroupID.y) * BATCHSIZE; for (coord.x = max(x - EXTENTS, 0); coord.x < min(x + BATCHSIZE + EXTENTS, outsize.x); ++coord.x) { for (coord.y = max(y - EXTENTS, 0); coord.y < min(y + BATCHSIZE + EXTENTS, outsize.y); ++coord.y) { color = imageLoad(imagearrayCube[inputimage], coord); samples[coord.x - int(gl_WorkGroupID.x) * BATCHSIZE + EXTENTS][coord.y - int(gl_WorkGroupID.y) * BATCHSIZE + EXTENTS] = color; } } This same technique will be used to make post-processing effects faster. I previously thought the speed of those would be pretty much the same in every engine, but now I see ways they can be improved significantly for a general-use speed increase. @klepto2 has been talking about the benefits of compute shaders for a while, and he is right, they are very cool. Most performance-intensive post-processing effects perform some kind of gather operation, so compute shaders can make a big improvement there.
    One issue with conventional VSMs is that all objects must cast a shadow. Otherwise, an object that appears in front of a shadow caster will be dark. However, I added some math of my own to fix this problem, and it appears to work with no issues. So that's not something we need to worry about.
    All around, variance shadow maps are a big win. They run faster, look better, and eliminate shadow acne, so they basically kill three birds with one stone.
  19. Josh
    Heat haze is a difficult problem. A particle emitter is created with a transparent material, and each particle warps the background a bit. The combined effect of lots of particles gives the whole background a nice shimmering wavy appearance. The problem is that when two particles overlap one another they don't blend together, because the last particle drawn is using the background of the solid world for the refracted image. This can result in a "popping" effect when particles disappear, as well as apparent seams on the edges of polygons.

    In order to do transparency with refraction the right way, we are going to render all our transparent objects into a separate color texture and then draw that texture on top of the solid scene. We do this in order to accommodate multiple layers of transparency and refraction. Now, the correct way to handle multiple layers would be to render the solid world, render the first transparency object, then switch to another framebuffer and use the previous framebuffer color attachment for the source of your refraction image. This could be done per-object, although it could get very expensive, flipping back and forth between two framebuffers, but that still wouldn't be enough.
    If we render all the transparent surfaces into a single image, we can blend their normals, refractive index, and other properties, and come up with a single refraction vector that combined the underlying surfaces in the best way possible.
    To do this, the transparent surface color is rendered into the first color attachment. Unlike deferred lighting, the pixels at this point are fully lit.

    The screen normals are stored in an additional color attachment. I am using world normals in this shot but later below I switched to screen normals:

    These images are drawn on top of the solid scene to render all transparent objects at once. Here we see the green box in the foreground is appearing in the refraction behind the glass dragon.

    To prevent this from happening, we need add another color texture to the framebuffer and render the pixel Z position into it. I am using the R32_SFLOAT format. I use the separate blend mode feature in Vulkan, and set the blend mode to minimum so that the smallest value always gets saved in the texture. The Z-position is divided by the camera far range in the fragment shader, so that the saved values are always between 0 and 1. The clear color for this attachment is set to 1,1,1,1, so any value written into the buffer will replace the background. Note this is the depth of the transparent pixels, not the whole scene, so the area in the center where the dragon is occluded by the box is pure white, since those pixels were not drawn.

    In the transparency pass, the Z position of the transparent pixel is compared to the Z position at the refracted texcoords. If the refracted position is closer to the camera than the transparent surface, the refraction is disabled for that pixel and the background directly behind the pixel is shown instead. There is some very slight red visible in the refraction, but no green.

    Now let's see how well this handles heat haze / distortion. We want to prevent the problem when two particles overlap. Here is what a particle emitter looks like when rendered to the transparency framebuffer, this time using screen-space normals. The particles aren't rotating so there are visible repetitions in the pattern, but that's okay for now.

    And finally here is the result of the full render. As you can see, the seams and popping is gone, and we have a heavy but smooth distortion effect. Particles can safely overlap without causing any artifacts, as their normals are just blended together and combined to create a single refraction angle.

  20. Josh

    Articles
    I'm wrapping up the terrain features now for the initial release. Here's a summary of terrain in Ultra Engine.
    Terrains are an entity, just like anything else, and can be positioned, rotated, or scaled. Non-square terrains are supported, so you can create something with a 1024x512 or whatever resolution. (Power-of-two sizes are required.)
    Editing Terrain
    Ultra Engine includes an API that lets you modify terrain in real-time. I took something very complicated and distilled it down to a very simple API for you:
    terrain->SetElevation(x, y, height); That's it! The engine will automatically update normals, physics, raycasting, rendering, and other data for you behind the scenes any time you modify the terrain, in a manner that maximizes efficiency.
    Materials
    Leadwerks supports 32 terrain texture layers. Unity supports eight. Thanks to the flexibility of shaders in Vulkan, any terrain in Ultra can have up to a whopping 256 different materials applied to it, and it will always be rendered in a single pass at maximum speed.
    To apply a material at any point on the terrain, you just call this method. The weight value lets you control how much influence the material as at that point, i.e. its alpha transparency:
    terrain->SetMaterial(x, y, material, weight) This example shows how to paint materials onto the terrain.

    I'm quite happy with how the documentation system has turned out, and I feel it is representative of the quality I want your entire user experience to be.

    All the API examples load media from the web, so there's no need to manually download any extra files. Copy and paste the code into any project, and it just works.
    //Create base material auto diffusemap = LoadTexture("https://raw.githubusercontent.com/Leadwerks/Documentation/master/Assets/Materials/Ground/river_small_rocks_diff_4k.dds"); auto normalmap = LoadTexture("https://raw.githubusercontent.com/Leadwerks/Documentation/master/Assets/Materials/Ground/river_small_rocks_nor_gl_4k.dds"); auto ground = CreateMaterial(); ground->SetTexture(diffusemap, TEXTURE_DIFFUSE); ground->SetTexture(normalmap, TEXTURE_NORMAL); Tessellation and Displacement
    Tessellation works great with terrain, which is modeled with quads. There are a lot of high-quality free PBR materials that are great for VR available on the web, so you will have no shortage of interesting materials to paint your terrains with. Here are a few sites for you to peruse:
    https://matlib.gpuopen.com/main/materials/all
    https://icons8.com/l/3d-textures
    https://ambientcg.com/
    https://polyhaven.com/textures
    https://freepbr.com/
    https://www.cgbookcase.com/textures

    Cutting Holes
    You can cut holes in the terrain by hiding any tile. This is useful for making caves, and can be used in the future for blending voxel geometry into a heightmap terrain.

    Shader Design
    Shaders in Ultra are big and quite complicated. I don't expect anyone to make 100% custom shaders like you could with the simpler shaders in Leadwerks. I've structured shaders so that the entry point shader defines a user function, then includes the base file, which calls the function:
    #version 450 #extension GL_GOOGLE_include_directive : enable #extension GL_ARB_separate_shader_objects : enable #define LIGHTING_PBR #define USERFUNCTION #include "../Base/Materials.glsl" #include "../Base/TextureArrays.glsl" void UserFragment(inout vec4 color, inout vec3 emission, inout vec3 normal, inout float metalness, inout float roughness, inout float ambientocclusion, in vec3 position, in vec2 texcoords, in vec3 tangent, in vec3 bitangent, in Material material) { // Custom code goes here... } #include "../Base/base_frag.glsl" This organizes shaders so custom behavior can be added on top of the lighting and other systems, and paves the way for a future node-based shader designer.
    Future Development
    Ideas for future development to add to this system include voxel-based terrains for caves and overhangs, roads (using the decals system), large streaming terrains, and seamless blending of models into the terrain surface.
  21. Josh
    Ultra Engine makes much better use of class encapsulation with strict public / private / protected members. This makes it so you can only access parts of the API you are meant to use, and prevents confusion about what is and isn't supported. I've also moved all internal engine commands into separate namespaces so you don't have to worry about what a RenderMesh or PhysicsNode does. These classes won't appear in the intellisense suggestions unless you were to add "using namespace UltraRender" to your code, which you should never do.
    There's still a lot of extra stuff intellisense shows you from Windows headers and other things:

    Using the __INTELLISENSE__ macro in Visual Studio, I was able to trick the intellisense compiler into skipping include files that link to third party libraries and system headers the user doesn't need to worry about. This strips all the unnecessary stuff out of the API, leaving just basic C++ commands, some STL classes, and the Ultra Engine API. With all the garbage removed, the API seems much friendlier:

    This is one more detail that I feel helps make C++ with Ultra Engine about as easy as C# programming. You can disable this feature by adding _ULTRA_SIMPLE_API=0 in your project macros.
  22. Josh

    Articles
    I've actually been doing a lot of work to finalize the terrain system, but I got into tessellation, and another rabbit hole opened up. I've been thinking about detailed models in VR. Tessellation is a nice way to easily increase model detail. It does two things:
    Curved surfaces get smoother (using point-normal triangles or quads) A displacement map can be used to make small geometric detail to a surface. These are really nice features because they don't require a lot of memory or disk space, and they're automatic. However, tessellation also has a nasty tendency to create cracks in geometry, so it tends to work best with organic models like rocks that are carefully wrapped. I was able to mitigate some of these problems with a per-vertex displacement value, which dampens displacement wherever there is a mesh seam:

    However, this does not help round edges become rounder. With this cylinder, because the faces are not a continuous rounded surface, a crack appears at the sharp edges:

    What if there was a way to properly tessellate mechanical shapes like this? Wouldn't it be great if all your models could just automatically gain new detail that scales with the camera distance, with no need for extra Lod versions?
    Well, I had an idea how it might be possible, and a few hours later I came up with something. Here is the original model with no tessellation applied:


    Here is the same model with my new and improved tessellation shader, suitable for organic and mechanical shapes:


    A closeup view reveals perfectly aligned geometry with no gaps or cracks:

    The distribution of polygons can probably be made more uniform with additional work. This feature uses two additional bytes in the vertex structure to control a normal for tessellated curves. I think it will be possible to automatically detect seams and assign tessellation normals in the final editor. If no separate tessellation normals are assigned, then the default tessellation behavior occurs. This is a very promising technique because it could allow you to add a lot of detail to any model very easily with barely any effort.
  23. Josh
    I'm finalizing the shaders, and I was able to pack a lot of extra data into a single entity 4x4 matrix:
    Orthogonal 4x4 matrix RGBA color Per-entity texture mapping offset (U/V), scale (U/V), and rotation Bitwise entity flags for various settings Linear and rotational velocity (for motion blur) Skeleton ID All of that info can be fit into just 64 bytes. The shaders now use a lot of snazzy new function like this:
    void ExtractEntityInfo(in uint id, out mat4 mat, out vec4 color, out int skeletonID, out uint flags) mat4 ExtractCameraProjectionMatrix(in uint cameraID, in int eye) So all the storage / decompression routines are in one place instead of hard-coding a lot of matrix offset in different shaders.
    A per-entity linear and angular velocity is being calculated and sent to the rendering thread. This is not related to the entity velocity in the physics simulation, although they will most often be the same. Rather, this is just a measure of the distance the entity moved since the last world render, so it will work with any motion, physics or otherwise.
    In the video below, the linear velocity is being added to the camera position to help predict the region of the GI lighting area, so the camera stays roughly in the center as the GI updates. A single 128x128x128 volume texture is being used here, with a voxel size of 0.25 meters. In the final version, you probably want to use 3-4 stages, maybe using a 64x64x64 texture for each stage.
    Performance of our voxel cone step tracing is quite good, with FPS in the mid-hundreds on a mid-range notebook GPU. The last big question is how to handle dynamic objects, particular fast-moving dynamic objects. I have some ideas but I'm not sure how hard this is going to be. My first attempt was slow and ugly. You can probably guess which dragon is static and which is dynamic:

    It is a difficult problem. I would like the solution to be as cool as the rest of this system as I finish up this feature. Also, I think you will probably want to still use SSAO so dynamic objects blend into the GI lighting better.
    Everyone should have access to fast real-time global illumination for games.
  24. Josh

    Articles
    I have more than one light bounce working now, and it looks a lot nicer than single-bounce GI. The ambient light here is pure black. All light is coming off from the direct light, and bouncing off surfaces.

    It will take some time to get the details worked out, and more bounces will require more memory. I'm actually kind of shocked how good looking it is. This is just a single 128x128x128 volume texture at 0.25 meters per voxel. Light leaks seem to be not a problem, even at that low resolution, even with thin walls.
    You can see that absolutely no direct light is hitting the red wall on the left, and the ambient light is pure black. Yet light is still bouncing off the floor onto the wall, and then lighting the dragon very faintly with red light. (The exposure here is higher than it should be so it looks a little washed out):

    There's a lot here for me to take in. Real-time global illumination is such a rabbit hole that keeps unfolding new layers of complexity and possibilities. It seems like every few days my mind is totally blown and I look at the system differently, and the new way makes so much sense. In the last three days I have said to myself "I'll just try one more thing..." and then that thing opens up a lot of new possibilities. April is nearly over, and I have been working on just this feature since December!
    I've got a bunch of stuff to work out, but it looks like this will run really really well in VR.
  25. Josh
    Since previously determining that voxels alone weren't really capable of displaying artifact-free motion, I have been restructuring the GI system to favor speed with some tolerance for latency. The idea is that global illumination will get updated incrementally in the background over the course of a number of frames, so the impact of the calculation per frame is small. The new GI result is then smoothly interpolated from the old one, over a period of perhaps half a second. Here's a shot of the restructured system sort-of-working, at least it's writing to and reading from the correct textures, though the lighting is not correct here:

    Diffuse lighting gets calculated for each voxel, for each of six directions, so the final scene render just takes the normal and uses the dot product to figure out which GI samples to use in the voxel that point intersects. This greatly lightens the load of the final scene render, since it no longer has to perform several costly cone-step rays. Since specular reflection is view-dependent, that part must be ray-traced in the final scene render, but there are some optimizations I have yet to make that will probably double the framerate here. My best estimate right now is that the GI system reduces framerate by 30% when enabled, and that it's not really dependent on scene complexity. In other words, as the scene gets more objects, the performance cost should stay about the same, and we have already seen that Ultra Engine scales extremely well with scene complexity. So everything looks very good.
    From what I have seen, real-time global illumination is the killer feature everyone wants. Spending a few extra weeks on this to get a better result makes sense. I don't think I've ever worked on a feature or system that required so much iteration. I get the system working, start using it in different scenes, testing performance, and then I realize what needs to be different. With this feature, that process seems very extreme, but I really think this is the last restructure needed to get the best result possible for your games. Delivering real-time global illumination while maintaining fast framerates is very much inline with the design goals of Ultra Engine.
×
×
  • Create New...