Jump to content
Ultra Engine Community

Josh

Staff
 View Profile  See their activity

  • Posts

    23,094

  • Joined

  • Last visited

 Content Type 

Blog Entries posted by Josh

  1. Josh

    Finishing Touches

    By Josh,


    Articles
    At long last, the engine that felt like it would never be done is here. This is all the result of an idea I had and wanted to put to the test. Of course I never anticipated it was possible to deliver a 10x improvement in rendering performance, but here we are with the benchmarks to prove it.
    Research & Development
    The last four years of research and development were an opportunity to rethink and redesign what a game engine should be. Along the way there were a few ideas I had that turned out to be not that great of an idea in practice:
    Variance shadow maps: Although I love the look of these under very controlled settings, the technique has extremely bad artifacts that can never be fixed. Depth shadow maps really are the best approach to texture-based shadows. They are also extremely slow to render shadow updates. I would like to see some better hardware filtering options in the future, though.
    Voxel-based indirect lighting: Compelling in theory but extremely messy in practice. I learned a lot and this prompted me to integrate compute shaders into the engine, but ultimately the results I got weren't compelling enough to turn into a finished product. Given the problems everyone else has with this approach, I think it makes sense to focus on hardware raytracing as the next step up in lighting.
    Novel entity component system: My idea was to make it so when you called a component method, all the components that had that method would get called. I had some complicated Lua code set up that handled this, and early builds of Ultra used a C++ preprocessor that analyzed header files and created some complicated wrappers to make C++ work in roughly the same way. Ultimately I decided the confusion wasn't worth the questionable benefit and I implemented a conventional multi-component system like everyone had asked for, without any preprocessor.
    Tabbed Asset Editor: Another thing that sounded good in theory but was not great in practice was to use a single asset editor window with tabs for the different opened files. All it took was a day of using this to realize how much better it was to open each item in its own window. Hunting and clicking file names in tabs is just not fun. I get claustrophobic just looking at this:

    The Results
    Besides the performance, here are the things that I think turned out better than I imagined:
    glTF Integration: Khronos glTF has proven to be a wonderful workflow, and eliminates any "black box" file formats for 3D content. Your game-ready files can always be pulled back into your modeling program, edited, and saved, with no need for secretive conversion pipelines. A wide variety of game-ready content is available for you to use with Ultra, so I am expecting to see no more programmer art!

    Physically-based Rendering: This aligns closely with glTF integration because the file format includes a well-defined PBR materials system. The secret to good PBR is to have good imagery for reflections. To achieve this, I put a lot of work into the volumetric environment probes system that first appeared in Leadwerks. Probe volumes can now be drawn like a brush object for easier creation and precise sizing in the editor. A configurable fade distance for each edge lets you control how probes blend together. Probes also incorporate the sky lighting so you can have seamless transitions between bright outdoor areas and dark enclosed spaces.

    Lua Integration: The VSCode Lua debugger turns Lua into a first-class game programming language and is a dream to work with. The editor scripting integration is in my opinion the best scripting integration of any 3D program I've ever seen. You can access the entire engine API from within editor scripts to add new functionality or modify the program. Additional documentation for the editor's innards will arrive in the coming days.
    Ultra GUI: This took a lot of time but having control over every pixel and event made it worthwhile. Taking control of the user experience by writing my own GUI was one of the best decisions I made. The interface doubles up as the editor's own UI drawn using GDI+ and the in-game UI rendered with Vulkan.

    C++ API: The most significant change is the user of shared pointers, which forever eliminate the problems with uninitialized and invalid pointers games were prone to as they grew more complex. Additionally, the API is generally better thought out and consistent than what we had in Leadwerks.
    Terrain: You wanted more terrain material layers, so you got them. Up to 256, in fact, with fast performance. You can create multiple terrains, position, rotate, and scale them, and even create non-square terrains to fill in any area.

     
    Player Physics: Yes, you can crouch now.
    Pathfinding: It's dynamic and you can create multiple navmeshes, for different areas or differently sized characters. Pathfinding and physics are now decoupled so you can have armies of characters that only use the pathfinding system for movement.
    3D Brush Editing: I was not planning on this feature until the last minute, but you can can create, move, scale, rotate, and shear brushes in any viewport, including the 3D view. the long-asked-for vertex tool is included as well as a face tool for adjusting texture mapping and geometry.
    Last but not least, the engine's multithreaded design is crazy advanced! Your game code runs on its own thread, giving you a full 16 milliseconds to operate without slowing down the renderer. Physics, pathfinding, animation, rendering, and culling all operate on separate threads. It's an amazing feat in engineering that was only possible because I had seen all the bottlenecks people could create in Leadwerks, so I could plan a way around them when designing Ultra.
    Early Access
    The 1.0 release will be marked "Early Access" because there are still a few extra features to add, and we will have a period where changes to the workflow can still be considered based on user feedback. Once decals, particle emitters, VR support, flowgraphs, and brush smooth groups are added, that will be considered version 1.1 (no "early access"). So there will still be a few months of time where I can revise any parts of the workflow you think can be improved. There are also still some compatibility issues with AMD cards that are being resolved, so please be aware of that.
    A big thanks goes out to all the testers who helped me smooth things out before the release. The good times are finally here!
  2. Josh

    Colliders in Ultra Engine

    By Josh,


    Articles
    I haven't been blogging much because I am making so much progress that there would be too much to cover. In this article I will talk about one little system I spent the last week on and show the improvements I have made to the workflow over Leadwerks.
    In the asset editor, when the root of a model is selected, a "Collider" field appears in the physics properties. You can use this to quickly calculate and apply a collider to the model.

    The convex hull collider includes an option for tolerance. You can adjust this to get a more or less precise convex hull around the model:

    There is also an "optimize" option for the mesh collider. When this is activated it will attempt to merge adjacent coplanar polygons into a single face, and can result in fewer seams on colliding faces.
    Two new shapes are included, the capsule and the chamfer cylinder, so now all the shapes supported by Newton Dynamics are available. (The chamfer cylinder is good for vehicle tires.)
     
    Unlike Leadwerks, you no longer need to specify the axis cylindrical shapes are oriented around. The editor will calculate which bounding box axes are most like each other, and use the third axis to orient the shape. This works for long skinny objects:

    And it also works for short fat objects:

    It will even detect which direction a cone should be facing, by calculating the total distance from the shape the model vertices are with each direction, and selecting the orientation with the smallest error value. (A single cone is probably not the best choice of shape for this model, but the point is that it correctly guesses which direction the cone should point based on the model vertices.)

    For more advanced geometry, the convex decomposition tool is available in the Utilities panel. This exposes all the parameters available in V-HACD 4.0 and processes the task on a separate thread so you can continue to use the editor as the task runs in the background. It usually only takes a few seconds to compete, but it is nice to not have your workflow interrupted:

    You can export colliders as model files, so it's also possible to just use this to generate convex hulls and export them to .obj. (I could get a better fitting shape if I spent more time adjusting the settings, but I just quickly made this shape with very low settings in order to get this screenshot.)

    The new .collider file format, like everything else in Ultra, is JSON-based. Simple shapes can be stored in pure ASCII text, so they can be edited by hand if needed:
    { "collider": { "parts": [ { "offset": [ 0.0024815797805786133, -0.18715500831604004, -1.055002212524414e-05 ], "rotation": [ 0.0, 0.0, 0.0 ], "shape": "BOX", "size": [ 1.8899539709091187, 2.0, 1.8019688129425049 ] } ] } } More complex shapes like convex hulls and meshes use binary data appended to the end of the file to store their contents, so that they load quickly in the engine:
    { "collider": { "parts": [ { "data": 0, "shape": "CONVEXHULL", "vertices": 61 } ] } }�n�2��<�5�j�k��#�I3��ch�������=b[��I<��~d�I�D�` �[�\{4���A���?XO3���>��վ�v!���f��� ?]����I�!�����V� >�?���mdG�t9�a��@���X?W��0o��e�-�D��h�P?>����>��׾���@��=T�U?��ž�{����w�� ���G��3��ޑ?>V?c�����;���6�s�/?&�=����?q�e?j��N���[#b��bݽ(��>��Ǿ�ڽ�E��MV�ַؽ��8<�G������D?BYT?I���@"N?6?�";-�$?��콜�";��t>8�"�t�";�z��3�0���";����ݐ�)�%=}P?��0?���=/�E?BYT?'>�=�E��MV����={; ?"����>�־��c� �>��?q�e?�$>�ӽ��P��@>��,?�j|=�2�>[o/���P�C��>��?>V?Z�>�������Қ�>��h���@�>��E�CO2��.�>T��[A�y]�>L��=T�U?w4�>���>�쾛?#o��e�-�r?�����U?�"?.�v>��!?�R?�zv����>�@?(�0�S �>.(?�n_�C�?�|-?d�[��}?��4?��>��վ��6?2M4��*?�>?P��=_�s��F?V ���ྲAm?�#�I3�Lr?����� Leadwerks .phy collider files can also be loaded.
    Notice in the shots above, in the drop-down menu there is also a "Browse" option, so you can create a collider file with a low-resolution model and then load the resulting collider to use with a high-resolution model.
    I think this design strikes the right balance to let you do simple things quickly, but still allow for more advanced options when you need them. In the future I would like to add an additional in-depth collider editing utility that would allow you to select each sub-object, select model polygons and generate shapes just from those polys, and other advanced features.
    The pick mode feature has been revised. The original design was based on Blitz3D, which did not support collision primitives at all. Instead of specifying a sphere or other shape for picking, you can now just tell the entity to use the collider for picking. (PICK_SPHERE is removed.)
    On an unrelated note, I had to adjust the camera range based on the distance from the model being viewed, in order to make very small objects visible. This allows extreme close-up shots like in the blog header image that you probably aren't used to seeing except in VR. It's kind of interesting.
  3. Josh
    Not long ago, I wrote about my experiments with AI-generated textures for games. I think the general consensus at the time was that the technology was interesting but not very useful in its form at the time. Recently, I had reason to look into the OpenAI development SDK, because I wanted to see if it was possible to automatically convert our C++ documentation into documentation for Lua. While looking at that, I started poking around with the image generation API, which is now using DALL-E 2. Step by step, I was able to implement AI texture generation in the new editor and game engine, using only a Lua script file and an external DLL. This is available in version 1.0.2 right now:

    Let's take a deep dive into how this works...
    Extension Script
    The extension is run by placing a file called "OpenAI.lua" in the "Scripts/Start/Extensions" folder. Everything in the Start folder gets run automatically when the editor starts up, in no particular order. At the top of the script we create a Lua table and load a DLL module that contains a few functions we need:
    local extension = {} extension.openai = require "openai" Next we declare a function that is used to process events. We can skip its inner workings for now:
    function extension.hook(event, extension) end We need a way for the user to activate our extension, so we will add a menu item to the "Scripting" submenu. The ListenEvent call will cause our hook function to get called whenever the user selects the menu item for this extension.  Note that we are passing the extension table itself in the event listener's extra parameter. There is no need for us to use a global variable for the extension table, and it's better that we don't.
    local menu = program.menu:FindChild("Scripting", false) if menu ~= nil then local submenu = menu:FindChild("OpenAI", false) if submenu == nil then submenu = CreateMenu("", menu)-- divider submenu = CreateMenu("OpenAI", menu) end extension.menuitem = CreateMenu("Text to Image", submenu) end ListenEvent(EVENT_WIDGETACTION, extension.menuitem, extension.hook, extension) This gives us a menu item we can use to bring up our extension's window.

    The next section creates a window, creates a user interface, and adds some widgets to it. I won't paste the whole thing here, but you can look at the script to see the rest:
    extension.window = CreateWindow("Text to Image", 0, 0, winx, winy, program.window, WINDOW_HIDDEN | WINDOW_CENTER | WINDOW_TITLEBAR) Note the window is using the WINDOW_HIDDEN style flag so it is not visible when the program starts. We're also going to add event listeners to detect when the window is closed, and when a button is pressed:
    ListenEvent(EVENT_WINDOWCLOSE, extension.window, extension.hook, extension) ListenEvent(EVENT_WIDGETACTION, extension.button, extension.hook, extension) The resulting tool window will look something like this:

    Now let's take a look at that hook function. We made three calls to ListenEvent, so that means we have three things the function needs to evaluate. Selecting the menu item for this extension will cause our hidden window to become visible and be activated:
    elseif event.id == EVENT_WIDGETACTION then if event.source == extension.menuitem then extension.window:SetHidden(false) extension.window:Activate() When the user closes the close button on the tool window, the window gets hidden and the main program window is activated:
    if event.id == EVENT_WINDOWCLOSE then if event.source == extension.window then extension.window:SetHidden(true) program.window:Activate() end Finally, we get to the real point of this extension, and write the code that should be executed when the Generate button is pressed. First we get the API key from the text field, passing it to the Lua module DLL by calling openal.setapikey.
    elseif event.source == extension.button then local apikey = extension.apikeyfield:GetText() if apikey == "" then Notify("API key is missing", "Error", true) return false end extension.openai.setapikey(apikey) Next we get the user's description of the image they want, and figure out what size it should be generated at. Smaller images generate faster and cost a little bit less, if you are using a paid OpenAL plan, so they can be good for testing ideas. The maximum size for images is currently 1021x1024.
    local prompt = extension.promptfield:GetText() local sz = 512 local i = extension.sizefield:GetSelectedItem() if i == 1 then sz = 256 elseif i == 3 then sz = 1024 end The next step is to copy the user's settings into the program settings so they will get saved when the program closes. Since the main program is using a C++ table for the settings, both Lua and the main program can easily share the same information:
    --Save settings if type(program.settings.extensions) ~= "userdata" then program.settings.extensions = {} end if type(program.settings.extensions.openai) ~= "userdata" then program.settings.extensions.openai = {} end program.settings.extensions.openai.apikey = apikey program.settings.extensions.openai.prompt = prompt program.settings.extensions.openai.size = {} program.settings.extensions.openai.size[1] = sz program.settings.extensions.openai.size[2] = sz Extensions should save their settings in a sub-table in the "extensions" table, so keep data separate from the main program and other extensions. When these settings are saved in the settings.json file, they will look like this. Although generated images must be square, I opted to save both width and height in the settings, for possible future compatibility.
    "extensions": { "openai": { "apikey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "prompt": "tiling seamless texture warehouse painted concrete wall abandoned dirty", "size": [ 1024, 1024 ] } }, Finally, we call the module function to generate the image, which may take a couple of minutes. If its successful we load the resulting image as a pixmap, create a texture from it, and then open that texture in a new asset editor window. This is done to eliminate the asset path, so the asset editor doesn't know where the file was loaded from. We also make a call to AssetEditor:Modify, which will cause the window to display a prompt if it is closed without saving. This prevents the user's project folder from filling up with a lot of garbage images they don't want to keep.
    if extension.openai.newimage(sz, sz, prompt, savepath) then local pixmap = LoadPixmap(savepath) if pixmap ~= nil then local tex = CreateTexture(TEXTURE_2D, pixmap.size.x, pixmap.size.y, pixmap.format, {pixmap}) tex.name = "New texture" local asseteditor = program:OpenAsset(tex) if asseteditor ~= nil then asseteditor:Modify() else Print("Error: Failed to open texture in asset editor."); end end else Print("Error: Image generation failed.") end The resulting extension provides an interface we can use to generate a variety of interesting textures. I think you will agree, these are quite a lot better than what we had just a few months ago.





    Of course the Lua debugger in Visual Studio Code came in very handy while developing this.

    That's pretty much all there is to the Lua side of things. Now let's take a closer look at the module code.
    The Module
    Lua modules provide a mechanism whereby Lua can execute C++ code packed into a dynamic linked library. The DLL needs to contain one retired function, which is luaopen_ plus the name of the module, without any extension. The module file is "openai.dll" so we will declare a function called luaopen_openai:
    extern "C" { __declspec(dllexport) int luaopen_openai(lua_State* L) { lua_newtable(L); int sz = lua_gettop(L); lua_pushcfunction(L, openai_newimage); lua_setfield(L, -2, "newimage"); lua_pushcfunction(L, openai_setapikey); lua_setfield(L, -2, "setapikey"); lua_pushcfunction(L, openai_getlog); lua_setfield(L, -2, "getlog"); lua_settop(L, sz); return 1; } } This function creates a new table and adds some function pointers to it, and returns the table. (This is the table we will store in extension.openai). The functions are setapikey(), getlog() and newimage().
    The first function is very simple, and just provides a way for the script to send the user's API key to the module:
    int openai_setapikey(lua_State* L) { APIKEY.clear(); if (lua_isstring(L, -1)) APIKEY = lua_tostring(L, -1); return 0; } The getlog function just returns any printed text, for extra debugging:
    int openai_getlog(lua_State* L) { lua_pushstring(L, logtext.c_str()); logtext.clear(); return 1; } The newimage function is where the action is at, but there's actually two overloads of it. The first one is the "real" function, and the second one is a wrapper that extracts the right function arguments from Lua, and then calls the real function. I'd say the hardest part of all this is interfacing with the Lua stack, but if you just go carefully you can follow the right pattern.
    bool openai_newimage(const int width, const int height, const std::string& prompt, const std::string& path) int openai_newimage(lua_State* L) This is done so the module can be easily compiled and tested as an executable.
    The real newimage function is where all the action is. It sets up a curl instance and communicates with a web server. There's quite a lot of error checking in the response, so don't let that confused you. If the call is successful, then a second curl object is created in order to download the resulting image. This must be done before the curl connection is closed, as the server will not allow access after that happens:
    bool openai_newimage(const int width, const int height, const std::string& prompt, const std::string& path) { bool success = false; if (width != height or (width != 256 and width != 512 and width != 1024)) { Print("Error: Image dimensions must be 256x256, 512x512, or 1024x1024."); return false; } std::string imageurl; if (APIKEY.empty()) return 0; std::string url = "https://api.openai.com/v1/images/generations"; std::string readBuffer; std::string bearerTokenHeader = "Authorization: Bearer " + APIKEY; std::string contentType = "Content-Type: application/json"; auto curl = curl_easy_init(); struct curl_slist* headers = NULL; headers = curl_slist_append(headers, bearerTokenHeader.c_str()); headers = curl_slist_append(headers, contentType.c_str()); nlohmann::json j3; j3["prompt"] = prompt; j3["n"] = 1; switch (width) { case 256: j3["size"] = "256x256"; break; case 512: j3["size"] = "512x512"; break; case 1024: j3["size"] = "1024x1024"; break; } std::string postfields = j3.dump(); curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers); curl_easy_setopt(curl, CURLOPT_URL, url.c_str()); curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postfields.c_str()); curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback); curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer); auto errcode = curl_easy_perform(curl); if (errcode == CURLE_OK) { //OutputDebugStringA(readBuffer.c_str()); trim(readBuffer); if (readBuffer.size() > 1 and readBuffer[0] == '{' and readBuffer[readBuffer.size() - 1] == '}') { j3 = nlohmann::json::parse(readBuffer); if (j3.is_object()) { if (j3["error"].is_object()) { if (j3["error"]["message"].is_string()) { std::string msg = j3["error"]["message"]; msg = "Error: " + msg; Print(msg.c_str()); } else { Print("Error: Unknown error."); } } else { if (j3["data"].is_array() or j3["data"].size() == 0) { if (j3["data"][0]["url"].is_string()) { std::string s = j3["data"][0]["url"]; imageurl = s; // I don't know why the extra string is needed here... readBuffer.clear(); // Download the image file auto curl2 = curl_easy_init(); curl_easy_setopt(curl2, CURLOPT_URL, imageurl.c_str()); curl_easy_setopt(curl2, CURLOPT_WRITEFUNCTION, WriteCallback); curl_easy_setopt(curl2, CURLOPT_WRITEDATA, &readBuffer); auto errcode = curl_easy_perform(curl2); if (errcode == CURLE_OK) { FILE* file = fopen(path.c_str(), "wb"); if (file == NULL) { Print("Error: Failed to write file."); } else { auto w = fwrite(readBuffer.c_str(), 1, readBuffer.size(), file); if (w == readBuffer.size()) { success = true; } else { Print("Error: Failed to write file data."); } fclose(file); } } else { Print("Error: Failed to download image."); } curl_easy_cleanup(curl2); } else { Print("Error: Image URL missing."); } } else { Print("Error: Data is not an array, or data is empty."); } } } else { Print("Error: Response is not a valid JSON object."); Print(readBuffer); } } else { Print("Error: Response is not a valid JSON object."); Print(readBuffer); } } else { Print("Error: Request failed."); } curl_easy_cleanup(curl); return success; } My first attempt at this used a third-party C++ library for OpenAI, but I actually found it was easier to just make the low-level CURL calls myself.
    Now here's the kicker: Any web API in the world will work with almost the exact same code. You now know how to build extensions that communicate with any website with a web interface, including SketchFab, CGTrader, itch.io, and easily interact with them in the Ultra Engine editor. The full source code for this and other Ultra Engine modules is available here:
    https://github.com/UltraEngine/Lua
    This integration of text-to-image tends to do well with base textures that have a uniform distribution of detail. Rock, concrete, ground, plaster, and other materials look great with the DALL-E 2 generator. It doesn't do as well with geometry details and complex structures, since the AI has no real understanding of the purpose of things. The point of this integration was not to make the end-all-be-all AI texture generation. The technology is changing rapidly and will undoubtedly continue to advance. Rather, the point of this exercise was to demonstrate a complex feature added in a Lua extension, without being built into the editor's code. By releasing the source for this I hope to help people start developing their own extensions to add new peripheral features they want to see in the editor and game engine.
     
  4. Josh

    Ultra Engine glTF extensions

    By Josh,


    Articles
    As I have explained before, I plan for Ultra Engine to use glTF for our main 3D model file format, so that your final game models can be easily loaded back into a modeling program for editing whenever you need. glTF supports a lot of useful features and is widely supported, but there are a few missing pieces of information I need to add into it. Fortunately, this JSON-based file format has a mechanism for extensions that add new features and data to the format. In this article I will describe the custom extensions I am adding for Ultra Engine.
    ULTRA_triangle_quads
    All glTF models are triangle meshes, but we want to support quad meshes primarily because its better for tessellation. This extension gets added to the primitives block. If the "quads" value is set to true, this indicates that the triangle indices are stored in a manner such that the first four indices of every six indices form a quad:
    "extensions": { "ULTRA_triangle_quads": { "quads": true } } There is no other glTF extension for quads, and so there is no way to export a glTF quad mesh from any modeling program. To get quad meshes into Ultra Engine you can load an OBJ file and then resave it as glTF. Here is a glTF file using quads that was created this way. You can see the tessellation creates an even distribution of polygons:

    For comparison, here is the same mesh saved as triangles and tessellated. The long thin triangles result in a very uneven distribution of polygons. Not good!

    The mesh still stores triangle data so the file can be loaded back into a 3D modeling program without any issues.
    Here is another comparison that shows how triangle (on the left) and quads (on the right) tessellate:

    ULTRA_material_displacement
    This extension adds displacement maps to glTF materials, in a manner that is consistent with how other textures are stored:
    "ULTRA_material_displacement": { "displacementTexture": { "index": 3, "offset": -0.035, "strength": 0.05 } } The extension indicates a texture index, a maximum displacement value in meters, and a uniform offset, also in meters. This can be used to store material displacement data for tessellation or parallax mapping. Here is a model loaded straight from a glTF file with displacement info and tessellation:

     
    If the file is loaded in other programs, the displacement info will just be skipped.
    ULTRA_vertex_displacement
    Our game engine uses a per-vertex displacement factor to control how displacement maps affect geometry. This extension adds an extra attribute into the primitives structure to store these values:
    "primitives": [ { "attributes": { "NORMAL": 1, "POSITION": 0, "TEXCOORD_0": 2 }, "indices": 3, "material": 0, "mode": 4, "extensions": { "ULTRA_vertex_displacement": { "DISPLACEMENT": 7 } } } } This can be used to prevent cracks from appearing  at texcoord seams.

    Here you can see the displacement value being loaded back from a glTF file it has been saved into. I'm using the vertex color to visually verify that it's working right:

    ULTRA_extended_material
    This extension adds other custom parameters that Ultra Engine uses. glTF handles almost everything we want to do, and there are just a few settings to add. Since the Ultra Engine material format is JSON-based, it's easy to just insert the extra parameters into the glTF like so:
    "ULTRA_extended_material": { "shaderFamily": "PBR", "shadow": true, "tessellation": true } In reality I do not feel that this extension is very well-defined and do not expect it to see any adoption outside of Ultra Engine. I made the displacement parameters a separate extension because they are well-defined, and there might be an opportunity to work with other application developers using that extension.
    Here we can see the per-material shadow property is disabled when it is loaded from the glTF:

    For comparison, here is what the default setting looks like:

    These extensions are simply meant to add special information that Ultra Engine uses into the glTF format. I do not currently have plans to try to get other developers to adopt my standards. I just want to add the extra information that our game engine needs, while also ensuring compatibility with the rest of the glTF ecosystem. If you are writing a glTF importer or exporter and would like to work together to improve the compatibility of our applications, let me know!
    I used the Rock Moss Set 02 model pack from Polyhaven in this article.
  5. Josh

    Building Universal Game Assets

    By Josh,


    Articles
    I have two goals for the art pipeline in the new game engine.
    Eliminate one-way 3D model conversions. Use the same game asset files on all platforms. In Leadwerks, 3D models get converted into the proprietary MDL format. Although the format is pretty straightforward and simple, there is no widespread support for loading it back into 3D modeling programs. This means you need to keep a copy of your 3D model in FBX and MDL format. You may possibly want to keep an additional file for the modeling program it was made in, such as 3D Studio Max MAX files. I wanted to simplify this by relying on the widely-support glTF file format, which can be used for final game-ready models, but can still easily loaded back into Blender or another program.
    Texture Compression
    Texture compression is where this all gets a lot more tricky. Texture compression is an important technique that can reduce video memory usage to about 25% what it would be otherwise. When texture compression is used games run faster, load faster, and look better because they can use higher resolution images. There are two problems with texture compression.
    Supported texture compression methods vary wildly across PC and mobile platforms. Many 3D modeling programs do not support modern texture compression formats. On the PC, BC7 is the best compression format to use for most images. BC5 is a two-channel format appropriate for normal maps, with Z reconstruction in the shader. BC6H can be used to compress HDR RGB images (mostly skyboxes). BC4 is a single-channel compressed format that is rarely used but could be useful for some special cases. The older DXTC formats should no longer be used, as they have worse artifacts with the same data size.
    Here are a few programs that do not support the newer BC formats, even though they are the standard for modern games:
    Windows Explorer Blender Microsoft 3D Object Viewer Now, the glTF model format does not actually support DDS format textures at all. The MSFT_texture_dds extension adds support for this by adding an extra image source into the file. glTF loaders that support this extension can load the DDS files, while programs that do not recognize this extension will ignore it and load the original PNG or JPEG files:
    "textures": [ { "source": 0, "extensions": { "MSFT_texture_dds": { "source": 1 } } } ], "images": [ { "uri": "defaultTexture.png" }, { "uri": "DDSTexture.dds" } ] I don't know any software that supports this extension except Ultra Engine. Even Microsoft's own 3D Object Viewer app does not support DDS files.
    The Ultra Engine editor has a feature that allows you to convert a model's textures to DDS. This will resave all textures in DDS format, and change the model materials to point to the DDS files instead of the original format the model uses (probably PNG):

    When a glTF file is saved after this step, the resulting glTF will keep copies of both the original PNG and the saved DDS files. The resulting glTF file can be loaded in Ultra Engine ready to use in your game, with DDS files applied, but it can still be loaded in programs that do not support DDS files, and the original PNG files will be used instead:

    The same model can be loaded back into Blender in case any changes need to be made. If you resave it, the references to the DDS images will be lost, so just repeat the DDS conversion step in the Ultra Engine to finalize the updated version of your model.

    To cut down on the size of your game files, you can just skip PNG files in the final packaging step so only DDS files are included.
    Basis Universal
    We still have to deal with the issue of hardware support for different codecs. This table from Unity's documentation shows the problem clearly:

    We do want to support some mobile-based VR platforms, so this is something that needs to be considered now. Basis Universal is a library from the author of Crunch that solves this problem by introducing a platform-agnostic intermediate compressed format that can be quickly transcoded into various texture compression formats, without going through a slow decompression and recompression step. We can generate Basis files in the Ultra Engine editor just like we did for DDS: There is a glTF extension that supports basis textures, so glTF models can be saved that reference the Basis files.

    The basis textures are stored in the glTF file the same way the DDS extension works:
    "textures": [ { "source": 0, "extensions": { "KHR_texture_basisu": { "source": 1 } } } ], "images": [ { "mimeType": "image/png", "bufferView": 1 }, { "mimeType": "image/ktx2", "bufferView": 2 } ] The resulting glTF files can be loaded as game-ready models with compressed textures on PC or mobile platforms, and can still be loaded back into Blender or another 3D modeling program. We don't have to store different copies of our textures in different compression formats or go through a slow conversion step when publishing the game to other platforms. If your game is only intended to run on PC platforms, then DDS is the simplest path to take, but if you plan to support mobile-based VR devices in the future then Basis solves that problem nicely.
    Here is our revised table:

    I was also working with the KTX2 format for a while, but I ended up only using it for its support for Basis compression. DDS and Basis cover all your needs and are more widely supported.
    Basis also supports a very lossy but efficient ETC1S method which is similar to Crunch, as well as uncompressed texture data, but in my opinion the one purpose for the Basis format is its UASTC format.
    This article features the lantern sample glTF model from Microsoft.
  6. Josh
    Package plugins are now supported in Ultra Engine 1.0.2. This allows the engine to handle other package formats besides just ZIP. In order to test this out, I created the Quake Loader plugin, which currently supports the following formats used in the original Quake game:
    PAK WAD BSP (textures) SPR LMP Why Quake? Well, the original Quake game is what got me into game development through modding, but Quake is also great for testing because it's so weird. This game was written before hardware accelerated graphics existed, so it has a lot of conventions that don't match modern 3D standards:
    All animation is vertex morphing (no skeletal animation) Texture coordinates are in pixels / integers instead of floating point values Three different types of "packages" (PAK, WAD, BSP), the latter two of which can be stored inside the first (packages-in-packages) Image files are stored seven different ways: LMP image files SPR sprite files Color pallette ('@') Mipmapped texture in WAD ('D') Console picture ('E') Status bar pictures ('B') Mipmapped textures stored in BSP Each of the image storage methods store data in slightly different ways. Sometimes the width and height are packed into the image data, sometimes they are found elsewhere in a separate structure, sometimes they are hard-coded, and sometimes you have to guess the image dimensions based on the data size. It's a mess, and it seems like this was made up on-the-fly by several people, but this was cutting-edge technology at the time and I don't think usability outside the specified game they were working on was a big concern.
    There are some interesting challenges there, but this forms the basis of several dated but influential games. As time went by, the design became more and more sane. For example, Quake 3 just stores textures as TGA images in a ZIP file with the extension changed to PK3. So I figure if I can make the plugin system flexible enough to work with Quake then it will be able to handle anything.
    Originally I tried implementing this using HLLib from Nem's Tools. This library supports Quake formats and more, but I had a lot of problems with it, and got better results when I just wrote my own PAK loading code. WAD files were an interesting challenge. There is no real Quake texture format, so loading the raw texture data from the package made no sense. The most interesting breakthrough in this process was how I handled WAD textures. I finally figured out I can make the WAD loader return a block of data that forms a DDS file when a texture is loaded, even through the texture name has no extension. This tricks the engine into thinking the WAD package contains a DDS file which can easily be loaded, when the reality is quite different. This introduces an important concept, that package plugins don't necessarily have to return the exact data they contain, but instead can return files in a ready-to-use format, and the game engine will never know the difference.
    The resulting plugin allows you to easily extract and load textures from the Quake game files. This code will extract textures from a single WAD file.
    #include "UltraEngine.h" using namespace UltraEngine; void main(int argc, const char* argv[]) { // Load Quake file loader plugin auto qplg = LoadPlugin("Plugins/QuakeLoader"); // Load FreeImage texture plugin auto fiplg = LoadPlugin("Plugins/FITextureLoader"); // WAD to download WString wadfile = "bigcastle"; // Download a WAD package DownloadFile("https://www.quaketastic.com/files/texture_wads/" + wadfile + ".wad", wadfile + ".wad"); // Load the package auto wad = LoadPackage(wadfile + ".wad"); // Create a subdirectory to save images in CreateDir(wadfile); // Read all files in the package auto dir = wad->LoadDir(""); for (auto file : dir) { // Load the image auto pm = LoadPixmap(file); // Save as a PNG file if (pm) pm->Save(wadfile + "/" + file + ".png"); } OpenDir(wadfile); } All the files found in the WAD package are saved as PNG images. BSP files will work exactly the same way.

    This code will load a Quake package, extract all the textures from all the maps in the game, and save them to a folder on the desktop. This is done by detecting packages-in-packages (WAD and BSP), which return the file type '3', indicating that they can be treated both as a file and as a folder. Since none of these package formats use any compression, pixmaps can be easily loaded straight out of the file without extracting the entire BSP. Since Quake PAK files don't use compression, the whole system just turns into a very complicated blob of data with pointers that store data all over the place:
    #include "UltraEngine.h" using namespace UltraEngine; void main(int argc, const char* argv[]) { // Load Quake file loader plugin auto qplg = LoadPlugin("Plugins/QuakeLoader"); // Load FreeImage texture plugin auto fiplg = LoadPlugin("Plugins/FITextureLoader"); // Path to Quake game WString qpath = "C:/Program Files (x86)/Steam/steamapps/common/Quake"; //WString qpath = "D:/SteamLibrary/steamapps/common/Quake"; // Load game package auto pak = LoadPackage(qpath + "/id1/PAK1.PAK"); //Change the current directory to load files with relative paths ChangeDir(qpath + "/id1"); // Create a folder to save images in WString savedir = GetPath(PATH_DESKTOP) + "/Quake"; CreateDir(savedir); // Load the main package directory auto dir = pak->LoadDir("maps"); for (auto file : dir) { // Print the file name Print(file); // Get the file type int type = FileType("maps/" + file); // If a package-in-a-package is found load its contents (BSP and WAD) if (type == 3) { auto subdir = pak->LoadDir("maps/" + file); for (int n = 0; n < subdir.size(); ++n) { // Load the texture (plugin will return DDS files) auto pm = LoadPixmap("maps/" + file + "/" + subdir[n]); // Save to PNG if (pm) pm->Save(savedir + "/" + subdir[n] + ".png"); } } } OpenDir(savedir); } Here is the result:

    Since all of these images can be loaded as pixmaps, does that mean they can also be loaded as a texture? I had to know the answer, so I tried this code:
    #include "UltraEngine.h" using namespace UltraEngine; int main(int argc, const char* argv[]) { //Get the displays auto displays = GetDisplays(); //Create a window auto window = CreateWindow("Ultra Engine", 0, 0, 1280, 720, displays[0], WINDOW_CENTER | WINDOW_TITLEBAR); //Create a world auto world = CreateWorld(); //Create a framebuffer auto framebuffer = CreateFramebuffer(window); //Create a camera auto camera = CreateCamera(world); camera->SetClearColor(0.125); camera->SetPosition(0, 0, -2); camera->SetFov(70); //Create a light auto light = CreateBoxLight(world); light->SetRotation(45, 35, 0); light->SetRange(-10, 10); light->SetColor(2); //Create a model auto model = CreateBox(world); // Load Quake file loader plugin auto qplg = LoadPlugin("Plugins/QuakeLoader"); // WAD to download WString wadfile = "bigcastle"; // Download a WAD package DownloadFile("https://www.quaketastic.com/files/texture_wads/" + wadfile + ".wad", wadfile + ".wad"); // Load the package auto wad = LoadPackage(wadfile + ".wad"); // Load a texture from the WAD package auto tex = LoadTexture("sclock"); // Create a material auto mtl = CreateMaterial(); mtl->SetTexture(tex); // Apply the material to the box model->SetMaterial(mtl); //Main loop while (window->Closed() == false and window->KeyDown(KEY_ESCAPE) == false) { model->Turn(0, 1, 0); world->Update(); world->Render(framebuffer); } return 0; } Here is the result, a texture loaded straight out of a Quake WAD file!

    To reiterate, testing with Quake files helped me to come up with two important design features:
    Packages-in-packages, indicated with the file type '3'. Packages do not necessarily have to return the same exact data they contain, and can instead prepare the data in a ready-to-use format, when that is desirable. The decision to base file type detection on the contents of the file instead of the file name extension worked well here, and allowed me to load the extension-less WAD texture names as DDS files. Some of these images came from "bigcastle.wad". I don't know who the original author is.
    If you are interested, you can read more about the weird world of Quake file formats here, although I must warn you that specification is not complete! 
  7. Josh

    Editor Development Continues

    By Josh,


    Articles
    I wanted to take some time to investigate geospatial features and see if it was possible to use GIS systems with Ultra Engine. My investigations were a success and resulted in some beautiful lunar landscapes in a relatively short period of time in the new game engine.

    I have plans for this, but nothing I can say anything specific about right now, so now I am working on the editor again.
    Leadwerks had a very well-defined workflow, which made it simple to use but also somewhat limited. Ultra goes in the opposite direction, with extremely flexible support for plugins and extensions. It's a challenge to keep things together because if you make things to flexible it just turns into indecisiveness, and nothing will ever get finished that way. I think I am finding a good balance or things that should be hard-coded and things that should be configurable.
    Instead of separate windows for viewing models, textures, and materials, we just have one asset editor window, with tabs for separate items. The texture interface is the simplest and just shows some information about the asset on the left-side panel.

    When you select the Save as menu item, the save file dialog is automatically populated with information loaded from all the plugins the engine has loaded. You can add support for new image formats at any time just by dropping a plugin in the right folder. In fact, the only hard-coded format in the list is DDS.

    You can save to an image format, or to the more advanced DDS and KTX2 formats, which include mipmaps and other features. Even more impressive is the ability to load Leadwerks texture files (TEX) and save them directly into DDS format in their original pixel format, with no loss of image quality. Here we have a Leadwerks texture that uses DXT1 compression, converted to DDS without recompressing the pixels:

    There are several tricks to make the new editor start up as fast as possible. One of these is that the various windows the editor uses don't actually get created until the first time they are used. This saves a little bit of time to give the application a snappier feel at startup.
    I hope the end result will provide a very intuitive workflow like the Leadwerks editor has, with enough power and flexibility to make the editor into a living growing platform with many people adding new capabilities for you to use. I think this is actually my favorite stuff to work on because this is where all the technology is exposed to you, the end user, and I get to carefully hand-craft an interface that makes you feel happy when you use it.
  8. Josh

    AI-Generated Game Textures and Concept Art

    By Josh,


    Articles
    Midjourney is an AI art generator you can interact with on Discord to make content for your game engine. To use it, first join the Discord channel and enter one of the "newbie" rooms. To generate a new image, just type "/imagine" followed by the keywords you want to use. The more descriptive you are, the better. After a few moments four different images will be shown. You can upsample or create new variations of any of the images the algorithm creates.

    And then the magic begins:

    Here are some of the images I "created" in a few minutes using the tool:

    I'm really surprised by the results. I didn't think it was possible for AI to demonstrate this level of spatial reasoning. You can clearly see that it has some kind of understanding of 3D perspective and lighting. Small errors like the misspelling of "Quake" as "Quke" only make it creepier, because it means the AI has a deep level of understanding and isn't just copying and pasting parts of images.
    What do you think about AI-generated artwork? Do you have any of your own images you would like to show off? Let me know in the comments below.
  9. Josh
    As I have stated before, my goa for this game enginel is not to build a marketplace of 3D models, but instead to just make sure our model loading code reliably loads all 3D models that are compliant with the glTF specification. I started testing more 3D models from Sketchfab, and found that many of them are using specular/gloss materials. At first I thought I could just fudge the result, but I wasn't getting very good results, and the Windows 10 3D Object Viewer was showing them perfectly. This made me very upset because I feel that the software I make for you should be the absolute best thing possible. So I went ahead and implemented the actual KHR_materials_pbrSpecularGlossiness extension. Here are the results:

    It looks quite good without breaking the specular/gloss approach. I think this would be an ideal way to upgrade the Leadwerks renderer without making changes that are too big.
    To implement this, I used the glTF sample viewer source code, which is a bit of a reference renderer for glTF materials. It has changed quite a lot since I first used it as a guide to implementing PBR materials. I decided I might as well integrated the latest code into our renderer. The shader code is very well organized, so it was pretty simple to integrate into my latest code. The glTF materials system is great because it provides a standardized materials system with a lot of advanced effects, along with a reference renderer that shows exactly how the materials are supposed to appear. This means that any engine or tool that is conforms to the glTF standard will always display the same materials the same way, while still allowing additional more specialized features like shadows and post-processing effects to be added.
    In addition to revising our metal-roughness code, I went ahead and added support for some more features. The KHR_materials_clearcoat extension basically makes cars look like cars:
    It adds an extra layer of reflections that gives a dark surface a big boost, and can even use a separate normal map for the clearcoat layer:

    The KHR_materials_sheen extension makes cloth look like cloth. You can see for yourself in the image below. The appearance kind of makes my skin crawl because I can't stand the feeling of velvet / velour. Apparently this is some kind of common tactile hypersensitivity and I actually get a feeling like electricity shooting through my spine just looking at this image, so I think the effect is working well:

    Another creepy example. Just the thought of running my hand over that fuzzy orange fabric is making me shiver:

    The KHR_materials_transmission extensions provides a physically accurate model for transparent surfaces, with support for refraction. Here's a shot of refraction bending the background image:

    This also supports roughness, for making the refracted background appear blurry like frosted glass:

    I feel the background is too pixellated and should be improved. This problem occurs in the reference renderer, and I have several ideas on how to solve it.
    The transparency system is very advanced, and introduces a lot of interesting new commands to the engine:
    Material::SetIndexOfRfraction(const float ior) Material::SetThickness(const float thickness) Material::SetTransmission(const float transmission) These features will give you reliable support for loading glTF models from any source, as well as the best image quality possible for your games. I think realistic materials in VR will also make a big difference in bridging the uncanny valley. And all of this is done with the fastest possible performance for VR.
    And yes, of course I used a gamepad to do all of this.
     
     
  10. Josh

    Ultra Engine Client App

    By Josh,


    Articles
    I've been working hard getting all the rendering features to work together in one unified system. Ultra Engine, more than any renderer I have worked on, takes a lot of different features and integrates them into one physically-based graphics system. A lot of this is due to the excellent PBR materials system that Khronos glTF provides, and then there are my own features that are worked into this, like combined screen-space and voxel ray traced reflections.
    Anyways, it's a lot of difficult work, and I decided to take a "break" and focus something else for a few days.
    Before Leadwerks was on Steam, it had a web installer that would fetch a list of files from our server and any files that were missing, or were newer than the locally stored ones. There was no system for detecting updates, you just pressed the update button and the updater ran. The backend for this system was designed by @klepto2 and it functioned well for what it needed to do.

    With Ultra App Kit, I created a simple tool to generate projects. This introduced the account authentication / signin system, which was sort of superfluous for this application, but it was a good way to test it out:

    With Ultra Engine, I wanted some characteristics of both these applications. I wrote a new backend in about a day that handles updates. A PHP script authenticates the user and verifies product ownership, fetches a list of files, and retrieves files. Since C++ library files tend to be huge, I found it was necessary to add a compression system, so the script returns a zip compressed file. Of course, you don't want the server to be constantly creating zip files, so it caches the file and updates the zip only when I upload a new copy of the file. There's also a cache for the info retrieval, which is returned in JSON format, so it's easy to read in C++.
    For the front end, I took inspiration from the Github settings page, which I thought looked nice:

    And here's what I came up with. Projects will show when they are outdated and need to be updated (if a file in the template the project came from was changed). Each of the sections contains info and links to various topics. There's a lot there, but none of it feels extraneous to me. This is all made with the built-in GUI system. No HTML is used at all:

    The Invision Power REST API is extremely interesting. It allows authentication of accounts and purchases, but it can be made to do a lot of other things.
    Post a forum topic: https://invisioncommunity.com/developers/rest-api?endpoint=forums/topics/POSTindex Upload an image to the gallery: https://invisioncommunity.com/developers/rest-api?endpoint=gallery/images/GETindex Download a file: https://invisioncommunity.com/developers/rest-api?endpoint=downloads/files/GETitem Unlock achievements: https://invisioncommunity.com/developers/rest-api?endpoint=core/members/POSTitem_achievements_awardbadge None of that is very important right now, but it does provide some interesting ideas for future development of the game engine.
  11. Josh

    Exporting a glTF model from 3ds Max

    By Josh,


    Articles
    Autodesk 3ds Max now supports export of glTF models, as well as a new glTF material type. The process of setting up and exporting glTF models is pretty straightforward, but there are a couple of little details I wanted to point out to help prevent you from getting stuck. For this article, I will be working with the moss rocks 1 model pack from Polyhaven.
    Getting geometry into 3ds Max is simple enough. I imported the model as an FBX file.

    To set up the material, I opened the compact material editor and set the first slot to be a glTF material.

    Press the button for the base color map, and very importantly choose the General > Bitmap map type. Do not choose OSL > Bitmap Lookup or your textures won't export at all.

    Select your base color texture, then do the same thing with the normal and roughness maps, if you have them. 3ds Max treats metal / roughness as two separate textures, although you might be able to use the same texture both if it grabs the data from the green (roughness) and blue (metal) channels. This is something I don't know yet.

    Select the File > Export menu item to bring up the glTF export dialog. Uncheck the "Export glTF binary" option because we don't want to pack our model and textures into a single file: I don't know what the baked / original material option does because I don't see any difference when I use it.

    At this point you should have a glTF file that is visible in any glTF model viewer.

    Now something slightly weird max does is it generates some new textures for some of the maps. This is probably because it is combining different channels to produce final images. In this case, none of our textures need to be combined, so it is just a small annoyance. A .log file will be saved as well, but these can be safely deleted.

    You can leave the images as-is, or you can open up the glTF file in a text editor and manually change the image file names back to the original files:
    "images": [ { "uri": "rock_moss_set_01_nor_gl_4k.jpg" }, { "uri": "M_01___Defaultbasecolortexture.jpeg" }, { "uri": "M_01___Defaultmetallicroughnesstex.jpeg" } ], Finally, we're going to add LODs using the Mesh Editor > ProOptimizer modifier. I like these settings, but the most important thing is to make sure "Keep textures" is checked. You can press the F3 key at any time to toggle wireframe view and get a better view of what the optimizer does to your mesh.

    Export the file with the same name as the full-resolution model, and add "_lod1" to the end of the file name (before the extension). Then repeat this process saving lod2 and lod3 using 25% and 12.5% for the vertex reduction value in the ProOptimizer modifier.
    Here is my example model you can inspect:
    mossrock1a.zip
    Now it is very easy to get 3D models from 3ds max to your game engine.
  12. Josh
    As the first release of Ultra Engine approaches, it seems clear that the best way to maximize its usefulness is to make it as compatible as possible with the Leadwerks game engine. To that end, I have implemented the following features.
    Native Loading of Leadwerks File Formats
    Ultra Engine loads and saves DDS, glTF, and OBJ files. Other formats are supported by plugins, both first and potentially third-party, for PNG, JPG, BMP, TGA, TIFF, GIF, HDR, KTX2, and other files. Additionally, all Leadwerks file formats are natively supported without requiring any plugins, so you can load TEX, MDL, PHY, MAT files, as well as full Leadwerks maps. You also have the option to save Leadwerks game engine formats in more up-to-date file formats such as glTF or DDS files with BC5/BC7 compression.

    Testing a scene from the excellent Cyclone game (available on Steam!)
    Classic Specular Shader Family
    PBR materials are wonderful but act quite differently from conventional specular/gloss game materials. I've added a "Classic" shader family which provides an additive specular effect, which may not be realistic but it makes materials expect the way you would expect them to, coming from Leadwerks game engine. When Leadwerks MAT files are loaded, this shader family is automatically applied to them, to get a closer match to their appearance in Leadwerks.
    Leadwerks Translation Layer
    I'm experimenting with a Leadwerks header for Ultra Engine. You can drop this into your Ultra Engine project folder and then start running code with the Leadwerks API. Internally, the Leadwerks game engine commands will be translated into the Ultra Engine API to execute your existing code in the new engine. I don't think this will be 100% perfect due to some differences in the way the two engines work, but I do think it will give you an easy way to get started and provide a gentler transition to Ultra.
    This code will actually work with both engines:
    #include "Leadwerks.h" using namespace Leadwerks; int main(int argc, const char* argv[]) { Window* window = Window::Create(); Context* context = Context::Create(window); World* world = World::Create(); Camera* camera = Camera::Create(); camera->Move(0, 0, -3); Light* light = DirectionalLight::Create(); light->SetRotation(35, 35, 0); Model* model = Model::Box(); model->SetColor(0.0, 0.0, 1.0); while (true) { if (window->Closed() || window->KeyDown(Key::Escape)) return false; model->Turn(0, Time::GetSpeed(), 0); Time::Update(); world->Update(); world->Render(); context->Sync(false); } return 0; } I hope these enhancements give you a more enjoyable experience using Ultra together with the Leadwerks Editor, or as a standalone programming SDK.
  13. Josh
    HDR skyboxes are important in PBR rendering because sky reflections get dampened by surface color. It's not really for the sky itself, but rather we don't want bright reflections to get clamped and washed out, so we need colors that go beyond the visible range of 0-1.
    Polyhaven has a large collection of free photo-based HDR environments, but they are all stored in EXR format as sphere maps. What we want are cubemaps stored in a single DDS file, preferably using texture compression.
    We're going to use this online tool to convert the sphere maps to cubemaps. There are other ways to do this, and some may be better, but this is what I was able to find right away. By default, the page shows a background we can use for testing:

    Let's process and save the image. Since we are just testing, it's fine to use a low resolution for now:

    The layout is very important. We will select the last option, which exports each face as a separate image:

    Processing and saving the image will result in a zip file downloaded to your computer, with six files:
    px.hdr nx.hdr py.hdr ny.hdr pz.hdr nz.hdr These images correspond to the positive and negative directions on the X, Y, and Z axes.
    We will start by just converting a single image, px.hdr. We will convert this pixmap to the format TEXTURE_BC6H, which corresponds to VK_FORMAT_BC6H_UFLOAT_BLOCK in Vulkan, or DXGI_FORMAT_BC6H_UF16 in the DDS file format. We need to load the FreeImage plugin for HDR file import, and the ISPC plugin for fast BC6H compression. The R16G16B16A16_SFLOAT is an intermediate format we need to convert the 32-bit floating point HDR file to, because the ISPC compressor expects this as the input format for BC6H compression:
    auto plg = LoadPlugin("Plugins/FITextureLoader"); auto plg2 = LoadPlugin("Plugins/ISPCTexComp"); auto pixmap = LoadPixmap(GetPath(PATH_DESKTOP) + "/px.hdr"); pixmap = pixmap->Convert(TextureFormat(VK_FORMAT_R16G16B16A16_SFLOAT)); pixmap = pixmap->Convert(TEXTURE_BC6H); pixmap->Save(GetPath(PATH_DESKTOP) + "/px.dds"); When we open the resulting DDS file in Visual Studio it appear quite dark, but I think this is due to how the HDR image is stored. I'm guessing colors are being scaled to a range between zero and one in the HDR file:

    Now that we know how to convert a save a single 2D texture, let's try saving a more complex cube map texture with mipmaps. to do this, we need to pass an an array of pixmaps to the SaveTexture function, containing all the mipmaps in the DDS file, in the correct order. Microsoft's documentation on the DDS file format is very helpful here.
    std::vector<std::shared_ptr<Pixmap> > mipchain; WString files[6] = { "px.hdr", "nx.hdr", "py.hdr", "ny.hdr", "pz.hdr", "nz.hdr" }; for (int n = 0; n < 6; ++n) { auto pixmap = LoadPixmap(GetPath(PATH_DESKTOP) + "/" + files[n]); pixmap = pixmap->Convert(TextureFormat(VK_FORMAT_R16G16B16A16_SFLOAT)); Assert(pixmap); mipchain.push_back(pixmap->Convert(TEXTURE_BC6H)); while (true) { auto size = pixmap->size; size /= 2; pixmap = pixmap->Resize(size.x, size.y); Assert(pixmap); mipchain.push_back(pixmap->Convert(TEXTURE_BC6H)); if (size.x == 4 and size.y == 4) break; } } SaveTexture(GetPath(PATH_DESKTOP) + "/skybox.dds", TEXTURE_CUBE, mipchain, 6); when we open the resulting DDS file in Visual Studio we can select different cube map faces and mipmap levels, and see that things generally look like we expect them to:

    Now is a good time to load our cube map up in the engine and make sure it does what we think it does:
    auto skybox = LoadTexture(GetPath(PATH_DESKTOP) + "/skybox.dds"); Assert(skybox); world->SetSkybox(skybox); When we run the program we can see the skybox appearing. It still appears very dark in the engine:

    We can change the sky color to lighten it up, and the image comes alive. We don't have any problems with the dark colors getting washed out, because this is an HDR image:
    world->SetSkyColor(4.0);
    Now that we have determined that our pipeline works, let's try converting a sky image at high resolution. I will use this image from Polyhaven because it has a nice interesting sky. However, there's one problem. Polyhaven stores the image in EXR format, and the cube map generator I am using only loads HDR files. I used this online converter to convert my EXR to HDR format, but there are probably lots of other tools that can do the job.
    Once we have the new skybox loaded in the cubemap generator, we will again process and save it, but this time at full resolution:

    Save the file like before and run the DDS creation code. If you are running in debug mode, it will take a lot longer to process the image this time, but when it's finished the results will be worthwhile!:

    This will be very interesting to view in VR. Our final DDS file is 120 MB, much smaller than the 768 MB an uncompressed R16G16B16A16 image would require. This gives us better performance and requires a lot less hard drive space. When we open the file in Visual Studio we can browser through the different faces and mipmap levels, and everything seems to be in order:

    Undoubtedly this process will get easier and more automated with visual tools, but it's important to get the technical basis laid down first. With complete and in-depth support for the DDS file format, we can rely on a widely supported format for textures, instead of black-box file formats that lock your game content away.
  14. Josh

    Finalizing Terrain

    By Josh,


    Articles
    I'm wrapping up the terrain features now for the initial release. Here's a summary of terrain in Ultra Engine.
    Terrains are an entity, just like anything else, and can be positioned, rotated, or scaled. Non-square terrains are supported, so you can create something with a 1024x512 or whatever resolution. (Power-of-two sizes are required.)
    Editing Terrain
    Ultra Engine includes an API that lets you modify terrain in real-time. I took something very complicated and distilled it down to a very simple API for you:
    terrain->SetElevation(x, y, height); That's it! The engine will automatically update normals, physics, raycasting, rendering, and other data for you behind the scenes any time you modify the terrain, in a manner that maximizes efficiency.
    Materials
    Leadwerks supports 32 terrain texture layers. Unity supports eight. Thanks to the flexibility of shaders in Vulkan, any terrain in Ultra can have up to a whopping 256 different materials applied to it, and it will always be rendered in a single pass at maximum speed.
    To apply a material at any point on the terrain, you just call this method. The weight value lets you control how much influence the material as at that point, i.e. its alpha transparency:
    terrain->SetMaterial(x, y, material, weight) This example shows how to paint materials onto the terrain.

    I'm quite happy with how the documentation system has turned out, and I feel it is representative of the quality I want your entire user experience to be.

    All the API examples load media from the web, so there's no need to manually download any extra files. Copy and paste the code into any project, and it just works.
    //Create base material auto diffusemap = LoadTexture("https://raw.githubusercontent.com/Leadwerks/Documentation/master/Assets/Materials/Ground/river_small_rocks_diff_4k.dds"); auto normalmap = LoadTexture("https://raw.githubusercontent.com/Leadwerks/Documentation/master/Assets/Materials/Ground/river_small_rocks_nor_gl_4k.dds"); auto ground = CreateMaterial(); ground->SetTexture(diffusemap, TEXTURE_DIFFUSE); ground->SetTexture(normalmap, TEXTURE_NORMAL); Tessellation and Displacement
    Tessellation works great with terrain, which is modeled with quads. There are a lot of high-quality free PBR materials that are great for VR available on the web, so you will have no shortage of interesting materials to paint your terrains with. Here are a few sites for you to peruse:
    https://matlib.gpuopen.com/main/materials/all
    https://icons8.com/l/3d-textures
    https://ambientcg.com/
    https://polyhaven.com/textures
    https://freepbr.com/
    https://www.cgbookcase.com/textures

    Cutting Holes
    You can cut holes in the terrain by hiding any tile. This is useful for making caves, and can be used in the future for blending voxel geometry into a heightmap terrain.

    Shader Design
    Shaders in Ultra are big and quite complicated. I don't expect anyone to make 100% custom shaders like you could with the simpler shaders in Leadwerks. I've structured shaders so that the entry point shader defines a user function, then includes the base file, which calls the function:
    #version 450 #extension GL_GOOGLE_include_directive : enable #extension GL_ARB_separate_shader_objects : enable #define LIGHTING_PBR #define USERFUNCTION #include "../Base/Materials.glsl" #include "../Base/TextureArrays.glsl" void UserFragment(inout vec4 color, inout vec3 emission, inout vec3 normal, inout float metalness, inout float roughness, inout float ambientocclusion, in vec3 position, in vec2 texcoords, in vec3 tangent, in vec3 bitangent, in Material material) { // Custom code goes here... } #include "../Base/base_frag.glsl" This organizes shaders so custom behavior can be added on top of the lighting and other systems, and paves the way for a future node-based shader designer.
    Future Development
    Ideas for future development to add to this system include voxel-based terrains for caves and overhangs, roads (using the decals system), large streaming terrains, and seamless blending of models into the terrain surface.
  15. Josh

    Smart Tessellation

    By Josh,


    Articles
    I've actually been doing a lot of work to finalize the terrain system, but I got into tessellation, and another rabbit hole opened up. I've been thinking about detailed models in VR. Tessellation is a nice way to easily increase model detail. It does two things:
    Curved surfaces get smoother (using point-normal triangles or quads) A displacement map can be used to make small geometric detail to a surface. These are really nice features because they don't require a lot of memory or disk space, and they're automatic. However, tessellation also has a nasty tendency to create cracks in geometry, so it tends to work best with organic models like rocks that are carefully wrapped. I was able to mitigate some of these problems with a per-vertex displacement value, which dampens displacement wherever there is a mesh seam:

    However, this does not help round edges become rounder. With this cylinder, because the faces are not a continuous rounded surface, a crack appears at the sharp edges:

    What if there was a way to properly tessellate mechanical shapes like this? Wouldn't it be great if all your models could just automatically gain new detail that scales with the camera distance, with no need for extra Lod versions?
    Well, I had an idea how it might be possible, and a few hours later I came up with something. Here is the original model with no tessellation applied:


    Here is the same model with my new and improved tessellation shader, suitable for organic and mechanical shapes:


    A closeup view reveals perfectly aligned geometry with no gaps or cracks:

    The distribution of polygons can probably be made more uniform with additional work. This feature uses two additional bytes in the vertex structure to control a normal for tessellated curves. I think it will be possible to automatically detect seams and assign tessellation normals in the final editor. If no separate tessellation normals are assigned, then the default tessellation behavior occurs. This is a very promising technique because it could allow you to add a lot of detail to any model very easily with barely any effort.
  16. Josh
    Ultra Engine makes much better use of class encapsulation with strict public / private / protected members. This makes it so you can only access parts of the API you are meant to use, and prevents confusion about what is and isn't supported. I've also moved all internal engine commands into separate namespaces so you don't have to worry about what a RenderMesh or PhysicsNode does. These classes won't appear in the intellisense suggestions unless you were to add "using namespace UltraRender" to your code, which you should never do.
    There's still a lot of extra stuff intellisense shows you from Windows headers and other things:

    Using the __INTELLISENSE__ macro in Visual Studio, I was able to trick the intellisense compiler into skipping include files that link to third party libraries and system headers the user doesn't need to worry about. This strips all the unnecessary stuff out of the API, leaving just basic C++ commands, some STL classes, and the Ultra Engine API. With all the garbage removed, the API seems much friendlier:

    This is one more detail that I feel helps make C++ with Ultra Engine about as easy as C# programming. You can disable this feature by adding _ULTRA_SIMPLE_API=0 in your project macros.
  17. Josh

    Shadow Filtering

    By Josh,


    Articles
    Happy Friday! I am taking a break from global illumination to take care of some various remaining odds and ends in Ultra Engine.
    Variance shadow maps are a type of shadowmap filter technique that use a statistical sample of the depth at each pixel to do some funky math stuff. GPU Gems 3 has a nice chapter on the technique. The end result is softer shadows that run faster. I was wondering where my variance shadow map code went, until I realized this is something I only prototyped in OpenGL and never implemented in Vulkan until now. Here's my first pass at variance shadow maps in Vulkan:

    There are a few small issues but they are no problem to work out. The blurring is taking place before the scene render, in the shadow map itself, which is a floating point color texture instead of a depth texture. (This makes VSMs faster than normal shadow maps.) The seams you see on the edges in the shot above are caused by that blurring, but there's a way we can fix that. If we store one sharp and one blurred image in the variance shadow map, we can interpolate between those based on distance from the shadow caster. Not only does this get rid of the ugly artifacts (say goodbye to shadow acne forever), but it also creates a realistic penumbra, as you can see in the shot of my original OpenGL implementation. Close to the shadow caster, the shadow is well-defined and sharp, but it gets much blurrier the further away it gets from the object:

    Instead of blurring the near image after rendering, we can use MSAA to give it a fine-but-smooth edge. There is no such thing as an MSAA depth shadow sampler in GLSL, although I think there should be, and I have lobbied on behalf of this idea.

    Finally, in my Vulkan implementation I used a compute shader instead of a fragment shader to perform the blurring. The advantage is that a compute shader can gather a bunch of samples and store them in memory, then access them to process a group of pixels at once. Instead of reading 9x9 pixels for each fragment, it can read a block of pixels and process them all at once, performing the same number of image writes, but much fewer reads:
    // Read all required pixel samples x = int(gl_WorkGroupID.x) * BATCHSIZE; y = int(gl_WorkGroupID.y) * BATCHSIZE; for (coord.x = max(x - EXTENTS, 0); coord.x < min(x + BATCHSIZE + EXTENTS, outsize.x); ++coord.x) { for (coord.y = max(y - EXTENTS, 0); coord.y < min(y + BATCHSIZE + EXTENTS, outsize.y); ++coord.y) { color = imageLoad(imagearrayCube[inputimage], coord); samples[coord.x - int(gl_WorkGroupID.x) * BATCHSIZE + EXTENTS][coord.y - int(gl_WorkGroupID.y) * BATCHSIZE + EXTENTS] = color; } } This same technique will be used to make post-processing effects faster. I previously thought the speed of those would be pretty much the same in every engine, but now I see ways they can be improved significantly for a general-use speed increase. @klepto2 has been talking about the benefits of compute shaders for a while, and he is right, they are very cool. Most performance-intensive post-processing effects perform some kind of gather operation, so compute shaders can make a big improvement there.
    One issue with conventional VSMs is that all objects must cast a shadow. Otherwise, an object that appears in front of a shadow caster will be dark. However, I added some math of my own to fix this problem, and it appears to work with no issues. So that's not something we need to worry about.
    All around, variance shadow maps are a big win. They run faster, look better, and eliminate shadow acne, so they basically kill three birds with one stone.
  18. Josh
    I'm finalizing the shaders, and I was able to pack a lot of extra data into a single entity 4x4 matrix:
    Orthogonal 4x4 matrix RGBA color Per-entity texture mapping offset (U/V), scale (U/V), and rotation Bitwise entity flags for various settings Linear and rotational velocity (for motion blur) Skeleton ID All of that info can be fit into just 64 bytes. The shaders now use a lot of snazzy new function like this:
    void ExtractEntityInfo(in uint id, out mat4 mat, out vec4 color, out int skeletonID, out uint flags) mat4 ExtractCameraProjectionMatrix(in uint cameraID, in int eye) So all the storage / decompression routines are in one place instead of hard-coding a lot of matrix offset in different shaders.
    A per-entity linear and angular velocity is being calculated and sent to the rendering thread. This is not related to the entity velocity in the physics simulation, although they will most often be the same. Rather, this is just a measure of the distance the entity moved since the last world render, so it will work with any motion, physics or otherwise.
    In the video below, the linear velocity is being added to the camera position to help predict the region of the GI lighting area, so the camera stays roughly in the center as the GI updates. A single 128x128x128 volume texture is being used here, with a voxel size of 0.25 meters. In the final version, you probably want to use 3-4 stages, maybe using a 64x64x64 texture for each stage.
    Performance of our voxel cone step tracing is quite good, with FPS in the mid-hundreds on a mid-range notebook GPU. The last big question is how to handle dynamic objects, particular fast-moving dynamic objects. I have some ideas but I'm not sure how hard this is going to be. My first attempt was slow and ugly. You can probably guess which dragon is static and which is dynamic:

    It is a difficult problem. I would like the solution to be as cool as the rest of this system as I finish up this feature. Also, I think you will probably want to still use SSAO so dynamic objects blend into the GI lighting better.
    Everyone should have access to fast real-time global illumination for games.
  19. Josh

    Multiple Light Bounces

    By Josh,


    Articles
    I have more than one light bounce working now, and it looks a lot nicer than single-bounce GI. The ambient light here is pure black. All light is coming off from the direct light, and bouncing off surfaces.

    It will take some time to get the details worked out, and more bounces will require more memory. I'm actually kind of shocked how good looking it is. This is just a single 128x128x128 volume texture at 0.25 meters per voxel. Light leaks seem to be not a problem, even at that low resolution, even with thin walls.
    You can see that absolutely no direct light is hitting the red wall on the left, and the ambient light is pure black. Yet light is still bouncing off the floor onto the wall, and then lighting the dragon very faintly with red light. (The exposure here is higher than it should be so it looks a little washed out):

    There's a lot here for me to take in. Real-time global illumination is such a rabbit hole that keeps unfolding new layers of complexity and possibilities. It seems like every few days my mind is totally blown and I look at the system differently, and the new way makes so much sense. In the last three days I have said to myself "I'll just try one more thing..." and then that thing opens up a lot of new possibilities. April is nearly over, and I have been working on just this feature since December!
    I've got a bunch of stuff to work out, but it looks like this will run really really well in VR.
  20. Josh
    Previously I wrote about introducing latency to the voxel cone step tracing realtime global illumination system. The idea here is to improve performance and quality, at the cost of a small delay when the GI calculation gets updated. The diffuse GI lighting gets cached so the final scene render is very fast.
    Here's what a gradual GI update does. Of course, this will be running unseen in the background for the final version, but this shows what is actually happening:

    My new video project1.mp4 Vulkan has a sophisticated system for supporting multiple device queues. I initially thought I could just run the GI update on a separate queue, like a low-priority CPU thread running in the background:

    Unfortunately, this is not really possible on today's hardware. Even with the lowest queue priority setting, the GI queue hogs the whole GPU and causes the rendering queue to stall out. So I had to stick with doing everything in the main render queue.
    The GI calculation only updates when the camera moves a certain distance. There is a latency setting that controls how many substeps the task is broken up into. High latency means many small steps, so the framerate will not dip much when GI is updating. Low latency means the GI will significantly decrease the framerate every time the camera triggers an update. It is possible to set up a combination of high resolution and low latency that will cause the render queue to stall out. If this happens the program will encounter a VK_ERROR_DEVICE_LOST error. I don't know how to prevent this for now, other than just don't use ridiculous settings.
    Here you can see the GI updating seamlessly as the camera moves around. I actually have to use three sets of volume textures, one for rasterization and direct lighting, and then the final GI data is stored in a set of two textures that alternate back and forth. This allows me to display the previous results while the next update is still processing, and the two textures get swapped when the update is finished.

    962180265_Mynewvideoproject1.mp4 A couple of small issues remain.
    The transition between textures needs to be smoothed, to handle changes to the environment. I need a way to calculate the diffuse GI for dynamic objects that don't appear in the GI reflections. (I have an idea.) The items are not too terribly difficult and they will be done in short order. I'm very happy with how this has turned out. It provides quality real-time global illumination without compromising performance, and it will work very well with VR.
  21. Josh
    Since previously determining that voxels alone weren't really capable of displaying artifact-free motion, I have been restructuring the GI system to favor speed with some tolerance for latency. The idea is that global illumination will get updated incrementally in the background over the course of a number of frames, so the impact of the calculation per frame is small. The new GI result is then smoothly interpolated from the old one, over a period of perhaps half a second. Here's a shot of the restructured system sort-of-working, at least it's writing to and reading from the correct textures, though the lighting is not correct here:

    Diffuse lighting gets calculated for each voxel, for each of six directions, so the final scene render just takes the normal and uses the dot product to figure out which GI samples to use in the voxel that point intersects. This greatly lightens the load of the final scene render, since it no longer has to perform several costly cone-step rays. Since specular reflection is view-dependent, that part must be ray-traced in the final scene render, but there are some optimizations I have yet to make that will probably double the framerate here. My best estimate right now is that the GI system reduces framerate by 30% when enabled, and that it's not really dependent on scene complexity. In other words, as the scene gets more objects, the performance cost should stay about the same, and we have already seen that Ultra Engine scales extremely well with scene complexity. So everything looks very good.
    From what I have seen, real-time global illumination is the killer feature everyone wants. Spending a few extra weeks on this to get a better result makes sense. I don't think I've ever worked on a feature or system that required so much iteration. I get the system working, start using it in different scenes, testing performance, and then I realize what needs to be different. With this feature, that process seems very extreme, but I really think this is the last restructure needed to get the best result possible for your games. Delivering real-time global illumination while maintaining fast framerates is very much inline with the design goals of Ultra Engine.
  22. Josh

    Finalizing Direct Lighting

    By Josh,


    Articles
    I have not used the engine outside the editor in a while, but I needed to for performance testing, so now I am back to real-time rendering. During the development of the GI system I broke most of the light types, so I had to spend a couple of days getting those to work again. While doing this, I decided to resolve some longstanding issues I have put off.
    First, the PBR lighting will use a default gradient in place of the skybox, if no reflection map is set for the world. This is based off the ambient light level, so it will blend in with any scene. You can disable this by setting a black skybox, but this will give some acceptable default reflectance when an empty scene is created and it prevents unlit areas from looking completely flat:

    Shadow acne will be a thing of the past. I found that using vkCmdSetDepthBias to use the hardware depth bias feature does a great job of eliminating shadow acne. This shot shows a light that is extremely close to the floor, yet no artifacts are visible:

    Of course the new strip lights are can be used for dramatic effects:

    My new video project1.mp4 I have not yet implemented shadows for directional lights. I was hoping this whole time I would come up with a brilliant idea that would make cascaded shadow maps obsolete, but that's the one are I haven't found any great new innovative technique to use.
    Box lights are another new type of light, and can be used to simulate directional lights coming into a window, for levels that are mostly indoors.
    The realtime GI system has been very popular. I am rolling around an idea to increase the quality and performance of the system, with the flip side of that being increased latency. But looking around at the world, the general global illumination never really changes suddenly all at once, so I think it is worth trying. The whole point of Ultra Engine is that it should run very fast, so if I can make the GI system favor performance, hopefully to where it has no impact at all on framerate, I think that is more consistent with our goals.
  23. Josh

    Voxel Cone Step Tracing Refinement

    By Josh,


    Articles
    Before proceeding with multiple GI volumes, I decided to focus on just getting the lighting to look as close to perfect as possible, with a single stage.
    Injecting the ambient light into the voxel data made flat-lit areas appear much more "3D", with color bleeding and subtle contours everywhere.
    Lighting only:

    Lighting + albedo

    Some adjustments to the way the sky color is sampled gave a more lifelike appearance to outdoor lighting.
    Before:

    After. Notice the shaded area still has a lot of variation:

    Initial performance testing gives results consistent with my expectations. I'm running at half of 1920x1080 resolution, on a GEForce 1660 TI, and performance is about a third what it would be without GI. At 1920x1080, that drops to 90 FPS. Because it is so resource-intensive, I plan to render the effect at half-resolution, then upscale it and use an edge detection filter to fill in info for any pixels that need it. This card has only 1536 stream processors, about half as much as a 2080.

    Further experiments with motion did not resolve the artifacts I was experiencing earlier, and in fact caused new ones because of the flickering introduced by the GPU voxelization. You can read a detailed discussion of these issues on the Gamedev.net forum. My conclusion for now is that moving objects should not be baked into the voxelized data, because they cause a lot of flashing and flickering artifacts. These could be added in the future by storing a separate voxel grid for each dynamic object, along with some kind of data structure the shader can use to quickly find the objects a ray can pass through.
    This is great though, because it means voxelization only has to be updated when the camera moves a certain distance, of if a new static object is created or deleted. You still have completely dynamic direct lighting, and the GI system just follows you around and generates indirect lighting on the fly. I could run the update in the background and then show a smooth transition in between updates, and all the flickering problems go away. Performance should be very good once I have further optimized the system. And every surface in your game can show reflections everywhere. Moving lights work really really well, as you have seen.
    The end is in sight and I am very pleased how this system is turning out. My goal was to create a completely dynamic system that provided better 3D reflections than cubemaps, and did not require manual placement or baking of probes, fast enough to use on older mid-range discrete GPUs, and that is what we got.
  24. Josh
    Until now, all my experiments with voxel cone step tracing placed the center of the GI data at the world origin (0,0,0). In reality, we want the GI volume to follow the camera around so we can see the effect everywhere, with more detail up close. I feel my productivity has not been very good lately, but I am not being too hard on myself because this is very difficult stuff. The double-blind nature of it (rendering the voxel data and then using that data to render an effect) makes development very difficult. The intermediate voxel data consists of several LODs and is difficult to visualize. My work schedule lately has been to do nothing for several days, or just putter around with some peripheral tasks, and then suddenly solve major problems in a short two-hour work session.
    Here you can see a single GI stage following the camera around properly. More will be added to increase the area the effect covers, and the edges of the final stage will fade out the effect for a smooth transition:

    My new video project1.mp4 This all makes me wonder what "work" is when you are handling extremely difficult technical problems. I have no problem working 8+ hours a day on intermediate programming tasks, but when it comes to these really advanced problems I can't really be "on" all day. This morning, I went for a walk, for seven miles. Was I subconsciously working during that time, so that I can later sit down and quickly solve a problem I was completely stuck on previously?
    I definitely underestimated the difficulty of making this feature work as a robust engine feature that can be used reliably. There is a lot of nuance and small issues that come up when you start testing in a variety of scenes, and this information could easily fill an hour-long talk about the details of voxel cone step tracing. However, there is just one more step, to make the moving volumes work with multiple GI stages. Once that is working I can proceed with more testing, look for artifacts to eliminate, and optimize speed.
    This is the last big feature I have to finish. It seems fitting that I should get one final big challenge before completing Ultra Engine, and I am enjoying it.
  25. Josh
    Adding emission into the cascaded voxel cone step tracing global illumination and dynamic reflections system (SEO ftw) was simple enough:
    There's some slight trailing but it looks okay to me. There is a bit of a "glitch" in that when the emissive surface gets near the wall, the ambient occlusion kicks in, even though the sphere is self-illuminating. This happens because the emission color is mixed with the light voxel during the rasterization step. I could fix this by storing emission in a separate texture, but I don't think the increase memory and processing time are justifiable and this is acceptable as-is.
    Changing the center of the voxel grid will probably cause the whole scene to display the same trailing, and at that point I think that would look really bad. I'm thinking the contents of the volume texture probably need to be shifted over when the camera moves one voxel unit. Can a texture be copied to itself in Vulkan? Let's see:
    Hmmm, that still does not tell me if I can copy a section of overlapping memory in a texture.  Hmmm, this might be what I am looking for...
    So I probably need to keep an extra "transfer" texture around to act as an intermediate image when copying the contents of the volume texture to itself.
Copyright © 2024 Ultra Software.
All rights reserved.
×
×
  • Create New...