This often goes unnoticed because support for viewing alpha channels is almost universally bad in all paint programs and image viewers. Here is what the image looks like with the alpha channel active:

Although Leadwerks is smart enough to weight pixels by their alpha value when creating the downsampled mipmap chain, the linear filter the GPU uses will still show the background color on the edge of the solid area. This results in a strange darkening or lightening of your vegetation. If you see the vegetation getting darker with distance, the background of the transparent area is probably black. If you see flickering white pixels, the transparent area is probably white.

The script below will downsample the image, using the alpha-weighted technique Leadwerks uses, and then apply the blurred color to the transparent background of the original image, resulting in an image that looks like this:

This will eliminate the artifacts you are seeing in your masked vegetation and provide clear but smooth alpha-masked surfaces, free of pixelation.

local file = RequestFile("Open File", CurrentDir().."/")
if file == "" then return end

local pixmap = LoadPixmap(file)
if pixmap == nil then return end

local mipmaps = pixmap:BuildMipchain()
local n = #mipmaps - 3
if n < 1 then n = 1 end

local x, y, rgba, r, g, b, a
local changed = false

for x = 0, pixmap.size.x - 1 do
    for y = 0, pixmap.size.y - 1 do
        rgba = pixmap:Sample(iVec2(x, y))
        if rgba.a < 1.0 then
            background = mipmaps[n]:Sample((x + 0.5) / pixmap.size.x, (y + 0.5) / pixmap.size.y)
            rgba.r = Mix(background.r, rgba.r, rgba.a)
            rgba.g = Mix(background.g, rgba.g, rgba.a)
            rgba.b = Mix(background.b, rgba.b, rgba.a)
            pixmap:WritePixel(x, y, rgba)
            changed = true
        end
    end
end

if changed then
    pixmap:Save(StripExt(file).."_fixed."..ExtractExt(file))
else
    Print("No transparent pixels")
end

The Breakable Model

As usual, I like to start with the artwork, because it's key to everything. I usually learn so much just by using actual game-ready artwork, rather than blocky "programmer art". Plus, at the end I get something I am proud of showing off. @reepblue created this model and provided it for use with Leadwerks, and @Rich D textured it and broke it up into fragments, for your destructive enjoyment.

In fact, breakable objects are primarily an art problem. The basic concept is very simple: When an object is damaged, hide it and replace it with a series of fragments that represent the object broken into pieces. This is not a complicated programming problem at all, but let's dig into the details and see what happens.

Colliders

The first question is how the colliders for the model fragments should be made.

If your colliders fit too perfectly, the broken object will form a strong interlocking structure, and won't fall apart unless a large force is applied to it. Even worse, if your fragment colliders intersect the moment they are enabled, the object will form explosive forces that could send the crate or the player flying! We definitely don't want this happening.

The main question is, should fragments of the same object even intersect at all? I was curious how Half-Life 2 handled this, so I fired it up, broke a crate, picked up one of the pieces, and saw that fragments of a broken object don't collide with each other at all! Here I am holding one piece, and it freely intersects the other fragments with no collision, even if I drop it. I have 36 hours of play time in this game, and I never noticed this until now!

Well, if this solution is good enough for Valve, then I don't feel a need to make it more complicated than it needs to be. This simplifies our goals because we can just set the collision type of the fragments to "Debris", which will collide with the Prop and Scene collision types, but not with the player, and not with other broken pieces. That makes it perfect for this use, and we don't have to worry too much about the exact shape of the colliders for each piece of our broken object. If the fragment colliders overlap, it won't cause explosive forces or lock the pieces into place like a jigsaw puzzle.

The next question is how should the content be structured? I have a single-mesh unbroken version of the model, as well as a version that contains all the fragments separated into child limbs. I could save each limb as a separate model and maybe use some naming scheme to identify them, like crate_frag1.mdl, crate_frag2.mdl, etc. However, I think the simplest way to handle this is to make the unbroken version of the model the top-level entity in the hierarchy, and then just add the fragments as children in the model file. The script will just make the assumption that all children in the model are fragments, and hide them when the game starts:

• Unbroken mesh
  • Fragment 1
  • Fragment 2
  • Fragment 3

I think this design accounts for the most likely use case, and making it more complicated would be more trouble than its worth.

I noticed that in the current build of Leadwerks Editor, the model editor only allows assigning a collider to the whole object. Let's fix that, so we can assign a separate collider to any object in the hierarchy!

I performed that for each limb in the broken crate model.

I then opened the unbroken crate model, and assigned a box collider to it, and saved it:

Merging the Models

Now how to we combine these two models? An artist might want to export the two models to glTF and combine them in Blender or another program, but since I am a programmer, the easiest solution for me is to just write a simple script.

local mdl1 = LoadModel(nil, "Models/Crate/crate01.mdl")
local mdl2 = LoadModel(nil, "Models/Crate/cratebroken01.mdl")

while #mdl2.kids > 0 do
    local child = mdl2.kids[1]
    child:SetParent(mdl1)
end

mdl1:Save("Models/Crate/Crate01_Combined.mdl")

The only thing to watch out for is that when were call SetParent we are modifying the list of objects we are iterating through. Therefore, it's best to use a while loop and just get the first child each time. This tripped me up for a few minutes because I started with a simple for loop!  Here's the wrong way that I tried first:

--Don't do this!
for n = 1, #mdl2.kids do
    local child = mdl2.kids[n]
    child:SetParent(mdl1)
end

I just copied and pasted my script code into the editor console, pressed enter, and it saves the model with the unbroken single mesh as the top-level limb, and all the fragments as children. Separate colliders for each limb are retained correctly:

Awesome!

Now that the artwork is correctly prepared, the actual code this requires will be very easy.

Breakable Object Script

Let's create a new entity script called "Breakable" in the Physics folder. We'll add a property for health, and another for the speed of a collision that will cause the object to break:

Breakable = {}
Breakable.health = 10 --"Health"
Breakable.breakspeed = 10 --"Break speed"

In the Start function, we'll just hide all the child limbs, since we have decided already that those will form the fragments of the object when it breaks:

function Breakable:Start()	
	local n
	for n = 1, #self.kids do --No problems using a for loop here since we are not modifying the entity limbs ;)
		self.kids[n]:SetMass(0)--This should already be zero, but if not let's set it automatically
		self.kids[n]:SetCollisionType(COLLISION_NONE)
		self.kids[n]:SetHidden(true)
		--self.kids[n]:SetColor(0,4,0,1) --optional visual hint for testing
	end
end

We will add a function so that the object will accept damage if something hits it. Most likely this will be caused by the player shooting or hitting object with a melee weapon, but if an enemy attacks the crate for any reason, it will also work.

function Breakable:Damage(amount)
    if self.health > 0 then
        self.health = self.health - amount
        if self.health <= 0 then
            self:Break()
        end
    end
end

In the collide function, we can add a check to see if the break speed was hit, and break the object if it was:

function Breakable:Collide(entity, position, normal, speed)
    if speed > self.breakspeed then
        self:Break()
    end
end

Finally, we add our Break function, where the object finally gets broken apart into pieces:

function Breakable:Break()
    if self:GetHidden() then return end --Make sure this only gets broken once
    if #self.kids == 0 then return end --If there are no kids then don't do anything

    --Hide the object
    self:SetHidden(true)
	
    --Calculate simple mass for each piece
    local mass = self:GetMass()
    if mass == 0 then
        mass = 5
    else
        mass = mass / #self.kids
    end
	
    --Create a table to store the fragments in
    self.fragments = {}
    while #self.kids > 0 do
        local child = self.kids[1]
        child:SetParent(nil)
        child:SetMass(mass)
        child:SetHidden(false)		
        child:SetCollisionType(COLLISION_DEBRIS)
        table.insert(self.fragments, child)
    end
end

Running the Game

I was very excited when I ran the game for the first time, in a level with a crate hooked up to this script. What would happen if I shot the object? Would it break into pieces and look natural, like I hoped it would? Here is the result:

Wow, on the first try it looks great! Adding sounds for collision and breaking would enhance the feel of it even more, and I plan to add this next. I also plan to try adding some velocity to the broken fragments, so the crate has more of an explosive appearance, instead of just falling apart in place.

Here is the complete script, with the accompanying properties definition file. The script requires a game created with the current "beta-e" branch on Steam:

Breakable.luaBreakable.json

And here are the model files I used. Crate01_Combined.mdl is the one that is ready-to-use with this script:

Crate01.zip

I even find the code itself for this very beautiful, for some reason. The weird comments with the @ signs are annotations that give Lua Language Server a hint about what type each parameter is, for nice auto-completion in our script editor. These comments are automatically generated when the script is first created.

---@class Breakable : Entity
Breakable = {}
Breakable.health = 10 --"Health"
Breakable.breakspeed = 10 --"Break speed"

---@param self Breakable
function Breakable:Start()	
    local n
    for n = 1, #self.kids do --No problems using a for loop here since we are not modifying the entity limbs ;)
        self.kids[n]:SetMass(0)--This should already be zero, but if not let's set it automatically
        self.kids[n]:SetCollisionType(COLLISION_NONE)
        self.kids[n]:SetHidden(true)
        --self.kids[n]:SetColor(0,4,0,1) --optional visual hint for testing
    end
end

---@param self Breakable
---@param entity Entity
---@param position Vec3
---@param normal Vec3
---@param speed number
function Breakable:Collide(entity, position, normal, speed)
    if speed > self.breakspeed then
        self:Break()
    end
end

---@param self Breakable
---@param amount number
function Breakable:Damage(amount)
    if self.health > 0 then
        self.health = self.health - amount
        if self.health <= 0 then
            self:Break()
        end
    end
end

---@param self Breakable
function Breakable:Break()
    if self:GetHidden() then return end --Make sure this only gets broken once
    if #self.kids == 0 then return end --If there are no kids then don't do anything

    --Hide the object
    self:SetHidden(true)
	
    --Calculate simple mass for each piece
    local mass = self:GetMass()
    if mass == 0 then
        mass = 5
    else
        mass = mass / #self.kids
    end
	
    --Create a table to store the fragments in
    self.fragments = {}
    while #self.kids > 0 do
        local child = self.kids[1]
        child:SetParent(nil)
        child:SetMass(mass)
        child:SetHidden(false)		
        child:SetCollisionType(COLLISION_DEBRIS)
        table.insert(self.fragments, child)
    end
end

If you are looking for custom artwork for your game, I can wholeheartedly recommend Rich DiGiovanni's services. I have worked with Rich on many projects, and I always get great results at a reasonable price: https://richdigiovanni.com/

Furniture is a great way to add physics interactions to your game, because each object can be set up one, and many instances can of the object can be placed in your levels. Cabinets and doors can swivel open and closed using a hinge joint. Drawers can slide in and out using a slider joint. This simple interactivity can turn a search for an important item into a mini-game, and encourages the player to explore the world in hopes of finding a valuable item.

I decided to start with a filing cabinet that @Rich created. Each drawer is a separate limb in the model, so they can move independently.

Colliders

This item will require five different oddly-shape colliders. One for the cabinet, and another for each of the four drawers. I want to be able to place items in the drawers and close the drawers, while still having collision with the exterior of the cabinet, so the colliders need to be pretty detailed.

The first thing I did was open the model in the model editor, delete everything except the bottom drawer, collapse it, then save that model as a separate drawer model.

Because the drawers are dynamic objects, they could not use polygon mesh colliders. There isn't a single collider shape that is appropriate for drawers, and a convex hull would not give me the abililty to place items in the drawer. What I needed was a multi-part collider made up of convex hulls.

I added the drawer model to a new scene, positioned it as (0,0,0), and started building a collision shape out of brushes. I saved this as a map file and will keep it in case I want to modify the collider a bit more in the future: My minimum thickness here is four centimeters, a number I will keep in mind in the future and revise if it turns out this is needed after play testing.

I selected all the brushes, right clicked in the scene editor, and selected the "Save as Model" popup menu item, and saved a MDL file. Now I had a model file with separate limbs for the different convex hulls I wanted to make the shape out of.

I did the same thing for the cabinet exterior. Entities that are connected to each other with a joint will not collide with each other, fortunately, so we don't have to worry about making the drawer and cabinet colliders match exactly. If the drawers intersect the file cabinet collider a little bit, it's not a problem. The individual drawers will collide with each other, since they are not connected to the same joint, so take care that they do not interpenetrate.

I am not sure yet if I want the file cabinets to be immovable or dynamic. Since it's not difficult, I will make the collider out of separate convex pieces so I can make it dynamic in the future if I decide I want to.

Now we are ready to add the colliders to the models. I opened the drawer model in the model editor and selected the new File > Load Dynamic Collider menu item. This will load a model and turn each limb into a separate convex hull, for multi-part colliders. It worked perfectly!

For this to work as expected, you usually need the root limb in your visual model to be positioned at (0,0,0), with rotation (0,0,0), and scale (1,1,1). You may get weirdly rotated, offset, or scaled results if this is not the case. Fortunately, you can reset the model's transformation matrix just by selecting the Tools > Reset Transform menu item. You might need to do this to the visual model, the collider model, or both!

Now that I had my individual models, with colliders, I added the file cabinet to the scene, added four drawers, positioned them each in the right place, and parented them to the cabinet. Positioning drawers was actually pretty easy and their placement does not have to be exact, so don't get obssessive about making them perfect.

I saved the assembly as a prefab, and now I can place the cabinet anywhere I like. I could even save this assembly as a new model file, and it would retain the colliders for each limb, but I think I am going to leave it in its separate pieces, referencing the file cabinet and draw models. This way I can easily modify those models in the future, if I am not happy with how their colliders are shaped.

Since I ended up just using one repeating drawer model, I recommend all furniture in the future be exported from the modeling program this way: the cabinet object with no drawers, doors, or whatever moving parts there are, and then one copy of the drawer or door. Since it's pretty easy to line the drawers up in the right place, you don't even need any nodes to hint where they are supposed to be placed.

Joints

The next step was to add a slider joint for each drawer. I added a new class that just creates a slider joint. There are two pieces of information I am interested in, the joint pin and the limits. The pin defines the vector the joint slides along, and the limits determine where the joint starts and stops. I want the drawer to be locked to the cabinet so it can't be pulled out all the way, and the joint limits will help me achieve that behavior. I could add more properties in the future, but for now let's just keep it simple.

SliderJoint.zip

In the Start method, a slider joint is created between the entity and its own parent. If there is no parent, this is actually okay. The joint will just be created as if a non-movable parent object is used.

void SliderJoint::Start()
{
    auto entity = GetEntity();
    if (entity->GetMass() == 0.0f)
    {
        Print("Error: Slider joint requires entity to have non-zero mass");
        return;
    }
    auto gpin = TransformVector(pin, entity, NULL);
    joint = CreateSliderJoint(entity->GetPosition(true), gpin, entity->GetParent(), entity);
    if (limits[1] > limits[0]) joint->SetLimits(limits[0], limits[1]);
}

I created a simple scene with a floor, a pivot for the player start position, and our filing cabinet. I selected one drawer, pulled it out a bit, set the pass to 2.0, and added the slider joint to it. If it works, I should be able to walk into the drawer and push it back into the cabinet.

When I run the game, I am able to push the drawer in a bit, but it seems like it is hitting something when it gets close to the filing cabinet. Since the drawer does not collide with the cabinet, it must be hitting the other drawers.

I went back to the editor and moved the drawer above and below out of the way, to make sure that was actually the problem:

When I ran the game I could push the drawer in all the way, and then some. Because there is no collision between the drawer and its parent, the cabinet itself, there is nothing to stop the drawer unless I set the joint limits. I'm not worried about this, so it's fine for now. I'm more worried about these collidring drawers!

If I pull the other drawers out all the way, the problem becomes can be easily seen. The front panel overlaps with the drawer above it. How should I resolve this? I already said I want 4 centimeters to be my minimum thickness, since very thin objects could potentially have problems with small objects falling through them.

Fortunately, I still have my map file where I created the drawer collider shape. I decided to just lower the height of the front piece so that it matched the sides and the back. Then I saved the brush objects as a model, and loaded it back into the fiel cabinet drawer model as a dynamic collider. We might change this design in the future, but for now I just want to get it working!

Here is the drawer with the new collider loaded:

When I load my scene back up, it looks like the drawers have plenty of clearance.

The game now works as expected. I can push the drawer in all the way without hitting anything.

I added the slider joint to the other drawers and set their mass to 2.0, and all the drawers could now be pushed freely.

I discovered an interesting problem while doing this! It's easy for the drawers to form a stair case shape. When this happens, the player will step almost instantly up all the drawers, to the top of the cabinet! In the future, maybe we will need a per-entity setting that makes it so an entity is non-steppable. For right now, I am not too concerned about it because I want to manually push and pull these drawers, not lean into them.

Tactile Physics

I love the player physics in Amnesia: The Dark Descent, and I wanted to bring this type of interaction to our game. It seemed like it would be achievable, since we are using the same excellent physics library, Newton Game Dynamics, that is in use by Frictional Games.

We already have logic in our player class for picking up objects, and to my delight this mostly worked with the drawers. When a new object is picked up, a kinematic joint is created. The kinematic joint will then exert force and torque on the object the player is grabbing.

if (pickedentity)
{
	float mass = pickedentity->GetMass();
	if (mass > 0.0f and mass < maxcarrymass + maxcarrytolerance)
	{
		carryjoint = CreateKinematicJoint(pickedentity->GetPosition(true), pickedentity);
		carryjoint->SetMaxForce(1000);
		carryjoint->SetMaxTorque(1000);
		carrycollisiontype = pickedentity->GetCollisionType();
		pickedentity->SetCollisionType(COLLISION_DEBRIS);
		carryposition = TransformPoint(pickedentity->GetPosition(true), NULL, camera);
		carryposition.z -= 0.1f;
		carryrotation = TransformRotation(pickedentity->GetQuaternion(true), NULL, camera);
		pickedobject.reset();
	}
}

I made a small improvement to disable torque if the picked up object has any joints attached to it. This makes sense, because you don't want to rotate the drawer, just move it on one axis:

if (not pickedentity->GetJoints().empty()) carryjoint->SetMaxTorque(0);

When the game is played, the physics basically work the way I wanted. Of course there are no joint limits here, so there is nothing from stopping the drawers from going through the cabinet, or being thrown very far away.

I set the joint limits property to [-40, 0], since the drawer starts closed by default.

When I ran the game again, here was the result. You can see the joint limits are working as intended.

The behavior is not perfect. It's jumpy and glitchy. I might need to treat jointed objects a little differently than objects you pick up. But it's basically working, and I am confident with more testing and refinement it will turn into a polished and fun game mechanic.

The last step was to save my assembly as a new prefab, now that I am pretty sure it's done.

Now I can place an infinite number of file cabinets, with an infinite number of interactive drawers, and force the player to search through them all! The true horror of the SCP beauracracy awaits you! Bwahahahahahaha!

As with all highly optimized code, the ray tracing commands are not quite as user-friendly as the rest of our API, but I will walk you through the steps to setting up a simple ray tracing test.

Getting Started

The first step is to create a ray trace device and a ray tracing scene:

// Create raytrace device    
auto rt = CreateRayTracer();

// Create RT scene
auto scene = CreateRTScene(rt);

A ray tracing scene is different from a Leadwerks Scene object. You can add mesh data directly to a scene:

//Create the scene ground
auto ground = CreateBox(NULL, 10, 0.2, 10);
auto rtground = scene->AddMesh(ground->GetMesh(0));

Instanced Geometry

Ray tracing scenes do not have any concept of entities like Leadwerks does. Any mesh you add to a scene just uses the mesh vertex positions without any transformation. However, you can make an instance of a scene, add it as a child of another scene, and apply a transform matrix to adjust its position and orientation.

Let's make a scene just to act as a container for a mesh we plan on making multiple instances of:

// Create RT scene that will be used as a container for the box mesh
auto meshcontainer = CreateRTScene(rt);

//Create a box mesh and add it to the container scene
auto box = CreateBox(NULL);
auto rtmesh = meshcontainer->AddMesh(box->GetMesh(0));

And now let's create an instance of the mesh container scene in the first scene we created:

// Create instances of the box scene and add them to the parent scene
auto A = CreateRTInstance(scene, meshcontainer);

We can now reposition instance A like so. We'll move it one meter to the left, and up 0.1 meters (half the ground height) plus 0.5 (half the box height):

A->SetPosition(-1, 0.6, 0);

Now let's create a second instance and move it one meter to the right:

// Create a second instance
auto B = CreateRTInstance(scene, meshcontainer);
B->SetPosition(1, 0.6, 0);

Tracing Rays

Now we have a scene with ground and two boxes, and we are ready to trace rays. Our ray tracing code is hyperoptimized, which means we need to feed it data in the exact format it wants. Instead of providing two points for the raycast, we are going to provide a starting point, a normalized direction vector, and a length for the direction vector.

// Define a ray
RTRay ray;
ray.origin = Vec3(-10, 0.6, 0);
ray.dir = Vec3(1, 0, 0);
ray.length = 20.0f;

// Perform single ray trace
auto rayinfo = scene->TraceRay(ray);

The returned rayinfo structure will give us information about the ray. The structure uses IDs to tell us what was hit.

• If the mesh ID equals RT_INVALID_ID then nothing was hit, otherwise something was hit.
• If the instance ID equals RT_INVALID_ID, then a mesh added directly to the scene was hit, and we can find it by the mesh ID.
• If the instance ID it not RT_INVALID_ID, then an instance of another scene that was added to this scene was hit.
• Note that RT_INVALID_GEOMETRY_ID does not equal zero, so a mesh ID of 0 indicates a valid mesh.

// Print out the ray hit information
if (rttrace.meshid == RT_INVALID_ID)
{
    Print("Ray missed");
}
else
{
    Print("Hit position: " + std::wstring(rttrace.position));
    if (rttrace.instid != RT_INVALID_ID)
    {
        if (rttrace.instid == A->id) Print("A hit (" + String(rttrace.instid) + ")");
        if (rttrace.instid == B->id) Print("B hit (" + String(rttrace.instid) + ")");
    }
    else
    {
        if (rttrace.meshid == rtground->id) Print("Ground hit (" + String(rtground->id) + ")");
    }
}

This is more complicated than the simpler Leadwerks raycasting commands, but this approach will also scale better when performing hundreds of thousands of raycasts.

Here is the complete code for the ray trace test program:

#include "Leadwerks.h"

using namespace UltraEngine;

int main(int argc, const char* argv[])
{
    // Create raytrace device    
    auto rt = CreateRayTracer();

    // Create RT scene
    auto scene = CreateRTScene(rt);

    //Create the scene ground
    auto ground = CreateBox(NULL, 10, 0.2, 10);
    auto rtground = scene->AddMesh(ground->GetMesh(0));

    // Create RT scene that will be used as a container for the box mesh
    auto meshcontainer = CreateRTScene(rt);

    //Create a box mesh and add it to the container scene
    auto box = CreateBox(NULL);
    auto rtmesh = meshcontainer->AddMesh(box->GetMesh(0));
    
    // Create instances of the box scene and add them to the parent scene
    auto A = CreateRTInstance(scene, meshcontainer);
    A->SetPosition(-1, 0.6, 0);

    // Create a second instance
    auto B = CreateRTInstance(scene, meshcontainer);
    B->SetPosition(1, 0.6, 0);
    
    // Define a ray
    RTRay ray;
    ray.origin = Vec3(-10, 0.6, 0);
    ray.dir = Vec3(1, 0, 0);
    ray.length = 20.0f;
    
    // Perform single ray trace
    auto rttrace = scene->TraceRay(ray);
    
    // Print out the ray hit information
    if (rttrace.meshid == RT_INVALID_ID)
    {
        Print("Ray missed");
    }
    else
    {
        Print("Hit position: " + std::wstring(rttrace.position));
        if (rttrace.instid != RT_INVALID_ID)
        {
            if (rttrace.instid == A->id) Print("A hit (" + String(rttrace.instid) + ")");
            if (rttrace.instid == B->id) Print("B hit (" + String(rttrace.instid) + ")");
        }
        else
        {
            if (rttrace.meshid == rtground->id) Print("Ground hit (" + String(rtground->id) + ")");
        }
    }

    return 0;
}

The program will output this text when run, indicating that instance A is the first hit object, along with the position the intersection occurs at.

Hit position: -1.5, 0.6, 0
A hit (1)

Ray Filtering

You can use the instance and mesh filter feature to make a ray that only intersects some geometry. Here we will set a bitwise mask so that instance A gets skipped, and instance B will be hit. Just add this code right before the scene->TraceRay() call.

// Assign obect masks
A->SetMask(1);
B->SetMask(2);

And then set an additional parameter in the TraceRay call to define the mask:

// Perform single ray trace
auto rttrace = scene->TraceRay(ray, 2);

Filtering will work on both a per-instance and per-mesh level. The default mask value for instances is 1 and the default mask value for raytrace meshes is -1u (all flags).

The output of our modified program shows that instance A is being ignored, and now instance B is the one that gets hit:

Hit position: 0.499999, 0.6, 0
B hit (2)

Advanced Usage

The code below demonstrates a very simple real-time lightmapper using the ray tracing system.

#include "Leadwerks.h"

using namespace UltraEngine;

int main(int argc, const char* argv[])
{
    auto displays = GetDisplays();
    auto win = CreateWindow("", 0, 0, 1280, 720, displays[0]);
    auto fb = CreateFramebuffer(win);
    auto world = CreateWorld();
    auto cam = CreateCamera(world);
    cam->SetClearColor(0.125f);

    auto ground = CreateBox(world, 10, 1, 10);
    auto mtl = CreateMaterial();
    ground->SetMaterial(mtl);
    
    int sz = 128;
    auto pixmap = CreatePixmap(sz, sz);
    auto tex = CreateTexture(TEXTURE_2D, sz, sz);
    mtl->SetTexture(tex);

    auto caster = CreateBox(world, 8, 1, 1);
    caster->SetPosition(0, 3, 0);
    caster->SetColor(0, 0, 1);

    cam->SetPosition(0, 2, -5);

    auto rt = CreateRayTracer();
    auto rts = CreateRTScene(rt);
    auto base = CreateRTScene(rt);
    auto rtm = base->AddMesh(caster->GetMesh(0));
    auto rti = CreateRTInstance(rts, base);

    RTRay ray;
    ray.dir = Vec3(0, 1, 0);
    ray.length = 20;

    while (not win->KeyHit(KEY_ESCAPE) and not win->Closed())
    {
        caster->Turn(0, 1, 0);
        rti->SetMatrix(caster->matrix);
        unsigned int color;

        for (int x = 0; x < sz; ++x)
        {
            for (int y = 0; y < sz; ++y)
            {
                ray.origin.x = (float(x) / float(sz) + 0.5f / float(sz)) * 10.0f - 5.0f;
                ray.origin.y = 0.5f;
                ray.origin.z = -((float(y) / float(sz) + 0.5f / float(sz)) * 10.0f - 5.0f);

                auto rayinfo = rts->TraceRay(ray);
                if (rayinfo.meshid != RT_INVALID_ID)
                {
                    color = Rgba(0, 0, 0, 255);
                }
                else
                {
                    color = Rgba(255, 255, 255, 255);
                }
                pixmap->WritePixel(x, y, color);
            }
        }
        tex->SetPixels(pixmap);

        world->Update();
        world->Render(fb);
    }
    return 0;
}

There's a lot that could be done to improve this basic implementation, but the idea works nicely.

Substituting the spinning box a more complex model, we can see the routine does not seem to be bothered much by mesh complexity. This model is over 100,000 polygons, and a 512x512 lightmap is updating in 30 milliseconds with one CPU core and unbatched raycasts.

Here it is in motion:

Here is an updated test using shadow maps for direct lighting and CPU ray tracing for ambient occlusion:

The original idea in Leadwerks 5 was to integrate Microsoft's Visual Studio Code as the IDE for the engine, rather than using the somewhat clunky built-in IDE we had in Leadwerks 4.

My eyes!

Visual Studio Code was the hot new text editor at the time, with cross-platform support and a dark theme that blended well with our new editor. A robust extensions system provided support for integrations with many languages including GLSL and Lua. There was even a Lua debugger with advanced features that seemed to turn Lua into a first-class programming language.

However, outsourcing a significant part of our user experience to a third party application is not without disadvantages. One day I was looking at some old Leadwerks 4 tutorials on Lua basics, and the simplicity of the debug interface really jumped out at me. This got me thinking about the whole user experience for new Lua developers.

Installation

The setup process for Visual Studio Code with Leadwerks 5 is not as streamlined as I would like.

1. Find and install Visual Studio Code (not to be confused with Microsoft Visual Studio).
2. Install the DevCat Lua Debugger extension from the Visual Studio Code marketplace (not any of the other hundreds of search results that appear when you search for this, and definitely not the actboy168 Lua debugger that will make your game crash).
3. Navigate four levels deep in the program settings to find and enable the "Allow Breakpoints Everywhere" option. If you don't do this, or if the setting gets disabled somehow, the debugger won't work.

The setup process is not too difficult, but I don't want a new user's first impression to be "Welcome to Leadwerks 5, the first thing you need to do is go install this other program and some weird things you've never heard of".

Welcome to...Leadwerks?

The alternative to this is to just provide a build-in IDE (Integrated Development Environment) as part of the level editor. This completely eliminates the need to install anything extra. Visual Studio Code can still be used, and you can choose how Lua files should be opened when you double-click them in the project browser. Three options are provided in the program settings:

• Open in IDE
• Open in Visual Studio Code (also opens the current folder as a project)
• System Default

It was not too difficult to get working, and after one week and 1500 lines of code I have a nearly-completed IDE working. The text editor is based on @klepto2's excellent Scintilla widget, which he has generously made available for free on Github under the MIT license.

Error Handling

When a coding error is encountered, the Visual Studio console dispays a rather verbose message.

The built-in IDE displays a clearly highlighted error message that tells you what the problem is. I even coded it to convert the first letter in the error message to uppercase, just to make it look a little nicer.

Both interfaces contain the same information, but with the built-in IDE you spend a lot less time scanning to find what you're looking for...which is a really good thing for an interface that gets heavily used!

Debugging

Visual Studio Code has some advanced features like hovering over a word in the text editor to display the value. However, the whole interface is flooded with Visual clutter. Type names include their namespace and things like shared_ptr, which is an underlying detail that is irrelavant to Lua.

The built-in IDE provides a more clear and concise view, with simplified type names and color-coded icons. It even displays an icon for each type of entity.

Window Management

Since Visual Studio Code is an external program, it does not stay on top of the editor window. To select the main editor window, VSCode, or game window, you need to click on their icons in the Windows taskbar or use Alt+Tab to cycle through windows.

When the built-in IDE is used, by default, the game window appears on top of the editor when the game runs.

With some clever Win32 programming, we can retrieve the game window from the running game's process handle. By responding to WINDOWSELECT and WINDOWDESELECT events, we can adjust the z-order of the game and editor windows to make the game and IDE act like they are both siblings parented to the main window. Whichever window is selected will appear on top, but both windows always stay on top of the main editor window.

If the IDE window is maximized, we can detect this event and then change the game window parent to the IDE window itself. This allows you to use the debugger in fullscreen, while the game remains on top.

With this design you will spend less time hunting for icons in the Windows taskbar to switch windows back and forth, and more time working on your game.

Finally, we are starting to see bloat creep into the Visual Studio Code user experience and it's no longer the light and fast text editor it began as.

Overall, the built-in IDE provides a smoother workflow with a much more focused presentation that highlights the information you are looking for with a lot less visual clutter. While the Visual Studio Code debugger will remain for those who wish to continue using it, I can already see that the built-in IDE will provide a massive boost in productivity to all our Lua developers, and a much improved user experience for new users.

My idea was to use my VR headset as a mocap system to record myself walking and get the headset orientation for a few seconds, then play that back in a loop when the first-person player moves around. I started with a simple scene. The strip on the ground gave me a clear indication of direction so I knew which way to walk.

The walkway is oriented along the X axis because the room I am using is wider than it is long!

Next I made a simple program that would just record the headset orientation for a few frames. I used the trigger button to control when recording occurs. The recorded data gets saved to mocap.json when the program ends. It's very easy to accidentally overwrite this file, so once you get a good recording, it's a good idea to just copy and paste the file in case you overwrite it accidentally.

Once I had the mocap data, I modified the FPSPlayer so it would get loaded and displayed.

I added two global variables at the top of the file:

table mocapdata;
int mocapindex = 0;

Then I added some code to load the mocap data in the Start method:

void FPSPlayer::Start()
{
	mocapdata = LoadTable("mocap.json");

Next I added this function. It's a function, not a class method, but I was just trying to get things working as fast as possible so I could see the result:

void UpdateBobble(shared_ptr<Entity> camera)
{
	int count = mocapdata.size();
	mocapindex = mocapindex % count;
	Vec3 offset;
	offset.x = mocapdata[mocapindex]["position"][0];
	offset.y = mocapdata[mocapindex]["position"][1];
	offset.y -= 1.65f;
	offset.z = 0;// mocapdata[mocapindex]["position"][2];
	camera->Move(offset);

	float pitch = camera->rotation.x;
	camera->SetRotation(0, camera->rotation.y, 0);

	offset.x = 0;// mocapdata[mocapindex]["rotation"][2];
	offset.y = float(mocapdata[mocapindex]["rotation"][1]) - 90.0f;
	offset.z = 0;// mocapdata[mocapindex]["rotation"][0];
	camera->Turn(offset);

	camera->Turn(pitch, 0, 0);


	++mocapindex;
}

Next I created another scene with an FPSPlayer and a long textured hallway to walk down.

I was worried that my head movements might be too subtle to show up in first-person player view, but it turns out I had nothing to worry about. Just some simple code produced the approximate effect I was hoping to see.

The technique basically works, but could be improved with more recordings in a bigger space, blending the start and the end of the loop to prevent the jump, and smoothing the camera movement a bit. Of course, you would want to footstep sounds to be timed to the footsteps in the mocap data. Still, the initial results are pretty good and I am encouraged to experiment with this more.

Here are the files I used. In main.cpp I added a definition for RECORDMODE, which controls whether the application is being run to record movement or play it back.

FPSPlayer.cpp main.cpp Maps.zip mocap.json

I found that a simple linear to sRGB conversion made the images look as I would expect.

I have read about sRGB and linear color spaces before, but for some reason this time it just clicked and I understood. Maybe since I am not longer dealing with low-level Vulkan nonsense, I have extra mental capacity to spend on more high-level concepts like this. In any case, I find this stuff is usually not explained very well and I want to show you how it works, in simple terms everyone can understand.

The basic idea of sRGB color space is that the way we perceive lightness in an image is wrong. Look at the gradient below. The change in the left-most 25% of the image looks much greater than the change in the right-most 25%, but it's not:

A long time ago, photography folks all agreed that these types of gradients look wrong. You will often see this referred to as crushed blacks, the opposite of overexposed bright colors, and it's usually considered to be a bad thing:

In 1996 HP and Microsoft came up with an equation that makes a gradient that looks "right". This is called sRGB color space, and it just shifts the darker tones to the right with a simple equation. The resulting gradient below looks more "balanced", even though it's really unbalanced:

The only problem is that since these colors are stretched, all our nice neat color math is no longer valid. Multiplying two colors together like we do for lighting won't produce the right result because the colors have all been shifted to the right by varying amounts. 

The solution is to unstretch all colors by converting from sRGB to linear, perform your color math like lighting, and then shift the result back from linear to sRGB. Fortunately you don't have to modify your textures and colors, because sRGB is considered the "correct" value that we want to see, and linear color space is a temporary intermediary used for math. We can convert between these color spaces in a shader with the code below.

const float GAMMA = 2.2;
const float INV_GAMMA = 1.0 / GAMMA;

vec3 linearTosRGB(vec3 color)
{
    return pow(color, vec3(INV_GAMMA));
}

vec3 sRGBToLinear(vec3 srgbIn)
{
    return vec3(pow(srgbIn.xyz, vec3(GAMMA)));
}

Let's take a look at some math to see how light and color interact with this approach:

float albedo = 0.5;
float light = 0.75;
    
float color = albedo * light;
    
Print("Simple lighting: " + String(color));

albedo = sRGBToLinear(albedo);
light = sRGBToLinear(light);

color = LinearTosRGB(color);

Print("sRGB lighting: " + String(color));

The output of this program shows that converting from sRGB to linear and back results in lighting that doesn't get smashed by the albedo color:

Simple lighting: 0.375
sRGB lighting: 0.640292

What this technique basically does is it reallocates precision away from the bright colors, which we do not have as accurate perception of, and gives it to the dark colors, which we can see between more easily. With standard linear color a simple scene looks both dull and heavily contrasted:

The same scene comes to life when colors are converted from sRGB to linear before lighting, and then transformed back for the final color. We can clearly see a blue tint from the PBR environment map mixed with the orange color, and the dark gray colors have much more definition now. I don't even understand how something can be blue and orange at the same time, but I can clearly see a blue reflection from the sky in the image below.

This operation doesn't change the colors, it just changes the way that colors combine. Overall this makes it easier for lights to illuminate dark areas without requiring extremely bright lights with strong contrast, and the PBR lighting is more strongly visible in colored objects. This update is available now on the beta branch, in the standalone and Steam versions of Ultra Engine.

 I used the Lua code below to create the gradient images. You can paste this into a Lua file and run it by selecting the Script > Run Script item in the main menu, or you can just paste this into the console and press enter, and it will run:

local pixmap = CreatePixmap(256, 16)

--Write linear gradient
for x = 0, pixmap.size.x - 1 do
    for y = 0, pixmap.size.y - 1 do
        pixmap:WritePixel(x, y, Rgba(x, x, x, 255))
    end
end
pixmap:Save(GetPath(PATH_DESKTOP).."/lineargradient.png")

--Convert to sRGB
pixmap = pixmap:LinearTosRgb()
pixmap:Save(GetPath(PATH_DESKTOP).."/srgbgradient.png")

I hope this explanation helps you to understand what sRGB and linear color spaces actually do and how this feature makes your games look better. It's actually a very straightforward concept that for some reason is poorly explained in most articles I have seen. Just remember, linear is the "squished" color for math, and sRGB is the "stretched" color that's easier to see (sRGB = standard stretched RGB).

Research & Development

The last four years of research and development were an opportunity to rethink and redesign what a game engine should be. Along the way there were a few ideas I had that turned out to be not that great of an idea in practice:

Variance shadow maps: Although I love the look of these under very controlled settings, the technique has extremely bad artifacts that can never be fixed. Depth shadow maps really are the best approach to texture-based shadows. They are also extremely slow to render shadow updates. I would like to see some better hardware filtering options in the future, though.

Voxel-based indirect lighting: Compelling in theory but extremely messy in practice. I learned a lot and this prompted me to integrate compute shaders into the engine, but ultimately the results I got weren't compelling enough to turn into a finished product. Given the problems everyone else has with this approach, I think it makes sense to focus on hardware raytracing as the next step up in lighting.

Novel entity component system: My idea was to make it so when you called a component method, all the components that had that method would get called. I had some complicated Lua code set up that handled this, and early builds of Ultra used a C++ preprocessor that analyzed header files and created some complicated wrappers to make C++ work in roughly the same way. Ultimately I decided the confusion wasn't worth the questionable benefit and I implemented a conventional multi-component system like everyone had asked for, without any preprocessor.

Tabbed Asset Editor: Another thing that sounded good in theory but was not great in practice was to use a single asset editor window with tabs for the different opened files. All it took was a day of using this to realize how much better it was to open each item in its own window. Hunting and clicking file names in tabs is just not fun. I get claustrophobic just looking at this:

The Results

Besides the performance, here are the things that I think turned out better than I imagined:

glTF Integration: Khronos glTF has proven to be a wonderful workflow, and eliminates any "black box" file formats for 3D content. Your game-ready files can always be pulled back into your modeling program, edited, and saved, with no need for secretive conversion pipelines. A wide variety of game-ready content is available for you to use with Ultra, so I am expecting to see no more programmer art!

Physically-based Rendering: This aligns closely with glTF integration because the file format includes a well-defined PBR materials system. The secret to good PBR is to have good imagery for reflections. To achieve this, I put a lot of work into the volumetric environment probes system that first appeared in Leadwerks. Probe volumes can now be drawn like a brush object for easier creation and precise sizing in the editor. A configurable fade distance for each edge lets you control how probes blend together. Probes also incorporate the sky lighting so you can have seamless transitions between bright outdoor areas and dark enclosed spaces.

Lua Integration: The VSCode Lua debugger turns Lua into a first-class game programming language and is a dream to work with. The editor scripting integration is in my opinion the best scripting integration of any 3D program I've ever seen. You can access the entire engine API from within editor scripts to add new functionality or modify the program. Additional documentation for the editor's innards will arrive in the coming days.

Ultra GUI: This took a lot of time but having control over every pixel and event made it worthwhile. Taking control of the user experience by writing my own GUI was one of the best decisions I made. The interface doubles up as the editor's own UI drawn using GDI+ and the in-game UI rendered with Vulkan.

C++ API: The most significant change is the user of shared pointers, which forever eliminate the problems with uninitialized and invalid pointers games were prone to as they grew more complex. Additionally, the API is generally better thought out and consistent than what we had in Leadwerks.

Terrain: You wanted more terrain material layers, so you got them. Up to 256, in fact, with fast performance. You can create multiple terrains, position, rotate, and scale them, and even create non-square terrains to fill in any area.

Player Physics: Yes, you can crouch now.

Pathfinding: It's dynamic and you can create multiple navmeshes, for different areas or differently sized characters. Pathfinding and physics are now decoupled so you can have armies of characters that only use the pathfinding system for movement.

3D Brush Editing: I was not planning on this feature until the last minute, but you can can create, move, scale, rotate, and shear brushes in any viewport, including the 3D view. the long-asked-for vertex tool is included as well as a face tool for adjusting texture mapping and geometry.

Last but not least, the engine's multithreaded design is crazy advanced! Your game code runs on its own thread, giving you a full 16 milliseconds to operate without slowing down the renderer. Physics, pathfinding, animation, rendering, and culling all operate on separate threads. It's an amazing feat in engineering that was only possible because I had seen all the bottlenecks people could create in Leadwerks, so I could plan a way around them when designing Ultra.

Early Access

The 1.0 release will be marked "Early Access" because there are still a few extra features to add, and we will have a period where changes to the workflow can still be considered based on user feedback. Once decals, particle emitters, VR support, flowgraphs, and brush smooth groups are added, that will be considered version 1.1 (no "early access"). So there will still be a few months of time where I can revise any parts of the workflow you think can be improved. There are also still some compatibility issues with AMD cards that are being resolved, so please be aware of that.

A big thanks goes out to all the testers who helped me smooth things out before the release. The good times are finally here!

I haven't been blogging much because I am making so much progress that there would be too much to cover. In this article I will talk about one little system I spent the last week on and show the improvements I have made to the workflow over Leadwerks.

In the asset editor, when the root of a model is selected, a "Collider" field appears in the physics properties. You can use this to quickly calculate and apply a collider to the model.

The convex hull collider includes an option for tolerance. You can adjust this to get a more or less precise convex hull around the model:

There is also an "optimize" option for the mesh collider. When this is activated it will attempt to merge adjacent coplanar polygons into a single face, and can result in fewer seams on colliding faces.

Two new shapes are included, the capsule and the chamfer cylinder, so now all the shapes supported by Newton Dynamics are available. (The chamfer cylinder is good for vehicle tires.)

Unlike Leadwerks, you no longer need to specify the axis cylindrical shapes are oriented around. The editor will calculate which bounding box axes are most like each other, and use the third axis to orient the shape. This works for long skinny objects:

And it also works for short fat objects:

It will even detect which direction a cone should be facing, by calculating the total distance from the shape the model vertices are with each direction, and selecting the orientation with the smallest error value. (A single cone is probably not the best choice of shape for this model, but the point is that it correctly guesses which direction the cone should point based on the model vertices.)

For more advanced geometry, the convex decomposition tool is available in the Utilities panel. This exposes all the parameters available in V-HACD 4.0 and processes the task on a separate thread so you can continue to use the editor as the task runs in the background. It usually only takes a few seconds to compete, but it is nice to not have your workflow interrupted:

You can export colliders as model files, so it's also possible to just use this to generate convex hulls and export them to .obj. (I could get a better fitting shape if I spent more time adjusting the settings, but I just quickly made this shape with very low settings in order to get this screenshot.)

The new .collider file format, like everything else in Ultra, is JSON-based. Simple shapes can be stored in pure ASCII text, so they can be edited by hand if needed:

{
	"collider": {
		"parts": [
			{
				"offset": [
					0.0024815797805786133,
					-0.18715500831604004,
					-1.055002212524414e-05
				],
				"rotation": [
					0.0,
					0.0,
					0.0
				],
				"shape": "BOX",
				"size": [
					1.8899539709091187,
					2.0,
					1.8019688129425049
				]
			}
		]
	}
}

More complex shapes like convex hulls and meshes use binary data appended to the end of the file to store their contents, so that they load quickly in the engine:

{
	"collider": {
		"parts": [
			{
				"data": 0,
				"shape": "CONVEXHULL",
				"vertices": 61
			}
		]
	}
}�n�2��<�5�j�k��#�I3��ch�������=b[��I<��~d�I�D�` �[�\{4���A���?XO3���>��վ�v!���f���	?]����I�!�����V� >�?���mdG�t9�a��@���X?W��0o��e�-�D��h�P?>����>��׾���@��=T�U?��ž�{����w�� ���G��3��ޑ?>V?c�����;���6�s�/?&�=����?q�e?j��N���[#b��bݽ(��>��Ǿ�ڽ�E��MV�ַؽ��8<�G������D?BYT?I���@"N?6?�";-�$?��콜�";��t>8�"�t�";�z��3�0���";����ݐ�)�%=}P?��0?���=/�E?BYT?'>�=�E��MV����={;
?"����>�־��c�
�>��?q�e?�$>�ӽ��P��@>��,?�j|=�2�>[o/���P�C��>��?>V?Z�>�������Қ�>��h���@�>��E�CO2��.�>T��[A�y]�>L��=T�U?w4�>���>�쾛?#o��e�-�r?�����U?�"?.�v>��!?�R?�zv����>�@?(�0�S
�>.(?�n_�C�?�|-?d�[��}?��4?��>��վ��6?2M4��*?�>?P��=_�s��F?V ���ྲAm?�#�I3�Lr?�����

Leadwerks .phy collider files can also be loaded.

Notice in the shots above, in the drop-down menu there is also a "Browse" option, so you can create a collider file with a low-resolution model and then load the resulting collider to use with a high-resolution model.

I think this design strikes the right balance to let you do simple things quickly, but still allow for more advanced options when you need them. In the future I would like to add an additional in-depth collider editing utility that would allow you to select each sub-object, select model polygons and generate shapes just from those polys, and other advanced features.

The pick mode feature has been revised. The original design was based on Blitz3D, which did not support collision primitives at all. Instead of specifying a sphere or other shape for picking, you can now just tell the entity to use the collider for picking. (PICK_SPHERE is removed.)

On an unrelated note, I had to adjust the camera range based on the distance from the model being viewed, in order to make very small objects visible. This allows extreme close-up shots like in the blog header image that you probably aren't used to seeing except in VR. It's kind of interesting.

Let's take a deep dive into how this works...

Extension Script

The extension is run by placing a file called "OpenAI.lua" in the "Scripts/Start/Extensions" folder. Everything in the Start folder gets run automatically when the editor starts up, in no particular order. At the top of the script we create a Lua table and load a DLL module that contains a few functions we need:

local extension = {}
extension.openai = require "openai"

Next we declare a function that is used to process events. We can skip its inner workings for now:

function extension.hook(event, extension)

end

We need a way for the user to activate our extension, so we will add a menu item to the "Scripting" submenu. The ListenEvent call will cause our hook function to get called whenever the user selects the menu item for this extension.  Note that we are passing the extension table itself in the event listener's extra parameter. There is no need for us to use a global variable for the extension table, and it's better that we don't.

local menu = program.menu:FindChild("Scripting", false)
if menu ~= nil then
    local submenu = menu:FindChild("OpenAI", false)
    if submenu == nil then
        submenu = CreateMenu("", menu)-- divider
        submenu = CreateMenu("OpenAI", menu)
    end
    extension.menuitem = CreateMenu("Text to Image", submenu)
end

ListenEvent(EVENT_WIDGETACTION, extension.menuitem, extension.hook, extension)

This gives us a menu item we can use to bring up our extension's window.

The next section creates a window, creates a user interface, and adds some widgets to it. I won't paste the whole thing here, but you can look at the script to see the rest:

extension.window = CreateWindow("Text to Image", 0, 0, winx, winy, program.window, WINDOW_HIDDEN | WINDOW_CENTER | WINDOW_TITLEBAR)

Note the window is using the WINDOW_HIDDEN style flag so it is not visible when the program starts. We're also going to add event listeners to detect when the window is closed, and when a button is pressed:

ListenEvent(EVENT_WINDOWCLOSE, extension.window, extension.hook, extension)
ListenEvent(EVENT_WIDGETACTION, extension.button, extension.hook, extension)

The resulting tool window will look something like this:

Now let's take a look at that hook function. We made three calls to ListenEvent, so that means we have three things the function needs to evaluate. Selecting the menu item for this extension will cause our hidden window to become visible and be activated:

    elseif event.id == EVENT_WIDGETACTION then
        
        if event.source == extension.menuitem then

            extension.window:SetHidden(false)
            extension.window:Activate()

When the user closes the close button on the tool window, the window gets hidden and the main program window is activated:

    if event.id == EVENT_WINDOWCLOSE then
        if event.source == extension.window then
            extension.window:SetHidden(true)
            program.window:Activate()
        end

Finally, we get to the real point of this extension, and write the code that should be executed when the Generate button is pressed. First we get the API key from the text field, passing it to the Lua module DLL by calling openal.setapikey.

        elseif event.source == extension.button then

            local apikey = extension.apikeyfield:GetText()
            if apikey == "" then
                Notify("API key is missing", "Error", true)
                return false
            end

            extension.openai.setapikey(apikey)

Next we get the user's description of the image they want, and figure out what size it should be generated at. Smaller images generate faster and cost a little bit less, if you are using a paid OpenAL plan, so they can be good for testing ideas. The maximum size for images is currently 1021x1024.

            local prompt = extension.promptfield:GetText()

            local sz = 512
            local i = extension.sizefield:GetSelectedItem()
            if i == 1 then
                sz = 256
            elseif i == 3 then
                sz = 1024
            end

The next step is to copy the user's settings into the program settings so they will get saved when the program closes. Since the main program is using a C++ table for the settings, both Lua and the main program can easily share the same information:

            --Save settings
            if type(program.settings.extensions) ~= "userdata" then
                program.settings.extensions = {}
            end
            if type(program.settings.extensions.openai) ~= "userdata" then
                program.settings.extensions.openai = {}
            end
            program.settings.extensions.openai.apikey = apikey
            program.settings.extensions.openai.prompt = prompt
            program.settings.extensions.openai.size = {}
            program.settings.extensions.openai.size[1] = sz
            program.settings.extensions.openai.size[2] = sz   

Extensions should save their settings in a sub-table in the "extensions" table, so keep data separate from the main program and other extensions. When these settings are saved in the settings.json file, they will look like this. Although generated images must be square, I opted to save both width and height in the settings, for possible future compatibility.

	"extensions":
	{
		"openai":
		{
			"apikey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
			"prompt": "tiling seamless texture warehouse painted concrete wall abandoned dirty",
			"size":
			[
				1024,
				1024
			]
		}
	},

Finally, we call the module function to generate the image, which may take a couple of minutes. If its successful we load the resulting image as a pixmap, create a texture from it, and then open that texture in a new asset editor window. This is done to eliminate the asset path, so the asset editor doesn't know where the file was loaded from. We also make a call to AssetEditor:Modify, which will cause the window to display a prompt if it is closed without saving. This prevents the user's project folder from filling up with a lot of garbage images they don't want to keep.

            if extension.openai.newimage(sz, sz, prompt, savepath) then
                local pixmap = LoadPixmap(savepath)
                if pixmap ~= nil then
                    local tex = CreateTexture(TEXTURE_2D, pixmap.size.x, pixmap.size.y, pixmap.format, {pixmap})
                    tex.name = "New texture"
                    local asseteditor = program:OpenAsset(tex)
                    if asseteditor ~= nil then
                        asseteditor:Modify()
                    else
                        Print("Error: Failed to open texture in asset editor.");
                    end
                end
            else
                Print("Error: Image generation failed.")
            end

The resulting extension provides an interface we can use to generate a variety of interesting textures. I think you will agree, these are quite a lot better than what we had just a few months ago.

Of course the Lua debugger in Visual Studio Code came in very handy while developing this.

That's pretty much all there is to the Lua side of things. Now let's take a closer look at the module code.

The Module

Lua modules provide a mechanism whereby Lua can execute C++ code packed into a dynamic linked library. The DLL needs to contain one retired function, which is luaopen_ plus the name of the module, without any extension. The module file is "openai.dll" so we will declare a function called luaopen_openai:

extern "C"
{
    __declspec(dllexport) int luaopen_openai(lua_State* L)
    {
        lua_newtable(L);
        int sz = lua_gettop(L);
        lua_pushcfunction(L, openai_newimage);   lua_setfield(L, -2, "newimage");
        lua_pushcfunction(L, openai_setapikey);   lua_setfield(L, -2, "setapikey");
        lua_pushcfunction(L, openai_getlog);   lua_setfield(L, -2, "getlog");
        lua_settop(L, sz);
        return 1;
    }
}

This function creates a new table and adds some function pointers to it, and returns the table. (This is the table we will store in extension.openai). The functions are setapikey(), getlog() and newimage().

The first function is very simple, and just provides a way for the script to send the user's API key to the module:

int openai_setapikey(lua_State* L)
{
    APIKEY.clear();
    if (lua_isstring(L, -1)) APIKEY = lua_tostring(L, -1);
    return 0;
}

The getlog function just returns any printed text, for extra debugging:

int openai_getlog(lua_State* L)
{
    lua_pushstring(L, logtext.c_str());
    logtext.clear();
    return 1;
}

The newimage function is where the action is at, but there's actually two overloads of it. The first one is the "real" function, and the second one is a wrapper that extracts the right function arguments from Lua, and then calls the real function. I'd say the hardest part of all this is interfacing with the Lua stack, but if you just go carefully you can follow the right pattern.

bool openai_newimage(const int width, const int height, const std::string& prompt, const std::string& path)
int openai_newimage(lua_State* L)

This is done so the module can be easily compiled and tested as an executable.

The real newimage function is where all the action is. It sets up a curl instance and communicates with a web server. There's quite a lot of error checking in the response, so don't let that confused you. If the call is successful, then a second curl object is created in order to download the resulting image. This must be done before the curl connection is closed, as the server will not allow access after that happens:

bool openai_newimage(const int width, const int height, const std::string& prompt, const std::string& path)
{
    bool success = false;
    if (width != height or (width != 256 and width != 512 and width != 1024))
    {
        Print("Error: Image dimensions must be 256x256, 512x512, or 1024x1024.");
        return false;
    }

    std::string imageurl;

    if (APIKEY.empty()) return 0;

    std::string url = "https://api.openai.com/v1/images/generations";
    std::string readBuffer;
    std::string bearerTokenHeader = "Authorization: Bearer " + APIKEY;
    std::string contentType = "Content-Type: application/json";

    auto curl = curl_easy_init();

    struct curl_slist* headers = NULL;
    headers = curl_slist_append(headers, bearerTokenHeader.c_str());
    headers = curl_slist_append(headers, contentType.c_str());

    nlohmann::json j3;
    j3["prompt"] = prompt;
    j3["n"] = 1;

    switch (width)
    {
    case 256:
        j3["size"] = "256x256";
        break;
    case 512:
        j3["size"] = "512x512";
        break;
    case 1024:
        j3["size"] = "1024x1024";
        break;
    }
    std::string postfields = j3.dump();

    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
    curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postfields.c_str());
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);

    auto errcode = curl_easy_perform(curl);
    if (errcode == CURLE_OK)
    {
        //OutputDebugStringA(readBuffer.c_str());
        trim(readBuffer);
        if (readBuffer.size() > 1 and readBuffer[0] == '{' and readBuffer[readBuffer.size() - 1] == '}')
        {
            j3 = nlohmann::json::parse(readBuffer);
            if (j3.is_object())
            {
                if (j3["error"].is_object())
                {
                    if (j3["error"]["message"].is_string())
                    {
                        std::string msg = j3["error"]["message"];
                        msg = "Error: " + msg;
                        Print(msg.c_str());
                    }
                    else
                    {
                        Print("Error: Unknown error.");
                    }
                }
                else
                {
                    if (j3["data"].is_array() or j3["data"].size() == 0)
                    {
                        if (j3["data"][0]["url"].is_string())
                        {
                            std::string s = j3["data"][0]["url"];
                            imageurl = s; // I don't know why the extra string is needed here...

                            readBuffer.clear();

                            // Download the image file
                            auto curl2 = curl_easy_init();
                            curl_easy_setopt(curl2, CURLOPT_URL, imageurl.c_str());
                            curl_easy_setopt(curl2, CURLOPT_WRITEFUNCTION, WriteCallback);
                            curl_easy_setopt(curl2, CURLOPT_WRITEDATA, &readBuffer);

                            auto errcode = curl_easy_perform(curl2);
                            if (errcode == CURLE_OK)
                            {
                                FILE* file = fopen(path.c_str(), "wb");
                                if (file == NULL)
                                {
                                    Print("Error: Failed to write file.");
                                }
                                else
                                {
                                    auto w = fwrite(readBuffer.c_str(), 1, readBuffer.size(), file);
                                    if (w == readBuffer.size())
                                    {
                                        success = true;
                                    }
                                    else
                                    {
                                        Print("Error: Failed to write file data.");
                                    }
                                    fclose(file);
                                }
                            }
                            else
                            {
                                Print("Error: Failed to download image.");
                            }

                            curl_easy_cleanup(curl2);
                        }
                        else
                        {
                            Print("Error: Image URL missing.");
                        }
                    }
                    else
                    {
                        Print("Error: Data is not an array, or data is empty.");
                    }
                }
            }
            else
            {
                Print("Error: Response is not a valid JSON object.");
                Print(readBuffer);
            }
        }
        else
        {
            Print("Error: Response is not a valid JSON object.");
            Print(readBuffer);
        }
    }
    else
    {
        Print("Error: Request failed.");
    }

    curl_easy_cleanup(curl);

    return success;
}

My first attempt at this used a third-party C++ library for OpenAI, but I actually found it was easier to just make the low-level CURL calls myself.

Now here's the kicker: Any web API in the world will work with almost the exact same code. You now know how to build extensions that communicate with any website with a web interface, including SketchFab, CGTrader, itch.io, and easily interact with them in the Ultra Engine editor. The full source code for this and other Ultra Engine modules is available here:

https://github.com/UltraEngine/Lua

This integration of text-to-image tends to do well with base textures that have a uniform distribution of detail. Rock, concrete, ground, plaster, and other materials look great with the DALL-E 2 generator. It doesn't do as well with geometry details and complex structures, since the AI has no real understanding of the purpose of things. The point of this integration was not to make the end-all-be-all AI texture generation. The technology is changing rapidly and will undoubtedly continue to advance. Rather, the point of this exercise was to demonstrate a complex feature added in a Lua extension, without being built into the editor's code. By releasing the source for this I hope to help people start developing their own extensions to add new peripheral features they want to see in the editor and game engine.

ULTRA_triangle_quads

All glTF models are triangle meshes, but we want to support quad meshes primarily because its better for tessellation. This extension gets added to the primitives block. If the "quads" value is set to true, this indicates that the triangle indices are stored in a manner such that the first four indices of every six indices form a quad:

"extensions": {
	"ULTRA_triangle_quads": {
		"quads": true
	}
}

There is no other glTF extension for quads, and so there is no way to export a glTF quad mesh from any modeling program. To get quad meshes into Ultra Engine you can load an OBJ file and then resave it as glTF. Here is a glTF file using quads that was created this way. You can see the tessellation creates an even distribution of polygons:

For comparison, here is the same mesh saved as triangles and tessellated. The long thin triangles result in a very uneven distribution of polygons. Not good!

The mesh still stores triangle data so the file can be loaded back into a 3D modeling program without any issues.

Here is another comparison that shows how triangle (on the left) and quads (on the right) tessellate:

ULTRA_material_displacement

This extension adds displacement maps to glTF materials, in a manner that is consistent with how other textures are stored:

"ULTRA_material_displacement": {
	"displacementTexture": {
		"index": 3,
		"offset": -0.035,
		"strength": 0.05
	}
}

The extension indicates a texture index, a maximum displacement value in meters, and a uniform offset, also in meters. This can be used to store material displacement data for tessellation or parallax mapping. Here is a model loaded straight from a glTF file with displacement info and tessellation:

If the file is loaded in other programs, the displacement info will just be skipped.

ULTRA_vertex_displacement

Our game engine uses a per-vertex displacement factor to control how displacement maps affect geometry. This extension adds an extra attribute into the primitives structure to store these values:

"primitives": [
	{
		"attributes": {
			"NORMAL": 1,
			"POSITION": 0,
			"TEXCOORD_0": 2
		},
		"indices": 3,
		"material": 0,
		"mode": 4,
		"extensions": {
			"ULTRA_vertex_displacement": {
				"DISPLACEMENT": 7
			}
		}
	}
}

This can be used to prevent cracks from appearing  at texcoord seams.

Here you can see the displacement value being loaded back from a glTF file it has been saved into. I'm using the vertex color to visually verify that it's working right:

ULTRA_extended_material

This extension adds other custom parameters that Ultra Engine uses. glTF handles almost everything we want to do, and there are just a few settings to add. Since the Ultra Engine material format is JSON-based, it's easy to just insert the extra parameters into the glTF like so:

"ULTRA_extended_material": {
	"shaderFamily": "PBR",
	"shadow": true,
	"tessellation": true
}

In reality I do not feel that this extension is very well-defined and do not expect it to see any adoption outside of Ultra Engine. I made the displacement parameters a separate extension because they are well-defined, and there might be an opportunity to work with other application developers using that extension.

Here we can see the per-material shadow property is disabled when it is loaded from the glTF:

For comparison, here is what the default setting looks like:

These extensions are simply meant to add special information that Ultra Engine uses into the glTF format. I do not currently have plans to try to get other developers to adopt my standards. I just want to add the extra information that our game engine needs, while also ensuring compatibility with the rest of the glTF ecosystem. If you are writing a glTF importer or exporter and would like to work together to improve the compatibility of our applications, let me know!

I used the Rock Moss Set 02 model pack from Polyhaven in this article.

• Eliminate one-way 3D model conversions.
• Use the same game asset files on all platforms.

In Leadwerks, 3D models get converted into the proprietary MDL format. Although the format is pretty straightforward and simple, there is no widespread support for loading it back into 3D modeling programs. This means you need to keep a copy of your 3D model in FBX and MDL format. You may possibly want to keep an additional file for the modeling program it was made in, such as 3D Studio Max MAX files. I wanted to simplify this by relying on the widely-support glTF file format, which can be used for final game-ready models, but can still easily loaded back into Blender or another program.

Texture Compression

Texture compression is where this all gets a lot more tricky. Texture compression is an important technique that can reduce video memory usage to about 25% what it would be otherwise. When texture compression is used games run faster, load faster, and look better because they can use higher resolution images. There are two problems with texture compression.

• Supported texture compression methods vary wildly across PC and mobile platforms.
• Many 3D modeling programs do not support modern texture compression formats.

On the PC, BC7 is the best compression format to use for most images. BC5 is a two-channel format appropriate for normal maps, with Z reconstruction in the shader. BC6H can be used to compress HDR RGB images (mostly skyboxes). BC4 is a single-channel compressed format that is rarely used but could be useful for some special cases. The older DXTC formats should no longer be used, as they have worse artifacts with the same data size.

Here are a few programs that do not support the newer BC formats, even though they are the standard for modern games:

• Windows Explorer
• Blender
• Microsoft 3D Object Viewer

Now, the glTF model format does not actually support DDS format textures at all. The MSFT_texture_dds extension adds support for this by adding an extra image source into the file. glTF loaders that support this extension can load the DDS files, while programs that do not recognize this extension will ignore it and load the original PNG or JPEG files:

"textures": [
    {
        "source": 0,
        "extensions": {
            "MSFT_texture_dds": {
                "source": 1
            }
        }
    }
],
"images": [
    {
        "uri": "defaultTexture.png"
    },
    {
        "uri": "DDSTexture.dds"
    }
]

I don't know any software that supports this extension except Ultra Engine. Even Microsoft's own 3D Object Viewer app does not support DDS files.

The Ultra Engine editor has a feature that allows you to convert a model's textures to DDS. This will resave all textures in DDS format, and change the model materials to point to the DDS files instead of the original format the model uses (probably PNG):

When a glTF file is saved after this step, the resulting glTF will keep copies of both the original PNG and the saved DDS files. The resulting glTF file can be loaded in Ultra Engine ready to use in your game, with DDS files applied, but it can still be loaded in programs that do not support DDS files, and the original PNG files will be used instead:

The same model can be loaded back into Blender in case any changes need to be made. If you resave it, the references to the DDS images will be lost, so just repeat the DDS conversion step in the Ultra Engine to finalize the updated version of your model.

To cut down on the size of your game files, you can just skip PNG files in the final packaging step so only DDS files are included.

Basis Universal

We still have to deal with the issue of hardware support for different codecs. This table from Unity's documentation shows the problem clearly:

We do want to support some mobile-based VR platforms, so this is something that needs to be considered now. Basis Universal is a library from the author of Crunch that solves this problem by introducing a platform-agnostic intermediate compressed format that can be quickly transcoded into various texture compression formats, without going through a slow decompression and recompression step. We can generate Basis files in the Ultra Engine editor just like we did for DDS: There is a glTF extension that supports basis textures, so glTF models can be saved that reference the Basis files.

The basis textures are stored in the glTF file the same way the DDS extension works:

"textures": [
    {
        "source": 0,
        "extensions": {
            "KHR_texture_basisu": {
                "source": 1
            }
        }
    }
],
"images": [
    {
        "mimeType": "image/png",
        "bufferView": 1
    },
    {
        "mimeType": "image/ktx2",
        "bufferView": 2
    }
]

The resulting glTF files can be loaded as game-ready models with compressed textures on PC or mobile platforms, and can still be loaded back into Blender or another 3D modeling program. We don't have to store different copies of our textures in different compression formats or go through a slow conversion step when publishing the game to other platforms. If your game is only intended to run on PC platforms, then DDS is the simplest path to take, but if you plan to support mobile-based VR devices in the future then Basis solves that problem nicely.

Here is our revised table:

I was also working with the KTX2 format for a while, but I ended up only using it for its support for Basis compression. DDS and Basis cover all your needs and are more widely supported.

Basis also supports a very lossy but efficient ETC1S method which is similar to Crunch, as well as uncompressed texture data, but in my opinion the one purpose for the Basis format is its UASTC format.

This article features the lantern sample glTF model from Microsoft.

• PAK
• WAD
• BSP (textures)
• SPR
• LMP

Why Quake? Well, the original Quake game is what got me into game development through modding, but Quake is also great for testing because it's so weird. This game was written before hardware accelerated graphics existed, so it has a lot of conventions that don't match modern 3D standards:

• All animation is vertex morphing (no skeletal animation)
• Texture coordinates are in pixels / integers instead of floating point values
• Three different types of "packages" (PAK, WAD, BSP), the latter two of which can be stored inside the first (packages-in-packages)
• Image files are stored seven different ways:
  • LMP image files
  • SPR sprite files
  • Color pallette ('@')
  • Mipmapped texture in WAD ('D')
  • Console picture ('E')
  • Status bar pictures ('B')
  • Mipmapped textures stored in BSP

Each of the image storage methods store data in slightly different ways. Sometimes the width and height are packed into the image data, sometimes they are found elsewhere in a separate structure, sometimes they are hard-coded, and sometimes you have to guess the image dimensions based on the data size. It's a mess, and it seems like this was made up on-the-fly by several people, but this was cutting-edge technology at the time and I don't think usability outside the specified game they were working on was a big concern.

There are some interesting challenges there, but this forms the basis of several dated but influential games. As time went by, the design became more and more sane. For example, Quake 3 just stores textures as TGA images in a ZIP file with the extension changed to PK3. So I figure if I can make the plugin system flexible enough to work with Quake then it will be able to handle anything.

Originally I tried implementing this using HLLib from Nem's Tools. This library supports Quake formats and more, but I had a lot of problems with it, and got better results when I just wrote my own PAK loading code. WAD files were an interesting challenge. There is no real Quake texture format, so loading the raw texture data from the package made no sense. The most interesting breakthrough in this process was how I handled WAD textures. I finally figured out I can make the WAD loader return a block of data that forms a DDS file when a texture is loaded, even through the texture name has no extension. This tricks the engine into thinking the WAD package contains a DDS file which can easily be loaded, when the reality is quite different. This introduces an important concept, that package plugins don't necessarily have to return the exact data they contain, but instead can return files in a ready-to-use format, and the game engine will never know the difference.

The resulting plugin allows you to easily extract and load textures from the Quake game files. This code will extract textures from a single WAD file.

#include "UltraEngine.h"

using namespace UltraEngine;

void main(int argc, const char* argv[])
{
    // Load Quake file loader plugin
    auto qplg = LoadPlugin("Plugins/QuakeLoader");

    // Load FreeImage texture plugin
    auto fiplg = LoadPlugin("Plugins/FITextureLoader");

    // WAD to download
    WString wadfile = "bigcastle";

    // Download a WAD package
    DownloadFile("https://www.quaketastic.com/files/texture_wads/" + wadfile + ".wad", wadfile + ".wad");

    // Load the package
    auto wad = LoadPackage(wadfile + ".wad");

    // Create a subdirectory to save images in
    CreateDir(wadfile);

    // Read all files in the package
    auto dir = wad->LoadDir("");
    for (auto file : dir)
    {
        // Load the image
        auto pm = LoadPixmap(file);

        // Save as a PNG file
        if (pm) pm->Save(wadfile + "/" + file + ".png");
    }

    OpenDir(wadfile);
}

All the files found in the WAD package are saved as PNG images. BSP files will work exactly the same way.

This code will load a Quake package, extract all the textures from all the maps in the game, and save them to a folder on the desktop. This is done by detecting packages-in-packages (WAD and BSP), which return the file type '3', indicating that they can be treated both as a file and as a folder. Since none of these package formats use any compression, pixmaps can be easily loaded straight out of the file without extracting the entire BSP. Since Quake PAK files don't use compression, the whole system just turns into a very complicated blob of data with pointers that store data all over the place:

#include "UltraEngine.h"

using namespace UltraEngine;

void main(int argc, const char* argv[])
{
    // Load Quake file loader plugin
    auto qplg = LoadPlugin("Plugins/QuakeLoader");

    // Load FreeImage texture plugin
    auto fiplg = LoadPlugin("Plugins/FITextureLoader");

    // Path to Quake game
    WString qpath = "C:/Program Files (x86)/Steam/steamapps/common/Quake";
    //WString qpath = "D:/SteamLibrary/steamapps/common/Quake";
  
    // Load game package
    auto pak = LoadPackage(qpath + "/id1/PAK1.PAK");

    //Change the current directory to load files with relative paths
    ChangeDir(qpath + "/id1");

    // Create a folder to save images in    
    WString savedir = GetPath(PATH_DESKTOP) + "/Quake";
    CreateDir(savedir);

    // Load the main package directory
    auto dir = pak->LoadDir("maps");
    for (auto file : dir)
    {
        // Print the file name
        Print(file);

        // Get the file type
        int type = FileType("maps/" + file);

        // If a package-in-a-package is found load its contents (BSP and WAD)
        if (type == 3)
        {
            auto subdir = pak->LoadDir("maps/" + file);
            for (int n = 0; n < subdir.size(); ++n)
            {
                // Load the texture (plugin will return DDS files)
                auto pm = LoadPixmap("maps/" + file + "/" + subdir[n]);

                // Save to PNG
                if (pm) pm->Save(savedir + "/" + subdir[n] + ".png");
            }
        }
    }
    OpenDir(savedir);
}

Here is the result:

Since all of these images can be loaded as pixmaps, does that mean they can also be loaded as a texture? I had to know the answer, so I tried this code:

#include "UltraEngine.h"

using namespace UltraEngine;

int main(int argc, const char* argv[])
{
    //Get the displays
    auto displays = GetDisplays();

    //Create a window
    auto window = CreateWindow("Ultra Engine", 0, 0, 1280, 720, displays[0], WINDOW_CENTER | WINDOW_TITLEBAR);

    //Create a world
    auto world = CreateWorld();

    //Create a framebuffer
    auto framebuffer = CreateFramebuffer(window);

    //Create a camera
    auto camera = CreateCamera(world);
    camera->SetClearColor(0.125);
    camera->SetPosition(0, 0, -2);
    camera->SetFov(70);

    //Create a light
    auto light = CreateBoxLight(world);
    light->SetRotation(45, 35, 0);
    light->SetRange(-10, 10);
    light->SetColor(2);

    //Create a model
    auto model = CreateBox(world);
    
    // Load Quake file loader plugin
    auto qplg = LoadPlugin("Plugins/QuakeLoader");

    // WAD to download
    WString wadfile = "bigcastle";

    // Download a WAD package
    DownloadFile("https://www.quaketastic.com/files/texture_wads/" + wadfile + ".wad", wadfile + ".wad");

    // Load the package
    auto wad = LoadPackage(wadfile + ".wad");

    // Load a texture from the WAD package
    auto tex = LoadTexture("sclock");

    // Create a material
    auto mtl = CreateMaterial();
    mtl->SetTexture(tex);

    // Apply the material to the box
    model->SetMaterial(mtl);

    //Main loop
    while (window->Closed() == false and window->KeyDown(KEY_ESCAPE) == false)
    {
        model->Turn(0, 1, 0);
        world->Update();
        world->Render(framebuffer);
    }
    return 0;
}

Here is the result, a texture loaded straight out of a Quake WAD file!

To reiterate, testing with Quake files helped me to come up with two important design features:

• Packages-in-packages, indicated with the file type '3'.
• Packages do not necessarily have to return the same exact data they contain, and can instead prepare the data in a ready-to-use format, when that is desirable. The decision to base file type detection on the contents of the file instead of the file name extension worked well here, and allowed me to load the extension-less WAD texture names as DDS files.

Some of these images came from "bigcastle.wad". I don't know who the original author is.

If you are interested, you can read more about the weird world of Quake file formats here, although I must warn you that specification is not complete! 

I have plans for this, but nothing I can say anything specific about right now, so now I am working on the editor again.

Leadwerks had a very well-defined workflow, which made it simple to use but also somewhat limited. Ultra goes in the opposite direction, with extremely flexible support for plugins and extensions. It's a challenge to keep things together because if you make things to flexible it just turns into indecisiveness, and nothing will ever get finished that way. I think I am finding a good balance or things that should be hard-coded and things that should be configurable.

Instead of separate windows for viewing models, textures, and materials, we just have one asset editor window, with tabs for separate items. The texture interface is the simplest and just shows some information about the asset on the left-side panel.

When you select the Save as menu item, the save file dialog is automatically populated with information loaded from all the plugins the engine has loaded. You can add support for new image formats at any time just by dropping a plugin in the right folder. In fact, the only hard-coded format in the list is DDS.

You can save to an image format, or to the more advanced DDS and KTX2 formats, which include mipmaps and other features. Even more impressive is the ability to load Leadwerks texture files (TEX) and save them directly into DDS format in their original pixel format, with no loss of image quality. Here we have a Leadwerks texture that uses DXT1 compression, converted to DDS without recompressing the pixels:

There are several tricks to make the new editor start up as fast as possible. One of these is that the various windows the editor uses don't actually get created until the first time they are used. This saves a little bit of time to give the application a snappier feel at startup.

I hope the end result will provide a very intuitive workflow like the Leadwerks editor has, with enough power and flexibility to make the editor into a living growing platform with many people adding new capabilities for you to use. I think this is actually my favorite stuff to work on because this is where all the technology is exposed to you, the end user, and I get to carefully hand-craft an interface that makes you feel happy when you use it.

And then the magic begins:

Here are some of the images I "created" in a few minutes using the tool:

I'm really surprised by the results. I didn't think it was possible for AI to demonstrate this level of spatial reasoning. You can clearly see that it has some kind of understanding of 3D perspective and lighting. Small errors like the misspelling of "Quake" as "Quke" only make it creepier, because it means the AI has a deep level of understanding and isn't just copying and pasting parts of images.

What do you think about AI-generated artwork? Do you have any of your own images you would like to show off? Let me know in the comments below.

It looks quite good without breaking the specular/gloss approach. I think this would be an ideal way to upgrade the Leadwerks renderer without making changes that are too big.

To implement this, I used the glTF sample viewer source code, which is a bit of a reference renderer for glTF materials. It has changed quite a lot since I first used it as a guide to implementing PBR materials. I decided I might as well integrated the latest code into our renderer. The shader code is very well organized, so it was pretty simple to integrate into my latest code. The glTF materials system is great because it provides a standardized materials system with a lot of advanced effects, along with a reference renderer that shows exactly how the materials are supposed to appear. This means that any engine or tool that is conforms to the glTF standard will always display the same materials the same way, while still allowing additional more specialized features like shadows and post-processing effects to be added.

In addition to revising our metal-roughness code, I went ahead and added support for some more features. The KHR_materials_clearcoat extension basically makes cars look like cars:

It adds an extra layer of reflections that gives a dark surface a big boost, and can even use a separate normal map for the clearcoat layer:

The KHR_materials_sheen extension makes cloth look like cloth. You can see for yourself in the image below. The appearance kind of makes my skin crawl because I can't stand the feeling of velvet / velour. Apparently this is some kind of common tactile hypersensitivity and I actually get a feeling like electricity shooting through my spine just looking at this image, so I think the effect is working well:

Another creepy example. Just the thought of running my hand over that fuzzy orange fabric is making me shiver:

The KHR_materials_transmission extensions provides a physically accurate model for transparent surfaces, with support for refraction. Here's a shot of refraction bending the background image:

This also supports roughness, for making the refracted background appear blurry like frosted glass:

I feel the background is too pixellated and should be improved. This problem occurs in the reference renderer, and I have several ideas on how to solve it.

The transparency system is very advanced, and introduces a lot of interesting new commands to the engine:

Material::SetIndexOfRfraction(const float ior)
Material::SetThickness(const float thickness)
Material::SetTransmission(const float transmission)

These features will give you reliable support for loading glTF models from any source, as well as the best image quality possible for your games. I think realistic materials in VR will also make a big difference in bridging the uncanny valley. And all of this is done with the fastest possible performance for VR.

And yes, of course I used a gamepad to do all of this.

Anyways, it's a lot of difficult work, and I decided to take a "break" and focus something else for a few days.

Before Leadwerks was on Steam, it had a web installer that would fetch a list of files from our server and any files that were missing, or were newer than the locally stored ones. There was no system for detecting updates, you just pressed the update button and the updater ran. The backend for this system was designed by @klepto2 and it functioned well for what it needed to do.

With Ultra App Kit, I created a simple tool to generate projects. This introduced the account authentication / signin system, which was sort of superfluous for this application, but it was a good way to test it out:

With Ultra Engine, I wanted some characteristics of both these applications. I wrote a new backend in about a day that handles updates. A PHP script authenticates the user and verifies product ownership, fetches a list of files, and retrieves files. Since C++ library files tend to be huge, I found it was necessary to add a compression system, so the script returns a zip compressed file. Of course, you don't want the server to be constantly creating zip files, so it caches the file and updates the zip only when I upload a new copy of the file. There's also a cache for the info retrieval, which is returned in JSON format, so it's easy to read in C++.

For the front end, I took inspiration from the Github settings page, which I thought looked nice:

And here's what I came up with. Projects will show when they are outdated and need to be updated (if a file in the template the project came from was changed). Each of the sections contains info and links to various topics. There's a lot there, but none of it feels extraneous to me. This is all made with the built-in GUI system. No HTML is used at all:

The Invision Power REST API is extremely interesting. It allows authentication of accounts and purchases, but it can be made to do a lot of other things.

• Post a forum topic: https://invisioncommunity.com/developers/rest-api?endpoint=forums/topics/POSTindex
• Upload an image to the gallery: https://invisioncommunity.com/developers/rest-api?endpoint=gallery/images/GETindex
• Download a file: https://invisioncommunity.com/developers/rest-api?endpoint=downloads/files/GETitem
• Unlock achievements: https://invisioncommunity.com/developers/rest-api?endpoint=core/members/POSTitem_achievements_awardbadge

None of that is very important right now, but it does provide some interesting ideas for future development of the game engine.

Getting geometry into 3ds Max is simple enough. I imported the model as an FBX file.

To set up the material, I opened the compact material editor and set the first slot to be a glTF material.

Press the button for the base color map, and very importantly choose the General > Bitmap map type. Do not choose OSL > Bitmap Lookup or your textures won't export at all.

Select your base color texture, then do the same thing with the normal and roughness maps, if you have them. 3ds Max treats metal / roughness as two separate textures, although you might be able to use the same texture both if it grabs the data from the green (roughness) and blue (metal) channels. This is something I don't know yet.

Select the File > Export menu item to bring up the glTF export dialog. Uncheck the "Export glTF binary" option because we don't want to pack our model and textures into a single file: I don't know what the baked / original material option does because I don't see any difference when I use it.

At this point you should have a glTF file that is visible in any glTF model viewer.

Now something slightly weird max does is it generates some new textures for some of the maps. This is probably because it is combining different channels to produce final images. In this case, none of our textures need to be combined, so it is just a small annoyance. A .log file will be saved as well, but these can be safely deleted.

You can leave the images as-is, or you can open up the glTF file in a text editor and manually change the image file names back to the original files:

  "images": [
    {
      "uri": "rock_moss_set_01_nor_gl_4k.jpg"
    },
    {
      "uri": "M_01___Defaultbasecolortexture.jpeg"
    },
    {
      "uri": "M_01___Defaultmetallicroughnesstex.jpeg"
    }
  ],

Finally, we're going to add LODs using the Mesh Editor > ProOptimizer modifier. I like these settings, but the most important thing is to make sure "Keep textures" is checked. You can press the F3 key at any time to toggle wireframe view and get a better view of what the optimizer does to your mesh.

Export the file with the same name as the full-resolution model, and add "_lod1" to the end of the file name (before the extension). Then repeat this process saving lod2 and lod3 using 25% and 12.5% for the vertex reduction value in the ProOptimizer modifier.

Here is my example model you can inspect:

mossrock1a.zip

Now it is very easy to get 3D models from 3ds max to your game engine.

Native Loading of Leadwerks File Formats

Ultra Engine loads and saves DDS, glTF, and OBJ files. Other formats are supported by plugins, both first and potentially third-party, for PNG, JPG, BMP, TGA, TIFF, GIF, HDR, KTX2, and other files. Additionally, all Leadwerks file formats are natively supported without requiring any plugins, so you can load TEX, MDL, PHY, MAT files, as well as full Leadwerks maps. You also have the option to save Leadwerks game engine formats in more up-to-date file formats such as glTF or DDS files with BC5/BC7 compression.

Testing a scene from the excellent Cyclone game (available on Steam!)

Classic Specular Shader Family

PBR materials are wonderful but act quite differently from conventional specular/gloss game materials. I've added a "Classic" shader family which provides an additive specular effect, which may not be realistic but it makes materials expect the way you would expect them to, coming from Leadwerks game engine. When Leadwerks MAT files are loaded, this shader family is automatically applied to them, to get a closer match to their appearance in Leadwerks.

Leadwerks Translation Layer

I'm experimenting with a Leadwerks header for Ultra Engine. You can drop this into your Ultra Engine project folder and then start running code with the Leadwerks API. Internally, the Leadwerks game engine commands will be translated into the Ultra Engine API to execute your existing code in the new engine. I don't think this will be 100% perfect due to some differences in the way the two engines work, but I do think it will give you an easy way to get started and provide a gentler transition to Ultra.

This code will actually work with both engines:

#include "Leadwerks.h"

using namespace Leadwerks;

int main(int argc, const char* argv[])
{
    Window* window = Window::Create();
  
    Context* context = Context::Create(window);
  
    World* world = World::Create();
    
    Camera* camera = Camera::Create();
    camera->Move(0, 0, -3);
    
    Light* light = DirectionalLight::Create();
    light->SetRotation(35, 35, 0);

    Model* model = Model::Box();
    model->SetColor(0.0, 0.0, 1.0);

    while (true)
    {
        if (window->Closed() || window->KeyDown(Key::Escape)) return false;

        model->Turn(0, Time::GetSpeed(), 0);

        Time::Update();
        world->Update();
        world->Render();
        context->Sync(false);

    }
    return 0;
}

I hope these enhancements give you a more enjoyable experience using Ultra together with the Leadwerks Editor, or as a standalone programming SDK.

Polyhaven has a large collection of free photo-based HDR environments, but they are all stored in EXR format as sphere maps. What we want are cubemaps stored in a single DDS file, preferably using texture compression.

We're going to use this online tool to convert the sphere maps to cubemaps. There are other ways to do this, and some may be better, but this is what I was able to find right away. By default, the page shows a background we can use for testing:

Let's process and save the image. Since we are just testing, it's fine to use a low resolution for now:

The layout is very important. We will select the last option, which exports each face as a separate image:

Processing and saving the image will result in a zip file downloaded to your computer, with six files:

• px.hdr
• nx.hdr
• py.hdr
• ny.hdr
• pz.hdr
• nz.hdr

These images correspond to the positive and negative directions on the X, Y, and Z axes.

We will start by just converting a single image, px.hdr. We will convert this pixmap to the format TEXTURE_BC6H, which corresponds to VK_FORMAT_BC6H_UFLOAT_BLOCK in Vulkan, or DXGI_FORMAT_BC6H_UF16 in the DDS file format. We need to load the FreeImage plugin for HDR file import, and the ISPC plugin for fast BC6H compression. The R16G16B16A16_SFLOAT is an intermediate format we need to convert the 32-bit floating point HDR file to, because the ISPC compressor expects this as the input format for BC6H compression:

auto plg = LoadPlugin("Plugins/FITextureLoader");
auto plg2 = LoadPlugin("Plugins/ISPCTexComp");

auto pixmap = LoadPixmap(GetPath(PATH_DESKTOP) + "/px.hdr");
pixmap = pixmap->Convert(TextureFormat(VK_FORMAT_R16G16B16A16_SFLOAT));
pixmap = pixmap->Convert(TEXTURE_BC6H);
pixmap->Save(GetPath(PATH_DESKTOP) + "/px.dds");

When we open the resulting DDS file in Visual Studio it appear quite dark, but I think this is due to how the HDR image is stored. I'm guessing colors are being scaled to a range between zero and one in the HDR file:

Now that we know how to convert a save a single 2D texture, let's try saving a more complex cube map texture with mipmaps. to do this, we need to pass an an array of pixmaps to the SaveTexture function, containing all the mipmaps in the DDS file, in the correct order. Microsoft's documentation on the DDS file format is very helpful here.

std::vector<std::shared_ptr<Pixmap> > mipchain;
WString files[6] = { "px.hdr", "nx.hdr", "py.hdr", "ny.hdr", "pz.hdr", "nz.hdr" };
for (int n = 0; n < 6; ++n)
{
    auto pixmap = LoadPixmap(GetPath(PATH_DESKTOP) + "/" + files[n]);
    pixmap = pixmap->Convert(TextureFormat(VK_FORMAT_R16G16B16A16_SFLOAT));
    Assert(pixmap);
    mipchain.push_back(pixmap->Convert(TEXTURE_BC6H));
    while (true)
    {
        auto size = pixmap->size;
        size /= 2;
        pixmap = pixmap->Resize(size.x, size.y);
        Assert(pixmap);
        mipchain.push_back(pixmap->Convert(TEXTURE_BC6H));
        if (size.x == 4 and size.y == 4) break;
    }
}
SaveTexture(GetPath(PATH_DESKTOP) + "/skybox.dds", TEXTURE_CUBE, mipchain, 6);

when we open the resulting DDS file in Visual Studio we can select different cube map faces and mipmap levels, and see that things generally look like we expect them to:

Now is a good time to load our cube map up in the engine and make sure it does what we think it does:

auto skybox = LoadTexture(GetPath(PATH_DESKTOP) + "/skybox.dds");
Assert(skybox);
world->SetSkybox(skybox);

When we run the program we can see the skybox appearing. It still appears very dark in the engine:

We can change the sky color to lighten it up, and the image comes alive. We don't have any problems with the dark colors getting washed out, because this is an HDR image:

world->SetSkyColor(4.0);

Now that we have determined that our pipeline works, let's try converting a sky image at high resolution. I will use this image from Polyhaven because it has a nice interesting sky. However, there's one problem. Polyhaven stores the image in EXR format, and the cube map generator I am using only loads HDR files. I used this online converter to convert my EXR to HDR format, but there are probably lots of other tools that can do the job.

Once we have the new skybox loaded in the cubemap generator, we will again process and save it, but this time at full resolution:

Save the file like before and run the DDS creation code. If you are running in debug mode, it will take a lot longer to process the image this time, but when it's finished the results will be worthwhile!:

This will be very interesting to view in VR. Our final DDS file is 120 MB, much smaller than the 768 MB an uncompressed R16G16B16A16 image would require. This gives us better performance and requires a lot less hard drive space. When we open the file in Visual Studio we can browser through the different faces and mipmap levels, and everything seems to be in order:

Undoubtedly this process will get easier and more automated with visual tools, but it's important to get the technical basis laid down first. With complete and in-depth support for the DDS file format, we can rely on a widely supported format for textures, instead of black-box file formats that lock your game content away.

Terrains are an entity, just like anything else, and can be positioned, rotated, or scaled. Non-square terrains are supported, so you can create something with a 1024x512 or whatever resolution. (Power-of-two sizes are required.)

Editing Terrain

Ultra Engine includes an API that lets you modify terrain in real-time. I took something very complicated and distilled it down to a very simple API for you:

terrain->SetElevation(x, y, height);

That's it! The engine will automatically update normals, physics, raycasting, rendering, and other data for you behind the scenes any time you modify the terrain, in a manner that maximizes efficiency.

Materials

Leadwerks supports 32 terrain texture layers. Unity supports eight. Thanks to the flexibility of shaders in Vulkan, any terrain in Ultra can have up to a whopping 256 different materials applied to it, and it will always be rendered in a single pass at maximum speed.

To apply a material at any point on the terrain, you just call this method. The weight value lets you control how much influence the material as at that point, i.e. its alpha transparency:

terrain->SetMaterial(x, y, material, weight)

This example shows how to paint materials onto the terrain.

I'm quite happy with how the documentation system has turned out, and I feel it is representative of the quality I want your entire user experience to be.

All the API examples load media from the web, so there's no need to manually download any extra files. Copy and paste the code into any project, and it just works.

//Create base material
auto diffusemap = LoadTexture("https://raw.githubusercontent.com/Leadwerks/Documentation/master/Assets/Materials/Ground/river_small_rocks_diff_4k.dds");
auto normalmap = LoadTexture("https://raw.githubusercontent.com/Leadwerks/Documentation/master/Assets/Materials/Ground/river_small_rocks_nor_gl_4k.dds");
auto ground = CreateMaterial();
ground->SetTexture(diffusemap, TEXTURE_DIFFUSE);
ground->SetTexture(normalmap, TEXTURE_NORMAL);

Tessellation and Displacement

Tessellation works great with terrain, which is modeled with quads. There are a lot of high-quality free PBR materials that are great for VR available on the web, so you will have no shortage of interesting materials to paint your terrains with. Here are a few sites for you to peruse:

https://matlib.gpuopen.com/main/materials/all
https://icons8.com/l/3d-textures
https://ambientcg.com/
https://polyhaven.com/textures
https://freepbr.com/
https://www.cgbookcase.com/textures

Cutting Holes

You can cut holes in the terrain by hiding any tile. This is useful for making caves, and can be used in the future for blending voxel geometry into a heightmap terrain.

Shader Design

Shaders in Ultra are big and quite complicated. I don't expect anyone to make 100% custom shaders like you could with the simpler shaders in Leadwerks. I've structured shaders so that the entry point shader defines a user function, then includes the base file, which calls the function:

#version 450
#extension GL_GOOGLE_include_directive : enable
#extension GL_ARB_separate_shader_objects : enable

#define LIGHTING_PBR
#define USERFUNCTION

#include "../Base/Materials.glsl"
#include "../Base/TextureArrays.glsl"

void UserFragment(inout vec4 color, inout vec3 emission, inout vec3 normal, inout float metalness, inout float roughness, inout float ambientocclusion, in vec3 position, in vec2 texcoords, in vec3 tangent, in vec3 bitangent, in Material material)
{
	// Custom code goes here...
}

#include "../Base/base_frag.glsl"

This organizes shaders so custom behavior can be added on top of the lighting and other systems, and paves the way for a future node-based shader designer.

Future Development

Ideas for future development to add to this system include voxel-based terrains for caves and overhangs, roads (using the decals system), large streaming terrains, and seamless blending of models into the terrain surface.

• Curved surfaces get smoother (using point-normal triangles or quads)
• A displacement map can be used to make small geometric detail to a surface.

These are really nice features because they don't require a lot of memory or disk space, and they're automatic. However, tessellation also has a nasty tendency to create cracks in geometry, so it tends to work best with organic models like rocks that are carefully wrapped. I was able to mitigate some of these problems with a per-vertex displacement value, which dampens displacement wherever there is a mesh seam:

However, this does not help round edges become rounder. With this cylinder, because the faces are not a continuous rounded surface, a crack appears at the sharp edges:

What if there was a way to properly tessellate mechanical shapes like this? Wouldn't it be great if all your models could just automatically gain new detail that scales with the camera distance, with no need for extra Lod versions?

Well, I had an idea how it might be possible, and a few hours later I came up with something. Here is the original model with no tessellation applied:

Here is the same model with my new and improved tessellation shader, suitable for organic and mechanical shapes:

A closeup view reveals perfectly aligned geometry with no gaps or cracks:

The distribution of polygons can probably be made more uniform with additional work. This feature uses two additional bytes in the vertex structure to control a normal for tessellated curves. I think it will be possible to automatically detect seams and assign tessellation normals in the final editor. If no separate tessellation normals are assigned, then the default tessellation behavior occurs. This is a very promising technique because it could allow you to add a lot of detail to any model very easily with barely any effort.

There's still a lot of extra stuff intellisense shows you from Windows headers and other things:

Using the __INTELLISENSE__ macro in Visual Studio, I was able to trick the intellisense compiler into skipping include files that link to third party libraries and system headers the user doesn't need to worry about. This strips all the unnecessary stuff out of the API, leaving just basic C++ commands, some STL classes, and the Ultra Engine API. With all the garbage removed, the API seems much friendlier:

This is one more detail that I feel helps make C++ with Ultra Engine about as easy as C# programming. You can disable this feature by adding _ULTRA_SIMPLE_API=0 in your project macros.

Variance shadow maps are a type of shadowmap filter technique that use a statistical sample of the depth at each pixel to do some funky math stuff. GPU Gems 3 has a nice chapter on the technique. The end result is softer shadows that run faster. I was wondering where my variance shadow map code went, until I realized this is something I only prototyped in OpenGL and never implemented in Vulkan until now. Here's my first pass at variance shadow maps in Vulkan:

There are a few small issues but they are no problem to work out. The blurring is taking place before the scene render, in the shadow map itself, which is a floating point color texture instead of a depth texture. (This makes VSMs faster than normal shadow maps.) The seams you see on the edges in the shot above are caused by that blurring, but there's a way we can fix that. If we store one sharp and one blurred image in the variance shadow map, we can interpolate between those based on distance from the shadow caster. Not only does this get rid of the ugly artifacts (say goodbye to shadow acne forever), but it also creates a realistic penumbra, as you can see in the shot of my original OpenGL implementation. Close to the shadow caster, the shadow is well-defined and sharp, but it gets much blurrier the further away it gets from the object:

Instead of blurring the near image after rendering, we can use MSAA to give it a fine-but-smooth edge. There is no such thing as an MSAA depth shadow sampler in GLSL, although I think there should be, and I have lobbied on behalf of this idea.

Finally, in my Vulkan implementation I used a compute shader instead of a fragment shader to perform the blurring. The advantage is that a compute shader can gather a bunch of samples and store them in memory, then access them to process a group of pixels at once. Instead of reading 9x9 pixels for each fragment, it can read a block of pixels and process them all at once, performing the same number of image writes, but much fewer reads:

// Read all required pixel samples
x = int(gl_WorkGroupID.x) * BATCHSIZE;
y = int(gl_WorkGroupID.y) * BATCHSIZE;
for (coord.x = max(x - EXTENTS, 0); coord.x < min(x + BATCHSIZE + EXTENTS, outsize.x); ++coord.x)
{
    for (coord.y = max(y - EXTENTS, 0); coord.y < min(y + BATCHSIZE + EXTENTS, outsize.y); ++coord.y)
    {
        color = imageLoad(imagearrayCube[inputimage], coord);
        samples[coord.x - int(gl_WorkGroupID.x) * BATCHSIZE + EXTENTS][coord.y - int(gl_WorkGroupID.y) * BATCHSIZE + EXTENTS] = color;
    }
}

This same technique will be used to make post-processing effects faster. I previously thought the speed of those would be pretty much the same in every engine, but now I see ways they can be improved significantly for a general-use speed increase. @klepto2 has been talking about the benefits of compute shaders for a while, and he is right, they are very cool. Most performance-intensive post-processing effects perform some kind of gather operation, so compute shaders can make a big improvement there.

One issue with conventional VSMs is that all objects must cast a shadow. Otherwise, an object that appears in front of a shadow caster will be dark. However, I added some math of my own to fix this problem, and it appears to work with no issues. So that's not something we need to worry about.

All around, variance shadow maps are a big win. They run faster, look better, and eliminate shadow acne, so they basically kill three birds with one stone.