Render breaks after several map loads

[Dreikblack]

Can't reproduce it yet in simple example since it tends to happen with big maps with many entities.

In my game it can be reproduced by:

1. New game or Choose Map ->Easy Map -> Ok

2. F5 to quick save

3. After a second (have a delay to prevent other issue with double click) press F9 for quick load.

4, Do Quick Load 5+ times and eventually it either HUD became invisible fully or partly (buttons still can be pressed so it's not sudden UI->SetHidden anywhere) or game stuckes on Loading screen, but in fact game is just not being rendered - "Error: Invalid value" in console after every world->Render()

Another way is starting new game, returning to main menu and starting again.

Also beside HUD also anything transparent seems to be non rendered - shotgun cones, fog cubes, blood decals. And for some reason gibs invisible too (they only thing with a physics tho).

[Dreikblack]

After UI update HUD is working, but is not rendered anymore after 5+ loads

[klepto2]

Might be related to this:

it happens as soon as generated meshes or instances go beyond a certain value.

[Josh]

On 2/12/2025 at 1:05 PM, klepto2 said:

Might be related to this:

it happens as soon as generated meshes or instances go beyond a certain value.

This issue is resolved.

@DreikblackDo you still experience this problem with the current build?

[Dreikblack]

6 hours ago, Josh said:

@DreikblackDo you still experience this problem with the current build?

Yes. Now it's takes twice longer and before it happens another render bug happens:

1. Random visual artifacts - black strip, cubes. Visible only at some angles.

2. No outlines.

3. No sprites.

4. No brush circles under characters.

[Josh]

Okay, thanks for the info. I am going through all reports.

[Dreikblack]

Tried in debug with Easy map - render broke on 11th reload (only HUD visible, like in 2nd post).

No errors in console. btw memory usage keep increasing a bit with load, was something like 2 GB when issue appeared.

[Dreikblack]

Tried to reproduce it in FPS template. For some reason loading background is not stuck as in my game, but rendering in general breaks in strange ways after 20-30 loads:

Just replace main in FPS template project and keep hitting F9 after load until you get you similar result:

#include "Leadwerks.h"
#include "ComponentSystem.h"
#include "Encryption.h"

using namespace Leadwerks;

shared_ptr<Scene> scene;
shared_ptr<Framebuffer> framebuffer;
shared_ptr<World> world;
shared_ptr<World> loadingWorld;
shared_ptr<Interface> ui;
shared_ptr<Font> font;
iVec2 framebufferSize;
shared_ptr<Widget> btn;

shared_ptr<Sprite> loadingBackground;
shared_ptr<Camera> loadingCamera;


void initScene() {
    loadingWorld->Render(framebuffer);
    world = CreateWorld();
    WString mapname = "Maps/start.map";
    scene = LoadScene(world, mapname);
    for (auto const& entity : world->GetEntities()) {
        auto camera = entity->As<Camera>();
        if (camera) {
            ui = CreateInterface(camera, font, framebufferSize);
            ui->root->SetColor(0.0f, 0.0f, 0.0f, 0.0f);
            btn = CreateButton("Test",100, 100, 100, 100, ui->root);
            break;
        }
    }
}

int main(int argc, const char* argv[])
{
    RegisterComponents();
    auto displays = GetDisplays();
    auto window = CreateWindow("Leadwerks", 0, 0, 1280 * displays[0]->scale, 720 * displays[0]->scale, displays[0], WINDOW_CENTER | WINDOW_TITLEBAR);
    framebuffer = CreateFramebuffer(window); 

    font = LoadFont("Fonts/arial.ttf");
    framebufferSize = framebuffer->GetSize();

    loadingWorld = CreateWorld();
    loadingBackground = CreateSprite(loadingWorld, framebuffer->size.width, framebuffer->size.height);
    loadingBackground->SetColor(0.3f, 0.2f, 0.2f);
    loadingCamera = CreateCamera(loadingWorld, PROJECTION_ORTHOGRAPHIC);
    loadingCamera->SetPosition(framebufferSize.x * 0.5f, framebufferSize.y * 0.5f, 0);

    initScene();

    //Main loop
    while (window->Closed() == false and window->KeyDown(KEY_ESCAPE) == false)
    {
        if (window->KeyDown(KEY_F9)) {
            initScene();
        }
        world->Update();
        world->Render(framebuffer);
    }

    return 0;
}

 

[Josh]

It looks like the directional lights have a memory leak, as this simple example shows. If the sunlight color is set to black the memory usage is stable.

This also occurs if I add a point or spot light (although the latter is harder to see the change with). I tried adding a point light and disabling shadows on it, and the memory usage stayed constant.

I need to take a closer look at how bindless texture handles are being released.

start.zip

#include "Leadwerks.h"
#include "ComponentSystem.h"
#include "Encryption.h"

using namespace Leadwerks;

int main(int argc, const char* argv[])
{
    RegisterComponents();

    auto cl = ParseCommandLine(argc, argv);
    
    //Get the displays
    auto displays = GetDisplays();

    //Create a window
    auto window = CreateWindow("Leadwerks", 0, 0, 1280 * displays[0]->scale, 720 * displays[0]->scale, displays[0], WINDOW_CENTER | WINDOW_TITLEBAR);

    //Create a framebuffer
    auto framebuffer = CreateFramebuffer(window);

    //Create a world
    auto world = CreateWorld();
    
    //Load the map
    auto scene = LoadScene(world, "Maps/start.map");
    auto tm = Millisecs();

    //Main loop
    while (window->Closed() == false and window->KeyDown(KEY_ESCAPE) == false)
    {
        auto now = Millisecs();
        if (now - tm > 2000)
        {
            scene = LoadScene(world, "Maps/start.map");
            tm = now;
        }

        world->Update();
        world->Render(framebuffer);
    }
    return 0;
}

 

[Josh]

My test cases above are fixed, but more testing is required...

[Josh]

Testing with the FPS example map now, no problems. I think the problem was that textures weren't being freed correctly, so you eventually ran out of VRAM.

I will update the engine tomorrow with this fix.

[Josh]

All my tests are working correctly and I believe this is fixed. Update is available now on the beta branch.

[Dreikblack]

Still same issue after 10-12 reloads:

And in next load 3D render does not work at all.

[Josh]

Is the memory increasing each time?

[Dreikblack]

Not much. VRAM at first load ~3.2 GB and when issue occurs it's 3.6 Gb out of 16 (checked via Task Manager, without game 2.6 Gb). RAM goes from 900 Mb to ~2 Gb

[Josh]

Since the texture problem is fixed, it seems likely that your game code contains circular references that are keeping objects in memory. Otherwise my example above would not be working.

https://www.leadwerks.com/community/topic/67399-render-breaks-after-several-map-loads/#findComment-318192

 

[Dreikblack]

2 minutes ago, Josh said:

Since the texture problem is fixed, it seems likely that your game code contains circular references that are keeping objects in memory

Even if it's a case it should not affect render at all (and pretty sure it's not a case, would be like few dozen MB for very few little things that could left). Since when game engine suppose to totally break when PC have 80% free VRAM and even more RAM?

[Dreikblack]

19 minutes ago, Josh said:

Otherwise my example above would not be working.

VRAM (5.1 -> 5.3 at lows) and RAM (1400 -> 2000) usage increases with this code with FPS map as well. Only changed timer to 5000 ms. Used Release more.

Was not able to break render yet (with fps map), maybe will test more tomorrow

[Josh]

You also might see occasional one-time increases in memory, where the system or driver decides it will do a one-time allocation of more memory. Maybe it crossed a certain threshold and the driver just decided "this app might need more memory in the future" and it pre-allocates a bigger block.

What you want to look out for is constant memory increases that consistently happen each time. If it doesn't happen each time, it's probably just the system adjusting.

[Josh]

There's definitely a mem leak, but I don't know how big an issue it is, and have not been able to break the renderer yet.

I do not see any changes in the framerate each time the game is reloaded, which is good, because that means we aren't leaving a bunch of cameras of lights in memory. Looks like memory usage increases by about 100 MB each reload, which honestly isn't the end of the world.

[Josh]

Ah, finally, after about ten reloads:

It appears that the main camera is being disabled somehow. The framerate jumps way up, which indicates it isn't even being drawn. There's a "hall of mirrors" effect if you move, which indicates no camera is being drawn, rather than a different camera overwriting the screen.

Can you tell me where in the code this camera is created? Does it get recreated on a game load, or is it persistent in memory?

[Josh]

I looked at the number of cameras being rendered, and it does not change when the error begins. System is reporting 10 cameras are in use...

This occurs with MSAA enabled or disabled.

I am very curious to know where the main 3D camera is being created.

I ran the game with the debugger enabled, but it takes 10-15 minutes to make the bug appear that way, and once I am in, I don't see anything strange.

I saw the same exact behavior on an AMD card, which indicates this isn't due to a driver bug.

It's difficult to tell what to look for when there are so many lights and cameras in use. If it is possible to simplify the problem, that would be much easier to work with.

[Dreikblack]

5 hours ago, Josh said:

I am very curious to know where the main 3D camera is being created.

It's very simple - in map via the Editor (with TacticCamera component)

[Josh]

Hmmm, recreating the cameras each time seems like something I would not do. The system should be able to handle that, but obviously something is wrong, and currently it's almost impossible to debug due to the amount of cameras and lights there are.

[Dreikblack]

Tried with simple level without lights except sun - same issue