Jump to content

Multithreaded Architecture in Leadwerks Game Engine 5

Josh

1,953 views

Leadwerks Game Engine 5 is a restructuring of Leadwerks Game Engine 4 to adapt to the demands of virtual reality and leverage the full capabilities of modern and future hardware.

Basically, the main idea with VR is that if you don't maintain a steady 90 FPS in virtual reality, you will throw up.  Nausea may be the worst physiological feeling you can experience.  In fact, nausea has been rated by cancer patients as worse than pain.  Being sensitive to motion sickness myself, this is a problem I am very motivated to solve.

In a conventional renderer, running both your game logic and rendering at 60 hz (frames per second) seems reasonable.  However, when we up the framerate to the 90 hz required for fluid virtual reality experiences, it seems like an excessive demand on the game code.  Game logic normally handles AI, player input, and various other tasks, and those things don't have to be updated quite that often.

Distributing Tasks
The solution I have come up with is to decouple the game loop from the renderer.  In the diagram below, the game loop is running at only 30 hz, while the physics, culling, and rendering loops are running at independent frequencies.

multithread1.png.9a5b81c1fd8ac31eac590a913b23894c.png

Think of this as like gears on a bicycle.  Your pedals move slowly, but your wheels spin very fast.  The game logic loop is like your pedals, while the rendering loop is like the rear wheel it is connected to.

gears.jpg.2a420b3c735ed09ebaad61597144f6b8.jpg

Previously, your game logic needed to execute in about 8 milliseconds or it would start slowing down the framerate.  With my design here, your game code gets more than 32 milliseconds to execute, a lifetime in code execution time, while a steady framerate of 90 or 60 FPS is constantly maintained.

I actually came up with this idea on my own, but upon further research I found out this is exactly what Intel recommends.  The technique is called Free Step Mode.  The diagram below does not correspond to our engine design, but it does illustrate the idea that separate systems are operating at different speeds:

7951-2.jpg.a224d98361a9c2890ece33362afa653f.jpg

If all threads are set to execute at the same frequency, it is called Lock Step Mode.

7952.jpg.913116238ba12998a0b351549d732e16.jpg

Data Exchange
Data in the game loop is exchanged with the physics and navmesh threads, but is passed one-way on to the culling loop, where it is then passed in a single direction to the rendering loop.  This means there will be a slight delay between when an event occurs and when it makes its way to the rendering thread and the final screen output, but we are talking times on the level of perhaps 10 milliseconds, so it won't be noticeable.  The user will just see smooth motion.

multithread2.png.5f514bfdc821b4e6404455b91c10248a.png

Rather than risk a lot of mutex locks, data is going to be passed one-way and each thread will have a copy of the object.  The game loop will have the full entity class, but the rendering threads will only have a stripped-down class, something like this:

class RenderObject
{
public:
	Mat4 matrix;
	AABB aabb;
	std::vector<Surface*> surfaces;
};

The original entity this object corresponds to can be modified or deleted, without fear of affecting downstream threads.  Again, Intel confirmed what I already thought would be the best approach:

Quote

In order for a game engine to truly run parallel, with as little synchronization overhead as possible, it will need to have each system operate within its own execution state with as little interaction as possible to anything else that is going on in the engine. Data still needs to be shared however, but now instead of each system accessing a common data location to say, get position or orientation data, each system has its own copy. This removes the data dependency that exists between different parts of the engine. Notices of any changes made by a system to shared data are sent to a state manager which then queues up all the changes, called messaging. Once the different systems are done executing, they are notified of the state changes and update their internal data structures, which is also part of messaging. Using this mechanism greatly reduces synchronization overhead, allowing systems to act more independently.

-Designing the Framework of a Parallel Game Engine, Jeff Andrews, Intel
https://software.intel.com/en-us/articles/designing-the-framework-of-a-parallel-game-engine

But wait, isn't latency a huge problem in VR, and I just described a system that adds latency to the renderer?  Yes and no.  The rendering thread will constantly update the headset and controller orientations, every single frame, at 90 hz.  The rest of the world will be 1-2 frames behind, but it won't matter because it's not connected to your body.  You'll get smooth head motion with zero delays while at the same time relieving the renderer of all CPU-side bottlenecks.

Even for non-VR games, I believe this design will produce a massive performance boost unlike anything you've ever seen.

  • Upvote 5


6 Comments


Recommended Comments

I don't understand, why render another frame if gameplay logic wasn't updated yet and nothing in the world changed? In non-VR games you will see two identical frames?

Will we be able to increase gameplay refresh rate? I think gameplay should be updated faster than rendering. My rhythm game feels much more responsive on 200 FPS than on 60 even though I have 60 hz monitor.

Share this comment


Link to comment
Just now, Genebris said:

I don't understand, why render another frame if gameplay logic wasn't updated yet and nothing in the world changed? In non-VR games you will see two identical frames?

Will we be able to increase gameplay refresh rate? I think gameplay should be updated faster than rendering. My rhythm game feels much more responsive on 200 FPS than on 60 even though I have 60 hz monitor.

The renderer can interpolate between two frames of data, creating a new frame in-between.

The frequencies of each system can probably be made adjustable.

Share this comment


Link to comment

Genebris,

While nothing in the game world may have changed, the camera may turn suddenly and a new rendered frame is necessary, one with out lag. I don't want game logic impacting movement of camera. 

 

 

Share this comment


Link to comment
1 minute ago, wayneg said:

Genebris,

While nothing in the game world may have changed, the camera may turn suddenly and a new rendered frame is necessary, one with out lag. I don't want game logic impacting movement of camera. 

 

 

That's the one point in the renderer where a mutex will lock, and a callback will be used to update the camera rotation and position.  The only thing that has to be instantaneous are the camera and VR controllers (if present).

Share this comment


Link to comment

I got goosebumps reading this. I am so excited to be able to leverage all of these great features. I cannot tell you how excited I am to have this the software in this architecture. It is really going to answer a lot of difficult questions for us in the engineering field as well. Well done leadwerks.

Share this comment


Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Create Your Account

Sign in

Already have an account? Sign in here.

Sign In Now
×