Jump to content

Five Programming Changes You'll See in Leadwerks 5

Josh

2,546 views

Leadwerks 4.x will see a few more releases, each remaining backwards compatible with previous versions.  Leadwerks 5.0 is the point where we will introduce changes that break compatibility with previous versions.  The changes will support a faster design that relies more on multi-threading, updates to take advantage of C++11 features, and better support for non-English speaking users worldwide.  Changes are limited to code; all asset files including models, maps, shaders, textures, and materials will continue to be backwards-compatible.

Let's take a look at some of the new stuff.

Shared Pointers
C++11 shared pointers eliminate the need for manual reference counting.  Using the auto keyword will make it easier to update Leadwerks 4 code when version 5 arrives.  You can read more about the use of shared pointers in Leadwerks 5 here.

Unicode Support
To support the entire world's languages, Leadwerks 5 will make use of Unicode everywhere.  All std::string variables will be replaced by std::wstring.  Lua will be updated to the latest version 5.3.  This is not compatible with LuaJIT, but the main developer has left the LuaJIT project and it is time to move on.  Script execution time is not a bottleneck, Leadwerks 5 gains a much longer window of time for your game code to run, and I don't recommend people build complex VR games in Lua.  So I think it is time to update.

Elimination of Bound Globals
To assist with multithreaded programming, I am leaning towards a stateless design with all commands like World::GetCurrent() removed.  An entity needs to be explicitly told which world it belongs to upon creation, or it must be created as a child of another entity:

auto entity = Pivot::Create(world);

I am considering encapsulating all global variables into a GameEngine object:

class GameEngine
{
public:  
  std::map<std::string, std::weak_ptr<Asset> > loadedassets;
  shared_ptr<GraphicsEngine> graphicsengine;
  shared_ptr<PhysicsEngine> physicsengine;
  shared_ptr<NetworkEngine> networkengine;
  shared_ptr<SoundEngine> soundengine;
  shared_ptr<ScriptEngine> scriptengine;//replaces Interpreter class
};

A world would need the GameEngine object supplied upon creation:

auto gameengine = GameEngine::Create();
auto world = World::Create(gameengine);

When the GameEngine object goes out of scope, the entire game gets cleaned up and everything is deleted, leaving nothing in memory.

New SurfaceBuilder Class
To improve efficiency in Leadwerks 5, surfaces will no longer be stored in system memory, and surfaces cannot be modified once they are created.  If you need a modifiable surface, you can create a SurfaceBuilder object.

auto builder = SurfaceBuilder::Create(gameengine);
builder->AddVertex(0,0,0);
builder->AddVertex(0,0,1);
builder->AddVertex(0,1,1);
builder->AddTriangle(0,1,2);
auto surface = model->AddSurface(builder);

When a model is first loaded, before it is sent to the rendering thread for drawing, you can access the builder object that is loaded for each surface:

auto model = Model::Load("Models\box.mdl", Asset::Unique);
for (int i=0; i<model->CountSurfaces(); ++i)
{
	auto surface = model->GetSurface(i);
	shared_ptr<SurfaceBuilder> builder = surface->GetBuilder();
  	if (builder != nullptr)
	{
		for (int v=0; v < surface->builder->CountVertices(); ++v)
		{
			Vec3 v = builder->GetVertexPosition(v);
		}
	}
}

98% of the time there is no reason to keep vertex and triangle data in system memory.  For special cases, the SurfaceBuilder class does the job, and includes functions that were previously found in the Surface class like UpdateNormals().  This will prevent surfaces from being modified by the user when they are in use in the rendering thread.

A TextureBuilder class will be used internally when loading textures and will operate in a similar manner.  Pixel data will be retained in system memory until the first render.  These classes have the effect of keeping all OpenGL (or other graphics API) code contained inside the rendering thread, which leads to our next new feature...

Asynchronous Loading
Because surfaces and textures defer all GPU calls to the rendering thread, there is no reason we can't safely load these assets on another thread.  The LoadASync function will simply return true or false depending on whether the file was able to be opened:

bool result = Model::LoadASync("Models\box.mdl");

The result of the load will be given in an event:

while (gameengine->eventqueue->Peek())
{
	auto event = gameengine->eventqueue->Wait();
	if (event.id == Event::AsyncLoadResult)
	{
		if (event.extra->GetClass() == Object::ModelClass)
		{
			auto model = static_cast<Model>(event.source.get());
		}
	}
}

Thank goodness for shared pointers, or this would be very difficult to keep track of!

Asynchronous loading of maps is a little more complicated, but with proper encapsulation I think we can do it.  The script interpreter will get a mutex that is locked whenever a Lua script executes so scripts can be run from separate threads:

gameengine->scriptengine->execmutex->Lock();
//Do Lua stuff
gameengine->scriptengine->execmutex->Unlock();

This allows you to easily do things like make an animated loading screen.  The code for this would look something like below:

Map::LoadASync("Maps/start.map", world);
while (true)
{
	while (EventQueue::Peek())
	{
		auto event = EventQueue::Wait();
		if (event.id == Event::AsyncLoadResult)
		{
			break;
		}
	}
	
	loadsprite->Turn(0,0,1);
	world->Render();// Context::Sync() might be done automatically here, not sure yet...
}

All of this will look pretty familiar to you, but with the features of C++11 and the new design of Leadwerks 5 it opens up a lot of exciting possibilities.



22 Comments


Recommended Comments

Another nice thing about shared pointers is we can safely return a Map object from Map::Load that contains a list of entities, and the user doesn't have to delete it.  You can use it or ignore it and let it go out of scope:

auto map = Map::Load("Maps/start.map",world);
for (int i=0; i<map->CountEntities(); i++)
{
	auto entity = map->GetEntity(i);
}
//nothing to delete or clean up!

 

Share this comment


Link to comment

With the creation of a GameEngine class, does this mean we could have two worlds loaded at once?

Say for example, have level 1 loaded and playing, then in the background, spawn a new GameEngine to load level 2 for an almost seamless transition?

Share this comment


Link to comment
2 minutes ago, martyj said:

With the creation of a GameEngine class, does this mean we could have two worlds loaded at once?

Say for example, have level 1 loaded and playing, then in the background, spawn a new GameEngine to load level 2 for an almost seamless transition?

I don't recommend creating multiple game engine objects because they could not exist in the same graphics context, but I believe you could create another world and safely load another map into that world in the background.  This is part of the reason the idea of a "current" world is going away.

Share this comment


Link to comment

That async loading style seems to be a pain. Ideally I'm guessing there would be one place where the event loop exists, but loading of models may exist in another place. So how do you get the resulting model thst was loaded to the place that called the load async function?

C# handles this with an idea of async/await. It's a form of coroutine where you prefix an async function with await and it'll leave that function and continue with the main app loop and enter back in where it left off when that operation is finished. From the programmers point of view you write sequential code that's doing asynchronous stuff. Sadly C++ doesn't have this directly but it is proposed. However all that's really doing is a state machine behind the scenes to make it seem easy. It seems like C++ has some built in libraries that can mimic this behavior in c++11. It's called future library. I haven't used it but it may help get a more streamlined way of loading stuff async without needing some kind of event loop to go through which messes up the program flow and will probably result in bad designs and/or global variables required.

 

Share this comment


Link to comment

Can you make Version 4.X (starting from 4.3 at least) available in the archive when version 5 arrives since those 2 versions are essentially different programs? I'm not really keen on changing my 10,000 line code framework to fit the new formula.

Share this comment


Link to comment
25 minutes ago, Rick said:

It seems like C++ has some built in libraries that can mimic this behavior in c++11. It's called future library. I haven't used it but it may help get a more streamlined way of loading stuff async without needing some kind of event loop to go through which messes up the program flow and will probably result in bad designs and/or global variables required.

 

I will look into this.  Another way is to create a separate thread and execute the loading command on that.

Maybe something like this would be better:

auto task = Model::AsyncLoad("Models\box.mdl");
while (!task->Completed())
{
	//Do some other stuff or return to the main loop
}
if (task->Result())
{
	auto model = static_cast<Model>(task->LoadedObject());
}
19 minutes ago, jen said:

Can you make Version 4.X available in the archive when version 5 arrives since those 2 versions are essentially different programs. I'm not really keen on changing my 10,000 code framework to fit the new formula.

Leadwerks 4.x will always be available on Steam.  The documentation will be moved to a "learn/4" folder on the site.  Leadwerks 5 will probably be a new app ID, if it is even released on Steam (it probably will but there's no telling yet where Steam will be by that point).

Keep in mind this is all a long ways off, but it's never too soon to start thinking about the future.  I definitely would not put any projects on hold for version 5, because it could be a long way away.

Share this comment


Link to comment

With the removal of world:getcurrent it would make scripts like my grenade script harder to make. I would need to know ahead of time, pre compile what world a prefab/entity will be loaded into, instead of asking what world spawned an entity. Also keep update,render,sync separate. A game server usually does not need to render or sync, but games where you might use buffers for game mechanics might need to renderer and not sync.

Share this comment


Link to comment
4 minutes ago, Einlander said:

With the removal of world:getcurrent it would make scripts like my grenade script harder to make. I would need to know ahead of time, pre compile what world a prefab/entity will be loaded into, instead of asking what world spawned an entity. 

I cannot find where your script is using the World::GetCurrent() commend.  You will still be able to retrieve the world an entity is created in.

Share this comment


Link to comment

A entity would need to explicitly need to be told what world to spawn into. Well when I load the prefab, to what world is it assigned to? It is not a child of any other entity. A drop in script would not have any clue what world it is being loaded into. So unless I can find what world I'm in then I would be stuck.

Share this comment


Link to comment
5 minutes ago, Einlander said:

A entity would need to explicitly need to be told what world to spawn into. Well when I load the prefab, to what world is it assigned to? It is not a child of any other entity. A drop in script would not have any clue what world it is being loaded into. So unless I can find what world I'm in then I would be stuck.

You would just do this:

function Script:Start()	
	self.GrenadePrefab = Prefab:Load(self.GrenadePrefabPath, self.entity.world)

And PickupWeapon.lua would pass it along like this:

function Script:Start()
	if self.vwepfile~="" then
		local prefab = Prefab:Load(self.vwepfile, self.entity.world)

I am not trying to make every command thread-safe, but eliminating bound states like this will make it simpler for me to program the internals of the engine, and it does allow some new multithreading techniques you can probably use.

:mellow:

Share this comment


Link to comment

So what about bound-states like your GraphicsDriver and what not? Will you remove these? I feel like that'd be a slightly harder one to remove cause it's require lots of design changes, but on the flip side it may not be neccesary to remove it because you won't have the main thread writing to the GraphicsDriver at any point. (I don't think.)

I'm more asking, do you plan to remove ALL bound states?

Share this comment


Link to comment
3 hours ago, Crazycarpet said:

So what about bound-states like your GraphicsDriver and what not? Will you remove these? I feel like that'd be a slightly harder one to remove cause it's require lots of design changes, but on the flip side it may not be neccesary to remove it because you won't have the main thread writing to the GraphicsDriver at any point. (I don't think.)

I'm more asking, do you plan to remove ALL bound states?

Not sure yet.  If I use a gameengine object then it would contain the graphicsengine (graphicsdriver) as a member.  In multithreaded programming this makes life a lot easier, but I don't want to pedantically enforce something just because.

Share this comment


Link to comment

Thing that worries me about this subject is I don't have a clue what any of you are talking about. Given a choice I will just use old fashioned c language. Its easy to learn, its fast and could do the job.

Share this comment


Link to comment
1 hour ago, cassius said:

Thing that worries me about this subject is I don't have a clue what any of you are talking about. Given a choice I will just use old fashioned c language. Its easy to learn, its fast and could do the job.

It's basically this without delete.

Share this comment


Link to comment
20 hours ago, Josh said:

Not sure yet.  If I use a gameengine object then it would contain the graphicsengine (graphicsdriver) as a member.  In multithreaded programming this makes life a lot easier, but I don't want to pedantically enforce something just because.

I get why you'd want to do this but would this not be the same thing in principal because your "GameEngine" object would nwo be the bound state? This would have the same implications n a multithreaded environment as GetCurrent() because like the latter, only one thread could write to this "GameEngine" state at a time. Reading is always thread safe.

I guess it doesn't matter because users don't have to play with that stuff, I just figure why not make life easier for yourself? :P

Share this comment


Link to comment
11 hours ago, jen said:

I would love to see support for double precision.

Enterprise edition may have this, both for physics and optional graphics, together with a floating-point depth buffer.

Share this comment


Link to comment

I'm curious, is there a point carrying on with Leadwerks 4.5/6 development if you're going to recreate the application in version 5 anyway?

Share this comment


Link to comment
50 minutes ago, jen said:

I'm curious, is there a point carrying on with Leadwerks 4.5/6 development if you're going to recreate the application in version 5 anyway?

Josh has an existing customer group that is using the engine for their (commercial) games. He can't simply say "alright I am going to put all my time in to this new program, that might be out in 1/2/3 years from now.".  Some time will go in maintaining the current engine and some time will go in to planning/making the new engine/editor.

Share this comment


Link to comment
5 hours ago, jen said:

I'm curious, is there a point carrying on with Leadwerks 4.5/6 development if you're going to recreate the application in version 5 anyway?

Everything new will carry over to version 5.  The only thing I am reluctant to put a lot of time into is additional features in the editor, since that code is eventually going to be replaced.

Share this comment


Link to comment

@Josh Are you going to keep the project library feature? Each project has it in its root folder like brecon.lib. I'm guessing all the functions in my game are in this library?

It's useful for studios. The lib can be shared with other developers in a team instead of the raw source code. I guess this was the purpose of the library in the first place.

Share this comment


Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Blog Entries

    • By Josh in Josh's Dev Blog 5
      I did a little experiment with FPS Creator Pack #75 by upsampling the images with Gigapixel, which uses deep learning to upsample images and infer details that don't appear in the original pixels. The AI neural network does a pretty impressive job, generating results that are look better than a simple sharpen filter: I doubled the size of the textures to 1024x1024. Then I generated normal maps from the high-res images using AMD's TGA to DOT3 tool, and saved the normal maps with BC5 DDS compression. The diffuse textures were saved with BC7 DDS compression. The images below are using a 4x magnification to demonstrate the difference.


      As you can see, the image that is upsampled with deep learning looks normal and the resized image looks like there is butter on the lens! It's hard to believe the above images came from a 256x128 section of an image.
      The workflow was pretty tedious, as I had to convert images to TGA, then to uncompressed or BC5 DDS, and then to BC7 in Visual Studio. Each BC7 texture took maybe 5-10 minutes to compress! So while this set represents the optimum quality for 2019 game tech, and the format for assets we want to use in LE5, the workflow has a lot of room for improvement.
      You can download the full package here:
      FPSCPack75TexturesHD.zip
    • By Josh in Josh's Dev Blog 0
      A new beta is available with the following changes:
      Script prefixes are now changed to lowercase entity:Update(), entity:Start(), etc., as well as widget:Draw(), etc. This is because Entity() and Widget() are going to be functions to cast an object to that type. Sprites are now created on a sprite layer object. A sprite layer is created in a world and added to a camera. This allows control over what camera sees what set of sprites. See the examples for details. GUI system is partially working. Resizing the window won't reposition items correctly. Only four widget types are supported, panel, button, hyperlink, and label. Example in the FPSGame demo. The game menu system is packed into an entity script and works really nicely. Widget scripts are a little harder to develop now since they use a system of persistent objects, but performance is very much better than LE4. An interesting detail is that you get free interpolation of position and color at a rate of 60 hz. A lot of work was done to improve the Lua binding system. See details here. Error reporting and handling is much improved. No work was done on sound. No work has been done to the multicam demo, which some people had trouble with. Actors crashing / Lua stack error bug fixed. Changed .bat files to use release version of EXE instead of debug. New commands EmitEvent() and GetEvent(). If the returned event type is EVENT_NONE, there is no event. EVENT_QUIT can be emitted anywhere in the program to signal the main loop to exit. Typical usage is like this: while window:Closed() == false do while true do local event = GetEvent() if event.id == EVENT_QUIT then return end if event.id == EVENT_NONE then break end if event.id == EVENT_WINDOW_CLOSE and Window(event.source) == window then return end--you don't need this when Window:Closed() is being checked for already end world:Update() world:Render(framebuffer) end  
    • By Josh in Josh's Dev Blog 2
      DPI scaling and the 2D drawing and GUI system were an issue I was a bit concerned about, but I think I have it worked out. This all goes back to the multi-monitor support that I designed back in September. Part of that system allows you to retrieve the DPI scale for each display. This gives you another piece of information in addition to the raw screen resolution. The display scale gives you a percentage value the user expects to see vector graphics at, with 100% being what you would expect with a regular HD monitor. If we scale our GUI elements and font sizes by the display scale we can adjust for screens with any pixel density.
      This shot shows 1920x1080 fullscreen with DPI scaling set to 100%:

      Here we see the same resolution, with scaling set to 125%:

      And this is with scaling set to 150%:

      The effect of this is that if the player is using a 4K, 8K, or any other type of monitor, your game can display finely detailed text at the correct size the user expects to see. It also means that user interfaces can be rendered at any resolution for VR.
      Rather than trying to automatically scale GUI elements I am giving you full control over the raw pixels. That means you have to decide how your widgets will be scaled yourself, and program it into the game interface, but there is nothing hidden from the developer. Here is my code I am working with now to create a simple game menu. Also notice there is no CreatePanel(), CreateButton(), etc. anymore, there is just one widget you create and set the script for. I might add an option for C++ actors as well, but since these are operating on the main logic thread there's not really a downside to running the code in Lua.
      local window = ActiveWindow() if window == nullptr then return end local framebuffer = window:GetFramebuffer() if framebuffer == nil then return end self.gui = CreateGUI(self.guispritelayer) --Main background panel self.mainpanel = CreateWidget(self.gui,"",0,0,framebuffer.size.x,framebuffer.size.y) self.mainpanel:SetScript("Scripts/GUI/Panel.lua", true) local scale = window.display.scale.y local w = 120 local h = 24 local sep = 36 local x = framebuffer.size.x / 6 local y = framebuffer.size.y / 2 - sep * 3 self.resumebutton = CreateWidget(self.mainpanel,"RESUME GAME",x,y,w,h) self.resumebutton:SetScript("Scripts/GUI/Hyperlink.lua", true) self.resumebutton:SetFontSize(14 * window.display.scale.y) y=y+sep*2 self.label2 = CreateWidget(self.mainpanel,"OPTIONS",x,y,w,h) self.label2:SetScript("Scripts/GUI/Hyperlink.lua", true) self.label2:SetFontSize(14 * window.display.scale.y) y=y+sep*2 self.quitbutton = CreateWidget(self.mainpanel,"QUIT", x,y, w,h) self.quitbutton:SetScript("Scripts/GUI/Hyperlink.lua", true) self.quitbutton:SetFontSize(14 * window.display.scale.y) w = 400 * scale h = 550 * scale self.optionspanel = CreateWidget(self.mainpanel,"QUIT", (framebuffer.size.x- w) * 0.5, (framebuffer.size.y - h) * 0.5, w, h) self.optionspanel:SetScript("Scripts/GUI/Panel.lua", true) self.optionspanel.color = Vec4(0.2,0.2,0.2,1) self.optionspanel.border = true self.optionspanel.radius = 8 * scale self.optionspanel.hidden = true  
×
×
  • Create New...