Jump to content

Five Programming Changes You'll See in Leadwerks 5

Josh

2,434 views

Leadwerks 4.x will see a few more releases, each remaining backwards compatible with previous versions.  Leadwerks 5.0 is the point where we will introduce changes that break compatibility with previous versions.  The changes will support a faster design that relies more on multi-threading, updates to take advantage of C++11 features, and better support for non-English speaking users worldwide.  Changes are limited to code; all asset files including models, maps, shaders, textures, and materials will continue to be backwards-compatible.

Let's take a look at some of the new stuff.

Shared Pointers
C++11 shared pointers eliminate the need for manual reference counting.  Using the auto keyword will make it easier to update Leadwerks 4 code when version 5 arrives.  You can read more about the use of shared pointers in Leadwerks 5 here.

Unicode Support
To support the entire world's languages, Leadwerks 5 will make use of Unicode everywhere.  All std::string variables will be replaced by std::wstring.  Lua will be updated to the latest version 5.3.  This is not compatible with LuaJIT, but the main developer has left the LuaJIT project and it is time to move on.  Script execution time is not a bottleneck, Leadwerks 5 gains a much longer window of time for your game code to run, and I don't recommend people build complex VR games in Lua.  So I think it is time to update.

Elimination of Bound Globals
To assist with multithreaded programming, I am leaning towards a stateless design with all commands like World::GetCurrent() removed.  An entity needs to be explicitly told which world it belongs to upon creation, or it must be created as a child of another entity:

auto entity = Pivot::Create(world);

I am considering encapsulating all global variables into a GameEngine object:

class GameEngine
{
public:  
  std::map<std::string, std::weak_ptr<Asset> > loadedassets;
  shared_ptr<GraphicsEngine> graphicsengine;
  shared_ptr<PhysicsEngine> physicsengine;
  shared_ptr<NetworkEngine> networkengine;
  shared_ptr<SoundEngine> soundengine;
  shared_ptr<ScriptEngine> scriptengine;//replaces Interpreter class
};

A world would need the GameEngine object supplied upon creation:

auto gameengine = GameEngine::Create();
auto world = World::Create(gameengine);

When the GameEngine object goes out of scope, the entire game gets cleaned up and everything is deleted, leaving nothing in memory.

New SurfaceBuilder Class
To improve efficiency in Leadwerks 5, surfaces will no longer be stored in system memory, and surfaces cannot be modified once they are created.  If you need a modifiable surface, you can create a SurfaceBuilder object.

auto builder = SurfaceBuilder::Create(gameengine);
builder->AddVertex(0,0,0);
builder->AddVertex(0,0,1);
builder->AddVertex(0,1,1);
builder->AddTriangle(0,1,2);
auto surface = model->AddSurface(builder);

When a model is first loaded, before it is sent to the rendering thread for drawing, you can access the builder object that is loaded for each surface:

auto model = Model::Load("Models\box.mdl", Asset::Unique);
for (int i=0; i<model->CountSurfaces(); ++i)
{
	auto surface = model->GetSurface(i);
	shared_ptr<SurfaceBuilder> builder = surface->GetBuilder();
  	if (builder != nullptr)
	{
		for (int v=0; v < surface->builder->CountVertices(); ++v)
		{
			Vec3 v = builder->GetVertexPosition(v);
		}
	}
}

98% of the time there is no reason to keep vertex and triangle data in system memory.  For special cases, the SurfaceBuilder class does the job, and includes functions that were previously found in the Surface class like UpdateNormals().  This will prevent surfaces from being modified by the user when they are in use in the rendering thread.

A TextureBuilder class will be used internally when loading textures and will operate in a similar manner.  Pixel data will be retained in system memory until the first render.  These classes have the effect of keeping all OpenGL (or other graphics API) code contained inside the rendering thread, which leads to our next new feature...

Asynchronous Loading
Because surfaces and textures defer all GPU calls to the rendering thread, there is no reason we can't safely load these assets on another thread.  The LoadASync function will simply return true or false depending on whether the file was able to be opened:

bool result = Model::LoadASync("Models\box.mdl");

The result of the load will be given in an event:

while (gameengine->eventqueue->Peek())
{
	auto event = gameengine->eventqueue->Wait();
	if (event.id == Event::AsyncLoadResult)
	{
		if (event.extra->GetClass() == Object::ModelClass)
		{
			auto model = static_cast<Model>(event.source.get());
		}
	}
}

Thank goodness for shared pointers, or this would be very difficult to keep track of!

Asynchronous loading of maps is a little more complicated, but with proper encapsulation I think we can do it.  The script interpreter will get a mutex that is locked whenever a Lua script executes so scripts can be run from separate threads:

gameengine->scriptengine->execmutex->Lock();
//Do Lua stuff
gameengine->scriptengine->execmutex->Unlock();

This allows you to easily do things like make an animated loading screen.  The code for this would look something like below:

Map::LoadASync("Maps/start.map", world);
while (true)
{
	while (EventQueue::Peek())
	{
		auto event = EventQueue::Wait();
		if (event.id == Event::AsyncLoadResult)
		{
			break;
		}
	}
	
	loadsprite->Turn(0,0,1);
	world->Render();// Context::Sync() might be done automatically here, not sure yet...
}

All of this will look pretty familiar to you, but with the features of C++11 and the new design of Leadwerks 5 it opens up a lot of exciting possibilities.



22 Comments


Recommended Comments

Another nice thing about shared pointers is we can safely return a Map object from Map::Load that contains a list of entities, and the user doesn't have to delete it.  You can use it or ignore it and let it go out of scope:

auto map = Map::Load("Maps/start.map",world);
for (int i=0; i<map->CountEntities(); i++)
{
	auto entity = map->GetEntity(i);
}
//nothing to delete or clean up!

 

Share this comment


Link to comment

With the creation of a GameEngine class, does this mean we could have two worlds loaded at once?

Say for example, have level 1 loaded and playing, then in the background, spawn a new GameEngine to load level 2 for an almost seamless transition?

Share this comment


Link to comment
2 minutes ago, martyj said:

With the creation of a GameEngine class, does this mean we could have two worlds loaded at once?

Say for example, have level 1 loaded and playing, then in the background, spawn a new GameEngine to load level 2 for an almost seamless transition?

I don't recommend creating multiple game engine objects because they could not exist in the same graphics context, but I believe you could create another world and safely load another map into that world in the background.  This is part of the reason the idea of a "current" world is going away.

Share this comment


Link to comment

That async loading style seems to be a pain. Ideally I'm guessing there would be one place where the event loop exists, but loading of models may exist in another place. So how do you get the resulting model thst was loaded to the place that called the load async function?

C# handles this with an idea of async/await. It's a form of coroutine where you prefix an async function with await and it'll leave that function and continue with the main app loop and enter back in where it left off when that operation is finished. From the programmers point of view you write sequential code that's doing asynchronous stuff. Sadly C++ doesn't have this directly but it is proposed. However all that's really doing is a state machine behind the scenes to make it seem easy. It seems like C++ has some built in libraries that can mimic this behavior in c++11. It's called future library. I haven't used it but it may help get a more streamlined way of loading stuff async without needing some kind of event loop to go through which messes up the program flow and will probably result in bad designs and/or global variables required.

 

Share this comment


Link to comment

Can you make Version 4.X (starting from 4.3 at least) available in the archive when version 5 arrives since those 2 versions are essentially different programs? I'm not really keen on changing my 10,000 line code framework to fit the new formula.

Share this comment


Link to comment
25 minutes ago, Rick said:

It seems like C++ has some built in libraries that can mimic this behavior in c++11. It's called future library. I haven't used it but it may help get a more streamlined way of loading stuff async without needing some kind of event loop to go through which messes up the program flow and will probably result in bad designs and/or global variables required.

 

I will look into this.  Another way is to create a separate thread and execute the loading command on that.

Maybe something like this would be better:

auto task = Model::AsyncLoad("Models\box.mdl");
while (!task->Completed())
{
	//Do some other stuff or return to the main loop
}
if (task->Result())
{
	auto model = static_cast<Model>(task->LoadedObject());
}
19 minutes ago, jen said:

Can you make Version 4.X available in the archive when version 5 arrives since those 2 versions are essentially different programs. I'm not really keen on changing my 10,000 code framework to fit the new formula.

Leadwerks 4.x will always be available on Steam.  The documentation will be moved to a "learn/4" folder on the site.  Leadwerks 5 will probably be a new app ID, if it is even released on Steam (it probably will but there's no telling yet where Steam will be by that point).

Keep in mind this is all a long ways off, but it's never too soon to start thinking about the future.  I definitely would not put any projects on hold for version 5, because it could be a long way away.

Share this comment


Link to comment

With the removal of world:getcurrent it would make scripts like my grenade script harder to make. I would need to know ahead of time, pre compile what world a prefab/entity will be loaded into, instead of asking what world spawned an entity. Also keep update,render,sync separate. A game server usually does not need to render or sync, but games where you might use buffers for game mechanics might need to renderer and not sync.

Share this comment


Link to comment
4 minutes ago, Einlander said:

With the removal of world:getcurrent it would make scripts like my grenade script harder to make. I would need to know ahead of time, pre compile what world a prefab/entity will be loaded into, instead of asking what world spawned an entity. 

I cannot find where your script is using the World::GetCurrent() commend.  You will still be able to retrieve the world an entity is created in.

Share this comment


Link to comment

A entity would need to explicitly need to be told what world to spawn into. Well when I load the prefab, to what world is it assigned to? It is not a child of any other entity. A drop in script would not have any clue what world it is being loaded into. So unless I can find what world I'm in then I would be stuck.

Share this comment


Link to comment
5 minutes ago, Einlander said:

A entity would need to explicitly need to be told what world to spawn into. Well when I load the prefab, to what world is it assigned to? It is not a child of any other entity. A drop in script would not have any clue what world it is being loaded into. So unless I can find what world I'm in then I would be stuck.

You would just do this:

function Script:Start()	
	self.GrenadePrefab = Prefab:Load(self.GrenadePrefabPath, self.entity.world)

And PickupWeapon.lua would pass it along like this:

function Script:Start()
	if self.vwepfile~="" then
		local prefab = Prefab:Load(self.vwepfile, self.entity.world)

I am not trying to make every command thread-safe, but eliminating bound states like this will make it simpler for me to program the internals of the engine, and it does allow some new multithreading techniques you can probably use.

:mellow:

Share this comment


Link to comment

So what about bound-states like your GraphicsDriver and what not? Will you remove these? I feel like that'd be a slightly harder one to remove cause it's require lots of design changes, but on the flip side it may not be neccesary to remove it because you won't have the main thread writing to the GraphicsDriver at any point. (I don't think.)

I'm more asking, do you plan to remove ALL bound states?

Share this comment


Link to comment
3 hours ago, Crazycarpet said:

So what about bound-states like your GraphicsDriver and what not? Will you remove these? I feel like that'd be a slightly harder one to remove cause it's require lots of design changes, but on the flip side it may not be neccesary to remove it because you won't have the main thread writing to the GraphicsDriver at any point. (I don't think.)

I'm more asking, do you plan to remove ALL bound states?

Not sure yet.  If I use a gameengine object then it would contain the graphicsengine (graphicsdriver) as a member.  In multithreaded programming this makes life a lot easier, but I don't want to pedantically enforce something just because.

Share this comment


Link to comment

Thing that worries me about this subject is I don't have a clue what any of you are talking about. Given a choice I will just use old fashioned c language. Its easy to learn, its fast and could do the job.

Share this comment


Link to comment
1 hour ago, cassius said:

Thing that worries me about this subject is I don't have a clue what any of you are talking about. Given a choice I will just use old fashioned c language. Its easy to learn, its fast and could do the job.

It's basically this without delete.

Share this comment


Link to comment
20 hours ago, Josh said:

Not sure yet.  If I use a gameengine object then it would contain the graphicsengine (graphicsdriver) as a member.  In multithreaded programming this makes life a lot easier, but I don't want to pedantically enforce something just because.

I get why you'd want to do this but would this not be the same thing in principal because your "GameEngine" object would nwo be the bound state? This would have the same implications n a multithreaded environment as GetCurrent() because like the latter, only one thread could write to this "GameEngine" state at a time. Reading is always thread safe.

I guess it doesn't matter because users don't have to play with that stuff, I just figure why not make life easier for yourself? :P

Share this comment


Link to comment
11 hours ago, jen said:

I would love to see support for double precision.

Enterprise edition may have this, both for physics and optional graphics, together with a floating-point depth buffer.

Share this comment


Link to comment

I'm curious, is there a point carrying on with Leadwerks 4.5/6 development if you're going to recreate the application in version 5 anyway?

Share this comment


Link to comment
50 minutes ago, jen said:

I'm curious, is there a point carrying on with Leadwerks 4.5/6 development if you're going to recreate the application in version 5 anyway?

Josh has an existing customer group that is using the engine for their (commercial) games. He can't simply say "alright I am going to put all my time in to this new program, that might be out in 1/2/3 years from now.".  Some time will go in maintaining the current engine and some time will go in to planning/making the new engine/editor.

Share this comment


Link to comment
5 hours ago, jen said:

I'm curious, is there a point carrying on with Leadwerks 4.5/6 development if you're going to recreate the application in version 5 anyway?

Everything new will carry over to version 5.  The only thing I am reluctant to put a lot of time into is additional features in the editor, since that code is eventually going to be replaced.

Share this comment


Link to comment

@Josh Are you going to keep the project library feature? Each project has it in its root folder like brecon.lib. I'm guessing all the functions in my game are in this library?

It's useful for studios. The lib can be shared with other developers in a team instead of the raw source code. I guess this was the purpose of the library in the first place.

Share this comment


Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Blog Entries

    • By Josh in Josh's Dev Blog 7
      You might have seen this graphic comparing the size of the world in different games. I've played Fuel, and never reached the end of the world in that game. You can drive for a very long time on those roads.

      We want to use the new engine for realistic simulations of air and ground movements. At normal cruising altitude of a commercial airliner, the pilot has a view range of about 400 kilometers. The image below shows that area (800 x 800 km). You can see the areas of the biggest games ever fit neatly into the corner of just our visible area.

      The gray space above is not the total world size, it is just the area you can see at once from high altitude. The total world size is about 50 times bigger.
      This is what I am working on now.
    • By Josh in Josh's Dev Blog 26
      Gamers have always been fascinated with the idea of endless areas to roam.  It seems we are always artificially constrained within a small area to play in, and the possibility of an entire world outside those bounds is tantalizing.  The game FUEL captured this idea by presenting the player with an enormous world that took hours to drive across:
      In the past, I always implemented terrain with one big heightmap texture, which had a fixed size like 1024x1024, 2048x2048, etc.  However, our vegetation system, featured in the book Game Engine Gems 3, required a different approach.  There was far too many instances of grass, trees, and rocks to store them all in memory, and I wanted to do something really radical.  The solution was to create an algorithm that could instantly calculate all the vegetation instances in a given area.  The algorithm would always produce the same result, but the actual data would never be saved, it was just retrieved in the area where you needed it, when you needed it.  So with a few modifications, our vegetation system is already set up to generate infinite instances far into the distance.

      However, terrain is problematic.  Just because an area is too far away to see doesn't mean it should stop existing.  If we don't store the terrain in memory then how do we prevent far away objects from falling into the ground?  I don't like the idea of disabling far away physics because it makes things very complex for the end user.  There are definitely some tricks we can add like not updating far away AI agents, but I want everything to just work by default, to the best of my ability.
      It was during the development of the vegetation system that I realized the MISSING PIECE to this puzzle.  The secret is in the way collision works with vegetation.  When any object moves all the collidable vegetation instances around it are retrieved and collision is performed on this fetched data.  We can do the exact same thing with terrain   Imagine a log rolling across the terrain.  We could use an algorithm to generate all the triangles it potentially could collide with, like in the image below.

      You can probably imagine how it would be easy to lay out an infinite grid of flat squares around the player, wherever he is standing in the world.

      What if we only save heightmap data for the squares the user modifies in the editor?  They can't possibly modify the entire universe, so let's just save their changes and make the default terrain flat.  It won't be very interesting, but it will work, right?
      What if instead of being flat by default, there was a function we had that would procedurally calculate the terrain height at any point?  The input would be the XZ position in the world and the output would be a heightmap value.

      If we used this, then we would have an entire procedurally generated terrain combined with parts that the developer modifies by hand with the terrain tools.  Only the hand-modified parts would have to be saved to a series of files that could be named "mapname_x_x.patch", i.e. "magickingdom_54_72.patch".  These patches could be loaded from disk as needed, and deleted from memory when no longer in use.
      The real magic would be in developing an algorithm that could quickly generate a height value given an XZ position.  A random seed could be introduced to allow us to create an endless variety of procedural landscapes to explore.  Perhaps a large brush could even be used to assign characteristics to an entire region like "mountainy", "plains", etc.
      The possibilities of what we can do in Leadwerks Engine 5 are intriguing.  Granted I don't have all the answers right now, but implementing a system like this would be a major step forward that unlocks an enormous world to explore.  What do you think?

    • By Haydenmango in Snowboarding Development Blog 6
      So I've been researching snowboarding lately to get an idea of what animations and mechanics I need to create for my game.  I have learned lots of interesting things since I've only seen snow once or twice in my entire life and have never even tried snowboarding or any other board sports (skateboarding, surfing, etc.) for that matter.
       
      Snowboarding tricks are quite interesting as they are mostly derived from skateboarding.  Snowboarding tricks pay homage to their equivalent skating tricks by sharing many concepts and names.  For example basic grabs in snowboarding share the same concepts and names as skateboarding: indy, mute, method, stalefish, nosegrab, and tailgrab.  Something interesting to note is in snowboarding you can grab Tindy or Tailfish but this is considered poor form since these grabs can't be done on a skateboard (due to the board not being attached to the skaters feet) and grabbing these areas is generally something a novice snowboarder does when failing or "half-assing" a normal grab.  Check out this diagram to see how grabs work -
       
       
      So, after reading lots of text descriptions for tricks I was still confused by what all these terms meant and how they were actually applied.  So my next step was to look up these tricks actually being done and I found some really cool videos showing off how to do various tricks.  This video in particular is the best reference material I've found as it contains nearly every trick back to back with labeled names and some tweaks -
       
      Sadly my rigged model doesn't handle leg animations with the snowboard that well so I can't animate as many tricks as I want to.  Regardless there will still be around 15 total grab/air tricks in the game.  Now it's time for me to stop procrastinating and start animating!  
×
×
  • Create New...