Jump to content

Lua binding in Leadwerks 5

Josh

2,283 views

The Leadwerks 5 API uses C++11 smart pointers for all complex objects the user interacts with. This design replaces the manual reference counting in Leadwerks 4 so that there is no Release() or AddRef() method anymore. To delete an object you just set all variables that reference that object to nullptr:

auto model = CreateBox();
model = nullptr; //poof!

In Lua this works the same way, with some caveats:

local window = CreateWindow()
local context = CreateContext(window)
local world = CreateWorld()

local camera = CreateCamera(world)
camera:SetPosition(0,0,-5)

local model = CreateBox()

while true do
	if window:KeyHit(KEY_SPACE) then
		model = nil
	end
	world:Render()
end

In the above example you would expect the box to disappear immediately, right? But it doesn't actually work that way. Lua uses garbage collection, and unless you are constantly calling the garbage collector each frame the model will not be immediately collected. One way to fix this is to manually call the garbage collector immediately after setting a variable to nil:

if window:KeyHit(KEY_SPACE) then
	model = nil
	collectgarbage()
end

However, this is not something I recommend doing. Instead, a change in the way we think about these things is needed. If we hide an entity and then set our variable to nil we can just defer the garbage collection until enough memory is accrued to trigger it:

if window:KeyHit(KEY_SPACE) then
	model:Hide()-- out of sight, out of mind
	model = nil
end

I am presently investigating the sol2 library for exposing the C++ API to Lua. Exposing a new class to Lua is pretty straightforward:

lua.new_usertype<World>("World", "Render", &World::Render, "Update", &World::Update);
lua.set_function("CreateWorld",CreateWorld);

However, there are some issues like downcasting shared pointers. Currently, this code will not work with sol2:

local a = CreateBox()
local b = CreateBox()
a:SetParent(b)-- Entity:SetParent() expects an Entity, not a Model, even though the Model class is derived from Entity

There is also no support for default argument values like the last argument has in this function:

Entity::SetPosition(const float x,const float y,const float z,const bool global=false)

This can be accomplished with overloads, but it would require A LOT of extra function definitions to mimic all the default arguments we use in Leadwerks.

I am talking to the developer now about these issues and we'll see what happens.

  • Like 1


54 Comments


Recommended Comments



Hiding and then setting to nil isn’t ideal from a usability standpoint. How about creating a base class method called Delete() which does this stuff for us in c++ as a more intuitive manner.

Share this comment


Link to comment
6 minutes ago, Rick said:

Hiding and then setting to nil isn’t ideal from a usability standpoint. How about creating a base class method called Delete() which does this stuff for us in c++ as a more intuitive manner.

All this would do is call Hide(). Maybe if we find other things this needs to do then it will be appropriate to have a Delete() method in the entity class.

Share this comment


Link to comment

The lua object is just a reference to C++ object right? Couldn't the C++ Delete() function delete itself? It's been awhile since I've used C++ but you could do:

class Entity{
  void Delete(Entity* e){
    e = nullptr;
  }
}

// whatever the smartpointer syntax is
Entity* e = new Entity();

e->Delete(e);

per https://stackoverflow.com/questions/1208961/can-an-object-instance-null-out-the-this-pointer-to-itself-safely

That way we have a standard interface of deleting objects between C++ and Lua.

Share this comment


Link to comment
6 minutes ago, Rick said:

The lua object is just a reference to C++ object right? Couldn't the C++ Delete() function delete itself? It's been awhile since I've used C++ but you could do:


class Entity{
  void Delete(Entity* e){
    e = nullptr;
  }
}

// whatever the smartpointer syntax is
Entity* e = new Entity();

e->Delete(e);

per https://stackoverflow.com/questions/1208961/can-an-object-instance-null-out-the-this-pointer-to-itself-safely

That way we have a standard interface of deleting objects between C++ and Lua.

Well actually the C++ class does not need anything to delete the object. Again, you just set the variable to nullptr:

auto model = CreateBox();
model = nullptr; //poof!

It's gone!

Share this comment


Link to comment
1 minute ago, Josh said:

Well actually the C++ class does not need anything to delete the object. Again, you just set the variable to nullptr:


auto model = CreateBox();
model = nullptr; //poof!

It's gone!

I get that's the case, but I'm trying to help find a consistent interface for this between the 2 languages because I think that's a better design and has value. You could make a global DeleteEntity(e) function perhaps that does that for both C++ objects and Lua objects so people just have to know 1 thing between the 2 languages and documentation can be shared to explain it between the 2 languages.

  • Like 1

Share this comment


Link to comment
1 hour ago, Rick said:

I get that's the case, but I'm trying to help find a consistent interface for this between the 2 languages because I think that's a better design and has value. You could make a global DeleteEntity(e) function perhaps that does that for both C++ objects and Lua objects so people just have to know 1 thing between the 2 languages and documentation can be shared to explain it between the 2 languages.

But then you're walking around with a variable that is no longer a valid entity, which is exactly what smart pointers seek to avoid:

auto box = CreateBox()
box:Delete()
box:SetPosition(1,2,3)

Moreover, one part of your code might be still using an entity when another part is done. This is why we don't have any explicit Delete(), Destroy(), Release() etc, function.

Share this comment


Link to comment
1 hour ago, Josh said:

But then you're walking around with a variable that is no longer a valid entity, which is exactly what smart pointers seek to avoid:


auto box = CreateBox()
box:Delete()
box:SetPosition(1,2,3)

Moreover, one part of your code might be still using an entity when another part is done. This is why we don't have any explicit Delete(), Destroy(), Release() etc, function.

I would assume you can make a global function that does the exact same thing as setting to nullptr. Delete(box); This way both lua and C++ can do the same thing. It's literally exactly the same thing just a more consistent interface between the 2 languages.

Share this comment


Link to comment
27 minutes ago, Rick said:

I would assume you can make a global function that does the exact same thing as setting to nullptr.

You can't. :) That's the beauty of this system. The scene Octree stores weak pointers to the entity and when the entity no longer exists, the weak pointers fail to convert into a shared_ptr, so they are skipped and removed, and the entity is no longer rendered. It removes a TON of error-prone cleanup of pointers I previously had to be very meticulous about.

It is also impossible to have an invalid pointer. :D

There is something weird to consider.

You don't want to call Hide() or Delete() or anything too aggressive because it is possible the entity might be in use somewhere else by some other script. But if you don't do this then the object will hang around until the garbage collector is called. So the correct way to handle this sometimes actually is to run the garbage collector after an entity is removed:

if window:KeyHit(KEY_SPACE) then
	model = nil
	collectgarbage()
end

This actually isn't so bad because in real life it is best to not be dynamically creating and deleting a lot of entities as the game is running.

I think in some situations you will simply set the variable to nil as if to say "I am done with this" while in other situations you will want to call Hide().

Share this comment


Link to comment

I could add a Destroy() method that removes all resources of any SharedObject (entities, materials, textures, etc.) but the object would still be valid and usable:

void Entity::Destroy()
{
	Hide();
	SetParent(nullptr);
	SetScript(nullptr);
	SetShape(nullptr);
	SetCollisionType(0);
	while (CountChildren())
	{
		GetChild(0)->Destroy();
	}
}

For example, the Lua code below would be perfectly legitimate:

local model = CreateBox()
model:Destroy()
model:Show()
local surf = model:AddSurface()
surf:AddVertex(0,1,1)
surf:AddVertex(0,-1,1)
surf:AddVertex(0,1,-1)
surf:AddTriangle(0,1,2)

And then you have a visible model again.
 

Share this comment


Link to comment
Quote

The scene Octree stores weak pointers to the entity and when the entity no longer exists, the weak pointers fail to convert into a shared_ptr, so they are skipped and removed

And a weak pointer fails to exist when the last variable that points to it sets it to nullptr?

Share this comment


Link to comment
19 minutes ago, Rick said:

And a weak pointer fails to exist when the last variable that points to it sets it to nullptr?

A weak pointer stops working when all shared pointers no longer exist. It's like a shared pointer that doesn't keep the object alive.

Share this comment


Link to comment

I think this is how you'd do it.

 

#include <memory>
#include <string>
#include <iostream>

using namespace std;

class Test{
private:
	string name;
public:
	Test(string _name){
		name = _name;
	}

	void Print(){
		cout << name;
	}
};


void Delete(shared_ptr<Test>& obj)
{
	obj.reset();
}

int main(){
	shared_ptr<Test> sptr(new Test("Test"));

	Delete(sptr);
	//sptr = nullptr;

	sptr.get()->Print();

	int input;

	cin >> input;

	return 0;
}

You can do an if(sptr) before trying to anything with it and that still works to make sure it's valid. If you do that on the C++ side for every command then in lua it'll never fail if we have multiple variables pointing to the one we deleted. The calls simply won't do anything, but at least they won't fail. You could make another function we can check perhaps like Exists(obj) that does the if statement on the shared_ptr and if not return false else return true? Just thinking high level on that one in case we want to know on the lua side if something was deleted.

Share this comment


Link to comment

What if there are other shared pointers pointing to the same object? I think reset will only invalidate that one variable:

int main(){
	shared_ptr<Test> sptr(new Test("Test"));
	auto a = sptr;
	Delete(sptr);
	//sptr = nullptr;
  	
	a->Print();

BTW you can just do sptr->Function() instead of sptr.get()->Function(). :)

Share this comment


Link to comment

From my testing reset() actually removes all references.

 

void Delete(shared_ptr<Test>& obj)
{
	long count = obj.use_count(); // will show 2

	obj.reset();

	count = obj.use_count(); // will show 0
}

int main(){
	shared_ptr<Test> sptr(new Test("Test"));

	shared_ptr<Test> ptr = sptr;

	Delete(sptr);
	//sptr = nullptr;

	sptr.get()->Print();

	int input;

	cin >> input;

	return 0;
}

Note the key is passing the shared_ptr by reference to the Delete() function. So you'd probably need 2 functions. The lua one we call that gets the shared pointer object we passed in and then another that takes the object by reference. This would be the case IF we can't pass by reference from lua. Not sure if that works with the bindings or not. If it does then great, only need the 1 delete.

Share this comment


Link to comment

If that were the case I would expect a to be NULL when it is printed here:

shared_ptr<std::string> ptr = make_shared <std::string>("Hello.");
auto a = ptr;
ptr.reset();
Print(*(a.get())); //print "Hello."

I think the object count is 0 because the shared_ptr no longer points to an object, not because all the other shared_ptrs were reset.

Share this comment


Link to comment

You may want to check that with any of the stuff you're currently doing as well. The following won't print because ptr doesn't pass the if check and count is 0 after the reset(). Perhaps some difference between objects and the string class? I mean you've seen it work with objects which you're creating in the engine, but this is a different usage with whatever string is doing behind the scenes. Or perhaps make_shared()?

	shared_ptr<std::string> ptr = make_shared <std::string>("Hello.");

	auto a = ptr;

	long count = ptr.use_count();

	ptr.reset();

	count = ptr.use_count();

	if(ptr)
		cout << *(a.get());

 

Share this comment


Link to comment

@Rick The behavior above makes perfect sense.

Smart pointers are like two people who know a language.

Let's say you and I are the only two people in the world who can speak German. If I forget German one day, the language does not die as long as one person knows it. I won't be able to tell anyone what Kartoffelsalat means, but you still know German and the language is not dead.

If we both forget to speak German then the language is dead like Latin.

Here's the equivalent code with manual reference counting:

std::string* ptr = new std::string("Hello.");
long count = 1;

auto a = ptr;
count++;

ptr = NULL;
count--;
	
if (a)
	cout << *(a);

delete a;
a = NULL;
count--;

 

Share this comment


Link to comment

But the above is saying if I asked you if you know German you're saying no I don't. The if statement check. Even though you really do, that check says you don't. However, as in our example with the Test class, the language is actually gone (and error happens when we try to access it's function because the memory doesn't seem to be there anymore). So why is the if check acting as it doesn't exist anymore which in the case of the Test() class shared_ptr it doesn't, but in this string case it does. You should always use the if check before trying to access a shared_ptr right? That's the benefit, that you can tell if it's valid or not anymore where straight pointers you can't.

Share this comment


Link to comment

Why are you basing your knowledge of one person's language skills on another person? :D

That's like saying if Shadmar's car is blue then you will drive Aggror's blue car.

Reseting a shared pointer is the equivalent to setting a pointer to NULL. You can't get any info on the object after that:

Entity* a = CreatePivot();
Entity* b = a;
a = NULL;
b->SetPosition(1,2,3);
delete b;
b = NULL;

Or a safer way in Leadwerks 4:

Entity* a = CreatePivot();

Entity* b = a;
b->AddRef();

a->Release()
a = NULL;

b->SetPosition(1,2,3);
b->Release();
b = NULL;

Now we simply do this:

shared_ptr<Entity> a = CreatePivot();
shared_ptr<Entity> b = a;
a = nullptr;
b->SetPosition(1,2,3);
b = nullptr;

If you want the removal of a to force b to be invalid, we use a weak pointer:

shared_ptr<Entity> a = CreatePivot();

weak_ptr<Entity> b = a;

a = nullptr;

shared_ptr<Entity> ptr = b.lock(); //creates a shared_ptr with a null value

if (ptr) ptr->SetPosition(1,2,3);

 

Share this comment


Link to comment

Correct, but when crossing language domains (C++/LUA) we can't set it to equal nullptr in lua directly like we can in C++. That's the entire point of my post. There is now a way to do that with the Delete() example I showed. That was why I showed it. Lua now has a way to be equal to C++ in the removal of the object vs hiding or even calling garbage collecting manually, both which are pretty hacky for when you want an object gone given you don't do that from C++ to remove the object.

Share this comment


Link to comment
2 hours ago, Rick said:

Correct, but when crossing language domains (C++/LUA) we can't set it to equal nullptr in lua directly like we can in C++. That's the entire point of my post. There is now a way to do that with the Delete() example I showed. That was why I showed it. Lua now has a way to be equal to C++ in the removal of the object vs hiding or even calling garbage collecting manually, both which are pretty hacky for when you want an object gone given you don't do that from C++ to remove the object.

Ah, I see what you mean. I will try it out and see if this works.

Share this comment


Link to comment

It sort of works. The object is deleted immediately, but Lua holds onto the invalid pointer. I am not sure how that is possible.

I think different copies of the Lua userdata actually share the same shared_ptr object, and it is only decremented after the GC collects all the userdata objects.

I will keep experimenting with this.

Share this comment


Link to comment

What do you mean by saying lua holds onto the invalid pointer? Do you mean that the lua variable doesn’t turn nil after Delete is called? I wouldn’t think it would given that lua variable is a shared pointer right? However if there was an Exists(obj) function that checked the lua shared ptr was valid and returns true or false then if we know a lua variable might be shared in our code like in the tower defense situation where 2 towers point to same obj then we use that Exists() to see if it’s actually valid on the c++ side

Share this comment


Link to comment
5 minutes ago, Rick said:

What do you mean by saying lua holds onto the invalid pointer? Do you mean that the lua variable doesn’t turn nil after Delete is called? I wouldn’t think it would given that lua variable is a shared pointer right? However if there was an Exists(obj) function that checked the lua shared ptr was valid and returns true or false then if we know a lua variable might be shared in our code like in the tower defense situation where 2 towers point to same obj then we use that Exists() to see if it’s actually valid on the c++ side

The object is deleted and its destructor is called but the shared_ptr somehow holds onto the pointer and keeps the same memory address. I have no idea how.

Normally your Exists() function is a check to see if something equals nil.

Share this comment


Link to comment
38 minutes ago, Josh said:

The object is deleted and its destructor is called but the shared_ptr somehow holds onto the pointer and keeps the same memory address. I have no idea how.

Normally your Exists() function is a check to see if something equals nil.

What's the negative aspect of the first part of what you said?

I would think because the C++ side is deleting things the Lua side isn't going to know about it so the lua variable won't be nil, which is why I was thinking the Exist() function is needed because it would do the if(shared_ptr) check in C++ which should correctly show it's no longer valid.

Share this comment


Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Blog Entries

    • By Josh in Josh's Dev Blog 5
      You might have seen this graphic comparing the size of the world in different games. I've played Fuel, and never reached the end of the world in that game. You can drive for a very long time on those roads.

      We want to use the new engine for realistic simulations of air and ground movements. At normal cruising altitude of a commercial airliner, the pilot has a view range of about 400 kilometers. The image below shows that area (800 x 800 km). You can see the areas of the biggest games ever fit neatly into the corner of just our visible area.

      The gray space above is not the total world size, it is just the area you can see at once from high altitude. The total world size is about 50 times bigger.
      This is what I am working on now.
    • By Josh in Josh's Dev Blog 26
      Gamers have always been fascinated with the idea of endless areas to roam.  It seems we are always artificially constrained within a small area to play in, and the possibility of an entire world outside those bounds is tantalizing.  The game FUEL captured this idea by presenting the player with an enormous world that took hours to drive across:
      In the past, I always implemented terrain with one big heightmap texture, which had a fixed size like 1024x1024, 2048x2048, etc.  However, our vegetation system, featured in the book Game Engine Gems 3, required a different approach.  There was far too many instances of grass, trees, and rocks to store them all in memory, and I wanted to do something really radical.  The solution was to create an algorithm that could instantly calculate all the vegetation instances in a given area.  The algorithm would always produce the same result, but the actual data would never be saved, it was just retrieved in the area where you needed it, when you needed it.  So with a few modifications, our vegetation system is already set up to generate infinite instances far into the distance.

      However, terrain is problematic.  Just because an area is too far away to see doesn't mean it should stop existing.  If we don't store the terrain in memory then how do we prevent far away objects from falling into the ground?  I don't like the idea of disabling far away physics because it makes things very complex for the end user.  There are definitely some tricks we can add like not updating far away AI agents, but I want everything to just work by default, to the best of my ability.
      It was during the development of the vegetation system that I realized the MISSING PIECE to this puzzle.  The secret is in the way collision works with vegetation.  When any object moves all the collidable vegetation instances around it are retrieved and collision is performed on this fetched data.  We can do the exact same thing with terrain   Imagine a log rolling across the terrain.  We could use an algorithm to generate all the triangles it potentially could collide with, like in the image below.

      You can probably imagine how it would be easy to lay out an infinite grid of flat squares around the player, wherever he is standing in the world.

      What if we only save heightmap data for the squares the user modifies in the editor?  They can't possibly modify the entire universe, so let's just save their changes and make the default terrain flat.  It won't be very interesting, but it will work, right?
      What if instead of being flat by default, there was a function we had that would procedurally calculate the terrain height at any point?  The input would be the XZ position in the world and the output would be a heightmap value.

      If we used this, then we would have an entire procedurally generated terrain combined with parts that the developer modifies by hand with the terrain tools.  Only the hand-modified parts would have to be saved to a series of files that could be named "mapname_x_x.patch", i.e. "magickingdom_54_72.patch".  These patches could be loaded from disk as needed, and deleted from memory when no longer in use.
      The real magic would be in developing an algorithm that could quickly generate a height value given an XZ position.  A random seed could be introduced to allow us to create an endless variety of procedural landscapes to explore.  Perhaps a large brush could even be used to assign characteristics to an entire region like "mountainy", "plains", etc.
      The possibilities of what we can do in Leadwerks Engine 5 are intriguing.  Granted I don't have all the answers right now, but implementing a system like this would be a major step forward that unlocks an enormous world to explore.  What do you think?

    • By Haydenmango in Snowboarding Development Blog 6
      So I've been researching snowboarding lately to get an idea of what animations and mechanics I need to create for my game.  I have learned lots of interesting things since I've only seen snow once or twice in my entire life and have never even tried snowboarding or any other board sports (skateboarding, surfing, etc.) for that matter.
       
      Snowboarding tricks are quite interesting as they are mostly derived from skateboarding.  Snowboarding tricks pay homage to their equivalent skating tricks by sharing many concepts and names.  For example basic grabs in snowboarding share the same concepts and names as skateboarding: indy, mute, method, stalefish, nosegrab, and tailgrab.  Something interesting to note is in snowboarding you can grab Tindy or Tailfish but this is considered poor form since these grabs can't be done on a skateboard (due to the board not being attached to the skaters feet) and grabbing these areas is generally something a novice snowboarder does when failing or "half-assing" a normal grab.  Check out this diagram to see how grabs work -
       
       
      So, after reading lots of text descriptions for tricks I was still confused by what all these terms meant and how they were actually applied.  So my next step was to look up these tricks actually being done and I found some really cool videos showing off how to do various tricks.  This video in particular is the best reference material I've found as it contains nearly every trick back to back with labeled names and some tweaks -
       
      Sadly my rigged model doesn't handle leg animations with the snowboard that well so I can't animate as many tricks as I want to.  Regardless there will still be around 15 total grab/air tricks in the game.  Now it's time for me to stop procrastinating and start animating!  
×
×
  • Create New...