Can Leadwerks do what I need it to do? Procedural Meshes and Collisions. Multithreading.

[Slight0]

I've built a prototype voxel engine in Unity using threads to do mesh generation and voxel data generation. My main goal is generation speed. I've run into issues with Unity's MeshCollider as I cannot generate it in a thread due to Unity's single-thread nature and it is a time consuming task to create each MeshCollider.

 

Here is my thread with more details if you want specifics.

 

Anyway, I noticed Leadwerks has a Surface class which I presume will let me control meshes from code. What about collisions? Chunk generation speed is my priority so how fast is the creation of collision meshes? More importantly, can I offload this process in another thread so that the main thread is unhindered by collision mesh creation?

 

Any info is appreciated.

[DerRidda]

I haven't looked into these things but one thing I can guarantee is that you will need the Standard Edition which supports C++ for that. The Indie Edition is limited to Lua only which doesn't really do multithreading (yet). Look at this for further reference: http://www.leadwerks.com/werkspace/page/documentation/_/command-reference/thread/

[Slight0]

Ah that's disappointing. Strange there isn't a full evaluation for the C++ version to get a hang of the workflow; I'm not big on gambling on game engines. Still, I'd be curious if what I mentioned is possible.

[Josh]

It's not really possible to make a time-limited C++ trial.

 

I think the collision mesh creation would be trivial relative to the other calculations. It's not easy to split the physics off onto a separate thread, but I always assumed it wouldn't be an issue for Leadwerks with this type of game. Can't really say for sure since I've never tried it though.

 

How many polygons do you reckon an average chunk consists of?

[Slight0]

Appreciate the response.

 

It's not really possible to make a time-limited C++ trial.

 

Perhaps a free version of the engine is in order then? One with restricted advanced features like Unity3d does? Afterall, what does limiting engine access really get you anyway?

 

I think the collision mesh creation would be trivial relative to the other calculations. It's not easy to split the physics off onto a separate thread, but I always assumed it wouldn't be an issue for Leadwerks with this type of game. Can't really say for sure since I've never tried it though.

 

How many polygons do you reckon an average chunk consists of?

 

As long as the collision mesh, I assume the class is btBvhTriangleMeshShape, doesn't access other objects like the world object or what have you, whilst the Bounding Volume Hierarchy optimization and other caching is taking place, threading should be trivial. I don't need to simulate physics in another thread, just to create the object and then pass it to the main thread.

 

I'm benchmarking with about 1536 triangles per chunk, but the actual geometry will probably yield about 400-800 triangles per chunk. This is without any sort of mesh optimization applied.

[Olby]

Perhaps a free version of the engine is in order then? One with restricted advanced features like Unity3d does? Afterall, what does limiting engine access really get you anyway?

 

There is a free demo on Steam but like DerRidda mentioned its LUA only. This means you don't get access to Mutex and Threading libraries, however the rest of the engine is fully accessible (unlike the other game engine you mentioned). C++ version adds templates for VS (or CB on Linux) option when creating a new project.

[Brutile]

I've implemented a voxel engine in Leadwerks, and I can say that it can be done, but I've found that the time it takes to generate the collision is quite long. I've had a lot better luck in Unity. From what I've found, Leadwerks is faster to generate the mesh, but Unity is faster in the collision, but not that much slower in gerenating the mesh, so Unity is a better option IMO for a voxel engine. Hope this helps

[xtreampb]

I'm not completely familiar with this subject, but isn't the speed at which the collision is generated dependent upon the physics engine used? If so, I know that others have replaced the standard physics engine that comes with leadwerks with one that was more suited to their needs.

[Slight0]

There is a free demo on Steam but like DerRidda mentioned its LUA only. This means you don't get access to Mutex and Threading libraries, however the rest of the engine is fully accessible (unlike the other game engine you mentioned). C++ version adds templates for VS (or CB on Linux) option when creating a new project.

 

Yes, I'm aware. Threading is paramount to what I'm doing. Secondly, I'm a programmer and, though lua is nice for quick prototypes or games like garry's mod and world of warcraft, it's not a good game development language. I'd cite Natural Selection II as evidence of that. Not that it's a bad game, but a real pain to modify (hint: a lot of the game core is written in lua). No types, no good IDEs, debugging is challenging, no classes, it's just too flexible to keep organized with.

 

Unity has a 30-day full featured trial that you can email them and have extended for another 30 days, after that it's still usable, just without the advanced features that you really don't need for core development. I prefer Unity's limited model to this model because I can do everything I need to do, then buy the engine when I'm nearing a public release and work in the relevant advanced graphics/animation features.

 

Leadwerks does look promising with a superior editor toolset to say the least, but switching from lua to C++ is non-trival.

 

I've implemented a voxel engine in Leadwerks, and I can say that it can be done, but I've found that the time it takes to generate the collision is quite long. I've had a lot better luck in Unity. From what I've found, Leadwerks is faster to generate the mesh, but Unity is faster in the collision, but not that much slower in gerenating the mesh, so Unity is a better option IMO for a voxel engine. Hope this helps

 

Would you be able to tell me the timing it takes you to generate the collision mesh for a single chunk in your engine as well as the number of triangles in said collision mesh?

 

I'm not completely familiar with this subject, but isn't the speed at which the collision is generated dependent upon the physics engine used? If so, I know that others have replaced the standard physics engine that comes with leadwerks with one that was more suited to their needs.

 

Right, but replacing the physics system is something I can do with other engines like Unity. It's a bit of work and if I have to do it, I might as well do it in the engine I'm already familiar with.

[Brutile]

Would you be able to tell me the timing it takes you to generate the collision mesh for a single chunk in your engine as well as the number of triangles in said collision mesh?

 

Collider(86 faces): 8ms

Collider(135 faces): 14ms

Collider(146 faces): 15ms

Collider(31 facecs): 3ms

Collider(107 faces): 10ms

Collider(142 faces): 14ms

Collider(70 faces): 8ms

 

Those are some times to generate colliders. It ranges from around 3ms to 20ms.

The time taken to generate the mesh ranges from 0ms to 1ms.

Chunks are 16x16x16

[Einlander]

A 200x200 grid plane takes around 20seconds to generate a collision, 100x100 is from .4 to 4 seconds.

[Slight0]

Collider(86 faces): 8ms

Collider(135 faces): 14ms

Collider(146 faces): 15ms

Collider(31 facecs): 3ms

Collider(107 faces): 10ms

Collider(142 faces): 14ms

Collider(70 faces): 8ms

 

Those are some times to generate colliders. It ranges from around 3ms to 20ms.

The time taken to generate the mesh ranges from 0ms to 1ms.

Chunks are 16x16x16

 

Thanks and holy ****! I was expecting more from bullet physics. In the thread I posted in the OP, a chunk with about 1500 triangles takes about 6ms to generate a mesh collider for. Mesh generation is always under 1ms, usually like 300 microseconds.

 

Are you using RigidBodys or static colliders? Make sure you did it with static colliders as those should be faster.

 

I don't think my CPU is that amazing either; Intel Core i7-2670QM @ 2.20GHz 4 cores (8 threads).

 

This does not bode well because I'm thinking about integrating Bullet physics into Unity.

 

A 200x200 grid plane takes around 20seconds to generate a collision, 100x100 is from .4 to 4 seconds.

 

Interesting, but without the exact number of triangles there's little one can do with that information.

[Rick]

With Lua you can use coroutines to simulate threads. Basically you can split out the task of generating all these things over a longer period of time without blocking the main game.

 

If you aren't familiar with coroutines, they maintain state inside a function that you can yield out of and resume back into. So basically if you were looping through some list you can yield out after a set amount of time is reached and when you resume back in you'll start where it left off.

 

Of course Unity has coroutines also so maybe you deemed them not good for this?

[Crazycarpet]

With Lua you can use coroutines to simulate threads. Basically you can split out the task of generating all these things over a longer period of time without blocking the main game.

 

If you want you could use LuaJit's FFI api in combination with Lua C API to multithread Lua using Mutex. You shouldn't need much C code, only abit to start the threads and their main functionallity.

 

I've never tried it but a friend of mine on steam linked me this:

https://github.com/ColonelThirtyTwo/LuaJIT-Threads

 

I currently use one I made that uses coroutines, but its extremely slow and soon I'll be writing a real multithreading library.

[Rick]

Wouldn't you run into the same issues with LE that Unity has in that creating entities on a thread other than the main thread won't work? I seem to recall others running into this issue with LE.

[Einlander]

 

 

Thanks and holy ****! I was expecting more from bullet physics. In the thread I posted in the OP, a chunk with about 1500 triangles takes about 6ms to generate a mesh collider for. Mesh generation is always under 1ms, usually like 300 microseconds.

 

Are you using RigidBodys or static colliders? Make sure you did it with static colliders as those should be faster.

 

I don't think my CPU is that amazing either; Intel Core i7-2670QM @ 2.20GHz 4 cores (8 threads).

 

This does not bode well because I'm thinking about integrating Bullet physics into Unity.

 

 

 

Interesting, but without the exact number of triangles there's little one can do with that information.

 

Leadwerks only deals with triangles so the exact number would be 400^2

 

Also Leadwerks uses Newton physics.

[Josh]

I was curious if there was some way to handle this so I asked on the forum here:

http://newtondynamics.com/forum/viewtopic.php?f=9&t=8741

 

It looks like the Newton user mesh is best for this because you can just provide some callbacks without having to deal with building new physics geometry at all. However, it's a rather advanced feature.

[Crazycarpet]

I was curious if there was some way to handle this so I asked on the forum here:

http://newtondynamics.com/forum/viewtopic.php

 

I think you missed /lots/ of the link, lol.

[Josh]

I just removed all that pesky nonsense at the end to clean it up a bit.

[Josh]

I'll add this if people really intend to build Minecraft-like games, since those are cool right now. A community lib for this would be pretty nice if anyone feels like it.

 

Basically you'll have to supply some C callbacks that handle the collision data dynamically. Voxel geometry is really ideal for this because it can be calculated really quickly for small collision areas:

NEWTON_API NewtonCollision* NewtonCreateUserMeshCollision (const NewtonWorld* const newtonWorld, const dFloat* const minBox,
const dFloat* const maxBox, void* const userData, NewtonUserMeshCollisionCollideCallback collideCallback,
NewtonUserMeshCollisionRayHitCallback rayHitCallback, NewtonUserMeshCollisionDestroyCallback destroyCallback,
NewtonUserMeshCollisionGetCollisionInfo getInfoCallback, NewtonUserMeshCollisionAABBTest getLocalAABBCallback,
NewtonUserMeshCollisionGetFacesInAABB facesInAABBCallback, NewtonOnUserCollisionSerializationCallback serializeCallback, int shapeID);

 

So as a result the physics generation time will fall to zero, because there will be no data stored. You will just calculate the results dynamically in small areas where physics are occuring, which will be very fast for the small amount of polygons I think your collisions will actually hit.

[Josh]

Here's the code. THis is a start:

       void UserGridCollision::CollideCallbackDescrete(NewtonUserMeshCollisionCollideDesc* const collideDesc)
  {
// this is your same original function
       }

// this is fo continue collsion
  void UserGridCollision::CollideCallbackContinue(NewtonUserMeshCollisionCollideDesc* const collideDesc, const void* const continueHandle)
  {
     // get the aabb
     dVector p0(collideDesc->m_boxP0[0], collideDesc->m_boxP0[1], collideDesc->m_boxP0[2]);
     dVector p1(collideDesc->m_boxP1[0], collideDesc->m_boxP1[1], collideDesc->m_boxP1[2]);

     // translate the box to target 
     dVector transtionDist (collideDesc->m_boxDistanceTravel[0], collideDesc->m_boxDistanceTravel[1], collideDesc->m_boxDistanceTravel[2]);
     dVector q0(p0 + transtionDist);
     dVector q1(p1 + transtionDist);

     p0.m_x = dMin (p0.m_x, q0.m_x); 
     p0.m_y = dMin (p0.m_y, q0.m_y); 
     p0.m_z = dMin (p0.m_z, q0.m_z); 

     p1.m_x = dMax (p1.m_x, q1.m_x); 
     p1.m_y = dMax (p1.m_y, q1.m_y); 
     p1.m_z = dMax (p1.m_z, q1.m_z); 

     collideDesc->m_faceCount = 0;
     collideDesc->m_vertexStrideInBytes = sizeof(dVector);
     collideDesc->m_faceIndexCount = &m_faceIndices[0];
     collideDesc->m_faceVertexIndex = &m_indexArray[0];
     collideDesc->m_vertex = &m_collisionVertex[0][0];

//      int collisionVertexIndex = 0;
//      dVector* polygon;

     // min voxel position
     Vector3i minBlock((int)p0.m_x, (int)p0.m_y, (int)p0.m_z);

     // max voxel position
     Vector3i maxBlock((int)ceil(p1.m_x), (int)ceil(p1.m_y), (int)ceil(p1.m_z));


     int stack;
     Vector3i stackPool[64][2];

     stackPool[0][0] = minBlock;
     stackPool[0][1] = maxBlock;
     stack = 1;

     while (stack)
     {
        stack --;
        Vector3i minBlock (stackPool[stack][0]);
        Vector3i maxBlock (stackPool[stack][1]);

        //see this cell is visible to the collision path
        dVector p0 (minBlock.m_x, minBlock.m_y, minBlock.m_z, 0.0f);
        dVector p1 (maxBlock.m_x, maxBlock.m_y, maxBlock.m_z, 0.0f);
        if (NewtonUserMeshCollisionContinueOveralapTest (collideDesc, continueHandle, &p0[0], &p1[0])) {

           if (minBlock.m_x == maxBlock.m_x) {
              if (minBlock.m_y == maxBlock.m_y) {
                 if (minBlock.m_z == maxBlock.m_z) {
                    //here we found a visible cell, submit this face


                 } else {
                    // split alone z axis 
                    int z = (minBlock.m_z + maxBlock.m_z) / 2;
                    stackPool[stack][0] = minBlock;
                    stackPool[stack][1] = Vector3i(minBlock.m_x, minBlock.m_y, z);
                    stack ++;
                    dAssert (stack < 64);

                    stackPool[stack][0] = Vector3i(minBlock.m_x, minBlock.m_y, z + 1);
                    stackPool[stack][1] = maxBlock;
                    stack ++;
                    dAssert (stack < 64);
                 }
              } else {
                 if (minBlock.m_z == maxBlock.m_z) {
                    // split alone y only
                    int y = (minBlock.m_y + maxBlock.m_y) / 2;
                    stackPool[stack][0] = minBlock;
                    stackPool[stack][1] = Vector3i(minBlock.m_x, y, minBlock.m_z);
                    stack ++;
                    dAssert (stack < 64);

                    stackPool[stack][0] = Vector3i(minBlock.m_x, y + 1, minBlock.m_z);
                    stackPool[stack][1] = maxBlock;
                    stack ++;
                    dAssert (stack < 64);

                 } else {
                    // slit alone y and z
                    int y = (minBlock.m_y + maxBlock.m_y) / 2;
                    int z = (minBlock.m_z + maxBlock.m_z) / 2;

                    stackPool[stack][0] = minBlock;
                    stackPool[stack][1] = Vector3i(minBlock.m_x, y, z);
                    stack ++;
                    dAssert (stack < 64);

                    stackPool[stack][0] = Vector3i(minBlock.m_x, y + 1, z);
                    stackPool[stack][1] = Vector3i(minBlock.m_x, maxBlock.m_y, z);
                    stack ++;
                    dAssert (stack < 64);


                    stackPool[stack][0] = Vector3i(minBlock.m_x, minBlock.m_y, z);
                    stackPool[stack][1] = Vector3i(minBlock.m_x, y, maxBlock.m_z);
                    stack ++;
                    dAssert (stack < 64);

                    stackPool[stack][0] = Vector3i(minBlock.m_x, y + 1, z + 1);
                    stackPool[stack][1] = maxBlock;
                    stack ++;
                    dAssert (stack < 64);
                 }
              }
           } else {
              // x's are different 
              if (minBlock.m_y == maxBlock.m_y) {
                 if (minBlock.m_z == maxBlock.m_z) {
                    // split alone x axis 
                    int x = (minBlock.m_x + maxBlock.m_x) / 2;
                    stackPool[stack][0] = minBlock;
                    stackPool[stack][1] = Vector3i(x, minBlock.m_y, minBlock.m_z);
                    stack ++;
                    dAssert (stack < 64);

                    stackPool[stack][0] = Vector3i(x + 1, minBlock.m_y, minBlock.m_z);
                    stackPool[stack][1] = maxBlock;
                    stack ++;
                    dAssert (stack < 64);
                 } else {
                    // split alone x and z axis 
                    int x = (minBlock.m_y + maxBlock.m_y) / 2;
                    int z = (minBlock.m_z + maxBlock.m_z) / 2;

                    stackPool[stack][0] = minBlock;
                    stackPool[stack][1] = Vector3i(x, minBlock.m_y, z);
                    stack ++;
                    dAssert (stack < 64);

                    stackPool[stack][0] = Vector3i(x + 1, minBlock.m_y, z);
                    stackPool[stack][1] = Vector3i(minBlock.m_x, maxBlock.m_y, z);
                    stack ++;
                    dAssert (stack < 64);

                    stackPool[stack][0] = Vector3i(minBlock.m_x, minBlock.m_y, z);
                    stackPool[stack][1] = Vector3i(x, minBlock.m_y, maxBlock.m_z);
                    stack ++;
                    dAssert (stack < 64);

                    stackPool[stack][0] = Vector3i(x + 1, minBlock.m_y, z + 1);
                    stackPool[stack][1] = maxBlock;
                    stack ++;
                    dAssert (stack < 64);
                 }
              } else {
                 if (minBlock.m_z == maxBlock.m_z) {
                    // split alone x, and y
                    int x = (minBlock.m_x + maxBlock.m_x) / 2;
                    int y = (minBlock.m_y + maxBlock.m_y) / 2;
                    //....


                 } else {
                    // slit alone x, y and z
                    int x = (minBlock.m_x + maxBlock.m_x) / 2;
                    int y = (minBlock.m_y + maxBlock.m_y) / 2;
                    int z = (minBlock.m_z + maxBlock.m_z) / 2;
                    //...
// here you push the 8 quadrant on the stack

                 }
              }

           }
        }
     }
  }


  static void UserGridCollision::CollideCallback(NewtonUserMeshCollisionCollideDesc* const collideDesc, const void* const continueHandle)
  {
     UserGridCollision* const me = (UserGridCollision*)collideDesc->m_userData;
     if (continueHandle) {
        me->CollideCallbackContinue(collideDesc, continueHandle);
     } else {
        me->CollideCallbackDescrete(collideDesc);
     }
  }