Jump to content

Building a Network Test App with Leadwerks



Thought I could write my first Blog entry and report about my ventures into network programming with Leadwerks, ENet and Lua. Also, provide some source code. Okay, here we go:


What I wanted to do


My test app should be a client-server setup where clients can connect to the server, see each other's avatar, send chat lines and move around their avatar on a plane. Nothing fancy really, just to get into the topic.

The ENet network library (http://http://enet.bespin.org/) should be used, and network functions should be exposed from C to Leadwerk's lua engine (as I wanted to do the main 'game coding' in lua).

The clients are intended to run on Windows, while the server should first be run as console app on Win, might later be moved to a remote rented Linux server.



The Network Concept


For the first testing, I tried to make the server part very "light-weight", and leave many things to the client(s). Probably not the right thing to do for a full scale MMOG (mainly due to cheat concerns) - but, well, this test app is far from that anyway ;-)

So, the server will - aside from some administrative tasks - mainly receive messages from a client and distribute to the others. As for the communication, a rather simple protocol layer was created on top of ENet (i.e., a definition of message formats and identifiers).

Note that not everything a client sends will be distributed - it would first be checked for (some very basic) consistency.





Network Messages


Seven network messages were defined, some for client->server or server->client only, some for both ways. The messages are wrapped into ENet's 'ENetPacket' and sent, to be evaluated by the receiver.

  • MSG_TYPE_ID_REQ(a clients requests a unique identification number from the server)
  • MSG_TYPE_NAME (a client sends its chat name)
  • MSG_TYPE_POS (a client or the server sends info about position/rotation of an avatar)
  • MSG_TYPE_CHAT (a client or the server sends info about a chat line)
  • MSG_TYPE_CLIENT_ON (the server sends info about a new client online - also used to inform a new client about existing ones)
  • MSG_TYPE_CLIENT_OFF (the server sends info about a client leaving)
  • MSG_TYPE_CLIENT_ID (the server sends a unique identification number to the client that requsted one)

The message formats used would look like so (complete source available below):

struct MSG_POS // Sending a client pos and rot info
UINT16 ClientID;
float fPosX, fPosY, fPosZ;
float fRotX, fRotY, fRotZ;



The Server


This is a win console C app, which should - with some tweaking - compile and run on Linux also. Actually, I started out with doing it on Ubuntu on a VMWare Player, but moved to Win console later on. Just for the ease of use.

The server will detect when a new client connected (using ENet functionality), and wait for the initial 'request for ID' message. After assignment, it informs the other clients about the new one (and vice versa). On server side, only the unique ID and the client's chat name are stored.

When receiving e.g. a "Chatline" message, the server would check if the sender's client ID and the message type both are valid, then distribute the message to all clients. Upon disconnection of a client (again using ENet functionality), remaining clients would be informed also.



The Client


Networking here is done as a two-stage concept: Extending the Leadwerks App with some networking C functions and exposing some of them to lua (where the main 'game programming' is to happen). Note that Leadwerks also comes with the ENet headers and libs, so it is not necessary to include it here. You can just use it in your C code.

As for the connection between C and lua, two possibilities must be considered: Calling a (new) network function from lua and calling back from C into lua when e.g. a certain message arrived.


Exposing a new function to lua would look like this (returning the connected state of the network):


lua_register(Interpreter::L, "NW_IsConnected", NW_IsConnected);

extern "C" int NW_IsConnected(lua_State* L)
lua_pushboolean(L, gl_Networking.m_bConnected);
return 1;


Preparing a callback from C into lua looked like so (this is a callback when a new client is reported by the server). Note that (obviously) the global function "NW_Callback_Connected" must be present in the lua code, to be called.


void Callback_Connected(unsigned short int iID)
lua_getglobal(Leadwerks::Interpreter::L, "NW_Callback_Connected")
if (!lua_isfunction(Leadwerks::Interpreter::L, -1))
{ lua_pop(Leadwerks::Interpreter::L, 1); return; }

lua_pushnumber(Leadwerks::Interpreter::L, iID);

/* do the call (1 argument, 0 result) */
if (lua_pcall(Leadwerks::Interpreter::L, 1, 0, 0) != 0)
 printf("NW_Callback_Connected ERROR: %s\n", lua_tostring(Leadwerks::Interpreter::L, -1));


For the test app, several functions were exposed to lua from the added network C code, and also several possible callbacks are provided. To avoid any multi-threading issues, the function "NW_Update()" must periodically called from lua (e.g. in WorldUpdate). Within there, callbacks are then triggered.


Callbacks, invoked from network (when NW_Update() called)

NW_Callback_Connected(id) // called when we are connected to the network
NW_Callback_ClientJoined(id) // called when a client joined the network
NW_Callback_ClientLeft(id) // called when a client left the network
NW_Callback_ClientPosRot(id,Px,Py,Pz,Rx,Ry,Rz) // called when a client pos/rot message arrived
NW_Callback_ClientName(id,name) // called when a client sent its name
NW_Callback_ClientChat(id,chatline) // called when a client sent a chatline


Functions, to be called from lua

NW_Connect() // do this once, e.g. in Script:Start()
NW_IsConnected() // call this to see if server connection is established
NW_Update() // do this regularly, e.g. in Script:UpdateWorld()
NW_SendPosRot(Px,Py,Pz,Rx,Ry,Rz) // when pos/rot changes, send all e.g. 200ms
NW_SendName(name) // send your client name once (15 chars max);
NW_SendChatline(chatline) // send chatlines (255 chars max) to other clients


Within lua, it is then the client's task to respond to e.g. a 'NW_Callback_ClientJoined' message. Like, create a new avatar and move it around.

The client is sending its own position every 200ms, to be distributed further by the server. Within the client, only very few measures have been taken to compensate for network lag. It will interpolate incoming positions for remote avatars (moving them with constant speed), but does not really apply such high-sophisticated methods like movement prediction and whatnot.


To move one's own avatar, use WASD. To submit a chat line, hit enter (submit with enter again, cancel with esc). Note that as for now, a connection attempt is only done at startup. So the server must be started before any client. Also, 'localhost' as server IP is currently hardcoded.



Final Words


While I did dive into ENet before, this was my rather first time with all that lua mumbo-jumbo. Actually, after getting used to it, writing scripts in lua is quite fast (relaxing, even). Think I will stick to the idea of doing some basic stuff in C, but adding game logic in lua. Now on to that movement prediction et al!





A short video, showing the whole mess in action:

Link to executables (server+client): http://www.mikoweb.eu/tmp/LWEN_Networking_Exe.zip (10MB)

Link to Source (VS2013+LW): http://www.mikoweb.eu/tmp/LWEN_Networking_Source.zip (25MB)





The test app (here, 3 clients and the server running on Win)




Recommended Comments

Uhm, I've got the standard here, so can't really check. But as some C compiling is involved (also for the client), I'd assume the full version be needed...

Share this comment

Link to comment

I would love this if it also worked on linux,

as I'm trying to develop for the linux community first.

Share this comment

Link to comment

Will this work with the indie edition or only if you have the full standard edition?

If indie edition can call c functions from lua code you can compile your own external client library somewhere else.

Share this comment

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Blog Entries

    • By Josh in Josh's Dev Blog 1
      I started to implement quads for tessellation, and at that point the shader system reached the point of being unmanageable. Rendering an object to a shadow map and to a color buffer are two different processes that require two different shaders. Turbo introduces an early Z-pass which can use another shader, and if variance shadow maps are not in use this can be a different shader from the shadow shader. Rendering with tessellation requires another set of shaders, with one different set for each primitive type (isolines, triangles, and quads). And then each one of these needs a masked and opaque option, if alpha discard is enabled.
      All in all, there are currently 48 different shaders a material could use based on what is currently being drawn. This is unmanageable.
      To handle this I am introducing the concept of a "shader family". This is a JSON file that lists all possible permutations of a shader. Instead of setting lots of different shaders in a material, you just set the shader family one:
      shaderFamily: "PBR.json" Or in code:
      material->SetShaderFamily(LoadShaderFamily("PBR.json")); The shader family file is a big JSON structure that contains all the different shader modules for each different rendering configuration: Here are the partial contents of my PBR.json file:
      { "turboShaderFamily" : { "OPAQUE": { "default": { "base": { "vertex": "Shaders/PBR.vert.spv", "fragment": "Shaders/PBR.frag.spv" }, "depthPass": { "vertex": "Shaders/Depthpass.vert.spv" }, "shadow": { "vertex": "Shaders/Shadow.vert.spv" } }, "isolines": { "base": { "vertex": "Shaders/PBR_Tess.vert.spv", "tessellationControl": "Shaders/Isolines.tesc.spv", "tessellationEvaluation": "Shaders/Isolines.tese.spv", "fragment": "Shaders/PBR_Tess.frag.spv" }, "shadow": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "Shaders/DepthPass_Isolines.tesc.spv", "tessellationEvaluation": "Shaders/DepthPass_Isolines.tese.spv" }, "depthPass": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "DepthPass_Isolines.tesc.spv", "tessellationEvaluation": "DepthPass_Isolines.tese.spv" } }, "triangles": { "base": { "vertex": "Shaders/PBR_Tess.vert.spv", "tessellationControl": "Shaders/Triangles.tesc.spv", "tessellationEvaluation": "Shaders/Triangles.tese.spv", "fragment": "Shaders/PBR_Tess.frag.spv" }, "shadow": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "Shaders/DepthPass_Triangles.tesc.spv", "tessellationEvaluation": "Shaders/DepthPass_Triangles.tese.spv" }, "depthPass": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "DepthPass_Triangles.tesc.spv", "tessellationEvaluation": "DepthPass_Triangles.tese.spv" } }, "quads": { "base": { "vertex": "Shaders/PBR_Tess.vert.spv", "tessellationControl": "Shaders/Quads.tesc.spv", "tessellationEvaluation": "Shaders/Quads.tese.spv", "fragment": "Shaders/PBR_Tess.frag.spv" }, "shadow": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "Shaders/DepthPass_Quads.tesc.spv", "tessellationEvaluation": "Shaders/DepthPass_Quads.tese.spv" }, "depthPass": { "vertex": "Shaders/DepthPass_Tess.vert.spv", "tessellationControl": "DepthPass_Quads.tesc.spv", "tessellationEvaluation": "DepthPass_Quads.tese.spv" } } } } } A shader family file can indicate a root to inherit values from. The Blinn-Phong shader family pulls settings from the PBR file and just switches some of the fragment shader values.
      { "turboShaderFamily" : { "root": "PBR.json", "OPAQUE": { "default": { "base": { "fragment": "Shaders/Blinn-Phong.frag.spv" } }, "isolines": { "base": { "fragment": "Shaders/Blinn-Phong_Tess.frag.spv" } }, "triangles": { "base": { "fragment": "Shaders/Blinn-Phong_Tess.frag.spv" } }, "quads": { "base": { "fragment": "Shaders/Blinn-Phong_Tess.frag.spv" } } } } } If you want to implement a custom shader, this is more work because you have to define all your changes for each possible shader variation. But once that is done, you have a new shader that will work with all of these different settings, which in the end is easier. I considered making a more complex inheritance / cascading schema but I think eliminating ambiguity is the most important goal in this and that should override any concern about the verbosity of these files. After all, I only plan on providing a couple of these files and you aren't going to need any more unless you are doing a lot of custom shaders. And if you are, this is the best solution for you anyways.
      Consequently, the baseShader, depthShader, etc. values in the material file definition are going away. Leadwerks .mat files will always use the Blinn-Phong shader family, and there is no way to change this without creating a material file in the new JSON material format.
      The shader class is no longer derived from the Asset class because it doesn't correspond to a single file. Instead, it is just a dumb container. A ShaderModule class derived from the Asset class has been added, and this does correspond with a single .spv file. But you, the user, won't really need to deal with any of this.
      The result of this is that one material will work with tessellation enabled or disabled, quad, triangle, or line meshes, and animated meshes. I also added an optional parameter in the CreatePlane(), CreateBox(), and CreateQuadSphere() commands that will create these primitives out of quads instead of triangles. The main reason for supporting quad meshes is that the tessellation is cleaner when quads are used. (Note that Vulkan still displays quads in wireframe mode as if they are triangles. I think the renderer probably converts them to normal triangles after the tessellation stage.)

      I also was able to implement PN Quads, which is a quad version of the Bezier curve that PN Triangles add to tessellation.

      Basically all the complexity is being packed into the shader family file so that these decisions only have to be made once instead of thousands of times for each different material.
    • By Josh in Josh's Dev Blog 0
      I'm back from I/ITSEC. This conference is basically like the military's version of GDC. VR applications built with Leadwerks took up about half of Northrop Grumman's booth. There were many interesting discussions about new technology and I received a very warm reception. I feel very positive about our new technology going forward.

      I am currently reworking the text field widget script to work with our persistent 2D objects. This is long and boring but needs to be done. Not much else to say right now.
    • By Josh in Josh's Dev Blog 4
      Here are some screenshots showing more complex interface items scaled at different resolutions. First, here is the interface at 100% scaling:

      And here is the same interface at the same screen resolution, with the DPI scaling turned up to 150%:

      The code to control this is sort of complex, and I don't care. GUI resolution independence is a complicated thing, so the goal should be to create a system that does what it is supposed to do reliably, not to make complicated things simpler at the expense of functionality.
      function widget:Draw(x,y,width,height) local scale = self.gui:GetScale() self.primitives[1].size = iVec2(self.size.x, self.size.y - self.tabsize.y * scale) self.primitives[2].size = iVec2(self.size.x, self.size.y - self.tabsize.y * scale) --Tabs local n local tabpos = 0 for n = 1, #self.items do local tw = self:TabWidth(n) * scale if n * 3 > #self.primitives - 2 then self:AddRect(iVec2(tabpos,0), iVec2(tw, self.tabsize.y * scale), self.bordercolor, false, self.itemcornerradius * scale) self:AddRect(iVec2(tabpos+1,1), iVec2(tw, self.tabsize.y * scale) - iVec2(2 * scale,-1 * scale), self.backgroundcolor, false, self.itemcornerradius * scale) self:AddTextRect(self.items[n].text, iVec2(tabpos,0), iVec2(tw, self.tabsize.y*scale), self.textcolor, TEXT_CENTER + TEXT_MIDDLE) end if self:SelectedItem() == n then self.primitives[2 + (n - 1) * 3 + 1].position = iVec2(tabpos, 0) self.primitives[2 + (n - 1) * 3 + 1].size = iVec2(tw, self.tabsize.y * scale) + iVec2(0,2) self.primitives[2 + (n - 1) * 3 + 2].position = iVec2(tabpos + 1, 1) self.primitives[2 + (n - 1) * 3 + 2].color = self.selectedtabcolor self.primitives[2 + (n - 1) * 3 + 2].size = iVec2(tw, self.tabsize.y * scale) - iVec2(2,-1) self.primitives[2 + (n - 1) * 3 + 3].color = self.hoveredtextcolor self.primitives[2 + (n - 1) * 3 + 1].position = iVec2(tabpos,0) self.primitives[2 + (n - 1) * 3 + 2].position = iVec2(tabpos + 1, 1) self.primitives[2 + (n - 1) * 3 + 3].position = iVec2(tabpos,0) else self.primitives[2 + (n - 1) * 3 + 1].size = iVec2(tw, self.tabsize.y * scale) self.primitives[2 + (n - 1) * 3 + 2].color = self.tabcolor self.primitives[2 + (n - 1) * 3 + 2].size = iVec2(tw, self.tabsize.y * scale) - iVec2(2,2) if n == self.hovereditem then self.primitives[2 + (n - 1) * 3 + 3].color = self.hoveredtextcolor else self.primitives[2 + (n - 1) * 3 + 3].color = self.textcolor end self.primitives[2 + (n - 1) * 3 + 1].position = iVec2(tabpos,2) self.primitives[2 + (n - 1) * 3 + 2].position = iVec2(tabpos + 1, 3) self.primitives[2 + (n - 1) * 3 + 3].position = iVec2(tabpos,2) end self.primitives[2 + (n - 1) * 3 + 3].text = self.items[n].text tabpos = tabpos + tw - 2 end end  
  • Create New...