Jump to content

GMF2, Plugins, and Modding

Josh

641 views

I wanted to get a working example of the plugin system together. One thing led to another, and...well, this is going to be a very different blog.

GMF2

The first thing I had to do is figure out a way for plugins to communicate with the main program. I considered using a structure they both share in memory, but those always inevitably get out of sync when either the structure changes or there is a small difference between the compilers used for the program and DLL. This scared me so I went with a standard file format to feed the data from the plugin to the main program.

GLTF is a pretty good format but has some problems that makes me look for something else to use in our model loader plugins.

  • It's text-based and loads slower.
  • It's extremely complicated. There are 3-4 different versions of the file format and many many options to split it across multiple files, binary, text, or binary-to-text encoding.
  • It's meant for web content, not PC games.
  • Tons of missing functionality is added through a weird plugin system. For example, DDS is supported through a plugin, but a backup PNG has to be included in case the loaded doesn't support the extension.

The GMF file format was used in Leadwerks. It's a custom fast-loading chunk-based binary file format. GMF2 is a simpler flat binary format updated with some extra features:

  • Vertices are stored in a single array ready to load straight into GPU memory.
  • Vertex displacement is supported.
  • Compressed bitangents.
  • Quad and line meshes are supported.
  • LOD
  • Materials and textures can be packed into the format or loaded from external files.
  • PBR and Blinn-Phong materials
  • Mesh bounding boxes are supported so they don't have to be calculated at load time. This means the vertex array never has to be iterated through, making large meshes load faster.

I am not sure yet if GMF2 will be used for actual files or if it is just going to be an internal data format for plugins to use. GLTF will continue to be supported, but the format is too much of a mess to use for plugin data.

Here's a cool five-minute logo:

gmf2.png.e4f8a73b44c3737353080d4dcb82b16c.png

The format looks something like this:

char[4] ID "GMF2"
int version 200
int root //ID of the root entity
int texture_count //number of textures
int textures_pos //file position for texture array
int materials_count //number of materials
int materials_pos //file position for materials
int nodes_count //number of nodes
int nodes_pos //file position for nodes

As you can see, it is really easy to read and really easy to write. There's enough complexity in this already. I'm not bored. I don't need to introduce unnecessary complexity into the design just so I can show off. There are real problems that need to be solved and making a "tricky" file format is not one of them.

In Leadwerks 2, we had a bit of code called the "GMF SDK". This was written in BlitzMax, and allowed construction of a GMF file with easy commands. I've created new C++ code to create GMF2 files:

//Create the file
GMFFile* file = new GMFFile;

//Create a model
node = new GMFNode(file, GMF_TYPE_MODEL);

//Add an LOD level
GMFLOD* lod = node->AddLOD();

//Add a mesh
GMFMesh* mesh = lod->AddMesh(3); //triangle mesh

//Add a vertex
mesh->AddVertex(0,0,0, 0,0,1, 0,0, 0,0, 255,255,255,255);
mesh->AddVertex(0,0,1, 0,0,1, 0,0, 0,0, 255,255,255,255);
mesh->AddVertex(0,1,1, 0,0,1, 0,0, 0,0, 255,255,255,255);

//Add a triangle
mesh->AddIndice(0);
mesh->AddIndice(1);
mesh->AddIndice(2);

Once your GMFFile is constructed you can save it into memory with one command. The Turbo Plugin SDK is a little more low-level than the engine, so it includes a MemWriter class to help with this, since the engine Stream class is not present.

As a test I am writing a Quake 3 MD3 import plugin and will provide the project and source as an example of how to use the Turbo Plugin SDK.

bitterman.jpg.93ad860beedacf9cd9033e5a3584bc94.jpg

Packages

The ZIP virtual file system from Leadwerks is being expanded and formalized. You can load a Package object to add new virtual files to your project. These will also be loadable from the editor, so you can add new packages to a project, and the files will appear in the asset browser and file dialogs. (Package files are read-only.) Plugins will allow packages to be loaded from file formats besides ZIP, like Quake WADs or Half-Life GCF files. Notice we keep all our loaded items in variables or arrays because we don't want them to get auto-deleted.

//Load Quake 3 plugins
auto pk3reader = LoadPlugin("Plugins/PK3.dll");
auto md3loader = LoadPlugin("Plugins/MD3.dll");
auto bsploader = LoadPlugin("Plugins/Q3BSP.dll");

Next we load the game package files straight from the Quake game directory. This is just like the package system from Leadwerks.

//Load Quake 3 game packages
std::wstring q3apath = L"C:/Program Files (x86)/Steam/steamapps/common/Quake 3 Arena/baseq3";
auto dir = LoadDir(q3apath);
std::vector<shared_ptr<Package> > q3apackages;
for (auto file : dir)
{
	if (Lower(ExtractExt(file)) == L"pk3")
	{
		auto pak = LoadPackage(q3apath + L"/" + file);
		if (pak) q3apackages.push_back(pak);
	}
}

Now we can start loading content directly from the game.

//Load up some game content from Quake!
auto head = LoadModel(world, "models/players/bitterman/head.md3");
auto upper = LoadModel(world, "models/players/bitterman/upper.md3");
auto lower = LoadModel(world, "models/players/bitterman/lower.md3");
auto scene = LoadScene(world, "maps/q3ctf2.bsp");

Modding

I have a soft spot for modding because that is what originally got me into computer programming and game development. I was part of the team that made "Checkered Flag: Gold Cup" which was a spin-off on the wonderful Quake Rally mod:

I expect in the new editor you will be able to browse through game files just as if they were uncompressed in your project file, so the new editor can act as a modding tool, for those who are so inclined. It's going to be interesting to see what people do with this. We can configure the new editor to run a launch script that handles map compiling and then launches the game. All the pieces are there to make the new editor a tool for modding games, like Discreet's old Gmax experiment.

gmaxsplash.jpg.3afd806e4ce5cbd58dc1d0e03c696b98.jpg

I am going to provide official support for Quake 3 because all the file formats are pretty easy to load, and because gmax can export to MD3 and it would be fun to load Gmax models. Other games can be supported by adding plugins.

So here are some of the things the new plugin system will allow:

  • Load content directly from other games and use it in your own game. I don't recommend using copyrighted game assets for commercial projects, but you could make a "mod" that replaces the engine and requires the player to own the original game.
  • You could probably safely use content from any Valve games and release a commercial game on Steam.
  • Use the editor as a tool to make Quake or other maps.
  • Add plugin support for new file formats.

This might all be a silly waste of time, but it's a way to get the plugin system working, and if we can make something flexible enough to build Quake maps, well that is a good test of the robustness of the system.

 

  • Like 4


12 Comments


Recommended Comments

25 minutes ago, aiaf said:

We would be able to load gltf models? 

GLTF is already available, right now. It will continue to be supported.

Share this comment


Link to comment

I shared this in the Quake modding discord server since you mentioned quake 3 mapping a few times. Hopefully it drives some more interest into turbo :)

  • Like 1
  • Haha 1

Share this comment


Link to comment
40 minutes ago, Mr_SquarePeg said:

I shared this in the Quake modding discord server since you mentioned quake 3 mapping a few times. Hopefully it drives some more interest into turbo :)

Quake is what got me into computer programming in the first place, and now I work with NASA. I am very fond of that whole scene.

Share this comment


Link to comment

Notice there is actually no one talking about a game-ready model format, because I guess that is boring or something. OpenGEX, Collada, and FBX are "exchange formats" for moving data between modeling programs, and I guess this USD format is too. GLTF is a scene (not model) format for web graphics, which is why it uses JPG and PNG textures. No one even considers making something for normal 3D games.

What do you think the advantage of this USD format is? GLTF is nice because at least you can load all the Sketchfab content up easily.

Share this comment


Link to comment

USD is used to transfer, well, anything you can do in a 3D modeling program. So, it doesn't matter if you use 3DS Max, Maya, or even Blender. If you can import/export out USD, you can bring the entire scene (models, materials, animations (including bones, etc.), lights ... everything, in one shot (if you want). Unity already supports USD. Unreal is getting ready to (if they don't already). And most 3D modeling programs do, too. Apparently it has advantages over GLTF. 

USD Primer

Share this comment


Link to comment

What is USD?

Pipelines capable of producing computer graphics films and games typically generate, store, and transmit great quantities of 3D data, which we call "scene description".  Each of many cooperating applications in the pipeline  (modeling, shading, animation, lighting, fx, rendering) typically has its own special form of scene description tailored to the specific needs and workflows of the application, and neither readable nor editable by any other application.   Universal Scene Description (USD) is the first publicly available software that addresses the need to robustly and scalably interchange and augment arbitrary 3D scenes that may be composed from many elemental assets.  

From: https://graphics.pixar.com/usd/docs/index.html

Share this comment


Link to comment

@Monkey Frog Studio You've convinced me to make GMF2 our default file format. If another standard is coming along that replaced GLTF, yet still does not give us fast-loading models, we need to invent our own. We will still provide direct loading and optional conversion of GLTF files, and maybe also this new format, but some of these huge CAD files we are working with would take ages to load if we have to sort through all the data. I think we will be able to show some loading speed comparisons between GLTF and GMF2, and the difference will be massive.

I am going to introduce the file format at GDC 2020, along with tools for working with it.

Share this comment


Link to comment

 

Quote

You've convinced me to make GMF2 our default file format.

Well, firstly, I didn't ask for it to be the DEFAULT file format, but recommended it as a format that could be supported by Leadwerks. And that was at least a year ago, too. Time moves on. Things progress. And as you worked with GLTF you found things you both liked and did not like. Due to the things you didn't like about GLTF (some of them mentioned in your post here), I then recommended USD as a possibility. Some are saying there are advantages USD has over GLTF. So, is there really any harm in looking at it? If you find that GLTF is great for what you want, then good. You've already done the work (or most of it), right? If you find USD is a viable option and provides benefits, then wouldn't you be happy to know about it?
 

Quote

If another standard is coming along that replaced GLTF, yet still does not give us fast-loading models, we need to invent our own.

And when you do that, you then need a way for artists to get their content into your engine. That means exporting from their 3D package of choice into your custom file format. And that means someone creating and maintaining these export plugins for the various packages, such as Maya, 3DS Max, C4D, MODO, and Blender. 

Quote

We will still provide direct loading and optional conversion of GLTF files, and maybe also this new format,

Glad to hear it.

Quote

but some of these huge CAD files we are working with would take ages to load if we have to sort through all the data.

That's a standard issue with CAD files in a non-CAD application.

Share this comment


Link to comment

Sorry, I just meant that in a matter-of-fact way.

The way the plugin system is set up, the data is always run through the GMF2 data format. So we can have a plugin that loads a GLTF or USD or other files and gives it to the engine in memory as a GMF2 file, or we can make it save the GMF2 for faster loading. Bottom line is if we have an import plugin for any format, we also automatically have a converter. Plus the file format is so simple I think we will see more community-made exporters than we did with GMF1, which had a "tricky" chunk structure.

Plus, we have the GMF2 SDK, which makes it easy to just create some data structures and then the code saves a file for you. So GMF2 and its SDK is a big funnel we can easily pipe all kinds of data into and either save it or load it from memory.

I think loading speed is a significant consideration we should be paying attention to. This seems like a really basic consideration we should be designing file formats around, for all games. I guess I need to get some definite numbers on what mesh data load times are like under realistic use cases.

Share this comment


Link to comment

 That all sounds pretty awesome, actually. Thanks for taking the time to fill me in. Selfishly, all I care about, as an artist, is getting my assets into the engine as easily as possible. The less hoops I have to jump through, the better. :)

  • Haha 1

Share this comment


Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Blog Entries

    • By tipforeveryone in tipforeveryone's Blog 5
      II. Implement advanced actions
      Previous tutorial guided you to create your own FPS Character Controller with basic actions: Move, Look.
      This tutorial will help you add some advanced actions like Jump, Run, Crouch, and Lean
      *Note: Below steps will use code from previous tutorial (CharacterController.lua)
      1. Jump / Run / Crouch
      Firstly, build obtacle with space under it to test crouching, the space height shoud be at least 120cm (leadwerks editor will show you this height in other viewports) You can not crouch and pass space under 120cm, this was hardcoded


      Next, modify the code.
      Add new variables under --Character movement variables comment. Note that putting script variables at the top of script will allow you to modify character stat easier and faster than finding variables deep inside script's functions
      --Character movement variables Script.playerSpeed = 2 --higer = faster Script.playerRunSpeedMultipier = 2 --if this variable < 1, you can make slower movement Script.playerJumpForce = 7 --This defines how high you can jump Add new funtion to smooth movement of character components function Script:SmoothPosition(position,entity,rate) local smoothX = Math:Curve(position.x,entity:GetPosition().x,rate) local smoothY = Math:Curve(position.y,entity:GetPosition().y,rate) local smoothZ = Math:Curve(position.z,entity:GetPosition().z,rate) entity:SetPosition(smoothX,smoothY,smoothZ) end Replace code in Character_Movement() function by this code function Script:Character_Movement() local playerSpeed, playerJumpForce, playerCrouch --Press Space bar to Jump if window:KeyHit(Key.Space) then playerJumpForce = self.playerJumpForce else playerJumpForce = 0 end --Hold Ctrl key to Crouch if window:KeyDown(Key.ControlKey) then playerCrouch = true else if self.playerHeadBlocked then playerCrouch = true else playerCrouch = false end end --Hold Shift key + AWSD to Run if window:KeyDown(Key.Shift) then playerSpeed = self.playerSpeed * self.playerRunSpeedMultipier else playerSpeed = self.playerSpeed end local playerMove = ((window:KeyDown(Key.W) and 1 or 0) - (window:KeyDown(Key.S) and 1 or 0)) * playerSpeed local playerStrafe = ((window:KeyDown(Key.D)and 1 or 0) - (window:KeyDown(Key.A) and 1 or 0)) * playerSpeed --Using local playerSpeed varialbe instead of self.playerSpeed in the old code local playerTurn = self.playerNeck:GetRotation(true).y self.playerBase:SetInput(playerTurn,playerMove,playerStrafe,playerJumpForce,playerCrouch) end Replace code in Bind_Character_Components_Together() function too function Script:Bind_Character_Components_Together() --Must use this reposition process because playerBase is not playerNeck's parent, they are indipendent. local basePos = self.playerBase:GetPosition(true) local height if window:KeyDown(Key.ControlKey) then height = basePos.y + (self.playerHeight - self.neckLength) / 2 --You can adjust this variable to get desired crouch height for playerNeck (and playerEyes too) else if self.playerHeadBlocked then height = basePos.y + (self.playerHeight - self.neckLength) / 2 else height = basePos.y + self.playerHeight - self.neckLength end end self:SmoothPosition(Vec3(basePos.x,height,basePos.z),self.playerNeck,10) end Add a new function to script, this will keep player crouching if something block above character when release Ctrl key, prevent from being pushed around by obtacle when standing. function Script:Check_Head_Block() --We can use a raycast to check if something block character head when crouching local pickInfo = PickInfo() local point1 = self.playerNeck:GetPosition(true) local point2 = self.playerBase:GetPosition(true) + Vec3(0,self.playerHeight,0) if world:Pick(point1,point2,pickInfo,0.3,true) then self.playerHeadBlocked = true else self.playerHeadBlocked = false end end Update Script:UpdateWorld() function with new Check_Head_Block() function function Script:UpdateWorld() self:Bind_Character_Components_Together() self:Check_Head_Block() self:Character_Look() self:Character_Movement() end Now, you can crouch under obtacle, release Ctrl Key and you are still crouching. Continue moving for auto standing. 2. Leaning
      In Character_Movement() function, add this code after Crouching code
      --Hold Q/E to lean Left/Right local leanDistance = 0.025 --leanDistance should be < 0.025 or camera will pass through wall when you get too close if window:KeyDown(Key.Q) then self.playerEyes:Move(-leanDistance,-0.01,0) end if window:KeyDown(Key.E) then self.playerEyes:Move(leanDistance,-0.01,0) end And add this line at the bottom of Bind_Character_Components_Together() function self:SmoothPosition(Vec3(0,self.neckLength,0),self.playerEyes,10) You can lean now, so smooth. Final script file was attached to this entry. You can apply it to a pivot point (PlayerControl).
      CharacterController.lua
    • By 💎Yue💎 in The shock absorbers 2
      Coming to the end of my prototype of a 3d game, and with background music by Hanz Zimmer ( Time ). I saw my progress in many aspects, always something to learn, always something to improve, I didn't intend to make a game, that has never been the goal. 
      Rather, the effort and dedication immeasurably, was to improve on something learned. And here I was with the powerful leadwerks engine, where his greatest power lay in making everything very easy.  
      About the project
      The prototype is very simple, a third person character goes through a stage, an orbital camera that follows him with many interesting things when scripting.  A character who runs, walks, ducks, jumps, and suffers damage when he falls from different heights. 
      The interesting thing about all this is that just like when you're little and learn to write, repetition is key to learning to program, understand concepts and improve.  So as I've always said, you learn to program by programming, although I sincerely think that lua script is not programming, but the experts say it is, so I go into that elitism of those of us who think we are programmers. 
      The final part of the project consists of creating a death animation, this will be activated when the player falls from a high part and separates the legs (that is very cruel). 
      But that feeling of improvement is the same that I feel when in the mornings when I have a coffee I solve a riddle of the newspaper, but I don't know when I'm going to stop, it turns out and it happens that technology advances very fast and this continues and continues without stopping. Starting with LE 5, and the only thing that can happen is that I die or that my old computer doesn't work anymore. 
      Translated with www.DeepL.com/Translator





    • By 💎Yue💎 in The shock absorbers 1
      It's interesting that when you become an expert on something, you're not sparing any effort to see how something works, but rather you're focusing on creating something. And so everything becomes easier.
      At this point of learning there is a glimpse of a low idea of creating a game, but the secret of all this is to keep it simple and to be very clear that a game is a game, and not an exact simulation of the real world. For example anyone who has a low idea of the red planet, will understand no matter the colors of the scene that is a terrain of Mars, even if it is not very real what is transmitted, a game, that's just it.
      At this point I already have an astronaut character who runs from one place to another on a very large 4096 x 4046 terrain that would surely take a long walk. My previous prototype projects involve a vehicle, but I didn't get the best implementation prospect in that time and I always found performance problems in my machine, something that isn't happening with the character controller for a third person player. 
      As always, I think I'm a scavenger looking for game resources, that's where this community exposes links to websites with interesting hd textures, and one or another model searched on the net, but what I've greatly improved is learning to write code, I have a better workflow, writing Lua code focused on the paradigm of object programming.



      Something interesting is the system of putting rocks, all very nice from the point of implementing them. And it works very well with the character controller if you put collision in cube form.
      I've been thinking about implementing a car system, I think it would be necessary in such a large terrain, but I think it's not the time, my previous experience, involves deterioration in performance and something I think is the physics of the car with respect to the terrain and rocks that in the previous project involve deterioration in the fps. Although if you implement a car would have an option would be to remove the rocks, but I prefer not to have a car and if you have rocks. 
       
       
       
       
×
×
  • Create New...