Jump to content

Single-file SPIR-V shaders for Vulkan

Josh

438 views

It is now possible to compile shaders into a single self-contained file that can loaded by any Vulkan program, but it's not obvious how this is done. After poking around for a while I found all the pieces I needed to put this together.

Compiling

First, you need to compile each shader stage from a source code file into a precompiled SPIR-V file. There are several tools available to do this, but I prefer GLSlangValidator because it supports the Google #include extension. Put your vertex shader code in a text file named "shader.vert" and your pixel shader code in a file called "shader.frag". Create a .bat file in the same directory with the following contents:

glslangValidator.exe "shader.vert" -V -o "vert.spv"
glslangValidator.exe "shader.frag" -V -o "frag.spv"

Run the bat file and two .spv files will be saved.

Linking

Now we want to combine our two files representing different shader stages into a single file. This is done with the link tool from Khronos. Add the following lines to your .bat file to compile the two .spv files into one. It will also delete the existing files to clean things up a little:

spirv-link "vert.spv" "frag.spv" -o "shader.spv"
del "vert.spv"
del "frag.spv"

This will save a single file named "shader.spv" that you can load as one shader module and use for different stages in Vulkan.

Here are the required executables and a .bat file:
BuildShader.zip

Parsing

If you always use vertex and fragment stages then there is no problem, but what if the combined .spv file contains other stages, or is missing a fragment stage? We can easily account for this with a minimal SPIR-V file parser. We're not going to include any big bloated libraries to do this because we only need some basic information about what stages are contained in the shader. Fortunately, the SPIR-V specification is pretty simple and it doesn't take much code to extract the information we want:

std::string entrypointname[6];

auto stream = ReadFile(L"Shaders/shader.spv");

// Parse SPIR-V data
Assert(stream->ReadInt() == 0x07230203);
int version = stream->ReadInt();
int genmagnum = stream->ReadInt();
int bound = stream->ReadInt();
int reserved = stream->ReadInt();

bool stages[6] = {false,false,false,false,false,false};		

// Instruction stream
while (stream->Ended() == false)
{
	int pos = stream->GetPos();
	unsigned int bytes = stream->ReadUInt();
	int opcode = LOWORD(bytes);
	int wordcount = HIWORD(bytes);
	if (opcode == 15)
	{
		int executionmodel = stream->ReadInt();
		Assert(executionmodel >= 0);
		if (executionmodel < 6)
		{
			stream->ReadInt(); // entry point
			stages[executionmodel] = true;
			entrypointname[executionmodel] = stream->ReadString();
		}
	}
	stream->Seek(pos + wordcount * 4);
}

This code even retrieves the entry point name for each stage, so you can be sure you are loadng the shader correctly.

Here are the different shader stages from the SPIR-V specification:

  • 0: Vertex
  • 1: TessellationControl
  • 2: TessellationEvaluation
  • 3: Geometry
  • 4: Fragment
  • 5: GLCompute

That's it! We now have a standard single-file shader format for Vulkan programs. Your code for creating these will look something like this:

VkShaderModule shadermodule;

// Create shader module
VkShaderModuleCreateInfo shaderCreateInfo = {};
shaderCreateInfo.sType = VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO;
shaderCreateInfo.codeSize = bank->GetSize();
shaderCreateInfo.pCode = reinterpret_cast<const uint32_t*>(bank->buf);
VkAssert(vkCreateShaderModule(device->device, &shaderCreateInfo, nullptr, &shadermodule));

// Create vertex stage info
VkPipelineShaderStageCreateInfo vertShaderStageInfo = {};
vertShaderStageInfo.sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO;
vertShaderStageInfo.stage = VK_SHADER_STAGE_VERTEX_BIT;
vertShaderStageInfo.module = shadermodule;
vertShaderStageInfo.pName = entrypointname[0].c_str();

VkPipelineShaderStageCreateInfo fragShaderStageInfo = {};
if (stages[4])
{
  // Create fragment stage info			
  fragShaderStageInfo.sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO;
  fragShaderStageInfo.stage = VK_SHADER_STAGE_FRAGMENT_BIT;
  fragShaderStageInfo.module = shadermodule;
  fragShaderStageInfo.pName = entrypointname[4].c_str();
}

// Create your graphics pipeline...

 



3 Comments


Recommended Comments

How will it work with custom shaders from the user? Suppose they will need to compile them.

Share this comment


Link to comment

I plan to add a tool for Visual Studio Code that will automatically call the required steps.

Then in the engine you can just call this:

auto shader = LoadShader("Shaders/myshader.spv");

 

  • Like 1

Share this comment


Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Blog Entries

    • By Josh in Josh's Dev Blog 8
      You might have seen this graphic comparing the size of the world in different games. I've played Fuel, and never reached the end of the world in that game. You can drive for a very long time on those roads.

      We want to use the new engine for realistic simulations of air and ground movements. At normal cruising altitude of a commercial airliner, the pilot has a view range of about 400 kilometers. The image below shows that area (800 x 800 km). You can see the areas of the biggest games ever fit neatly into the corner of just our visible area.

      The gray space above is not the total world size, it is just the area you can see at once from high altitude. The total world size is about 50 times bigger.
      This is what I am working on now.
    • By Josh in Josh's Dev Blog 26
      Gamers have always been fascinated with the idea of endless areas to roam.  It seems we are always artificially constrained within a small area to play in, and the possibility of an entire world outside those bounds is tantalizing.  The game FUEL captured this idea by presenting the player with an enormous world that took hours to drive across:
      In the past, I always implemented terrain with one big heightmap texture, which had a fixed size like 1024x1024, 2048x2048, etc.  However, our vegetation system, featured in the book Game Engine Gems 3, required a different approach.  There was far too many instances of grass, trees, and rocks to store them all in memory, and I wanted to do something really radical.  The solution was to create an algorithm that could instantly calculate all the vegetation instances in a given area.  The algorithm would always produce the same result, but the actual data would never be saved, it was just retrieved in the area where you needed it, when you needed it.  So with a few modifications, our vegetation system is already set up to generate infinite instances far into the distance.

      However, terrain is problematic.  Just because an area is too far away to see doesn't mean it should stop existing.  If we don't store the terrain in memory then how do we prevent far away objects from falling into the ground?  I don't like the idea of disabling far away physics because it makes things very complex for the end user.  There are definitely some tricks we can add like not updating far away AI agents, but I want everything to just work by default, to the best of my ability.
      It was during the development of the vegetation system that I realized the MISSING PIECE to this puzzle.  The secret is in the way collision works with vegetation.  When any object moves all the collidable vegetation instances around it are retrieved and collision is performed on this fetched data.  We can do the exact same thing with terrain   Imagine a log rolling across the terrain.  We could use an algorithm to generate all the triangles it potentially could collide with, like in the image below.

      You can probably imagine how it would be easy to lay out an infinite grid of flat squares around the player, wherever he is standing in the world.

      What if we only save heightmap data for the squares the user modifies in the editor?  They can't possibly modify the entire universe, so let's just save their changes and make the default terrain flat.  It won't be very interesting, but it will work, right?
      What if instead of being flat by default, there was a function we had that would procedurally calculate the terrain height at any point?  The input would be the XZ position in the world and the output would be a heightmap value.

      If we used this, then we would have an entire procedurally generated terrain combined with parts that the developer modifies by hand with the terrain tools.  Only the hand-modified parts would have to be saved to a series of files that could be named "mapname_x_x.patch", i.e. "magickingdom_54_72.patch".  These patches could be loaded from disk as needed, and deleted from memory when no longer in use.
      The real magic would be in developing an algorithm that could quickly generate a height value given an XZ position.  A random seed could be introduced to allow us to create an endless variety of procedural landscapes to explore.  Perhaps a large brush could even be used to assign characteristics to an entire region like "mountainy", "plains", etc.
      The possibilities of what we can do in Leadwerks Engine 5 are intriguing.  Granted I don't have all the answers right now, but implementing a system like this would be a major step forward that unlocks an enormous world to explore.  What do you think?

    • By Haydenmango in Snowboarding Development Blog 6
      So I've been researching snowboarding lately to get an idea of what animations and mechanics I need to create for my game.  I have learned lots of interesting things since I've only seen snow once or twice in my entire life and have never even tried snowboarding or any other board sports (skateboarding, surfing, etc.) for that matter.
       
      Snowboarding tricks are quite interesting as they are mostly derived from skateboarding.  Snowboarding tricks pay homage to their equivalent skating tricks by sharing many concepts and names.  For example basic grabs in snowboarding share the same concepts and names as skateboarding: indy, mute, method, stalefish, nosegrab, and tailgrab.  Something interesting to note is in snowboarding you can grab Tindy or Tailfish but this is considered poor form since these grabs can't be done on a skateboard (due to the board not being attached to the skaters feet) and grabbing these areas is generally something a novice snowboarder does when failing or "half-assing" a normal grab.  Check out this diagram to see how grabs work -
       
       
      So, after reading lots of text descriptions for tricks I was still confused by what all these terms meant and how they were actually applied.  So my next step was to look up these tricks actually being done and I found some really cool videos showing off how to do various tricks.  This video in particular is the best reference material I've found as it contains nearly every trick back to back with labeled names and some tweaks -
       
      Sadly my rigged model doesn't handle leg animations with the snowboard that well so I can't animate as many tricks as I want to.  Regardless there will still be around 15 total grab/air tricks in the game.  Now it's time for me to stop procrastinating and start animating!  
×
×
  • Create New...