Jump to content

Upload bug


Slimwaffle
 Share

Recommended Posts

So I found what I think is a bug. My game that I uploaded to steam and is for sale on steam forces the customer to download the full size of the game whenever I upload a new build. It shouldn't be doing this because steam should scan the game files and determine which ones have changed. Then I came across something interesting in the steamworks documentation saying that if the game uses encrypted or compressed files/ folders steams scanning won't work. And I know that when I publish with leadwerks standalone that it compresses the game files/folders to a zip file.

Is there a way to make leadwerks not compress /encrypt these files. Because customers downloading the full game each time is excessive.

Here is the info from steamworks;

Building Efficient Depots for SteamPipe

The old Steam content system would patch updates on a file level, which meant that if a single byte in a file changed, the entire new file would be downloaded by all users. This was especially inefficient if the game used pack files, which are collections of game content files in a single big file. Pack files can easily exceed 1 GB, so updates often led to unnecessarily large downloads. A common way to avoid these large downloads was to add new pack files that overrode content of already shipped pack files. That worked for updates, but it hurt new users long-term, since they ended up downloading unused, already-patched content.

The new content system fixes this problem by splitting each file into roughly 1-MB chunks. Each chunk is then compressed and encrypted before being distributed by the Steam content system. If the game content has large redundant parts, these chunks are reused and the user only has to download each repeated chunk once. However, the real strength of this system is building efficient update patches. While the system is building a patch, the new content is scanned for already known chunks. If it finds them, it reuses them. This means if you change or inject a few bytes in a big file, the user only has to download the changes.

This works well in most cases, but there are still a few pitfalls that need to be avoided when designing the content layout of a game. Do not compress or encrypt your game data. This is already done for in-flight downloads and retail discs by the Steam content system. If you do it too, you will reduce the effectiveness of delta patching. If you package multiple data files in a single pack file, make sure that with each re-packaging, no unnecessary changes are made. One problematic practice is including the full name of the original source files on disk, because the names may change, depending on the build machine. Another bad pattern is including build time stamps for each file. If possible, always add new content to the end of your pack files and keep the order of existing files. Also, keep your pack file’s metadata (offset and sizes to individual files) in one place and don’t intersperse it with content files. Use a binary diff’ing tool like BeyondCompare to look at two builds of your pack files to make sure that hundreds of unwanted changes don’t show up.

If you follow these rules you will minimize patch sizes and only new content will need to be downloaded. Your customers will thank you for that and you will be able to increase the quality of your product by shipping more updates.

Link to comment
Share on other sites

The publish step takes all the files in your project and puts them into a zip file. You are saying you do not want to compress your project into a zip file. So just upload your game to Steam without the publish step in Leadwerks.

  • Like 1

My job is to make tools you love, with the features you want, and performance you can't live without.

Link to comment
Share on other sites

ok because when I generate a build using visual studio to a folder on my desktop it just adds 3 files.

Waffles.exp, Waffles (Application) and Waffles.lib

So then I can add the rest of my project to this folder? And I just remove all my scripts?

Because they are all in lua and the cpp files just launch them using the Interpreter.

I am just unsure of which files you are referring to when you say Source code files.

 

Link to comment
Share on other sites

You need the following file types:

  • map
  • lua
  • tex
  • mat
  • mdl
  • ttf
  • wav
  • ogg
  • shader
  • pfb
  • phy

And of course your executable (Waffles.exe).

No other files need to be included.

  • Like 1

My job is to make tools you love, with the features you want, and performance you can't live without.

Link to comment
Share on other sites

So I tried this and it worked great for fixing the patch sizes. But doing this way offers like no file protection. Steam say they encrypt files. But I can't see any evidence of that. Is there a way to keep my file protection and fix the patch sizes?

So I had a look into how other games on other engines do it. And I think I came up with a middle ground that would work. But I'm not sure if its possible to do on my end.

So I noticed other games create package files. These act almost exactly the same way as the data zip file created when I use the publish standalone feature in leadwerks.

What I would like to do if possible is configure the leadwerks publish feature so that instead of creating a single encrypted zip file for all files in the project. It creates one for each of the sub folders in the main folder. For example, (My project Folder is called Waffles) configure it so that it creates a zip for; Addons folder, Prefabs, Scripts and so on and so forth until it has everything contained in a subfolder in an encrypted/compressed zip of the same name.

That way when patching; Steam says that it can't scan any folder that is encrypted or compressed. But it should only replace the ones that have changed in size. Because it replaces folders and files that have changed in size.

Is this possible? And would it even work?

Link to comment
Share on other sites

Sure, the specific function usage is here: https://www.leadwerks.com/community/topic/17491-loading-from-the-encrypted-data-file/?do=findComment&comment=113622

Some more code here: https://www.leadwerks.com/community/topic/16566-reading-a-text-file-from-a-package/

And you can find another thread or two if you just search the forums for the word password.

  • Like 1
  • Thanks 1
Link to comment
Share on other sites

Thank you soo much guys. Absolute legends.

Using this system of compressing sub-folders works perfectly.

Now when I make a small change to the game it will only upload a few mb.

Also using this system greatly reduces the upload time to steam after the initial build is uploaded.

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...