Zip format for mods

If mod size is small, this doesn’t matter. If mod size is large, you’ll might step into fragmentation problem as well. How are you going to mmap() files within archive or random access some specific data file? And why do you insist on no compression, while when proper alorithm is choosen (like LZO) there’s no speed penalny, but bonus (from faster I/O)?

Actually, you should abandon archive idea and leave such ‘optimizations’ to filesystem.

Self-modified files should ALWAYS go to user’s home directory. Distributed mods should have version ID (and all the files verified by MD5/SHA via package manager). Data files are loaded when used, so no such tracking is required.

Why was it requested? My guess is to obsuscate contents and hide files from users.

Have you ever seen any open-source program using archives for regular work? What kind of program and what kind of data it was?

What size you consider “big”? For example new town mods (which can be considered big) are ~ 20 Mb. I don’t think that files of such size will have severe fragmentation. But such mods also contain hundreds(!) of files. If they will be scattered around hard drive it will take a while to read all of them.

Because that’s what I got from profiler - ~90% of data loading is unpacking files from .lod. Which uses deflate just like .zip’s. Using another archive format has another downside: availability. Zip is the only format available on any platform out of the box.

And how you will differ self-modified files from work-in-progress mod that may be already available in repository?
Zip already has CRC checksum calculated by archiver so all VCMI needs to do is to compare it to value from previous start - I have no interest in writing file-by-file verification.

Hiding files in format that all OS’s recognize as archive? Not sure how this supposed to work?
And I can’t remember exact quote but it was something like “ease of use, opposed t ohuge number of files”.

OpenArena (and probably all other quake3-based projects) - use .zip’s with changed extension to store data.
OpenTTD - uses some custom archive format(!) for game data
I can search for more if you’re interested.

+1 for zipped mods as an option. They are just easier to handle in certain scenarios (already mentioned above).

You’re going to load ALL of these into memory? That hurts performance for sure. If you have hundreds of files they should be loaded on demand, not on startup (i.e. map them into memory, let the OS do the job including caching and swapping out when necessary as a bonus).

Well… first of all, do not load all the files at once, they are probably not required in the moment (player starts with one town without buildings or army, sees nothing on adventure map - unless loading save, which always takes longer, but still there is a chance that low-rank buildings or army won’t be accessed). And compare, for the maximum-entropy file:

$ cd /dev/shm
$ dd if=/dev/urandom of=test bs=1M count=50
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 7.20605 s, 7.3 MB/s
$ cp test test2
$ gzip test2
$ lzop test
$ ls -la
total 153708
drwxrwxrwt  2 root  root       100 May 27 03:30 .
drwxr-xr-x 24 root  root     94208 May 25 09:39 ..
-rw-r--r--  1 gotar users 52428800 May 27 03:29 test
-rw-r--r--  1 gotar users 52436824 May 27 03:29 test2.gz
-rw-r--r--  1 gotar users 52431246 May 27 03:29 test.lzo
$ time zcat test2.gz > /dev/null
zcat test2.gz > /dev/null  0.79s user 0.03s system 99% cpu 0.828 total
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; )  0.25s user 0.07s system 95% cpu 0.334 total
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; )  0.27s user 0.04s system 93% cpu 0.333 total
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; )  0.26s user 0.06s system 94% cpu 0.339 total
$ time zcat test2.gz > /dev/null
zcat test2.gz > /dev/null  0.73s user 0.02s system 98% cpu 0.759 total
$ time zcat test2.gz > /dev/null
zcat test2.gz > /dev/null  0.72s user 0.01s system 98% cpu 0.739 total
$ time zcat test2.gz > /dev/null
zcat test2.gz > /dev/null  0.76s user 0.00s system 99% cpu 0.761 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; )  0.69s user 0.07s system 99% cpu 0.763 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; )  0.77s user 0.03s system 97% cpu 0.818 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; )  0.68s user 0.04s system 96% cpu 0.748 total

and something that compresses well:

$ dd if=/dev/zero of=test bs=1M count=50
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0912174 s, 575 MB/s
$ cp test test2
$ gzip test2
$ lzop test
$ ls -la
total 51556
drwxrwxrwt  2 root  root       100 May 27 03:41 .
drwxr-xr-x 24 root  root     94208 May 25 09:39 ..
-rw-r--r--  1 gotar users 52428800 May 27 03:40 test
-rw-r--r--  1 gotar users    50919 May 27 03:40 test2.gz
-rw-r--r--  1 gotar users   210446 May 27 03:40 test.lzo
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; )  0.52s user 0.00s system 99% cpu 0.522 total
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; )  0.51s user 0.01s system 99% cpu 0.523 total
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; )  0.45s user 0.00s system 98% cpu 0.459 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; )  0.63s user 0.00s system 98% cpu 0.640 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; )  0.62s user 0.01s system 98% cpu 0.640 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; )  0.62s user 0.01s system 99% cpu 0.633 total

Everything between would be 20%-100% faster.

Every WIP should use ‘disable all the caches’ knob, KISS (developer’s host). Every mod released into repository MUST HAVE unique version ID. Uploading a mod with the same version MUST NOT be allowed. That’s why there were 0.xx releases, that’s why you can use RCS-generated ID for nightly builds (like git hash).

Remember to check mtime, ctime and file size as well, as CRC may be ‘tuned’ to match previous. It was not intended to provide signature, but integrity, Application should trust version strings.

Tell me, what’s the point of any reloading if the only thing that got changed was some graphics? It will be replaced anyway, no need to alert user.

97% of people (might be 99,5% of gamers) can’t repack this.

Original HoMM3 was written 14 years ago for Win95 with strict dataset. Write loading code well, using modern techniques, you’ll not suffer from such issues.

GL textures need to be loaded anyway. As for OpenTTD - isn’t that original data format? So - not the case, not your rationale.

Not exactly. VCMI will not load whole archive in memory but this most likely will be done by OS. Caching of one file works much better than caching of whole directory. But main point is that one (unfragmented) archive will be located in continuous area on hard drive while multiple files are usually not grouped that well.

Consider this to be like a “hint” to OS - that all/some of content in that archive is likely to be requested soon.

Of course. But then player opens fort window. Which requires loading ~1 Mb of data (background + 7-8 creature animations). Or starts battle. That hurts speed quite a lot - I actually started profiling tests solely because of slow loading of fort screen.

Such bursts of data requests are the biggest problem. Thus keeping them in memory (cached by OS for example) helps a lot with performance. Actually right now we have small cache to keep unpacked data - that was my solution to slow loading.

Current solution is even simpler - there is no need to set such flag at all.
Yes. Mods in repo MUST have unique versions at least for proper updating but I don’t see any need to block changes in mods on client side. Only exception is multiplayer but in this case comparing mod version + zip checksum should be enough.

…and CRC is not extremely reliable anyway. Yes I know. My intent was to use CRC as indication that archive has changed since previous launch - not as way to detect changes at all. If user want to break his game by hacking checksum - let him. Client-server architecture won’t let him to make any changes to multiplayer anyway.

In this case 90% of people won’t manage to navigate to mod directory or edit mod anyway.

AFAIK most of archive managers support editing files in archive (by unpacking them into temporary directory and watching for changes). So in terms of data accessibility this won’t change much.

What’s the difference? Town needs 8 icons, ~40 animated buildings (~10 frames each), 14 animated creatures (~50 frames in animation) with 2 icons, 16 heroes with 4 icons each. This can be reduced by using spritesheets but this will make mods more difficult to make.

Formats like mng/apng can also help here but they have extremely bad support in image editors.

OpenArena packs everything in archives. Including maps, config files, models, scripts, sounds. Quite similar to VCMI I think.
OpenTTD - so should we switch back to .lod archives? Or more precisely - to 3 arhive formats used for different types of data?

Scourched3D - downloading mod gave me file in some unknown format. May be legacy of original game though. But no plain directory.

Wesnoth - plain directory. But also - “too slow” complains from devs (not related to mods but to game data structure in general)

So far haven’t found any other open-source games that have mod support - usually content made by artists goes straight to main game VCS. And VCS’s are not designed to keep archives.

Definitely one could add .

What I meant is that you cannot mmap() the exact required parts of the zip compressed archive and extract thousands of single files ON DEMAND in efficient way without writing your own caching subsystem.

Without compression you can use zip indices to directly pinpoint some part of archive, but this will break miserably if any compression gets enabled (consider users yelling that they’ve repackaged some archive and it doesn’t work - zip is zip, isn’t it?).

I know what you mean, but I think you don’t understand me:) How are you going to open such archive and not load it’s contents? Enforcing store-only?
If you map zillion of files, they are NOT loaded from disk at all until used. So at the game start you map EVERYTHING required (you can skip parts that are not - e.g. some castles not existing on the specific map) and that’s all. The only I/Os required are that from readdir and that can be the issue for thousands of files. On the other hand, you solution requires loading hundreds of MBs that might be not used at all (consider user with 10 mods - new castles, creatures etc. playing some regular map).

The golden mean would be to concatenate files that are loaded among (e.g. all the data for a single creature).

Data is loaded just once (until system runs out of memory and flushes file-backed maps, but in low RAM situations this is good to reload some files instead regular swapping), so when game starts from begin it doesn’t matter (each creature or building gets loaded as is developed), only when loading from savegame it might get laggy in the first action (and it will anyway somewhere during the loading or the game).

That’s exactly why mmap was invented. Otherwise you’ll sooner or later run into OOM conditions. However, file mapping requires plain data (may be archived, might not be compressed *]). I don’t know nor haven’t found a library handling compressed-file-backed mmap, but according to contemporary OS-engineering this is filesystem’s job (like per-file compression, e.g. in BTRFS).

Starting the game you have to load how much, 5% of this? Next building and creatures require loading one at a time when bought. You don’t load 5/10 frames of course, so you shoud have 40+14+16 archives (or sprites, doesn’t matter from I/O point of view). Not a single archive with all these data, as some buildings or creatures might not get bought at all.

To be honest - I have no idea. I hope you can do:) some tests on real data considering my post and you’ll choose well. There are many projects disregarding RAM usage and startup time due to broken data management, and it’s too late to switch (for VCMI it’s going to be even harder due to 3rd party mods). If sb asked me I’d focus on incremental loading, possibly file-backed using OS mechanisms and disregarding compression (leave it to filesystem), but supporting it by grouping data into objects loaded among (i.e. mmaped raw data in per-object archive).

  • just to make it clear - no compression means no compression, including data-specific (PNG). In order to use raw data they must be raw, not initialized and loaded well-known images, as that’s the moment they will be read from disk and will occupy memory. I might guess only, that this is the main reason for archives like lod to be created by game developers. Summarizing (IMHO):
  • per-file data compression (image format, like PNG): too much processing, breaks mmap, RAM starvation; better to distribute raw,
  • single file archive: good as long as it’s not compressed (any compression breaks mmap), so zip might confuse users: enforce no compression by any means, like switching zip to tar,
  • if mmap won’t be used and data is well compressible, switch zip to LZO (saves I/Os, very fast decompression).

Maybe it’s better to create separate tool for proper packaging (including data format conversion)?

Insert my 5 coins:
Windows already has mechanism to make folders “compressed”. So place on disk in Windows already can be optimized without adding some new functions.
As for Linux, I don’t know if such mechanisms exist there.

PS For example, i’m using SSD as my hard disk, so speed of loading files into memory is not harming me.
Imagine around 160-180 mods, all zipped. It will consume all memory and processor.
I think it’s more wise to implement needed game features at first, and then play with mods sizes. :smiley:

U used to download GIGABYTES of mods and maps for various games. Typical map for 3D game now takes 30-50 megabytes alone and people pick them by dozens. I think you underestimate capabilities of modern computers :stuck_out_tongue: No need to optimize a game that is 14 years old, especially not in terms of file size.

It may be an issue on portable devices, but:

  • Each of them has different hardware and operating system, they also evolve rapidly.
  • People may live without 20 new towns on their tablet for a while from now. Before such amonut of playable content emerges, hardware will advance by a few generations, anyway.

Why you think of mmap as the only way to load data? It is completely possible to load one specific file from zip without any significant memory overhead.

I already did those. And more than once.

Like Wesnoth which uses system exactly like the one you’re describing?

Not exactly true. Centralized repository also means that there is single center from which players receive mods. And we can make all necessary adjustments before allowing mod to be uploaded to repo.

Or keep it simple enough so this tool is not needed?

System I’m proposing is quite similar to one used by some of commercial games. And I’m talking about AAA-grade projects where speed is essential factor.

Keyword: “me”. Don’t forget that most of users don’t have SSD


BTRFS supports per-file compression flag using LZO or zlib. However, random access to compressed files introduces some overhead, that’s why it’s better to have more library files instead one big archive.

But having 1 GB of RAM used might harm. Especially when playing at work, where you need fast switching just after boss-key press;)

Not file size, but loading procedure. Pick some 300 MB files out of hard drive, measure time, check buffers/cache. This 14 yo game used lod archives, replacing them careless with ‘something made easy’ might easily kill modern hardware, especially when multitasking with some Fx, Office or dozens of other desktop apps. If you want to have enjoyable game, you need to run smoothly on decent hardware. The cheapest is storage - both desktops and mobiles (HDD or microSDHC card), you don’t need to save it. But you should save:

  1. I/Os, as this storage is slow (except SSD) and not scalable (multitasking kills it with 2 apps),
  2. CPU power, as their performance has stalled - unless you make use of many cores or GPU acceleration (the way current computers evolve since a few years),
  3. RAM - which might get cheap, but is still hardware limited (and eaten by everything around),
  4. power! - remember, every I/O or CPU drains you battery (laptop/mobiles) and despite the current weather here in Poland I’d prefer not to get burns from my terminal:)

OpenGL apps might not care about 100 MBs of textures as they are processed by GPU, but until you go multithreaded and accelerated, don’t scare off modders please. Forget about new hardware, it requires appropriate code and definitely won’t solve performance problems itself (before graphene comes into play or some other brand new technology).

Because it’s the only OS-assisted way.

You need to actually load it - this is overhead. Unless you have written your own cache and can load such files on demand, ONLY when requested and not already there - if so, then this discussion is meaningful (just point me out the code, please).

Last time I’ve seen wesnoth data there were PNGs (not raws), which must be loaded and processed to be usable - the same way you’ll have to cope with data. Whether they are in separate files or single archive is only tiny piece of optimizations I’ve described (not to mention I’ve suggested grouping entire objects per file).

There ain’t no such thing as a free lunch. The cost of data processing is always equal. If you don’t do this once, you’ll have to do this every other time. The question is: is that acceptable? Wesnoth carries around 400 MB of data and is slow. You’ve mentioned 20 MB mod, and this might be acceptable. But then you DO talk about performance, so is it?

lod’s are almost identical to zip’s. Both use deflate compression, both keep file index (albeit in different location). H3 uses two lod archives to store all game data. Why using zip instead of lod’s will have any performance inpact?

What’s the difference? You still need to load data from hard drive or from memory-mapped file into video memory (instead of ram).

And why (indirect) usage of system cache is not OS-assisted? Besides - mmap is unix-only function. This means that other platforms like Win would need custom solutions -> more code to maintain.

Some files are loaded on start and stored till the end of the game - this is true (e.g. data in client/Graphics). But most data will be loaded only when needed - see client/CDefHandler and client/CBitmapHandler for examples.

So far I don’t see any impact on loading time that comes from mods. By this I mean that time needed to load game without mods is roughly same as loading game with mods. I think that this can be called as acceptable.

It’s you who said lods were loading slow (OK, due to compression). Difference is lods had fixed size, while mods can be much larger (in total).

You need to load files into video RAM, you don’t need to read files into virtual memory until accessed. So the difference is about what can be archived/improved/made better in this particular usage pattern.

Really? There’s no such thing as OS-assisted buffer (not cache!) lazy preloading - there’s only small prefetch at block device layer or explicit reading (which definitely needs some code to work in non-idle time). File-backed memory might be easy flushed and restored from disk (instead swapping out). Dumb prefetching a file would give you some random contents (in order of archive), not the files you might need in near future. Sorry, but when to use or not use mmap are essentials I thought you’re aware of…

I don’t think so

Don’t loose from sight, that last two-three years desktop sellings are falling down, and now most popular form-factor is netbook/tablet pc. And there goes SSD by default.

I was talking about mmap “function”, not “functionality”. To do the system your way I would need to:

  1. Write OS-independent memory mapping. I think boost have something like that so this may not be needed
  2. Write zip library that uses #1
  3. Write tool that will create correctly organized archives

In case of my proposal all I need to do is write some bits of glue code to connect zip library with our code.

Your solution may be faster but it also much more difficult to implement. And I never heard of any project that uses something like this.

True. But in case of mmap() we would have to manually deallocate some data in case of running out of memory. In case of caching this will be done by OS.

For example if player opened allied town ONCE during the game for mmap this would mean that this data will remain in memory till the end of the game (unless there will be way to detect that player will not visit that town anytime soon).
In case of caching these images will soon be removed from cache in favor of something else without any hints from our code.

While majority of H3 players still uses 10+ years old Win XP…
SSD are becoming popular it it will take quite a while until they will replace HDD’s.
And why you haven’t answered on “why?”? With .zip’s you will get some speedup when using HDD but this does not means that there will be slowdown on SSD’s.


Because when there is a lot mods, VCMI starts to slow down and works with delays right now. If you add additional zip functions, it only will get slower.
PS If I was developer, i’d rather load graphics and def in process when they are needed.

Why you sure about 10+ years old Win XP?
Is there some statistics of this site visitings?
Maybe create poll?

Actually there are. :wink:

However they cover the whole period since the opening, so they can be misleading. In reality there is a notable boost of Win7 users the last couple of years, while WinXP users are gradually decreasing. But same as we still had Win98/Me users in 2008, we’ll most probably continue to have WinXP users for many years still, even if just a minority.

Allowing packaging mods in zip sounds reasonable. Zip decompression is really fast. I don’t think compression will be the chokepoint anyway, things that gotar proposes seem to me a premature optimization. *

For the time being let’s do it simple. If the speed issues will arise, then let’s profile it and fix it.
We are fine with breaking mod compatibility. We are planning to have a central repo for mods, so we can automatically repackage them if the container shape is changed.

For the last month, Google analytics says:
Windows: 73,45% => among that: Win7 68%, WinXP 17%, Win8 11%, Vista 3%
Linux: 14,33%
Android 6,34%
Macintosh 4,17%
iOS: 1,32%

WinXP current share is as low as 12,5%.


Are you referring to the startup times (from starting appp till main menu shows) or in-game speed? (Battles, moving hero, etc.)
VCMI is basically sluggish in-game because it doesn’t use hardware acceleration for drawing. There is undergoing effort to introduce OpenGL.*

  1. or use own (flat?) archive (just like bunch of commercial games do),
  2. yes, this is what I’ve meant taking about conversion tool.

Nope, that doesn’t work this way - mmaped (file-backed) data will be automatically flushed from physical RAM when OS decides it needs more of it. It resides in virtual memory and uses only your address space (even 3 GB should be enough for VCMI). Cache on the contrary contains random pieces of storage that you’ve already read so until run low memory you should NOT read these data again - even from the cache (consider all the initialization time or PNG decompression).

So the difference is: cache contains lots of garbage and shrinks when required (best effort), while mmap MAKES USE of the same cache. Cache is related to filesystem in general and you have absolutely no control over it, while r/o mmap SAVES YOU FROM SWAPPING (writing MBs of data to a storage).