You’re going to load ALL of these into memory? That hurts performance for sure. If you have hundreds of files they should be loaded on demand, not on startup (i.e. map them into memory, let the OS do the job including caching and swapping out when necessary as a bonus).
Well… first of all, do not load all the files at once, they are probably not required in the moment (player starts with one town without buildings or army, sees nothing on adventure map - unless loading save, which always takes longer, but still there is a chance that low-rank buildings or army won’t be accessed). And compare, for the maximum-entropy file:
$ cd /dev/shm
$ dd if=/dev/urandom of=test bs=1M count=50
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 7.20605 s, 7.3 MB/s
$ cp test test2
$ gzip test2
$ lzop test
$ ls -la
total 153708
drwxrwxrwt 2 root root 100 May 27 03:30 .
drwxr-xr-x 24 root root 94208 May 25 09:39 ..
-rw-r--r-- 1 gotar users 52428800 May 27 03:29 test
-rw-r--r-- 1 gotar users 52436824 May 27 03:29 test2.gz
-rw-r--r-- 1 gotar users 52431246 May 27 03:29 test.lzo
$ time zcat test2.gz > /dev/null
zcat test2.gz > /dev/null 0.79s user 0.03s system 99% cpu 0.828 total
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; ) 0.25s user 0.07s system 95% cpu 0.334 total
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; ) 0.27s user 0.04s system 93% cpu 0.333 total
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; ) 0.26s user 0.06s system 94% cpu 0.339 total
$ time zcat test2.gz > /dev/null
zcat test2.gz > /dev/null 0.73s user 0.02s system 98% cpu 0.759 total
$ time zcat test2.gz > /dev/null
zcat test2.gz > /dev/null 0.72s user 0.01s system 98% cpu 0.739 total
$ time zcat test2.gz > /dev/null
zcat test2.gz > /dev/null 0.76s user 0.00s system 99% cpu 0.761 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; ) 0.69s user 0.07s system 99% cpu 0.763 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; ) 0.77s user 0.03s system 97% cpu 0.818 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; ) 0.68s user 0.04s system 96% cpu 0.748 total
and something that compresses well:
$ dd if=/dev/zero of=test bs=1M count=50
50+0 records in
50+0 records out
52428800 bytes (52 MB) copied, 0.0912174 s, 575 MB/s
$ cp test test2
$ gzip test2
$ lzop test
$ ls -la
total 51556
drwxrwxrwt 2 root root 100 May 27 03:41 .
drwxr-xr-x 24 root root 94208 May 25 09:39 ..
-rw-r--r-- 1 gotar users 52428800 May 27 03:40 test
-rw-r--r-- 1 gotar users 50919 May 27 03:40 test2.gz
-rw-r--r-- 1 gotar users 210446 May 27 03:40 test.lzo
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; ) 0.52s user 0.00s system 99% cpu 0.522 total
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; ) 0.51s user 0.01s system 99% cpu 0.523 total
$ time ( cat test.lzo | lzop -d >/dev/null )
(; cat test.lzo | lzop -d > /dev/null; ) 0.45s user 0.00s system 98% cpu 0.459 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; ) 0.63s user 0.00s system 98% cpu 0.640 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; ) 0.62s user 0.01s system 98% cpu 0.640 total
$ time ( cat test2.gz | gzip -d >/dev/null )
(; cat test2.gz | gzip -d > /dev/null; ) 0.62s user 0.01s system 99% cpu 0.633 total
Everything between would be 20%-100% faster.
Every WIP should use ‘disable all the caches’ knob, KISS (developer’s host). Every mod released into repository MUST HAVE unique version ID. Uploading a mod with the same version MUST NOT be allowed. That’s why there were 0.xx releases, that’s why you can use RCS-generated ID for nightly builds (like git hash).
Remember to check mtime, ctime and file size as well, as CRC may be ‘tuned’ to match previous. It was not intended to provide signature, but integrity, Application should trust version strings.
Tell me, what’s the point of any reloading if the only thing that got changed was some graphics? It will be replaced anyway, no need to alert user.
97% of people (might be 99,5% of gamers) can’t repack this.
Original HoMM3 was written 14 years ago for Win95 with strict dataset. Write loading code well, using modern techniques, you’ll not suffer from such issues.
GL textures need to be loaded anyway. As for OpenTTD - isn’t that original data format? So - not the case, not your rationale.
No need to optimize a game that is 14 years old, especially not in terms of file size.