Now that the modding system is in place wouldn’t be nice to have a SoD in a mod format that has nicely organized all the original H3 data in VCMI mod format?
Here my idea: Upon first run VCMI processes the files from the /Data folder and moves them to appropriate locations. Then the original packages are just discarded.
I’ve already done some experiments and grouping the files to the right folders is not that complicated. Theoretically you just need some json files (that in part exist) to map the resources to the right places and some simple functions to move things around.
Advantages:
there are many duplicate files in the game because of AB. Getting rid of those would speed up loading times.
there are tons of quite large campaign specific files that serve no purpose when you are not campaigning.
modding would become more straight forward for inexperienced users (like me). You will not have a huge pile of files with cryptic names and sometimes content thrown together in 1-2 compressed folders.
special tools for packing/unpacking stuff would be drastically reduced.
we will have a base for mods derived from original factions.
things from original H3 could be turned on and off again like any other mod.
The result would be a nice clean structure that would improve modding experience under VCMI by a ton.
In part? What’s missing? Right now mods share quite a lot with main game:
filesystem: mod.json may have “filesystem” entry instead of Content folder (default). VCMI uses config/filesystem.json for this.
json configs: All mod formats on wiki are also used by main game. Added in 0.93 config overriding used here as well - H3 configs are converted into json and merged with our data in config directory.
Are you sure? IIRC most costly operation here is scanning directories for files. Parsing archives is much faster than that.
This part is not as simple as it looks like. If (for example) Dungeon is removed from the game what would happens if player will load map with Dungeon? Or with dungeon creatures? Or with Medusa Bank?
Besides - I’m not sure if VCMI can handle “missing” objects - ID’s of H3 objects are hardcoded and must be continuous.
IMO moving some of config files into “core” mod may be a good idea but everything else should be kept as it.
And what’s the problem?
If users make standart towns external, they will prepare to not play carts made for turned-off factions.
In part of towns I agree that standart towns/creatures must be moved to mods. Only difficulty is that now some abilities are hardcoded.
In this realization we can improve mod management system - it will begin being helping, when there is a chance to turn off towns.
I am talking here about campaigns and a way to distinguish some of the sound files, like creature abilities and interface/battle sounds.
Here I can only guess what the difference in speed between removing duplicates and spliting the rest into several separate archives over having all in one place. Maybe it will increase, maybe it will decrease but you will have the chance to easily turn on and off again individual parts that you do not need.
I think this can be/should be fixed. ( I for one would very much like to play without sharpshooters and enchantress).
Disabling original towns and factions should be supported by VCMI. Since a totally separated faction like Cove is possible why shouldn’t the rest of factions be treated the same way?
Here we disagree. A structured and ordered filesystem (like the one in place for cove town) would help modders and new developers in better understanding the components of the game and will ease up their work a ton. Like showed in Cove already it is only natural to store creature sounds/ animations/ stats in a common folder.
Mods are the driver force for this game, the life of modders should be made as easy as possible.
What is the motivation behind this? What are the downsides? I can think only of upsides.
Arrogance, aka “default map” requires at least Necropolis and Conflux despite not being made for specific faction(s).
Disabling one of standard factions will break huge number of standard maps. And there is no way to determine what factions are used on map without starting it first.
While I’m not against concept of disabling factions or moving them into mods I don’t think that we can implement this right now without running into endless bugs.
Alternative solution is to implement field “special” for towns. But it also have some limitations - random towns and dwellings have set of allowed factions. What should we do if none of these factions are available?
It most likely will increase CPU usage during loading while insignificantly decreasing memory usage (10-20 bytes per file). So there is no point in disabling these parts - when unused they have no notable effect on speed.
IMO we should keep installation as simple as possible. Meaning that all data should be used as it without conversions.
Feel free to make patch for this. From my point of view this is not essential feature so I’d rather spend my time on other areas.
And what are your upsides? Especially regarding better filesystem?
Keep it simple. Such rearranging of files means more code to write, more code to support (more bugs! yay!).
One of main ideas of modding system is that modder does not need to know how H3 filesystem works. You want to change creature animation? Replace entry /“graphics”/“animation” - no need to guess name of that .def or finding out where that data is stored.
So such rearranging should not be needed for vcmi.
I agree that some areas like GUI still have “guess file name” problem. But that’s mostly because we don’t have real GUI modding right now.
First to make 1 think 100% clear. There are no conversions involved! Data remains in the same format and with the same names but not in 4-5 chunks instead it is moved in the proper places.
Cpu usage will increase a lot when you will ad zip to mods. But that is the whole point. By making usage of more CPU power you gain ~2-3 times more speed when loading files. But you have a point without testing it is hard to say if less files but more disperse will make it faster or worse. So me “guessing” you will actually gain some speed could be wrong.
The whole disabling factions/creatures/towns thing is a bonus that people could use. I see it come into play very nicely in RMG for example.
Another disabling usage would be in mods with lot of contend for existing towns. Let’s say you want to mod Castle and touch every creature, every building. The easy and reasonable way would be to duplicate the castle and modify there.
If you want to mod something usually you modify! existing things. To do that you need to be able to find them and see how you can mod them. Having a nice structure that makes files consistent, easy to find for everyone and without the help of special tools can only help with that. Where to put that having SoD as a “mod” will create a great example base for future modding. Right now all the answers for modding help are on the form: “take a look at mod x,y and see how it is done there”
Codding effort for moving files in the right places is not that great. You “just” look into some json files. I offer to to that. VCMI I understood has already a mechanism to look for files in more than just data folder, but if any adjustments are needed in that part someone else needs to help.
Actually no. Lod archives use exactly same compression algorithm. So speed will remain the same.
And after moving data into mods it will change into “take a look on mod with Castle/Dungeon/Rampart/etc”. Or am I missing something?
Moving towns into separate mods is an interesting feature but I don’t think that it can be implemented safely at least right now. (for reasons I mentioned earlier)
What about keeping this file moving optional or (even better) - as a separate utility that modders may use if they want to?
I wanted to say that now you do not compress mods and that CPU usage will increase once you do.
The idea was that you will have a lot more examples to point the people to. All sorts of creatures…
This is the answer I was expecting all along. That this makes some sense but there are some hurdles atm.
How about this: I implement the whole moving thing (did some parts already as test) as an VCMI_ckient.exe option but leave the original files in place! and see how people like it. (so when you run client with a specific option SoD mod is created with files in the right places but inactive.) If by any chance VCMI code will never support this, at least modders could have a nice reference.
For this I will need very little help, mainly in the form of: “is this file structure ok?”: or “what is the purpose of these files?”.
You may want to make it as separate binary (linked to vcmi library to get support for H3 formats).
VCMI should support resulting directory structure - filesystem support is quite flexible here. But I’d like to keep this separate (unless other devs will propose to include these changes in vcmi).
krs, I think you can try to move at least 1 town to mods and see, if this works (disabled town in .config, move to mod only json configs, but not yet graphics)
If this will work, this can be used for all other standart towns.
First step is to dump all of the original resource files: xxx.snd xxx.vid sprites.lod bitmap.lod in a temporary folder. Armaghedon’s Blade files will overwrite where appropriate.
Second Step a function will parse these folders and move file to appropriate locations. Everything that is faction related will go to that folder the remaining files… I do not know yet.
Temp folder structure will be:
data_temp\
sprites\
images\
sounds\
videos\
misc\
I’ve separated the original H3 files into logical groups. Some more detailed separation can be done in the future if really needed, but for now it looks cohesive enough (EG: group terrains by type, group map editor objects, group interface elements).
For complete separation of towns there are 2-3 def files that need splitting. (EG: Creature portraits are all in one def, town icons are all in one def and I think there was another one).
And a lot of icons: artifacts, buildings, luck/morale, primary/secondary skills, hero specialties, spells, resources, dwellings.
Those are icons for starting bonuses of campaign scenarios. Apart from these bitmaps there are also several .def files for same purpose (can be identified easily by yellow-ish background and “bo(n)” in their names)
Here the .json for Pikeman. Can someone take a look for something wrong? (Not numbers but overall structure).
I cannot seem to be able to find big icons for creatures. (Only for small “CPRSMALL.def”). Is there any documentation/example on splitting a .def file like this?
Abilities are missing?
You may want to dump creature JSON structure somewhere in CCreatureHandler::loadObject - this will give you complete structure including parts imported from crtraits.txt
File names - since you’re extracting everything anyway why not switch to more readable names? E.g.
“castle/PIKEATTK.wav” ->“castle/pikeman/attack.wav”
TWCRPORT.def
You can access frame in .def file using syntax : but there is no straightforward way to extract them using VCMI code.
Perhaps load .def using CDefHandler? This will give you SDL_Surface’s that can be saved as bmp
The Def Preview program as standalone is not helping because there should be no game files redistributed. VCMI has to come with a program that does the splitting automatically from ORIGINAL files.
That being said are there any source codes for Def Preview? Those can be used as model.
Renaming them will cause problems with Original H3 mods. And the renaming gains are not that great since you have a meaningful folder structure with reduced number of files (EG: castle/pikeman/pikeattack.wav anyone can tell what this is).
While it would be great to have nice names for files… the trouble that will cause with original mods is just too much imo. (Maybe a mapping system could be used?..).
like any other mod.