itoijala,
There will be issues with existing content - how VCMI can load maps from WoG if load order (and as result - numeric ID’s) is undefined? Same goes to original game as well as for such mods like HoTA.
And there are may be some more issues of which I am not aware right now.
We should assume that original / WoG maps need original / WoG content present, which still will be read from config files as it is now. So numeric ids will remain constant.
I’m working on Bonus format parser. It is necessary for creature abilities, but also Artifacts.
If it is possible for a modder to make the attribute non-static the code must be able to cope with this. KING1 is an example where this is possible. DOUBLE_WIDE I think not so much. The graphics just will not fit. CATAPULT could be non-static. KING1 and CATAPULT are great examples for possible upgrades from stack experience.
It is impossible to rely entirely on load order. For example the town config will say that this building produces this creature. But the creature config will say that it belongs to this town. There is no right order to load them.
Another example is two creatures in the same file. B is the upgrade of A and A hates B. The modder cannot choose an order that makes this work if the system linearly reads the file. Such a loader would complain about creature B missing when reading creature A.
Loading the original configs is ok as long as it is optional. If they are not found in the active mods the game should still work.
There is no right choice for enormous number of things that need to be standardized. Loader must load everything in linear order anyway. A modder should be aware of it but don’t think about it until something goes wrong. Loader should not complain about missing creature references before all creatures are loaded, no matter what choice on loading order is made.
My point was that if a modder manages to create config files where anything changes if the entries are processed in a different order, then this is a bug in VCMI. If something goes wrong then the modder should file a bug and not try to work around the problem. When the bug is fixed the workaround could stop working.
itoijala, completely agree. ANY load order should work. One of reason I proposed to turn “id” into field is because loading alphabetically is not logical - modders would expect loading to be serial from top to bottom.
And don’t forget about maps - I don’t think that they will have any replacement any time soon and we definitely have to support them. With undefined load order this can be problematic.
Yup. H3 configs will be used to fill empty fields in vcmi configs (just like now). Mods or some kind of total conversions will have complete configs on their own so they won’t need any of h3 txt’s.
itoijala, I finally understand what you mean. I assumed that, since you discuss it, it’s important – perhaps a script whose execution interlaces creature loading could be run. If it’s assumed otherwise, then OK, but Ivan’s point seems invalid. Map format needs to be changed to support mods anyway.
This is how I think loading should be done (python-like pseudo-code):
mods = "A", "B", "C"]
tmpCreatures = dict()
# more config types
for mod in reversed(mods):
creatureFiles = get_creature_files_in_dir(mod + "/config") # effectively mod/config/*.creature.json
# ...
for creatureFile in creatureFiles:
creatures = parse_json_file(creatureFile)
for creature, data in creatures:
if not creature in tmpCreatures:
tmpCreatures[creature] = data
else:
merge_json_data(tmpCreatures[creature], data) # add new keys from data to object, replace existing keys
# ...
for creature, data in tmpCreatures.items():
if not all_referenced_objects_in_tmp_dicts(data):
error()
# ...
creatures = dict()
# ...
for creature in tmpCreatures.keys():
creatures[creature] = Creature() # new empty Creature object
# ...
for creature in creatures.keys():
fill_in_all_data(creatures[creature], tmpCreatures[creature]) # error on false or missing data, warning on unknown data (typo in key?)
# ...
# all objects are now available and fully loadedLoading order (except for entire mods) is specified (or needed). All errors are caught before the game starts. After the end of the code all objects are completely loaded. Maps from string ids to the objects are also created (needed for scripting, …). Loading order inside mods is not specified (probably pseudo-random due to dicts).
If the same object is specified twice in the same mod the order will be effectively random. I think the same creature twice in the same file should not be supported (it is made impossible by the current format). Files in the same mod could be loaded alphabetically (theoretically dependent on locale, need to use C locale for sorting).
I think that creating a new map format should be a priority after all config formats are agreed upon because it is needed for testing. The new map format would also be json so it can be hand-edited. This means that there is no rush to create a new map editor because test maps can be coded by hand.
Tow dragon,
Not necessary. New creatures or artifacts can be placed during map randomization, new heroes or new towns can available for selection in pregame.
This may need some minor tweaks to list of banned items but everything else should work.
New map format would need a new map editor so this is a very long-term goal.
If existing content (H3 or Wog) will be handled by this new creature config as well - then undefined load order issue will be present even without mods. (And I don’t think that supporting both old and new configs is a good idea)
itoijala,
for mod in reversed(mods):Why reversed? If A depends on B we should load A first. Right?
creatureFiles = get_creature_files_in_dir(mod + "/config") # effectively mod/config/*.creature.jsonIMO file name(s) of creature files should be either specified in main mod file or be hardcoded (e.g. mod/config/creatures.json). Our filesystem can get such list almost immediately (no searching through huge amount of files).
merge_json_data(tmpCreatures[creature], data) # add new keys from data to object, replace existing keysThis may be the most complex part of loading actually.
Look on creature config for example. There is field “cost” and field “bonuses”. Both of them are vectors but first one should be overriten by mod while second one should be appended.
This approach (generate huge single json and feed it to engine) is the best one so far though.
warning on unknown data (typo in key?)Not necessary typo. VCMI have separate client and server so all gui-specific data should be only read by client (and not by library handlers). There should be way to send this json to client to finish loading. Currently this is not strict so don’t look on existing code but this should be fixed at some point.
Wesnoth map format is text. HoMM5 map format is text. But due to huge amount of data writing them by hand is extremely complex. At least h3m -> json converter is must-have.
No. If A depends on B then B needs to be loaded first. This way A can change the creatures from B in the merging step.
I think having a list in the mod’s config file is ok but hardcoding is not. If only one file is allowed they will become very large and hard to read and edit. For example if I add two towns I would put the creatures from the towns in two separate files.
I don’t think it is so difficult. In fact, I kind of designed the format to support this easily. There are three cases:
- for keys where the value is a primitive just overwrite it
- for vectors also overwrite (only used for upgrades, damage, amm, missileFrameAngles). It makes sense to overwrite these because they will almost always want to be completely changed.
- for objects ({…}) do this recursively
The abilities need to wait for a bonus parser to be written (Warmonger suggested this) before settling on the final syntax. Abilities will probably become a vector so the above algorithm might need special cases.
Perhaps it’s not so easy after all…
I agree. This is the reason I wanted to put all graphics and sounds together in the format. They can be easily given to a client loader.
I think converters for all formats are needed. This is also a good test that no needed information is missing. I don’t think that writing test maps by hand would be too difficult because they only need a few objects for the specific test.
The code will need to be reorganized a bit if the above loading separation (server/lib and client) is done. The map editor will also need the graphics that only the client would load. Perhaps it is better to leave all loading in the lib but just not load the graphics in the server. Something like
if (!is_server())
{
// load graphics and sound
}On the other hand loading everything is easier. I don’t think parsing the graphics and souns will take very long. The json format has already been parsed by that point so all that would be saved is writing the attributes to the Creature object.
Another thing is that this loading plan (pseudo-code from above) needs config files to be managed differently in the filesystem. All config files need to be loaded, even if a mod has a config file with the same name (likely). Example: Mod A has Creature C in file C.creature.json. Mod B mods the creature and also calls the file C.creature.json (makes sense). The file from mod A must still be loaded. The layered filesystem should only be used for graphics and sound. This is what modders would expect.
I suggest pre-load all the mods and then reconfigure them in next step, replacing string identifiers with generated ids.
But you know, I made that parser just yesterday :P. Check trunk.
I think we need the string ids for scripting anyway.
{
"type": "BONUS_TYPE",
"subtype: 0,
"val" : 0,
"valueType", "VALUE_TYPE",
"addInfo" : 0,
"duration" : "BONUS_DURATION",
"turns" : 0,
"sourceType" : "SOURCE_TYPE",
"sourceID" : 0,
"effectRange" : "EFFECT_RANGE",
"limiter" : "LIMITER_TYPE", optional_parameters (...)]
"propagator" : "PROPAGATOR_TYPE", optional_parameters (...)]
"description" : ""
}Are duration and turns really needed (I assume they control how long the bonus is active)? I think it only makes sense to configure permanent bonuses in the creature config. The same goes for sourceType, sourceID, limiter and propagator (I think).
So a smaller format for creature configs would be
{
"type" : "BONUS_TYPE",
"subtype: 0,
"val" : 0,
"valueType", "VALUE_TYPE",
"addInfo" : 0,
"effectRange" : "EFFECT_RANGE",
"description" : ""
}We still need a syntax to remove bonuses.
My bad. Should be “If B depends on A”. With reversing “B” will be loaded before "A"
Until we get proper dependences support using list where second mod may depend on previous ones is correct order IMO -> no need for reversing
… and in separate mods. What if someone wants to have only one town?
Anyway storing file list somewhere (for example in “master” file modConfig.json) is better than searching.
This may work for some mods. But it won’t work with H5-style alternative upgrades - for this mod merging vectors is needed.
So yes - it is not that simple
I still like this method - for example its first stage may be reading H3 configs (if present) into json and then overwrite it with data from mods.
Even bare map would be quite big - terrain (over 1000 tiles for 36x36 map), player settings, several essential objects like towns or heroes.
if (!is_server())
{
// load graphics and sound
}No. What if somebody will create their own client or just modify existing one?
There should be some kind of callback to pass this json structure to application. And VCMI lacks anything like that right now.
Filesystem shouldn’t be a problem - files from mod can be separated from game, merged with dependencies, etc - all of this is quite simple to implement.
Something like this?
{
...
"configs" :
{
"creature" : "A.creature.json", "B.creature.json"],
"town" : ...
}
...
}You are right. I forgot about the terrain.
All parameters but type are optional. What I showed is general config for all bonuses possible (for now).
Why?
In fact, removing bonuses in code as such is very rare, they usually remove themselves when needed.
I think we should focus on one task at time, for example making the creature config work. We are not going to create complete mod system out of thin air.
An example: The Pikeman has the ability CHARGE_IMMUNITY. A modder doesn’t want the pikeman to have this ability. So they make a creature config file with the contents
{
"Pikeman" :
{
????
}
}I have written a full implementation of the pseudo-code I posted in an earlier post. The code is in Python because I can’t really write C++ :(. But the point was to see if the system could work like this. I think it can. The code is 350 lines and includes full (probably only almost full) error checking for all attributes. I used dummy formats for mods, towns and resources to complete the mod system and error checking (check that the referenced town exists). Code is attached along with two example mods.
load.zip (6.29 KB)
Given the complexity of Bonus System, I’d leave it to scripts. Python scripts. Additional parsing for rarely used feature is not good idea.
That’s why we already Json parser with tons of examples in the code 
But certainly no Python interpreter for a long time from now.
I appreciate your effort, but this python code has no use apart from being a pseufo-code example. It is even far from our existing solutions.
Also, there are no mod examples in the package, just empty folders I’m afraid.
Validation can be done using schema - check buildings.json or defaultSettings.json.
Similar one can be made for creatures.
One more thing in format: cost.
Short version is fine. But long one?
"cost": 0, 0, 0, 0, 0, 0, 1000]I want to make this unit cost 5 sulfur. Which value should I change? I need to learn order of resources? Too complicated.
This should be replaced with this:
"cost": { "gold" : 1000, "sulfur":5}More readable, support for any additional resources (e.g. mithril), have functionality of long version and quite small so it can replace small one as well (and less checks in our code).
Regarding parser:
Right now I’m working on something similar for towns and town buildings (I know this code better than creatures). This would need quite a lot of changes so it will take a while.
So far it looks that converting parsed txt’s to json and merging it with configs from vcmi is indeed the best solution - it would solve all issues I’m aware about and can be reused for mods.
Can’t say when I’ll upload code - it is quite messy right now.
Yes, that is probably a better idea. But the schema should not be in the file. It should be in another file or embedded in the code.
This is already how it is :). Currently the format would support a single integer meaning gold or the dict. WHat should happen in this situation?
"cost" :
{
"Mercury" : 5,
"Gold" : 100
}mod:
"cost" : 200How should the values be merged? Does the unit still cost some mercury?
I would suggest to specify and review every format before writing code is started. You might want to make a wiki page and post a link here.
The point is: the complexity of the bonus system doesn’t matter because the bonus system will only ever see the finished product. All adding, removing and changing of products is done before they are loaded by the bonus system.
The two parts that were still a bit unclear were how to merge upgrades and abilities.
Mods should be able to add and remove upgrades. So with the base creature like this
"upgrades" : ["Halberdier"]a mod could do this
"upgrades" : ["-Halberdier", "Marksman"]so that what is seen by the game is then
"upgrades" : ["Marksman"]The same can be done for abilities. Example: base
"abilities" :
{
"ChargeImmune" : {"type" : "CHARGE_IMMUNITY"},
"HateDevils" : {"type" : "HATE", "val" : 150, "subtype" : <Devil>}
}mod
"abilities" :
{
"-ChargeImmune" : {},
"HateDevils" : {"val" : 500},
"HateAngels" : {"type" : "HATE", "val" : 150, "subtype" : <Angel>}
}the system sees
"abilities" :
{
"HateDevils" : {"type" : "HATE", "val" : 500, "subtype" : <Devil>},
"HateAngels" : {"type" : HATE", "val" : 150, "subtype" : <Angel>}
}This uses a slightly different format for abilities. The keys are names given by the modder. They can then be used by other modders to refer to the ability. The names are never seen by the bonus system. They are also local to the creature.
Because all of the transformations are done before the bonus system sees anything, changing things is easy. It can be done the same way that the rest of the data is merged. I don’t think that the feature would not be used by modders. I would think that changing a creature’s abilities would be common.
The bonus format has to be changed to use string ids for creatures and not numbers. See subtype above, how can the modder refer to creatures if he does not know the id (for any creature not in the original game)?
Quite the other way round: the Bonus System is a base mechanism and it’s really difficult to tell which exactly Bonus we want to remove. If always requires some lines of C++ code and existing mechanism is not universal.
The problem is that “subtype” parameter can as well refer to spell type or resource type, to name a few. It’s simply not worth to think about every possible case in advance to create a parser. We need mechanisms which can handle any situation, not one particular case. Some things may be impossible for now and can wait until scripting is implemented.
Also, the modders are suppose to create mods in first place. Let’s make a FIRST one, then a bunch, and only then we will try to handle incompatibility or cross-reference. All this discussion leads only to conclusion that we can’t predict all possible conflicts in advance.