Modding system discussion

International board Development
#201

So let’s consider this:
script A: 2x hp for flying creatures, flying taken away.
script B: 3x hp for flying creatures, flying taken away.
What happens if you load both of them? HP x2? HP x3? HP x6?

Good idea, it will make IDs more consistent.
Small suggestion: WoG is going to be a mod, so I’d put WoG and SoD objects in separate namespaces.

#202

I think should be x6. But in fact will be x2 or x3, depending on load order, because after first bonus change flying bonus will be deleted.

To fix this solutions may too difficult: smth like “mod should not see changes in bonus system from unrelated mods”, but this by-turn in not desirable behavior.

#203

That’s a lot of… data mlox.googlecode.com/svn/trunk/data/mlox_base.txt
Luckily biggest issues in TES modding are not applicable to our modding system so at least we’re safe from something like that.

While we don’t have any tool for auto-downloading providing support for something like that is not a problem.

Currently mods consist from two elements:

  1. mod.json with some basic description of mod
  2. Directory with all content of mod

mod.json acts as small file with mod information (e.g. version) and directory with content can be compressed into .zip to turn mod in just 2 files.

Problem: accessing files in compressed mod can be too slow. Can be avoided by using this archive only to store mods (no compression) BUT distribute mods in archives that consist from mod.json + uncompressed archive.

So mod ready to distribution will look like this:

modArchive.zip - file that user must download
    /mod.json - mod information
    /Content.zip - archive with no compression, contains mod data

Then we may also add repository - server that contains all available mods. To avoid writing any specialized software it will store such archives + index file - json that contains same data as individual mod.json files.

So autodownloader/mod manager will get this file, shows it to user. User selects mods he wants to install\update, mod manager downloads them and unpacks.

Probably should depend on how script was written. It may have some internal state to determine current state (on\off) - this would mean no to 6x HP.
As for 2x vs 3x - I don’t see correct answer here to begin with.

This should be either left to random or there should be some way to say “mod X should be loaded after Y, if present” without any clear mod order or “priority” field - it will act too erratic due to need for reordering of dependencies.

#204

@AVS, Ivan

I think that the result, above all, should be deterministic (always or by default)*. These mods are incompatible (one is an alternative to the other) but this fact doesn’t have to be written in mod config file (in conflicts with section) nor can be automatically detected in general. HP x 6 is indeed the most reasonable solution here but I don’t think ensuring it is worth the effort.

*Oh! My script works! HP x3! [runs the game once again] What? HP x2? What a ***! [censored]

#205

To make load order deterministic we should define default load order (smth like order by mod name) + take into account user defined order and dependencies. the question is: can we design less complex system than Mlox.

#206

My algorithm for deterministic load order (assumes mods to load are already selected):

  1. create directed acyclic graph of mods and dependencies, e.g.
A -> B
 \    /
  \ /
   C

D

means B depends on A and C, C depends on A, D is independent
2) add ‘base’ node; connect ‘base’ to all nodes without incoming edges

base -> A -> B
    \    \    /
     \     \ /
      \     C
       D
  1. perform BFS from ‘base’, at each arrival at a node remember distance to base
       [1]   [2,3]
base -> A -> B
    \    \    /
     \     \ /
      \     C [2]
       D [1]

assign each mod a sequence number – maximum from remembered distances
A, D - 1
C - 2
B - 3
The sequence of loading is determined by:

  1. sequence numbers,
  2. if sequence numbers are equal, then lexicographical ordering of mod names.
    So the final load order is A,D,C,B.

I hope we will never have to use expert system to fix loading order.

EDIT:

I’m an idiot :blush: – this algorithm won’t work. We need Bellman-Ford from base with all edges having weight -1 to obtain longest paths (sequence numbers).

#207

My idea for my ‘hypothetical’ IDE for baldur’s gate trilogy (literally, since it would use the netbeans platform), was much the same. First sort by a deterministic order all mods that are in the mod database; database would include locations to the download of mods, the folder structure to use inside of them :unamused: , shortcuts for the thread relating to the mod on the appropriate forums with a preprocessor replace. And some ‘runtime code units’ that would essentially use groovy before installing the mod (programming is sometimes needed by BGT - this project would essentially replace BGT, which is a bat file script that mucks around with a single defined order for installs).

The actual order rules and conflicts would be on a separate file like that mlox_base there and wouldn’t imply any base order.

Then it would (hopefully), check for new versions download all that into a cache, check for the rules and rebuild the graph and install or reinstall (Weidu is neat for that, it allows ‘rollback’), from the first change onwards.

Problems are many. For instance, did you wonder that mlox is using [size=] + name to identify mods? Yeah… TES has the same problem, of modders thinking that versions are optional stuff. You can detect something is different in the rules… but can’t detect if it is newer (without storing the old ones at least :unamused:)

If it ever gets to a functional stage however, it’s likely to be much much faster than install on the bat file, for various boring reasons, relating to weidu invocation merging (the rules are checked before creating the install list, so there aren’t any ‘if’ dividing in two a perfectly good weidu invocation - weidu ‘mods’ are more like a collection of ‘submods’ that can often be installed separately or not installed)[/size]

#208

I think a hash would be better for telling the difference actually. But the problem is real. If we had a server for mods (possibly distributed via p2p) this would reduce this problem.

#209

Main problem here is not modders but Bethesda. From what I see they still don’t have field “version” in their mod files (.esp). So versioning can be done only by renaming mod file each time (and break all mods that depend on it).

Mod hosting sites usually handle versioning quite well but this information is lost after install.

P2P? Certainly an interesting feature but is it really worth it? With large number of users it would be great but othervice it may not be reliable - too slow or not functioning at all.

#210

What do you mean by lost after install?

Our server could be a P2P peer too, then I see no reason for it to be slower and it would function all the time. I’m sure a few VCMI enthusiast could seed the content significantly lowering server bandwidths (which is primary motivation for this feature). Look at our download statistics, then at Android version statistics, then estimate monthly bandwidth per player (I think 10 - 50 MB/month per at least casual player is reasonable guess). Then 10 GB per month is minimum in mid-term and may reach 1 TB per month should VCMI become popular.

#211

I’m talking about TES modding. By itself mod file may contain only a small (up to 512 bytes) description and no version information at all. This is how mod file header looks like: uesp.net/wiki/Tes5Mod:Mod_File_Format/TES4

Mod hosting sites provide much more information (large description, version, download URL, etc) but most of tools don’t have access to it because it is not part of mod file.

So it is hard to say what’s the problem here - modders who don’t use versioning correctly or badly designed mod format. Second one certainly can be avoided in vcmi.

I’d rather avoid such calculations until we get some real statistics. Certain amount of players will want to play good old Heroes 3 (with or without WoG) and won’t be interested in mods. This is especially true for those who can’t play original game (e.g. Android users).

I won’t surprise if downloads of vcmi itself will require more bandwidth than mods.

#212

Actually updates could be distributed via P2P too. Anyway, P2P can be added at any time so we can just wait as long as Tow’s server is enough.

#213

I got one idea.
In WOG there was script, so when you attack neutral monsters, some other creatures were on their side. It’s very treacherous, that other creatures were not shown on adventure map (cause they were generated by script on time of battle).

My idea - on map gathered bands will display as some tent, right-clicking on it will bring down the list of creatures in it. So there could be neutral hero also.
When they are defeated, tent disappears, and hero may pass.
Also on defeat some artifact may be passed.

Creatures and artefacts may be random (from one fraction, generated on start-up). Or they also may be written in external mod.

#214

In H4 and H5 it is simply possible to have multiple types of creatures in wandering stack. It is displayed as the strongets creature on adventure map. In H5 it also has some additional banner to indicate there are more creatures.

And monsters can give artifacts on deafeat already, it’s all possible in OH3.

#215

So will wait for this feature :smiley:

#216

Trying to finish editing H3 objects via mods feature.

  1. It seems that “full ID” must have different separator between mod name and object name, e.g. colon:
    "modName:creature.someName"
    Othervice it is not always clear whether first section is mod name or type of object.

  2. Scope resolution (see page 14). I’ll take metadata approach - JsonNode will have metadata field which will be used (in this case) to store name of source mod.

EDIT:

  1. Original objects (for now - including wog) will be in namespace “core” to differentiate from objects local to mod

  2. mod load order will be determined strictly based on dependencies:

  1. Sort all mods by name (to avoid hash map having any effect on this)
  2. Use topological sort to load dependencies first
#217

Which topological sort? And what does point (1) actually do? Keep in mind I’ve already suggested a solution for this: [forum.vcmi.eu/t/modding-system-discussion/451/206) (the proper way to calculate sequence numbers is in edit). This is basically topological sort but with all the details specified.

#218

CModHandler::resolveDependencies
It was present even before 0.92 and this discussion.

Right now initial list of mods used as input to topological sort is not ordered alphabetically - it uses order in which these mods were found during scan of Mods directory.

Along with my changes final algorithm should be like this:

sort( input )

while (input is not empty)
    resolved mods = output
    for ( mods in input )
        if ( all dependencies are in list of resolved )
            add mod to output

Do we need anything more complex than that? Results should be the same as with your algorithm.

#219

Bellman-Ford is faster and produces different order. Your algorithm is fast enough anyway. And order… I don’t know which one is better – if there is such thing as better or worse order here.

BTW, you seem to have forgotten to check if the dependency graph is actually acyclic. But it’s simple.

#220

Good part is that right now we don’t need to care much about load order - most of mod system will work just as well without it.

What can be affected by it is:

  • order in which files are overridden by mods (if same file present in two mods)
  • order in which patches* are applied to mods (only if patch affects same config entries)
    And both of these situations are already a conflict between mods that must be solved by author(s)
  • by patches I mean accessing namespace of another mod

No I didn’t. Check CModHandler::hasCircularDependency
Straightforward approach as well so this is not the fastest way but at least it simple.

I just don’t want to spend too much extra time here to write faster algorithms properly - it is unlikely that this place can become a bottleneck in our loading code.