Modding system discussion

[AVS]

Numeric IDs are imortant as you don’t know which mods will be used at run time. Imagine the script which generates random army of tier 7 creatures. You can use a function (or even simple iteration at full range) to get an array of numeric IDs of all tier 7 creatures. Strings won’t be of any use in this situation.

In such situation no ID`s needed, only instance pointers.

[Ivan]

Imagine the script which generates random army of tier 7 creatures. You can use a function (or even simple iteration at full range) to get an array of numeric IDs of all tier 7 creatures.

2. Accessing all creatures available in game

for creature in engine.creatures: 
     if creature.level == 8: 
         doSomething()

You don’t need numeric ID here. Pointer to class identifier is much better - OOP style, easy validness check (just compare it to NULL\None), type safety (id #4 can be 4th creature as well as 4th artifact while pointer already have specific type).

[Warmonger]

Pointers? In scripts? How do you imagine that?

Even then, pointers STILL need to have some user-accessible name.

Here is Heroes V scripting manual for Lua. If you scroll it down, there are IDs for all game objects in two forms - numeric ID and string ID, which work VERY well. The only difference is that they are enumerative, while in VCMI we need to assign them synamically at the beginning of the game.

[AVS]

Pointers? In scripts? How do you imagine that?

object variables, more precise.

So don’t need to know, what numeric|string IDs of 7lvl creatures in your example. You gen objects defining creatures in some list, and when use them as you wish.

And yes in fact it is a pointer to CCreature wrapped in script object.

[Warmonger]

You gen objects defining creatures in some list, and when use them as you wish.

But the discussed issue is what kind of list (not “some”) it is and how (not “as you wish”). they will be used. It’s easy to tell that scripting engine should be “nice”, but it is not enough to implement it. Details, please.

[Warmonger]

You gen objects defining creatures in some list, and when use them as you wish.

But the discussed issue is what kind of list (not “some”) it is and how (not “as you wish”) they will be used. It’s easy to tell that scripting engine should be “nice”, but it is not enough to implement it.

[Ivan]

Pointers? In scripts? How do you imagine that?

My bad. Not pointer but reference to python object. From what I see in Boost.Python docs it can create wrappers for C++ objects.
So there can be python class CreatureType which is wrapper for our C++ class CCreature.
mod.creatures is a python map of string <-> CreatureType.
engine.creatures is a python list of CreatureType
so in this code

creature = mod.creatures"name"]

creature becomes reference (or separate instance, not sure how python works here) of python class CreatureType which wil give modder access to our internal C++ CCreature class.

[AVS]

You gen objects defining creatures in some list, and when use them as you wish.

But the discussed issue is what kind of list (not “some”) it is and how (not “as you wish”) they will be used. It’s easy to tell that scripting engine should be “nice”, but it is not enough to implement it.

What list? In current version of engine its CCreatureHandler::creatures (it should be accessible from script). Iterate list, select creatures you need (as wrappers to CCreature*) and do whatever you want

PS BTW I didnt say that IDs are useless, I say that manipulations with objects should be through wrapped pointers but not ID`s. ID should be only to find particular object.

PPS. see Ivan`s answer.

[Warmonger]

Well, Ivan’s proposal makes much sense for me. That was quite an initial though, but I’m not sure if exported classes are exactly what beginning modders may look for. It’s good for definition (just to create new type of creature), but may be confusing for some operations, modifications, comparison or iteration / array reference. You know, object-oriented architecture may be difficult to understand as a primary way of scripting / modding, especially as creatures are quite simple and uniform objects.

[AVS]

Well, Ivan’s proposal makes much sense for me. That was quite an initial though, but I’m not sure if exported classes are exactly what beginning modders may look for. It’s good for definition (just to create new type of creature), but may be confusing for some operations, modifications, comparison or iteration / array reference. You know, object-oriented architecture may be difficult to understand as a primary way of scripting / modding, especially as creatures are quite simple and uniform objects.

I see no problem to implement procedural event-driven layer (like ERM) inside script library over object oriented event-driven intermediate engine-script layer (as proposed by Ivan).

Some common tasks anyway need high-level helpers.

In comparison to ERM, low level tasks, which most of all implemented in ERM through direct memory access should be implemented in VCMI with direct access to object wrappers. High level functions (most ERM receivers) should be implemented in higher level. It`s the same architecture as WoG in fact, but most features hardcoded in WoG can be scripted in VCMI.

[AVS]

1. Files location
Each mod has separate subfolder inside “mods” folder, and identifies by this folder name as string ID .([size=75]From ERAII[/size]). With “data”, “config”, “scripts”(for python, data/s for ERM) subfolders.

2. WoG
WoG is a mod, all original datafiles remains unchanged. ([size=75]From ERAII[/size]). This make possible to test modding system w|o need to write additional mods.
All WoG mechanics, and then nearly all game mechanics should be rewritten in script language with the lapse of time.

Nearly all WoG mechanics is optional, script should not rely on that particular option is enabled. Other mods may have optional mechanics and may have no configurable option at all. Handling this is not the purpose of Modding system.

3. Conflict resolution
Mods should be as isolated as possible. Mod runs as if there are mo more mods except its dependencies. Engine should resolve conflicts if possible (for resources f.e. see Resource override priority). If conflict cant be resolved conflicting mods are marked as conflicted to each other and then never runs simultaneously. In current session newer (by release date) mod runs.
[size=75]Adding HotA we can test mod isolation (HotA is addon to SoD, not WoG addition). If we will be able to run HotA maps with WoG gameplay (which is not possible by design: WoG require “commanders” defined for all towns, HotA has new town w|o commander) then we have good modding system.[/size]

Special case: if mod A depends on B&C and B conflicts with C, then A responsible to resolve incompatibility between B&C and then A is enabled B&C are not treated as conflicting.

4. Resources
Each mods may have several data packs (LOD & may be own format in future) and unpacked files in data folder.
4.1 Resource override priority (high to low)
[ul]
unpacked data in mods "data" packed data in mods "data"
unpacked data in deps “data” (topological sort + order in dependency list)
packed data in deps “data” (topological sort + order in dependency list)
original unpacked (in fact=VCMI)
original packed
[/ul]
4.2
Resources are identified by filename (with override priority) or by URI ({file|lod}://modname/[lod_name|subfolder_name]/filename )

5. Configuration
Each mod must have main config file (JSON) defining (*=optional)
[ul]
mod description, release date and version
required VCMI version (min, max])

• conflicting mods
• other required mods with version.[size=75] order of mods in this list influences override priority (4.1)[/size]
  *additional config filenames to add|alter settings for game objects
  *JSON object with additional mod specific configuration
  *main script filename (for python scripts)
  [/ul]

6. Scripts
Each mod may have scripts to alter game mechanics.
6.1 Each mod may have ERM scripts in data/s folder(for compatibility). They are loaded automatically. Script overrides script from required mod if filename matches.

6.2 Each mod may have Python scripts in “scripts” subfolder.

7. Localization
Localization of game in a whole or particular mod is a mod itself. Mod name for mod localization is spetial mod_name.locale. Only localization may have extension in name.
Engine should have option to enable all localizations for particular or system default locale. (All other localization mods are disabled). In some cases localization may require more than just resouruce and config override, so localization is full-function mod.

8. Maps

Map shall define required mods, but other mods can be manually enabled before game start.

To be continued…

[Ivan]

1. Files location
I have a bit different proposition that will make mods more modular.
Instead of plain “1 dir = 1 mod” every directory can have “submods” and each submod may have its own submods and so on.
Each of those will have its own main config and can be enabled\disabled from some kind of mod manager separately.

3. Conflict resolution
Looks to be perfect system which will take infinite time to implement it correctly. As Warmonger said: “Details, please”. We need to have some basic system to see what actually causes conflicts and how they can be detected\fixed.

4. Resources
4.1 - This is pretty much what we’ll get in beegee’s system - all folders in Data (except Saves I think) will be part of our internal virtual file system which will support all those overrides.
4.2 - URI - bad idea, was already discussed some time ago.

7. Localization
There are already mature localization systems like gettext. Perhaps we should use one of those instead of writing our own homebrew system.

[AVS]

1. Files location
   I have a bit different proposition that will make mods more modular.
   Instead of plain “1 dir = 1 mod” every directory can have “submods” and each submod may have its own submods and so on.
   Each of those will have its own main config and can be enabled\disabled from some kind of mod manager separately.

Mods are too global to have detailed hierarchy, I think.

3. Conflict resolution
   Looks to be perfect system which will take infinite time to implement it correctly. As Warmonger said: “Details, please”. We need to have some basic system to see what actually causes conflicts and how they can be detected\fixed.

Automatic conflict resolution is long-term target. I wrote how to resolve resource|config conflicts, let script conflict question remain open.

4. Resources

4.1 Good, one point is partially implemented.
4.2 OK. Overridden version of resource become inaccessible.

7. Localization
   There are already mature localization systems like gettext. Perhaps we should use one of those instead of writing our own homebrew system.

Gettext is good for texts and what about graphics?

[Ivan]

LOD & may be own format in future

No more custom formats please. If we’ll need some kind of archive we can always use zip.

Mods are too global to have detailed hierarchy, I think.

What about WoG? Commanders, stack exp, new creatures, etc. All of this can be put in separate mods so one part won’t affect another.
Or modA can have compatibility patch with modB as submod.
This should be quite easy to implement and may be useful so why not?

Gettext is good for texts and what about graphics?

Good point. IIRC Wesnoth have support for localized images. Worth checking how it works there.
BTW - localization to specific language can be implemented as separate submod

[lord_haart]

I have read all previous posts and although I am not a programmer, but I would like to make some suggestions.

In wog all scripts can be turned on and off (some scripts are not compatible and there is a check that they were not included, well, you know), the scripts are divided into categories and pretty much still have the option to save the script settings in the file, and then load it.

I liked the system of mutators in the game unreal tournament (1999 issue), if you were playing, then you will understand. There used mutators, which are slightly changed gameplay (almost like a wog scripts), also sometimes was important the order in which mutators applied to the game. And they used to write a special language - unreal script. Of course there was mods for this game, but they usually were more global.

It would be nice to get the above strengths in VCMI. For example, would be a separate list of mutators for the game between man and man, the other for man and computer, and the third between man and computer, but at a high complexity. Of course, if we saving game, then we must provide for the saving applied mutator list, so you do not get confused. Yet it would be useful, but it will be done when the network part of the game, transmit such lists mutator (with options) on the network directly in the game. I think many wog options could be such mutators.

[AVS]

LOD & may be own format in future

No more custom formats please. If we’ll need some kind of archive we can always use zip.

Mods are too global to have detailed hierarchy, I think.

What about WoG? Commanders, stack exp, new creatures, etc. All of this can be put in separate mods so one part won’t affect another.
Or modA can have compatibility patch with modB as submod.
This should be quite easy to implement and may be useful so why not?

Gettext is good for texts and what about graphics?

Good point. IIRC Wesnoth have support for localized images. Worth checking how it works there.
BTW - localization to specific language can be implemented as separate submod

Localization as submod is a good idea. (And a good model: submod implicitly depends on parent mod, parent mod is not depends on submod) But dividing WoG into submods is not as useful as it seems, if we want to maintain compatibility with existing scripts|maps. They in fact depends on WoG in a whole, tracking detailed dependencies is a waste of time. (On the other hand, on implementation level such decomposition will be useful)

*I think, there is a terminology problem: module, namespace, component in case of C++, python, VCMI with mods in a whole.

In C++:
compilation unit (.cpp file) = module;
Namespase = just namespace - scope organization;
several classes, templates etc. acting as comprehensive whole = component (f.e. boost in a whole is component, separate boost library is a component too, they have corresponding namespaces, but it`s not mandatory, its just useful)

in python:
module = module AND namespace
package= component AND namespace (f.e. python libraries)

in VCMI in a whole:
to be decided

[AVS]

8. Maps

Map shall define required mods, but other mods can be manually enabled before game start.

8.1 Map itself is a mod too.

8.2 Map is packed in (zip) archive with or w|o compression. Each map archive:

1. shall contain map itself. One file H3M|H3C|Json format named map.*
2. shall contain main config file (as other mods). (mod description, release date and version are ignored)
3. may contain resources, additional configs, scripts as regular mods. They have highest priority.

[majaczek]

I think 7z would be better than zip.
it uses lzma algorithm, and has high compression ratio, fast decompress, and consumes LITTLE resources WHILE DECOMPRESSING.
well it consumes pretty much resources while compressing but it doesn’t matter.

7z has free packers and unpackers for both linux and windows, and sources are open (so for systems which isn’t default supported you can recompile it, or even you could include un/packer code in your open-source project)

[AVS]

I think 7z would be better than zip.
it uses lzma algorithm, and has high compression ratio, fast decompress, and consumes LITTLE resources WHILE DECOMPRESSING.
well it consumes pretty much resources while compressing but it doesn’t matter.

7z has free packers and unpackers for both linux and windows, and sources are open (so for systems which isn’t default supported you can recompile it, or even you could include un/packer code in your open-source project)

If 7z really faster in decompress than zip then it`s a good choice. But peazip.org/peazip-benchmark-3.html says that ZIP is 2 times faster.

[majaczek]

well it’s peazip program. native 7-zip binary would perform better for this format.
and yes it’s quick but not fastest. the real strenght is high compression ratio and low memory use on decompression (I believe a few times less than zip), so it’s client friendly.
the not-fastest decompress time is pay-off fro the two strenghts.

The beauty of 7z is you can compress it to the max on PC and then download and decompress on Mobile Phone without problem (mobile phones have low memory, and downloading from internet is matter of file size). Note that high memory use is only on compression, and it have perhaps the best compression ratio. I think 7z is best format for files comprssed once (a few times) and decompressed always (many times).

EDIT: notice that on those page test may be tendencial. each compress program has a counter where we can choose between better compress ratio or faster operation, and the tests doesn’t include value of the counter. if the decompress time is too much we can move counter a notch from default one. also on computer with no more than 256MB memory 7z will be fastest because of low memory use (and almost no swapping)

PS: the cpu use on decompress is also low. the 7z format has been thought to move the stress to the packaging, and have problemless decompression.