Modding system discussion

#1

Today I started writting down the concept of Modding System. The amount of text will grow rapidly, so the article is placed on Wiki for clarity and convenience.

Of course, all suggestions, questions and implementation proposals are mostly appreciated.

#2
  1. Intermediate engine-script layer
    Will be good to have exported C (not C++) interface allowing writing mods in almost any language (anyway, I agreed that is enough to have build-in scripting support only for ERM and Python)
  2. Direct memory access is often used in ERM, and is not impossible to partially support it by mapping UN:C f.e. calls to safe access to game objects (which by-turn may be implemented in python). I dont think that it should be implemented by team, but should be stated that its possible.
  3. Defining all new|modified game object directly in script code may be significantly slow. A suggest use json configs for them.
#3
  1. We can always create any number of layers and levels of abstraction, but it’s nice to finally have something actually working and useful.
  2. Ok, makes sense if you know it by heart :wink:
  3. Not all behaviour cna or should be defined by set of parameters. Some of them, especially adventure map objects, need to have custom script in form of script.

Also, using script as a universal way of describing mods may be convenient for users. You may clearly tell what happens when you append more scripts to the file, but it’s less intuitive what happens when you add more configuration files to already existing pile of configuration files.

#4

If WoG will be mount as mod for vcmi, vcmi will require SoD and wog data will be repackaged in vcmi mod format?

#5

I’ve thought of creating such a thread for a few months, but haven’t had time for it. We should talk about this issue thoroughly. It will have a major impact on future development of VCMI.

In my opinion there should be 3 different ways of extending VCMI.

  1. Technique: Resource override mechanism. A file called CORC.DEF in folder /Mods/[MOD_NAME]/… overrides original CORC def file for example.
    Features: + add/replace animations, bitmaps, music, sounds, campaigns, config files(creature stats)… => even a new town/faction could be added like this way(everything should be layed out in config files,…)

One problem: If two mods want to change in each case one different creature stat(e.g. attack), it will result in a conflict.
Simple solution: a) You should choose either mod A or B. b) File diff tool could merge those different two config files(automatically, at vcmi startup) c) Those two mod developers should work together:)

For config files the add_ prefix can be used to expicitely say that you only want to add a few lines to original config file. e.g. add_creatures.json

  1. Technique: Python support only as a CAMPAIGN/MAP scripting language like ERM.
    Features: Scripted events(adventure map, battles), simple actions should be possible like the following: give resources from player A to B under XY pre-conditions, teleport hero from location A to B,… don’t have enough ideas :slight_smile:

  2. Technique: Extending the core of VCMI. If a modder wants to add reasonable, smaller and mostly non-gameplay-changes like improving the minimap(toogle on/off enemies), adding the possibility to build troops on a new week siege battle then the modder should be allowed to make a branch and then commit this to trunk. This shouldn’t be done via Python or CPP Reflection or anything else as such a possibility would be nearly impossible to implement, worrysome, buggy,… Such mods like anything else which differs from OH3 should be made completely optionally. Useful good implemented mods with a high quality(graphical quality too) should be activated by default. (there are shipped with vcmi)
    Mid-controversial mods like stack experience, wog features should be integrated in trunk too, but not activated by default.
    Controversial mods like re-building a town from scratch which you could done via WOG shouldn’t be supported at all in my opinion. As such things would blow up our source code significantly. => they should program a fork of VCMI then…
    Features: CPP land. Everything is possible.:slight_smile: Every change to trunk should be discussed, so you can’t work completely independently.

These 3 points should be relatively straightforward to implement. These techniques are robust and simple to understand from a modders point of view. Point 1) should be implemented in a couple of weeks.:), for Point 3) we have to implement nothing specific. A mod browser, conflict checker and such things are needed for point 1 and 3. Python scripts are no real mods, they extend the OH3 MAP format possibilities.

#6

begee I agree mostly but (1) and (2) should be proxied through nice mod system, so uninstalling a mod wouldn’t be a hell, changes will be grouped into packages to maintain a whole package at a time, and to be posssible to enforce compability between clients while still allowing for skinning.

I agree mostly for (3) but i think it should be splitted between things that gets incorporated into client/server executable and things added via dynamic libraries (.dll, .so) like new AIs and all things that can be easily mounted into engine without to have recompile client/server executable

#7

Some interesting links:
forum.df2.ru/index.php?showtopic=26467 new version of Era - 2.0 ENG (most interesting is that WoG is now a plug-in)

wogarchive.ru/file.php?id=189 full sources of WoG.

#8
  1. nearly all new factions|towns adds new creature abilities at least - impossible to implement w|o programming.

  2. IMO if we support such language like python, we can rewrite all mechanics in it. Only time-critical parts should remain in C++ with the lapse of time.

#9

@AVS

  1. I’m actually against this. Python (or lua) + ERM (for WoG compatibility) should be more than enought for most modders + less languages we have more easier creation of guides, examples, support, etc will be.
  2. Agree with this one. Configuration files are more easy to edit + at some point we’ll be able to add some gui for it.

Checked wiki:
@Mod system proposal
Conversions - may be possible to replace with incomplete config files. Graphics can be replaced on filesystem level (can be done with beegee’s system)

Language - python is OK but what about lua? IIRC it is used in HoMM5 and gaming in general - so some modders may know it already.

New map format - care to fix the red link? :wink:

Scripts online - what about separating server-side and client-side scripts + allow some interaction with them like sending python data structures over network? Will work in both Single and Multi as long as mods are in sync.

@Packages
main file IMO should be JSON which specify language, script files (if any) along with name, version, description, etc.

New objects:
Json may be better here - I don’t like specifying name\description\images in code. For complex objects modders may specify name of python class with required actions.
Some of our c++ classes can be rewritten in python - will work as examples for modders + allow some un-hardcoding of mechanics.

@Mod handler
Apart from prerequisites there should be list of conflicting mods.

Object identifiers - numeric ID will be needed for SoD objects but what about using strings as ID instead of number at least for objects added in mods:
WOG.creatures(“supreme archangel”)
this ID should either match name of python class or specified in config file. (separate from readable name to allow localization). May even work with subID’s. For example new object with id = “Mine” and subID=“Mithril” can be added by engine with same ID as H3 mine but new subID.

@ERA
Looks to be simple resource overriding. We’ll need more than that. Nice mod BTW.

#10

@Ivan@scripts_online
i was first (see wiki discussion) but of course totally agree

#11

@Ivan

If use string IDs we should be of format.
Capital case with “_” and w|o spaces, camel case etc. Or engine should handle most of all cases (“supreme archangel”=“supreme_archangel”=“SupremeArchangel” …)

#12

Thank you for your contribution, this discussion makes more sense than I expected :smiley:

  1. Lua is popular, but already obsolete. It doesn’t seem to be developed much further.
  2. Python has more advanced features, like explicit class and metaclass support, which may be useful for modding.
  3. Python comes with tons of documentation, examples and rich library in case you may need to write something ambitious.

Lua implementation in H5, for example, was just a set of fixed functions… and you couldn’t go beyond them with your own code.

I agree with that. Json files may be easier to scan and manage in first place.

That makes sense in general, but there’s an issue: client-side scripts (GUI) which result in change of game state, such as garrison drag-and-drop. This must be done in a smart way and I can’t help much with network programming.

Well, I believe some definictions can be conditional or generated at run time - for example, in a middle of a map you may want to create new artifact depending on player’s choice. Or multiply certain type of creature, each with new abilities.

#13

Every mod resides in its own folder under [game_dir]/mods/… Uninstalling shouldn’t be a problem. Original files remain unchanged.

For AI it may be OK. Client/Server/Lib executables shouldn’t be replaced by mods. This would be the false direction. How to support several mods instantly then?

No problem. You can use both point 1(resource override) and point 3(changes to trunk).

AFAIK python can be connected with C with just a macro in the class/function definition. So calling a C++ function from Python shouldn’t be a problem. The big problem is that we have to define places in VCMI code where we are likely to expect an addition to the normal behaviour(invoke predefined python methods from cpp). This can be a major task. Don’t know if data type conversions will cause trouble. It would be nice especially for this discussion if someone could look how VCMI CPP/Python Integration could be done. Which steps are necessary exactly to call a function from cpp and the other way round? Where are performance bottlenecks? Which other risks are there? How could a creature ability be added? What is easily possible, what takes more work?
Developing mods in Python, than directly in c++ is more complicated as there is one additional layer anyway. Python as a map scripting language seems to be quite fine, but don’t know if it makes sense to introduce gameplay changes which affects lib, client, server and ai.

This makes sense. We should define a limited set of method declarations which can be overriden by Python mods dynamically. That methods have to be invoked from VCMI Core at specific positions.

#14

About defining objects in python:
RMG & map editor shouldn`t have dependency on python engine. Every object in mod should be accessible w|o executing scripts.

#15
  • Give to scripts access to CCallback - will allow client-side scripts to safely modify game state just like in our C++ gui code.
  • For interaction between client and server parts of mod something like this should work:
  1. on client convert python data to c++ string (in scripting system wrapper)
  2. client sends message (modID + string) to server
  3. convert c++ string back to python data on server
  4. pass received data to python script on server

I know that there is Boost.Python which may work for us. Don’t know anything about this lib though.

Asking modders to use C++ is even worse - it almost requires some computer science knowledge. Writing every ability in C++ by VCMI devs isn 't an option too.
Using python for gameplay mods shouldn’t be that difficult - we already have clearly defined events (net packs). All we need is way to pass this events to scripts.
For example some kind of “Random encounters” mod will have script attached to “Hero moved” game event and will be triggered by engine each time a hero moves.

#16

I vote YES for python.
I vote for Dynamic Library Plugins also (similair way as AIs are done in engine) with dynamic loading (you may enable/disable module)

“Things Integrated In Client/Server Executable” in my sentence was the same as your “Include major mod mods in trunk” but worded different (and also allows for branches/forks). Who for the hell would be patching binary code if we have the source open for everybody :stuck_out_tongue: ?

#17

@JSON
i think there should be configuration files as for map editor independence from python, but i think there could be a few types of configuration files

  1. those loaded at mod loading (name it “autoloaded settings”)
  2. those loaded on demand by python script function (or API call for binary plugins), call them “dynamic settings”
  3. those loaded on demand to patch any of (1) (2), call them “settings patches”

since all config can be autloaded and demand-loaded there is no reason to not define objects directly from python code without use of JSON files (but those objects would be unavailible in map editor until we have placeholder objects in JSON configuration which will be changed in runtime)

#18

You’re right! I’ve said above in point 2 that we should use Python only as a map scripting language, but I think for specific purposes we should provide modders a easy way to realize the most common needs. The interface and the framework should be documented, clean and stable. With those Python extensions you should be possible to do e.g. such tasks(common tasks for 90% of modders):

  • Add/modify creature abilities
  • Add/modify secondary skills
  • Add/modify (special) artifacts

In order that this works well we should implement standard creature abilities in python too. I think this would make development more straightforward.

If someone wants to do greater modifications e.g. to increase the size of the battlefield, introduce a new skill system, town conversion,… we should not blow our code that such things would be possible in python. As this would be a major task and combined with much probability. What if the modder wants then to have feature XY to be moddable? Then we’ll come to point 3, where an experienced modder could implement that features what he wants directly in the source code. Therefore as said in my first post the changes should be of high quality, stable, separated from OH3 logic and optionally.

#19

The problem is that development of these abilities was NOT straightforward. Each of them needs an access to callback for checks and triggers or condictions placed in the very different places of code. Adding remaining spells will also require brand new pieces of code and reorganization. These mechanics are simply very complex and hard to plan in advance.

On the other hand, Nix’s ability from HOTA is one-liner in current C++ code.

Ahh, so how’s the development of siege going? If you had a look at that code, you’d see how complex it is and how many things are fixed or hard to figure out at all. It can’t be done “from outside” as the code inside is a real puzzle on its own.

#20

In order that we can provide ability modding we have to re-arrange a lot of source code. But perhaps it may be possible. Even for spells. My initial thought was that it is by far too much complicated to do such gameplay mods(spells,abilities NOT radical changes!). Not only for us developers, but for modders too. They have to override several methods to get this working. That’s why I’ve thought that python would be used best as a map scripting language only where you could do simple but more advanced things compared to what OH3 offers. I don’t want to step to deep into technical details, as I just wanted to talk about the rough picture of the modding system. So HOW gameplay mods could be realized from a technical point of view, needs much further investigation. BUT adding new abilities may be more probable as it’s more centered and bordered than town conversion or such things.

:mrgreen: So true! Siege and battle interface code have to be re-factored anyway. Even a perfectly modular written code wouldn’t allow easily a greater battlefield without any changes to source code in that case.