Hi, i saw many plans about adding scripting language like LUA or python, but none of them sadly seems to be realized. I had some idea`s about devloping python api for VCMI. Something like GemRB does, maybe not with ushc powerful control. For start i was thinking about exporting methods from CGameCallback to be scriptable in python 3. What do you think about it. Should scripts be executed on clientside or on server side? Are there any concept design documents about scripting language?
Depending on purpose of such script I guess. Usually - on server but GUI scripts should obviously be client-side:
- GUI script. Executed on client, provides features like “recruit all” button. Have access only to data known by player. Integrating such script with our GUI code may be most difficult part (a lot of old code in need of rewrite).
- Server script that changes gameplay. Such script would have full access to game state. Complex to implement since such script should be able to override game handler logic - access to game callback may not be enough.
- Some module:
-
- Scriptable map objects would be great and should be easy to implement - give access to callback and add some way to implement inheritance from our map objects (e.g. CGObjectInstance). Possible problem - AI would have no idea on what’s going on in such object. Providing info to AI would be responsibility of modder.
-
- Scriptable AI - I remember request for that but not sure if it is a good idea - we already have performance issues from C++ AI. Implementing AI in “slow” scripting languages may make this issue even bigger.
I don’t think so. So far no-one from our devs have shown interest in actually implementing this feature. We do have ERM parser/interpreter but actual script-game interaction was not implemented (this was university project of Tow).
If you think you can handle this - then go ahead.
Some suggestions:
- using wrapper like boost::python may be a good idea to speed things up.
- perhaps you should implement scriptable map objects as a first part. This is (relatively) small feature that was asked a lot so we can use it as our scripting showcase - to see how interaction works, what areas should be improved and so on.
boost::python was exactly of i was thinking with some addition of static module methods should do the trick. I`ll try to create some prototype of scriptable maps with OO access to game callback. How do you see integrating script with map? Writing python code inside events like it is for ERM or completely separate file in format {MAPNAME}.py? We can discuss details on private
For map scripts separate file is probably a better idea - right now plans for “VCMI map format” is to have .zip archive with several text/json files in it. So map script can be represented as separate file in such archive.
For now since we don’t have own map format I guess we should look for script in maps/.py as you suggested.
As for scripting API - callback is well, callback. Implementing python interface for it should be straightforward. But to use those script must be activated in one way or another (like via triggers). How are you going to implement them? Track received/sent network packages?
No need to hide this - most (if not all) of our discussions are public so let’s keep it that way. Besides - I’m not the only developer here.
-
Scriptable objects is a good starting point. Map objects - thanks to Ivan - just have been heavily refactored and on the whole are solid, sufficiently isolated part of engine.
-
Yes, most significant design problem with scripts is GameHandler: nearly any mechanics change requires overriding it
s functionality, but you cant override it in more than one class - it is singleton. Even if youll manage to do that (with f.e. single huge “proxy” and then providing API to several scripts for changing individual aspects) you will have to reimplement complex mechanics for just a tiny change. I have some ideas and plans now to solve that, but this a difficult task. -
Its better to get rid of GameHandler before trying to implement complete script support.
There are singinifact troubles concernning safety, synchronization, GUI and assymetric access between Client and Server. Tow explained some years ago why adding scripts is not a simple task and since then probably not much changed.
I agree, however, that adding simple scripts to support well-defined adventure map objects is resonable aim. They contain no interactive GUI and generate no further events, as for now.
Better to have barely useful Python API which waits for better days than to have none at all.
For now ill try to create scriptable extension of CGObjectInstance which will have scriptable its virtual methods. I guess with some twists its possible to call python implemented virtual functions seamlessly to game engine.
OK, sounds good. Although I suggest looking into IObjectInterface methods first - these are core methods needed for an object (including static callback field through which object may access game state). CGObjectInstance API needs cleanup - some fields that can be made virtual are not marked as such while some fields that should be internal are marked as virtual. Will fix this soon.
Another issue is that for anything related to granting creatures or fighting guards you need to use CArmedInstance. So it is a good idea to use it as base for “moddable object”. So inheritance should be like this:
CPythonScriptObject -> CArmedInstance -> CGObjectInstance -> IObjectInterface
We have more interfaces (f.e. IMarket) that may be needed by scripts.
Should we:
- merge all of them into IObjectInterface
- add all of them as CPythonScriptObject ancestors
- introduce some kind of Interface Querying (on the IObjectInterface level)?
Fun thing that I also have been thinking about that. In my case set of “interfaces” as of now consists of market, shipyard and army/creature set. Although “object modules” is probably a better term for them. Something similar to interface querying I guess.
Another related issue is that right now we have duplication between town buildings and adventure map objects - most of buildings have analog on map but their handling is usually separate. This is because town buildings can not derive from CGObject while map object must derive from it.
After that I noticed how our shipyard works - it derives from IObject but receives CGObject in constructor so it has access to CGObject without directly deriving from it. Perhaps there is some way to use such approach on bigger scale?
Each of such modules is stored as part of CGObject or IObject with method like tryGetInterface() to query for presence of specific interface. In the same time each module have access to its parent object (and all calls like onHeroVisit are transferred by parent to modules).
What next - I’m not sure yet. One approach would be to create set of such modules big enough to cover all generic functionalities another one is to convert all town buildings into such modules. It may be a good idea or it may be a bad idea - I’m not sure yet but it definitely sounds interesting for me.
Next? Adding new modules by scripts. It should be discussed early as it influence querying mechanism (at least static polymorphism - templates - will not be possible)
UPD. This is similar concept to new spell system which I`m designing (spell itself - SCpell - and spell mechanics are different entities)
For scriptable modules we can (at least at first) just provide them with common interface (like onHeroVisit() method) and be done with it - so this should be easy part.
But there are more than enough questions even without scripts:
- which objects should be turned into such modules?
- how objects should work after this change? Turning for example Shroud of Darkness into module for Necropolis means that map object would need changes - module can’t act as independent object.
- how objects should interact with such modules like army or harbor? For some modules common interface won’t be enough. For example to pass call like “create ship of type X” to harbor we would need to find harbor module first.
Probably better to split this discussion into separate thread.
How do you think should be new scriptable objects added? I started to implement it this way. The event on map contains in its message identifier like Python:ClassName and it identifies that in file {mapname.py} is its definition. I know itsnot elegant way to intercept the events, if you have better suggestions which will be easier to maintain in future, feel free to share. I see there is a comment about possible extension when creating TreasureChest object so maybe thats the spot.
For now we should stick with with new objects as opposed to editing/rewriting existing objects. And by new I mean object with its own type/subtype values.
To do this in a proper way you need:
-
Create own type handler - object that implements AObjectTypeHandler interface, see lib/mapObjects/CommonConstructors file for examples.
Key methods in this class:
void initTypeData(const JsonNode & input); <- this is when you will receive configuration of your object, including path to script.
CGObjectInstance * create(ObjectTemplate tmpl) const; <- this is when you need to create your object and pass script to it. Note that this function may be called before map loading so you should not access game state from it. -
Register your handler in CObjectClassesHandler constructor with name “scriptable”:
SET_HANDLER_CLASS("scriptable", CScriptableObjectConstructor);- Add code to load your object from H3M. Something like this should work:
case 140:
nobj = VLC->objtypeh->getHandlerFor(objTempl.id, objTempl.subid)->create(objTempl);
break;(This is ugly workaround since map objects refactoring is still not 100% finished. Later this won’t be needed)
- Download this mod and unpack it into Mods/ directory. You should end up with file “Mods/mapObjects/mod.json”. It already contains configuration for your object as well as test map that contains such objects but script is empty.
dropbox.com/s/tiwh29hs3e6i3 … tsMods.zip
If you wish I can make a more detailed guide on how to create such map but for now this should be enough.
Now start “New objects test Arrogance” map and you should have your object next to starting town (they use graphics from H3 subterranean gates)
(Actually there are 5 different objects, their type/subtype is written on nearby signs)
If something in our code does not works as expected or you run into some issues - upload you code to github so I can take a look on what’s going on here.
UPD: and make sure to register new file extension (".py") in EResTypeHelper::getTypeFromExtension method. Othervice VCMI will ignore such files during loading
Thanks, i managed already to register file type and invoke some arbitral scripts, but what you just showed is exactly what i need.
Small update from my side - fixes for H3M loader as well as CGObjectInstance API cleanup are now in github repo.
So you no longer need that H3M loader hack for new objects and CGObjectInstance API is back into stable state.
I managed to do some prototype, and here is what i got. With script like this:
import VCMI
class Scripted(VCMI.CGPythonObjectInstance):
def __init__(self):
super(Scripted,self).__init__()
self.messages="Oh, you scripted me!","Ha, scripted me again","Wow, such scripter","Stop it! Now!"]
self.i=0
def onHeroVisit(self,hero):
if self.i < len(self.messages):
VCMI.showInfoDialog(hero,self.messages[self.i])
self.i=self.i+1
You can view the effect on attached files. Nothing fancy, but from now it`s just matter of exposing different functions and classes, which will require some automation, to keep up with C API changes. I will loon on tools like Py++ or pygccxml. Also for python is better to keep callback as part of VCMI module than to hook to static fileds of IGameCallback but it is all to be discussed.
Report from work in progress.
[ul]
- Assumptions
[list]
a)There will be one built in mode named “vcmi”, available from every user script.
b)User scripts should be able to access only:
[list]
-Main vcmi module.
-Own mod module, and its submodules.
-Allowed builtin functions(“super”,“len”).
-Allowed python standard library modules.
[/ul]
c)User will have disabled import function, he will list used modules in json and have them imported via native code.
d)Provided library scripts will be precompiled(*.pyc) and their checksums will be hardcoded in vcmi to verify them.
e)If user wants to provide his custom class for mapobject he must list this class in object configuration, end write this class in his module like SomeObject(CGPyObjectInstance)
[/list:u] - Current state of work
[ul]
a)Added new mod element script:
[list]
"scripts":"path/to/your/scriptconfig1.json","path/to/your/scriptconfig2.json"]{
"scriptableObject" :
{
"index" : 140,
"handler": "scriptable",
"types" :
{
"testObject1" :
{
"index" : 0,
"class" : "Scripted"
}
}
}
}{
"root" :
{
"script":"scripts/objects.py",
"type":"classes",
"imports":"random","hashlib"]
}
}f)If module name is set to root it gets same name as mod. Other names are translated to modName.moduleName
[/ul]
b)Exported:
[ul]
-virtual methods of CGObjectInstace to be extended in python.
-CGHeroInstance but only class without methods, to suppress errors, because onHeroVisit is taking hero as parameter.
-Some methods of IGameCallack accessed as vcmi.getCallback()
[/ul]
c)Created singleton CScriptManager class responsible for all script operations, creating modules, creating and exporting objects, managing security by running untrusted code in separate namespaces.
d)On initialization CScriptManager stores normal python builtins as an object. Next it creates empty namespace and iterates over allowed builtins and sets them accessible to this new namespace.
e)When mod containing scripts is created new namespace with modName is created containing mod definitions and added to mod namespace which is member of vcmi namespace so it is accessible as vcmi.mod.modName
[/list:u]
- Todos
[ul]
a)Implement mechanism to load precompiled library scripts and verify them.
b)Automate python wrapper creation(CppHeaderParser)
c)Improve reference counting mechanisms to avoid future crashes.
d)Make code more readable. For example every python object is presented as boost::python::object. I think python modules should be wrapped to somehing like:
CPythonModule{
CPythonModule(boost::python::object){
this.object=object;
}
object getNamespace(){return object.attr("__dict__")}
};
e)Provide option to execute arbitrary python commands on runtime, maybe through chat menu?
[/ul]
[/list:u]
I listen to every ide how to improve this and what features will be useful. And if somedy know better tools to parse classes let me know. I already know pygcxml and Py++ but i cant get them to work.
Oh now, python!! Kill me!
PS Just joking. Will have to learn more about Python…
Will take a closer look on all of this tomorrow.
Perhaps we should turn this into a method of CGObjectInstance (or CPyObjectInstance) and export it?
Other types of scripts (e.g. GUI script) may have other means to access gamestate so exposing callback to all scripts may not be a good idea.
That would be cool, probably can be implemented as a “cheat code” that toggles python console on and sends all input to server for execution.