Better serializer for network protocol

I spent several hours on the study of the serializer vcmi. And I want to say that it looks good for academic research, but it looks just awful for real projects.

  1. It is support different binary architecture (Little Endian & Big Endian). 21 th century and a strange protocol. All binary network protocols set hard rule - all data must be in Little Endian or Big Endian, but not variable type.

  2. It is very rigidly dependent on the C ++ language. This is the second protocol on my memory with tight binding to the language (the first was Freeciv and the C language). And this is not praise.

  3. Serialization pointers and smart pointers in both directions with different types and ability disables serialization. I guess what it is, but it broke my brain.

Why do so hard ???

My suggestion:

  1. Use a text protocol. As containers used json (benefit json serializer you already use).

Network packets can be split by byte zero (text does not contain zero bytes).

  1. Pass only what you need! No pointer serializations that can be switched off.

Benefits:

  1. Detaching from the processor architecture. No headache (Little Endian or Big Endian).

  2. Extensibility protocol. Inside the binary protocol harder to add a variable. Inside json-object additional variable is inserted easily.

  3. Readability protocol. All developers will thank you for it in the debugging process, because the json - a very simple format and easy read by human.

  4. Unbinding from C ++. Minimum templates. Minimum modern C ++ 11 features in the base serialization.

Disadvantages:

  1. Increase the overhead of serialization. But we all have modern computers, is not it? We do not use “coffins” of the 70-ies of the last century. We write turn-based strategy, rather than on a microcontroller program, where a delay of 1 microsecond can be critical.

From a developers point of view, the ability to serialize polymorhpic objects and smart pointers without backend considerations is very convenient. It’s also extensible.

We do not extend anything in binary, we extend it on high-level - just where game mechanics belongs. Json would be just another layer of indirection which we would have to write by hand and parse again. If it isn’t broken, don’t fix it.

Turn-based is what makes this game unpopular, especially in multiplayer. We wanted to implement simultanous turns with cheat-proof satefy checks, so quick transfer over network is important.

Also keep in mind that serializer til now was used mainly for game saving and loading, since multiplayer is officially not upported.

Not to mention serializer was originally written many years ago, as a basis of the engine.

Warmonger,

I have one more question.

You have two main projects - the client and the server.
Are there any plans to completely separate logic on the client and server?
Example:

Client - display & sound logic (load multimedia resources), clicks logic and communicate with the server. Nothing more.

Server - implementation mechanics of the game (loading resources required for this, but not for display), random map creation, communication with clients. Nothing more.

I mean that even for a single game client will need to run the server to perform the mechanics of the game.

This greatly simplifies the writing of clients.

Yes, that’s how it was done by design.

The issue is that if all game logic was checked only on server side, client would be completely unresponsive when playing online. Imagine when you want to rearrange the army and need to wait until server allows client to enable interface. That is why such checks are doubled on client and server side.

I’ll also comment on this one, but keep in mind I’m not talking from standpoint of programmer, but as someone who want to get VCMI into playable state ASAP.

One and most important advantage our current serialization have: it’s actually working with almost no effort. Any change in that area would require massive engine rework that will probably throw us back to many months.

Another nice feature is that current serializer will let us easily implement passing whole gamestate over network on client reconnection or host migration.

Sorry, but the last thing we need is our own implementation of some new protocol. If we ever decide to rework whole serialization we’ll likely use something like protobuf.

Keep in mind that VCMI is developed by mortals and it’s not too hard to make inefficient code that will be slow on any hardware. In fact VCMI is slow enough already.

For instance things like AI require a lot of CPU time especially since AI also work over network and I personally don’t want this to change. So serialization must be efficient.

While Warmonger already answered there is two different approaches and likely both going to be implemented at some point. From one point it’s would be good to move everything to server include some way to attach some headless client for AIs there. This would help us avoid dependency on “host client”. Also we can apply gamestate changes on server only and then sent diffs to client. That let us avoid many issues like RNG platform dependency (we do have issue with floats right now, MSVC and GCC have different results).

Though in same time it’s would be great to keep server as lightweight as possible. When server use just 20-30MB and use very little of CPU it’s can be hosted anywhere. If we ever going to have active multiplayer this would be really helpful since then we can for instance host a lot of games with very little resource usage.

Do you know many open source games that have more than one client? Any games at all? That’s would be really advantage that have no practical use at all.

Nowadays you can just compile normal C++ app into web application so there is very little reason make client completely standalone. And if you want think about using it for writing AI client in non-C++ then it’s will be much easier to just create bindings to for current client.

And what I really don’t get why you want client that is not in C++ at all? Even if VCMI will eventually replace original H3 completely for whole H3 community there is like 0% chance it’s going to be base to non-H3 game.