Forum index VCMI Project - Heroes 3: WoG recreated
Forum of the project aiming to recreate best turn-based strategy ever!

FAQFAQ  SearchSearch  MemberlistMemberlist  UsergroupsUsergroups  StatisticsStatistics
RegisterRegister  Log inLog in  AlbumAlbum  DownloadDownload

Previous topic :: Next topic
Better serializer for network protocol
Author Message
heretic13

Joined: 20 Nov 2016
Posts: 2
Posted: 2016-11-20, 13:34   Better serializer for network protocol

I spent several hours on the study of the serializer vcmi. And I want to say that it looks good for academic research, but it looks just awful for real projects.

1. It is support different binary architecture (Little Endian & Big Endian). 21 th century and a strange protocol. All binary network protocols set hard rule - all data must be in Little Endian or Big Endian, but not variable type.

2. It is very rigidly dependent on the C ++ language. This is the second protocol on my memory with tight binding to the language (the first was Freeciv and the C language). And this is not praise.

3. Serialization pointers and smart pointers in both directions with different types and ability disables serialization. I guess what it is, but it broke my brain.

Why do so hard ???

My suggestion:

1. Use a text protocol. As containers used json (benefit json serializer you already use).

Network packets can be split by byte zero (text does not contain zero bytes).

2. Pass only what you need! No pointer serializations that can be switched off.

Benefits:

1. Detaching from the processor architecture. No headache (Little Endian or Big Endian).

2. Extensibility protocol. Inside the binary protocol harder to add a variable. Inside json-object additional variable is inserted easily.

3. Readability protocol. All developers will thank you for it in the debugging process, because the json - a very simple format and easy read by human.

4. Unbinding from C ++. Minimum templates. Minimum modern C ++ 11 features in the base serialization.

Disadvantages:

1. Increase the overhead of serialization. But we all have modern computers, is not it? We do not use "coffins" of the 70-ies of the last century. We write turn-based strategy, rather than on a microcontroller program, where a delay of 1 microsecond can be critical.
 
     
Warmonger 
VCMI programmer


Age: 28
Joined: 07 Jun 2008
Posts: 1545
Location: Warsaw, Poland
Posted: 2016-11-20, 14:18   

Quote:
And I want to say that it looks good for academic research, but it looks just awful for real projects

From a developers point of view, the ability to serialize polymorhpic objects and smart pointers without backend considerations is very convenient. It's also extensible.
Quote:
Inside the binary protocol harder to add a variable. Inside json-object additional variable is inserted easily.

We do not extend anything in binary, we extend it on high-level - just where game mechanics belongs. Json would be just another layer of indirection which we would have to write by hand and parse again. If it isn't broken, don't fix it.
Quote:
We write turn-based strategy, rather than on a microcontroller program, where a delay of 1 microsecond can be critical.

Turn-based is what makes this game unpopular, especially in multiplayer. We wanted to implement simultanous turns with cheat-proof satefy checks, so quick transfer over network is important.

Also keep in mind that serializer til now was used mainly for game saving and loading, since multiplayer is officially not upported.

Not to mention serializer was originally written many years ago, as a basis of the engine.
_________________
Think twice if you really need to send me private message. Use public forum for general questions.
DJ Warmonger blog
beegee wrote:
Warmonger, you are the best!
 
 
     
heretic13

Joined: 20 Nov 2016
Posts: 2
Posted: 2016-11-20, 14:42   

Warmonger,

I have one more question.

You have two main projects - the client and the server.
Are there any plans to completely separate logic on the client and server?
Example:

Client - display & sound logic (load multimedia resources), clicks logic and communicate with the server. Nothing more.

Server - implementation mechanics of the game (loading resources required for this, but not for display), random map creation, communication with clients. Nothing more.

I mean that even for a single game client will need to run the server to perform the mechanics of the game.

This greatly simplifies the writing of clients.
 
     
Warmonger 
VCMI programmer


Age: 28
Joined: 07 Jun 2008
Posts: 1545
Location: Warsaw, Poland
Posted: 2016-11-20, 16:52   

Quote:
Are there any plans to completely separate logic on the client and server?

Yes, that's how it was done by design.

The issue is that if all game logic was checked only on server side, client would be completely unresponsive when playing online. Imagine when you want to rearrange the army and need to wait until server allows client to enable interface. That is why such checks are doubled on client and server side.
_________________
Think twice if you really need to send me private message. Use public forum for general questions.
DJ Warmonger blog
beegee wrote:
Warmonger, you are the best!
 
 
     
SXX 
VCMI programmer

Age: 25
Joined: 04 Jul 2014
Posts: 389
Posted: 2016-11-20, 17:00   Re: Better serializer for network protocol

I'll also comment on this one, but keep in mind I'm not talking from standpoint of programmer, but as someone who want to get VCMI into playable state ASAP.

heretic13 wrote:
I spent several hours on the study of the serializer vcmi. And I want to say that it looks good for academic research, but it looks just awful for real projects.
One and most important advantage our current serialization have: it's actually working with almost no effort. Any change in that area would require massive engine rework that will probably throw us back to many months.

Another nice feature is that current serializer will let us easily implement passing whole gamestate over network on client reconnection or host migration.

heretic13 wrote:
1. Use a text protocol. As containers used json (benefit json serializer you already use).
Network packets can be split by byte zero (text does not contain zero bytes).
Sorry, but the last thing we need is our own implementation of some new protocol. If we ever decide to rework whole serialization we'll likely use something like protobuf.

heretic13 wrote:
Increase the overhead of serialization. But we all have modern computers, is not it? We do not use "coffins" of the 70-ies of the last century. We write turn-based strategy, rather than on a microcontroller program, where a delay of 1 microsecond can be critical.
Keep in mind that VCMI is developed by mortals and it's not too hard to make inefficient code that will be slow on any hardware. In fact VCMI is slow enough already.

For instance things like AI require a lot of CPU time especially since AI also work over network and I personally don't want this to change. So serialization must be efficient.
 
     
SXX 
VCMI programmer

Age: 25
Joined: 04 Jul 2014
Posts: 389
Posted: 2016-11-20, 17:28   

heretic13 wrote:
Client - display & sound logic (load multimedia resources), clicks logic and communicate with the server. Nothing more.

Server - implementation mechanics of the game (loading resources required for this, but not for display), random map creation, communication with clients. Nothing more.
While Warmonger already answered there is two different approaches and likely both going to be implemented at some point. From one point it's would be good to move everything to server include some way to attach some headless client for AIs there. This would help us avoid dependency on "host client". Also we can apply gamestate changes on server only and then sent diffs to client. That let us avoid many issues like RNG platform dependency (we do have issue with floats right now, MSVC and GCC have different results).

Though in same time it's would be great to keep server as lightweight as possible. When server use just 20-30MB and use very little of CPU it's can be hosted anywhere. If we ever going to have active multiplayer this would be really helpful since then we can for instance host a lot of games with very little resource usage.

heretic13 wrote:
This greatly simplifies the writing of clients.
Do you know many open source games that have more than one client? Any games at all? That's would be really advantage that have no practical use at all.

Nowadays you can just compile normal C++ app into web application so there is very little reason make client completely standalone. And if you want think about using it for writing AI client in non-C++ then it's will be much easier to just create bindings to for current client.

And what I really don't get why you want client that is not in C++ at all? Even if VCMI will eventually replace original H3 completely for whole H3 community there is like 0% chance it's going to be base to non-H3 game.
 
     
Display posts from previous:   
Reply to topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum
Add this topic to your bookmarks
Printable version

Jump to:  

Powered by phpBB modified by Przemo © 2003 phpBB Group

Hosting provided by DigitalOcean
Page generated in 0.03 second. SQL queries: 12