Unit tests

I think we need some kind of automatic testing, especially to catch regressions (and some future objectives - cheat proof and scripting - will require heavy refactoring).

I propose to use boost::test with Turtle( turtle.sourceforge.net/) as mock library.

I agree about the concept, but have no idea what and how to test. Heroes is a very complex game with lots of quirks and mutual dependencies, some issues can be observed just once in a lifetime :wink:

This just means that full test coverage of game mechanics will take much time, but not that testing whole mechanics is impossible. At first test should be written to

  1. currently fixing bugs (avoiding regressions is priority target)
  2. current refactorings (they are … dangerous w|o tests)
  3. recently added features
  4. known bugs

IOW write test for that you currently changing.

Hmm… I though about it many times. Every single time I concluded that our design of code (and the mechanics of game too) is way too coupled to have reasonable units with reasonable tests in most cases.

It doesn’t mean we cannot have any kind of automated testing. I’d suggest going into design-by-contract-like testing, with the possibility of two (or more) AI fighting each other to detect regressions automatically. A well-designed set of invariants (or semi-invariants) for classes would be a nice thing to have. Additionally we don’t test input arguments in every place we should do it – it should change.

Unit tests test single functions. To do so, we need coherent and admissible input for them. The simplest way to get it is to just run a game.

There are a couple of reasons to use unit testing in general. One important reason is to find bugs more quickly. For example, if you refactor some code, fix a bug or improve sth. in existing code, then you can run some unit tests. If these tests fail and due to you know what you’ve changed in the last few hours, it is very likely that you find the bug. Finding a bug is often difficult and time-consuming. To fix a bug is mostly easy. When unit tests can save time, then there very useful.

Sadly it is often not easy to find a piece of software where unit tests are reasonable and useful. There is the pareto principle which claims that only 20% of the source code are responsible for 80% of the bugs of a software. So there are mostly just a few classes which come into consideration to be tested by unit tests.

After the development of the new logging API, I’ll start to write a unit test for the CMapEditManager and its functionality to select the correct terrain tile view images. This part of software is very suitable for unit testing as it is complex and error-prone. If you change anything in this code or in the .json config files you mostly have no chance to test it extensive. It is a little bit like lottery. I’ll use boost.test as a unit testing library due to the reason we already have boost and it’s header-only. So there are no additional dependencies and this library seems to be fairly enough what we need. I’ll discuss details later:) It has to be clarified where to put test code (in subfolder test in the root source folder) and so on.

To get some feeling how I want to test the CMapEditManager, a few notes:

  1. A prepared map with a lot of different terrain view images serves as the base of the unit test. I’ll write down some positions in the map which I want to test later. The map resides in the /test/xyz folder and during the setup phase it will be copied to the /Maps folder. After the test I’ll be deleted.
  2. The map will be loaded at startup. A copy is made of the CMap object. It remains const and won’t be changed. The numbers which represent the terrain view images of the original CMap object are set to 0.
  3. The CMapEditManager calculates the terrain view images of the whole map.
  4. Now we come to the important part. Terrain tiles at specific locations come into question to be compared, those which I’ve chosen in the map editor before. The integer value of the re-calculated terrain view and the integer value of the const terrain view are compared. If it’s the same it’s ok and all is fine.

High coupling is second candidate for refactor (after code duplication). And yes, integration of unit tests will require much changes to codebase, don`t be afraid of it - this is a price of quality.

AI fighting each other is good but its rather time-consuming operation. Its better to have more than one level of testing (unit tests on function/class level, integration tests on component level and acceptance tests - this is the place for AI battles - on entire engine level). Unit tests + integration tests should have 100% coverage together.

Having X levels of testing and Y layers of abstraction is of course possible, but I’d like to see one example of AI tests that actually does test something uncommon and is not a simple assert or output log. Anyone?

I’m just not convinced that the quality gain would be significant enough to justify the enormous work that needs to be done. Anyway, it’s more of excuse for not wanting to do it myself than discouraging you from doing it. And lowering coupling… it’s a tricky thing. I’d advise to find a way to live with high (even if a bit lower than now) coupling either way.

I’d estimate that VCMI with good coverage of unit tests would take comparable amount of time to be tested.

We can prepare a predefined battle file (we have working AI-only duels mode) and implement special AIs that execute predefined set of actions. It shouldn’t be that hard to cover practically all battle mechanics. (The only issue is the non-deterministic character of battle but that can be easily solved by using a single generator and predefined seed.)
AI shall call various callback methods and check if the results are valid plus validate if battle goes as expected.
I believe something like that would be much easier to implement than tests giving comparable coverage and help more in detecting bugs.

Though the same can be done with adventure map, that eg. visits all kind of objects and checks the results. I guess we can test that way pretty much everything except of GUI.

I’m not sure how much could be achieved with unit tests, I think that most of bugs are rather in bad cooperation between various elements, not the elements itself.

I’ve added a separate project /test for unit testing. It uses the Boost.Test library and adds no additional dependency. There is a CMakeLists for the CMake users to compile and execute tests. Make sure that you set ENABLE_TEST variable to enable testing. A project file for Visual Studio is missing. If there is anyone who wants to create one and setup unit testing, you can take a look on this:
blog.yastrebkov.com/2010/07/boos … usage.html