Forum index VCMI Project - Heroes 3: WoG recreated
Forum of the project aiming to recreate best turn-based strategy ever!

FAQFAQ  SearchSearch  MemberlistMemberlist  UsergroupsUsergroups  StatisticsStatistics
RegisterRegister  Log inLog in  AlbumAlbum  DownloadDownload

Previous topic :: Next topic
SoD files mod like
Author Message
krs


Joined: 04 Sep 2012
Posts: 374
Posted: 2013-06-06, 10:01   SoD files mod like

Now that the modding system is in place wouldn't be nice to have a SoD in a mod format that has nicely organized all the original H3 data in VCMI mod format?

Here my idea: Upon first run VCMI processes the files from the /Data folder and moves them to appropriate locations. Then the original packages are just discarded.

I've already done some experiments and grouping the files to the right folders is not that complicated. Theoretically you just need some json files (that in part exist) to map the resources to the right places and some simple functions to move things around.

Advantages:

- there are many duplicate files in the game because of AB. Getting rid of those would speed up loading times.

- there are tons of quite large campaign specific files that serve no purpose when you are not campaigning.

- modding would become more straight forward for inexperienced users (like me). You will not have a huge pile of files with cryptic names and sometimes content thrown together in 1-2 compressed folders.

- special tools for packing/unpacking stuff would be drastically reduced.

- we will have a base for mods derived from original factions.

- things from original H3 could be turned on and off again :) like any other mod.

The result would be a nice clean structure that would improve modding experience under VCMI by a ton.
 
     
Ivan 
VCMI programmer

Age: 25
Joined: 08 May 2009
Posts: 1438
Location: Ukraine
Posted: 2013-06-06, 11:07   

Quote:
Theoretically you just need some json files (that in part exist) to map the resources to the right places and some simple functions to move things around.

In part? What's missing? Right now mods share quite a lot with main game:
- filesystem: mod.json may have "filesystem" entry instead of Content folder (default). VCMI uses config/filesystem.json for this.
- json configs: All mod formats on wiki are also used by main game. Added in 0.93 config overriding used here as well - H3 configs are converted into json and merged with our data in config directory.
Quote:
- there are many duplicate files in the game because of AB. Getting rid of those would speed up loading times.

Are you sure? IIRC most costly operation here is scanning directories for files. Parsing archives is much faster than that.
Quote:
- things from original H3 could be turned on and off again :) like any other mod.

This part is not as simple as it looks like. If (for example) Dungeon is removed from the game what would happens if player will load map with Dungeon? Or with dungeon creatures? Or with Medusa Bank?

Besides - I'm not sure if VCMI can handle "missing" objects - ID's of H3 objects are hardcoded and must be continuous.


IMO moving some of config files into "core" mod may be a good idea but everything else should be kept as it.
 
     
Macron1 

Joined: 02 Apr 2013
Posts: 576
Posted: 2013-06-06, 11:22   

Ivan wrote:

This part is not as simple as it looks like. If (for example) Dungeon is removed from the game what would happens if player will load map with Dungeon? Or with dungeon creatures? Or with Medusa Bank?

Bank system is not modded yet, right?

And what's the problem?
If users make standart towns external, they will prepare to not play carts made for turned-off factions.
In part of towns I agree that standart towns/creatures must be moved to mods. Only difficulty is that now some abilities are hardcoded.
In this realization we can improve mod management system - it will begin being helping, when there is a chance to turn off towns.
_________________
I'm not a member of VCMI developer group and my posts are not official. I'm just a fan.
 
     
krs


Joined: 04 Sep 2012
Posts: 374
Posted: 2013-06-06, 13:18   

Ivan wrote:
In part? What's missing? Right now mods share quite a lot with main game:


I am talking here about campaigns and a way to distinguish some of the sound files, like creature abilities and interface/battle sounds.

Quote:
Quote:
- there are many duplicate files in the game because of AB. Getting rid of those would speed up loading times.

Are you sure? IIRC most costly operation here is scanning directories for files. Parsing archives is much faster than that.


Here I can only guess what the difference in speed between removing duplicates and spliting the rest into several separate archives over having all in one place. Maybe it will increase, maybe it will decrease but you will have the chance to easily turn on and off again individual parts that you do not need.

Quote:
Quote:
- things from original H3 could be turned on and off again :) like any other mod.

Besides - I'm not sure if VCMI can handle "missing" objects - ID's of H3 objects are hardcoded and must be continuous.

I think this can be/should be fixed. ( I for one would very much like to play without sharpshooters and enchantress).
Disabling original towns and factions should be supported by VCMI. Since a totally separated faction like Cove is possible why shouldn't the rest of factions be treated the same way?

Quote:
IMO moving some of config files into "core" mod may be a good idea but everything else should be kept as it.

Here we disagree. A structured and ordered filesystem (like the one in place for cove town) would help modders and new developers in better understanding the components of the game and will ease up their work a ton. Like showed in Cove already it is only natural to store creature sounds/ animations/ stats in a common folder.

Mods are the driver force for this game, the life of modders should be made as easy as possible.

Quote:
IMO moving some of config files into "core" mod may be a good idea but everything else should be kept as it.


What is the motivation behind this? What are the downsides? I can think only of upsides.
 
     
Ivan 
VCMI programmer

Age: 25
Joined: 08 May 2009
Posts: 1438
Location: Ukraine
Posted: 2013-06-06, 17:35   

Quote:
If users make standart towns external, they will prepare to not play carts made for turned-off factions.

Arrogance, aka "default map" requires at least Necropolis and Conflux despite not being made for specific faction(s).

Disabling one of standard factions will break huge number of standard maps. And there is no way to determine what factions are used on map without starting it first.

While I'm not against concept of disabling factions or moving them into mods I don't think that we can implement this right now without running into endless bugs.

Alternative solution is to implement field "special" for towns. But it also have some limitations - random towns and dwellings have set of allowed factions. What should we do if none of these factions are available?
Quote:
Maybe it will increase, maybe it will decrease but you will have the chance to easily turn on and off again individual parts that you do not need.

It most likely will increase CPU usage during loading while insignificantly decreasing memory usage (10-20 bytes per file). So there is no point in disabling these parts - when unused they have no notable effect on speed.

IMO we should keep installation as simple as possible. Meaning that all data should be used as it without conversions.
Quote:
I think this can be/should be fixed.

Feel free to make patch for this. From my point of view this is not essential feature so I'd rather spend my time on other areas.
Quote:
What is the motivation behind this? What are the downsides? I can think only of upsides.

And what are your upsides? Especially regarding better filesystem?

- Keep it simple. Such rearranging of files means more code to write, more code to support (more bugs! yay!).

- One of main ideas of modding system is that modder does not need to know how H3 filesystem works. You want to change creature animation? Replace entry <creature>/"graphics"/"animation" - no need to guess name of that .def or finding out where that data is stored.
So such rearranging should not be needed for vcmi.

I agree that some areas like GUI still have "guess file name" problem. But that's mostly because we don't have real GUI modding right now.
 
     
krs


Joined: 04 Sep 2012
Posts: 374
Posted: 2013-06-06, 19:05   

First to make 1 think 100% clear. There are no conversions involved! Data remains in the same format and with the same names but not in 4-5 chunks instead it is moved in the proper places.

Cpu usage will increase a lot when you will ad zip to mods. But that is the whole point. By making usage of more CPU power you gain ~2-3 times more speed when loading files. But you have a point without testing it is hard to say if less files but more disperse will make it faster or worse. So me "guessing" you will actually gain some speed could be wrong.

The whole disabling factions/creatures/towns thing is a bonus that people could use. I see it come into play very nicely in RMG for example.

Another disabling usage would be in mods with lot of contend for existing towns. Let's say you want to mod Castle and touch every creature, every building. The easy and reasonable way would be to duplicate the castle and modify there.

If you want to mod something usually you modify! existing things. To do that you need to be able to find them and see how you can mod them. Having a nice structure that makes files consistent, easy to find for everyone and without the help of special tools can only help with that. Where to put that having SoD as a "mod" will create a great example base for future modding. Right now all the answers for modding help are on the form: "take a look at mod x,y and see how it is done there"

Codding effort for moving files in the right places is not that great. You "just" look into some json files. I offer to to that. VCMI I understood has already a mechanism to look for files in more than just data folder, but if any adjustments are needed in that part someone else needs to help.
 
     
Ivan 
VCMI programmer

Age: 25
Joined: 08 May 2009
Posts: 1438
Location: Ukraine
Posted: 2013-06-06, 20:27   

Quote:
There are no conversions involved!

OK. Replace "conversions" with "moving".
Quote:
Cpu usage will increase a lot when you will ad zip to mods.

Actually no. Lod archives use exactly same compression algorithm. So speed will remain the same.
Quote:
Right now all the answers for modding help are on the form: "take a look at mod x,y and see how it is done there"

And after moving data into mods it will change into "take a look on mod with Castle/Dungeon/Rampart/etc". Or am I missing something?

Moving towns into separate mods is an interesting feature but I don't think that it can be implemented safely at least right now. (for reasons I mentioned earlier)

What about keeping this file moving optional or (even better) - as a separate utility that modders may use if they want to?
 
     
krs


Joined: 04 Sep 2012
Posts: 374
Posted: 2013-06-06, 20:55   

Quote:
Actually no. Lod archives use exactly same compression algorithm. So speed will remain the same.


I wanted to say that now you do not compress mods and that CPU usage will increase once you do.

Quote:
And after moving data into mods it will change into "take a look on mod with Castle/Dungeon/Rampart/etc". Or am I missing something?


The idea was that you will have a lot more examples to point the people to. All sorts of creatures...

Quote:
What about keeping this file moving optional or (even better) - as a separate utility that modders may use if they want to?


Quote:
Moving towns into separate mods is an interesting feature but I don't think that it can be implemented safely at least right now


This is the answer I was expecting all along. That this makes some sense but there are some hurdles atm.

How about this: I implement the whole moving thing (did some parts already as test) as an VCMI_ckient.exe option but leave the original files in place! and see how people like it. (so when you run client with a specific option SoD mod is created with files in the right places but inactive.) If by any chance VCMI code will never support this, at least modders could have a nice reference.

For this I will need very little help, mainly in the form of: "is this file structure ok?": or "what is the purpose of these files?".
 
     
Ivan 
VCMI programmer

Age: 25
Joined: 08 May 2009
Posts: 1438
Location: Ukraine
Posted: 2013-06-06, 21:58   

Quote:
How about this: I implement the whole moving thing (did some parts already as test) as an VCMI_ckient.exe option but leave the original files in place!

You may want to make it as separate binary (linked to vcmi library to get support for H3 formats).

VCMI should support resulting directory structure - filesystem support is quite flexible here. But I'd like to keep this separate (unless other devs will propose to include these changes in vcmi).
 
     
Macron1 

Joined: 02 Apr 2013
Posts: 576
Posted: 2013-06-06, 22:49   

krs, I think you can try to move at least 1 town to mods and see, if this works (disabled town in .config, move to mod only json configs, but not yet graphics)
If this will work, this can be used for all other standart towns.
_________________
I'm not a member of VCMI developer group and my posts are not official. I'm just a fan.
 
     
krs


Joined: 04 Sep 2012
Posts: 374
Posted: 2013-06-07, 11:44   

First step is to dump all of the original resource files: xxx.snd xxx.vid sprites.lod bitmap.lod in a temporary folder. Armaghedon's Blade files will overwrite where appropriate.

Second Step a function will parse these folders and move file to appropriate locations. Everything that is faction related will go to that folder the remaining files... I do not know yet.

Temp folder structure will be:
Code:
data_temp\
    sprites\
    images\
    sounds\
    videos\
    misc\


I've separated the original H3 files into logical groups. Some more detailed separation can be done in the future if really needed, but for now it looks cohesive enough (EG: group terrains by type, group map editor objects, group interface elements).

For complete separation of towns there are 2-3 def files that need splitting. (EG: Creature portraits are all in one def, town icons are all in one def and I think there was another one).

Code:
//////////////////////////////////////////////////
SPRITES - sources: H2spite.lod

AdventureMapObjects
    Towns
        Castle
        Conflux
        .......
    Creatures
        Castle
        Conflux
        ......
    Dwellings
        Castle
        Conflux
        ......
    CreatureBanks
    Artifacts
CombatHeroes
    Castle
    Conflux
    .....
CombatObstacles
Creatures
    Castle
    Conflux
    .....
Cursors
Heroes
    Castle
    Conflux
    .....
Interfaces
Spells
Terrains


//////////////////////////////////////////////////
IMAGES - sources: H3bitmap.lod, H3AB_bmp.lod

Interface
Campaigns
Town Screens
    Castle
    Conflux
    ......
Town Building Areas
    Castle
    Conflux
    .....
Town Building Borders
    Castle
    Conflux
    .....
Town Puzzle Maps
    Castle
    Conflux
    .......
Creature Backgrounds
    Castle
    Conflux
    .......
Battle Siege Objects
    Castle
    Conflux
    .......
Battle Backgrounds
Battle Obstacles
Hero Prtraits (Big, Small)
    Castle
    Conflux
    .........


////////////////////////////////////////////////////
MISC - sources: H3bitmap.lod, H3AB_bmp.lod

Fonts
Campaign Files (h3c)
Configs (text files)
BoXXX.bmp files (What are these? Some/all go under BITMAPS)


//////////////////////////////////////////////////
SOUNDS - sources: Heroes3_snd, H3ab_ahd_snd

Creature_sounds (attack, defend, move, wince, shoot, kill)

Spell_sounds (including creature abilities)

Interface_Sounds (battle interface too)

AdventureMapObjects_Sounds


//////////////////////////////////////////////////
VIDEOS - sources: H3ab_ahd_vid, VIDEO_VID

Campaign Videos
Misc Videos
 
     
Ivan 
VCMI programmer

Age: 25
Joined: 08 May 2009
Posts: 1438
Location: Ukraine
Posted: 2013-06-07, 12:14   

Quote:
Creature portraits are all in one def, town icons are all in one def and I think there was another one

And a lot of icons: artifacts, buildings, luck/morale, primary/secondary skills, hero specialties, spells, resources, dwellings.

Quote:
BoXXX.bmp files (What are these?

Those are icons for starting bonuses of campaign scenarios. Apart from these bitmaps there are also several .def files for same purpose (can be identified easily by yellow-ish background and "bo(n)" in their names)
 
     
krs


Joined: 04 Sep 2012
Posts: 374
Posted: 2013-06-07, 13:14   

There are too many to be campaign bonuses. I a talking specifically about files like: Bo01SumB.bmp... etc

These I think are the Campaign bonuses: CBONE1A1.bmp... etc
 
     
Ivan 
VCMI programmer

Age: 25
Joined: 08 May 2009
Posts: 1438
Location: Ukraine
Posted: 2013-06-07, 15:22   

Those spell icons seems to be unused and identical to icons in SpellBon.def
 
     
krs


Joined: 04 Sep 2012
Posts: 374
Posted: 2013-06-07, 17:36   

Here the .json for Pikeman. Can someone take a look for something wrong? (Not numbers but overall structure).

I cannot seem to be able to find big icons for creatures. (Only for small "CPRSMALL.def"). Is there any documentation/example on splitting a .def file like this?

Code:
{
    "pikeman":
    {
        "faction" : "castle",
        "level" : 1,
        "cost" :
        {
            "gold" : 60
        },
        "speed" : 4,
        "hitPoints" : 10,
        "attack" : 4,
        "defense" : 5,
        "damage" :
        {
            "min" : 1,
            "max" : 3
        },
        "doubleWide" : false,
        "growth" : 14,
        "horde" : 0,
        "aiValue" : 80,
        "fightValue" : 100,
        "advMapAmount" :
        {
            "min" : 20,
            "max" : 50
        },
        "graphics" :
        {
            "animation" : "castle/battle/CPKMAN.def",
            "map" : "castle/map/AvWPike.def",
            "iconSmall" : "castle/iconsSmall/xxxxx.bmp",
            "iconLarge" : "castle/iconsBig/xxxxx.bmp",
            "timeBetweenFidgets" : 1,
            "animationTime" :
            {
                "walk" : 1.15,
                "attack" : 1,
                "flight" : 1
            }
        },
        "sound" :
        {
            "attack" : "castle/PIKEATTK.wav",
            "defend" : "castle/PIKEDFND.wav",
            "killed" : "castle/PIKEKILL.wav",
            "move" : "castle/PIKEMOVE.wav",
            "wince" : "castle/PIKEWNCE.wav"
        }
    }
}
 
     
Display posts from previous:   
Reply to topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum
Add this topic to your bookmarks
Printable version

Jump to:  

Powered by phpBB modified by Przemo © 2003 phpBB Group

Hosting provided by DigitalOcean
Page generated in 0.05 second. SQL queries: 13