I was unable to find an open source tool able to create DEF files. All I could find were open source DEF readers. But until the animation format can be used everywhere, there must be a way for artists on all platforms to convert their content into the DEF format.
I’m probably missing something obvious, so could you show me a DEF creating tool which is either already cross platform or at least open source so that it can be ported?
As I explained in the other thread I already tried myself at creating a DEF writer based on the vcmi DEF reading code. I tested my DEF writer by doing the following:
extract frames from all DEF files
replace all colors with values greater than 7 (values of 7 and below denote transparency, right?) with a solid color
write the modified images back to their original DEF files
The resulting DEF files can be read by my DEF file reader but with vcmi they create mixed results. The DEF files I created work well on the adventure map but they do not for buttons, town buildings and battle creature animations. Here is how it looks like:
When repacking the DEF I save all frames uncompressed (format zero) for simplicity.
It would be best if an open source tool for creating DEF files from individual images already existed. I dont want to spend my time on doing something that’s already done.
As far as I know, DEF is proprietary 3DO format, so you won’t find open-source general-purpose tools for it. However, they are available for everyone: download.vcmi.eu/tools/
I know but why does that mean I wont find open-source general-purpose tools for it? The microsoft doc format is proprietary as well and still I can load and save it in libre office. In case of vcmi, DEF is proprietary but still vcmi can load it through an open source implementation.
Those are only binaries. They dont tell me anything about the algorithms behind. Not even the functionality is available to everyone. It is only available to those running the right operating system.
The fact that these tools exist tell me that enough has been reverse engineered of the format to create a DEF writer/packer.
Pretty much all pre-VCMI content for H3 was created by Russians who have mastered reverse-engineering, hex-editing (sic) and tend to keep their secrets rather than share as open-source. You probably would have to ask one of tool’s authors.
it’s also obvious to me but if I do not ask then maybe I might be missing efforts somebody else already made and would thus waste my time. Hence I wanted to ask first.
Thanks for pointing me to the wiki article though obviously it doesnt give me more information than I already had by reading the vcmi sources.
You may try contacting author of deftool for source code. This seems to be his homepage: sites.google.com/site/sergroj/wog
(most likely written in Pascal/Delphi)
Not exactly. Values below 7 are special colors - they are used for shadows and for “glue” selection around units in battle.
This may be a problem - at least for battles vcmi needs either format 1 or format 3 (can’t remember which one - check client/battle/creatureAnimation.cpp). Yet another issue I still haven’t fixed.
Ah I see - what makes the transparency then? I must’ve misunderstood what I read in the vcmi sources.
Ah, then what I observe makes more sense.
Then I have to extend my python scripts so that they can also write format 1 and format 3 files. How do I determine in which format to safe? By the object id in the very first header? Or by which format would result in the smallest filesize?
I also have thee questions about the json animation format.
Firstly, is it a hard task to make the json format be supported by everything instead of the def format? If it’s an easy task which can be accomplished soon-ish then I will not bother with spending time on a DEF file writer.
Secondly, does the json format work as a drop-in replacement just as the pcx files can actually be png files? Essentially, what do I have to do to replace a def file in the Data/ directory by a json animation? Is it more than just putting a json file with the same basename as the def file into Data/?
Thirdly I have a question about the json format. It uses relative paths. To what are the paths relative? To the current directory? To the directory the json file lives in? To the Data/ directory?
Take a look here: github.com/vcmi/vcmi/blob/maste … n.cpp#L288
Color #0 - full transparency
Colors #1-#4 - semi-transparent shadow
Color #5 - glue selection in battle
Colors #6-#7 - glue selection on top of shadow
Colors #8-#9 are also used in H3 in terrain def’s to determine color of tile on minimap. VCMI ignores them.
Rechecked code - for creatures in battles def must have format “1”. VCMI checks specifically for format type and ignores “object ID” field but it should be same for all H3 creatures - check any of C*.def files
As for Json animation:
Far from easy. Current code relies too much on direct access to underlying SDL_Surface and assumes that all image have palette. In some parts of our code this has been fixed, introducing Json format alongside. Long-term goal is to get rid of .def requirements but I have no idea when this will happen.
Yes. For images VCMI always loads latest file. For def/json this is a bit complex but idea is same:
vcmi parses def file (if present) and keeps list of available frames
vcmi loads all json files with such name (multiple overrides are possible via mods) and updates animation index replacing any frames from def*
Relative to Data/ or Sprites/ directory - either one will work. If you wish to make your own directory structure you can specify base path in json.
BTW - if you wish you can edit wiki (login via forum account) to make documentation clearer where necessary.
I see. Yes, that’s why for my tests I was replacing all colors except 0-7 with a solid color. It would leave the stencil of the object.
Okay. Then when unpacking the original def I will also save the format so that when repacking the def I can use the exact same as the original def had.
Okay, then it will make sense for me to improve the def writer.
Sounds nice! I’ll try that out for some town buildings and GUI elements. I can then keep a list of defs which do not need to be repacked as defs but which just need a small json file to be created alongside them.
I tried to login with my forum account credentials but got rejected. Maybe it doesnt work because I recently did a password reset and created a new password?
As you can see, now everything ingame seems to work just as expected. There are only some tiny things left:
I dont know what to do about format 2. As far as I understood it’s a format which does not animate by cycling through multiple frames but by modifying the palette assignments. It also seems that the affected graphics (adag.def, clrrvr.def, cobbrd.def, dirtrd.def, gravrd.def, icyrvr.def, lavrvr.def, mudrvr.def, tshre.def) cannot just be replaced by ones that are compressed in formats 0, 1 or 3. Is that observation correct? At least only these 9 graphics use format 2 so the problem is very limited. For now I’m using the original def files for all of them.
Currently I unpack the def files into a directory with one png file per frame inside. Metadata is encoded in the filenames such that a working def can be created from such a directory. It would be better to use the JSON animation format instead but it does not allow to encode some information that is necessary to recreate a working def from it. For example as we discovered it is important that some graphics are saved in the same format. Though I suppose that the vcmi json parser will not complain if it finds some unknown dictionary keys? I would then just overload the animation format with all additional information that is needed to be encoded.
Other files that give me a headache are sgtwmta.def and sgtwmtb.def as they specify an offset which is greater than the image dimensions. And also ovslot.def is problematic because it contains frames with different total size. How does vcmi deal with these anomalities?
Another thing I noticed is that one actually has to make an effort to keep the filesize small using the RLE formats 1 and 3 because if one fails then it’s easy for length and offset values to grow larger than they are allowed to. Unfortunately lots of values are saved in only 8 or 16 bits. The “compression” algorithm I came up with seems to be enough for the existing assets.
on format 2 - apart from creatures in battle any type should work anywhere. So you don’t need to implement all of them. VCMI decodes def file and loads everything into SDL_Surface’s, any palette swaps necessary for animation will be performed on SDL_Surface during rendering. So def itself may contain any format, just make sure that palette is not lost (and not reordered) during conversions.
We do have schema validation that can detect things like this for most of formats exposed to mods. Except for animation format.
As for files:
sgtwmt* - those contain gfx for mine field (probably for Tower sieges). Unused since there is separate set of graphics for “place mines” (spell).
ovslot - by total size you mean x/y dimensions? In this case - VCMI uses size of individual frames and ignores size from def header.
“creatures in battle” is a strange exception. Okay, I will try again and just convert the type 2 format to type 0.
If it does schema validation for other things then surely that will also be introduced for the animation format sooner or later.
Are there other obvious ways on how I could store the extracted frames other than the animation format, given that it might not allow to store all necessary information to successfully repack the def in the future?
Okay. Then I delete them.
Okay. Some of the windows based def readers do not ignore the header information though. Initially I was setting the header information to zero because ovslot.def showed that the information might not match. But when I ran into troubles I just picked one of the frames and used that one as source for the info. I guess I can find a way to fix the problem because ovslot.def is the only case where this information does not match.
This is what I mentioned before:
Rechecked code - for creatures in battles def must have format “1”. VCMI checks specifically for format type and ignores “object ID” field but it should be same for all H3 creatures - check any of C*.def files
Even if/when it will be introduced it won’t be enforced - right now its main purpose is to notify modders for any possible issues in mod.
If you wish - you may keep all your data in comments - our json parser accept C+±style comments “//”
Yes. It turns out that all def files starting with c except for clrdelt*.def have format 1. clrdelt*.def are river deltas in format 3. Funnily in contrast to the other rivers (which exist in format 2) they do not use palette swapping to create the animation.
Okay. With that in mind I did some more updates to my code. Now it is possible to extract all files from lod and pac archives (pcx images are saved as png automatically), convert def animations to the json format and convert from the json format back to the def. I need to store two additional bits in the json data to do the conversion to the def which is the object id and the format type. Since buildings understand the json animations, I tried to only leave the tb*.json files in my Data/ directory and remove all the tb*.def files. It worked perfectly! Until other parts of vcmi support the json animation format, they can be converted back to the def format. A json file as created by my scripts look like this:
You see that I decided to put all frames for each def animation into their own subdirectory. I did this to avoid name clashes and to avoid the Data/ directory growing too big. You can also see that I prefixed these directories with “.dir” to make it easy to remove them all at once if desired. The def format saves filenames for the individual frames. I drop them as they do not seem to be used by vcmi.
Other improvements include that I can now fully support extracting and saving format 2 def animations and that I fixed my implementation of the writer for format 3. My initial implementation was based on the vcmi reader but it seems that in the original files, format 3 encodes 32 pixel long blocks seperately. Vcmi does not care about this restriction but other editors might enforce this.
I now deem this project quite complete. What would be needed to make it actually useful for people besides me? Would it make sense to integrate these packing/unpacking scripts into vcmi? If yes, should I rewrite them in C?
I have also no clue how all this works on platforms that are not Linux/BSD/Hurd which are the only platforms on which I tested the code. Sadly I do not own any license for either Windows or MacOS.