Forum index VCMI Project - Heroes 3: WoG recreated
Forum of the project aiming to recreate best turn-based strategy ever!

FAQFAQ  SearchSearch  MemberlistMemberlist  UsergroupsUsergroups  StatisticsStatistics
RegisterRegister  Log inLog in  AlbumAlbum  DownloadDownload

Previous topic :: Next topic
New mods repository management / downloads concept
Author Message
SXX 
VCMI programmer

Age: 25
Joined: 04 Jul 2014
Posts: 358
Posted: 2017-05-23, 06:35   New mods repository management / downloads concept

Please check my concept of new mods repository:
http://wiki.vcmi.eu/index...Mods_Repository

Keep in mind I just registered organization for to make more clear how it's should work, but nothing else is done. Feel free to leave any ideas here.
 
     
SXX 
VCMI programmer

Age: 25
Joined: 04 Jul 2014
Posts: 358
Posted: 2017-05-27, 00:19   

Some implementation idea. Advantage of having repositories on GitHub would be ability to verify mods integrity with Travis, e.g:
  • Verify repository JSON integrity.
  • Verify JSON integrity of mod itself since Travis can clone mod from git.
  • Depend on what mod is changing we can run several vcmi tests too. At least make sure it's won't crash on loading.
 
     
misiokles 

Joined: 12 Nov 2015
Posts: 168
Posted: 2017-05-27, 10:04   

Your proposals sound good enough. And yeah, few persons should be choosen to ensure intergity of all vcmi mods (for cross testing to find all possible conflicts for example and alterative mods of given mod).

Also content folder should be in .zip form - it's easier to update mods in this form when - for example - new mod version will be refractored.

And launcher should have few additional features. For me, better funcionality to "conflict" command should be implemented.
 
     
SXX 
VCMI programmer

Age: 25
Joined: 04 Jul 2014
Posts: 358
Posted: 2017-05-27, 16:28   

misiokles wrote:
Also content folder should be in .zip form - it's easier to update mods in this form when - for example - new mod version will be refractored.
I not quite get what you mean, but it's not good idea to store archives in Git and let's say even WoG would need to be unpacked properly even if this would require additional overrides support in engine.

GitHub have 1GB size limit (include history) per repository which is sufficient for almost every H3 mod, but we're better not to abuse it without a reason. Then launcher will always download mods as ZIPs so for end-user it's doesn't matter how exactly assets stored in Git.

misiokles wrote:
And launcher should have few additional features. For me, better funcionality to "conflict" command should be implemented.
Launcher really need a lot of work, but so far I not sure there anyone want to work on it.
 
     
misiokles 

Joined: 12 Nov 2015
Posts: 168
Posted: 2017-05-28, 08:56   

SXX wrote:
misiokles wrote:
Also content folder should be in .zip form - it's easier to update mods in this form when - for example - new mod version will be refractored.
I not quite get what you mean, but it's not good idea to store archives in Git and let's say even WoG would need to be unpacked properly even if this would require additional overrides support in engine.

From http://wiki.vcmi.eu/index...ding_guidelines
Code:
All content of your mod should go into Content directory, e.g. Mods/myMod/Content/. In future it will be possible to replace this directory with single .zip archive.

HotA mod is build like this.
 
     
SXX 
VCMI programmer

Age: 25
Joined: 04 Jul 2014
Posts: 358
Posted: 2017-05-28, 13:10   

If we going to store mods in Git then we would not compress anything in it and will need to actually unpack everything from mods like WoG. Compression would be handled by github when mod is downloaded.
 
     
Brutus5000

Joined: 30 May 2017
Posts: 1
Posted: 2017-05-30, 00:21   

Hi guys,

I'm following your projects for a long time and I just stumbled upon this thread. I am a developer at faforever.com (Supreme Commander Forged Alliance community).

We have a comparable issue with mod repository versioning. As a solution I developed a software called BiReUS. It creates binary patches using the bsdiff4 algorithm and is also capable of handling zip-files (extract, compare, generate patch --> extract, apply and zip again). It is developed in Python 3.x. The server part works like any unix cli tool generating the patches after you copied a new version into the repository folder. The files need to be serverd by an http server.

The client has a git-like interface but can also be embedded as a library. The client compares checksums before and after patching to ensure nothing went wrong.

We are not using it in our main client yet, because it did not made the transition to Python 3, so there is no productive experience yet, but I tested it succesfully on our cores repository (around 50 versions, patches between 200kb to 40mb depending on the changes between).

You can find it in my git repository under http://github.com/Brutus5000/BiReUS , there is also a basic Qt-based launcher application and a Java client library

If you are interested, but have additional requirements, I am sure we can find a solution for that as well.

Cheers
Brutus5000
 
     
SXX 
VCMI programmer

Age: 25
Joined: 04 Jul 2014
Posts: 358
Posted: 2017-05-30, 04:04   

Welcome. Thanks for the idea, but I don't think we trying to solve similar problems since mods for FA can be large since they include heavy models and a lot of textures. So there can be actual reason to use binary diff.

In case of H3 largest mods are towns like HotA that take up to 50MB and they rarely change binary assets so we really can store them in git uncompressed. If GitHub limit wouldn't be enough we can simply move some of them to GitLab since they allow 10GB repositories. And for the same reason download size isn't issue for us too: mods just too small.
 
     
Asgraf


Joined: 05 Nov 2012
Posts: 37
Location: Poland
Posted: 2017-06-10, 23:35   

I think you should take a look at IPFS protocol:
https://ipfs.io/
https://www.youtube.com/watch?v=HUVmypx9HGI
It provides all the tools you will ever need for a good mod repository management

tl;dr:
IPFS = HTTPS + SFS + GIT + BITTORRENT + DHT
  
 
     
SXX 
VCMI programmer

Age: 25
Joined: 04 Jul 2014
Posts: 358
Posted: 2017-06-11, 16:58   

Asgraf wrote:
I think you should take a look at IPFS protocol
I like IPFS as concept, but that would mean anyone who want to install mods would need IPFS installed and this is no go. :-(
 
     
Asgraf


Joined: 05 Nov 2012
Posts: 37
Location: Poland
Posted: 2017-06-18, 20:23   

SXX wrote:
I like IPFS as concept, but that would mean anyone who want to install mods would need IPFS installed and this is no go. :-(

IPFS is not a software. IPFS is a protocol (same way as HTTP is protocol).
There are multilple implementations of this protocol in different languages.
Implementations are modular so they can be used standalone or as a part of any software.
So i see no reason why not use it in VCMI.

List of existing VCMI implemetations
Please note that currently only Javascript and Go implementations are mark as stable.
 
     
SXX 
VCMI programmer

Age: 25
Joined: 04 Jul 2014
Posts: 358
Posted: 2017-06-18, 21:15   

Asgraf wrote:
IPFS is not a software. IPFS is a protocol (same way as HTTP is protocol).
Terminology makes little difference because we still have to ship implementation that have to work on Windows, Linux, MacOS and Android.

Asgraf wrote:
Implementations are modular so they can be used standalone or as a part of any software.
So i see no reason why not use it in VCMI.
Integrating it would require someone who actually know what he's doing and my own experience in Go limited to writing easy API web server and some file management tools. So reason is super simple: we have no developer who have time to handle task like that. :-(

And even if we would use something like IPFS for content delivery we still need to use git for contribution management and tracking of changes.
 
     
Asgraf


Joined: 05 Nov 2012
Posts: 37
Location: Poland
Posted: 2017-06-18, 23:13   

SXX wrote:
we still need to use git for contribution management and tracking of changes.

Here is git extension that adds full IPFS support for your command line git
https://github.com/larsks/git-remote-ipfs

SXX wrote:
we still have to ship implementation that have to work on Windows, Linux, MacOS and Android

I think the simplest solution is to recreate new Mod manager as a standalone web app written in entirely in JavaScript (HTML5 + CSS for GUI).
JavaScript should parse VCMI json files natively (as long as they are truly valid JSON) and IPFS has stable JavaScript implementation.
Tell me if such a idea appeals to you.
If yes, i can squeeze some of my free time to create some proof of concept app.
 
     
SXX 
VCMI programmer

Age: 25
Joined: 04 Jul 2014
Posts: 358
Posted: 2017-06-18, 23:50   

Asgraf wrote:
Here is git extension that adds full IPFS support for your command line git
https://github.com/larsks/git-remote-ipfs
Problem is that people who going to develop mods or manage repository shouldn't be required to use it. I see teaching them how to use Git and doable since there GUI tools and most of mod changes can be even made using GitHub website, but anything else is just way too hard.

Also plan is to not provide anyone access to our servers / keys to anyone at all so it's should be possible to automate all interactions will IPFS.

Asgraf wrote:
Tell me if such a idea appeals to you.
If yes, i can squeeze some of my free time to create some proof of concept app.
I like IPFS as idea. Problem is that I want implementation where mod creators (and maintainers) as well as players would never have to even think how mods are distributed and should never ever have issues with it's not working.

If you have time to explain how do you see it's going to work I'll appreciate if you could jump on our Slack:
https://slack.vcmi.eu/
  
 
     
Display posts from previous:   
Reply to topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum
Add this topic to your bookmarks
Printable version

Jump to:  

Powered by phpBB modified by Przemo © 2003 phpBB Group

Hosting provided by DigitalOcean
Page generated in 0.02 second. SQL queries: 13