Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 Encoder GUIs

Reply
 
Thread Tools Search this Thread Display Modes
Old 4th August 2017, 14:46   #1  |  Link
stax76
staxrip & mpv.net
 
Join Date: Jun 2002
Posts: 5,247
How to make StaxRip download easier for people with low bandwidth?

Is there a simple way I can use to make it easier for people with low bandwidth to update to the latest test build?

Right now StaxRip includes 82 tools and is 97 MB packed, about 400 MB unpacked.
__________________
staxrip is in need of a new maintainer since I would like to retire.

Last edited by stax76; 4th August 2017 at 14:49.
stax76 is offline   Reply With Quote
Old 4th August 2017, 15:07   #2  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 4,983
- slim packages down (e.g. only compile ffmpeg with the libraries used by StaxRip)
- divide package into small ones for every tool and only update those that have changed and only those the user actually uses
- implement some diff update (e.g. based on xdelta3 - probably some tools out there)

Xdelta3 test:
Patch from StaxRip-x64-1.6.0.1-test.7z to StaxRip-x64-1.6.0.2-test.7z is only 827 kB.
sneaker_ger is offline   Reply With Quote
Old 4th August 2017, 17:24   #3  |  Link
mariush
Registered User
 
Join Date: Dec 2008
Posts: 563
Put the files in an archive/package file you can easily control/manipulate .. for example TAR files (512 byte header for each file/folder, then file content with padding to 512 bytes chunk) ... when you release an update you can make new tar while keeping the file names and order (in tar file) the same to make the diff (with tool like xdelta3) easier...

Download diff, apply diff to old tar file, delete previous version of the tar file. Optionally, compress the diff with 7zip or some other open source compression scheme.

+ agree with slim version of ffmpeg

stay away from compressing binaries with tools like upx , too much hassle with antivirus programs to be worth it and would just lower your diff efficiency
mariush is offline   Reply With Quote
Old 5th August 2017, 10:58   #4  |  Link
stax76
staxrip & mpv.net
 
Join Date: Jun 2002
Posts: 5,247
Xdelta3 would be a option, the initial tar would be about 400 MB though. Custom builds would be too much work, so far I only make a custom installer for avisynth and sometimes build mp4box (there is Visual Studio project in the repo which builds out of the box). At the moment it's already difficult constantly updating everything and answer the requests which lately are many. Another option would be uploading a light version excluding VapourSynth plugins, mpv and the hardware encoders and other things to cut the size minimum in half, this would only be available for test builds and people could overwrite the full version letting windows file explorer integrate it.
__________________
staxrip is in need of a new maintainer since I would like to retire.
stax76 is offline   Reply With Quote
Old 5th August 2017, 13:41   #5  |  Link
mariush
Registered User
 
Join Date: Dec 2008
Posts: 563
You shouldn't really care so much about how big files are, in the age of TB sized hard drives and games that use 60-80 GB.

People only care about how big an application is, if they have small capacity SSD drives as their boot drives.

You have to AUTOMATE things.

For example, every time your application changes either due to new versions of your tools or the main application itself, it gets a new release version, a unique ID. Let's say 2017080501 (for 2017 08 05 1st release this day). If tomorrow you update x265 in the tools folder, you'll have version 2017080601.
Now you can write something that will automatically create those "update packages" ... the tool can build the 7z file like you have it on your website, but can also go back and create DIFFs for let's say the last 10 versions of updates... diff from 2017080501 to 2017080601 and so on...
You can then write a small update tool and bundle it in the package. The tool can query a website and download the latest update information, which could be something like (of course encoded in a better way like using json or xml) :

update version=2017080501 type=full size=400MB url=xyz sha256=
update version=2017080501 type=partial version_previous=2017080401 size=10MB url=xyz
update version=2017080501 type=partial version_previous=2017080301 size=15MB url=xyz

So your update tool can then either download the diff from a version to the next one, or choose to download the full package.
If applying diffs, your tool most likely will need to hash check each file that would be changed by the diff to make sure it's the proper version. If any of the files is modified the diff can't be applied so the update tool will have to give up and download the full package.

You could look at torrent files to inspire from ... they're basically a collection of file paths along with several sha256 hashes for each file (one sha256 hash for each chunk of data in the file, where the chunk could be from 4K to 16 MB in size)
mariush is offline   Reply With Quote
Old 12th August 2017, 13:12   #6  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 5,561
Assuming you don't care about a few false-positive results from virus scanners use upx or mpress to pack some of of the binaries.

Cu Selur
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 12th August 2017, 13:57   #7  |  Link
mariush
Registered User
 
Join Date: Dec 2008
Posts: 563
could just as easily make the installer enable ntfs compression on the folder for transparent real time compression (to 50-70% of original size)
mariush is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 04:35.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.