The best compression lossless codec ;-)

[DmitriyV2]

MSU Lossless Video Codec 0.2.4:

Giving the best compression ratio in the lossless video compression
field! (If any codec better compress - please send it to us! :))
This codec is intended for lossless video compression (used for saving
master copies or intermediate rendering results).

The project is in beta-version stage (0.2.4 version). Good results in
compression are already achieved, now working on codec's speed.
Next major version will have about 10 times faster compression.

Installer avalable from:
http://www.compression.ru/video/ls-codec/index_en.html

Enjoy! ;)

[Tommy Carrot]

Seems promising, in my limited test it seems on par with FFV1, sometimes a little better, sometimes a little worse, which is actually quite a big accomplishment, as FFV1 is far the most efficient lossless codec so far. If the speed issues will be addressed, it'll be very useful for many of us.

Keep up the good work.

[Wilbert]

Giving the best compression ratio in the lossless video compression field! (If any codec better compress - please send it to us!
If you make claims like this you should at least:

1) provide the evidence (yes, I have seen your test somewhere ...). Just give a link here.

2) tell which lossless codecs you used for the comparison.

3) mention for what type of sources it gives the best compression ratio. Clean, noisy, etc ...

edit: the test is in the link. Sorry :)

[Kurtnoise]

Please, could you add VBLE (http://divxevolution.online.fr/modules.php?name=Downloads&d_op=getit&lid=49) in your tests. Thanks.:cool:

[Arachnotron]

What is it about the divx of the matrix that makes it so difficult to compress?

[LordRPI]

DivX is a lossy codec. While it strives to retain as much detail as possible, it does not preserve every aspect of the source film.

[DmitriyV2]

Originally posted by Tommy Carrot
Seems promising, in my limited test it seems on par with FFV1, sometimes a little better, sometimes a little worse... Keep up the good work.
Tommy send me detailed info how it's better to test FFV1 (version, mode) and we will add it to comparison.

Big thanks again! ;)

[DmitriyV2]

Originally posted by Kurtnoise13
Please, could you add VBLE (http://divxevolution.online.fr/modules.php?name=Downloads&d_op=getit&lid=49) in your tests. Thanks.:cool:
Please, could you explain how to use it in lossless mode? ;)
(on our sequences it change frame, but we test only RGB now)

[DmitriyV2]

Originally posted by Wilbert
If you make claims like this you should at least:
1) provide the evidence (yes, I have seen your test somewhere ...). Just give a link here.
2) tell which lossless codecs you used for the comparison.

Please visit http://www.compression.ru/video/ls-codec/index_en.html ! ;)
All tested codecs (of course!) pointed there.

We are going to expand this list with new codecs. Please, send us links!

3) mention for what type of sources it gives the best compression ratio. Clean, noisy, etc ...

For clean source our advantage is bigger (up to 3.5 time better compression than HuffYUV), for noisy - lower, but better.

Especially for noisy source (and some users :)) we add "visually lossless" compression, but for comparison only mathimatically lossless mode was used.

edit: the test is in the link. Sorry :)
Ok! :)

[DmitriyV2]

Originally posted by LordRPI
DivX is a lossy codec. While it strives to retain as much detail as possible, it does not preserve every aspect of the source film.
Sure. For us in this situation DivX works like big denoiser. ;)

Video after MPEG-2 (DVD) or from DV also will compressed with better advantage, than captured from analog input.

We use this analog captured noisy sequences (foreman, flower garden, bus, ...) becouse they are standard, available and widly used in scientific articles for codecs comparison.

[unmei]

afaik and for the version i have ("pretty old"), VBLE is for YV12 color space only. So, for now you cannot make comparisions to it if you don't have a YV12 mode in yours.

But, as it seems you want the codec to be attractive to DVD backup people, a YV12 mode in your codec is very welcome. People who use DVD source and AviSynth for later compression in mpeg-4 codecs may not want to store a intermediate result in RGB color space as the entire other processing chain is pure YV12..

[Doobie]

I was reading a bit about the compression levels. This is part of what is says for Medium compression. "Information losses that are visually unnoticeable... In many cases image quality increases." I guess the imagine quality doesn't improve enough to notice. :D

[Arachnotron]

@LordRPI
DivX is a lossy codec. While it strives to retain as much detail as possible, it does not preserve every aspect of the source film. Sorry, should have formulated it differently. I can understand why a source that has already been passed through a lossy codec contains less detail and therefore can be compressed further.

What I did not understand is why this codec profits so much from denoising. The compression ratio for the other codecs goes up by a factor of about two, while for the MSU codec the increase is by more than 3.

By now, Dimitry already explained this codec gives much better compression on clean sources relative to the other codecs
I suppose asking why is the same as asking how the codec works. And I probably would not understand that even if Dimitry explained it :)

[DmitriyV2]

Originally posted by unmei
afaik and for the version i have ("pretty old"), VBLE is for YV12 color space only. So, for now you cannot make comparisions to it if you don't have a YV12 mode in yours.

But, as it seems you want the codec to be attractive to DVD backup people, a YV12 mode in your codec is very welcome. People who use DVD source and AviSynth for later compression in mpeg-4 codecs may not want to store a intermediate result in RGB color space as the entire other processing chain is pure YV12..

Our main priorities now:
* Codec speed (serious improvement is possible)
* Bigger compression (it's also possible)
* YV12 support ;)

YV12 - simplest task (we already have common YUV support).

Thanks for info about VBLE!

[DmitriyV2]

Originally posted by Doobie
I was reading a bit about the compression levels. This is part of what is says for Medium compression. "Information losses that are visually unnoticeable... In many cases image quality increases." I guess the imagine quality doesn't improve enough to notice. :D

We have algorithms that improve enough:
http://www.compression.ru/video/old_film_recover/index_en.html ;;)

But if we implement them everyone will say - this is not visually the same video! :)

Main goal of this lossless codec - maxixum compression with zero (fully lossless) or minimun frame changing.

[DmitriyV2]

Originally posted by Arachnotron
What I did not understand is why this codec profits so much from denoising. The compression ratio for the other codecs goes up by a factor of about two, while for the MSU codec the increase is by more than 3. [...] I suppose asking why is the same as asking how the codec works. And I probably would not understand that even if Dimitry explained it :)
I try to explained as simple as possible. :)

Global compression principle: it's difficult (impossible) to compress random data (noise).

You can generate file with rand() function and compress it with any archiver (low compression), or get *.txt file (bigger compression), or generate file with zeros (only zeros) with the same length (maximum compression).

So for all codecs (especially for lossless) denoising seriously increase compression.

[Doobie]

The irony here is that information and garbage are the same thing. They cannot be compressed. Only redundancy can be compressed. A good filter tries to remove garbage without removing information so that the garbage doesn't take up space. Only redundancy can be compressed.

[RadicalEd]

Test results for 86 seconds of smooth animated content:

RGB YUV 4:2:2 YUV 4:2:0
KB % KB % KB %
raw: 1866657 1244457 932907
MSU: 308109 16.50 198717 15.97
FFV1: 254370 13.62 (82.56) 170451 13.70 (85.78) 142071 15.23
( ) is percent of MSU
FFV1 settings: coder: AC, context: 1, keydist: 300, FFDShow 10/12/04

edit: settings
edit 2: FFV1 yv12 for great justice

[DmitriyV2]

Originally posted by RadicalEd
Test results for 86 seconds of smooth animated content:
And what was smoothing level? It's looks like big (7 times lossless compression). Can you put this video on some FTP (or HTTP)?
Thank you in advance for the help in new version creation! :)

Edit: Thanks!

[DmitriyV2]

Originally posted by Doobie
The irony here is that information and garbage are the same thing. They cannot be compressed. Only redundancy can be compressed.
Sure.
But for such type of explanation its reasonable to define wnat is information (at least). And how calculate amount of information in video? ;)
And why new fully lossless video codecs again and again increase lossless compression ratio.

[RadicalEd]

The avisynth script was a mess, but a masterpiece. I couldn't compile it into any one form anymore. I just know it was done somehow, it was beautiful, and I've used the product for testing all kinds of things ever since. Filtering was mostly a combo of deen and mftoon, IIRC.
I don't have anywhere I could host 150 mb at, but if you want to give me an ftp address, I can upload it. It'd take about 2 and a half hours though :/

[DmitriyV2]

Originally posted by RadicalEd
The avisynth script was a mess, but a masterpiece. I couldn't compile it into any one form anymore. I just know it was done somehow, it was beautiful, and I've used the product for testing all kinds of things ever since. Filtering was mostly a combo of deen and mftoon, IIRC.
I don't have anywhere I could host 150 mb at, but if you want to give me an ftp address, I can upload it. It'd take about 2 and a half hours though :/
Thank you for greate job with filtering, to filter video to state where MSU is bigger than FFV1! :) Resulting video with only 2-3 second, with such effect also is very useful! Also it's 7 times compressed (with MSU it's enough ;) ), so resulting size will be small (1Mb) and acceptable via mail. Is it possible? Thank you in advance again! :)

[Atamido]

Don't forget about CorePNG (http://forum.doom9.org/showthread.php?s=&threadid=64139) . While it is extremely slow, it will typically get the "Best Lossless Compression" ratio. On the fourth page of that thread is a comparison of it and a few other codecs. When you test the codec, please include samples and the length of time it took to compress for each codec.

[RadicalEd]

Incidentally, there is a 3.42 second clip from it that I traditionally use for short testing. Here (http://misatokatsuragi.home.comcast.net/sub_msu.avi)'s the YUY2 MSU clip, weighing in at 10,997 KB, compared to the FFV1 clip at 9,799 KB.

[Wilbert]

Incidentally, there is a 3.42 second clip from it that I traditionally use for short testing. Here's the YUY2 MSU clip, weighing in at 10,997 KB, compared to the FFV1 clip at 9,799 KB.
Nice! Please refresh my memory. What codec was 'Giving the best compression ratio in the lossless video compression field!'? :)

[DmitriyV2]

Originally posted by RadicalEd
Incidentally, there is a 3.42 second clip from it that I traditionally use for short testing. Here (http://misatokatsuragi.home.comcast.net/sub_msu.avi)'s the YUY2 MSU clip, weighing in at 10,997 KB, compared to the FFV1 clip at 9,799 KB.
Thanks a lot for your help! Wating for new version of MSU! ;)
(In our plans were speed, bigger compression, YV12,
so currently it's bigger compression, speed, YV12 :)).

unmei! Sorry for delay. RadicalEd found an interesting work for us! ;)

[RadicalEd]

Please keep speed top priority anyway, it took over an hour and a half to encode the entire 86 second clip each time on my poor 1ghz athlon :/

Anyway, just to show the reason it's so clean and heavily filtered, here (http://misatokatsuragi.home.comcast.net/sauce.m2v)'s the sample from the original (R1 DVD). Have fun with that one.

[cypher_soundz]

link no workie :(
Regards
cyph

[Tommy Carrot]

I don't want to ruin the mood here (:D), but i found a lossless codec called 'snow' (well, not really a lossless codec, but it has a lossless mode) which outperforms FFV1 and MSU by quite a large margin (if it's truely lossless, i cannot check it, but it supposed to be). It uses yv12 colorspace, so i cannot directly compare it with MSU, but anyway, here are the results (from yv12 source, a chapter from the fellowship of the ring):

ffv1: 688,066k
msu : 669,351k
snow: 583,940k
It's currently only usable with the mencoder and mplayer(see this thread (http://forum.doom9.org/showthread.php?s=&threadid=83039&perpage=20&pagenumber=2)), and this limits the usability, but still, very impressive.

[RadicalEd]

Originally posted by cypher_soundz
link no workie :(
Regards
cyph

Thanks, set the permissions after my connection had already expired ;)

I kinda suspected Snow would be the most advanced lossless codec to date from what I've heard of it before. I plan on doing some pretty extensive testing later today, so expect those results sometime soon (hopefully... it's the weekend, never know when I'll have time :D )

[DmitriyV2]

Originally posted by Tommy Carrot
I don't want to ruin the mood here (:D), but i found a lossless codec called 'snow' (well, not really a lossless codec, but it has a lossless mode) which outperforms FFV1 and MSU by quite a large margin (if it's truely lossless, i cannot check it, but it supposed to be).
It's very common way currently - to create "visually lossless" codecs.
(Such mode quite enough and usefull for common users). Be vigilant. ;)

[DmitriyV2]

Originally posted by RadicalEd
Please keep speed top priority anyway, it took over an hour and a half to encode the entire 86 second clip each time on my poor 1ghz athlon :/
Ok! Only for you! ;)
Anyway, just to show the reason it's so clean and heavily filtered, here (http://misatokatsuragi.home.comcast.net/sauce.m2v)'s the sample from the original (R1 DVD). Have fun with that one.
Certainly! :) My deep appreciations!

[RadicalEd]

I'm afraid that while snow's wavelet based coding may be superior for natural images, it bombshells on the sharp-lined smooth-surface content of animated material.

YUV 4:2:0
KB %
raw : 932907
snow: 192125 20.59
MSU*: 154059 16.51
FFV1: 142071 15.23

settings yielding lowest lossless size: qpel, 5/3 wavelets, mb compare (qpel:pred=1:mbcmp=1)

*Of course, there is no YV12 mode in MSU currently. The above is an estimation based on the RGB result with the assumption that the YV12 result would be about RGB/2. This is likely because MSU scales from RGB to YUY2 almost exactly proportional to bitdepth.

I have some LA material I'll test my hypothesis on next...

edit:
Oddly enough, LA material (at least my DV source) is also much smaller with ffv1. 124 vs 138 mb. Settings should be the best... encoding is YV12... so I'm not sure what's going on really. Does anyone get these results?