Which "least lossy" would you recommend me for permanent storage?

[Chainmax]

For anime, cartoons and CG material I guess that CorePNG is probably the best alternative, since it compresses like a mofo while still being lossless. But for real life footage CorePNG is no good (it gets as big as huffYUV, if not bigger), so I wanted to know which lossy codec is the best for permanent storage. So far, I have thought of two alternatives:

XviD @ Quantizer 2, no b-frames, VHQ = 4, Andreas78 or MPEG+QPel.

MJPEG @ q19 with luminance quality @2 and chrominance quality @3.

Are there any other good alternatives? If not, which one of these two would you use and why?

[Sirber]

RV9@CQ90

[Tommy Carrot]

Originally posted by Chainmax

XviD @ Quantizer 2, no b-frames, VHQ = 4, Andreas78 or MPEG+QPel.



That's it, but without qpel. ;)

[Sirber]

QPel gives better motion right? so why no q-pel?

[Tommy Carrot]

Because it creates visible artifacts, which is a big NO for archiving. :)

[iago]

To be on the safer side concerning decoding issues (at least if you wanna use ffdshow-libavcodec for decoding) not Andreas78 either.

XviD, Q2, Chroma Motion, VHQ4, MPEG ;)

regards,
iago

[Chainmax]

First of all, I'd like to thanks everyone for the quick replies :).

Sirber, you seem to have a different opinion than Tommy Carrot or iago. Since I know nothing about RV9, could you give me a short but throughout explanation about why RV9@CQ90 would be better than XviD@quant2 or MJPEG at the settings I described?

Tommy: I understand that QPel would be an issue for permanent storage or raw clips. That's mostly what I intend to do, but if I do filter the clip beforehand is QPel still a no-no?

iago: what problems can Andreas78 cause when decoding with ffdshow? What if I decode using Nic's decoder? BTW, I do intending to use MSP6-ultra high+chroma motion+chroma optimizer. I didn't want to make the list longer than it had to be ;).


So, what would you say is the approximated average bitrate XviD at quant=2 uses?

[Tommy Carrot]

Originally posted by Chainmax

Tommy: I understand that QPel would be an issue for permanent storage or raw clips. That's mostly what I intend to do, but if I do filter the clip beforehand is QPel still a no-no?
I wouldn't use it, but i'm very allergic to the artifacts of qpel, even at quant 2, so my opinion probably doesn't count. :D If it doesn't bother you, feel free to use.

So, what would you say is the approximated average bitrate XviD at quant=2 uses?
It's hard to estimate, but if the source is clean, it will give much lower bitrates than mjpeg, not to mention the lossless codecs.

[iago]

Originally posted by Chainmax
iago: what problems can Andreas78 cause when decoding with ffdshow? What if I decode using Nic's decoder?Afair, as discussed and reported repeatedly in the XviD forum, libavcodec has/had problems with quantization tables containing coefficients lower than 16 or 12. Since Andreas78 matrix uses such low values, with libavcodec decoding you might occasionally come across some dequantization errors, such as big blocks! ;)

If you use Andreas78, you'd better use XviD instead of libavcodec to decode.

regards,
iago

[Prettz]

Why not just go with good old MPEG-2 if you want "archival quality"? You can just keep ramping up the bitrate with MPEG-2 and the quality will keep getting better. Even Xvid seems to sometimes reach a quality "ceiling" at quant 2 and MPEG matrix.

[Sirber]

I read somewhere there is a MPEG2 with I frames only (lossless) for AVI.

[mf]

I'm now encoding the revolutions trailer at 1024x432, with settings GMC+CM+Trellis+B-Frames_2-100-100, and with my just invented matrix "mf-insanity", which is filled with only eights. Current estimated filesize is 270MB. I'll calculate later what bitrate that calculates to. :D

[mf]

Ok, I've finished encoding.

Filesize: 240MB
Length: 98 seconds
Bitrate: 19.591Mbit/s
Resolution: 1024x432

:D

[Chainmax]

Prettz: well, I am asking for opinions. I haven't decided yet what to use.

mf: GMC? Are you using devapi-4?


While we're at it, what matrices do you guys use for über high bitrate encodes and why?

[mf]

Originally posted by Chainmax
mf: GMC? Are you using devapi-4?


While we're at it, what matrices do you guys use for über high bitrate encodes and why?
Yep. And I mentioned above that I used my own "mf-insanity" matrix for the encode.

Edit: I was afraid the blocks I saw when I screened my encode were from XviD, but further examination showed the blocks were 11x8 pixels, (the DVD is anamorphic), meaning they can't possibly come from XviD. So I think it's safe to say these settings are fine for near-lossless storage, since all the DVD's artifacts are still there, and I can't seem to find any new ones. (Proves that DVD quality still sucks, even for "high-quality" titles like The Matrix Reloaded - if I had the original digital transfer, I could probably fit much higher MPEG-4 quality at the same resolution on that stupid DVD. MPEG-2 is so inferior :D.)

[ChristianHJW]

I wonder what the opinion of the current XviD dev team is on quant = 1 ?

Especially together with Tim Jansen's XviD DShow encoder filter it could give nice results in either constant quantizer or constant quality mode for analog capturing ?

[mf]

Originally posted by ChristianHJW
I wonder what the opinion of the current XviD dev team is on quant = 1 ?

Especially together with Tim Jansen's XviD DShow encoder filter it could give nice results in either constant quantizer or constant quality mode for analog capturing ?
As far as I can see quant 2 with my insanity matrix is good enough as well. Better than any lossy MJPEG anyway.

[Chainmax]

I wish devapi4 was released to the general public...I'm only so impatient because I figure it's going to be teh shiznit :).

[Tommy Carrot]

Originally posted by Chainmax
I wish devapi4 was released to the general public...I'm only so impatient because I figure it's going to be teh shiznit :).

There is at least one public build. You just have to find it. :D

/hint: GomGom's build/

[outlyer]

Originally posted by Tommy Carrot
There is at least one public build. You just have to find it. :D
But it's still in pre-release status, or is it?

[SeeMoreDigital]

Originally posted by Chainmax
... So far, I have thought of two alternatives:

XviD @ Quantizer 2, no b-frames, VHQ = 4, Andreas78 or MPEG+QPel.

MJPEG @ q19 with luminance quality @2 and chrominance quality @3.

Are there any other good alternatives? If not, which one of these two would you use and why? If the purpose here is to find out which lossy codec looks best at the lowest bitrate then it's probably RV9. However VP6 looks very good too!

That said, do you have an image pixel frame size (resolution) in mind, that you want to use in order to save your treasured clips?

If bitrate is not a problem I would have thought, once you get around 3000kbps any of the usual suspects (DivX, XviD, RV9, WMV9 .wmv or WMV9 VCM) will be very much the same.

Cheers

[iago]

If bitrate is not a problem I would have thought, once you get around 3000kbps any of the usual suspects (DivX, XviD, RV9, WMV9 .wmv or WMV9 VCM) will be very much the same.As mf also mentioned (though his matrix consisting of only eights is really an insane one :D), with a high-bitrate matrix -such as Andreas78- and where bitrate is not an issue, XviD quality will definitely surpass the highest quality results of all the other alternatives you mentioned.

regards,
iago

[Didée]

edit: after some typing and some more thinking, I hit <subit>, and then got "the forum is closed for backup". Bad Luck :(

iago:
Nice to see you around again!

All:
Quality-wise, something like an all-8 matrix is quite reasonable for capturing (if you have enough CPU horsepower) and archiving. BUT I want to remind once more that one might run into playback problems with an all-8 matrix: ffdshow most likely will mess up on playback. One will need to use another decoder, like xvid.ax itself (but devAPI4 ??), perhaps 3ivx decoder, or another generic mpeg-4 decoder.

Plus, all-8 matrices give *huge* filesizes at quant 2:[mf]
Ok, I've finished encoding.

Filesize: 240MB
Length: 98 seconds
Bitrate: 19.591Mbit/s
Resolution: 1024x432Very nice, mf!
... So you are going to put that movie on 4 DVDs, or what ?!? :)

BTW: My suggestion for maximum, ffdshow compatible, high bitrate matrix:

16__8__8__9__9_10_10_11
_8__8__9__9_10_10_11_11
_8__9__9_10_10_11_11_12
_9__9_10_10_11_11_12_12
_9_10_10_11_11_12_12_13
10_10_11_11_12_12_13_14
10_11_11_12_12_13_14_15
11_11_12_12_13_14_15_16

for both inter and intra (intra with "8" in the upper-left cell, instead "16", of course).
That matrix is compatible with ffdshow, is - at quant 2 - visually not at all to distinguish from "all-8", but comes out at a more reasonable bitrate.

- Didée


P.S.

@ mf: Shame on you! Stand in the corner! ;)
You won't seriously call an "all-8" matrix "your invention", won't you ;) - Or shall I present my latest invention: "Didée's ultra insanity matrix"! It took me several days to figure it out: after lots of tweaking, testing, tweaking again - the "all-1" matrix was born !!!!!
:)

[mf]

Originally posted by Didée
BUT I want to remind once more that one might run into playback problems with an all-8 matrix: ffdshow most likely will mess up on playback. One will need to use another decoder, like xvid.ax itself (but devAPI4 ??), perhaps 3ivx decoder, or another generic mpeg-4 decoder.
This has been mentioned before. My clip is pretty crappy with ffdshow, but after disabling it I found out the Nero MPEG decoder took over, and it worked great (I only have my self-compiled api4 build installed, so I don't have any xvid.ax installed). I don't know about other decoders though, but dshow can always fallback on the VfW-to-DShow wrapper, using the xvid vfw dll.
Plus, all-8 matrices give *huge* filesizes at quant 2:Very nice, mf!
... So you are going to put that movie on 4 DVDs, or what ?!? :)
Of course not. One fixed HD of course :D.
@ mf: Shame on you! Stand in the corner! ;)
You won't seriously call an "all-8" matrix "your invention", won't you ;) - Or shall I present my latest invention: "Didée's ultra insanity matrix"! It took me several days to figure it out: after lots of tweaking, testing, tweaking again - the "all-1" matrix was born !!!!!
:)
Show me former art then :D. I was definitely first, right ? :rolleyes: Only someone like me can be insane enough to invent such a matrix, and yours is only a derivative :p.

[iago]

Didée,

Interesting that this half-insane matrix of yours is libavcodec compatible. I'm gonna give it a try and compare the results visually with those of the all-8 matrix.

But now I guess it has become even more difficult to theorize the incompatibility issue with ffdshow. Formerly, you know, this incompatibility was generally attributed to quantization values lower than 16 or 12. However, here, in your matrix, we have almost all coefficients lower than 16. Any ideas on that? Or, in other words, what and where is the borderline that makes a matrix (in)compatible with libavcodec?

regards,
iago


P.S.

* If only you attached it as a zipped/rarred .txt file! ;)
* Btw, DivX 5.1.1 decoder seems to decode all XviD files properly (except those with Qpel and GMC), even the ones created with an all-8 matrix (dev-api-3).

[Didée]

iago:

All I can report on that topic is resulting from try-and-error ... I have no hard facts about how and why ...
(and I cannot attach anything to my posts btw, just like you, since we're no moderators.)

Some outlines for ffdshow compatible custom quant tables:

1. Inter matrix DC >= 16 (topmost-leftmost)

2. From top-left to bottom-right, the values must raise by a certain minimal amount (compare: "all-16" matrix is failing sometimes). I don't know where the "critical point" for the "flatness" lies, though.

3. If you happen to have used a matrix that produces only sporadic, seldom errors, then sometimes these few errors vanish if you check "workaround encoder bugs - autodetect" -- but this works not all the time.

For "real" high bitrate encodings, I still use my personal matrix SixOfNine, slightly modded according to the actual source. ( mf: :p - ;) )

Unless some experienced codec devel' looks into that ffdshow-matrix-miracle, I guess we "Joe Advanced User"'s won't figure it definetly out.


@ mf:

(I hope we both agree it's all kidding...)

>> I was definitely first, right ?
>> Only someone like me can be insane enough to invent
>> such a matrix, and yours is only a derivative.

This I doubt. When I first used an all-8 matrix, XviD's B-frames were still in the far future ... so that was a looong time ago.

- Damn, if I only had let it patented by that time! :(

Of course, I won't argue about you being ulimately insane! And be assured, once you made it into the madhouse, I will visit you frequently :D

- Didée

[iago]

I cannot attach anything to my posts btw, just like you, since we're no moderators.I see; well, you know I'm a bit behind the recent developments! ;)

OK, here are some results and personal visual inspections based on a comparison with the original, using a 40 sec. source, a plain avs script, XviD q2 with Koepi's 24062003-1 (dev-api-3), and ffdshow-20030523 for decoding.

Maybe it provides some help to Chainmax, the thread starter! ;)

----------------------------------------
loadplugin("C:\DECODER\mpeg2dec3dg.dll")
mpeg2source("D:\SAMPLE\SAMPLE.d2v")
crop(8,16,-8,-20)
LanczosResize(640,352)
----------------------------------------

all-8 matrix
------------
* filesize: 33364 kb
* cannot be decoded with libavcodec (XviD and DivX 5.1.1 decodes fine)
* fine details, source grain, etc. retained - quality close to the original

Didee's matrix
--------------
* filesize: 20708 kb
* decoded properly with libavcodec
* fine details, source grain, etc. retained - quality close to the original

Andreas78
---------
* filesize: 10142 kb
* decoded properly with libavcodec
* fine details, source grain, etc. considerably lost - quality very good

MPEG (built-in)
---------------
* filesize: 6508 kb
* decoded properly with libavcodec
* fine details, source grain, etc. considerably lost - quality good


regards,
iago


p.s. regarding the cuckoo's nest, hehe, definitely, I'll also keep you company there! :D

[mfluder]

Although this isn't that high bitrate matrix like mf and Didée posted, it's a very good matrix that I use for all my encodings. You can find it here (http://forum.doom9.org/showthread.php?threadid=33499). It was posted a long time ago by ReferenceDivx, the guy who made that famous HVS matrices (good, better and best). It isn't his creation but a matrix that he found in some IEEE paper. It is also created with HVS (Human Visual System) in mind and IMHO it does a great job at that. Filesize is larger than with standard MPEG matrix at the same quantizer, but it also keeps more fine details. It is very good when used with b-frames (I use 2,150,100,10) and qpel. With this matrix and these settings I get a very crisp image with lots of fine details (qpel helps here a lot) that I wouldn't get if I used H.263 or MPEG quantization.

I recommend everyone to try it. It doesn't matter if you have a lower compressibility with it, you will end up with a better looking encode (but only in case fine details are what you prefer the most).

mfluder

[mikeson]

@mf:
with settings GMC+CM+Trellis+B-Frames_2-100-100, and with my just invented matrix "mf-insanity"
Does Trellis work with custom matrices too? Last time I checked changelog Trellis with MPEG quantization was added.

[mf]

Originally posted by mikeson
@mf:

Does Trellis work with custom matrices too? Last time I checked changelog Trellis with MPEG quantization was added.
H.263 divides all coefficients equally. Thus, trellis quantization is scaled for the entire macroblock. MPEG quantization allows for different scaling per coefficient, so scaling it equally for the whole macroblock would not be the right way. However, custom matrices filled with 1 value work the same as H.263, and thus Trellis can be used.
May devs correct me where I'm wrong ;).