Faster to first encode HuFFYUV when using dust?

[kingmob]

I've just been fidling with the new DivX 5.2 codec. I was trying it on my latest movies, on which i had to do some heavy filtering and i also used pixiedust on it, resulting in a slow avs file obviously.
Now all encodes are equally slow, no matter if I use slow or fast mode and with 3 passes this means long waiting times.
So now i was thinking, wouldn't it be better, when using mulitple passes in CCE or DivX for instance, to first encode to HuffYUV (which, as i understand it, preserves the quality exact) and then encode the HuffYUV file? I'm guessing this could greatly increase my speed when using slow filters like dust, but am i also right this would not have any impact on picture quality?

Oh, and i apologise for my many, probably very simple, questions, being a new member and all.

[kassandro]

You certainly can do that. Especially since Huffyuv encoding is extremely fast and Dust uses YUY2 anyway. Consequently you need to run the ultra slow Dust only once, which clearly saves you a lot of time and you suffer no quality loss either, because the yuy2 conversion was necessary anyway. But you need a lot of free disk space for this endeavour (for a 90 minute movie about 20 GB).

[kingmob]

Ok thx, i wanted to make sure if i wouldn't waste any quality, since my sources aren't that great already.

[kassandro]

Actually this is a nice trick for any slow avs script, if you have enough free disk space.

[scharfis_brain]

I always encode this way, cause all my AVS scripts are slow :devil:
(especially the restore24 ones)

[Didée]

Yeah, that's currently my mostly used X-pass encoding method, too. But I don't use Huffy, but VBLE instead. Or FFV1, sometimes. Final encoding is done to YV12 anyways, so I prefer the gain from lossless YV12 over lossless YUY2.

>> (scharfis_brain)
>> all my AVS scripts are slow (especially the restore24 ones)

Pfffh. Restore24 is "lightning fast". Seems you're not using iiP or LimitedSharpen :D

A typical processing chain:

Source -> (Restore24) -> Temp1(VBLE) -> TweakScript*) + iiP -> Temp2(VBLE) -> XviD 2-pass
+ -> CCE 3-pass

*) corrects few leftover hickups from Restore24 found by manual scanning

- Didée

[Lefungus]

What's the size of those temporary lossless files, usually ?

[Didée]

Depends alot on the input (surprise ;) ).

In my case, for a 40min-episode, VBLE compresses Restore24's output (704*432, no other processing) to something like 5 GB. The 2nd, iiP-processed one (704*528), usually is in the 8~10 GB range.

[kassandro]

If one uses Dust, it would be desirable to have also lossless conversion filters. With the builtin filters ConvertToYUY2().ConvertToYV12 vertically blurs the chroma. If there would be alternative conversion filters based on the mpeg1 instead of the mpeg2 specification, the above combination would be lossless and faster.

It would also be even more efficient, if XviD would have an option for lossless output during the first pass.

[Mug Funky]

lossless xvid... interesting. that sounds like a plan actually (so long as somebody's willing to implement it :))

how does mpeg-4 perform with i-frames only and with such low quantizers that the stream is practically lossless (i imagine there can be a loss of precision if it's less than 0.5/256)?

[kassandro]

Originally posted by Mug Funky
lossless xvid... interesting. that sounds like a plan actually (so long as somebody's willing to implement it :))

I don't know of concret attempt in this direction. For my encodes I usually prefer 1-pass encodes with fixed quantisers. Only for about 20% i use two passes. So for me personally there is no need for a lossless xvid.


how does mpeg-4 perform with i-frames only and with such low quantizers that the stream is practically lossless (i imagine there can be a loss of precision if it's less than 0.5/256)?

Actually I think of a lossless XviD with P-frames (and may be even B-frames). You simply make lossless compression of motion compensated frame differences instead of lossless compression of single frames as done by all current lossless codecs. The resulting file sizes should be significantly smaller, though compression will also be much slower with motion compensation. But XviD computes the motion vectors anyway during the first pass. Thus I would like to reuse most of the information gathered by XviD during pass 1 to get also a much more compact lossless compression, which then can be reused for a second pass without running the avs script again.

[morsa]

But in such a case, the compression could be called lossless?
I mean, if you compare your compressed video with the original, would it give an exact match??

[kassandro]

Originally posted by morsa
But in such a case, the compression could be called lossless?
I mean, if you compare your compressed video with the original, would it give an exact match??
As said already, this is a purely hypothetical discussion. The loss in JPEG and MPEGx comes through the DCT, which is used to drop high frequency details of frame blocks in the case of JPEG and (motion compensated) differences of frame blocks in the case MPEGx (non I-frames). My criticism of current lossless codecs is simply that unlike lossy codecs they ignore the continuity of movie pictures and encode all frames as still pictures, which is of course quite inefficient.

[Leak]

Originally posted by kassandro
My criticism of current lossless codecs is simply that unlike lossy codecs they ignore the continuity of movie pictures and encode all frames as still pictures, which is of course quite inefficient.

CorePNG 0.8.2 supports delta frames. It says "(unstable)" in the config dialog, but I haven't had any problems with them...

np: Sixtoo - Storm Clouds & Silver Linings (Chewing On Glass & Other Miracle Cures)

[kassandro]

Originally posted by Leak
CorePNG 0.8.2 supports delta frames. It says "(unstable)" in the config dialog, but I haven't had any problems with them...

np: Sixtoo - Storm Clouds & Silver Linings (Chewing On Glass & Other Miracle Cures)
Good to know that. Probably CorePNG doesn't use motion compensation, because it would be too costly. But for any XviD pass1 a motion search has to be done anyway and then these motion vectors could also be used for a better lossless compression. That was the point, which I wanted to make at the beginning.

[Didée]

(I'm no expert at all in lossless compression!)

The topic 'Lossless compression and ME/MC' was alread discussed thouroughly here on Doom9, quite some time back.

IIRC, the conclusion was that motion compensation is almost useless for lossless compression.

Why? By ME+MC, you hardly ever get a perfect match on block texture (except perhaps for *extremely* clean cartoon or CGI stuff, clips with lots of DUP frames, etc.).
On "normal" content, all you get by MC is image data with a different frequency level. In mpeg-4, the effect is so that this resulting different frequency level makes it more easy to drop certain frequency information without hurting the image too much.
But in lossless compression, you have to keep every single bit anyways - there's no information to throw away. And: You have to safe the motion vectors too. Either within the stream, or external - in both cases, they need space.
The deal is, through the need of keeping every single bit, the mere compression advantage of using motion vectors is so small, that the savings roughly equal the needed space for the motion vectors...


A small test on HP-II Trailer, ~3200 frames:

XviD: Quant=1. Matrix: "8 1 1 1 1..." / "1 1 1 1 1..." (The most "lossless" I can think of)
I-frame (ME=0): 501 636 kB
I+P, ME=4, VHQ0: 652 032 kB
I+P, ME=6, VHQ0: 660 176 kB
I+P, ME=6, VHQ4: 486 802 kB

Note #1:

"normal" search effort seems even contra-productive. Filesize gets bigger, instead of smaller, compared to I-frame only compression. Only with "extreme" search effort, there is a *small* advantage over I-frame compression.


Note #1a:

XviD's streams encoded with q=1, matrix="all=1" are *de*coded with artefacts by both XviD itself and ffdshow. So I'm not totally sure if above numbers make any sense at all. In relation to each other, they do make sense. But for the absolute values ... ?

Compare above numbers with

Note #2:

Same video stream:
Huffyuff YUY2 : 298 696 kB
ffdshow FFV1 : 168 982 kB (!)
(VLC, keydist=1)

I leave the analyzing to someone with better knowledge on this topic than me.


- Didée



edit: Oooops ... :o

[sh0dan]

ok - please open a new thread in New A/V Formats - Codecs (http://forum.doom9.org/forumdisplay.php?s=&forumid=54), as this hasn't related to AviSynth for a while. I'm leaving the thread open, if you want to post a link to the new thread.