Lossless encoding and first pass XviD at one pass?

[pogo stick]

I had to do some encodes with veeery slow avs scripts lately. Usually I just encoded with lossless codec first and then with XviD. And I thought that maybe there is or could be a way to combine lossless encoding with first pass of XviD that I am not aware of. Or maybe it sounds crazy, but is something like XviD lossless mode or AviSynth's filter for first pass file gathering theoretically possible?

[akupenguin]

I can think of two easy ways.

Method 1 (preferred): Allow multiple output streams in avs2yuv (http://students.washington.edu/lorenm/src/avisynth/avs2yuv/). Pipe one to mencoder doing huffyuv, the other to another instance of mencoder with xvid. Under Linux this is trivial; under Windows it might require fork/exec'ing an mencoder from within avs2yuv, due to lack of adequate redirection (fifos).

Method 2: An avisynth plugin could do either or both encodes. See SClavc (http://students.washington.edu/lorenm/src/avisynth/sclavc/) for a proof of concept. (This one uses libavcodec, so it would most easily apply to the lossless part, unless you're willing to switch mpeg4 codecs. But the idea would hold for xvid, too.)

[pogo stick]

Thank you! :)
As I understand, SClavc is not for XviD, so I will be trying multiple outputs in avs2yuv.
What is fork/exec'ing?
Is mencoder the only program on windows that avs2yuv can pipe to?

[akupenguin]

MEncoder is the only encoder program for Windows I know of that supports pipes. (All others use one or more of vfw, directshow, or files.) This is probably because there's no good way to do Unix-style pipes in a GUI.


If you could use fifos, I would hypothetically do:
mkfifo a.yuv b.yuv
avs2yuv foo.avs -o a.yuv -o b.yuv &
mencoder a.yuv -o hfyu.avi -ovc lavc -lavcopts vcodec=huffyuv:<other_opts> &
mencoder b.yuv -o /dev/null -ovc xvid -xvidencopts pass=1:<other_opts>
mencoder hfyu.avi -o out.avi -ovc xvid -xvidencopts pass=2:<other_opts>
But since you don't have fifos on Windows, I would have to hard-code the huffyuv command into avs2yuv, and instead use
avs2yuv foo.avs -o - -hfyu hfyu.avi | mencoder - -o NUL: -ovc xvid -xvidencopts pass=1:<other_opts>
mencoder hfyu.avi -o out.avi -ovc xvid -xvidencopts pass=2:<other_opts>
, which would internally call the commands from the first version (that is fork/exec). More work for me, less work but less flexibility for you.

edit: just to be clear, neither -o nor -hfyu are implemented yet.

[pogo stick]

I will try it.
Thank you, again. You are like inexhaustible source of information. :)

[lark]

strange, but i had pretty much the same idea (or wish) and was wondering, if this could be achieved with AVS2AVI:
1st pass for XviD could be done with -o n (w/o any output) and
the only output would be
- .pass file
- Huffy AVI

regards
t :)

[symonjfox]

Originally posted by lark
strange, but i had pretty much the same idea (or wish) and was wondering, if this could be achieved with AVS2AVI:
1st pass for XviD could be done with -o n (w/o any output) and
the only output would be
- .pass file
- Huffy AVI

regards
t :)
Who knows? Will it be difficult to implement?

It's a very good idea, very interesting and very useful.

[HalfHuman]

yup! interesting ideea.
the xvid makers should take a look at the ideea.

[akupenguin]

Ok, both modes are now implemented in avs2yuv 0.21 (students.washington.edu/lorenm/src/avisynth/avs2yuv/), exactly as I proposed above. (The fifo mode doesn't work under Windows, and the -hfyu mode doesn't work under Wine.)

[pogo stick]

Great! :)
I am trying it right now. First pass seems to work good so far.
I wonder what prediction huffyuv uses with avs2yuv. Is it median? And I hope it's YV12.
Also because I don't have much experience with MEncoder I have some questions in connection with XviD:
I am using this:

avs2yuv G:\BTW\avs\btw05.avs -hfyu G:\BTW\ll\llx.avi -o - | mencoder - -o NUL: -ovc xvid -xvidencopts pass=1:max_bframes=2:quant_type=h263
mencoder G:\BTW\ll\llx.avi -o G:\BTW\x\x5.avi -ovc xvid -xvidencopts pass=2:gmc:trellis:chroma_me:chroma_opt:vhq=4:bvhq=1:max_bframes=2:par=ext:par_width=20:par_height=19:bitrate=1500:packed:quant_type=h263

Am I right in thinking that cartoon, interlacing, max_bframes and quant_type must be set in first pass? And qpel, gmc, trellis, chroma_me, chroma_opt, vhq and bvhq can be set only at second pass without reducing quality?
"-passlogfile x.log" doesn't seems to matter in the first pass. It's not a problem, but why?
Is first pass running at quantizer=2, like it is in vfw version?
hq_ac (Activates High Quality AC coefficient prediction from neighbor blocks) is new to me. Is it recommended?
Are there some other things that I am not aware of, but I should?
I am using Celtic Druid's build from 2004.10.26 for P4. There is a new one, but I can not download it right now.
Hope that I am not too bothering and thanks in advance. :)

[pogo stick]

There is something wrong with first pass huffyuv file. :(
Instead of 21001 frames only 3691 can be decoded.
Then I repeated first pass with 4001 frame file, but mplayer/mencoder are telling:

AVI file format detected.
AVI: ODML: Building odml index (1 superindexchunks)

while trying to open it and everything stops.
VDub and DS players can not open both files.

[akupenguin]

Originally posted by pogo stick
I wonder what prediction huffyuv uses with avs2yuv. Is it median? And I hope it's YV12. Yes, I use median prediction and YV12.

Am I right in thinking that cartoon, interlacing, max_bframes and quant_type must be set in first pass? Cartoon shouldn't make much difference on the first pass, but it also shouldn't take any extra time, so yes, set it on. The others must be the same on both passes.

And qpel, gmc, trellis, chroma_me, chroma_opt, vhq and bvhq can be set only at second pass without reducing quality? Right.

"-passlogfile x.log" doesn't seems to matter in the first pass. It's not a problem, but why? Despite it being a mencoder global option, the xvid wrapper ignores passlogfile and uses it's own hardcoded value. It might have happened when we imported xvid 1.1

Is first pass running at quantizer=2, like it is in vfw version? Yes, same as vfw (which is I,P quant=2, B quant=4.)

hq_ac (Activates High Quality AC coefficient prediction from neighbor blocks) is new to me. Is it recommended? I think it's been available for a while, but not documented until recently.
It should always help (2nd pass), though I can't tell you how much.

[akupenguin]

Originally posted by pogo stick
Instead of 21001 frames only 3691 can be decoded.
Is frame #3691 (in the huffy) exactly on the 1GB boundary?
Do all large huffys produce the same error, or do some play partially while others crash mplayer?

[pogo stick]

Originally posted by akupenguin
Is frame #3691 (in the huffy) exactly on the 1GB boundary?
I think it is, probably.
Originally posted by akupenguin
Do all large huffys produce the same error, or do some play partially while others crash mplayer?
I only had a time test 2 files.
First one played partially - only 3691 frames.
Second (4001 frames one) didn't work at all.

I had to go to work and now I am away from my PC.
I will test it more when I will come home.

[lark]

thx a million for the new feature in avs2yuv. haven't had a chance to test it yet (i'm currently at work...), but looking at the src, i spotted one potential bug.

end = seek + atoi(argv[i+1]);

this would mean that if -seek is after -frames in the command line, seek is not taken into account when setting the end. i think this could be fixed by adding

if(end > 0) end = seek + end;
to the -seek's if branch...

regards
t :)

[akupenguin]

Originally posted by lark
this would mean that if -seek is after -frames in the command line, seek is not taken into account when setting the end.
fixed.

[lark]

quoting myself ;-)


strange, but i had pretty much the same idea (or wish) and was wondering, if this could be achieved with AVS2AVI:


i finally managed to steal some time for this idea and i implemented it based on avs2avi 1.39 source... and it seems to work. i haven't had basically any time for heavy testing or benchmarking.

as mentioned earlier, this one doesn't output anything for the other encoding (just generates the XviD, DivX, whatever compressor was chosen 1st pass or equivalent stats file). does anyone see that as a problem? i think, it should be possible to generate full AVI output for both encodes.

the advantage over avs2yuv is obviously that there's a dialog box where the user can select which compressors to use and set their parameters.
disadvantage is obviously that it's only for windows...

if someone is interested, i can post the proggy for your disposal.

regards
t :)

[NuPogodi]

I did brief test whether one could use the 1st-pass output as the source for the 2nd-pass or not... it might be not so elegant solution as to separate the avs-output, but unfortunately i have not too large HDD for Huffy's output. OK, what did i exactly?

XviD parameters: AS@L5, GMC, QuantMatrix=H.263, BVOPs=2, Motion Search=6, VHQ=1, Croma Motion=On, Trellis=On.
1-pass settings: Full 1st Pass=On, Discard 1st Pass=Off, Zone Options -> Quantizer 1.0, 1.5 & 2.0
2-pass settings: the same except for Zone Options -> Weight = 1

In other words, 1st-pass was used to generate 3 output files (Q=1.0, Q=1.5 and Q=2.0). Their bitrates, encoding time, PSNR are

BR Time(mm:ss) Size -> PSNR(Y,U,V)
4338kbps 4:28 68,0MB -> 43.39, 49.49, 48.77
2855kbps 4:00 44,3MB -> 42.37, 49.15, 48.45
1549kbps 3:28 24,0MB -> 41.39, 48.31, 47.72

Then, each of these files as well as initial video were used as the source for the 2nd pass, one used stats generated during the 1st pass. PSNR of these 4 files were afterwards compared. Ok, to the results... Large difference in PSNR is seen at relatively large 2nd-pass bitrates (when the losses during the 1st pass are essential). But when the 2nd-pass is performed at low bitrates, there are nearly no matter which source has to be used for the 2nd-pass. Intermediate bitrates (say, 1000kbps) of the 2nd-pass: the PSNR difference is about 0.3dB if the 1st-pass is encoded at Q=1...

Pro: no need to filter videosource twice and no need to repeat the 1st-pass as required, say, for Huffy-output.
Contra: there are losses especially visible for high-BR encodings.

[akupenguin]

Even if the average PSNR is only .3dB worse, I would bet that the distribution of quality over time is farther off. Q=1 doesn't do such a good job of estimating complexity when the 2nd pass averages Q=3.