XviD-21092002-1

[Koepi]

XviD-21092002-1:
- Updated to Statsreader 1.9.
- Added Foxer's modified 2pass code - internal scaling
should work as with statsreader now.

Foxer implemented the min[i|p]framesize formulas into XviD's internal 2pass code and modified the overflow treatment a little.

Give it a try, with these modification it the StatsReader's scaling capabilities may be obsolete! :)

Just use internal 2nd pass, disable alt CC and use the regular curve compression with high/low set to 0 - should be as good as StatsReader's curves!

Best regards,
Koepi

[iago]

@Koepi

Sorry to ask that question again ;), but 21092002-1 build works with 04092002-1 1stPass stats files with full compatibility (considering the latest changes), doesn't it?

If OK, I'll start another second pass using regular CC with altCC disabled and hi-lo: 0-0 this time ;), with the custom-matrix I tried using default altCC before (due to statsreader crash) ;)

Thanks a lot, btw ;)

regards,
iago

[Koepi]

Yes, I just changed the 2pass code which doesn't affect anything but the 2nd pass ;)

I'm eager to hear from your results! :)

Best regards,
Koepi

[TheUnforgiven]

very nice..
thanx koepi and foxer.
lets wait iago test result to confirm that.

[Koepi]

I'll post my first result in ~1 1/2h from now here :)
(Currently testing Monster's Inc. with it, first pass size 1.383MB, scaled to 646.374kb using the new internal scaling.)

Best regards,
Koepi

[iago]

@Koepi,

That's great! ;) Anyway, I've already started the new second pass with 21092002-1 internalCC, (linear scaling the curve by disabling altCC and setting hi-lo: 0-0 in regular curve compression - after so many repetitions, no one should be doing any more mistakes in these settings I guess! LOL! :D), since I was actually expecting this reply from you (they call it anticipation, no?) ;)...

I'll also post the results in 2/2.5 hours I guess, also comparing it to the second pass results I got with default altCC. (BTW, custom quantization matrix (which I've been using for both of the above mentioned encodes) based on HVS seems to do a really good job imho.)

best regards,
iago

[Koepi]

[4032] Quantizer distribution for 2nd pass:
[4032] Q:2:266
[4032] Q:3:12630
[4032] Q:4:96413
[4032] Q:5:12828
[4032] Q:6:372
[4032] Q:7:21

That looks pretty good for me :) the quant 6+7 frames where almost all during the start of the movie which has some complicated bitrate situations, so now I'm going to do a visual test - but I'm quite confident that the movie looks wonderful ;)

Regards,
Koepi

[MaTTeR]

I have a full movie comparison test running now also of "Heather's". Not the best DVD source, very soft and moderate noise. I'll post results in an hour or two. Thanks for the continued work guys.

[Dali Lama]

Koepi,

I am interested to know what Quantization Matrix you used for Monsters Inc. Also, which resize algorithm and to what dimension.

Thanks,

Dali

P.S. Sorry if that question bothers you. If you don't want to answer its ok. I am also working on that one.

[MaTTeR]

Did the modified 2 pass code in this build change the behavior of "ModHQ"? I'm noticing all the quants of 2 and 3 are MPEG where as before they were H263. Perhaps I've missed some news regarding this issue;)

[Koepi]

ARGS - yes, if i just replace my 2pass.c with foxer's one, there is no code for new mod HQ :-/ So it's not functional at the moment. I'll build another binary tonight/tomorrow which includes that code again.

dali,
I wasn't too impressed by the visual quality of that encode so I'm redoing the movie with other settings now, the settings above were:
- credits range 122530-132632 @50%
- h263
- Crop(10,0,702,576)
- Trim(0,122530).LanczosResize(512,384)+Trim(122530,0).BilinearResize(512,384).Convolution3d(0, 8, 8, 8, 8, 2.8, 0)

Now I'm doing credits without Conv3D and the main movie with neutral bicubic resize, h263 quant again, lumi masking,...

Regards,
Koepi

[MaTTeR]

Originally posted by Koepi
I'll build another binary tonight/tomorrow which includes that code again.


Thanks man, I'm starting to really like this ModHQ mode for lengthy movies going onto 1CD. It seems to cut down slightly on noise and slight ringing from what I've seen so far.

Edit- FWIW, ising the mode currently just reverts the entire 2nd pass to MPEG quants it seems.

[iago]

Hello again,

Movie: Salvador (1hr-57min) PAL / 512*272 / SimpleResize / Target Size: 665000kb
1st Pass: Koepi's 04092002-1 build
2nd Pass: Koepi's 21092002-1 build
MPEG-Custom / No lumi / Max-Min I-frame intervals: 300-5 / Quantizers capped: 2-6 - 2-16 / Payback proportionally / Payback delay: 300 / credits: 20quant


1stPass Size: 2165 mb

2ndPass results (internal CC, altCC disabled, hi-lo: 0-0)

Q:2:13
Q:3:770
Q:4:33274
Q:5:91594
Q:6:40538
Q:7:4161
Q:8:232
Q:9:3

(Average Quantizer Used for Movie : 5.086)


2ndPass results (internal CC with default AltCC):

Quant 2 Used : 131 Times, Percentage Used : 0.08%
Quant 3 Used : 3736 Times, Percentage Used : 2.19%
Quant 4 Used : 44102 Times, Percentage Used : 25.86%
Quant 5 Used : 73548 Times, Percentage Used : 43.12%
Quant 6 Used : 40827 Times, Percentage Used : 23.94%
Quant 7 Used : 7415 Times, Percentage Used : 4.35%
Quant 8 Used : 699 Times, Percentage Used : 0.41%
Quant 9 Used : 72 Times, Percentage Used : 0.04%
Quant 10 Used : 28 Times, Percentage Used : 0.02%
Quant 11 Used : 1 Times, Percentage Used : 0.00%

(Average Quantizer Used for Movie : 5.036)



These are some interesting results in trems of quantizers, open to discussion I guess ;).

Visuals: Well, though not much impressing, both are OK imho, watchable and acceptable, as much as they can be with these high average quantizer values.

best regards,
iago

[Mefisto]

Hi!

Can I use the ALT CURVE method with this versions?

Regards
Mefisto

[Dali Lama]

Hmm, I see.

Koepi,

I am encoding the Widescreen Version and going for 1-CD so...

Resize = 576x320 with Lanczos3
Quantization Matrix = CG/Animation
No Lumi
Quants I = 2-6
P = 2-12

I like the sharpness of Lanczos3 and The CG Matrix really brings out all of the detail. Note, these settings are only useful for extremely low noise CG films like Monsters.

I will let you know how it comes out.

Dali

Edit: Koepi, I think you are resizing incorectly, because Monsters Inc. only comes in Widescreen. When you watch it, you can choose either Full or Wide, because they crop the edges out or something during playback. Just wanted to let you know.

[Peters]

@iago

It seems that some frames are lost beetween the two tests :D


(170585 in first test, 170559 in second)

[iago]

@Peters,

Q:2:13
Q:3:770
Q:4:33274
Q:5:91594
Q:6:40538
Q:7:4161
Q:8:232
Q:9:3

Total: 170585 (without credits) is absolutely correct ;). And debugview spits it out correctly.

The next one (with lacking frames ;)) is from MoonWalker & Marc FD's XviD analyzer (new version 0.14: xda.exe) that still has a framecount problem I guess. (Unfortunately I don't have the original debugview log file any more.) I'd reported the problem to them before but it seems it's still going on. (The actual frame number for this one is 170585 as well, but "xda.exe" simply reports it wrong sometimes.)

regards,
iago

[Koepi]

dali,

well, my dvd player (dvdplay run from start-run) played it with this aspect ratio... I'll check widescreen now, let's see if it looks better then :)

Regards,
koepi

[Koepi]

I wasn't mistaken:

Technische Daten DVD: Bildformat: 1:1,33 für 4:3

(technical data DVD: aspect ratio 1:1,33 for 4:3)

You may own the special edition that has a 1:1.85 AR (Technische Daten DVD Deluxe: Bildformat: 1:1,85 für 16:9)

Regards,
Koepi

[rui]

Well, i made some tests this afternoon :) I am at home now, and here i don't have any comp problems (those were at my work).

First test: LOTR, using mpeg quantization and the latest statsreader 1.9, with 4/9 build:
Quantizers Used For Movie :
------------------------------
Quant 2 Used : 3505 Times, Percentage Used : 1.43%
Quant 3 Used : 165176 Times, Percentage Used : 67.27%
Quant 4 Used : 76124 Times, Percentage Used : 31.00%
Quant 5 Used : 695 Times, Percentage Used : 0.28%
Quant 6 Used : 37 Times, Percentage Used : 0.02%

Average Quantizer Used for Movie : 3.302

Quantizers Used For Credits :
--------------------------------
Quant 2 Used : 11059 Times.

Now, i am absolute certain that i configured the credits for 20 quantizer.

So, i made two quick tests using the The Replacements trailer, and Koepi's latest build, one using internal curve, and the other external.
The good news is that using internal is almost equal as using external with statsreader.
The bad news is statsreader 1.9 is making the credits to use quantizer 2 all the time :( :

Internal Quantizers Used For Movie :
------------------------------
Quant 2 Used : 68 Times, Percentage Used : 1.94%
Quant 3 Used : 2 Times, Percentage Used : 0.06%
Quant 4 Used : 24 Times, Percentage Used : 0.69%
Quant 5 Used : 244 Times, Percentage Used : 6.97%
Quant 6 Used : 418 Times, Percentage Used : 11.94%
Quant 7 Used : 704 Times, Percentage Used : 20.11%
Quant 8 Used : 1067 Times, Percentage Used : 30.49%
Quant 9 Used : 530 Times, Percentage Used : 15.14%
Quant 10 Used : 317 Times, Percentage Used : 9.06%
Quant 11 Used : 97 Times, Percentage Used : 2.77%
Quant 12 Used : 21 Times, Percentage Used : 0.60%
Quant 13 Used : 4 Times, Percentage Used : 0.11%
Quant 14 Used : 4 Times, Percentage Used : 0.11%

Average Quantizer Used for Movie : 7.656

Quantizers Used For Credits :
--------------------------------
Quant 20 Used : 70 Times.

External (using statsreader) Quantizers Used For Movie :
------------------------------
Quant 2 Used : 68 Times, Percentage Used : 1.94%
Quant 3 Used : 2 Times, Percentage Used : 0.06%
Quant 4 Used : 23 Times, Percentage Used : 0.66%
Quant 5 Used : 245 Times, Percentage Used : 7.00%
Quant 6 Used : 418 Times, Percentage Used : 11.94%
Quant 7 Used : 700 Times, Percentage Used : 20.00%
Quant 8 Used : 1062 Times, Percentage Used : 30.34%
Quant 9 Used : 531 Times, Percentage Used : 15.17%
Quant 10 Used : 319 Times, Percentage Used : 9.11%
Quant 11 Used : 100 Times, Percentage Used : 2.86%
Quant 12 Used : 25 Times, Percentage Used : 0.71%
Quant 13 Used : 4 Times, Percentage Used : 0.11%
Quant 14 Used : 3 Times, Percentage Used : 0.09%

Average Quantizer Used for Movie : 7.665

Quantizers Used For Credits :
--------------------------------
Quant 2 Used : 70 Times.

[Dali Lama]

I see then.

Sounds good, I'll keep you posted of the results in like 6hrs.

Take Care,

Dali

[Blight]

Koepi:
You need to update the settings guide on your site, it's completely out of date.

Also, I haven't really been following xvid all that well, so I'm somewhat missing your references regarding Alt-CC,
There is no such options as far as I can see, theres "Use Alternative Curve System", is this what you meant to disable?

[Defiler]

I thought I would post the results of my comparison between the use of StatsReader 1.9 and the new internal scaling from 21092002.
The target filesize was 620,000,000 bytes. The output files differ in size by only 600KB. I used the same first pass file for both.

First, the external scaling: (Build 09-04-2002)
------------------------------
Quant 2 Used : 942 Times, Percentage Used : 0.88%
Quant 3 Used : 28676 Times, Percentage Used : 26.90%
Quant 4 Used : 60102 Times, Percentage Used : 56.39%
Quant 5 Used : 15479 Times, Percentage Used : 14.52%
Quant 6 Used : 1368 Times, Percentage Used : 1.28%
Quant 7 Used : 18 Times, Percentage Used : 0.02%
Average Quantizer Used for Movie : 3.885
Compressibility : 37.13%
Relative Quality of XviD avi : 51.48%
Absolute Quality of XviD avi : 94.35%


Next, internal scaling. (Build 09-21-2002)
The curve settings looked like this. I assume this is correct?
http://hellninjacommando.com/misc/xvid-09-21.png
------------------------------
Quant 2 Used : 1 Times, Percentage Used : 0.00%
Quant 3 Used : 17874 Times, Percentage Used : 16.77%
Quant 4 Used : 77392 Times, Percentage Used : 72.61%
Quant 5 Used : 10830 Times, Percentage Used : 10.16%
Quant 6 Used : 360 Times, Percentage Used : 0.34%
Quant 7 Used : 82 Times, Percentage Used : 0.08%
Quant 8 Used : 29 Times, Percentage Used : 0.03%
Quant 9 Used : 17 Times, Percentage Used : 0.02%
Average Quantizer Used for Movie : 3.945
Compressibility : 37.14%
Relative Quality of XviD avi : 50.70%
Absolute Quality of XviD avi : 94.17%

Edit:
I decided to make a third test, using internal scaling, with Alt CC set to "Low" aggression.
Here are the results: (Build 09-21-2002)
------------------------------
Quant 2 Used : 1068 Times, Percentage Used : 1.00%
Quant 3 Used : 32767 Times, Percentage Used : 30.74%
Quant 4 Used : 55799 Times, Percentage Used : 52.35%
Quant 5 Used : 12009 Times, Percentage Used : 11.27%
Quant 6 Used : 4358 Times, Percentage Used : 4.09%
Quant 7 Used : 582 Times, Percentage Used : 0.55%
Quant 8 Used : 2 Times, Percentage Used : 0.00%
Average Quantizer Used for Movie : 3.883
Compressibility : 37.14%
Relative Quality of XviD avi : 51.50%
Absolute Quality of XviD avi : 94.35%

[Koepi]

XviD-22092002-1:
- Fixed 2pass code - it was totally messed up, credits/external now works again.
- New mod. HQ quant type reimplemented.

I just realized this stuff as my movie came out 50MB oversized - my credits were all quant 2 instead of quant5 as well :(

Well, it's fixed now, and i've a little cutting/joining to do now.

Regards,
Koepi

PS: Blight, feel free to update the documentation, it's outdated in a week again or in two if new API3 arrives...

[Koepi]

Ok, fixed the credits of my movie, now I used ext. scaling again and hit +4kb (right on spot) my desired size.

[3236] Quantizer distribution for 2nd pass:
[3236] Q:2:439
[3236] Q:3:25163
[3236] Q:4:84649
[3236] Q:5:12081
[3236] Q:6:198

This time I used mpeg quant - but most important, I used MarcFD's mpeg2dec.dll with postprocessing. The movie has much blockyness from DVD(!), setting postprocessing to 6 solved these issues partially (look at frames 11756-11762, the red car passing by...).

The visual quality is somewhat amazing, though I don't like the mosquito noise introduced by using mpeg quant type. Redoing a h263 encode now with the new XviD binary.

Regards,
Koepi

[MoonWalker]

Originally posted by Koepi
The visual quality is somewhat amazing, though I don't like the mosquito noise introduced by using mpeg quant type. Redoing a h263 encode now with the new XviD binary.

Koepi, you can try TemporalSoften2(1,5,0), it removes much from the MPEG noise and it doesn't loose any detail(or very little).

MoonWalker

[Koepi]

Hm, I could do that later when this 2pass is ready. But I prefer not to filter my sources from DVD, the post processing is already too much for my taste on usual movies (e.g. matrix), so I think I'll hate the result with temporal soften ;)

Are you positive that this line removes blockyness in the sources?

Thanks,

best regards,
Koepi

[iago]

@Koepi

I guess the messed up part of 21092002-1 binary was 2ndPass when used with external CC, right? Since I had no problems using it with 2ndPass internal, altCC disabled and regular CC set to hi-lo: 0-0 for linear scaling of the curve (credits 20quant just as set, and no oversize/undersize, and I really liked the result I got using h263 and simple resize). Can you confirm this please? Otherwise, I'll start a reencode with the 22092002-1 binary? ;)

thanks,
iago

[Koepi]

Yeah, it concerns 2nd pass external mode, confirmed. You don't need to reencode ;)

Regards,
Koepi

[iago]

Thanks! ;) So time to burn it and start another encode with the 22092002-1 binary now! ;)

iago

[OUTPinged_]

koepi, is it possible to find out what did foxer change in 2pass curve scaling code without messing with sources again?

Can you or foxer state there what does "modified min[i|p]framesize formulas into XviD's internal 2pass code and modified the overflow"

I mean, how is now min framesize calculated and what is changed in overflow distro behavior.


And one more question. Can anybody prove that using sharp resize algo (lanczor in example) coupled with h23 quantizer is more efficient than soft bicubic+mpeg quant? I mean, is it possible to have no mosquito noise and still have sharp picture :-|

[Koepi]

outpinged:

follow the stats reader thread, it's explained very well there.

In short, frames got scaled no matter what size they had, but there is a natural minimum frame size - for iframes and for pframes.

This code limits the downscaling to this minimum frame size, thus in low bitrate scenes there won't be unneccessary stack-up of overflow which can destroy some frames/scenes after this stack-up occured.

Koepi

[MoonWalker]

Originally posted by Koepi
Hm, I could do that later when this 2pass is ready. But I prefer not to filter my sources from DVD, the post processing is already too much for my taste on usual movies (e.g. matrix), so I think I'll hate the result with temporal soften ;)

Are you positive that this line removes blockyness in the sources?

Thanks,

best regards,
Koepi

I have testeded it on Matrix.No postproccesing.Just the usual mpeg2dec.dll...It removes much of the MPEG noise XviD generate, it filters only the luma channel (as Acaila has told me) and it preserver much from the original source.

MoonWalker

[kilg0r3]

i don't see why one would only want to filter the luma chanel when detail loss on chroma is less noticable to the eye. furthermore acaila's filter chain also involves chrome denoising with cnr2

[Acaila]

The filter chain that was discussed in the avisynth forum has undergone some changes as time passed.

Basically I now filter as little as possible, however to get rid of the noise created by MPEG (which appears to be mostly luma noise) I use TemporalSoften2(1,5,0) like Moonwalker said. It doesn't get rid of all of it, but a big portion of it. I don't believe it works very well against blocking, just mosquito noise, as a strength of 5 is quite low.
I only filter luma because I want to get rid of the noise, so there's not much reason to filter chroma too as the detail loss can only increase.

But let's not turn this thread into a filter discussion please.

[MoonWalker]

Originally posted by kilg0r3
i don't see why one would only want to filter the luma chanel when detail loss on chroma is less noticable to the eye. furthermore acaila's filter chain also involves chrome denoising with cnr2

I didn't I used acaila's filter chain..I said Acaila suggest me to use this..I liked the results I saw, so I suggest the same..

MoonWalker

[MoonWalker]

WOW...Simultanious answer :)

MoonWalker

[Dali Lama]

Originally posted by Koepi

The visual quality is somewhat amazing, though I don't like the mosquito noise introduced by using mpeg quant type. Redoing a h263 encode now with the new XviD binary.

Regards,
Koepi

I don't know if you are reffering to the Monster Inc. Encode, but if you are, try the CG/Animation Matrix I posted in the Custom Matrix thread and you should see an improvemnt in ringing.

Also, if you are doing a regular movie, like Matrix, try the Ultimate Matrix...(I laugh everytime I say that).

As far as the visual result of my Monster Inc. encode, its really amazing. So much sharpness is retained with Lanczos resize and CG/Animation Matrix. Although ringing and blocking is held to a minimum. This is 1-CD too. I don't use debugview, so I can't say the quant distribution. However, I changed the caps to

I frame = 2-5 and P frame = 2-10

Happy testing,

Dali

[Koepi]

Dali,

you should redo that movie when h's code is in CVS (didn't do a CVS checkout yet...) - he fixed the issue, resulting in 10mb more for movie data which can improve the quality a little bit more ;)

I'll try the CG matrix, thanks :)

Regards,
Koepi