Cvs 2002-03-02

[-h]

Changes:

- Keyframe boost % setting added (internal 2-pass)
- File size fixes
- Undersizing/crash with credits quant mode fixed
- I-frames ignoring quant restrictions fixed
- Extreme underflow prevention
- Quantizer smoothing now always enabled
- External 2-pass mode should now work again

Fingers crossed (as usual)...

Also, ignore the "Custom" option in the quantization mode drop-down, and the new button on the Debug tab. It's only there to help suxen_drol work on some core stuff, and won't do anything at the moment (apart from crash, probably).

-h

[-h]

I haven't tested this myself, but you should be able to get a bit more speed out of XviD by changing core slightly - at the moment, all the DEBUG(xxx) stuff is enabled in portab.h, and leads to quite a few branch mispredictions and other hits in speed (all the coefficient overflow checks, etc.).

If you feel like commenting out the debug stuff in portab.h the same way it is in codec.h, it might be a fun experiment.

-h

[Klumsy]

What is keyframe boost used for?

[-h]

If you set the boost to be 20%, then I-frames will receive 20% more bits. It's a "better way" of increasing the quality of I-frames, as it's more relative to the movie itself than the I-frame min/max quant controls are.

-h

[Dali Lama]

Is Nic or Koepi going to compile this?

No rush,

Dali

:)

[-h]

I'm told it's on uManiac's site.

-h

[Koepi]

XviD-02032002-1:
- the last versions had some troubles with wrong initialised variables, these issues
should be solved in the new CVS.
- re-implemented quantizer type modulation.

Find it following the XviD link in my signature.

Regards,
Koepi

[CavalloPazzo]

Why Quantizer modulation isn't added to the CVS and Koepi have to add it every time?

[-h]

Adding it to the CVS would break compatibility with "other" programs that happen to be using XviD's core at the moment. The CVS commit is waiting on a new official core API, it shouldn't be long now.

-h

[Nic]

Difficult for me to compile over the weekend :(

So, ill leave it until Monday, when hopefully ill be able to add my DShow filter to it :)

"Quantizer smoothing now always enabled" ?? Is that really wise? (im not challenging you on it, but because although I understand the idea, im not sure how it will effect the quality) :)

Hope the testing goes well :)

Cheers,
-Nic

ps
I got the DShow filter compiling in VC7 (had to go change a header file in the DX8 SDK to work (the Platform SDK (comes with Visual Studio .net) & DX8 have incompatible headers (!?!?)

[-h]

"Quantizer smoothing now always enabled" ?? Is that really wise? (im not challenging you on it, but because although I understand the idea, im not sure how it will effect the quality) :)

I think it's universally ugly to have the quantizer fluctuating like crazy. If someone complains, we'll put the checkbox back in :)

It eliminates one of those things that someone could uncheck, not fully understanding it, then get confused at the weird jumps in quality after I-frames. And then post a question here about it :)

-h

[Teegedeck]

I love that new way to treat i-frames! Thank you!

[Ripe73]

I try to download Nic's latest build on http://xvid.stormpages.com/
but i see this
Dont right-click on this file, left click to download! (Might only work in IE!))
XVID - 18/02/02

Is that build old??
Or have you not changed the name?

[rui]

Originally posted by -h
If you set the boost to be 20%, then I-frames will receive 20% more bits. It's a "better way" of increasing the quality of I-frames, as it's more relative to the movie itself than the I-frame min/max quant controls are.

-h

One question: If one uses this 20% boost option, it's not a good idea to setup in the debug tab a fixed I frame quantizer (like 2-6), right?
One would have the possibility that the I-frames got oversized a lot, correct?

[-h]

One question: If one uses this 20% boost option, it's not a good idea to setup in the debug tab a fixed I frame quantizer (like 2-6), right?

If you're restraining I-frames, I'd set the kf boost to 50%, and leave the I-frame quants at 1-31. Then.. well look at the dbgview output and see if it's what you wanted :)

-h

[Koepi]

You can still constrain your IFrames. This keyframe boost gave me more results like this:

02/03 28/02
kf 2 3
df 4 4
df 4 3

...
it's once more a matter of taste.
But if you leave away keyframe strictness, I dunno if it would always use e.g. quant 2 for keyframes....

Ah well, I have to check that ;)

Regards,
Koepi

[Ripe73]

I guess i have to go for Koepi's build then http://xvid.stormpages.com/ seems to be the wrong build

[Nic]

Yup, its best to go for Koepi's for now :)

-Nic

[Actron]

Just a first experience of the newest build (cvs 3.2.)

ive just finished the "bridget jones diary" with 2-pass int

-size is very accurate :
aimed at 615.520 kb and it came out with 615.616 kb

-quality is absolutly perfect, i cant see any blocks during the whole film (even with the xvid decoder ! )

-very sharp with lots of details (i know, this is relativ, but it was really very very good ! )



sets as following :
avisynth
2 pass int.
quant. modulated
low 15% high 25% kf boost 20%
i frame limit 2-5
frame limit 1-10


i think there is still room for improvements in my config...

[easyfab]

@ -h
i've just tried your XviD-0203-2002-2 build.

With the first file (fist of legend) it was perfect, but with the second one (unbreakable) i got just 1 I-frame after 600 frames and no one after this.
I've used old stats files (from a XviD.09.02.02 build) and my settings are : no credits, no luma, I frame min 0 max 600 ...
When i've returned to XviD-26022002-1 build with the same settings all perfect.
Are my older stats files obsolete (but with my first XviD.09.02.02 build stats file there is no problems)? Or is there another problem?

[Koepi]

Can you please, just to make sure, run the first pass as well?

Maybe after that the problem doesn't appear... never experienced something like this before.

Btw., XviD-02032002-2 is exactly "my" latest versionnumber :P

Regards,
Koepi

[wing1]

@-h

First I would like to say thank for yours and your crew on the endless works on improving a great codec, and they are improved tremendously every time they come out. The speed of the codec is getting faster and faster as well. However, there are those of us that are using the codec for capture purposes and would like to see deinterlace option implemented. How soon can we hope to see this option turn on in the codec?

[Koepi]

As -h wrote VERY often, he waits for the new core to incorporate it.
So it doesn't depend on him now...
Tmorrow there will be a core change which got announced by suxen_drol, but it concerns custom quantization matrix'es...

regards,
Koepi

[rui]

Originally posted by Koepi
You can still constrain your IFrames. This keyframe boost gave me more results like this:

02/03 28/02
kf 2 3
df 4 4
df 4 3

...
it's once more a matter of taste.
But if you leave away keyframe strictness, I dunno if it would always use e.g. quant 2 for keyframes....

Ah well, I have to check that ;)

Regards,
Koepi

Well, i am just finishing a 1 cd rip from the movie The Replacements, and just want to share two things.

First, i didn't lock I-frames this time, i only used the Keyframe boost feature (i choosed 50%, like -h said). But i noticed in debugview that almost all the keyframes now have a 5 quantizer.
And the other thing: time to time there is a line in debugview saying : SYMRDR: No redirection entry found for this address..
I don't know what this means.

Regarding the quality, considering that it was a 1 cd rip from a movie that always gave me a hard time to encode using divx4, i believe that it's quite nice. Certainly much better than divx4. And i am not using any kind of post processing, because i'm using xvid ds filter. Monday should bring a new light to this rip ;)
And to end this post, i wanted a file size of 614689 and got 614818.
What more can i guy ask? :D

[sierrafoxtrot]

i've noticed that (for the replacements anyway, 65% through 2nd pass ATM) that (at the risk of repetition) KF boost is lowering DRF for KF but causing subsequent drops in the following frames. i know this is the only way bitrate gets evened out, but does having really nice KFs and crap interframes help much? anyway, just a thought.

and this quant table thing ... will it be like DRF>3 use H.263, DRF<3 use MPEG? i mean ... we only have to ways of quantisation right? or do we? ;)

[Nic]

ps
Ive finished the filter, ill put it up tonight or tomorrow

-Nic

[SleepEXE]

ps
Ive finished the filter, ill put it up tonight or tomorrow

Yeah, post script.

Nice attempt at playing it down. ;)

--SleepEXE

[Nic]

Rofl :D

Go get it at www.freewebz.com/xvid

Please report bugs & crashes.

Also, remember that it wont clear all the blocks away, but it does exactly what I want (which is clear up just a little)

(ill update the codec & stormpages.com monday)

Have Fun,
-Nic

[Acaila]

Great job Nic.

And you're right about deringing taking a lot of processor power...totally messes up my playback. :(

Deblocking is great though.

[Actron]

deringing runs fine at 1800+ xp :)

[Nic]

How fast is your computer?
Ill start work on optimising it tomorrow, if you look at it though (even though its too slow to view normally on slow comps) it does a great job!

-Nic

ps
Cool to here its good on the XP! I hope the quality is to your liking! :)

[Actron]

Quality is nice ! But it smoothes the detailed parts of the movie more than the walls etc...Need a bit improvement i think :)

Anyway, this was a great job !

thanks for your work !


actron



p.s.:
i dont use any postprocessor anyway, because i like it to have sharp pictures and my encoded films dont have any blocks anyway...

but one suggestion :

Could you make an option to "draw the black parts in true black" or something to remove the blocks in very dark parts ?

[Acaila]

@Nic
I'm using a P3 575MHz. Not that fast, but no probs with DivX4's post-processing at level 6....


@Actron
I agree on the black part thing. Maybe something like if the color is almost black, turn it completely black instead. That should remove the macroblocks on black areas I believe.

[wing1]

quality is excellent. used options(Y) only and all blocks disappeared however, details deminished a little ( not a big deal ... better then divx4.12 filter ). Only concern is the CPU usage: 1900+XP AMD is running at 90%+! Divx4.12 at setting of 4 uses about 50% of the CPU. Thanks for the filter :):D

[Actron]

Again for the black parts thing :

im not sure, but i think there has allready been some develepment for this during the nandub time....but it was not finished (dont know why not...)

anyway, i would be happy, when it would be implemented in any kind

[Koepi]

That black-thing will come sooner or later and will get called "contrast" I think ;)

Nic, do you mind if I spread your DSF along with my binary?

(btw., and sorry for not asking the first time with the brightness control within the filter, but it was the only addon I was missing for switching totally over to XviD and uninstalling DivX4 :) )

Regards,
Koepi

[Nic]

@Koepi: Of course you can :) Ill put out the source soon. I didn't figure contrast would help the black macroblock situation.

@acton & acailia:Ill work on what you mention (about turning near black to complete black....but thats a dangerous thing to do, it would be best to analyse the picture first to decide...(i.e. only turn black to black if all around it is black)

But ill add contrast tomorrow (I didn't bother as I didn't think it was useful...Doh)

Ill work the optimising, but that really isn't my strong point....also ill compile with VC7 on monday & that should help.

Anyone else got any ideas for improvements?

Cheers,
-Nic

[Koepi]

Thanks Nic :)

Well, such an adaptive algo is really nice :)

But if you play around on an old DivX and increase brightness&contrast, you start to get rid of odd looking areas again...
but surely it can't compete with an intelligent algorithm, it's just a more-or-less fast fix ;)

Regards,
Koepi

[Klumsy]

Maybe you should let the user decide how much near black is black enough to be black, if you know what i mean ;)

deringing slows down my old Athlon700c so i hope it's optimized soon ;)

Good work Nic

[Emp3r0r]

should I even try with any of this if I have a p2 400? Actually, the last xvid filter in an ogg media stream with ogg sound would run fine except every now and then cpu usaged would shoot up from around 40-50% to 100% and I would get chops during playback. Guess it is time to reinstall everything from scratch and see if that fixes my slow computer problems.

[sierrafoxtrot]

@nic

you're a star!! thanks again for post-proc, eveything's beautiful, just a bit cpu-hungry ... :p

@rui

the replacements seems impossible to encode without noise filters, so i'm doing that now. my 2nd encode was at 464x256 bilinear, H263, CC 10/20 luma off, KF Boost 30% and most of it was watchable, not very many macroblocks, but everything's very blurry. i'm thinking that medium noise will blur things even more. anyway, i read in some thread soemwhere that if you resize down using bilinear, and then resize up using bicubic, you might get better results. anyone with experience with this?

in other news, i've done two movies with koepi's 02032002-2 build, filesize prediction is bang on and quality is amazing (Mod Quant), but i think i have a minor bug. VDub still crashes at the end of 2nd pass. i'm using const quant 20 for credits and i've noticed the overflow decreasing more and more (makes sense) but it always crashes, and hence never gets to the next job. does anyone else have similar problems?

cheers.

[-h]

in other news, i've done two movies with koepi's 02032002-2 build, filesize prediction is bang on and quality is amazing (Mod Quant), but i think i have a minor bug. VDub still crashes at the end of 2nd pass. i'm using const quant 20 for credits and i've noticed the overflow decreasing more and more (makes sense) but it always crashes, and hence never gets to the next job. does anyone else have similar problems?

Darn, I thought that would've been fixed in the last build. Ah well, must be something different.

Just out of curiousity, what does the overflow reach when it crashes?

-h

[sierrafoxtrot]

@-h the overflow went as far as -130234. is that normal? i just thought that was the bit-savings from the credits ie, how much is _not_ being used for the credits. does that make any sense?

anyway, that overflow was for credits for In the Mood for Love, a chinese movie with non-scrolling credits, so it may have been low.

i'm currently doing two back-to-back 2nd passes of the replacements (with full-on scrolling credits) to see if the second will start once the first finishes. and i'll keep an eye on the overflow.

BTW, this may sound like a stupid question, but when using the end credits option, i just copy the relevant frame numbers in from GKnot or VDub, so the end frame of credits is identical to the end frame of the movie. would setting the last 5 frames or so to non-credits frames make any difference? just a thought.

thanks for doing a great job guys! (developers can never hear enough of that :p )

cheers.

[rui]

First I want to congrat Nic for his outstanding work. I just watched a bit of my The Replacements 1 cd encoding with his filter, and it was a lifesaver for that movie.

Sierrafoxtrot, I already gave up on doing a 1 cd rip of this movie.
The last one I did, using Nic's filter is probably the best I can do with using only 1 cd. IMHO you are lowering the resolution too much. It's a personal taste, but 464x256, and using bilinear, could get much too smooth.

To Nic: I tried to install your filter by using the regsvr32.exe that came with your latest build, but it didn't work. Only when I copied the xvid.ax to Koepi's folder and then installed the xvid codec all over again, the filter got installed. Koepi's build install automatically the xvid filter.
I don't know what I did wrong when I tried to install it manually, but the fault must be mine. It's just that could be others like me that try to installing it manually. Maybe you could include in the readme the right way of installing it.
Using the filter with only Deblock option activated i get a cpu usage of 25/35%. But when activating Deringing too, i got, in some scenes, up to 95%. I have an Athlon XP 1,55Ghz.

[libredr]

I have compressed Mission to Mars on 1 CD and I have to say that the quality is excellent, a lot better than DivX4. Moreover, with the new post-processing filter, the viewing is very nice with a modulated encoding.

Here are my parameters :
I used GKnot to calculate and save the .avs file and used virtual dub to compress (using Fast compress) with Koepi build.
Size 608*256
1st pass
MSP 6-Ultra High
Quant Modulated
Min/max 10/300
Tuning min/max 1/16, no luminance
2nd pass internal
No luminance filter
Size 640000
Keyframe Boost 50%
Curve compression BP delay 240, HB 25%, LB 15%
Iframe min/max 1/5 (there is no box for the smooth change anymore. I guess it is enabled ?)

The obtained size was 642912ko. Maybe the Keyframe boost of 50% was a little too high to maintain a good predectibility. However, this is acceptable, using overburning with my CD writer.
Image is sharper than with DivX4. Hope to see a future version of GKnot fully supporting Xvid now !
I will do some tests with 2CD encodings now. I am sure the result will be awesome, considering the quality of the 1st pass file (1.2Go)
Thank you to the developers and congratulations, this work is awesome.

[sierrafoxtrot]

@-h

got the damn jobqueue thing to work, but i had to set the last 5 frames of the movie to non-credits. at least it works for me now. :)

the output of debug view for the last few frames ...

[328] 2nd-pass: quant:20 type:.h263 inter stats1:69 scaled:69 actual:69 overflow:-175110 credits
[328] 2nd-pass: quant:20 type:.h263 inter stats1:69 scaled:69 actual:69 overflow:-175134 credits
[328] 2nd-pass: quant:20 type:.h263 inter stats1:69 scaled:69 actual:69 overflow:-175158 credits
[328] 2nd-pass: quant:20 type:.h263 inter stats1:73 scaled:73 actual:73 overflow:-175182 credits
[328] 2nd-pass: quant:20 type:.h263 inter stats1:69 scaled:69 actual:69 overflow:-175206 credits
[328] 2nd-pass: quant:4 type:.h263 inter stats1:1517 scaled:748 actual:1315 overflow:-175797 movie

i'm still not sure what that means, you'll probably make more sense of it than me ... :p

@rui

yeah, i have to agree with you, there's no way the replacements will ever look good on 1CD. with medium noise, the blockiness is replaced by an all-pervading fuzz. depends on what you prefer really, but if you ask me, it's bad either way.

and i completely agree with lowering the res too much, i try never to use bilinear resizing, nor let horizontal resolution go below 544 but i was trying to see if i could get rid of some _horrific_ blocking in my 560*xxx rip. anyway, it was worth a try ...

cheers

[Nic]

@Rui:
I thought that I should have included a "how-to" install after I posted it....Sometimes regsvr32 does that for me too, with all different types of filters. Well, it will be installed by most people with Koepi's updates, so that should solve it :)

@Acaila + all:
The Deblocking should be ok because thats almost all MMX code, the SSE MMX code for the deringer is broken, so its a pure C version...im working on making it better.

Cheers,
-Nic

[Koepi]

@Nic:
hmmm, this leads me to the question, is the colour modifier code MMXified as well already?
Maybe my poor compi is simply too slow, but I thought it might be unoptimized code that sometimes causes stuttering on playback (nothing dramatically, but... the users don't like it ;) )

[Usual sidenote: I still think that there must be a way to control the overlay directly, dunno how much CPU time that could save if you do the colour shifting via VGA hardware compared to processing every frame... this is no offense nic, i love your work. But i _hate_ this directX stuff that doesn't want to be simple to code ;) ]

Keep up the good work!

Regards,
Koepi

EDIT: this always sounds so demanding. Believe me, it isn't meant that way. Before you hate me Nic, I'll try to explain why I keep on this idea: it's simply because I _know_ it can be done by the hardware. I even have weirder ideas like using Direct3D surface for showing the video and using the hardware's bilinear/trilinear filters to smoother the video, but I don't even dare suggesting something like that :)

[Nic]

No offense at all, it would be far nicer to do it the DirectX way...I wish I knew how, but I can't find any examples & there is no reference to anything like that in the DX8 SDK help. I think somehow you have to get a pointer to the video renderer then queryinterface to get the correct class (like the overlay one) and change that....but how you do that is beyond me.

& now im going to say thank you (because you mentioned on xvid.org that the brightness code was slow)! I looked through my code for optimisation and the brightness control was very very badly handled. So thats now a hell of a lot faster, also ive optimised bits of the deringing function (now not many multiplications/divides only bit shifts).

Its quite a bit faster now, still with all the processing & brightness turned up, I get the odd skip on my Duron 800. But before it was unwatchable using those settings, so im getting there slowly.

Ill put up the slightly faster version tomorrow.

Cheers,
-Nic

[Koepi]

PLEEEEEEEEEEEEASE put it up now :)

I'm going to watch a movie in ~45 minutes and would LOVE to have your dll :)

Or mail it to me to the mailadress in my profile please ;)))

Thanks a million for your work, it's really impressive how you manage to workaround the directX oddness via internal functions :)

Best regards,
Koepi