XviD-19062003-1

[Koepi]

Aloha,

a whole month without a new build, amazing :) XviD's getting somewhat stable I think :) (I'd nearly say sorry for the delay, but I wanted to test that build a little before releasing it. It works absolutely fine for me.)

XviD-19062003-1:
- CVS update.
- sysKin's new i/p/b-decision. New thresholds: -40 disables bframes, +90 produces only bframes ;)
- sysKin's new VHQ fixes.

There are some features in which I already forgot etc., but you can read most of them in the CVS changelog at umaniac's :)

Please do some testing like you did with the last build, the PSNR-tables were pretty impressive.

Find the build clicking on the "www"-botton below this post or use the link in my signature.

Best regards
Koepi

[The Edge]

Cheers Koepi.
I can almost smell stable XviD 1.0 :)
Will test tonight.


Bren

[JagPanzer]

Aloha! :D

Thank you, thank you, thank you... :cool:

It's time to do some tests now...

[Koepi]

A first observation:

bframe-settings: 2/150/75/0 seem to give impessing results (currently compressing 1GB 1st pass to 650mb 2nd pass at average quant <2.5 :) ).

Just as a hint.

regards
Koepi

[kaitsuburi]

Thank you Koepi!

Just finished a quick 2-pass encode of "The Flight of the Osiris" -- worked like a charm. The quality is awesome!

-kaitsuburi

[Garfield]

Thanks Koepi, another update of our favorite toy


I just have a few dumb questions :

-40 disables bframes, +90 produces only bframes

Does that mean that 90 is the highest treshold value ?

sysKin's new VHQ fixes.

So now it is safe, stable and good in any mode for sure ? (:))

[Koepi]

You can add higher values as well, but you won't notice much difference. That comment is there to make the new range clear.

It was safe before to use VHQ, it didn't break anything (badly). It should be safer now, only tests will show (I use vhq 4 [or 0 when i'm in a hurry] without problems).

Regards
Koepi

[kilg0r3]

1.)
anything new about trellis quant (don't quite remeber what the problem was)? The changelogs at umaniac's do not contain all insta builts, seems to me, at least.

Sorry, it's very hot in here ... praying for a nice thunderstorm ...

2.)
Something from the mpeg2-quality thread
originally posted by trbarry And I don't know if it is my imagination or not, but it does seem that it's not really possible with Xvid anymore to get really detailed high bit rate HDTV encodings. But I've wondered this before and found it was just me, screwing up my encoding parameters, so maybe it is again. But something does seem different recently.

[sh0dan]

Looking good!

Minor stuff: The text displayed, when hovering over VHQ mode should probably be updated (as quality doesn't decrease with VHQ > 1). Or am I mistaking?

[Koepi]

Oops, update needed there indeed :)

Thanks for pointing that out :)

Trellis quant is still the same code as in my last build, nothing changed there.

regards
Koepi

[Selur]

1st thx for the build
2nd since latest changelog just came:


23.6.2003 17:40:
U xvidcore/src/motion/motion_est.c (rev.1.71) syskin:
- ugly bugs fixed, R-D works better now
U xvidcore/src/motion/motion_est.h (rev.1.9) syskin:
- ugly bugs fixed, R-D works better now


I wonder if you could update to XviD-19062003-2, so we might even test a bit more with R-D ;)


Cu Selur

[Koepi]

Those bugfixes are the same as in my build, the only thing that _may_ have changed is thresholds for bframes-decision... which is just a bit more of testing IMO as sysKin tends to optimize with single sources and not different ones ;) Just kidding, I still think it's worth to test the difference on _many more_ sources.

regards
Koepi

[Selur]

Oh,.. if those fixes are included I'm happy :D
Thx for the info :)

Cu Selur

[Teegedeck]

Hello Koepi,

thanks for the new build (and many thanks to sysKin for his work, too!)!

First impressions of an encoding with your build, compared against an identical encoding performed with uManiac's CVS-checkout from 11th, are very positive. Considerably less blocks in a 'too-low-bitrate' encoding. May not be too low, soon. :D

Anyway, one thing I wanted to ask: Do you still have that tweaked lambda-value in your build? I'd thought sysKin said something of the sort 'VHQ isn't there for saving space anymore'?

[Selur]

and a little questions about "+90 produces only bframes":

Does this mean "maximum Bframes" will not be respected and there'll be no more Pframes ?

What if "maximum bframes" is set to zero ?
- Will the bframe threshold still produce 'only bframes'?

How is the bframe threshold scaled?
(I mean earlier it was scaled in percent (as far as I know), but how is it scaled now? )

Cu Selur

[plazz2000]

What is happening with GMC? Will it be fixed for 1.0?

[Koepi]

Selur:

take a test sequence of 20 frames and find out for yourself (and report back here of course). I don't think the new ipb-decision will mess up that badly. it will be more like closed GOP.

plazz2k:
This thread is about this actual build. So please stay on-topic. I think skal and gruel have some nice GMC code at hands in dev-api-4(=will become 1.0), but it's not backported. Don't expect much efficiency out of GMC though.

Teegedeck:
that#s a good question - all the file-circling and updating make it impossible to track every change. It's very likely that I have other values in the sources now (sysKin's tweaks :) ). I'll go and check that - later, tomorrow.

Regards
Koepi

[Teegedeck]

Whatever lambda, the results are great! Closer examination just reinforces my first impression: A good quality boost, again! This may be the breakthrough for B-frames that should prompt even anime-encoders to switch them on, finally. ;)

[Selur]

did a smal 30frame sequnce.
(monotone scene someone speaking, no backround action)

Does this mean "maximum Bframes" will not be respected and there'll be no more Pframes ?
No, "maximum bframes" (in a row) will still be respected, so a 30frame sequence could turn up with 1 Ifram, 15 Pframes and 14bframes (if max bframe is set to 1) ;)

But what confuses me is that if I set "maximum bframes" to 10 and threshold to +90, I still ed up with 1 Iframe, 12 pframes and 17 bframes,.. (thought "DX50 copatibility" caused this, but disabling it, didn't change the distribution)

What if "maximum bframes" is set to zero ?
- Will the bframe threshold still produce 'only bframes'?
If "maximum bframes" is set to zero, no bframes will be produced. ;)

How is the bframe threshold scaled?
No clue,.. especially since I can't say: "+90 produces only bframes" :(

Cu Selur

Ps.: got the distribution infos from statsview,..
(hope that's okay)

[sysKin]

Hi,
Since I'm here, instead of learning for tommorow:

About 'max bframes' - the statement "will produce only bframes" means that the decision itself will always say 'B' or 'I'. After the 'max' has been reached, B turns to P anyway. 'Max' is always respected.
But the statement is not 100% true anyway, because every next b-frame has its threashold decreased by 20. If you always want 10 bframes in a row, the threshold will have to be 90+9*20 = 270 ;)))

In any case, threshold above 40..60 will just remove any smartness of the decision. If you want to experiment, go for +/- 10 first.

About lambdas and VHQ: VHQ still decreases filesize. It wasn't the case in the last build becuase of the bugs. Lambda is still 1.00 , I don't think changing it will have any big effect (on constant-bitrate/constant-filesize. It will have effect on constant quant, but that's not what we do*) :)

I have a favour to ask - could you please check the new keyframe-decision? Does it miss any scenechangs? If yes, are the scenes dark or bright?
Does it put any unexpected keyframes, which hasn't been there in last build**?

Thanks,
Radek

* well, if you do 'compressability test', you care about constant quantizer result. Another prove that compressability value just stopped showing anything after divx3.11.

** like at the beginning, or in the middle, of any fade

[Selur]

@syskin: thx for clearing this up

about the keyframes, as far as I can tell form some small clips Iframe distribution is better than earlier and one doesn't have to 'activate' bframes for not ending up with only getting Iframes at the maximum Iframe intervall,..

I'll do further test later, have to hurry to university,.. ;)

CU Selur

[BoNz1]

Okay, I have got a test ready to go for VHQ 0-4 and I have calculated PSNR for 1-4 and it increases. However, I have been unable to calculate PSNR for VHQ 0 so I don't think I can post my little test. psnr4avi just gives back a whole bunch of garbarge letters for the last few frames and gives me a average and total PSNR values that are nonsensical letters. Sorry, I can't post it, hopefully I can figure out what is wrong. :(

[Didée]

Koepi,

what IDCT is this build actually using? The changelog on uManiac's site reports that simple was switched over to Walken again.
And WHY? After all the noise in the past about simple idct being "the one", I'm a little confused now.

- Didée

[Morbo]

Nice build Koepi!!

Could anyone else confirm that its slightly slower...

Im not complaining,but Id like to know if its just me.

Thanks again K!!

[Koepi]

This build is using SimpleIDCT.
Isibaar switched back to walken idct as he feels in theory it shouldn't be wrong. Well, please test umaniacs build and see if divx/ffdshow decoders produce qpel smearing again.

The speed isn't much of a difference here, can't say much about it.

Regards
Koepi

[JohnMK]

Thanks Koepi, I appreciate this new build very much! I have a question. When I run a compressibility test, I get a result of 86% for VHQ-0, and 71% for VHQ-4. This indicates that within a fixed filespace, VHQ-0 will have lower quantizers than VHQ-4! I thought I should be expecting the reverse.

[Malevolent]

@Morbo:

Hmm, actually it seems to be a bit faster.
I encoded earlier today one futurama episode i had been meaning to encode for weeks now, and it took ~2h15min. /pass.
After that i checked the forum, and noticed a new build (now why didn't i see it earlier? Doh!), and decided to encode again with the same script, just to compare.
I had a 300 frame testclip to check filters etc. , so i used it to rough out the b-frame setup (previously i used 3,150,100,200 - now i used 3,150,100,40), and the overall result was quite pleasing.
Time consumed was ~2h5min. /pass. & the result looked a bit better in my eyes.
:devil:

[Lobuz]

@sysKin
@Koepi

Could that b-frames quantizer decision be more fluent.
For example if there are I,P frames with q2 and there is a frame which couldn't be a B-frame with q4 cause of threshold. But there is a room for lower quantizer q3 with that threshold and it still could be smaller then p-frame.

Could it be done? Or am I wrong?

And I'm curious what is the status of VHQ for B-frames(does it work)?

Regards
Lobuz

[Gaia]

compressibility test

Little off the topic but i think compressibility tests are useless and don't tell much about final quality. Anyway that's just my opinion.

[Wuntvor]

@Morbo & Malevolent

I think I remember that the more bframes, the faster, because VHQ doesnt do any work on Bframes yet.

Or maybe this has changed?

regards
/Wuntvor

[OUTPinged_]

Just tested "Evil overflow" thing is still there. Expect bad results from 2pass encodes if you dont cap them.

[Koepi]

Outpinged,

while you're on the mission to spread your observation (which is ok) you should remember one thing: if you keep the sound of your posts like this, you're just insulting.

Just set your overflow redistribution from 60 to 30, and you're done. Or turn of bframes and you get what you want. But stop putting things in such unkind words. (Ungratefulness is a habbit of yours it seems. Always just complaining without helping with a single line of c++. What can I say? That's the spirit! That's how you have to take a gift!)

Koepi

[Teegedeck]

Reminds you of the Linux Router Project, doesn't it? ;)

OT: That gave me the idea we should have something like a 'XviD Hall of Fame' here with the names of everyone who contributed to the project and a special mention to active developers in it and a big THANK YOU! from all of us on the forum.

[bilu]

Originally posted by Gaia
Little off the topic but i think compressibility tests are useless and don't tell much about final quality. Anyway that's just my opinion.

Compressibility tests have shown in this case that a first pass done with VHQ-4 is harder to compress than VHQ-0 because it gets a bigger filesize.

VHQ used to help compressibility. :confused:


Bilu

[Teegedeck]

Hi Bilu,

haven't you read the whole thread? :)

sysKin:
* well, if you do 'compressability test', you care about constant quantizer result. Another prove that compressability value just stopped showing anything after divx3.11.

I mean, if sysKin says that, I believe him! How small a movie can be compressed at constant quant=2 isn't related closely to how well the movie can be compressed in 2-pass -- that's the essence of my experience with XviD, too. [Edited further, but that's the last OT thing that I write in this thread:] XviD isn't tuned to deliver optimum results at quant=2.

[bilu]

You're right, I didn't :o

But now that I did, He also says:

About lambdas and VHQ: VHQ still decreases filesize. It wasn't the case in the last build becuase of the bugs.

Maybe JohnMK was using this buggy build? :confused:

About compressibility: well, it's an approach just like PSNR.
Of course it's not necessarily precise but it gives a safe margin to work with :)


Bilu

[Teegedeck]

Please see my edited post above.

[bilu]

@Teegedeck

Safety margin methods are important, that's why people use PSNR and comp.tests.
But this is OT and it sure deserves another thread discussing reliable methods.

Could you start that thread? I'd like to see an intro from you :D

Best regards,
Bilu

[sysKin]

Originally posted by JohnMK
I have a question. When I run a compressibility test, I get a result of 86% for VHQ-0, and 71% for VHQ-4. This indicates that within a fixed filespace, VHQ-0 will have lower quantizers than VHQ-4! I thought I should be expecting the reverse. Ok this _is_ a problem. And I think I might have a theory now.
Can you tell us if you have P4?
Also, can you make a simple, short test, and encode something at fixed quantizer, with VHQ 0 and 1, and tell us if 1 had smaller filesize?

One of the problems with the previous VHQ was that one asm-ed function was not working as I expected. It was intra-block dequantization (both mpeg-type and h263-type) if I remember. It is ok for normal coding and decoding, but I had decided to be 'smart' and save us entire 128 bytes of memory, and the function stopped working. I changed it to plain C and it was ok with me.
However, perhaps there is more functions which can't work properly that way, and I just don't use them? Like P4 functions.

Anyway, if VHQ 1 results in bigger filesize, do a final test with disabled CPU optimizations. If it's ok, I'll change the whole thing ASAP.

Thanks,
Radek

PS. about iDCT - Koepi, you might have missed it, but I think there is a 'special' code in decoder which forces Walken iDCT if stream ID is below 10 (or something). This means that _de_coder from your build will use Walken, which is different than encoder uses.

This makes PSNR results deadly (I fell for this trap, it's 3-4 dB loss)

PS v2.0 : are you sure you can do PSNR calculations with psnr4avi, if your avi has bframes? The problem is the delay. If you can, please make sure that PSNR of the first frame is reasonable.

if you want, this is my PSNR code:
name = "00.avi"

a = avisource("monsters.avi").assumefps(100).converttoyuy2()
b = avisource(name).assumefps(100).converttoyuy2().trim(1,0)

compare(b, a, "", name+".txt", false)
It works like charm. Remove the "trim" if you have no bframes. Credit for "trim" thingy goes to mf.

[Teegedeck]

Originally posted by bilu
@Teegedeck

Safety margin methods are important, that's why people use PSNR and comp.tests.
But this is OT and it sure deserves another thread discussing reliable methods.

Could you start that thread? I'd like to see an intro from you :D

Best regards,
Bilu
Oooooh, if quant=2 should still be smaller, I was plain wrong! :p :o

[ssjkakaroto]

syskin you said that a threshold above 40..60 will just remove any smartness of the decision, so what would you recommend to extremely low bitrate scenarios (the old 3/150/100/255)?

tia

[Koepi]

Dilemma, dilemma:

I could set the bitstream version a number back.
I could switch to Walken IDCT.
I could remove the check from the decoder.

I think I'll go for the first option ;)

Regards
Koepi

[sysKin]

Originally posted by ssjkakaroto
syskin you said that a threshold above 40..60 will just remove any smartness of the decision, so what would you recommend to extremely low bitrate scenarios (the old 3/150/100/255)? In my humble opinion 255 was very wrong, just like 80 (100?) would be wrong here.

But as for recommendation, I don't have it: this new decision is experimantal, I haven't even checked thresholds different than 0 ;)

Try 10, 20, 30 and share your impressions with us.


/me should _really_ be learning now ;)

[Isibaar]

Originally posted by Koepi
Dilemma, dilemma:

I could set the bitstream version a number back.
I could switch to Walken IDCT.
I could remove the check from the decoder.

I think I'll go for the first option ;)

Regards
Koepi

Koepi, never ever switch the bitstream version back! Use higher numbers and tell us which version numbers we should reserve for you but never switch back version numbers. This destroys any chance of automatically activating workarounds for both XviD and libavcodec decoders.

Regarding idct: I posted several mails about this issue to the team mailing list which you have hopefully read. I concluded that simple idct is no good choice for encoding: Walken idct did _not_ cause qpel noise or the infamous smearing problems. These problems appeared when playing back XviD content with ffdshow and were caused by a bug in libavcodec's qpel code. I pointed this out already back in january on the XviD developer's mailing list:

http://list.xvid.org/pipermail/xvid-devel/2003-January/001762.html

This bug has been fixed in the mean-time and the most recent versions of ffdshow should decode XviD qpel flawlessly.

Different idct implementations however are indeed a problem, for qpel but for normal halfpel as well. If possible, the same idct should be used for decoding than was used for encoding. This means that Walken idct material should be played back with Walken, simple idct material should be decoded with simple idct.

The problem with using simple idct for encoding is the fact that this idct implementation isn't used in any other popular decoder besides ffdshow. This means that XviD content encoded with simple idct is _not_ correctly played back by DivX, 3ivX and EnvivioTV. However XviD Walken idct material is decoded correctly in all popular decoders besides ffdshow (because it uses simple idct by default). However ffdshow leaves the option to switch to XviD idct (Walken) which will fix potential problems. Also Walken idct could be automatically activated in ffdshow in the future depending on the XviD bitstream version.

That's why I think that using Walken idct for encoding is the best solution because it causes the least interoperability troubles and existing incompatibilities (with ffdshow) can be quite easily fixed.

[kilg0r3]

1.)
Originally posted by sysKin
name = "00.avi"

a = avisource("monsters.avi").assumefps(100).converttoyuy2()
b = avisource(name).assumefps(100).converttoyuy2().trim(1,0)

compare(b, a, "", name+".txt", false)

Would I have to adjust the trim value according to the number of b-frames?

2.)
some VHQ file sizes with constant quant 2 on an Athlon XP18000 (250frames)

vhq 0 2490KB (w/o cpu opt.-s 2494KB)
vhq 1 2420KB
vhq 2 2366KB
vhq 3 2348KB
vhq 4 2328KB (w/o cpu opt.-s 2332KB)

[Koepi]

Thanks for clearing this up isibaar,

i'll simply remove simple_idct again then with the next build.

EDIT:

LOL! Just checked my sources, and guess what: as long as you have a MMX capable processor and didn't switch off the MMX-features in the CPU-tab, Walken IDCT gets used. (note to myself: don't compile builds in a hurry.)

I'll clean that up in my next build anyways, so switching off MMX in the next build won't use MMXed simple_idct (did I eat something bad that day? That's so incredibly wrong but has no effect in "usual usage" *phew* ).

EDIT2:
Further comparing my sources with CVS showed up that on P4, fdct_sse2 gets used in my build. This maybe could explain some of the file size mismatch as well.

Regards
koepi

[trbarry]

Further comparing my sources with CVS showed up that on P4, fdct_sse2 gets used in my build. This maybe could explain some of the file size mismatch as well.

Koepi -

You mean it gets used on P4's when MMX features are switched off, right? Or is it used all the time on P4's?

- Tom

[Koepi]

Heya Tom,

no, they just got used when SSE2 optimizations were selected on the debug tab (like with autodetection).

@All:

XviD-24062003-1.exe (389kb)
Changelog:
- Fixed P4 issues and the use of SimpleIDCT when unchecking all CPU optimizations. Walken gets always used again.
Remember to switch IDCT to XviD when decoding with ffdshow.

Well, some issues are solved easily ;)

Regards
Koepi

[Didée]

Originally posted by kilg0r3
some VHQ file sizes with constant quant 2 on an Athlon XP18000

I want that machine, too!! :D

How many milliseconds did one pass take?

[vass-iliskus]

Originally posted by sysKin
I have a favour to ask - could you please check the new keyframe-decision? Does it miss any scenechangs? If yes, are the scenes dark or bright?
Does it put any unexpected keyframes, which hasn't been there in last build**?


Well, I did some checks, and here is what we have (1-pass q2):

vhq0, boff(-1): 7 keyrames (misses 2 dark scenechanges)
vhq1-4, boff(-1): 2 keyframes (misses all scenechanges)
vhq0-4, bon(0,1): 7 keyframes (all scenechanges have kf)


[1] has two groups of 2 succcessive keyframes, while [3] has only one.
The source is BW and rather dark, although case [2] happens in more lightened scenes too.

UPDATE: Checked the previous build too (koepi's 14052003).
Same results: when VHQ is in use and b-frames are OFF it misses all the scenechanges, except for the case [3]: when b-frames are ON (0 or greater), the result looks similar to [1] - both dark scenechanges are missed, one additional scenechange is missed, and one of the succcessive keyframes is missing too (just like [3] in the new build), totalling to 5 keyframes.

All in all: new build with b-frames ON has better scene detection.