XviD-03052003-1

[Koepi]

Heya folks,

since you all get worried about new builds...

XviD-03052003-1:
- Fresh CVS checkout.
- sysKin tweaked bframe decision and VHQ a bit.
- Disabled "hinted ME" as it's broken for a long time now.
- Completely untested build.

There are now even some tweaks for iframe decision thresholds added, which sysKin already applied to CVS after that as well.

Regards
Koepi

[Mel Maconoo]

thx for the new build Koepi i'll get right on testing it.. ^_^

[JagPanzer]

Koepi thanx! I'm going to do some tests with Minority Report now (very hard movie to compress).

bye

[Koepi]

If it's hard to compress and you're heading for 1CD you should use 2 bframes and a threshold of 255 ;) SysKin doesn't like that high threshold, but for 1CD rips it really helps a lot. I didn't get my hands on that particular DVD yet so I don't know how much detail and noise is in there, but I'd recommand

Avisynth 2.5x, YV12
- LumaFilter()
- Convolution3d(preset="movieHQ") # only if there's little noise,
# sometimes preset="movieLQ" is necessary - movieHQ doesn't harm
# quality at least
- UnFilter(-5,-5) # for very little oversharped movies, else more
- UnDot()
- LanczosResize()(+bilinear resize for credits range)

If you still don't hit desired low bitrate you may try Bicubic Resize(x,y,0,0.5) instead of Lanczos.

VDubMod for encoding in YV12

If the movie has plenty details, try qpel. ChromaME is mandatory of course, it's just slowing down the encoding a little but usually helps where you else would've gotten blocks since it takes colour information into account as well.

...so nobody can complain I'd hide my favorite settings.

Hope this helps,

Regards
Koepi

[filewalker]

:)
Thanks Koepi for your new build and your really nice recommendation about your favorite settings...it's exactly that what Xvid newbies(like me) like to see...

thanks:)

Cu filewalker

[JagPanzer]

Tnx Koepi, that really helps a lot. I made a compressibility test (only cropping and Bicubic Resize(640,272,0,0.5), no b-frames, no qpel, only VHQ=1 and ChromaME=ON) and it came out 3223 MB (!!). But then I lowered resolution (576*240), and I used you settings together with hvs-good-picture-matrix, and it came out 1224 MB. Waw, this is really great. Now I am able to do 2CD rip with AC3. :)

Once again big thanks to all great XviD developers for their efforts in making the best MPEG-4 codec in the universe. :D

[HarryM]

@Koepi: Thanks for new builds!

[frodoontop]

Thanks for the new build Koepi!

@Jagpantzer:
Minority report is hard to compress, but not impossible. I encoded it a few weeks ago for two XCD's, with only unfilter(-5,-5). I resized with Lanczos to 512x288. At quant 2, the resulting file was 1.457.190.912 bytes. (By the way: I never encode credits, at least not the video.) So there was plenty of MB's left for a nice .ogg soundtrack.

At that time I used Umaniacs 5-4-2003, 16:20, which I still find to have the best compresibility yet. But I haven't test Koepi's build yet:).

Xvid settings were:
Qpel, GMC, B-frames 3-100-200, Chroma Motion, discard first pass unchecked and the rest were default values.

[Koepi]

Frodo:

please don't use GMC yet, it's yet useless except in some special cases (where full movies don't belong). Better use VHQ=4 (safe with my latest build) and the bframe settings I suggested. The Undot() filter does some magic on compressability, too.

EDIT: @all...
Forgot to mention that I usually use VHQ now as well. ;) And don't forget to hit "load defaults" before starting your encoding sessions. *hint*

Koepi

[kastro68]

Thanks for the tips.

I would have been still using vhq=1 if i did not read your post.

I just have a quick question, do you recommend bframes=2 for all encodes? Because I normally use more bf when the source is longer, say 3hrs or more onto 1cd.

Well, the question is rather trivial, and I dont think it would make that much of a difference. Xvid gives good results even with bad settings.

[Koepi]

I tested quite a lot in the last weeks (well, you know, that c't comparison) and found that 3 bframes tend to make the overall image impression a little "dirty" (not as in: noise that suggests details where there are none, but like in: dirty).

This still depends on your source, maybe it's time we start to find out how the relation between "X bframes" and "subjective detail loss/dirt raise" impression is so we could find a more abstract formula for it.

Feel free to help us out! :)

Regards
Koepi

[corigan]

Thanks koepi for the new builds and all your hard-work. I find that all the devlopmental "unstable" builds are quite stable.. ecspecially when compared to other mpeg-4 codecs. I have made hundreds of xvid encodes and love the codec. I'm an advocate in some places on the net, and hope all the great developers like yourself keep up the hard-work for all of us that appreciate it.

Corigan

[Gazza]

Koepi

Does your build also include syskin's latest update to include intra frames at each scene change (as seen on umaniacs site)?

[Koepi]

Originally posted by Gazza
Koepi

Does your build also include syskin's latest update to include intra frames at each scene change (as seen on umaniacs site)?

Now I'm a little insulted: how is it possible that you don't read the initial post? Read it again.

Originally posted by Koepi
There are now even some tweaks for iframe decision thresholds added, which sysKin already applied to CVS after that as well.

...3 minutes which I could've better spent than with repeating my initial words.

[bodega]

i tested in virtualdub 1.5.2 and virtualdubmod 1.5.1.1a and first pass created large .avi file ... 4gb+ (and its dont play) ... bug ? :confused:

[Koepi]

FAT32 file size limit is at 4GB. No xvid bug.

[bodega]

Originally posted by Koepi
FAT32 file size limit is at 4GB. No xvid bug.
2gb dude .. and i using ntfs ... this bug is on virtualdub ?

[Koepi]

Take this: FAT16 has a 2GB file size limit, FAT32 has 4GB as its limit. (do some research and you'll see I'm right.)

AVI1.0 has a 2GB limit as well. Carefully written programs could extend that to 4GB - but those avis aren't standard compliant anymore then.

AVI2.0 allows internal 64bit counters, thus 18.000.000.000 GB would be possible.

Maybe you choose an avi1.0 file in vdub (dunno where exactly that option is to find, try for yourself, ). Try to produce an avi bigger than 4GB with another codec and see if that happens as well (best would be: another mpeg4 codec to have a direct comparison within the same standard). I for one have no problems with >4GB avis.
I'll look into the code again but i'm quite positive that's nothing wrong with xvid.

Please take the time to read the "how to report bugs"-sticky and see why my answer was that bad :P

Koepi

[bodega]

http://www.inter.brturbo.com/1pass.jpg
11mb ... i tested now with XviD-05042003-1 and ... vdub encodes work .avi ... strange or i crazy ? :)

[nexus]

@bodega: My first 1stPass with this build seems really good. It has the size it was predicted to have. I also used VirtualDubMod v1.5.1.1a.
@devs: Great work!
--
nexus

[Koepi]

Oh my god. Ok, we have a misunderstanding here.

I thought this was a bug report about an xvid-file which grew > 4GB during first pass and was unplayable then. Since I didn't see anything strange in xvid/vfw sources I even wrote a mail to the developer list to check if the standard doesn't allow for mpeg4-bitstreams > 4GB, or if we somewhere use 32bit pointers/counters where they don't belong.

You maybe didn't properly setup VDub and accidently hit "cancel" or hit another codec in the hurry and ended up with some weird uncompressed avi or something.
So this is definatly no xvid bug.

Regards
Koepi

[Gazza]

koepi

Hum, ok I owe you an apology.

Gazza

[Selur]

Seems like 1pass quantizer doesn't differ between quantizer 1 and quantizer 2. Or one feature has this effect.

At least if I encode in 1pass quantizer mode (+luma+qpel+vhq(4)+ch_mo+ch_opt), the filesize doesn't change no matter if I select quantizer 1 or 2.
(I did this 3 times to be sure I didn't mess up the settings some how.)

Cu Selur

Ps.: I'll do some more tests to check if this happens in general or just if one enables a specific feature,..

Never the less could someone else plz also check this, to be sure it's not just me or my system.

[Koepi]

We more or less hardcoded min quant 2 ;) It's quite possible that even in 1pass quant mode it's upping it to 2.

(Since it's consensus amongst the devs that quant1 is of no use and led to a few bug reports which were easy to solve by limiting quantizer range to 2-31...).

Regards
Koepi

[Selur]

"We more or less hardcoded min quant 2"
does this also imply that if I change the quantizer range (in 2pass encoding) to "1 to 31", quantizer 1 wouldn't be considered and it's the same as if I would have choosen the default setting ("2 to 31")?

Cu Selur

Ps.: seems like it only happens with a specific feature or feature combination. (I'll keep on testing)
(1pass+vhq or 1pass+chroma optimizer work fine)

....

found it:
If one enables luma masking 1pass quantizer doesn't differ between quant 1&2 :)

[Koepi]

Don't use luma masking. ATM it doesn't give any positive results.

And your conclusion about the quant range is correct, which i wanted to imply with that "hardcoded" sentence.

Koepi

[sungey]

just tested this new build 3-5-2003

all default settings .... different VHQ mode
Chroma ME = ON
Max bframe = 1

First pass sizes
VHQ 0 = 221 mb
VHQ 1 = 212 mb
VHQ 2 = 211 mb
VHQ 3 = 210 mb
VHQ 4 = 209 mb

Another test

VHQ 1 (1 bframe) = 212 mb
VHQ 1 (2 bframe) = 208 mb
VHQ 1 (3 bframe) = 218 mb :(

Seems like 2 bframe is optimal ? :)


Cheers ^^

[crusty]

@Koepi:
Thanks Koepi, both for the build and your settings. I'll ask you sometime about the B-frame ratios maybe so I could understand them (and then write another guide about them :D )

@Sungey:
How is the visual quality with different b-frames and vhq settings ?
What's the best visually speaking?

:D

[Kamui-Dash]

Thanks for the new build, just ran sum test and it works excellent ^^
Keep up the good work :D

[Morbo]

Funny that Doom's comparo uses Futurama....

Is it happenstance that XVID shines on it:D

Cheers@Koepi

[Prettz]

Originally posted by Koepi
Frodo:

please don't use GMC yet, it's yet useless except in some special cases (where full movies don't belong). Better use VHQ=4 (safe with my latest build) and the bframe settings I suggested. The Undot() filter does some magic on compressability, too.

EDIT: @all...
Forgot to mention that I usually use VHQ now as well. ;) And don't forget to hit "load defaults" before starting your encoding sessions. *hint*

Koepi
Wait a sec, what's the deal now with GMC? Was it modified?
And VHQ > 1 now *should* no longer decrease quality?

[Koepi]

SysKin tweaked VHQ quite a bit, it's no longer "decreasing quality" (in 2pass modes it never was.)

2bframes aren't always optimal, some sources arebetter with 3 bframes or 1 bframe.

GMC in general is having no positive effect on "every day content" yet as it's still unoptimized 2 warp point gmc. This will hopefully change with gruel's/skal's 3 warp point GMC, but that has to be tweaked and optimized like hell before it's "usable" as well.

Crusty: search for "bframe quant ratio" and you'll get the whole nice formula.

Regards
Koepi

[begu]

Sorry to bother with slight off-topic question, but here it goes:

Hinted ME is no more present in Your (Koepi) recent build.

I tried "hinted ME" in nic's build 30_03_2003 and it didn't screw up the encoding or the result. (I tested it the first time) But now I'm not sure if it worked correctly, maybe there was something I didn't notice, when watching the result.

So, how it is broken? If I could get proper result, (playable with software and hardware decoder) maybe it affects somehow to quality and I didn't look careful enough. The hinted ME speeded up encoding a little. So could someone tell how the "broken" hinted ME affects to encoding / video quality?

Thanks for info, just trying to find out that should I use the hinted ME in nic's build or not.

[tiki4]

I think the Hinted ME problem is of the following form:

If you use B-frames the decision whether a frame becomes B- or P-coded maybe (or is) different for first and second pass. I'm not sure if this also affects encoding without B-frames, but as the second pass uses different quantizers from the first pass the motion vectors may also be slightly different. (By the way: the same feature is also disabled in the last DivX 5.0.4 and 5.0.5).

This may not be the whole truth but hopefully part of it.

tiki4

[begu]

Ok, I'm getting it. I didn't use b-frames in my test, so maybe that explains it. But maybe there is really issue with slightly different motion vectors etc. After all the speed increase wasn't so big. It speeded up my test clip encoding time from 1:30 to 1:24, about 7 %.

Thanks for info, I'll leave the hinted MV option unchecked from now on.

[mpeg4_tech]

Originally posted by Koepi
SysKin tweaked VHQ quite a bit, it's no longer "decreasing quality" (in 2pass modes it never was.)
Koepi

Koepi, Hello nice to meet you, I'm fairly new with the Xvid codec, been using divX for over a year. I will be following the postings on this forum and visiting other websites with knowledge on Xvid so I can learn more. However, can you please take a minute to look over my settings and give me a simple thumbs up or down?

FYI, ALL my sources come from DVD and I can usually force film since the source in 95% or higher Film.
I will be using your latest xvid release.
I also usually aim for a 2 CD encode, bitrate usually average 1200kbps



Motion search Precision: 5-Very High
Quantization type: H.263
FourCC used: XVID
VHQ mode: 4-Wide Search?????
Max I-Frame interval 300
Min I-Frame interval 1
Use chroma motion (check)

Max B-frames: 2
B-frame quantizer ratio %: 150
B-frame quantizer offset: 100
B-frame threshold: 0

DX50 B-VOP compatibility (check)


Thumbs Up or down?

[Koepi]

Where the heck do you guys get that Search Precision=5 from? 6 isn't anyhow noticable slower than 5, but shoulkd have a small gain in achieved quality.

Else I think a "thumbs up" is ok since those settings don't show up anything anormal on first, slightly drunk sight ,9

I'll look over them tomorrow again.

Best regards
Koepi

[mpeg4_tech]

Originally posted by Koepi

I'll look over them tomorrow again.
Best regards
Koepi

Thanks Koepi, any help or advise would be very much appreciated. This entire forum is filled with a wealth of knowledge and for that I am grateful.

As for the Search Precision=5, I was kinda going by what doom's guide suggested for a high bitrate (2 cd) encode. I'll kick it up to 6 if you say it'll help. Thanks again
;)

[Assault]

@ mpeg4_tech

With a bitrate of 1200 kbit I would use mpeg as quantization matrix instead of h.263. It retains more details.

Regards
Assault

[Sigmatador]

can i suggest HVS Good matrix ? :)

[Prettz]

I recently used the 05042003-1 release to encode the credits for a movie (4 and a half minutes, standard b&w scrolling credits with very small text) as a seperate 2-pass encode.
My settings were: search precision 5 (using 6 caused the encode to completely and utterly fail quality-wise), MPEG matrix, VHQ 0, grayscale, QPel, GMC, max overflow improvement and degradation 80%.

The result was 8MB, and the quality could best be described as stupifying. The video was even undersized (meaning I could jack up the desired size for the main movie!) In short though, the awesome quality was completely due to GMC, because it was able to clean up the trails that the text would have otherwise left behind all over the screen.
This story is kind of off topic for this thread, but I felt the urge to share these settings with everyone (including Koepi, in case there are better settings to use with the new release), because this experience was so good that from now on I will encode the credits for all movies I do as seperate 2-pass clips.

[Big_Berny]

Why do you use MPEG-Quant instead of Modulated(HQ)?

[Assault]

@ Sigmatador

I like HVS Good Matrix very much for 1CD rips but I prefer MPEG or HVS Best Matrix for 2CD rips. If you encode with an even higher bitrate then you can use Andreas_78er Matrix.

@ Big_Berny

I wouldn't use Modulated(HQ) because you can't(shouldn't) use b-frames with it and it isn't mpeg4 compliant.

Regards
Assault

[mpeg4_tech]

Originally posted by Assault
@ mpeg4_tech

With a bitrate of 1200 kbit I would use mpeg as quantization matrix instead of h.263. It retains more details.


Assault, thanks for the advice. I do usually aim for a 2 cd encode at bitrates 1200kbps or higher. I'll use the quantization matrix at mpeg the next time I encode.
Regards

[Sharro]

TIP: Do not be afraid of using mpeg quant for 1 cd rips of Disney's Animation and alikes.

All the best,

Sharro

[crusty]

Koepi:
SysKin tweaked VHQ quite a bit, it's no longer "decreasing quality"
That's great to hear. I'm gonna test it as soon as a I burned 50+ movies to CD....:D
Crusty: search for "bframe quant ratio" and you'll get the whole nice formula.
I did, and I think I got it. Also tried to explain it in some more userfriendly terms. I got a 2 for maths on school so I'm not to keen on formulas. :D :D :D

Could someone please remind me..was it b-frames with GMC or VHQ with GMC that was supposed not to work/be compatible with eachother?

So, to sum up, stuff that's OK:
VHQ=4
Motion precision=6
Chroma Optimizer and ChromaME
Mpeg, H263 and custom matrices

Stuff that's so-so:
Lumi masking
GMC
modulated quant(HQ) and new modulated quant

Stuff that will work but that has it's tricks and caveats:
b-frames
Qpel

Right?

[Assault]

@ crusty

It's VHQ and GMC that doesn't work together and I don't think it's very hard to achieve good results with b-frames. ;)

Regards
Assault

[RadicalEd]

well :o I just ran a big test with umaniac's 5/05/03 instabuild (sorry koepi ;p) and did a bunch of decoder compatibility tests and came up with this:
Default settings were motion search: 6, quant: H.263, vhq: 4, chroma motion: on (everything else off), max B-frames: 2, packed bitstream and b-vop compatibility on, chroma optimizer on.
Tweaks were (each separate with default settings for everything else):
quant: mpeg
chroma optimizer: off
vhq: 0
vhq: 1
lumimasking: on
qpel: on
gmc: on
b frames: off
b frames: max 1


# = plays back buggy
* = freezes
^ = refuses to play (no support)

3ivX:
gmc #
qpel #

mpegable:
plays none #

wmp4player:
gmc *
qpel #

quicktime:
gmc^
qpel^
mpeg quant^

envivo:
2 bframes#

libavcodec:
decodes all

DivX 5.05:
qpel#

:\
I'd say everything except qpel and gmc works ok for the most part.

[Rash]

Sorry my completely noob question, but you have to choose "MPEG-Custom" to make your custom Quantizer matrix work, right? :)

Thanks

[Teegedeck]

Yes, exactly.