XviD-05042003-1

[Koepi]

XviD-05042003-1:
- Fresh CVS checkout.
- sysKin has made refdivx' lumimasking P4-compatible
- the same applies to VHQ
- some speedups by sysKin, he replaced ABS-macros for better compiler optimizations
- inspired by Cruncher: removed fdct_sse2, he suggested it produces worse quality
- still no negative thresholds for bframes
- statsreader updated to 2.0a (merging stats support)

Thanks Cruncher for all the useful hints.

Best regards
Koepi

[JagPanzer]

Koepi, thanx for a new build. : )
I will do some tests now.

[Didée]

>> statsreader updated to 2.0a (merging stats support)

Mmmmmmhhhh...
that's really nice. "Bundesverdienstkreuz" for Koepi!

[Lefungus]

Thanks Koepi and all others xvid developers. I'm a P4 user and xvid has been buggy with these cpus for a long time. Well i'm now able to test all its features again :)

[jarthel]

I assume this build has the same changes as umaniacs. right?

The problem with Umaniacs changelog is that it's too technical. Unless you are developer, you wouldn't have a clue.

Thanks

Jayel

[Koepi]

Sorryx, but what's not to understand about XviD-05042003-1:
- Fresh CVS checkout. ? Of course it has the same changes - and more.

Sorry if it's too technical for you. But unfortunately the changelog is from developers for developers.

Regards
Koepi

[Sharro]

A long thread with just thanks is of no use to anybody, still... not thanking all the work you guys do is rude.

Thanks a million Koepi, Syskin and Cruncher!!

Sharro

[Didée]

@ Koepi

Just an idea:
In addition to the new "merge" support of statsreader, a possibility to "save range" of a statsfile could come handy.
E.g. you made a 2-pass encoding, and then find that several scenes apparently could've needed more bitrate ... then you could build a "cusomized" statsfile with not too much effort.

(Yes, in my mind there is floating something similar to Nandub's stats editor, or even divx's EKG application. Please don't flame me ... I simply like to do some things manually, on occasion)

Didée

[Koepi]

Didee,

that's a good idea indeed. If I find some time (damn work all the tme :( ) I'll implement it.

All:

Thanks for the thanks :)

Regards
Koepi

[gotaserena]

It would be a dream to have a CCE-like interface to make judgments over bitrate distribution. A bit far-fetched, but a dream nevertheless.

Good build, koepi, hasn't crashed with me so far. I hope that sse2 comes back soon.

[kilg0r3]

high as it seems xvid does not insert i frames at the correct position. i just skipped through a movie looking for a good cut point. the iframes i could find always were positioned in a range of 1 frame before to 6 frames after the actual scene change. I checked this in vdub and with ffdshow. furthermore, when loading the statsfile into vdubmod, i could see that the iframes reported internally by vdubmod were not at the same position as those reported by the stats file.
http://www.mynetcologne.de/~nc-allgeife8/img/capture.jpg
however, the iframes in the stats file also were not located at the correct positions. at one occasion e.g. the iframe in the stream was the frame before the scene change, while the stats file reported an i frame one frame after the SC.

btw, is it save to use packed bitstream?

The stream i used for this testing was the first-pass of an encode which i have done with the following settings. this might explain some of the problems but not all. i will do the second pass today.

XviD BUILD (05.04.2003 18:17:50)
Mode: 2 Pass - 1st pass
Discard 1st Pass: OFF
Motion Search Precision: 6 - Ultra
Quantization Type: MPEG
FourCC Used: XVID
Max I-frame Interval: 400, Min I-frame Interval: 10
Lumimasking: ON
Interlacing: OFF
Greyscale: OFF
Max B-frames: 3
B-frames Quantizer Ratio: 126%
Packed Bitstream: OFF
DX50 B-VOP Compatibility: ON
Print debug info on each frame: OFF
Min I-frame Quantizer: 2, Max I-frame Quantizer: 31
Min P-frame Quantizer: 2, Max P-frame Quantizer: 31
Start Credits 0-589, End Credits: 135131-140772
Encode credits in greyscale: OFF
Credits I-frame Quantizer: 20, Credits P-frame Quantizer: 20

[sungey]

hmm my encode without bframe/gmc/qpel with this build has the I-frame inserted correctly.... (at the scene change)...

[kilg0r3]

in a second attempt, a partial encode of 1500 frames with _enabled_ option 'packed bitstream', the iframe seems to set at the right position now. yet, the divergence between stats file and video stream still persists.

therefore, again, is it safe to use packed bitstream?

[Koepi]

Erm. There wasn't mentioned that 2nd pass dynamic IPB-decision is fixed yet. Thus you can insert whatever you want into the statsfile, the frame_type_ won't be respected.
This is not the case when disabling bframes (max bframes: -1) as there the good old working code still is used there.

The difference between the stats and vdub-reported frame-positions is due to padframes at the beginning of the stream, internally xvid handles that correctly.

Koepi

[kilg0r3]

Originally posted by Koepi
Erm. There wasn't mentioned that 2nd pass dynamic IPB-decision is fixed yet. okay, this one of those bugs that had eluded me:)

The difference between the stats and vdub-reported frame-positions is due to padframes at the beginning of the stream, internally xvid handles that correctly.

Shouldn't the diffrence be constant then? Well heck, it's working:)

btw i managed to cut the stream properly. i don't know why, but it was even harder to find the right position this time. this cutting issue really is a pain in the ... But that's not really an xvid issue, is it?

cheers and sorry for the fuzz.

[drebel]

@ kilg0r3
I know i'm getting OT now,but i couldnt avoid noticing some "strange" values in your settings

- In Xvid sticky there's a suggestion of Unfilter(3,3).I thought the filter goes in steps of 5(0,5,10 etc.(Is the nearest value used?)
- B-frames Quantizer Ratio: 126% .I thought it goes in steps of 50,but i might be mistaken .Pls correct me if i'm wrong :)

regards,
george

[kilg0r3]

(1.)
@drebel
i am just guessing, but i don't think there is something like a 50% step.

(2.)

Just to make a bit more noise ;). Cuold somebody explain me what a message 'BrokenB-Frame, TPP=0 TBP=2' means. I get it when seeking frame-by-frame in a stream encoded with the above mentioned settings.http://www.mynetcologne.de/~nc-allgeife8/img/Capture2.jpg

[Selur]

Little bug report:

VHQ>1 + qpel causes flowting walls
(VHQ)>1 on it's own doesn't)

I know this has den reported earlier, but then VHQ>1 itself caused this, but now it only apears when vhq&qpel are enabled.
So could it be that qpel is not acurate enough, and we need something like 'simple qpel' ;)?

Cu Selur

ps.: @kilg0r: got that Broken-B-Frame-problem, some time back, with an earlier version of VirtualDubMod, but it doesn't apear with the current version.

[kilg0r3]

@Selur
I am using the latest version of vdm, strange.

@All

I have compiled a little collection of clips i encoded with different combinations of bframes vhq and qpel. Have a look (http://www.mynetcologne.de/~nc-allgeife8/downloads/material.rar) and judge yourself. The encodes are done with constant quant 2. When using bframes the settings were 3,BFR 100,BFO 100. These are very mild settings not stressing the codec very much.

The file is about 2mb. I recommend looking at in vdm at 2x zoom.

[Selur]

tried your clips and for me it gives the same error and I'm able to reproduce this with clips I encoded myself,.. (same error if you try it with VirtualDub and an avi file)

Strange, now I'm confused. I'm sure I had this problem before and it disapeared, but maybe I just didn't scroll so much in clip I used B-frames and VHQ,...

Cu Selur

[kilg0r3]

Perhaps it is specific to the resent xvid builds or(!) to not using DX50 compatibility mode. When i seek through a movie with a player, the playback starts with a broken frame (incoherent blocks). I cant describe it more exactly because it is visible only for a fraction of a second (presumable 1/25 sec :D).

Anyway for me it is not a problem, just being curious.

[Prettz]

What is the correct term for when, during fades (particularly fade-ins), bright areas in the video appear to expand/contract or "bulge", rather than fading in or out normally?
Xvid is the only codec that shows this problem, and even then I only see it when decoding with ffdshow; when I check "Use Xvid" in ffdshow (I'm using Koepi's latest Xvid build) the problem disappears.

[PowerMacG4]

YAY. B-Frames successfully stay disabled in this build.

[gruntle8]

Prettz: I may be wrong, but I believe that's a lumi masking artifact.

[Sharro]

Originally posted by Prettz
when I check "Use Xvid" in ffdshow (I'm using Koepi's latest Xvid build) the problem disappears.

Errr.. Wouldn't you say ...that this is a decoding problem not related to Xvid?

Try ffdshow forum...OR... Decode with xvid.dll

I think...

All the best,

Sharro

PS: I'm quite sure I read something here (or somewhere else) about ffdshow latest alpha not being able to properly decode some features of most recent Xvid builds. But I'm nobody....

[Prettz]

Originally posted by gruntle8
Prettz: I may be wrong, but I believe that's a lumi masking artifact.
I didn't use lumimasking in any of the xvid encodes where I saw the problem.


Originally posted by Sharro
Errr.. Wouldn't you say ...that this is a decoding problem not related to Xvid?

I mentioned it here because this problem only appears with xvid files, and it appears frequently in them (decoding with ffdshow, at least).

[sungey]

Prettz .. got a clip ?

[Koepi]

Turn off the automatic workarounds in ffdshow and try again decoding xvid with it. You may find that ffdshow is a bit outdated in the meantime.

Regards
Koepi

[kastro68]

I've compared several encodes of Hero using these two builds, I spent lots of time comparing the several scenes over and over.

My conclusion is that I think the 17-02-2003 build gives slightly better results than the 5-04-2003. It is still very hard to tell the difference between the two unless you watch the same scenes over and over and over.

Using 3 b-frames, vhq1, qpel, chroma me, everything else on default.

for 5-04-2003 I used a b-frame threshold of 50.

The 17-02-2003 build gave a first pass size of 1156 and the 5-04-2003 gave a 1st pass size of 1683.

Just and observation that I thought I should share.

[iago]

@Koepi

After a long long time, heya again my friend! Recently, time has really become my enemy! ;)

Now I'm searching through the threads, gathering information, and doing some tests with your new build, feeling a total newbie! ;)

ciao,
iago

[Didée]

Ahoy Iago!!!

Glad to see you back here - I was already worrying what might have happened.
As for encoding, you may relax: we're still using mpeg-4 techniques ... ;)

Didée

[Koepi]

Merhaba iago!

I'm happy to see you back on the xvid-track! ;) I hope there didn't happen too much in the meantime, I can't remember huge steps we made at least. Except for bframes are of amazing quality now, no flickering anymore, we use SimpleIDCT now, sysKin added VHQ modes,...

Looking forward to your impressions :)

Best regards dostum!
Koepi

[Sharro]

Iago,

Good to see that email kind of "awake" you!!!!!

Where have you been? Damn Life who doesn't leave us proper time for things we like eh?

All the best,


Sharro

PS: @koepi I asked Iago to show up on irc!!! Hope he has time for it.

[JasonFly]

I didn't use qpel very often but i did some tests in the lasts weeks and I saw some artifacts which seems to be related to qpel.

I have tested with différent config(bframe+qpel+vhq,Bframe+Qpel without vhq, qpel alone....) and the only clips which had some artifacts were those were I used qpel.

The artifacts I m talking about are the those:
Some parts of the image seem to "slide" and that's not very beautiful.This appears more otfen in movies with noise but also in others.
This also happens when camera make some "reverse" movement(it moves up and then it moves down for example)

This phenomenon is more visible in high quant(even in q=6, the result is very ugly).

I haven't read all xvid threads (or I have read them but don't remember such a comment on qpel) so I not sure if this problem is already known or if this only happens with me.

[cca]

I also have noticed the sliding problem JasonFly mentions. Maybe there is still bugs with Qpel.

[Sigmatador]

Originally posted by JasonFly
Some parts of the image seem to "slide" and that's not very beautiful.This appears more otfen in movies with noise but also in others.
This also happens when camera make some "reverse" movement(it moves up and then it moves down for example)

This phenomenon is more visible in high quant(even in q=6, the result is very ugly).

i notice this problem too

[BoNz1]

What are you guys using to decode your movie? I was using an older version of ffdshow which had this problem but I switched to Nic's directshow and now it is gone. If you are reporting a bug you should really give more info on how you are decoding since it is quite possible this is the problem.

[Sigmatador]

i know how to decode a xvid ^^:
ffdshow20030103 with "xvid" and "use xvid" enabled (and xvid for idct)

but right i didn't try last nic's dshow, i'm going to ^^

but i notice this problem too with a divx502 movie (with divx502 dshow decoder, but worth than xvid ^^) so i think i'm not a qpel fan (well, most of my encode are anime so it's really useless ^^)
on a movie with h.263 matrice it's not really noticiable, it's a perfectionnist critic ^^

[drebel]

i know how to decode a xvid ^^:
ffdshow20030103 with "xvid" and "use xvid" enabled (and xvid for idct)

but right i didn't try last nic's dshow, i'm going to ^^

I'm sure you do.But to eliminate qpel smearing problem u must have the latest alpha ffdshow 18.04.04 by athos and use simple IDCT...
...or Nic's decoder ;)

regards,
george

[cca]

Well, I AM using the latest alpha ffdshow 18.04.04 by athos with simple IDCT selected! The sliding problem is there...

[JasonFly]

The problem also exists with nic latest dec build.I usually decode with Nic's decoder.I havn't tested with ffdshow.

Does ffdshow really is better than Nic's decoder or is it just better for qpel decoding that there is a difference?

EDIT
I have just tested with the latest ffdshow build and the result seems to be good(no more "slidding" effects in the two movies I had noticed this problem.

So far, I didn't found any interest in ffdshow over Nic's decoder but qpel decoding is definitely in favour of ffdshow.
I haven't tested with older builds of ffdshow but Sigmatador said that this didn't work well with them too.

[Sigmatador]

Originally posted by drebel
I'm sure you do.But to eliminate qpel smearing problem u must have the latest alpha ffdshow 18.04.04 by athos and use simple IDCT...
...or Nic's decoder ;)

regards,
george

i'm going to test with the 21.04.04 one because 18.04.04 doesn't work properly on my computer (2k adv server).

[wotef]

unfortunately, i agree - still get qpel smearing with the newest ffdshow - and btw, doesn't "use xvid" just give the decoding job back to the codec? i.e. you guys are getting different results simply cause you have different xvid binaries installed, i.e. using different xvid.ax files

for now, i'm going to use the old trick of installing newer binaries, but overwriting my xvid.ax with nic's 16-feb version; it has the best qpel decoding

[Menion2k]

Koepi this build crashes between the first and the second pass with Virtualdubmod v2, as one older build (I don't remeber exactly the build, but there is a thread about this). Tested with Virtualdubmod v2 (latest binaries), video imported via avisynth 2.5.1 (latest build), Athlon XP 2700+, WinXP SP1 Italian, 512Mb RAM. The different xvid settings don't seem to influence the crash. Bye!

[Koepi]

Menion,

please check the location of your stats-file: is the space for it sufficient? Does the path exist?

You're the first one to report this behaviour, so it's unlikely to be a xvid bug.

Best regards
Koepi

[Menion2k]

stats file is in D:\, the space if more than sufficient and it was created succesfull, in fact the first pass was correctly completed. Now I restarted the second pass, and the encoding is proceding correctly, later I'll post the result. Bye!

Update: The second pass was succeffully created!

[InfoCynic]

When I was home for Easter, I saw this behavior at my friend's house (Athlon XP 2200+, WinXP SP1, AviSynth 2.5, VDubMod latest). VDubMod would throw an error & crash after each first pass. It was after the pass, because the stats file was complete and the second pass completed fine after that. I'll see if I can get him to post here with more details. He would have been using XviD-22032003-1.exe.

[PerCIVaL]

are you using the joblist?

With virtualdubmod and said koepi build, vdbmod crashes after the first pass (first job) finishes and the second pass (second job) is about to start with me.

Same with divx 5.04 btw.

[InfoCynic]

That was using GKnot 0.28 beta 3, IIRC. (Which is to say, no, b/c GKnot does not use VDM's job list.)

[Koepi]

And I always thought GKnot setups VDub via the joblist because that's what it's there for.

Please take a look at the vdub forum and see if it is somewhere mentioned (some internal variables changed versus original vdub, thus it might be that gknot messes with the wrong ones).

And: do your encoding properly, without one-click-and-go-tools. As you can see, they aren't adopted to the new builds of the used tools and can't unleash the full features available, or mess up otherwise.

Still no xvid bug. It's gknot or vdubmod which are failing.

Koepi

[iago]

Hello Koepi (and all),

I have finished my second encode with the 05042003-1 build, using very plain avs scripts for both encodes including only UnDot() and Limiter(), and aiming for 1CD with ogg audio. I am really glad with the results I have got with the following encoding parameters, though it takes a damn long time on my poor system using all those amazing advanced features:

h263
vhq1
cm
qpel
b-frames 3/150/100/0
chroma optimizer

Using ffdshow latest to decode and it's great not to have the qpel smearing anymore.

best regards,
iago

[Koepi]

Thanks for your test iago! Nice to see that you're back on xvid-track again ;)

I've new settings which usually give very nice results:

in the avs script, i use LumaFilter(), UnFilter(values depending on the source), UnDot() and on noisy DVDs Convolution3d(values depending on the movie). LanczosResize() if possible, else bicubic(..,..,0,0.5).

I nearly always use the hvs-good-picture-matrix.

bframes: 2/150/100/255 (!)

I found that 3 bframes harm quality considerably, maybe you want to retest with 2 bframes, iago :)

Chroma ME

(seldom qpel, helps (de-)ringing a lot)

VHQ1

linear scaling, payback proportional (looks better to me in the meantime)

The secret is to tune the avs-script so that ringing and noise gets reduced. With this procedure, my encodes look really amazing.

I think i'll set those values as default in my next builds ;)

Best regards
Koepi

[iago]

Hehe, heya mate, thanks a lot for sharing your settings. ;)

b-frame threshold: 255

Wow! Though I really like the quality of b-frames in the latest build, it is really a quite heavy b-frame setting, isn't it? Still, the possible drawbacks are perhaps compensated with the use of 2 b-frames instead of 3, as you have mentioned. However, I really like the p-b frame decision in this build with the default threshold of 0 ;).

payback proportional

That's what I had always recommended in my previous posts, but it was you then who had discouraged me, insistently suggesting payback with bias! ;)

vhq1 - cm - linear scaling

Yeah, I guess these three are already commonly agreed upon feaures, aren't they?

q-pel

As long as it doesn't pose smearing (seems that the problem is finally resolved in the latest ffdshow together with the use of simple idct in your builds), I won't give up on it.

And, for 1CD encodes, I usually prefer to stay within the limits of BicubicResize, not more than (..,..,0,0.5), unless the movie is "very" compressible.

Finally, yeah, fine tuning the avs scripts with LumaFilter, UnFilter, etc. has always been a great pleasure for me too! ;)

Thanks a lot one more time for all your effort and great work, my friend (and of course, all other developers of XviD).

ciao,
iago

edit: and the good old h263 is still my favourite for 1CD encodes, though I agree with you that hvs-good-matrix produces pretty nice results as well.

[manono]

Hi-

b-frame threshold: 255

Wow!

Yes, that one surprised me also. I really like having the threshold feature, and I'm very conservative with the B-Frames (I only use 1, with thresh=10). But it was good to see Koepi's preferred settings. Thanks! Now you've got no more secrets to hide. :)

And it's really nice to see you around more often, iago.

[kastro68]

Can I ask what the default B-frame threshold for Koepi's 17-02-2003 build is.

Because I tried the 5-04-2003 build with a B-threshold of 50 and still liked the 17th build more.

[iago]

Koepi,

You are definitely not only a great "coder" but also a great "encoder", my friend! Yeah, now I really like using b-frames (2-150-100) to its full extent with a threshold of 255! And yes, quality is still there! ;)


Heya manono,

Yes, it's really nice to be with all the great people around here again. Btw, you should try this setting as well! ;)

best regards,
iago

[PerCIVaL]

I was meaning to ask this,

The hvs-good-picture-matrix, what are the benefits of this matrix?

can anyone explain? I usually go with the standard h263 or modulated hq matrices of xvid.

Ow, and where can I get it?

[iago]

PerCIVaL,

You can find it here:
http://forum.doom9.org/showthread.php?threadid=34493&highlight=hvs+matrix

And imho it's better to try yourself and do your own comparison to see the differences.

iago

[PerCIVaL]

thanks iago,

I'm trying it now.

[corigan]

And I always thought GKnot setups VDub via the joblist because that's what it's there for.

Please take a look at the vdub forum and see if it is somewhere mentioned (some internal variables changed versus original vdub, thus it might be that gknot messes with the wrong ones).

And: do your encoding properly, without one-click-and-go-tools. As you can see, they aren't adopted to the new builds of the used tools and can't unleash the full features available, or mess up otherwise.

Still no xvid bug. It's gknot or vdubmod which are failing.

----------------

I have used the 0504 codec a few times without crashes and no problems. I use gknot just to make the simple avs script, cropping, and bitrate calc. Then I go and edit the avs scripts by hand and add in all the features I want that gknot doesn't offer. I then go and encode by hand in vdub/vdubmod. I installed the new gknot rip-pack/codec 1.1 pack two nights ago (0.28) and low and behold I now crash right after the first pass like previously stated in this thread. This definately shows me that it indeed isn't an xvid problem with the codec since I had used the new 0504 for encodes a few time with no problems (previous to install of rippack/codec pack) . Unfortunately this is just damn annoying with the crash right after the first pass. With virtualdub 1.51 I couldn't even go back in and do the 2nd pass by hand via the job control. I did go in with vdubmod and just do a flat out 2nd pass and use the stats file (that was created from the first pass before the crash) and it worked (not via the job control). I wonder if this has something to do with the job control, and my question would be what was re-installed/installed by the new gknot packs that is making this error occur. I know it's not anything that would deal with your codec koepi, just thought I would post my findings here and see if anybody else has any idea how to get rid of this error. Gknot also did not overwrite the specific build of vdubmod or vdub that I use (I keep them in seperate directories away from gknot) but now both seem to be plauged with this problem. Oh well, Hope someone can figure this out, I'm off to do some more testing to see if I can figure out a solution on my own.

Corigan

[corigan]

I'm lame, looks like it's ffvfw that is causing the crash due to B-Frames. I assume everyone else who is experiencing this crash has ffvfw installed and was trying to use b-frames in their encodes. I removed ffvfw and the crash on the 2nd pass (with bframes) and another error I was recieving (when selecting the compression menu, an err with WMVcore.dll) have both gone away. I guess it would help if I would read the first post of the ffvfw thread (you know, the one that says b-frames don't work in 2 pass mode yet.. :P) Oh well, sorry to spam your thread koepi, but this indeed does prove it's not a problem with your codec.

Corigan

[Menion2k]

what is ffvfw???

[Siku]

@Forum
I've made a dozen ecodes with this build and I've got pretty good results. I especially liked the result when I tested Koepi's suggestion of using b-frames 2/150/100/255.

@Menion2k
Check this (http://forum.doom9.org/showthread.php?s=&threadid=42710) out.

@Koepi
Is there coming a new build soon? :)

Regards,
Siku

[Menion2k]

weel i dind't install ffvfw, so my crashes between first and second pass is due to another cause. Bye!

[Rober2D2]

Just to make a bit more noise . Cuold somebody explain me what a message 'BrokenB-Frame, TPP=0 TBP=2' means. I get it when seeking frame-by-frame in a stream encoded with the above mentioned settings.

I have had this same problem, but only when I use Packed Bitstream. It appears a few times during a film. Problem seems to be solved in Umaniac's build from 14/04/2003, although I must make more test before assuring this.

[sungey]

Using Koepi 5-4-2003

Motion Precision = 6
Quantization Type = h.263
FourCC = DX50
VHQ = 1
Chroma ME = ON
Max-Min I frame = Default

Max B frame = <varies>
Bframe ratio/offset = default
Bframe threshold = 255

DX50 compatibility = ON

<Config> <First pass size>
Using 2 bframe = 207 mb
Using 3 bframe = 204 mb
Using 4 bframe = 204 mb
Using 5 bframe = 206 mb
Using 6 bframe = 207 mb

Note the changes of filesizes .. anyone know why is it like that ?

[Koepi]

The "usual" I/P motion detection gets less effective (remember, bframes derive from I/P-frames, but nothing derives from bframes) due to the fact that the frames are further distance which it has to work on.

Koepi