XviD CVS-Head build by Jawor (14.06.2004)

[Jawor]

Get it here (http://jawormat.republika.pl).

These builds include gruel's VBV code, so I've decided not to keep them private - probably many people are eager to test VBV :D
I've unlocked the DXN profiles due to their direct relation to VBV. Please remember that they're probably not fully DXN complaint due to lack of VCV and VMV.
@developers: if you don't want DXN profiles in my next builds, just tell me. I know you removed them from 1.0.x to avoid problems with DXN, but now, since we have VBV, I wanted to make creating test clips for standalones a little easier.

How to install:
After unzipping, right-click the file xvid.inf and choose Install.... Voila!

Disclaimer:
Please consider this build an experiment. If installing or using these binaries will cause any damage, don't blame me! Use it at your own risk.
This package does not contain a DirectShow decoder.

One last word, folks: VBV works only with two-pass encoding during the second pass.

[bond]

i just want to make clear that is is NOT XviD 1.1!

its a bad idea to label simple cvs compiles as "1.1" as 1.1 has NOT been release till now!
only compiles of the sources officially released with a version number on xvid.org should use the version numbering
call your build something like "cvs compile" or so

imagine all the newbies out there thinking that its official 1.1 and using it, a maybe heavily buggy version, instead of the stable 1.0.1!

[pernas]

Great !

I can't wait to test a few clips with Picard's BetaPlayer on my old PDA (Pocket LooX 600) with slow SD-card IO-access :p

Regards

[Jawor]

Originally posted by bond
i just want to make clear that is is NOT XviD 1.1!

its a bad idea to label simple cvs compiles as "1.1" as 1.1 has NOT been release till now!
call it something like "cvs compile" or so

imagine all the newbies out there thinking that its official 1.1 and using it, a maybe heavily buggy version, instead of the stable 1.0.1!
OK, I changed the title.
I want to state once again that the code from CVS head is not stable. It's only sort of 1.1 wannabe.

[bond]

Originally posted by Jawor
OK, I changed the title.
I want to state once again that [b]the code from CVS head is not stable.thanks :)
its important to keep the difference between officially released, stable builds with version numbering and simple testing builds

[BoNz1]

I don't think VBV works with this build either if it is just a straight vanilla build. VBV is not turned on by default in XviD ATM and nobody has done the vfw intergration. So unless you have done this Jawor it will not work. If you have done this your code should be made available.

[Jawor]

Originally posted by BoNz1
I don't think VBV works with this build either if it is just a straight vanilla build. VBV is not turned on by default in XviD ATM and nobody has done the vfw intergration. So unless you have done this Jawor it will not work. If you have done this your code should be made available.
I was in a hurry since many people wanted to test the CVS-Head code. gruel commited the VBV code to CVS head 5 days ago (see here (http://edu.bnhof.de/pipermail/xvid-devel/2004-June/004310.html)), so my build should include it, but I can see now that VBV is still off. Damn, I'll have to build the codec again with some changes. Sorry for that premature release.

[timeismoney]

Too many version

I think three: mmx/sse/sse is enough, athlon/athlon xp/athlon 64 use the above three tech, too

And I don't think a pentium mmx cpu can playback xvid...

[Jawor]

Originally posted by timeismoney
Too many version

I think three: mmx/sse/sse is enough, athlon/athlon xp/athlon 64 use the above three tech, too
Athlon and Duron CPUs use also 3DNow! instructions that aren't supported by Intel CPUs.
Originally posted by timeismoney
And I don't think a pentium mmx cpu can playback xvid...
This codec is meant for encoding only. It does contain a DirectShow decoder.

[Jawor]

Here (http://jawormat.republika.pl) it is - a build including my small n00bish patches that should force the usage
of VBV. I used VBV settings from DivX Home Theater Profile (the ones that gruel used in his VBV code), since most standalones probably wouldn't handle an avg. bitrate higher than 4000 kbps anyway. The source code is available

here (http://jawormat.republika.pl/XviD_15062004.tar.bz2), all my patches are marked.

The most important changes are included in src/plugins/plugin_2pass2.c:
#define VBV_FORCE 1
/* ===================================================================================== */
/* Jawor's patch - BEGINNING TAG ======================================================= */
/* changes: commenting the following code block, 15.06.2004 ============================ */
/* ===================================================================================== */
/*if (XVID_VERSION_MINOR(param->version) >= 1) {*/
/* ===================================================================================== */
/* Jawor's patch - END TAG ============================================================= */
/* ===================================================================================== */
rc->param.vbv_size = param->vbv_size;
rc->param.vbv_initial = param->vbv_initial;
rc->param.vbv_maxrate = param->vbv_maxrate;
rc->param.vbv_peakrate = param->vbv_peakrate;
/* ===================================================================================== */
/* Jawor's patch - BEGINNING TAG ======================================================= */
/* changes: commenting the following code block, 15.06.2004 ============================ */
/* ===================================================================================== */
/*}else{
rc->param.vbv_size =
rc->param.vbv_initial =
rc->param.vbv_maxrate =
rc->param.vbv_peakrate = 0;
}*/
/* ===================================================================================== */
/* Jawor's patch - END TAG ============================================================= */
/* ===================================================================================== */
I suppose now rc->param.vbv_size will never be set to 0. The parameters are as follows:
#ifdef VBV_FORCE
if (rc->param.vbv_size==0) {
rc->param.vbv_size = 3145728;
rc->param.vbv_initial = 2359296;
rc->param.vbv_maxrate = 4000000;
rc->param.vbv_peakrate = 10000000;
}
#endif

I wanted to check if VBV really works in this build, so I encoded about 2000 frames from The Matrix Reloaded (The

Burly Brawl) PAL edition with my builds from 14.06.2004 and 15.06.2004 (VDM 1.5.4.1, AviSynth 2.5.4, WinME).
XviD settings:
"fast" first pass
second pass target bitrate = 8000 kbps
quant matrix = SixOfNine max=20
MSP=6, VHQ=0, ChromaME ON
AQ, Interlace, Qpel, GMC, B-VOPs OFF
Trellis OFF, Quantizers 1-31
Turbo, Cartoon Mode, Chroma Optimizer, Greyscale OFF
Maximum I-frame interval 250, Frame drop ratio 0
I-frame Boost 5%, AltCC OFF
Overflow Treatment 5/5/5

I turned all these fancy things off because otherwise encoding would take ages on my Celeron 1.2 GHz :(

The AviSynth script looked like this:
MPEG2Source(...)
Tweak(bright=25)
LanczosResize(1025,576)
As you can see, it's a very demanding test clip.

The 14.06 build gave me a 76.3 MB clip, the one from 15.06 created a 34.2 MB clip, so my patch definitely changed

something ;)

Reports from the DRF Analyzer:
14.06.2004
DivX DRF Analyzer v0.9.5 Report!
File Name: D:\MATRIX_RELOADED_DISC_1\VIDEO_TS\VBV Test\no_vbv.avi
FourCC: XVID
Codec: XviD0035
Resolution: [ Width: 1024 Height: 576 ]
Frame Rate: 25.000 frames per second
The Video has 2001 frames [ 00:01:20 ]

Average Frame quality is HIGH [Average DRF/quantizer is 2.20]
Standard Deviation: Quality is HIGH [Std. Deviation is 0.40]
Image Resolution is HIGH

There are NO frame drops ( NO drops is better )

Recomended Resolution: [1152x640] (Target DRF/quantizer=2.8)
The filesize can be smaller!

Performance Caracteristics:
Macroblocks per frame: 2304 ( Poor Playback in Slow Computers, PIII450 or better required )
The Width is multiple of 32

Kilobits per Second: 7811.91
Kilobits per Frame: 312.31
Kilobits per Macroblock: 0.136
Bits per Pixel: 0.54

Frame Type Statistics :
I Frames: 1.90%
P Frames: 98.10%
B Frames: 0.00%
S Frames: 0.00%
N Frames: 0.00%
(More Advanced Codecs use B and S frames)
Frame Quality Statistics :

DRF=1&2: 1570 80.0%
DRF=3: 393 20.0%
DRF=4: 0 0.0%
DRF=5: 0 0.0%
DRF=6: 0 0.0%
DRF=7: 0 0.0%
DRF=8: 0 0.0%
DRF=9: 0 0.0%
DRF>9: 0 0.0%
KeyF/DeltaF: 1.94%
KeyDRF<4: 38
KeyDRF=4: 0
KeyDRF>4: 0

AverageKeyDRF: 2.11
MAXDRF: 3
AverageDRF: 2.20
Deviation: 0.40
15.06.2004
DivX DRF Analyzer v0.9.5 Report!
File Name: D:\MATRIX_RELOADED_DISC_1\VIDEO_TS\VBV Test\vbv.avi
FourCC: XVID
Codec: XviD0035
Resolution: [ Width: 1024 Height: 576 ]
Frame Rate: 25.000 frames per second
The Video has 2001 frames [ 00:01:20 ]

Average Frame quality is MEDIUM [Average DRF/quantizer is 4.27]
Standard Deviation: Quality is MEDIUM [Std. Deviation is 1.00]
Image Resolution is HIGH

There are NO frame drops ( NO drops is better )

Recomended Resolution: [816x464] (Target DRF/quantizer=2.8)
The filesize can be smaller!

Performance Caracteristics:
Macroblocks per frame: 2304 ( Poor Playback in Slow Computers, PIII450 or better required )
The Width is multiple of 32

Kilobits per Second: 3502.52
Kilobits per Frame: 140.02
Kilobits per Macroblock: 0.061
Bits per Pixel: 0.24

Frame Type Statistics :
I Frames: 1.90%
P Frames: 98.10%
B Frames: 0.00%
S Frames: 0.00%
N Frames: 0.00%
(More Advanced Codecs use B and S frames)
Frame Quality Statistics :

DRF=1&2: 68 3.5%
DRF=3: 383 19.5%
DRF=4: 679 34.6%
DRF=5: 633 32.2%
DRF=6: 200 10.2%
DRF=7: 0 0.0%
DRF=8: 0 0.0%
DRF=9: 0 0.0%
DRF>9: 0 0.0%
KeyF/DeltaF: 1.94%
KeyDRF<4: 7
KeyDRF=4: 11
KeyDRF>4: 20

AverageKeyDRF: 4.45
MAXDRF: 6
AverageDRF: 4.27
Deviation: 1.00
Unfortunately both files play jerky on my PC (ffdshow 20.05.2004, libavcodec), but that's normal with these resolutions. I have no standalone player, but I suppose none of the popular ones would play a clip with that resolution.

I took 2 pairs of screenshots with VDM 1.5.4.1:

14.06.2004
http://img16.imageshack.us/img16/9889/1112novbv.png
15.06.2004
http://img16.imageshack.us/img16/9618/1112vbv.png
14.06.2004
http://img16.imageshack.us/img16/984/1655novbv.png
15.06.2004
http://img16.imageshack.us/img16/3406/1655vbv.png

To everyone who'd like to test the new build: the VBV settings are hardcoded. Regardless the chosen profile & level,

VBV settings from DivX Home Theater Profile will be used. Unfortunately I don't have the time to code VBV settings for

all the other profiles (I suppose it's not difficult).

[Bogalvator]

Jawor, are you still building with MinGW?

[Jawor]

Originally posted by Bogalvator
Jawor, are you still building with MinGW?
Yeah, I do. I know there are better compilers for Win32 - the reason I use MingW32's is the GNU GPL. Unfortunately ICL requires a commercial license :(

[Bogalvator]

Well I've read a few times that the Intel compiler sometimes produces buggy compiles, so I'm happy that you provide an alternative.

I also think it's pretty cool that a free/open source compiler produces very competitive builds anyway. So in short - please do continue to use MinGW :)

[Koepi]

Originally posted by Bogalvator
Well I've read a few times that the Intel compiler sometimes produces buggy compiles, so I'm happy that you provide an alternative.


Well, then you should train your listening skills ;) (And stop using FUD techniques. We don't like that around here. Stick to the facts.)

ICL produces "buggy compiles" if at all then in the same situation where GCC fails way worse: using "over-optimization". So you might end up with something unusable when doing that (with both compilers - and as i already wrote, even worse with gcc).

Just compile xvid with gcc's /O6 (highest optimization). It'll be unnoticably faster but will produce crap output (if at all).

Be assured that my ICL compiles don't have buggy compiler optimisations, we often check the md5 sums (binary compatibility) of encodes with and without optimization, and if the resulting bitstream is the same, you don't have to fear anything from that side.

Regards
Koepi

[Jawor]

Here (http://jawormat.webpark.pl) is a new build (19.06.2004) that supports VBV at all profiles & levels (source code (http://jawormat.republika.pl/xvid_19062004.tar.bz2)).

[gatormac]

I've installed and tested the 19062004 build and it doesn't seem to work as far as VBV is concerned. I don't know if it is Gruels code or Jawors code, but I am still getting bitrate spikes on my Philips DVP642. I did a 2 pass encode with the DXN HT NTSC with generic settings on Band of Brothers Episode 3....I opened it in DFXanalyzer which showed that it was working in some places, but not in others. I am not a member of xvid.org so if someone could let Gruel know that would be great.

[Koepi]

Jawor just activated the code, it's gruel's code.

And this is why it is just in CVS and not yet ready for public testing: it doesn't really work, and gruel knows that.

Have phun,
Koepi

[Jawor]

gruel wrote
Please test, especially with difficult material, like high action
so I activated the VBV code that you could test it. That's all I did :D

@gatormac: Thanks for testing. Obviously VBV is buggy, like every new feature in XviD, ergo it really needs further testing.

[Bogalvator]

Well, the source of my "FUD technique" was the foobar2000 FAQ:
http://www.foobar2000.org/FAQ.html#FAQ_o3
a source which I considered reputable and had no reason to disbelieve.

[gatormac]

I figured it would still be buggy....I don't mind being a tester and wanted to let you all know the results. Keopi, I have no idea whether Gruel has been told or not....I read the developer mailing list on xvid.org and haven't seen any mention of VBV's functionality there or here on these boards since it has been added to CVS and just wanted to let him know my experience.

[Koepi]

gatormac:

thanks.

jawor:

the devel list is meant for developers (hence the name). so it should be tested by developers, which are capable of tracking down the issues, maybe even send in corrected code etc. I hope you start to see the difference.

Regards
Koepi

[Jawor]

I see the difference, Koepi. I just thought it would be good to see test results obtained by more people than just the developers. The code is GPL'ed after all...

I know that bug reports sent by the less experienced people are often incomplete and full of some weird ideas, but sometimes it's worth to take a chance.

Perhaps gruel should write what does he think about this?

[Koepi]

gruel often _badly_ attacked me in the past for early releasing code for public testing.

Maybe he has changed his mind since then? ;)

EDIT: Since XviD hit the 1.0 "barrier", I think we really have some responsibility about what we release and what we don't release. More people jumped on the train and it'll blast in our faces if there's something wrong with the encodes.
_Now_ I understand that point, before XviD went 1.0final I didn't - it was (pre-)alpha then, now it's final...

Regards
Koepi

[Jawor]

Originally posted by Koepi
gruel often _badly_ attacked me in the past for early releasing code for public testing.

Maybe he has changed his mind since then?
I hope he did ;)

Originally posted by Koepi
Since XviD hit the 1.0 "barrier", I think we really have some responsibility about what we release and what we don't release. More people jumped on the train and it'll blast in our faces if there's something wrong with the encodes.
_Now_ I understand that point, before XviD went 1.0final I didn't - it was (pre-)alpha then, now it's final...

I get the point. I try to warn people that my builds are completely unofficial and should be considered unstable. I hope that at least some of them read all these warnings :D

I'll put even more of them in the future, since obviously they're needed.

[Koepi]

That's a good idea, thank you.

I could send you my installer-script btw., so you can offer a full-featured binary. There you have the possibility to put some more warning messages as well ;) (It's innosetup which is very easy to setup)

Regards
Koepi

[AllTimeSToneD]

Well it does work for me :D I just encoded The Gladiator in 1280x720P@24fps and it played smooth in my standalone (xbox with xbmc) i encoded the same movie a few months ago using the same settings but at that time there was no VBV. And the Scenes with High Bitrate peaks killed the smoothness and it played more like a slideshow :D Big thx for this "test build" Jawor ;)

But still there is one question i have when i choose the "DXN HT NTSC" profile the Matrix drop down is grayed out, i would like to be able to use the HVS Best matrix for my encoding as i don't really care about compatibility when it comes for playback on "real" standalones (i don't count a modded xbox as a standalone player ;) All i need is the VBV enabled which currently works for me and the option to choose a custom matrix. I guess when the DXN HT NTSC profile is choosen its restricted to one matrix (H263?) Or am i wrong here?

[Jawor]

Unfortunately DXN profiles support only the H.263 quantization type. Many people would like to set VBV parameters by hand when using the "unrestricted" profile. I'll try code it if I find some time ;)

Originally posted by Koepi
I could send you my installer-script btw., so you can offer a full-featured binary. There you have the possibility to put some more warning messages as well (It's innosetup which is very easy to setup)
That would be great!

[AllTimeSToneD]

Isn't it easier just to enable the custom matrix on the dxn profiles :D Sure people would like to edit the VBV settings by hand but as long as it works with the current settings (well at least for me) all i need is the custom matrix enabled :devil: I know that its just a special "test build" but i really would love to have that option :devil: :D :devil: :D

[Jawor]

You can use the build from 15.06.2004 - it has hardcoded VBV settings from the DXN "Home Theater" profile regardless the chosen "Profile @ Level". You can choose AS@L5 there and load any custom quantization matrix you want :D

[AllTimeSToneD]

:D Aow didn't know! :D Big thanks! :D

[gatormac]

Try selecting your quant matrix before selecting your profile. Also, Gladiator wasn't a movie with alot of bitrate spikes anyway, so I'm certain it isn't working as correctly as you think. I've encoded Gladiator at a constant Q of 2 and don't have any problems playing it error-free on my standalone. Also, you are correct in that the XBOX isn't a standalone and isn't limited to the DIVX home theatre profile like other standalones, so the fact that you aren't experiencing bitrate spikes doesn't mean that it is working correctly for DIVX HT standalones.

[AllTimeSToneD]

Yeah i never said its working currently only "its working for me". My Gladiator encode is on a final video size of 4100MB at 720P resolution and thats alot of bitrate to chew even for a 733Mhz xbox. Like said before i encoded it a few months with the same settings before and it was more of a slideshow. With the VBV test builds all the scene are totally smooth in playback which were slideshows before. I also made a encode of a "slideshow scene" with koepi's latest build and its the same problem. But with Jawors VBV test builds it working like a charm :D So i guess its working but not as expected like for standalones :D

[Jawor]

Originally posted by gatormac
Try selecting your quant matrix before selecting your profile.
Somehow I doubt this will work with the DXN profiles - they just don't allow to use the MPEG quantization type and they're coded this way in XviD.

I just made a small patch that allow to customize the VBV settings (maximum bitrate and buffer size) at the "unrestricted" profile. The build should be available tomorrow. I would publish it today, but unfortunately I have no Internet access at home.

[Jawor]

Here (http://jawormat.republika.pl) is another build (25.06.2004) that supports VBV at all profiles & levels and allows to customize VBV parameters (source code (http://jawormat.webpark.pl/xvid_25062004.tar.bz2)). You won't have to wait until tomorrow ;)

[yidaki]

How are these builds concerning stability?
Which one is closest to 1.0.1 for instance.

[Jawor]

The one at the top of the page (1.06.2004). It was compiled from source code a bit older than Koepi's 1.0.1 - it does include the fix in Trellis but not the fixes in the bitrate calculator.

All other builds use sorce code from CVS-Head (the one that some day will become XviD 1.1) and they should be considered unstable.

[bond]

ok i now had the time to test the vbv myself, as i noticed some choppy playback in some high motion scenes and wanted to test, whether this can be avoided by using vbv

first of all my question:
why isnt vbv enabled already in the xvid cvs-head? would be great for being able to test this more heavily, or is gruel still working on it? doesnt seem so...

than i tested jawors XviD 25.06.2004 CVS-Head build, which i think is the latest one available with vbv enabled

here are my quick findings:
1) when enabling divx5's home theater profile it seems that simple profile @ level 3 gets indicated in the streams. i doubt that this is the correct equivalent to htp, also thinking about that i used b-frames, which is not covered by sp
maybe when enabling htp it should be done as divx5 does it: afaik it doesnt indicate a profile :D
2) xvid doesnt indicate the vbv parameters, like divx5, in the vol. i assume thats maybe also important for the playback in some players and should be added
3) even with the htp profile set, which should use pretty low vbv settings from my understanding, i got choppy playback playing the file from .avi on my hd with latest ffdshow. is this normal on a pentium3 866mhz? maybe something else is wrong? :(
any way for checking the actual framesizes?

[gatormac]

Thanks for bringing vbv up again bond. I've been wanting to find out how vbv is progressing and hope that somebody in the know will see this thread and respond. Keopi said it was broken and I havent seen anything in the mailing lists from or to gruel regarding vbv.

[Jawor]

Originally posted by bond
i tested jawors XviD 25.06.2004 CVS-Head build, which i think is the latest one available with vbv enabled
That's right :D
Originally posted by bond
maybe something else is wrong?
What resolution did you use? Did you use Qpel and/or GMC? They can slow down the decoding process significantly.
I had to ask that, but I suppose you know all that already ;)
Originally posted by gatormac
I've been wanting to find out how vbv is progressing
I didn't notice anything new in the VBV code lately.

[Dams]

@Jawor : nice work, I will try it if I have some high motion in some movies !

[bond]

Originally posted by Jawor
What resolution did you use? Did you use Qpel and/or GMC? They can slow down the decoding process significantly.
I had to ask that, but I suppose you know all that alreadyyeah i know that ;)
i tried it also without qpel and gmc and it was also choppy (res: 640x...) :(

[Dams]

@Jawor : how in general the VBV algorithm (or ur VBV algorithm) works ?
when a frame reach the limit, the difference between maximum and limit size is distributed to another frame in order to obtain better quality ?

[celtic_druid]

http://list.xvid.org/pipermail/xvid-devel/2004-June/004310.html

[Dams]

@celtic_druid : thxs a lot ! u're welcome :cool: