An XviD bug, a newer DShow filter & my new site

[Nic]

Hi,

First:
XviD Bug: This was what was crashing my DShow filter & possibly caused problems for milan with the new dev3 api. In xvid.c there is a line in xvid_init there is a line:
fprintf(stderr,"sad16v=XMM\n");

That is trying to write to stderr which is ok for most things apart from where stderr doesn't exist (like interfaces less programs, i.e. dshow filters). Remove that line, I could use my DShow filter again :) (minor stuff hey, but still took long enough to find)
(of course this isn't really a "bug", because its the dev api, but I thought it may be useful to report it anyway)

Second:
My new DShow filter works with dev3 api (YaY!) but it also now no longer relies on XviD.dll, this means it will no longer cause compatibility problems with each new version (& if you only needed xvid decoding then you could just install the .ax & not xvid.dll). I may add something to make it decode DivX if people want it.

Third:
Is my new site, http://nic.dnsalias.com a new dev3-api build is up (dont know how stable it is, but its there for testing) with the new DShow (src will be up soon). Hopefully my site is a bit easier to use & more useful.

Suggestions/bugs/comments welcome.

-Nic

[Koepi]

Nice work Nic! :)

Can you send me the xvid.ax only please so I can add that (and maybe even build a stand-alone decoder installer from it which I'll send back then of course)? I've plenty of stuff on HD for encoding so I don't have the option to install your build and take it from there (3 movies, some series,..).

Thanks,

Best regards,
Koepi

[Nic]

Sent to your roeder account :) Im hoping it wont get any annoying explorer.exe crashes anymore....

If anyone gets such a crash, could you post it here :) thanks.

-Nic

[Koepi]

Thanks a million Nic,

I'll build the installer ASAP... maybe I'm _really_ fast now ;)

Regards,
Koepi

[Koepi]

Hi Nic,

check your mail at that account you sent it from ;)

Best regards,
Koepi

[Nic]

Cool :) A 100kb XviD Decoder installer, no one can complain at that :D

-Nic

[milan]

ffdshow crashes when it uses xvid.dll from your newest build for decoding. After I saw your DirectShow filter sources I was able to access xvid quantizer table (better postprocessing) and motion vectors (ShowMV filter). Now when MACROBLOCK structure is changed it doesn't work anymore. But API version wasn't changed and I don't see any possilility to distinguish dev-api-3 and API 2.1. The only solution for now is to remove xvid internal structures access from ffdshow.

[Nic]

The way the XviD api works is being changed by gomgom so, hopefully youll be able to get round that soon :)

-Nic

[-h]

ffdshow crashes when it uses xvid.dll from your newest build for decoding. After I saw your DirectShow filter sources I was able to access xvid quantizer table (better postprocessing) and motion vectors (ShowMV filter). Now when MACROBLOCK structure is changed it doesn't work anymore. But API version wasn't changed and I don't see any possilility to distinguish dev-api-3 and API 2.1. The only solution for now is to remove xvid internal structures access from ffdshow.

Well, mapping handles to internal structs isn't really part of the API ;)

It's safe to assume that a 2.1-API DLL can't be mapped in this fashion, due to the gradual changes in structs like MACROBLOCK over time. However 3.0 should always be safe, as there are plans to export a standard array of quant/mv/dc/ac information.

-h

[HarryM]

Cool :)

I test it now...

[Shayne]

Site looks real good nic

[milan]

Well, mapping handles to internal structs isn't really part of the API


I know, but at least core_build could be changed.

[milan]

Cool A 100kb XviD Decoder installer, no one can complain at that


I'm sorry if you did this already, but for realy tiny decoding filter, you could try to remove references to encoding functions. Many of them are in xvid_init function, for example

quant_intra = quant_intra_c; - remove
dequant_intra = dequant_intra_c;
quant_inter = quant_inter_c; - remove
dequant_inter = dequant_inter_c;

Maybe few other changes will be needed, but linker dead code elimination should do the rest. When I was deciding to add xvid decoding routines to ffdshow is was able to compile 75 KB xvid_dec.dll library.

[Nic]

I was thinking the same thing milan :) ill try & cut down the size. Koepi do you have the nsis script you used? :)

Im going to add new luminance code, clean it up alot & add contrast & a few other bits. Then ill just put the source on my site for others to play with :)

Cheers,
-Nic

ps
I forgot the TOO_SMALL_LIMIT def so its probably set to 3 when it should be set to 1. If your planning to do any encoding this weekend, then I recommend Koepi's latest build rather than my own, ill fix that on my site at the latest monday.

[Nic]

New Build up: http://nic.dnsalias.com

Quarter Pel was added to the CVS on the 4th of this month. Ive tried to turn it on with Motion Level 6: Ultra High :) (experimental! But looks good to me!)

TOOSMALL_LIMIT set to 1 this time :)

Cheers,
-Nic

[-h]

Works well doesn't it Nic :)

Unfortunately there are some compatibility issues with inter4v+qpel - msfdam decodes it fine, but libavcodec/divx5/envivio don't like it. Yet.

-h

[Nic]

I know...concerning isn't it? Either Micheal's right and everyone else is wrong. or we've got it wrong.

Dont know about you but I trust Micheal & msfdam :)
(& if christoph gives it the nod & works on it....then YaY! :) )

Cheers,
-Nic

ps
Saw the australian pink floyd on Saturday. Weird hearing floyd songs sung with an aussie twang :)

pps
I assume by msfdam, thats the Microsoft Final Draft AMendment Reference Software?
(I know its obvious but im stupid.... )

[MaTTeR]

Worked very well for me on 2 test clips about 9mins long at 640 resolution. Details and color were superb with no artifacts. I decoded using the latest alpha of ffdshow with no post processing enabled.

Great work guys! Qpel was my long awaited love that I kept looking for, should be fun to test further now.

PS. Compile seems to be pretty fast as well using MPEG quants on both passes.

[Koepi]

Does it work with MVHints, too?
I'm writing out the MVhints now as well (hm. although I use my own compile ;) ), but it would be nice to know if this is a waste of space. I like the idea of quality improving MVhints, that's why I'd like to use it....

Thanks,

regards,
Koepi

PS: Nic, can you send me your updated DSF as well please? I don't know if your old DSF can decode QPEL already... :)

PPS: does it suffice to add a "frame.global |= QUARTERPEL;" after the "frame.global |= HALFPEL" line to activate qpel mode?

[-h]

Does it work with MVHints, too?

Very good question.. I haven't read up on the mechanics of it but since it just seems to increase MV resolution (1 = qpel, 2 = halfpel, 4 = pel) and functions like d_mv_bits() are unchanged, all MV/fcode storage for hints should be fully functional.

PPS: does it suffice to add a "frame.global |= QUARTERPEL;" after the "frame.global |= HALFPEL" line to activate qpel mode?

Yes, but you'll get better results with the qpel-specific REFINE16/REFINE8 flags.

-h

[Koepi]

Thanks Dan :)

*stopping encoding*
*rebuilding XviD*

:)

Thanks a million,
Koepi

EDIT: Hm. there's only halfpel-refine16|8 in XviD.h :-/
Restarting encoding...

EDIT2: stupid me. Ended up in the stable-tree sources on my HD %)
Recompiling with qpel refine flags...

[-h]

Has it crashed for either of you guys? I can only get a qpel encode to work ~5% of the time. Every other time it's an access violation on the first inter frame (?).

Looks like I'll have to start using binaries compiled by someone else :)

-h

[Koepi]

It works perfectly for me (sloooow, but effective in frame sizes...).
Never crashed no matter what I did, and I restarted encoding about 5-10 times now... no crashes.

I might send you a compile, if you like.

Best regards,
Koepi

[Acaila]

No problem for me either, but what do you mean with slow Koepi? I haven't noticed any slowdown at all compared to a normal build (at most 5% speed decrease, but that's all).

All we had to do to use QPel was motion search 6 right?

[reet]

Originally posted by -h
I can only get a qpel encode to work ~5% of the time.
-h

how can I check when a qpel worked??? maybe then i can help
cause i used the last binary on nic's site and everything went OK

i know how to use debugview but i can't seem to find any info about qpel in it :-(

greetz reet

[MaTTeR]

I didn't notice any encoding slowdowns at all. Furthermore I seen no mention of Qpel in debug view either, using MS6 I'm assuming it worked anyway:)

I just noticed a lot of access violations happening when using Decomb in my script now though. Like -h, it does seem to happen at first inter frame. Disabling decomb.dll and everything encodes perfectly though....very odd.

Encoding a full 4:3 DVD with it now using a very basic script, suspect it should be fine.

[Dali Lama]

Hi,

Can I ask what is QPEL used for. Is it a more accurate/precise Motion Estimation routine? My impressions of QPEL in Divx 5 at least was that at low bitrates (aka 1-CD) it increased artifacts, due to the more precise ME. Without it, things were a little bit more blurry but much less artifacts. Also, QPEL seemed to reduce filesize somewhat.

However, with my first impressions of Xvid QPEL, the quality seems to improve a little in complex scenes with fog/ghosting and it doesn't really sharpen. Also, I see no filesize reduction. I will have to test at lowbitrate. These observations are for Quant 2. Overall, I think this is good, because if QPEL is a more accurate ME, which is why Nic put it under that tab in Search=6, then I think QPEL will be used in all encodings.

Also, the speed of Xvid QPEL is ridiculously fast compared to Divx5, both in encoding and decoding.

Good work all,

Dali

[-h]

The goal of motion estimation is to re-create the current frame by copying blocks from a reference frame - the better job it does of re-creating the current frame (i.e. the closer the copied block is to the current one), the fewer bits will be needed to store what's left over.

Quarter-pel motion estimation is just another way of letting motion estimation find blocks which more closely match the current block. If a quarter-pel implementation is good enough, the result will always look better (for a given size) or require fewer bits (for a given quantizer) than an implementation without quarter-pel.

As for what it actually does, you'd have to first understand half-pel motion estimation. In that case, instead of just copying pixels from the reference frame to the current one, pixel values that are "half-way" between one pixel and another pixel are copied. These half-way values are derived by averaging two neighbouring values together, and improves compression by around 30%.

Quarter-pel makes things more complicated. Performing another averaging operation on the half-way values and the original full-pel values would give some kind of quarter-pel position, but would be weighted too far towards the full-pel value and not be terribly useful. The "interpolation" for quarter-pel values actually involves passing full-pel values through a lowpass filter, then performing further averaging between them. This extra lowpass step is part of the reason for quarter-pel mode being slower than full- or half-pel, but the fact that you're doing so many more SAD checks per block search also contributes to it.

All above should have "I think" appended, of course.

-h

[iago]

Hello all,

Currently I'm doing a full 2-pass encode with Nic's 07/10/02 build with motion search 6 and using MVHints too (with h263 quantization). I'll report back the results when finished.

best regards and thanks for the new Qpel feature addded,
iago

[Razor04]

Regarding MVHints...wasn't there a better version of them in the works? One that didn't lose detail but actually improved it? If there is one is it in this latest build from Nic? I can't wait to test this build when I get some free time (College doesn't leave much). Thx!

[Koepi]

@Razor:

if you'd snoop/lurk around a little on this forum, you'd know (a search could help you there, too!), that with sysKin's ME hintedMV is used for further precision in motion estimation during the 2nd pass, thus possibly increasing quality, at least not "damaging" it.

[Razor04]

@Koepi:

I know...I read about it on the dev mailing list...but didn't know whether it had actually been implemented or not. I read this forum regularly and didn't see anything mentioning that it had. Thx for the answer.

[HarryM]

@Nic:

I tested your build 07102002 with ME=5 (qpel not activated?) and ME=6 (qpel activated?).

I see no diferences!

The quality is comparable, the filesize_at_quant2 is very close identical.

I have experiences with using of qpel in divx5. Using qpel has much, much influence with filesize...

I have feeling, that qpel is'nt activated/ready, that I dont see any encoding speed slowdown too. ;)

[Nic]

Well, there is a possibility its not enabled, but it should be(?) But all I did was add the QPel refined flags to the motion 6 field of the vfw. So I may have missed something (I just didnt have the time at work to check).

Koepi/h do you know of anything I may have missed/done wrong?
(if not ill check it out again tomorrow & have a proper look)

Cheers,
-Nic

ps
@koepi: Cant send the filter today :( But I will tomorrow :)

[Koepi]

Nic:

the only thing that misses after that is that
frame.general |= XVID_QARTERPEL; // right after the line
frame.general |= XVID_HALFPEL;

This produces a 10-15% slower encoding here, together with MV hints the image is totally f***ed up, without MVhints it looks somewhat OK, but I don't have a propper qpel decoer at hand so I can't be sure it's right, latest ffdshow snapshot produces some small artefacts at least.
But as gruel et al discussed on devel list qpel is buggy (maybe at least), michael claimed that he didn't write it's bug-free ;)

At least XviD should decode that correctly... looking forward to get your actual standalone decoder :) I'll put up a new installer with that if you don't mind!

Regards,
Koepi

[Nic]

Cool :) Ill make a rebuild tomorrow & have another look at the code & make sure my dshow is decoding the QPel ok...
(ill put up the source tomorrow too :) & send you a copy).

Cheers,
-Nic

[Koepi]

Oopsi, so your build isn't doing quarterpel yet? Well, at least sysKin's ME can be tested for bugs that way! :)
I'm looking forward to the new decoder!

Thanks for your great work Nic,

best regards,
Koepi

[iago]

-> Oopsi, so your build isn't doing quarterpel yet? Well, at least sysKin's ME can be tested for bugs that way! :) (Koepi)

It seems with my full 2-pass encode that's about to finish I am testing only MVHints at the moment. Anyway, better than nothing! :) Looking forward to trying the new build with Qpel, Nic ;).

thanks and best regards,
iago

[HarryM]

Originally posted by Koepi
Oopsi, so your build isn't doing quarterpel yet? Well, at least sysKin's ME can be tested for bugs that way! :)
I'm looking forward to the new decoder!

Thanks for your great work Nic,

best regards,
Koepi


@Nic:
@Koepi:

I tested (on Nic's latest binaries) b-frames too (without packet bitstream). For decoding I use ffdshow (xvid decoder cant decode b-frames yet?).

Xvid with b-frames (I use max two b-frames) are very nice for looking ;)
Video is 'slightly' crappy at edges compare to b-frameless, but filesize is about 30% smaller (at quant=2)!

I think, that xvid will be prepare for regular use of b-frames (soon?)... :)

[Teegedeck]

B-frames look very nice to me, also! (Apart from the decoding-problem. As soon as that's solved, I'd vote for making them 'regular', too.)

Well-done all!

[iago]

hello all,

I don't know if it is due to Hinted ME or due to a high average quantizer of ~4, but the resulting encode is really NOT much fun to watch, with an unsatisfactory overall quality and excessive blockiness on the faces, etc...
(2hr-1min / 576*320 / SimpleResize / h263 / motion search 6 / internal linear scaling, no special credits treatment)

best regards,
iago

[Koepi]

@iago:

Try the second pass without MVhints, maybe that helps it :)

My Shanghai Noon encoding looks beautiful so far, when decoded with xvid.dll (via vdub), Nic's "old" standalone decoder produces some minor texture glitches (minor colour mismatches).

Regards,
Koepi

[TheXung]

I'm just jumping with joy now. Quarter pixel is the one feature that I've been waiting for. Now I can happily revert back to using XviD.

From a little test clip I did at quant 3/MPEG, there was a 6% reduction in filesize and the 1/4 pel clip was sharper. The sharpness is comparable to the sharpness gained using 1/4 on divx. It's also nice how there isn't that large of a performance hit. However there does seem to be some discrepencies with the decoders. Divx doesn't decode it right, ffdshow almost has it right except for some issues with the UV channels not being motion compensated right. Nic's Xvid works fine. However, it doesn't decode the bframes.

There also seems to be an issue with using bframes and 1/4 pel at the moment. It doesn't seem to be able to use both at the same time. Turning 1/4 pel on or off produced the same file when bframes is on. Does anyone know if this just hasn't been implemented yet or did I do something wrong. I don't know how to read the mailing list now ever since they took the website down and since I wasn't subscribed to it. What is the state of bframes with curve compression?

[MaTTeR]

@Nic

Possibility to have ModHQ also implemented on your next build?;)

[-h]

Quarter-pel is most likely buggy at the moment, but shouldn't be for long. B-frame quarter-pel support is not yet present (whether encoding or decoding).

-h

[Koepi]

@-h:

when did your crashes with the code occur?
I could reproduce an error when using more than 680MBs/image (e.g. 512x384 resolution). Results in a crash on the first pframe.

Regards,
Koepi

[octapus]

---TheXung
-------------------------------------------------------------------
I don't know how to read the mailing list now ever since they took the website down and since I wasn't subscribed to it.
-------------------------------------------------------------------

Me too. Can anyone post an address where mailing list for xvid can be viewed. I am talking about the mailing list which was inside www.xvid.org (the site is down since 22 august)

[Nic]

The one I check on is:
http://list.xvid.org/pipermail/xvid-devel/
is that the one?

-Nic

ps
Just compiled a new version, qpel looks good, but because it crashes on hi-res stuff, im going to try & debug & fix that before releasing a new build. (QPel decodes fine though it seems :) )

pps
Hmmm, crash occurs on a:
transfer_16to8copy(pV_Cur, &data[5 * 64], stride2);
But I dont know why???

(Actually its a memory error thats causing the debugger to output cr*p so, however, the bug doesnt seem to be there if you encode with bframes not set to -1)

[TheXung]

(Actually its a memory error thats causing the debugger to output cr*p so, however, the bug doesnt seem to be there if you encode with bframes not set to -1)

If you encode with bframes set to something not -1, then it starts using bframes and not qpel, thus no bug crash occurs. or at least that's how it seemed to me because turning the qpel flags off and compiling/redoing exactly the same thing will make a file of the exact same filesize as with qpel turned on.

[Nic]

Oh didnt realise that, I gave up at that point (Im at work & already took the morning off to goto the dentist, so my lunch break was very short). Sounds like isibaar is going to patch image.c soon anyway so that should help matters.

-Nic