Bug report: Xvid quant 1+trellis is broken!

[Sharktooth]

Look here: http://forum.doom9.org/showthread.php?s=&threadid=84975
I tested the quant 1 behaviour with TRELLIS ENABLED with 1.02, Koepi 1.1beta (first and second build).
In every case there was blocking with some custom matrices.
I'm trying to estabilish what "kind" of matrices does trigger that bug.
For sure EQM V3HR triggers the bug:
http://ebola.gamersrevolt.it/xvid_quant1_bug.avi
http://ebola.gamersrevolt.it/xvid_quant1_bug_2.avi

Disabling Trellis quantization fixes the blocking.

[COREiP]

Is this a decoder bug?

[Prettz]

Well, at least it's just when quant 1 is used. Not a show-stopper bug.

[*.mp4 guy]

Actually its not just at quant 1... I just reviewed a couple of old test encodes with Didee's sixofnine hvs matrice and I noticed faint flickering dark(er) blocks in all instances. It was much more pronounced on light and or smooth areas like wals, however im not sure if its fixed by disdabling trellis, as its a feature i always use. can someone else confirm that its not just quant 1?

[sysKin]

...and I was hoping we fixed all those stupid overflows in trellis...

[Teegedeck]

...nothing like that on my end! :confused: You sure it ain't a decoder thang?

No flickering blocks, nor mysterious blocking in my encodes (mostly SixOfNine custom quant, Trellis of course).

@*.mp4 guy: How old are those 'old test encodes'?

[Sharktooth]

It's not a decoder issue (tested with DivX, Xvid, FFDShow, 3ivx decoders). Using the same settings but disabling Trellis fixes the blocking. That means Trellis has still some bugs (overflows as syskin wrote).

[Teegedeck]

You only found those glitches in quant=1, didn't you, Sharktooth? As I don't use quant=1 and have never encountered that bug, I assume that what *.mp4 guy discovered there probably doesn't really concern this build?

[Sharktooth]

Sorry Teegedek i thought you were talking to me in your previous post.

[*.mp4 guy]

The encodes I was refering to were made with the 1.1 beta for vhq on b-frames. Its probably a different bug(b-frame vhq perhaps) but I thought there was a chance the two could be related.

[kurt]

I did also some quantizer1_test_encodes, but i can't confirm the trellis bug ... i used one of the latest 1.1 builds from celtic_druid (xvid.cvs.head.gcc.p4.2004.10.18.7z) with 6of9_hvs-matrix, anamporph in mkv....

here are the samples:
http://home.arcor.de/evil.bert/Xvid%20Test/

settings: single pass, q1, trellis/no trellis, gmc, vhq 4, vhq for b-frames, b-frames 3/1,5/1, qpel, glosed gov, chroma...
note: teaser.mkv is a two-pass file, targeting by 1024 kb, same settings (trellis enabled)...

[yaz]

Originally posted by kurt
I did also some quantizer1_test_encodes, but i can't confirm the trellis bug ... i used one of the latest 1.1 builds from celtic_druid (xvid.cvs.head.gcc.p4.2004.10.18.7z)... the same here with an athlon build and with mpeg, h263 and the (good-old-)vhs series. it seems as if it were a matrix problem ...

just occured ... the same problem came up with mpeg2 encoding when i tested a series of cqm. trellis didn't like some matrices, definitely. (a typical example was the (in)famous notch-matrix from kvcd) iirc, the problematic matrices had quite strange 'profiles'. say, most of them had very steep parts a/o local minima/maxima a/o drops/shifts instead of smooth transients ... just thinking aloud

the bests
y

[Sharktooth]

A matrix can't be "problematic"... The codec handling of those matrices IS "problematic"...
If the codec chooses the wrong quantizers it's a codec bug, not a matrix problem.
In the xvid case it's a trellis overflow.

[yaz]

Originally posted by Sharktooth
A matrix can't be "problematic"... The codec handling of those matrices IS "problematic"...
If the codec chooses the wrong quantizers it's a codec bug, not a matrix problem.
In the xvid case it's a trellis overflow. "if u say so and if it's true i will believe it." sure, i will :-) however ...
if a codec works fine with 10 different matrices but the 11th triggers some trouble i've called that matrix problematic (and i haven't used). now i will call it "problematic" (and i wont use:-). is that ok ? :-)

anyway, i talked about mpeg2 encoding with mencoder, so, there's nothing to do with xvid here. and, maybe, the problem is more general than u prompt. maybe.

the bests
y

[Sharktooth]

Oh well, maybe also other codecs have bugs.
what i want to say is a CQM is composed by 2 parts: the intra-matrix and the inter-matrix.
If the coefficients of both matrices are between 8 and 255 there is nothing "problematic" with that CQM.
I used that matrix (EQM V3HR) with other codecs and there were no problems at all.

[Ark]

I too experienced some blockiness on some encodes.
I don't know if it's CQM-related or Trellis/B-VHQ-related though...
I've done 2 full test encodings of Matrix, with same XviD settings, using EQM_V3LR for one and EQM_V3ULR for the other.

Xvid settings was:

2-pass
AQ/GMC/B-Frames default values
MSP6/VHQ4/B-VHQ/Chroma Motion
Trellis
Respect VBV buffer unchecked

The ULR one showed some blockiness only on scenechanges (I-frames), while LR was perfect.

Just to have another example... :)

EDIT: the blockiness happens on ALL scenechanges with ULR, so P and B frames after every I-frame are affected by a side-effect, being these based on a bad I-frame...

[Sharktooth]

At what bitrate you encoded the movie?
Did you used some smoothing/denoising filters?

[Ark]

Originally posted by Sharktooth
At what bitrate you encoded the movie?
Did you used some smoothing/denoising filters?

Bitrate is +/- 800kb, I used:

mpeg2source("matrix.d2v")
crop(8,80,704,416)
temporalsoften(1,2,4,15,2)
simpleresize(640,272)
undot()

[Sharktooth]

Thanks. Can you also provide a small clip?

[Ark]

Sure, i'll post it this evening (i've the encodings on home PC)!

[Ark]

Ok, here's a small clip with 2 I-frames, both of which are "bad".

[Sharktooth]

Thanks. V3 ULR is going to be revisioned.
EDIT: Download the new matrix :)

[COREiP]

Is the trellis overflow bug being fixed?

[minolta]

^^re-up^^

Scanned through my videos with ffdshow OSD display (frame type, frame quant). It seems all "P-frame; Quant 1" have ugly 8x8 blocks as SharkTooth describes.
-minolta

p.s. Yes, custom-matrix "adr99?" and trellis enabled. Also, Q1 I-frames were not affected, just Q1 P-frames (same with SharkTooth's examples).

[skal]

Originally posted by COREiP
Is the trellis overflow bug being fixed?


Note: Last time i checked, Trellis was only used for Inter
blocks, IIRC.
So i'm a little dubious about the I-frame bug being
related to trellis quant.


Note2: there's a #define TL_SHIFT in mbtransquant.c
in XviD's sources that controls the dynamic range
of calcs. Try reducing it...


Bye!

Skal

[Sharktooth]

Disabling Trellis fix the blocking... so the bug is somewhere in trellis code...

[Peter Cheat]

Changing:


#define TL_SHIFT 11


to


#define TL_SHIFT 10


solves the problem! Thanks skal.

[Koepi]

Silent update on my site: reduced the range from 11 to 10 as well. (in the 1.1.-127 test build)

Please download and test if this helps your problem!

Regards
Koepi

[Sharktooth]

It seems it fixes the blocking with my matrix.
Now i'm testing with other matrices...

[COREiP]

Is the fix in XviD 1.0.2 updated as well?

[Koepi]

Originally posted by COREiP
Is the fix in XviD 1.0.2 updated as well?

Originally posted by Koepi
(in the 1.1.-127 test build)

Why should I add that comment if it was not true?

[celtic_druid]

Wouldn't be v1.0.2 if it was updated anyway.

[Omni]

tested it with 2 CM's and i have the same old blocking problem @quant=1.3+trellis and quant=1+trellis
First matrix was my own tuned for a one-pass encode.
Second one was 6 of 9 (valid).
you can try a "all 8" matrix for intra and inter frame (for tuning the trellis settings).
normal mpeg looks fine now.
your turn again :)

[Koepi]

Not really. :(

With "valid" matrices the problem seems to be solved.

The trellis calculation precision has a range of 10 to 16 - so I will not set it to 9. Time to think about the implicated problems of custom "not quite standard conform" matrices per se ;)

Cheers
Koepi

[Omni]

well, sounds at least reasonable :)

[skal]

Originally posted by Koepi
Not really. :(

With "valid" matrices the problem seems to be solved.

Koepi


Well, there might be another way of fixing the problem than
decreasing the calc precision: we could switch trellis off
for Quant=1 or 2, since it's most probably CPU wasting
(all the more that there are a lot of non-zero coeffs to
scrutinize).

But, before: is the blocking bug only present when Quant = 1 or 2??

Skal

[Koepi]

The error is seen only at quant=1 and with extreme matrices if i'm not mistaken.

Regards
Koepi

[Omni]

right, mostly matrices which use values smaller than 12 or something like that (haven't tried it yet but at least every inter-matrix which has 8's in the upper left part fails definitely, at least in my case)

[skal]

Originally posted by Omni
right, mostly matrices which use values smaller than 12 or something like that (haven't tried it yet but at least every inter-matrix which has 8's in the upper left part fails definitely, at least in my case)


All right, so could someone change the source and
only call trellis for quant>2, e.g?

Namely, line 219 of mbtransquant.c should consist of:

if(sum && pMB->quant>2 && (frame->vop_flags & XVID_VOP_TRELLISQUANT))
{
...
}



Is there a quality impact? Is the bug gone?
Syskin are you there? ;)

Skal

[celtic_druid]

If anyone wants to test the above change:
http://celticdruid.no-ip.com/test/xvidcore.7z

Other than that it is a vanilla ICL7.1 cvs head compile.

[skal]

Originally posted by celtic_druid
If anyone wants to test the above change:
http://celticdruid.no-ip.com/test/xvidcore.7z

Other than that it is a vanilla ICL7.1 cvs head compile.

thanks very much for the build, Celtic-Druid

[Omni]

i'll test it later but if it works it's a good thing :)

[Koepi]

Since the error seems to only occur in q1 frames I'd say it should test for mbquant >1 (or >=2). The precision can be reset to #TL_SHIFT 11 then as well.

Regards
Koepi

[celtic_druid]

TL_SHIFT was already set to 11 in the last build.
How about something like TL_SHIFT is set to 10 for Q1 and 11 for Q2 and up? What about higher values for higher quants?

Almost forgot. Test build #2 (Q>=2)
http://celticdruid.no-ip.com/test/xvidcore2.7z

[Omni]

well works great but only made a short test.
for me it's really usefull to be able to use special CM's with q<1 since i sometimes have material (dark noisy videos) where it's impossible to kill al the noise and where without quants<2 the video gets blocky in those dark areas. Now i can use the override for those sections and set a quant of 1.5 which really improves the picture and eats up a lot of space ;) but the quality of the rest-video was more than maxed out with my CM so i don't mind the waste which q<2 will cause.

is there a way this "quickfix" will be static for future releases?
well now i'm going to encode some files again now :) thanks

oh and one idea^^ is it possible to implement a switch or some weightning for dark frames/scenes?
something like this:
frames lower than an average brightness 0-100 (%) get a bitrate bonus of x% (or average quants for the dark areas are x% smaller than the overall average quant).
could be really helpfull with dark and bright material. normally the bright areas are looking more than good but the dark ones are blocky.
a feature like this would help me a lot on with my encodes, too. otherwise i've to override those sections by hand :)

[IRONHiDE]

Sharktooth, your samples are not exists anymore. Could you answer please is the bug you sad like these?
http://i29.fastpic.ru/big/2011/1023/54/ff1ec0e2110ea0743c46d3b5f99f0454.png
http://i28.fastpic.ru/big/2011/1023/2f/859870007a92d22b736e5e973d56532f.png