Bug report: Xvid quant 1+trellis is broken!

Sharktooth · 6th November 2004, 15:57

Look here: http://forum.doom9.org/showthread.php?s=&threadid=84975
I tested the quant 1 behaviour with TRELLIS ENABLED with 1.02, Koepi 1.1beta (first and second build).
In every case there was blocking with some custom matrices.
I'm trying to estabilish what "kind" of matrices does trigger that bug.
For sure EQM V3HR triggers the bug:
http://ebola.gamersrevolt.it/xvid_quant1_bug.avi
http://ebola.gamersrevolt.it/xvid_quant1_bug_2.avi

Disabling Trellis quantization fixes the blocking.

COREiP · 6th November 2004, 21:31

Is this a decoder bug?

Prettz · 7th November 2004, 19:41

Well, at least it's just when quant 1 is used. Not a show-stopper bug.

*.mp4 guy · 8th November 2004, 10:01

Actually its not just at quant 1... I just reviewed a couple of old test encodes with Didee's sixofnine hvs matrice and I noticed faint flickering dark(er) blocks in all instances. It was much more pronounced on light and or smooth areas like wals, however im not sure if its fixed by disdabling trellis, as its a feature i always use. can someone else confirm that its not just quant 1?

sysKin · 8th November 2004, 11:10

...and I was hoping we fixed all those stupid overflows in trellis...

Teegedeck · 8th November 2004, 14:03

...nothing like that on my end!

You sure it ain't a decoder thang?

No flickering blocks, nor mysterious blocking in my encodes (mostly SixOfNine custom quant, Trellis of course).

@*.mp4 guy: How old are those 'old test encodes'?

Sharktooth · 8th November 2004, 14:25

It's not a decoder issue (tested with DivX, Xvid, FFDShow, 3ivx decoders). Using the same settings but disabling Trellis fixes the blocking. That means Trellis has still some bugs (overflows as syskin wrote).

Teegedeck · 8th November 2004, 15:20

You only found those glitches in quant=1, didn't you, Sharktooth? As I don't use quant=1 and have never encountered that bug, I assume that what *.mp4 guy discovered there probably doesn't really concern this build?

Sharktooth · 8th November 2004, 15:27

Sorry Teegedek i thought you were talking to me in your previous post.

*.mp4 guy · 9th November 2004, 04:53

The encodes I was refering to were made with the 1.1 beta for vhq on b-frames. Its probably a different bug(b-frame vhq perhaps) but I thought there was a chance the two could be related.

kurt · 11th November 2004, 11:12

I did also some quantizer1_test_encodes, but i can't confirm the trellis bug ... i used one of the latest 1.1 builds from celtic_druid (xvid.cvs.head.gcc.p4.2004.10.18.7z) with 6of9_hvs-matrix, anamporph in mkv....

here are the samples:
http://home.arcor.de/evil.bert/Xvid%20Test/

settings: single pass, q1, trellis/no trellis, gmc, vhq 4, vhq for b-frames, b-frames 3/1,5/1, qpel, glosed gov, chroma...
note: teaser.mkv is a two-pass file, targeting by 1024 kb, same settings (trellis enabled)...

yaz · 11th November 2004, 12:36

Quote:

Originally posted by kurt
I did also some quantizer1_test_encodes, but i can't confirm the trellis bug ... i used one of the latest 1.1 builds from celtic_druid (xvid.cvs.head.gcc.p4.2004.10.18.7z)...

the same here with an athlon build and with mpeg, h263 and the (good-old-)vhs series. it seems as if it were a matrix problem ...

just occured ... the same problem came up with mpeg2 encoding when i tested a series of cqm. trellis didn't like some matrices, definitely. (a typical example was the (in)famous notch-matrix from kvcd) iirc, the problematic matrices had quite strange 'profiles'. say, most of them had very steep parts a/o local minima/maxima a/o drops/shifts instead of smooth transients ... just thinking aloud

the bests
y

Sharktooth · 11th November 2004, 14:40

A matrix can't be "problematic"... The codec handling of those matrices IS "problematic"...
If the codec chooses the wrong quantizers it's a codec bug, not a matrix problem.
In the xvid case it's a trellis overflow.

yaz · 12th November 2004, 12:11

Quote:

Originally posted by Sharktooth
A matrix can't be "problematic"... The codec handling of those matrices IS "problematic"...
If the codec chooses the wrong quantizers it's a codec bug, not a matrix problem.
In the xvid case it's a trellis overflow.

"if u say so and if it's true i will believe it." sure, i will :-) however ...
if a codec works fine with 10 different matrices but the 11th triggers some trouble i've called that matrix problematic (and i haven't used). now i will call it "problematic" (and i wont use:-). is that ok ? :-)

anyway, i talked about mpeg2 encoding with mencoder, so, there's nothing to do with xvid here. and, maybe, the problem is more general than u prompt. maybe.

the bests
y

Sharktooth · 12th November 2004, 13:14

Oh well, maybe also other codecs have bugs.
what i want to say is a CQM is composed by 2 parts: the intra-matrix and the inter-matrix.
If the coefficients of both matrices are between 8 and 255 there is nothing "problematic" with that CQM.
I used that matrix (EQM V3HR) with other codecs and there were no problems at all.

Ark · 16th November 2004, 20:42

I too experienced some blockiness on some encodes.
I don't know if it's CQM-related or Trellis/B-VHQ-related though...
I've done 2 full test encodings of Matrix, with same XviD settings, using EQM_V3LR for one and EQM_V3ULR for the other.

Xvid settings was:

2-pass
AQ/GMC/B-Frames default values
MSP6/VHQ4/B-VHQ/Chroma Motion
Trellis
Respect VBV buffer unchecked

The ULR one showed some blockiness only on scenechanges (I-frames), while LR was perfect.

Just to have another example...

EDIT: the blockiness happens on ALL scenechanges with ULR, so P and B frames after every I-frame are affected by a side-effect, being these based on a bad I-frame...

Sharktooth · 17th November 2004, 03:47

At what bitrate you encoded the movie?
Did you used some smoothing/denoising filters?

Ark · 17th November 2004, 09:27

Quote:

Originally posted by Sharktooth
At what bitrate you encoded the movie?
Did you used some smoothing/denoising filters?

Bitrate is +/- 800kb, I used:

mpeg2source("matrix.d2v")
crop(8,80,704,416)
temporalsoften(1,2,4,15,2)
simpleresize(640,272)
undot()

Sharktooth · 17th November 2004, 13:04

Thanks. Can you also provide a small clip?

Ark · 17th November 2004, 13:08

Sure, i'll post it this evening (i've the encodings on home PC)!

6th November 2004, 15:57	#1 \| Link
Sharktooth Mr. Sandman Join Date: Sep 2003 Location: Haddonfield, IL Posts: 11,768	Bug report: Xvid quant 1+trellis is broken! Look here: http://forum.doom9.org/showthread.php?s=&threadid=84975 I tested the quant 1 behaviour with TRELLIS ENABLED with 1.02, Koepi 1.1beta (first and second build). In every case there was blocking with some custom matrices. I'm trying to estabilish what "kind" of matrices does trigger that bug. For sure EQM V3HR triggers the bug: http://ebola.gamersrevolt.it/xvid_quant1_bug.avi http://ebola.gamersrevolt.it/xvid_quant1_bug_2.avi Disabling Trellis quantization fixes the blocking. __________________ MPEG-4 ASP Custom Matrices: EQM V1(old), EQM AutoGK Sharpmatrix (aka EQM V2), EQM V3HR (updated 01/10/2004), EQM V3LR, EQM V3ULR (updated 04/02/2005), EQM V3UHR (updated 17/12/2004) and EQM V3EHR (updated 05/10/2004) Info about my ASP matrices. MPEG-4 AVC Custom Matrices: EQM AVC-HR Info about my AVC matrices My x264 builds. Mooo!!! Last edited by Sharktooth; 6th November 2004 at 16:08.

8th November 2004, 11:10	#5 \| Link
sysKin Registered User Join Date: Jun 2002 Location: Adelaide, Australia Posts: 1,167	...and I was hoping we fixed all those stupid overflows in trellis... __________________ Visit #xvid or #x264 at irc.freenode.net

8th November 2004, 14:03	#6 \| Link
Teegedeck Moderator, Ex(viD)-Mascot Join Date: Oct 2001 Posts: 2,564	...nothing like that on my end! You sure it ain't a decoder thang? No flickering blocks, nor mysterious blocking in my encodes (mostly SixOfNine custom quant, Trellis of course). @.mp4 guy: How old are those 'old test encodes'? __________________ It's a man's life in Doom9's 52nd MPEG division. "The cat sat on the mat." ATM I'm thoroughly enjoying the Banshee - a fantastic music player/ripper for Linux. Give it a whirl! Last edited by Teegedeck; 8th November 2004 at 14:05.*

8th November 2004, 14:25	#7 \| Link
Sharktooth Mr. Sandman Join Date: Sep 2003 Location: Haddonfield, IL Posts: 11,768	It's not a decoder issue (tested with DivX, Xvid, FFDShow, 3ivx decoders). Using the same settings but disabling Trellis fixes the blocking. That means Trellis has still some bugs (overflows as syskin wrote). __________________ MPEG-4 ASP Custom Matrices: EQM V1(old), EQM AutoGK Sharpmatrix (aka EQM V2), EQM V3HR (updated 01/10/2004), EQM V3LR, EQM V3ULR (updated 04/02/2005), EQM V3UHR (updated 17/12/2004) and EQM V3EHR (updated 05/10/2004) Info about my ASP matrices. MPEG-4 AVC Custom Matrices: EQM AVC-HR Info about my AVC matrices My x264 builds. Mooo!!!

8th November 2004, 15:20	#8 \| Link
Teegedeck Moderator, Ex(viD)-Mascot Join Date: Oct 2001 Posts: 2,564	You only found those glitches in quant=1, didn't you, Sharktooth? As I don't use quant=1 and have never encountered that bug, I assume that what *.mp4 guy discovered there probably doesn't really concern this build? __________________ It's a man's life in Doom9's 52nd MPEG division. "The cat sat on the mat." ATM I'm thoroughly enjoying the Banshee - a fantastic music player/ripper for Linux. Give it a whirl!

6th November 2004, 21:31	#2 \| Link
COREiP Registered User Join Date: Oct 2004 Posts: 65	Is this a decoder bug?

7th November 2004, 19:41	#3 \| Link
Prettz easily bamboozled user Join Date: Sep 2002 Location: Atlanta Posts: 373	Well, at least it's just when quant 1 is used. Not a show-stopper bug.

8th November 2004, 10:01	#4 \| Link
*.mp4 guy Registered User Join Date: Feb 2004 Posts: 1,348	Actually its not just at quant 1... I just reviewed a couple of old test encodes with Didee's sixofnine hvs matrice and I noticed faint flickering dark(er) blocks in all instances. It was much more pronounced on light and or smooth areas like wals, however im not sure if its fixed by disdabling trellis, as its a feature i always use. can someone else confirm that its not just quant 1?

8th November 2004, 15:27	#9 \| Link
Sharktooth Mr. Sandman Join Date: Sep 2003 Location: Haddonfield, IL Posts: 11,768	Sorry Teegedek i thought you were talking to me in your previous post. __________________ MPEG-4 ASP Custom Matrices: EQM V1(old), EQM AutoGK Sharpmatrix (aka EQM V2), EQM V3HR (updated 01/10/2004), EQM V3LR, EQM V3ULR (updated 04/02/2005), EQM V3UHR (updated 17/12/2004) and EQM V3EHR (updated 05/10/2004) Info about my ASP matrices. MPEG-4 AVC Custom Matrices: EQM AVC-HR Info about my AVC matrices My x264 builds. Mooo!!!

9th November 2004, 04:53	#10 \| Link
*.mp4 guy Registered User Join Date: Feb 2004 Posts: 1,348	The encodes I was refering to were made with the 1.1 beta for vhq on b-frames. Its probably a different bug(b-frame vhq perhaps) but I thought there was a chance the two could be related.

11th November 2004, 11:12	#11 \| Link
kurt sidekick Join Date: Apr 2004 Location: old Europe Posts: 610	I did also some quantizer1_test_encodes, but i can't confirm the trellis bug ... i used one of the latest 1.1 builds from celtic_druid (xvid.cvs.head.gcc.p4.2004.10.18.7z) with 6of9_hvs-matrix, anamporph in mkv.... here are the samples: http://home.arcor.de/evil.bert/Xvid%20Test/ settings: single pass, q1, trellis/no trellis, gmc, vhq 4, vhq for b-frames, b-frames 3/1,5/1, qpel, glosed gov, chroma... note: teaser.mkv is a two-pass file, targeting by 1024 kb, same settings (trellis enabled)... __________________ greets, kurt. Pioneer PDP-427 XA \| Popcorn Hour NMT C-200 \| Sony STR-DB 840 QS \| Canton Ergo 91 DC Last edited by kurt; 11th November 2004 at 11:15.

11th November 2004, 14:40	#13 \| Link
Sharktooth Mr. Sandman Join Date: Sep 2003 Location: Haddonfield, IL Posts: 11,768	A matrix can't be "problematic"... The codec handling of those matrices IS "problematic"... If the codec chooses the wrong quantizers it's a codec bug, not a matrix problem. In the xvid case it's a trellis overflow. __________________ MPEG-4 ASP Custom Matrices: EQM V1(old), EQM AutoGK Sharpmatrix (aka EQM V2), EQM V3HR (updated 01/10/2004), EQM V3LR, EQM V3ULR (updated 04/02/2005), EQM V3UHR (updated 17/12/2004) and EQM V3EHR (updated 05/10/2004) Info about my ASP matrices. MPEG-4 AVC Custom Matrices: EQM AVC-HR Info about my AVC matrices My x264 builds. Mooo!!!

12th November 2004, 13:14	#15 \| Link
Sharktooth Mr. Sandman Join Date: Sep 2003 Location: Haddonfield, IL Posts: 11,768	Oh well, maybe also other codecs have bugs. what i want to say is a CQM is composed by 2 parts: the intra-matrix and the inter-matrix. If the coefficients of both matrices are between 8 and 255 there is nothing "problematic" with that CQM. I used that matrix (EQM V3HR) with other codecs and there were no problems at all. __________________ MPEG-4 ASP Custom Matrices: EQM V1(old), EQM AutoGK Sharpmatrix (aka EQM V2), EQM V3HR (updated 01/10/2004), EQM V3LR, EQM V3ULR (updated 04/02/2005), EQM V3UHR (updated 17/12/2004) and EQM V3EHR (updated 05/10/2004) Info about my ASP matrices. MPEG-4 AVC Custom Matrices: EQM AVC-HR Info about my AVC matrices My x264 builds. Mooo!!!

16th November 2004, 20:42	#16 \| Link
Ark Aikidoka Join Date: Nov 2003 Location: Italy Posts: 216	I too experienced some blockiness on some encodes. I don't know if it's CQM-related or Trellis/B-VHQ-related though... I've done 2 full test encodings of Matrix, with same XviD settings, using EQM_V3LR for one and EQM_V3ULR for the other. Xvid settings was: 2-pass AQ/GMC/B-Frames default values MSP6/VHQ4/B-VHQ/Chroma Motion Trellis Respect VBV buffer unchecked The ULR one showed some blockiness only on scenechanges (I-frames), while LR was perfect. Just to have another example... EDIT: the blockiness happens on ALL scenechanges with ULR, so P and B frames after every I-frame are affected by a side-effect, being these based on a bad I-frame... Attached Images __________________ Xvid fan ;) Last edited by Ark; 16th November 2004 at 21:32.

17th November 2004, 03:47	#17 \| Link
Sharktooth Mr. Sandman Join Date: Sep 2003 Location: Haddonfield, IL Posts: 11,768	At what bitrate you encoded the movie? Did you used some smoothing/denoising filters? __________________ MPEG-4 ASP Custom Matrices: EQM V1(old), EQM AutoGK Sharpmatrix (aka EQM V2), EQM V3HR (updated 01/10/2004), EQM V3LR, EQM V3ULR (updated 04/02/2005), EQM V3UHR (updated 17/12/2004) and EQM V3EHR (updated 05/10/2004) Info about my ASP matrices. MPEG-4 AVC Custom Matrices: EQM AVC-HR Info about my AVC matrices My x264 builds. Mooo!!!

17th November 2004, 13:04	#19 \| Link
Sharktooth Mr. Sandman Join Date: Sep 2003 Location: Haddonfield, IL Posts: 11,768	Thanks. Can you also provide a small clip? __________________ MPEG-4 ASP Custom Matrices: EQM V1(old), EQM AutoGK Sharpmatrix (aka EQM V2), EQM V3HR (updated 01/10/2004), EQM V3LR, EQM V3ULR (updated 04/02/2005), EQM V3UHR (updated 17/12/2004) and EQM V3EHR (updated 05/10/2004) Info about my ASP matrices. MPEG-4 AVC Custom Matrices: EQM AVC-HR Info about my AVC matrices My x264 builds. Mooo!!!

17th November 2004, 13:08	#20 \| Link
Ark Aikidoka Join Date: Nov 2003 Location: Italy Posts: 216	Sure, i'll post it this evening (i've the encodings on home PC)! __________________ Xvid fan ;)