Transcoding Xvid to Xvid : issue, perhaps with IDCT or custom quant

[Manao]

In fact, I'm not sure if IDCT is problem, but perhaps you will know what happens...

I've been trying to reencode an Xvid movie ( to put it on 1 cd ), using the latest koepi's build. The source may have been encoded with Xvid simple profile (04/10/2002, but it's a guess ), and I used 'classical' settings for xvid ( bframes, chroma motion & optimizer, VHQ 4, hvs-good-matrix but no qpel ). The result was not as good as expected, because of huge smearing when there was a lot of motion ( see the first scrrenshot (http://jourdan.madism.org/~manao/xvid-smearing.jpg) ).

The compressibility of the movie was not in cause, since I got the same artifacts when encoding at quant 2 with only chroma motion and hvs-good-matrix. So I thought it was the cause of the change of the IDTC routine in the last build.

So I tried again to encode a portion where the problem was visible, this time not using avisynth and with full processing mode in vdubmod 1.5.1.a, in order to have the preview activated. And I was surprised to see the artifacts in the two windows.

When I tried a third time with divx3, there was no problem ( screenshot (http://jourdan.madism.org/~manao/xvid-divx3.jpg) )

Finally, I tried with h263 instead of hvs-good-matrix, the problem was gone.

So I don't know what is to blame. IDCT ? HVS-good-matrix ? I never experienced any problem with this matrix, that's why I didn't think it could be the problem at the beginning, and that's why even if all since to accuse it, I still mention the change in IDCT.

[DoW]

Reencoding something that has already been encoded will always result in artifacts. I really dont see a solution to this besides finding the DVD and doing it from there.

[JohnMK]

Encoding something to a compressed format results in, let's say, 20% quality loss. So you're left with (I'll just pull some number randomly out of my head) 80% of the original quality. If you again encode (the result this time, not the original source), you're doing 80% * 80% = 64%. This is known as transcoding and DoW correctly points out it's a big no-no. :)

[Manao]

I know that reencoding can result in a visual loss, but if you look at the screenshots, you will see that it's not the problem in this case. The source was very clean ( 1h55, low action movie, in two cds ) and was a very good candidate to be compressed in one cd. I have already done such an operation several times, and it's the first time I saw such artifacts.

I have reencoded the whole movie with H263 matrix and the result is fair. But I wonder why the hvs-good-matrix gave these huge artifacts, even at quant 2.

[sysKin]

Is there a way you could tell me if the original file was encoded with mpeg quant (+custom, maybe?). I can see a possibility that two codec instancies might have problems sharing quntization matrix.

Also, could you please try (just for tests):
- reading the original file with avisynth' directshowsource(), with ffdshow used as dshow decoder (you might even want to do that, because postprocessing can be useful for reencoding)
- reading the original file using divx5 decoder, after fourcc change.

If the two above help, there is a possibility you have found a bug.

BTW, a tip: if you resize the file you've read, you almost avoid reencoding problems - macroblocks no longer lie in the same places, this should help.

Radek

[Manao]

I tried what you said :

* No problem with DirectShowSource + ffdshow + hvsgood
* No problem with DivX 5 + hvsgood
+ Problem with Xvid + hvsgood ( I did the test again since I have an encoding process with Xvid + H263 running in parallel )
BTW, a tip: if you resize the file you've read, you almost avoid reencoding problems - macroblocks no longer lie in the same places, this should help. Thanks for the tip

Edit : I forgot, unless the matrix is written in the file, I can't know the quant matrix used.

[sysKin]

Alrighty, thank you Manao!

I'll look into it tommorow - there is probably something made static in the code (quant matrix itself, probably). I'm not an expert in this part (nor any other part, it seems ;p) so I'll learn something new :)

Regards,
Radek

[Didée]

Ah, one question:

How can the quantization matrix that was used for encoding be extracted again out of the final bitstream?
I remember, time ago, I peeked around in an avi file, and found something that looked similar to matrix values, but not exactly ... it was not even 128 Bytes.
Is the matrix written in some compressed manner?

- Didée

[sysKin]

Originally posted by Didée
How can the quantization matrix that was used for encoding be extracted again out of the final bitstream?
I remember, time ago, I peeked around in an avi file, and found something that looked similar to matrix values, but not exactly ... it was not even 128 Bytes.
Is the matrix written in some compressed manner?Yes it is. The easiest way to extract it would be to insert a small code to xvid which would print it to a text file for you (when decoding a keyframe, for example. Or anywhere else.)

If you want to read it yourself, consult bitstream.c::bs_get_matrix().

Actually I just looked at it - it's very very easy: 8-bit values in zigzag order, with value 0 meaning that all remaining values are equal to the last coded value. The only difficulty would be that values don't start byte-aligned, so you have to decode entire VOL header to find out where the matrix begins.

Regards,
Radek

[Selur]

If it's not that hard to do, could someone plz write such an app that get a file as input and gives a txt with the matrix vales as output? :D

From time to time I'm testing custom matrices, trying to optimize them for a specific source and sometimes I lose or forget to save all the different matrices I tested :( , so it would be helpful to get the custom matrices back. :)

Cu Selur