Arbitrariness in XviD

[Alice]

Hello to everyone,

First let me start with a complement on XviD codec to all developers, I used to use DivX to compress, but ever since I discovered XviD, I hardly ever use any other codec, your work guys is tremendous!

However, there is a little problem I have encountered, a fairly bothering one that I, for lack of a better term, called 'arbitrariness' in the codec. I know this is not the first thread tackling this issue:rolleyes: , but I will try to be sort of original.

I've been trying to compress a series of approximately 450 MB-sized Mpeg files, using 2-pass compression with Koepi's 24062003 version of the codec, converting audio to 128 kb/sec mp3, aiming at 350 MB files, using VirtualDub. I used H.263 quantization type, and ultra high (6)motion search precision.

However, the resulting xvid avi is never bigger than 270 MB, no matter what value I give for 'desired file size' in the 2nd pass. Originally, I started setting it to 350 MB, but I also get the same result with any value that's bigger than 350. I tried using other features, like quarterpel and global motion compensation, which, as expected, only made the size even smaller. (I even turned off VHQ to try to make the size bigger.)

Eventually, it seems that the codec uses the 'desired file size' value only as 'largest allowed file size', and compresses the file as well as it can, but if I allow extra megabites to 'spend' on bitrate it refuses to use them.:confused: :confused:
I even tested this (though with smaller file sizes on my friend's computer (similarly Win2000, same VDub version, different version of the codec), and we got the same problem occurring). I read in a current thread that smaller (few MBs) differences are possible, but 80 MB is quite a lot, especially if it is apparent from the result that the compression SHOULD use the extra MBs to make the avi look better and there really IS space to make use of.

I've read through the xvid guides, searched the forum, but did not quite get answer to this question. Is there any known reason for this phenomenon? Is it 'normal' with the codec? I'm not a dvd-rip expert, so I don't know if this is known 'characteristic' of the codec. Could somebody please help me and give some advice...?

Thank you in advance,
Alice

[KpeX]

This is quite normal, it sounds like what has happened is that you've maxed out the codec. Especially considering you're transcoding 450 mb mpeg-1 files, this is not surprising. Xvid is much more efficient than mpeg-1 for video compression, and for your two-pass encode you've 'maxed out' xvid, which means you have encoded with the highest quality possible. So why doens't it look perfect? Because your source is imperfect, you can't improve quality by reencoding, you'll actually lose quality. Considering you're only compressing 100 mb, you're much better off IMO keeping the original mpeg files. One way to be sure that this is what's happening is to do a full first pass encode and see what size that comes to. If it's the same as second pass, you've maxed out the encode.

[Assault]

Read this thread (http://forum.doom9.org/showthread.php?s=&threadid=24584) for an explanation and for ways to solve your problem. Although it was originally written for DivX 5 it also applies to XviD. (But I think KpeX is right. You shouldn't transcode an 450mb mpeg1 file to an 350mb mpeg4 one.) ;)

Assault

[Alice]

Thanks for the replies, now I see what causes the problem... Strange that DivX doesn't seem to get 'maxed out'... Otherwise, yes, I know that quality can only decrease compared to the original source during compression, all i hoped that the codec would make use of the extra Mb's and 2 compressed files would fill a cd... Anyways, thank you guys for the help and explanation, and also for the link, I think I'll keep the original files uncompressed.. :)

Alice

[Manao]

Kpex and Assault : 450 MB, you put one file per CD, 350 you put 2 files :). Plus, the loss in quality is almost invisible ( even when we reach 4 eps / CDs, depending on the compressibility, of course )

Alice : why not try to put 3 eps / CDs instead ? Except if the visual quality doesn't please you ( which I doubt ), you shouldn't have any problems.

[Alice]

Manao: Thanks for the comment:) , actually, after weeks of experimenting with both DivX and XviD, and with all the various features of both, I tend to believe that XviD is awesome for compressing video ripped from dvd, but not so great at compressing simple mpeg.:confused: (Not that I believe that any codec is better at it, for that matter...) I'm led to believe this, since I cannot really tell the difference between a dvd and a good 2 cd XviD, but I can differentiate my source and compressed 450mb mpeg videos, even though only when closely observing the details.

As you suggested it would be even better to burn 3 eps on 1cd, but then the cd has to be of 800Mb capacity (and even that is only sufficient with overburn), since I wouldn'like to make the files even smaller than 270Mb, which, as it seems maxes out the codec. (This fact, to be honest, is still something I don't fully understand: why can't I force XviD to actually USE those extra Mb's on the smallest details?)

[Soulhunter]

Have you also tryed MPEG quantization instead of H.263 ???

Should give more file size and more details...

Or is "approximately" (I don't know it) anime stuff ?

AFAIK its better than to stay with H.263...

Think Ive heard MPEG quant. looks bad with anime/cartoon stuff...

But have never found the time to test it ! :p

Correct me If I'm wrong ! ;)

PS: Still planed to capture all my DBZ eps. ! Damn I need more time... :D

Bye

[Alice]

Yep, already tried MPEG quantization, got a result that's some 300MB (that is still 50 Mb less than what I was aiming at: same problem there, it won't make a larger file size, codec again maxed out, i reckon). Furthermore, MPEG quantization, though gives somewhat bigger file size, gives much uglier picture in this case. (Maybe I should be using some extra feature of the codec there, or filter...?)
And no, it's not anime, god save us from that:D , it's the Brisco County Jr. series I'm downloading...(i.e. normal movie, not a cartoon) (I wonder who had the great idea to create 450MB files for 45 minutes videos...)
But thanks anyway!:)

[Assault]

You could try some custom matrices like HVS-best or andreas_78er. Then your file should get larger than 300MB.

Assault

[MfA]

If you want max size/quality in xvid use MPEG quantization with quantizer matrices consisting of all 8's and fixed quant 1, if that doesnt get you over 450 MB nothing will.

I would be very surprised if it didnt though, this basically disables quantization altogether (for that to be true for b-frames too you need to use a quantizer ratio of 1x and 0 offset of course).

[Alice]

I again appreciate your help guys, and thank you for taking the time and writing me your suggestions...:)

Assault: could you please perhaps give me a link where I could get further info about custom quant matrices, and download the ones you suggested, I didn't even know such things exist...:(
Do they only increase size, or also produce better results than normal H.263 in some respects? (Or is this a question that gave away my lack of insight to quant.types?:rolleyes: )

If you want max size/quality in xvid use MPEG quantization with quantizer matrices consisting of all 8's and fixed quant 1, if that doesnt get you over 450 MB nothing will.

MfA, I'm sorry, but I'm not such an expert at XviD...:scared: What does this exactly mean? (All 8's and fixed quant 1?) Where exactly can I set these things in the codec and what will they do differently?

[MfA]

Choose mpeg-custom as quantizer type, then edit the matrices and set every number to 8. Choose min/max quantizer of 1 for all frametypes (dont use trellis) in advanced options. Encode away and watch that file grow.

[Assault]

Nitrogen posted a collection of quantization matrices in this thread (http://forum.doom9.org/showthread.php?threadid=34493&perpage=20&pagenumber=2).
And yes, both the size AND the quality of your video should be bigger/better(of course). ;)

Assault

[Alice]

So, I have tried out different suggestions, and got the following results compressing my 450Mb Mpeg files to Xvid (Koepi's 24062003 version) using CUSTOM MPEG compression method (2 pass encoding), with various custom matrices:

1. Hvs-best: the file size is 320 Mb, and it's lower quality than my standard H.263 compression, which is 270 Mb.

2. Hvs-better: size about 270 Mb, and actually 'best' is better...

3. Bulletproof's High Quality Matrix: the file is only somewhat bigger than 270 Mb's, and it's a definitely not-so-high-quality one.


I have also tried out MfA's suggestion, custom matrices, all numbers set to 8, and all frames (min\max) to 1 and I watched the file grow, and it actually DID grow: 90 minutes later from my 450 Mb Mpeg-1 file I got a 2GB Mpeg-4 one. So, unfortunately this method is out of the question... (Even though it TRULY produced a bigger file.):)

The real problem is the obvious: especially compared to a dvd-rip, this mpeg, though is quite ok, it has some noise, which in the compressed file appears as a bunch of pulsating blurred squares...

I'm still trying out some other matrices, and should perhaps mess around with VHQ and/or quarterpel, but I'm gradually losing hope that anything can be done to improve these results...

[Manao]

I see no reason why you should not use VHQ 4. Unless you are short of time (which is obviously not the case), VHQ is a must. Of course, it may reduce slightly the size, but it will improve the quality.

Then, you seem to find that your source is slightly noisy. In that case, use a denoiser ( Convolution3D for example, with the setting preset="MovieHQ" if you want a soft effect ) Then again, your size will be reduced, but I would bet you'll find the quality better.

Finally, just make the following test. Configure XviD with VHQ 4, motion estimation 6, chroma motion and B-frame 2,150,75,10. Use either H263 or HVS-Good matrix. Use also avisynth in order to use convolution3d and postprocessing ( BlinPP=cpu2"xxxxoo" ). Make a first pass without discarding the result. The size should be around 200 MB. Now, compare the quality with what you obtained previously. The quality should be lower, it's normal, but do you really see the difference ? And then, you can put 3 ( even 4 ) eps / CD's.

And by the way, with your previous settings, the max size you obtained was 270 Mb, that means you could make a second pass, aiming at 230 Mb (or even less), without seeing a difference.

Please consider these options. I did a lot of transcoding MPEG-1 -> XviD, and I never put less than 3 eps / CD's.

[Koepi]

MPEG quantization type keeps the noise - but introduces ringing on oversharped sources. Use Unfilter(-10,-10) to reduce the sharpening side effects, this will again reduce your file size, but the result will be much more pleasant.

Koepi

[Alice]

I am deeply indebted for the detailed suggestions Manao, and it is an honor to get help from the great Koepi himself.

Forgive me for getting a little off-topic, but I have to say: despite all my difficuties (which I know are not the most 'elite' ones in Doom9 XviD forum history) I can't help falling in love with both XviD and the present community: it is simply the best and most helpful technical forum I have ever been to, and I stand amazed!

Since I have not yet been using Gknot, but only (unmodified) VDub for compression, and I realize that the suggestions refer to GKnot filters and options, I will need to have some time to get acquainted with the program, and test your valuable proposals. This I will do in the next few days, and will report back what final results I've got.

Furthermore, as it seems, I will need to re-adjust some of my older settings as well: I have had VHQ turned off so far:rolleyes: , because I read that its main function is to decrease size, which, in this case I didn't need.


Thank you once more until then,:)
Alice

[cult]

I think that the suggestions refer to avisynth.Gnot uses avisynth yes,but if you want to use alot of filters then use avisynth,For example BlinPP=cpu2"xxxxoo" cant be produced by gknot,only editing the avs.Also try to use vdmod or the newest vd which works with yv12.You would get better quality and speed