Xvid and multipass encoding...?

[trodas]

Well, this is relatively simple question, probably going directly to Koepi, the developer of Xvid codec :D (salute for great work, Koepi ;) )

Well, and now, as the subject telling us, why Xvid does not support ATM multiple passes encoding, to get the best possible results for desired bitrate :confused:

And if this is planed to be added into future...? :rolleyes:

I believe it could come handy.

...also i noticed that Xvid somewhat ineffectively using teh desired bitrate - i mean the desired size, what i set into second pass of 2 pass encoding...
The example is fairly simple. Convertion small movie from Sorenson3 QT video to avi - and Xvid codec, ofcourse. Got anything better? :D
So, i decompressed it to RGB24bit first and then encoded into VirtualDub Mod... I set the desired size same as orginal - eg. 18MB. But hell, Xvid created only 16MB size... Even i try it again, setting 20MB. The movie has got 3min 20sec with 224x180 pixels (eg. fairy small) and i believe that improvents still can be made, but... Xvid refuse, help?
(i mean comparing the decoded orginal and the encoded video make it obvios that Xvid is worser - no wonder, when we look at the size :devil: ...about 600MB to 16MB... But i offered more MB...?!)

Well, about settings, i leaved most at default, witch means that max. bitrate is still at 10000 (i believe that this is more that enought for the movie size :D ) and motion search precision at 6 - ultra high, quantization at MPEG (i heard that it produce sharpest results, is that true, or you recommending different one?), four CC as Xvid and VHQ mode as 4 - wide search. I enabled hinted ME files usage and that it is.
So, why Xvid somewhat uneffectivelly using the given bitrate and do there are plans of adding multiple passes encoding?

I know, what at some point, the gain it too low to bother, but sometimes even the "too low" making difference and the next passes go very quickly, so, i don't see a reason, why not allo even eight or ten passes to get the max out of the bitrate...

Whad do developer think? Koepi? :confused:

[Rober2D2]

Ok I will try to answer qickly:

1) When you set size in xvid, there is an acceptable error of up to 2MB. This is normal. It isn't much noticed if you encode a 700MB film, but of course it is a big difference when you try to make small files.

2) If you encode a movie, quality won't never be better than the original. It may have the same quality if you use losless compresion.

3) Of course improvements can be made :rolleyes:

4) Koepi will be very pleased :p, but he is not the developer of xvid. At least not the only one.

[JimiK]

Think I have to disagree (sometimes):
1. No, that would not be acceptable in this case. And XviD doesn't make such mistakes under normal circumstances. I would say the codec got saturated. After all we're talking about a clip thats only 3 min 20 secs long with a resolution of 224x180. I would bet you only get quant 2 frames. Did you discard the first pass? If not, how big was it? If you did discard it, you can still open the stats with Koepis Statsreader and check the first pass size.
2. Agreed ;)
3. Agreed again and it's already being worked at.
4. Right, but I also have to say: GREAT JOB GUYS!

5. It's never been a good idea to use hinted ME. This could also be a reason for a screwed up 2nd pass (size).
6. Multiple passes are not necessary if your scaling algorithm works fine. XviD can distribute bits very well after analyzing the first pass. In multiple passes (like DivX), the codec sees, 'oh, I didn't do well in the 2nd pass, maybe I should use some more bits here or there'. You cannot spend more bits, they just get distributed to other scenes. So, why do this multiple times when you can make a 'good guess' the first time?
Best regards,
JimiK

[Koepi]

Heh, no, I'm not at all pleased as I did and do only little development on XviD. Praise the core developers:

Isibaar
Gruel
SysKin
GomGom
...

and all who I forgot for their great work!

Regards
Koepi

[kl33per]

Originally posted by JimiK
6. Multiple passes are not necessary if your scaling algorithm works fine. XviD can distribute bits very well after analyzing the first pass. In multiple passes (like DivX), the codec sees, 'oh, I didn't do well in the 2nd pass, maybe I should use some more bits here or there'. You cannot spend more bits, they just get distributed to other scenes. So, why do this multiple times when you can make a 'good guess' the first time?
Obviously the more information you have, the better guess you can make, thus the more passes you do, the better the quality. Also with each subsequent pass, the increase in qualty becomes less noticeable. For CCE, more than about four passes is unnoticeable, for DivX 5.1 it's about three. It's possible that XviD won't noticeably benefit from multipass encoding, but I think some testing should be done to see if there are significant quality improvements when using multipass.

[sysKin]

Three-pass can aways work and can be done. Instead of asking "why it's not done", just do it yourself - plugin system of dev-api-4 makes it quite easy. Just don't forget to share it with everyone, just like we (developers) all do.

But no, next passes will not be faster. It's the first pass that should be faster (much faster) than final one. Hinted ME is gone and will not be back for a long time.

Radek

[Soulhunter]

Is it right that the the VHQ mode of XviD uses some sort of PSNR test to chose what frames are encoded @higherbitrate or so...? (can not explain it well, because I dont know about... sorry ! :D)

When yes, could this Info be saved for a 3rd Pass to "steal" bitrate from frames with higher PSNR, to give it to frames with lower one...?

Or is this what the VHQ mode in 2Pass allready does... ???

Bye

[sysKin]

Originally posted by Soulhunter
Is it right that the the VHQ mode of XviD uses some sort of PSNR test to chose what frames are encoded @higherbitrate or so...? (can not explain it well, because I dont know about... sorry ! :D)No, VHQ does something completely different. It doesn't work on frame basis, and has nothing to with bitrate given to a frame. VHQ is one extra alghorithm performed at the end of motion estimation, that uses brute-force computations in order to detect the optimal macroblock coding mode.
There are three (four with GMC) macroblock modes to choose from, so VHQ is useful.
The criteria is not PSNR, but something called 'rate-distortion' (R-D): number of bits used + total coding artifacts is the value to be minimalized.
When yes, could this Info be saved for a 3rd Pass to "steal" bitrate from frames with higher PSNR, to give it to frames with lower one...?Dev-api-4 has a PSNR plugin that can be used together with bitrate control plugin for this sort of things. It just has to be coded. Wait for XviD 1.0 first, then expect related experiments in 1.1 :)

Radek

[Soulhunter]

THX for the Info... (I was wrong as allways ! :D)

PS: Cant await the 1.0 final... ;) (2003->2004 maybe ??? :D)

Bye

[CruNcher]

Maybe 1.0 Xmas Edition who knows :D

[trodas]

So, let's get started:
Rober2D2
ad.1) - ok, i got it. But setting 20MB and geting nearly 15MB file i feel that there are a relatively BIG room for improvment. Especialy when i see compression artefacts, when elarged on fullscreen.

ad.2) - yes, im avare that i never ever can get quality of the orginal, especialy when considering the compression factor (decompressed about 600MB, compressed 15MB) but what i talking about is ineffective and impossible to reaching certain high quality.

ad.3) - improvents have to be made, this is not good at all.

ad.4) - Oooops... well, sorry must go for the rest then...

JimiK
Yes, that is it. In fact, i mistaken you, guys, its only 240x480 pixels, aftually, hehe. At 15fps. So, it should be nothing hard to compress into Xvid, right? ...sadly, wrong...
And no, i don't discard first pass, but its exactly, on byte same by the size as 2nd pass. Frankly, i never noticed a difference between these two, what suxx...
Well, its somehwat hard to say - great work and thanks very much, dudes, when things are far from expected state. Well, at least hi-quality and big movies get compressed quite well, so, thanks guys for this at least. Please pay attention to smaller movies too - not everyone like QuickTime crap...
What the hell is wrong with enabling hinted ME? I believed that it can help both speed and quality in 2nd pass.
Okay, disabled for testing reasons - exactly same results. Byte to byte. Happy now? And now pls explain WHY not using this options...? Anyone should say what thinking about this as well, developers? Power users...? Anyone?
And well, the multiple pass thing is just a cherry on top of cake. I mean - even good guess - its still only a guess and improvents can and have to be made there. It maybe don't bring there a BIG improvents (sure not), but it might cause the very little sharp thingies, what force someone switch from blurry DivX5 crap to Xvid. You never know. Frankly, i happily spend time on making ten passes encode, yust to get one pixel straight.
Im mad perfectionist. :devil:

Koepi - okay, sorry for the confusion and regards to Xvid developer team :D

kl33per - well, the improvents from multipass encoding never mentioned to be too big or noticable at first sight - but IMHO its worth to add this option to be possible get max from the desired bitreate.
But you missing the point. Where the 5MB go, with relatively small movie, huh? :rolleyes: First let Xvid to perfectly use all the available bites. Yes, 2MB error on 700MB movie i overlook gladly - but its hard to overlook setting 20MB as target size and getting only 15MB movie - hell, even 3MB are audio! - so, this IS ineffective.
And the point is, that the movie is noticably less quality that the decoded 24bit no compression source.
So, i trying to get nicer results and i hit the problem :mad:

sysKin - hello, core developer, god bless you :D And now the hard part :p I never programm a line into C or any other "typical language", so, im sorry, i don't want to hurt this project with my first programming experiments. Its not worth it, not even the multipass encoding.
And thanks for considdering this necessary feature.
About the speed - well, i noticed pretty well, that 2nd pass are about 2 to 3x faster that the first one. So i expected that third pass will be again at least a little-bit faster. Well, at least the 2nd pass speed will be, right? ...and this is f*cking fast! :devil:
And what's wrong with hinted ME? I though that this can help... nope? Please explain what happen to this... BTW, i using this option... wrong? :confused:

And last question - recently i tried to convert Matrix Revolutions teatrical trailer to Xvid and i got message, when trying to start compression of decompressed AVI about wrong inout format.
Ooops, its just a 24bit uncompressed avi, i used this metod before, so, what the hell...?
How i can determine the Xvid version and it's worth to update to most recent, right? :confused:

[trodas]

Ok, i downloaded latest Koepi unstable build there:
http://www.afterdawn.com/software/video_software/codecs_and_filters/xvid_koepi_unstable.cfm#download
but the results are still pretty much the same - okay, using MPEG-Custom (recommended for sharper-better results, is that right?) instead of MPEG increased the size to about 300K.

But still, i setting into 2nd pass size 20 000 and getting 15 185 920 (in case of MPEG-Custom 15 415 296)

Firs and last pass are exactly the same size :rolleyes:

[Koepi]

Why do you download my builds from such strange places?

My site _is_ the link in my signature (and the same as hitting the "www" button). This will make sure to have unaltered binaries.

Regards
Koepi

[JimiK]

O.K., I'll try to explain, even though I don't know where to start.
1. why is the quality so crappy?
XviD is a lossy codec, so it looses quality every time. On DVDs it's not very noticable, because they are quite clean (at least most new ones). So you're compressing what you want to compress. What you have is a already compressed source. Of course you 'unpacked' it to uncompressed RGB, but there are all the artifacts in the picture that came up with the QT-compression. XviD doesn't know it's artifacts and sees it as picture information. Now it tries to recreate the artifacts and it looses in other areas (lossy codec). That's why it looking much worse than the QT original.
2. why can't I reach the desired filesize?
The codec has been saturated. This means it already uses the highest possible quality (quantizer 2). That's why first and second pass are identical. First pass is always done with constant quant 2. In 2nd pass XviD looks where it could use 'higher' quants (>2 worse quality) to reach the desired (normally smaller filesize). Here it does not need to use higher quants, so it uses constant quant 2. What you can do about it: use for example the matrix Selur posted in this thread http://forum.doom9.org/showthread.php?s=&threadid=62762 (where you already posted). You'll still loose quality. Or you use a lossless codec. You cannot determine the filesize, it will be much bigger. The quality will exactly be as the QT original.
3. why not use hinted ME?
Hmm, that's what I understood from earlier discussions (could be it's complete crap): to reach a good compression Mpeg-codecs use Motion Estimation (ME). The picture gets divided into small blocks (called Macroblocks with size 8x8 I think, maybe 16x16. sysKin told me once in another thread). Because a picture and the picture following are most of the times not so different. There could be a camera move, but the scene remains the same. This means that blocks move from one position to another in the next picture. When you decribe this movement by a motion vector you don't have to compress the whole block, but only this vector -> you save a lot of bits. Now as I said before, first pass is done with constant quant 2, 2nd pass isn't. Hinted ME saves the 'motion' from first pass and just uses in 2nd pass. Now it could be that the block looks different in 2nd pass, because they have different quants in both pictures. So it would be better to do a new search with the actual data. It's just that you cannot trust that you get the same motion vectors when you're looking at a constant quant 2, than when using others quants (sounds strange, but I think it's correct). So it will harm the quality, because 'wrong' vectors were used in 2nd pass.
4. wouldn't 3rd, 4th pass... become constantly faster?
No, the speed increase you see comes from hinted ME. So 3rd, 4th... pass would be as fast as 2nd pass.

In the end I could say: you could have found all these information by doing a search, but I know, some threads get rather large. Still, next time you could try a bit harder (information on hinted ME would be rather old, because it has been banned for years now) ;)
Best regards,
JimiK

[trodas]

Koepi - because i not find then on SourceForge pages or Xvid org page, so... Thx for hint. Downloaded the latest snapshot from 24 06, but things aren't different, except that into 2nd pass it fail at Video compression error: An unknown error occured (may be corrupt data). (error code -100) :rolleyes:
Perhaps i take a look for more recent version of VirtualDumMod that 1.5.1.1a, but when first pass worked, then why...
At least i get 19MB, witch is more cose to 20... ;)

JimiK - well, i now don't know, how to tell you, that i know something about compressions and i never expected "better than orginal" results. Also, you probably misread or understand (probably because i did not properly mentioned it :rolleyes:) that i comparing already DOCODED QT, with the artefacts, and not the orginal.
About saturating the codec - well, i see i hit a problem there. But i don't want loseless coded, just fix Xvid to use more ites of what i willing to sacriface for the anim.
I understand perfectly the motion vectors hintfile problem, but the blocks into 2nd pass should not differ much from the used into 1st pass - they can get only nicer, thanks to redistributing bitrate where are awailable. So... :confused: Probably Xvid core coders having troubles with it?
Anyway, i can tell you, that 8x8 blocks are mpeg1 format. mpeg2 alow even 16x16 blocksize and IIRC, mpeg4 (witch is case of Xvid, DivX and all these ones) allow variable block size. :cool:

About treads - there is no single and simple answer about why codec get saturated and what Xvid developers planing to do with it todays ;)
(i mean - fix :D )

[trodas]

Koepi - well, trying second time after erasing all files, except the source ones, helps. Interesting quirk, don't happen first time for me, so i moved to VirtualDubMod 1.5.4.1 - list of bugfixes are impressive enought even for me, a VERY skeptical to upgrading programs, once they work, person... :D
And codec is still obviously saturated, so results are pretty uch the same. Okay, now 100kby smaller size with same settings.
There is still work to do... The first pass have to be HUGE ;) :D (well, carefully with the word huge, i have only there 120G drives... :rolleyes: )

[Acaila]

About treads - there is no single and simple answer about why codec get saturated and what Xvid developers planing to do with it todaysI wrote this (http://forum.doom9.org/showthread.php?s=&threadid=24584) a long time ago to explain some of the reasons why people get saturated videos (and some solutions). Perhaps it could be of some use to you too. Doesn't matter that it's originally about DivX 5, it applies to XviD just as well.

Almost all reasons for saturation are caused by video content, so there's really not much XviD developers can "fix" (although you could fool the codec yourself by applying noise before encoding).

[trodas]

Acaila - thx for hint :D So, since resizing and such actions (filters) aren't wellcome (as they only degrade the quality), i try disable B-frames at all and see, if things get more nicer :rolleyes:

Hoewer, i run into still the same problem with Xvid. I cannot compress big avi source. Yes, its over 2,3Gby.
No, its AVI 2.0 format.
No, don't have anny errors.
Yes, VirtualDubMod can process it and save it and so on...
No, its plain 24bit decopressed QT movie, no compression used.

Help come handy.
Even without audio this thing is 2.2G big, so... It can't get encoded :mad:

Help?

[trodas]

...well, as i see, the B-frames are at default set as -1, witch disabling them. So, there is not the problem. Then why the coded get saturadet too easily, for christ sake??? :rolleyes: :(

[communist]

I'm not sure what you're doing but I'm regularly encoding stuff with XviD (captures / DVD / stuff I created...) and I never (!) had too small filesize - using 1pass Q2 / 2pass VBR with all kind of combinations / 1pass CBR etc...
The only thing that ever happened is that my encodes ended up being 50% bigger but that was when I was still figuring out how to use GKnot.

I'm not sure but isnt there an option to allow the codec to max . increase / decrease the filesize somewhere in XviD?

[Danzel]

trodas: You will get a nicer end result if you dont do Unnecessary color space conversion.

To do this, you could have a avisynth script load your original 15meg QT3 source, and then open that script straight up with VDubMod.
(This will also mean you dont need that 2gig intermediate file)

IIRC, it is recommended to use bframes at 0 if you want them disabled rather than -1, as syskin(?) did some I/P/B frame decision improvements for bframe settings for 0 and above, which give much nicer results.

Also, you could try use the Custom matrixes found on the board, I think HVS-Best (or one of the HVS Matrixes) will give better results and use a higher bitrate, allowing you to hit your desired filesize.
(ah, I see you are already finding out about them :D)

Danzel.

[trodas]

communist - well, you are not sure, because you don't read what i doing. Its conversion of small video files, 240x480 15fps. At this case, Xvid simply seems to be very very problematic. Get saturated and suxx... :rolleyes:

Danzel - well, thx for the tip, but i believe that 24bit is enought ;) Quite frankly, as i said, i comparing the CONVERTED result to the compressed later one. Not the orginal to the Xvid!!! (this is IMHO very important note most ppls overlooked)

About the HVS-Best matrix, well, i foud it there:
http://www.mynetcologne.de/~nc-allgeife8/hvs-best-picture-by3.matrix
...and put it into usage. Immediatelly, first pass become very big (hooooray!) - instead of 15MB i got 38MB in size and the quality is nice, however there are very strange errors into the resulting video, so... help? :(
I just set the bframes to 0, and loaded the custom matrix. Well, i disabled the DX50 B-VOP compatibility, but since bframes is not used, it should not matter...??? :confused: :rolleyes:

Help, please.

[trodas]

Whooops, when i load the result back into Virtual Dub Mod, there is not a glitch! FFD show?! :( Grrr... Milan! :D
...even using Xvid IDCT, there are terrible bugs!!! Help! Fix!!!

[Assault]

Yes you have to decode your video with xvid's directshow filter.(Select use xvid in ffdshow) ;)
ffdshow can't decode some custom matrices properly. It's a very old and known bug.

Assault

[trodas]

Gosh, damn. So, i have nice video now, 3MB bigger that i specifiged, but well, i can't play it! Terrible :mad:

Sometimes i feel i fighting bug over a bug, bugs everywhere, bugs surrouding me... terrible... Its kinda hard to find something that working... :rolleyes:
For long time i tough that FFD show is, except two things, very good. Now the tough is gone :(

[trodas]

...and thanks for hint, settind Xvid as Codecs/format/decoder help like miracle and it working perfectly now. Hoooray. I can use settings 0 for bframes and so on, W/O gettings trashed video... :)