DX5 2-pass against qualitybased 1pass - worth it?

[Donald24]

Hi everybody,

when making rips for cd-recording, the final size of the movie is very important.

So I thought, the only reason for 2-pass encoding, was to analyze the movie, and to decrease the bitrate for any frame, so that the quality of the whole movie remains the same (percentage of quality) regarding the given bitrate.

I recently modded a xbox for Divx-play, and since a large harddisk is in there, I do not have the necessity to make 700/1400 or even 2100 MB-rips to make it fit on cds.

So I tried out one-pass encoding at a given quantisizer. My thought was, that the movie should be of the same quality, as a two-pass encode, and I would save alot of time, not having to deal with final sizes.

But I was wrong, I did make a test with a movie, first one-pass 2.5 quant, and did a two-pass of the movie, with the same file-size as the first.

The two-pass does look better, in any high-motion scenes less macro blocks. But I cannot explain this, is the bitrate somehow better distributed in two-pass?

I would like to make movies with a fixed quality, but I somehow think 1-pass quality based, is not the way to go, is it?

But first I would like to know, why does the two encodes differ that much?

What do you think?

Don

[jonny]

The two-pass does look better, in any high-motion scenes less macro blocks. But I cannot explain this, is the bitrate somehow better distributed in two-pass?

Yes, the bitrate have a better distribution on 2-pass mode, the codec is able to use the quantizers in an "intelligent" way.

I think 1pass qb is only useful if you use quant=2, for the others case, probably you'll obtain better results with 2-pass.

[Acaila]

What 2-pass does in addition to ending up with the filesize that you want is try to distribute quantizers so that everything looks as good as it can.
Fast motion frames need a high bitrate to look good. So in 2-pass mode these extra bits are taken from scenes that don't need a lot to look good, i.e. low motion scenes. So the first will end up with lower quantizers, the latter with higher quantizer but both will end up looking the best they can for that given filesize.

What 1-pass quantizer mode and quality mode do is encode the entire video with a fixed quantizer (although quality mode can deviate from this a bit). Thus high motion scenes get too few bits and low motion scenes get too much (well you can't really get too much, but relatively speaking). This is an inefficient waste of bitrate, but it's a lot faster that 2-pass mode.

[yingx2]

@Acaila
I disagree. I guess you can take a look at your old-fashioned 2-pass encodings again, and use ffdshow to figure out the quant distribution in high motion and low motion scenes.

@Donald24
If you don't care about filesize, quality-based mode is going to be your choice forever. Since no matter what filesize you ended up with, it's already in an optimal quality for that size, and, that filesize is also optimal for the quality you chose. (I know some people are still convinced that 1-pass qb mode is like stupid CBR, and multipass is smarter VBR) What happened to you seems very strange to me and to most people,I believe. If you only do 2 passes (not 3 or 4 passes ), DivX will use higher quants to encode high motion scenes than it does with low motion ones. (Enocde any movie and open its analyze.log , or use the OSD info feature in ffdshow to see what I said. ) The reason is: the codec doesn't learn enough necessary information after the 1st pass, but in order to achieve a specific filesize, better date rate control and better predictability are absolutely needed. Therefore, the codec has no choice but to do some "averaging" on the encoding, avoiding itself from taking "too many" bits to encode hard frames(high mo), as they're already taking a lot more bits than easy frames are, and "too few" to encode easy frames (low mo), as they're already taking a lot less bits than those hard frames. So, the bitrate distribution curve in a 2-pass encoding(again, I said "2-pass encoding" instead of 3 or 4-pass ones ) can't be climbing or falling too aggresively, making the rate conrol possible and easier in 2-pass mode. For example, the maximum and minumum bitrate might be 2500bps and 500kbps in a typical 2-pass encoding, but 9000kbps and 0kpbs in a quality-based encoding.(let's say they're in the same filesize). And I can't agree that we can waste bits on a movie: "How many bits you give to your movie" translates into "how nice your movie looks like".(noisy movies can't count, of course) The more bits you spend, the better it looks, and there is no miraculous way to spend bits intelligently. Unless you choose to believe DivX.com's marketing hype instead of your eyes.

@jonny.
You said the magical sentence "the bitrate have a better distribution on 2-pass mode, the codec is able to use the quantizers in an "intelligent" way". Most people will choose to follow you, I guess. But that cannot be true. If you think 2-pass mode has the better quant distribution than 1-pass qb mode, you must think 9-pass mode is abe to use quantizers in an even more intelligent way. I believe that a 9-pass encoding will look better than a 2-pass encoding, too, becuase almost all the frames of a 9-pass encoding are encoded with the same quantizer (you know that, right?).

1-pass QB mode does this for you, too, and you only need one pass if filesize means nothing to you. For those who are still wondering, yes, the ultimate goal of the multipass machanism is to aquire all the infomation it needs, and find a constant quantizer for every frame, and at the same time hit the target filesize.

[Acaila]

No matter what I say it seems to be coming out wrong lately. I'm going back to silent mode again...

[RMCF]

Originally posted by Donald24
Hi everybody,

when making rips for cd-recording, the final size of the movie is very important.

So I thought, the only reason for 2-pass encoding, was to analyze the movie, and to decrease the bitrate for any frame, so that the quality of the whole movie remains the same (percentage of quality) regarding the given bitrate.

I recently modded a xbox for Divx-play, and since a large harddisk is in there, I do not have the necessity to make 700/1400 or even 2100 MB-rips to make it fit on cds.

So I tried out one-pass encoding at a given quantisizer. My thought was, that the movie should be of the same quality, as a two-pass encode, and I would save alot of time, not having to deal with final sizes.

But I was wrong, I did make a test with a movie, first one-pass 2.5 quant, and did a two-pass of the movie, with the same file-size as the first.

The two-pass does look better, in any high-motion scenes less macro blocks. But I cannot explain this, is the bitrate somehow better distributed in two-pass?

I would like to make movies with a fixed quality, but I somehow think 1-pass quality based, is not the way to go, is it?

But first I would like to know, why does the two encodes differ that much?

What do you think?

Don



BTW 2GB is the limit for Avi's on the Xbox Hard Drive.

[jonny]

You said the magical sentence "the bitrate have a better distribution on 2-pass mode, the codec is able to use the quantizers in an "intelligent" way".

I say no magic sentence.
I prefer lower quants for low motion scenes and high quants for high motion scenes (where the quality degradation is less visible, since they are high motion scenes).
I don't care if the RC algorithm is not perfect, i still prefer this.

If you think 2-pass mode has the better quant distribution than 1-pass qb mode, you must think 9- pass mode is abe to use quantizers in an even more intelligent way.

I think more than 2 passes are a waste of time.

I believe that a 9-pass encoding will look better than a 2-pass encoding, too, becuase almost all the frames in a 9-pass encoding are already encoded with the same quantizer (you know that, right?).

This is a nice thing to know (if you have tested this).
But, as i've told you i still prefer a different quantizer allocations depending on fast / low motion.

Donald24 was using 2.5, this means 2-3-2-3-2-3-2-3 for frames quantizers.
Again, i prefer all quants=2 for low motion and quants=3-4 for high motion, in 2-pass you obtain someting like this (again, this is the ideal case, anyway i find 2-pass results better).


1-pass QB mode does this for you, too, and you only need one pass if you don't care about filesize. For those who are still wondering, yes, the ultimate goal of the multipass machanism is to aquire all the infomation it needs, and find a constant quantizer for every frame, and at the same time hit the target filesize.

If you like to find a costant quantizer to fit the target size, you'll probably know that you can do this with DivX5Enc (instead of wasting a lot of time with multipass)

[john blox]

I have also been trying to ascertain what 'good' settings can be used to give a consistent quality for a divx independant of filesize. I 'DO' feel that the previous aims of 1 or 2 cd rips is becoming an outdated aim with the advent of DVDR's.

Anyway I have spent a few days experimenting and also noticed that 2 pass seemed to look better than either of the single pass modes (for a similar end file size) which doesn't seem to make sense now - Now yingx2 obviously knows what he's talking about maybe he could elaborate on the ideal settings to get the smallest size before a noticeable degradation in quality is observed. I appreciate films are all different but surely there must be a way to get a consistent quality whereby the codec uses the minimum bits it requires to get good results. If two 90 min films end up at 900mb & 1100mb so be it.

[yingx2]

@Acaila
I have to let you know that you're one of the nicest people here whom I respect and trust. You always provide a lot of solid, good information I didn't even know. I am just offering the informaton which I think is correct.

@jonny
You're one of the nicest geniuses too, and I use your great tool on a daily basis.

Originally posted by jonny
This is a nice thing to know (if you have tested this).

Yes, I have tested it.
Originally posted by jonny
I prefer lower quants for low motion scenes and high quants for high motion scenes (where the quality degradation is less visible, since they are high motion scenes).

I thought that this "high-quants for complex frames" made sense for the same reason at first --we can not detect any ugly blocks when they are in the high motion frames as long as they move so fast that they're practically invisible. But the truth is: they're are very noticeable, especially if the high motion secene lasts long enough (5 to 10 seconds,for example). A collection of consecutive blocky frames for 5 seconds is terrible to look at. 0.5 second might be okay, but 5 seconds, no, it's not okay, at least for me because I believe my eyes.

However, If you eocode your movie at 1500kbps or so, the high motion scenes will look fine because the worst it can get is something like quant 4, which is still good enough as it is with low motion scenes. (quant 4 or 3 are excellent for low motion scenes, and they're considered "higher quantizers" if you stick to a 65%~75% comp value.)

I think 2-pass mode can't be intelligent, especially if the movie you want to encode is an extremely inconsistent one. For example, If it contains all its high motion scenes in the first 115 mins and all the low motion ones in the last 5 mins, you will get an undersized file no matter how hard you tweak the RC avraging period or other parameters. (and vice versa. You can add 10000 black frames to the end of a movie, and make the rate control ability of the DivX codec totally break down. But if you do a few more passes, you can still hit your desired filesize. )

My conlusion: 2-pass mode is intelligent only when your source is perfectly consistent in all its scenes.

[yingx2]

Originally posted by yingx2
And I can't agree that we can waste bits on a movie: "How many bits you give to your movie" translates into "how nice your movie looks like"

I have to explain this idea a little bit further.

If you use lower quants for low motion, the high motion will suffer; lower quants for high motion, the low motion suffers instead. However, abrupt quality degration or quality boost in a movie makes its viewers suffer. Besides, an abrupt quality degration is much more noticable than an abrupt quality boost. You can detect a suddenly-appearing 5-sec blocky scene very easily, but you will barely know that the "normal-looking" frames following it you're looking at right now are all encoded at quant 2.

That's why we're willing do so many passes just to find a fixed quatizer for a movie. Balanced, consistent quality perceiving is the most important thing eventually.

[jonny]

What you say have sense and i can agree.

But does not fit the way i encode. I usually target for comp.test >=80% and very high resolution (always 2 cd). This means that i can lower the max quant to something like 4 (or 3 sometimes), with divx 5.0.2. In this case i want my low motion scenes at quant=2, and the only way is to trust in the 2-pass mode. This not always work as we expect, low motion scenes with rain or smoke are encoded at higher quants and the result is not always perfect, but in the general case i think 2-pass works better if you target for high quality.
This is why i've choose 2-pass, for the original question that started this thread.

Another thing: sometime i try to help the codec using different filters for different parts of the movie, example: lanczos for low motion, bilinear (+ some tempsmooter filters if i think are necessary) for very high motion and so on... (the filters i use changes from movie to movie, there is no general case).
I know that this thing is really time consuming, but using this method, in some way, i move in an intelligent way the bits used by the codec (i hope the concept it's clear, it's hard sometime for me to find the correct english words)
I encourage you to make some test in this direction (i'm always happy with the results of this method)

To end all, i think there are too many different movies, different methods, different quality perceptions to be able to find a way that makes all happy :) (probably buying a dvdrw and encoding in qb100% is the only solution :))

Cheers
jonny

[Acaila]

yingx2 wrote:
I have to let you know that you're one of the nicest people here whom I respect and trust. You always provide a lot of solid, good information I didn't even know. I am just offering the informaton which I think is correct.Thank you for the kind words, but you were entirely correct. I wrote my first post before I thought it over...

Anyway, about the original poster's question, he says that he doesn't care (much) about filesize. What's wrong with encoding at 1-pass quantizer 2? Enable B-frames for a nice drop in filesize and your video will still look great (can B-frames be decoded on an xbox?). Average final size will be estimated between 1000-1500 MB, audio included, depending on the content.

[dTb]

I don't proclaim to be an expert but I thought the whole point was to vary the quantizers in 2-pass or multipass. I would be surprised to see a constant quant on a multipass encode unless the codec was maxing out. Apart from a case like this I don't think it would matter how many passes you did.

[jonny]

@dTb:
The only way you have to see what happen is testing (sometime "thinking" is not enough :))

[Donald24]

Thank you for your thoughts about it. I really appreciate that!

Now I think, experimenting is the way to go...but things got cleared up now a bit.

The xbox can decode any professional modes, even gmc and qpixel, but its decoding at 704@square pixels is absolutely at its processing limit without gmc and qpixels using b-frames. But I don't care for anything except b-frames. But their is still a problem with smoothness and dropped frames(very slightly). But its visual quality is really a lot better compared to any tv-out I tried.

But the people at Xboxmediaplayer are really doing a good job,constantly updating their project on a daily basis. (They've just implemented a visualizer for audio (mp3, ogg) in their player...)

THX again,

Don

[RMCF]

This is what I recommend. Encode movies at 640xyyy. No B-frames, no nothing. (macroblocks)

Use Convolution3D (moviehq) + Lanczos

Edit skin.xml and change resolution to 640x480. Calibrate screen (0.0) (0.0) (there was a bug, (0.1) if its not solved).

No post processing, no soften (for good quality movies).

That is the way, xbmp movies look pretty amazing.

[yingx2]

@dTb
(I am no expert, either. But I can tell you what I know.)
Varyng the quantizers for different frames of the same source makes no sense. Varying the bitrate, on the other hand, does, since the complexity of every frame is unique. 300kbps may work well for a sequence of easy frames, but it cannot handle an explosion scene undoubtedly. A quantizer determines the amount of information of a frame to be discarded. The same quantizer discards the same proportion of information. Frames encoded at the same quantizer present its viewers with the same visaul quality in theory. It somehow works like the compressibility value. Quant 2 is equal to a comp value of 100%. Quant 4, according to my experience, is usually somewhere between 44% - 49%.

Thus, if you vary the quantizers for different frames, you're actually varying the compression scale for them. Lets exaggerate the problem a little bit and make it simple: Can you imagine to divide a movie into 3 parts, encoding part A using a comp value of 100%, part B 70%, and part C 30%? If this is unacceptable, why is it acceptable to assign different quantizers to different scenes when quantizers and compessibility are significantly related? I'm sure it's not because DivX.com promotes multi-pass encoding instead of 1-pass QB mode, saying that 2-pass "can allocate data more efficiently to scenes that need more and less data to scenes that do not.", as it's sophiscated and fancy enough to fool its potential customers with little exprience into buying their product, while labeling 1-pass QB mode as "stupid newbie only". The truth is, when compared to 2-pass mode, 1-pass QB mode throws more bits on hard frames, and less bits on easy frames. (I know that some people don't like it)
Originally posted by jonny
Another thing: sometime i try to help the codec using different filters for different parts of the movie, example: lanczos for low motion, bilinear (+ some tempsmooter filters if i think are necessary) for very high motion and so on...
It's time consuming but it's the best. I bet that you will encode each micro block individually if you can.:)
Originally posted by jonny
but in the general case i think 2-pass works better if you target for high quality. This is why i've choose 2-pass

I know that 2-pass mode works pretty well for you in most cases. I'm just saying that it's still far from being intelligent. Can you make this clip (http://myweb.hinet.net/home1/joefox/test.avi) hit your disired filesize, say 10MB, using a "normal way" in 2 pass mode?

I was empahsizing the weakness of 2-pass mode by adding some black frames to it. Actually, any inconsistent source with very hard scenes and very compressible scenes unbalancedly located in it will suffer from different levels of "unintelligent encoding". If you use the typical way to encode this clip via 2-pass mode, you're destined to end up with an undersized file, and you know it's clearly not saturated. What's worse is, that the codec practically assigned all its quant 2 to those black frames.

Most movies are not that inconsistent like the case above, but they are always inconsistet to some degree, meaning that they will all suffer from different degrees of "unintelligent encoding" when you encode them using 2-pass mode.
Originally posted by jonny
probably buying a dvdrw and encoding in qb100% is the only solution :)

But at least, we can reach a consensus on what the ultimate soultion should be.(still, I would copy dvds rather than encode them if I have a DVD burner. :))

[yingx2]

"Prior to encoding, the video is analyzed to understand its complexity. We can then allocate data more efficiently to scenes that need more and less data to scenes that do not. This provides more accuracy than "1-Pass" encoding"

(directly quoted from the official DivX guide on DivX.com)

Sometimes, after all its great analyzing in the first pass, the codec determines that black frames should be one of those scenes that need more data, and assigns all its highest quality quantizers to encode them. Although they practically take zero bits to encode even at quant 2, still, it's ridiculous enough.

EDIT: By the way, even if your source is a perfectly consistent one, if you do only 2 passes, your encoding will still suffer from the unbalanced quality problem, since the DivX codec has the tendency to save bitrate in its early part of encoding process for convenience of its rate control, and unreasonably raises the average bitrate for the final portion of the encoding.

[dTb]

Originally posted by yingx2
[B]Varyng the quantizers for different frames of the same source makes no sense. Varying the bitrate, on the other hand, does, since the complexity of every frame is unique.[B]

It seems to make perfect sense to me, varying quantizers as well as bitrate is how you achieve high compression.
I never use 1-pass qb because it doesn't suit my purposes so I can't really comment much on it. I've been told from reliable sources that it's quite wasteful, maybe 2-pass qb would be useful.
2-pass may not be perfect but imo it is the best method for the majority of situations. You personally don't like the fact that bits are saved on high motion scenes but for me and I'm sure a lot of people it's the best way to go. You can adjust this to your liking in 5.0.3's multipass now as well.

[yingx2]

@dTb
I thought I made my point clear enough.

If varying quantizers makes perfect sense to you, the inventor of the multipass encoding system must be an idiot.

To make the quality of the video stream constant is the ultimate goal of multipass encoding method, including the 2-pass mode. The quantizers always jump up and down in a 2-pass encoding not because the codec tries to find the optimal quantizer for each frame ( See "black frames at quant 2" example) , but becauce it needs to decrease the difficulty of its data rate control job, since the information collected during the 1st pass is not accurate enough. So how does a codec produce a video stream in constant quality? The only way to do this so far is to quantify the complexity of each frame, preserving the same proportion of information in each frame by using the same quantizer. (Do you have a better way to make the quality of a stream constant?) Do you accept that certain part of your encoding is at quality 45, while the other part is at quality 75? If you do, yes, 2-pass mode is your winner, and the the inventor of multipass encoding system is a moron again.
2-pass may not be perfect but imo it is the best method for the majority of situations
2 pass mode is one of the mothods when you want size preditabilty, no more and no less. (Or go tell Gej, one of your "reiable sources" perhaps, that 2-pass is better than 3-pass)

Besides, I was not trying to encourage people to use DivX 5.03 or its multipass encoding method. I switched back to 5.02 right away when I finished testing the latest version. And personally, I use 1-pass QB at quant 2 for most cases. Apart from filesize predictability, the difference between a 1-pass QB encoding and a perfect multipass encoding (where each frame is being encoded at a fixed quantizer) is the bframe mutiplier. A bframe multiplier in multilpass mode is not necessarily 2.

By the way, 2-pass QB mode doesn't make any sense to me. Would you like to explain how do you think it should work? (And please tell me which part of my posts do you think is illogical.)

[N_F]

@yingx2

I have to say that at the beginning of this thread I didn't quite understand you and to be blunt I thought you were full of shit. :rolleyes: What you said went against everything I thought I knew about 2-pass encoding.

So yesterday I made some experiements with ffdshow's osd and discovered that you seeemed to be right. Usually (assuming no B-frames) there seems to be almost only 2 quants used, and one can almost see a pattern in them sometimes.

What you have also said later about quants also makes sense when you think further about it.


Thanks for bringing my understanding of encoding and quants to a further level. :) I don't claim to understand it completely yet, but hopefully I'm on my way.

BTW, am I the only one finding myself using the word "quants" 'cause I can't spell the correct full term? :D

[dTb]

Maybe I don't have a good enough understanding of quantizers but I thought that it was one way for the codec to help save bitrate. I thought it was only natural to up the quant say on certain frames or scenes to help compressibility where it could be done without noticeably affecting quality.
This is the first I've heard of a 2-pass or multipass encoding having the aim of achieving a constant quant throughout the whole encode. Did you actually achieve this with multipass in DivX 5.0.3 or is it something you figure will happen in the future?
My point about a possible 2-pass QB mode is that from what I've read (I'm aware you can't believe everything you're told or read) it's not very efficient, throwing bitrate at the video even if it's not needed. Having a second pass could cut out the wasted bitrate. If as you say multipass aims for a constant quant it almost becomes a quality based mode.
Maybe it's an abberation but Donald24 cited a case of a 2-pass encode with the same filesize as a 1-pass qb encode as having better quality.

[jonny]

It's time consuming but it's the best. I bet that you will encode each micro block individually if you can.

:D
note that in some way the method is really similar to what you are saying, because i'm manually "leveling" the complexity of the scenes (but the big difference it's that i am able to keep, lose details when i think it's needed).

I've read your post about all the quants=2 at the end of the clip some days ago. I understand you perfectly.
I'm not able to download the clip, too much for my 56k :(
The main point it's that i'll never go so low with compressibiliy to let the codec full use it's "intelligent" :D 2-pass mode. This is why for me it's enough.
I'll never be satisfied encoding at comp.test 50-60%, this means both for 2-pass or costant quant (anyway i'm really courious to compare the results now :))

It's nice to talk with you, yingx2.
Cheers
jonny

[jggimi]

Reminder: Let's be polite, people! Differences of opinion do not require verbal brutality.4) Be nice to each other and respect the moderator. Profanity and insults will not be tolerated. If you have a problem with another member turn to the respective moderator and if the moderator can't help you send a private message to Doom9.

[RMCF]

So, Do I understand this correctly?

Doing 9 passes on a movie encodes all frames in that movie at constand quant?

So in this case using the size prediction in Jonny's program is the way to do encoding since that equals 9 pass multipass.

Right?

[john blox]

Is there a conclusion to the original que then?

What are best divx5-pro settings to use when unconcerned with exact final file sizes but just aiming for a consistent quality to play back on an xbox. (Assuming 640x or lower is max horiz res xbox will cope with)

TIA

[RMCF]

Use qb=2 if you can. Don't lower your res less than 512yyy.

Anyway, I used Jonny's divx enc size prediction to a movie with quant 3 and it looked better that 3-pass multipass of the same movie (same size offcourse).

[DDogg]

Disregard.