how to predict quaity using x264

[PuzZLeR]

Thanks Akupenguin,

Nevertheless, regardless of distribution of benefits, it's clear that you'd, at least arguably, be better off with slower CRF settings than 2-pass in terms of value gained with respect to encoding time. It answers my question and should serve insight to the last question by the OP.

Hail to the x264 quantizer...:)

[SealTooGreat]

I guess you can get away with such bad encodes because the screen size is so small and the bad quality won't be as noticable. As well, it's great when you need a fast encode, at a certain size to meet bitrate limitations, and to have it ready for a trip or something in record time. You don't care to archive it, just watch it once or twice then delete it. ABR 1-pass is the BEST for this.

Other than that, I see very little use for ABR 1-pass.

You have made me thinking you have never used ABR in your life. Saying how CRF is great especially using 18 for VIP content, you are forgetting to say that ABR 1-pass has switch "--bitrate X" where you can type as much you can like ;)

Encode DVD content using CRF 18, take the info how much bitrate is used, type this value in ABR 1-pass cmd, encode and compare these 2 results, CRF and ABR. Do you think that ABR is bad? Let your eyes be the judge :)

I've never read that someone is openly saying that use CRF 31. According to that I can say CRF is bad, but is not. There is misunderstanding in all this story. Pls, separate words extra, good and shit (especially last 2 ones )

If this is how you make use of H.264, or you have limited resources, you seriously are better off with MPEG-4 Part 2.
Don't worry. ;)
I'm using both AVC and ASP.

[PuzZLeR]

Hey SealTooGreat,

Well I did do a comparison from CRF's resultant bitrate plugged into ABR 1-pass. Sorry. CRF, according to my eyes, was a considerably higher quality encode.

And yes, CRF=31, in my opinion is crap. But for a DvD type of resolution then so is ABR 1-pass, or ABR 2-pass, at 500kbps - pure crap.

Then again, I do admit, at higher bitrates, there may not be a huge difference between CRF and 1-pass ABR. If you're happy with 1-pass ABR, then by all means, if it serves your purposes then go for it.

And 1-pass ABR does give you the best speed and accuracy in file size, which CAN make it a good option depending on objectives.

To be consistent with the thread's title though "how to predict quality with x264", the encoding of CRF and then using that bitrate into 1-pass ABR would be a terrible idea though IMO, regardless of computing resources or objectives. I certainly hope, and assume, you will not do that.

CRF plugged into 2-pass ABR? We have a great argument. 1-pass ABR? Never.

Just my opinion my friend.:)

[Dark Shikari]

And yes, CRF=31, in my opinion is crap. But for a DvD type of resolution then so is ABR 1-pass, or ABR 2-pass, at 500kbps - pure crap.
Would you like me to upload a clip of my 1280x720 encode of 300 at 775kbps 2-pass ABR? :p

Or link you to my --crf 30 270MB, 50 minute long 720p EVE Online tournament video? :p

(Its weird, I'm obsessed with low file sizes rather than high quality...)

[Sagekilla]

@Sagekilla:
For MPEG-2 source, you can't do much better than CRF=18 from my experience. Even though my compression "sweet spot" is 19, 18 is for the VIP content...:)...but to each his/her own right?

Yeah, when I played around with the settings 19 is pretty much the sweet spot but I do have the storage space to spare so I shoot for the little extra. Plus most of the movies I'm ripping right now in crf 18 are my favorites. I don't think I'll have enough space to rip my entire collection, so I'm doing my favorites first.

[check]

don't forget you don't need to give integer values, 18.7 is legal, so is 19.65 or whatever :)

[((( atom )))]

si if i understand correctly, i can tweak other settings pretty much for better compression but stay at my crf 18 quality?

which settings are interesting?

doing that would surely make sense for less compressible movies, where my file might end up at 3gb or worse..

also it might be ok, since i have a core 2 quad waiting on my other desk to be tried out for encoding..

[check]

Yes, all the settings now basically just affect compression. This isn't entirely true, but it's pretty close to the truth.
As for interesting settings, read this (http://mewiki.project357.com/wiki/Video_configuration_dialog/X264_Configuration):)

[Dark Shikari]

si if i understand correctly, i can tweak other settings pretty much for better compression but stay at my crf 18 quality?

which settings are interesting?

doing that would surely make sense for less compressible movies, where my file might end up at 3gb or worse..

also it might be ok, since i have a core 2 quad waiting on my other desk to be tried out for encoding..
Some of them lower the bitrate by improving compression efficiency, while others are more likely to raise the quality a bit. Some do a combination. It depends on the situation.

For good quality at a good speed, use --subme 6 --trellis 1 --me hex --b-pyramid --weightb --bime --direct auto --bframes 16 --mixed refs --ref 3 --deblock 0:0 --partitions all --8x8dct and whatever ratecontrol and other options you need.

--subme 7, --me umh, and --no-fast-pskip will each give a 1.5-2% quality-per-bitrate improvement at the cost of a good bit of encoding speed. Raising --ref beyond 3 will help on animated footage.

[microchip8]

I read in the MEncoder manual page that one should not use p4x4 if you deal with big resolutions. I don't know why as they don't explain

if this is true, can someone explain why it "hurts" to use p4x4 on big resolutions?

[akupenguin]

p4x4 doesn't hurt, it just doesn't help. At large resolutions, individual objects in the movie are bigger than 4x4 pixels, so there's no point in partitioning mvs that small. Typically it might be enabled in 0.5% of macroblocks, and improve compression by 0.1%, making it one of the worst compression-per-cpu tradeoffs.

[microchip8]

ahh, now I get it, so what are large resolutions? Are we talking here about the common resolutions of today (720x576 or 720x480) or about the newly introduced HD resolutions 1080p/i, etc and where p4x4 "stops" working?

[akupenguin]

Just a matter of degree, but I don't use p4x4 even at 480p.

[((( atom )))]

so here we go: i just did "cinese box" and it hit 4.7gb for a 90 min movie without audio!
the source is very noisy although rather sharp. it simply contains extreme amounts of filmgrain.

now how could i deal with that best? i would not want to loose either resolution nor the noisy look, but some compromise will have to be made..

any hints?

[check]

you're right, a compromise must be made :) The two variables you are balancing are output filesize, and output quality.
If you want to reduce filesize, you will lose quality, no buts about it. Whether or not this quality loss is visible is another matter again, but in general, the higher the quality loss, the greater the drop in output size. Try these ideas to reduce the filesize:
o reduce resolution
o more heavily filter the clip (you should always do some basic filtering on all videos you encode, it can make a huge different to compressibility for almost no change in quality. I recommend fft3dfilter(sigma=0.3) or so, depending on the individual file. Test with AvsP)
o increase --crf target
o increase x264 quality settings
oo use a lower bitrate CQM. I'd suggest mp4guy's LRM, or possibly his MRM. http://forum.doom9.org/showthread.php?t=117041

If you're looking to

[Dark Shikari]

so here we go: i just did "cinese box" and it hit 4.7gb for a 90 min movie without audio!
the source is very noisy although rather sharp. it simply contains extreme amounts of filmgrain.

now how could i deal with that best? i would not want to loose either resolution nor the noisy look, but some compromise will have to be made..

any hints?
One choice is to remove the noise and re-add noise on playback using a simple filter. Reducing noise can drastically increase compression.

[((( atom )))]

that's no option for me, since i have a linux-player withoug a keyboard, only a remote control. also it is a tiny processor (via) wich can do x264 @ full pal with roundabout 75% load. so no filtering on a per movie basis :(

[Sagekilla]

Some of them lower the bitrate by improving compression efficiency, while others are more likely to raise the quality a bit. Some do a combination. It depends on the situation.

For good quality at a good speed, use --subme 6 --trellis 1 --me hex --b-pyramid --weightb --bime --direct auto --bframes 16 --mixed refs --ref 3 --deblock 0:0 --partitions all --8x8dct and whatever ratecontrol and other options you need.

--subme 7, --me umh, and --no-fast-pskip will each give a 1.5-2% quality-per-bitrate improvement at the cost of a good bit of encoding speed. Raising --ref beyond 3 will help on animated footage.

I was a bit interested in how setting B-Frames to 16 would help any.. I usually use 4 in my encodes with 4 reference frames. My methods may or may not be exactly the best (Someone please correct me in my procedure if you see anything amiss here) but from running 3 different test, with two settings each time I've compiled this much data. All settings listed in the image are the only ones changed. If a setting was unlisted, it meant I didn't use a switch to change it at all.

http://img.photobucket.com/albums/v621/Sagekilla/data.png

What I found to be a bit odd was the encoding speed actually -increased- slightly as I upped the number of B-frames to 16. It wasn't much, but it was fairly consistent in the fact that it did increase. Everything else remained pretty constant, with only a tiny drop in bit rate and corresponding tiny drop in average and global psnr, as well as SSIM.

I have the batch files used for running the encodes as well as the output stats and video files here (http://skdotnet.sytes.net:45312/x264/bframetest) if anyone is interested at taking a look.

[akupenguin]

A B-frame encodes faster than a P-frame, because most of the macroblocks in a B-frame are so obviously skips that it doesn't have to run motion estimation or any other decision.
Using B-frames may or may not be faser than not using B-frames, because B-adapt costs some time too. But if you're using B-adapt anyway, increasing the number of B-frames increases speed.

A B-frame decodes slower than a P-frame, because bidirectional motion compensation has to interpolate twice the samples.

[Sagekilla]

Hmm... gotcha, I guess I'd probably see the performance hit more on higher resolution or more complex scenes though. I just tested the video samples out and with task manager open, and --bframes 16 files seem to be only a bit more intensive then the regular ones.

I'm going to look around and see if I have any fraps recordings left to test this on, those are 1680x1050 native.

[Dark Shikari]

--bframes 16 is basically telling x264 to do whatever it thinks is best. This means that one doesn't have to think "Oh, this is a cartoon, it might need more b-frames": the encoder is just left free to do what it thinks is best.

[MarcioAB]

My 2 cents:

In my tests, for the same bitrate, Constant Quality ( --crf ) outperformed (OPSNR and SSIM) all the other modes.
It was a surprise to me that expected 2-pass should be better, but whatever so: crf rules !

BPP is nothing else than the compression index. I find it a very useful information.

Uncompressed YV12 (4:2:0) is 12 BPP, so a video with 0.20 BPP has a compression index of 60:1 ( 12/0.2 ).

More than sure that BPP is not an absolute value for visual quality but (at least for me) it help a lot:

- x264 videos below 0.16 BPP: high probability that it will suffer visual quality issues (imo).
- x264 videos above 0.20 BPP: high probability I am wasting bitrate.

Composing CRF + BPP + OPSNR + SSIM + your_PoV should give a good perspective.

( PoV = point of view )

[nurbs]

BPP is nothing else than the compression index. I find it a very useful information.

Uncompressed YV12 (4:2:0) is 12 BPP, so a video with 0.20 BPP has a compression index of 60:1 ( 12/0.2 ).

How often do you have an uncompresst YV12 source to work with? Most people here have already compressed sources (mpeg2, avc, vc1).

With enough experience you can estimate what bitrate and thus bpp a given source will need to look good, but it varies to much in my opinion to be any use as a rule of thumb. For instance I have very good quality Futurama encodes at about 0,06 bpp, but if I would try to encode Natural Borne Killers at the same bitrate it would look much worse. I was barely satisfied with 0,24 bpp on that movie.

[SealTooGreat]

How often do you have an uncompresst YV12 source to work with? Most people here have already compressed sources (mpeg2, avc, vc1).

With enough experience you can estimate what bitrate and thus bpp a given source will need to look good, but it varies to much in my opinion to be any use as a rule of thumb. For instance I have very good quality Futurama encodes at about 0,06 bpp, but if I would try to encode Natural Borne Killers at the same bitrate it would look much worse. I was barely satisfied with 0,24 bpp on that movie.

If we want questioning quality factors (CRF quantizer number) I can say I'm satisfied with CRF26 when encoding drama, but with the high motion action movies, 26 gives me worse quality. But it seems the people are rather to use CRF18 and saying how bpp needs balancing ;) If you want ultimate bpp than choose 0.3 and over and don't look back below. I've just encoded small sample (1min52sec) mixed high motion with short slow motion intervals. Qualty factor is 18 (CRF) and MediaInfo reports bpp=0.588 :) funny isn't that bpp enough for eyes's transparency. We are getting paradox - What is better CRF18 or ABR/bpp=0.588 ? As CruNcher says it's all about how you balance it in the end :)

[Plamen234]

that's no option for me, since i have a linux-player withoug a keyboard, only a remote control. also it is a tiny processor (via) wich can do x264 @ full pal with roundabout 75% load. so no filtering on a per movie basis :(

What is your system configuration?
I'm curious about mine...

[foxyshadis]

If we want questioning quality factors (CRF quantizer number) I can say I'm satisfied with CRF26 when encoding drama, but with the high motion action movies, 26 gives me worse quality. But it seems the people are rather to use CRF18 and saying how bpp needs balancing ;) If you want ultimate bpp than choose 0.3 and over and don't look back below. I've just encoded small sample (1min52sec) mixed high motion with short slow motion intervals. Qualty factor is 18 (CRF) and MediaInfo reports bpp=0.588 :) funny isn't that bpp enough for eyes's transparency. We are getting paradox - What is better CRF18 or ABR/bpp=0.588 ? As CruNcher says it's all about how you balance it in the end :)

If you're willing to dump enough bitrate on something to get it below CRF18, of course it's going to look as good or better than CRF18, that's a tautology. Did you check the average quants? You'd almost be better off keeping the original MPEG2 in that case - DVD9s usually have bpps around 1.0, and DVD5s around 0.5-0.6, and if made with a good encoder look about as good as mpeg-4 can most of the time.

BPP doesn't even come close to scaling with resolution - sagittaire conclusively established this years ago. BPP^2 does, sort of. But it's still more of a security blanket than a useful rule of thumb, when it's obvious that .2 bpp gives wildly different results. You can claim that experience gives you the ability to judge, but that can be reversed to experience allowing you to guess the bitrate that comes from a crf - not true in my experience, the codec tends to surprise you more often than not.

Have you actually tried the comp check by DDogg (which selects a quality based on a bitrate) or in automkv (which also selects resolution based on a quality & bitrate)?

[MarcioAB]

How often do you have an uncompresst YV12 source to work with? Most people here have already compressed sources (mpeg2, avc, vc1).
The point (12/BPP=compression ratio) is valid even if the source is already compressed.
(anyway it will be decompressed (uncompressed) in the middle of the process).

With enough experience you can estimate what bitrate and thus bpp ...
My point is the opposite. Example: assume you got the task to compress 1 hour of 1920x1080x60p source. What bitrate will you kick off ? No idea ... But assuming 0.2 BPP you will have a good starting point: the bitrate will be 0.2 x 1920 x 1080 x 60 = 24.88 Mbps, so expect 10.43 GB for the video part.

For instance I have very good quality Futurama encodes at about 0,06 bpp, but if I would try to encode Natural Borne Killers at the same bitrate it would look much worse. I was barely satisfied with 0,24 bpp on that movie.
I do not encode much anime but I think the BPP edges for anime should be lower than other types of movie. So for anime it could be something like this (for example):

- x264 anime videos below 0.060 BPP: high probability that it will suffer visual quality issues.
- x264 anime videos above 0.120 BPP: high probability it will be wasting bitrate.

- x264 extra-action below 0.180 BPP: high probability that it will suffer visual quality issues.
- x264 anime videos above 0.240 BPP: high probability it will be wasting bitrate.

[MarcioAB]

...Qualty factor is 18 (CRF) and MediaInfo reports bpp=0.588 :) funny isn't that bpp enough for eyes's transparency. We are getting paradox - What is better CRF18 or ABR/bpp=0.588 ? As CruNcher says it's all about how you balance it in the end :)

So, CRF 18 gave you 0.588 BPP. I suggest to try a much higher CRF, maybe 28, check the BPP and see if it please your eyes.

Let me suggest an hypothetical example just to exchange ideia:

film1 compressed at CRF 18 give 0.20 BPP and visual quality is good.
film2 compressed at CRF 28 give 0.20 BPP and visual quality is good.

This means film1 is "more compressible" (or "more easy") than film2. Maybe film1 is an anime, or low motion/low light ... does it make sense ?

[Sagekilla]

If we want questioning quality factors (CRF quantizer number) I can say I'm satisfied with CRF26 when encoding drama, but with the high motion action movies, 26 gives me worse quality. But it seems the people are rather to use CRF18 and saying how bpp needs balancing ;) If you want ultimate bpp than choose 0.3 and over and don't look back below. I've just encoded small sample (1min52sec) mixed high motion with short slow motion intervals. Qualty factor is 18 (CRF) and MediaInfo reports bpp=0.588 :) funny isn't that bpp enough for eyes's transparency. We are getting paradox - What is better CRF18 or ABR/bpp=0.588 ? As CruNcher says it's all about how you balance it in the end :)


Well constant rate factor is quite simply that -- constant.. It'll give you constant quality throughout. Even though it may report as being ".588 bpp" that's a global average across the whole video probably and isn't indicative of the bpp across various scenes in the movie. High motion will have higher bpp, low motion will have lower and so forth.

ABR on the other hand, gives you a little leverage to go slightly over or slightly under. But, the problem I see with it is that given a certain bit rate, say 3 mbps, in a low complexity scene it'll end up giving it way too much bit rate than actually needed and you'll have excessive quality. Whereas on a high motion scene it may not be giving enough and you'll start getting some video artifacts. But, like I said before, CRF will just give a uniform quality throughout and there won't ever be that "There's not enough bit rate for this scene" or "There's way too much quality and it's wasting bits in that scene."

'Course that's how it works, if anyone knows that any of that is incorrect feel free to correct me..

@MarcioAB: Your explanation seems a bit backward at the end of your post, I think you mean the other way around.. Lower CRF is higher quality, higher CRF is lower quality, so Film 2 should give better compressability since it doesn't need to use as low of a CRF.

[akupenguin]

ABR on the other hand, gives you a little leverage to go slightly over or slightly under. But, the problem I see with it is that given a certain bit rate, say 3 mbps, in a low complexity scene it'll end up giving it way too much bit rate than actually needed and you'll have excessive quality. Whereas on a high motion scene it may not be giving enough and you'll start getting some video artifacts.
That's CBR (i.e. ABR with one or more of: VBV, low --ratetol, or low --qcomp).
Lax ABR (i.e. --ratetol inf) tries to be CRF. It adjusts the RF value periodically to approximate the given filesize, but only the first few scenes have really variable quality. After that the variations in RF are progressively damped. ABR --ratetol inf still isn't as good as CRF, but there isn't such a huge difference. Otoh, --ratetol inf not only compromises between ABR and CRF in quality, it also compromises in filesize accuracy, being as much as +/-10% off.

[nurbs]

My point is the opposite. Example: assume you got the task to compress 1 hour of 1920x1080x60p source. What bitrate will you kick off ? No idea ... But assuming 0.2 BPP you will have a good starting point: the bitrate will be 0.2 x 1920 x 1080 x 60 = 24.88 Mbps, so expect 10.43 GB for the video part.


I do not encode much anime but I think the BPP edges for anime should be lower than other types of movie. So for anime it could be something like this (for example): ...

That was my point sort of. The bpp value that gives a good quality depends too much on the source to be of much help. Cartoons or stuff with low motion like talkshows or newsbroadcasts need less bitrate while an action movie or a movie from the thirties with the scratches that are common in old films need much more.

Then there is also the resolution that comes into play. The highter the resolution the lower the bitrate you need to get a certain quality. foxyshadis mentioned this thread (http://forum.doom9.org/showthread.php?t=95122) where they talk about this relationship. Also note post number 4 by akupenguin in the linked thread. So while for a given movie 0,2 bpp may be the optimum value at PAL resolution it will still be a waste of bitrate at 1080p.

[MarcioAB]

@MarcioAB: Your explanation seems a bit backward at the end of your post, I think you mean the other way around.. Lower CRF is higher quality, higher CRF is lower quality, so Film 2 should give better compressability since it doesn't need to use as low of a CRF.
Sagekilla, allow me to explain my point using the graphics (made by myself) and explanation below:
http://pws.prserv.net/marcioab/compressibilty.gif
The way I understand it (as a week-end hobbyist) is that CRF could be associated to the "compress force", so high CRF means higher compress force to video data.
BPP could be associated to "the result of such compress action", so high BPP means less compressed result.
Assuming that is correct, then a high compress force (high CRF) resulting in a low compressed file (high BPP) means that such video data has a LOW compression characteristic or that it is more difficult to compress.

This graphic has 3 dimensions. The 3rd dimension is subjective and assume that each pair ( CRF, BPP ) is associated to a film with good visual quality AND near the low-edge. To find the low-edge one should compress the video and check if result is good. Then compress more up to point it became not-so-good. The low-edge is the previous compression level where visual quality is still good.

So, back to film1 and film2:

film1 compressed at CRF 18 give 0.20 BPP and visual quality is good.
film2 compressed at CRF 28 give 0.20 BPP and visual quality is good.

I applied a "low force" into film1 than film2 and both became same size, so film1 is more compressible than film2.

Does it make sense ?
Thank you

[MarcioAB]

...Cartoons or stuff with low motion like talkshows or newsbroadcasts need less bitrate while an action movie or a movie from the thirties with the scratches that are common in old films need much more.
Yes. There may be possible to set some sort of classifications to films and using it plus the pair ( CRF, BPP) get better control. Should not be that difficult ...

Then there is also the resolution that comes into play. The highter the resolution the lower the bitrate ...That is a good point. I already noticed that, but I do not have access to HD source to check by myself. My only exposition to HD was downloading HD trailers (720p, 1080p) and short movies and check visual quality and BPP. I found most of this content at 0.16 BPP but I found visual quality was not good enough to my taste in certain parts (low motion / low light) but than I learn here that this is a very-known MPEG4 issue [ will it be resolved someday? ] and not lack of BPP. So 0.16 BPP seems to be a good spot to HD content as opposed to 0.18~0.20 to SD.

Regard the math in that thread ... interesting, but because we are now in a so much empiric environment, isolate behaviors in ranges/layers ( SD, HD, anime, high-action, dark-film, etc) could be easier to work (IMO).

[MarcioAB]

But, like I said before, CRF will just give a uniform quality throughout and there won't ever be that "There's not enough bit rate for this scene" or "There's way too much quality and it's wasting bits in that scene."
'Course that's how it works, if anyone knows that any of that is incorrect feel free to correct me..


Let me give view on CRF ( fell free to correct me also).

I set a Q target (lets say 20) and let it run. For each block in the frame x264 will kick a Q (lets say from 10 to 30) but in the end the overall frame-mean Q (of all these blocks) will not be much far from my requested Q target (lets say from 14 to 26) and in the end, the overall Q will be very close my requested Q target.

I wonder how x264 with just a single pass can distribute all those Qs in the very best way. Could a 2nd pass CRF-oriented even improve quality ?

[akupenguin]

CRF isn't perfectly optimal, but the optimal algorithm doesn't need 2 passes either. Read up on Lagrange multipliers.

Your graphic shows good ideas, but it's not quite right. The curves should be hyperbolas curving away from the origin and never quite touching the axes, not circles.

[shon3i]

Well 2pass VBR will be i think best rate control :). VBR/CRF 1pass never can give max quality for desired quality level.

[Dark Shikari]

Well 2pass VBR will be i think best rate control :). VBR/CRF 1pass never can give max quality for desired quality level.
But my tests show that CRF yields superior quality to VBR, though not by a gigantic margin.

[shon3i]

You think superior than 2pass ABR because 2pass in x264 (and others) is ABR not VBR ;)

Well quality is probably better on CRF encoding, but what with size? what you can say about size?

For same size 2pass will be better.

IMHO CRF/VBR is more wasting bits, than i get max quality for that size. Aslo is hard to say what quality level is good for some move without some analyses before ;)

I have ability to test VBR mode in Ateme Digital Encoder, and i must say it have some improvements over x264 CRF

[Dark Shikari]

You think superior than 2pass ABR because 2pass in x264 (and others) is ABR not VBR ;)

Well quality is probably better on CRF encoding, but what with size? what you can say about size?

For same size 2pass will be better.
Did you even read my posts? I did a CRF, and then did a 3-pass mode on the exact same bitrate that CRF gave. Not only did ABR fail to converge to that bitrate (it was about 5% over), but the quality was still worse.

And what do you mean by "its VBR, not ABR"? ABR and CRF are both VBR methods.

[Terranigma]

ABR and CRF are both VBR methods.
VBR = a method based on a quality level (so VBR in avc would be -qp and -crf, not abr)
What shon3i was trying to say was that doing 1 pass crf/qp is not using x264 at it's full efficiency, and that doing "1st pass vbr" followed by an additional abr pass would be better.

[microchip8]

doing "1st pass vbr" followed by an additional abr pass would be better.

that's what I always use here. First pass in CRF @ 19, then second pass with a bitrate. Doing only 1-pass CRF encodes is too unpredictable in terms of file size

[RaynQuist]

Now that is interesting: CRF easily outperformed 3-pass mode, because on even a 500-frame clip 3-pass was not only unable to hit the target bitrate, but got worse SSIM than CRF even though the bitrate was over the target!

Lax ABR (i.e. --ratetol inf) tries to be CRF. It adjusts the RF value periodically to approximate the given filesize, but only the first few scenes have really variable quality. After that the variations in RF are progressively damped.

I think that may be a bit of what's happening here. Dark Shikari your 500 frames might not be enough for abr to stabilize, even with 3 passes. On my 20000 frame sample there was not much variance in metrics between crf and all 3 passes. If anything I would say 2nd pass was better than CRF. It's anime though.


bitrate fps avgPSNR glbPSNR SSIM AvgIQP AvgPQP AvgBQP
CRF 20 683.53 8.37 47.690 47.213 0.9866877 17.11 18.84 21.01
1st pass 685.62 8.40 47.765 47.354 0.9872845 18.15 19.91 20.39
2nd pass 684.11 8.44 47.908 47.498 0.9873787 17.49 19.29 20.23
3rd pass 684.03 8.42 47.914 47.508 0.9873756 17.57 19.19 20.24

[Dark Shikari]

I think that may be a bit of what's happening here. Dark Shikari your 500 frames might not be enough for abr to stabilize, even with 3 passes. On my 20000 frame sample there was not much variance in metrics between crf and all 3 passes. If anything I would say 2nd pass was better than CRF. It's anime though.


bitrate fps avgPSNR glbPSNR SSIM AvgIQP AvgPQP AvgBQP
CRF 20 683.53 8.37 47.690 47.213 0.9866877 17.11 18.84 21.01
1st pass 685.62 8.40 47.765 47.354 0.9872845 18.15 19.91 20.39
2nd pass 684.11 8.44 47.908 47.498 0.9873787 17.49 19.29 20.23
3rd pass 684.03 8.42 47.914 47.508 0.9873756 17.57 19.19 20.24

Yeah, I would guess there is a length at which CRF is overtaken by ABR; it would be interesting to test this.

[MarcioAB]

CRF isn't perfectly optimal, but the optimal algorithm doesn't need 2 passes either.
Great. I like mystery :cool:
In the meantime, CRF is the champion (for the same bitrate, sure), not by far, but the champion.

Read up on Lagrange multipliers.Too dense for me (I tried on Wiki and was not able to understand even the "very simple example").

Your graphic shows good ideas, but it's not quite right. The curves should be hyperbolas curving away from the origin and never quite touching the axes, not circles.
Good. I'm interested to improve. If I understood you correctly the graph should be something like the graphic-B ?
http://pws.prserv.net/marcioab/compressibility2.gif
But, I do not agree the graphic-B represent the compressibility, but I like to discuss that.:o

EDIT: 2nd thought: You are very correct !! Great. Thank you.

[MarcioAB]

... If anything I would say 2nd pass was better than CRF. It's anime though.

RaynQuist could you post your command lines ? ( despite I use MeGUI ).

I would like to try it in 20000 frames of my MADAGASCAR video.

Thank you.

[akupenguin]

Good. I'm interested to improve. If I understood you correctly the graph should be something like the graphic-B ?
But, I do not agree the graphic-B represent the compressibility, but I like to discuss that.
Yes.
I am assuming that each line represents one video, and that "complexity" is an inherent property of the content, while the various points on the line come from the user's choice of bitrate/CRF values. Then the shape of the curves can be experimentally determined by encoding at various RFs, it's not up to anyone to decide. I was just telling you the result you'd get if you ran that experiment.
So if graph B doesn't represent complexity, then complexity can't be represented as a function of bitrate and RF.

[MarcioAB]

Yes.
I am assuming that each line represents one video, and that "complexity" is an inherent property of the content, while the various points on the line come from the user's choice of bitrate/CRF values. Then the shape of the curves can be experimentally determined by encoding at various RFs, it's not up to anyone to decide. I was just telling you the result you'd get if you ran that experiment.
So if graph B doesn't represent complexity, then complexity can't be represented as a function of bitrate and RF.

Not sure if I followed you ... but let me explain how I imagine this graph works ( the B one, that you proposed, and I guess is more correct now) :

A person compress a video "near the edge" using CRF and end up with a BPP. With this pair of coordinates (CRF, BPP) this person plot the point in the graph and concludes if the video is highly compressible, mid compressible or low compressible.
The 3 division lines are just for easy "classification" regard compressibility.

Need think how "complexity" relates with "compressibility" ...

[Terranigma]

p4x4 doesn't hurt, it just doesn't help. At large resolutions, individual objects in the movie are bigger than 4x4 pixels, so there's no point in partitioning mvs that small. Typically it might be enabled in 0.5% of macroblocks, and improve compression by 0.1%, making it one of the worst compression-per-cpu tradeoffs.

Well since p4x4 isn't really useful, would you say that an AVC Level of 4.1 is just as good, or almost as good as unrestricted?

With megui, these are the maximum settings I was able to use with a profile level of 4.1 without p4x4

--pass 3 --bitrate 100000 --stats ".stats" --level 4.1 --ref 16 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,-3 --subme 7 --trellis 2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-maxrate 50000 --b-bias 40 --me esa --merange 64 --threads auto --thread-input --cqmfile "Prestige CQM.cfg" --progress --no-dct-decimate --output ""ReallyInsaneSettings.mkv""

Of course i'd never use such settings, but I am quite curious as to what you think :cool:

[akupenguin]

@MarcioAB
In case it isn't obvious why graph A is ridiculous:
It says that lossless compression at 0.3 bpp is low compressibility? As is CRF28 at 0 kbps?

@Terranigma
There's no difference between 4.1 and 5.1 with regard to p4x4. Both forbid it. (To be precise: both forbid using p4x4 in all partitions of a macroblock at once, and I haven't bothered to support limited use of it.) There isn't any level that's a strict superset of all other levels.
I normally consider bitrate, vbv, dpb, and resolution to be the only important parts of level and ignore the rest, that's why I say 5.1 is the closest to unrestricted. But there are some other features that can only be enabled on lower levels.

[RaynQuist]

RaynQuist could you post your command lines ? ( despite I use MeGUI ).

I would like to try it in 20000 frames of my MADAGASCAR video.

Thank you.

--keyint 480 --min-keyint 12 --ref 16 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 7 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --merange 32 --threads auto --thread-input --progress --aq-strength 0.7