CRF translated to DABR graph and discussion

[DDogg]

I doubt this is anything new, but I'll flesh it out a little and see if anybody is interested in the topic. If this has been covered before please provide a link or two. I should warn that this stuff is all very tentative at the moment and needs much more testing. It is primarily just my old CCE stuff translated to x264 crf mode.

I spent some amount of time today translating a span of CRF 2% sample encodes into derived bitrate numbers (DABR). Much to my pleasure I found there seems to be a near perfect curve generated, and there also seems to be a strong possibility to use the Newton Ralphson convergence methods that are used to project CCE Q numbers into a predicted filesize.

Perhaps more importantly, it may provide a way to create branching logic in a program to allow the quality of a size constrained encode to be predicted before the actual encode is committed to.

As well this provides a replicable way to measure, quantify and predict the effect of various filters, matrices, etc. on final filesize.

Further, the accuracy of a 1% sample seems to be as effective as a 5% sample in my early tests. The DABR from the 1%, 2% and 5% were very close to each other.

There is a bitrate calculator spreadsheet attached to the post in my sig if anybody wants to replicate the numbers
and method used to create the graph. There may be some more information there that may be translatable for x264. Frankly I am just too new to x264 to know yet. Also, I have attached the spreadsheet data to the data post below if somebody wants to regraph it differently.

http://img96.imageshack.us/img96/734/graphda9.jpg (http://imageshack.us)

The cmdline used was:

C:\apps\x264.exe --crf 19 --ref 3
--mixed-refs --bframes 3 --b-pyramid --b-rdo
--bime --weightb --filter -2,-1 --subme 6 --trellis
1 --analyse all --8x8dct --vbv-maxrate 25000 --me
umh --threads 2 --thread-input --progress --no-psnr
--output "D:\temp\auto3\movienohrQ(xx).mp4"
"d:\temp\auto3\movie.avs"

The 2% range line was: SelectRangeEvery(600,12)

DABR conversion was done with - (((Sample_Size_In_Bytes*(100/Sample_Size_Percentage))*8)/1000)/(Total_Frames/Frame_Rate)



My plan is to set several encodes going tonight and see if the actual finished filesize ends up as predicted.


http://img96.imageshack.us/img96/734/graphda9.th.jpg (http://img96.imageshack.us/my.php?image=graphda9.jpg)

[foxyshadis]

Interestingly, although AVC is theoretically supposed to double in q steps of 6, this roughly doubles in q steps of 5, nearing 6 as it rises. I wonder if that has to do with crf, the options used, or just x264's implementation.

[akupenguin]

x264's bitrate predictor for 2pass assumes the step is 5.5, based on a similar measurement a long time ago.
My hypothesis is: The threshold for the magnitude of information to keep does double exactly every 6 qps. But the information in the source is not quite evently (well, laplacianly) distributed over the range of magnitudes. If there's a little more information in the noise end of the spectrum, which gets lost completely at high qp, then that increases the rate change of bitrate as a function of qp.

[DDogg]

All data below and attached in excel spreadsheet.

CRF 2% Size (BTs) DABR in KBPS Projected file size Actual Filesize dif1% dif2%

CRF15.mp4 32,951,805 2237 1,647,590,250 1,574,893,880 0.9559 4.41%
CRF16.mp4 28,046,445 1904 1,402,322,250 1,327,162,625 0.9464 5.36%
CRF17.mp4 24,013,257 1631 1,200,662,850 1,126,649,226 0.9384 6.16%
CRF18.mp4 20,662,192 1403 1,033,109,600 958,217,194 0.9275 7.25%
CRF19.mp4 17,846,928 1212 892,346,400 819,491,843 0.9184 8.16%
CRF20.mp4 15,469,190 1050 773,459,500 702,993,061 0.9089 9.11%
CRF21.mp4 13,472,796 915 673,639,800 605,788,784 0.8993 10.07%
CRF22.mp4 11,803,302 801 590,165,100 524,643,249 0.8890 11.10%
CRF23.mp4 10,338,978 702 516,948,900 456,028,249 0.8822 11.78%
CRF24.mp4 9,109,543 619 455,477,150 396,922,397 0.8714 12.86%
CRF25.mp4 8,012,111 544 400,605,550 346,629,200 0.8653 13.47%
CRF26.mp4 7,091,370 482 354,568,500 303,524,399 0.8560 14.40%
CRF27.mp4 6,263,265 425 313,163,250 266,540,478 0.8511 14.89%
CRF28.mp4 5,557,481 377 277,874,050 234,993,050 0.8457 15.43%
CRF29.mp4 4,923,517 334 246,175,850 207,760,736 0.8440 15.60%
CRF30.mp4 4,391,764 298 219,588,200 183,939,507 0.8377 16.23%






http://img504.imageshack.us/img504/6796/predrea5od0.jpg (http://imageshack.us)

[akupenguin]

Did all your tests use the same 2% of the movie? What's the distribution of projected sizes if you run one Q lots of times with different random 2%s?

[virus]

Interestingly, although AVC is theoretically supposed to double in q steps of 6, this roughly doubles in q steps of 5, nearing 6 as it rises. I wonder if that has to do with crf, the options used, or just x264's implementation.
Also the type of source plays an important role. I've seen several cases where the "-6 QP -> 2x rate" rule of thumb holds decently (say, within 10-15%), but also a case where going from QP 16 to QP 10 brought a 3.8x increase in bitrate. I have no idea whether CRF's rates would differ significantly from constant QP in that specific case, though.

[DDogg]

Did all your tests use the same 2% of the movie? What's the distribution of projected sizes if you run one Q lots of times with different random 2%s? Sorry, I don't completely follow your question. Presently I am just using the method of sampling we always used for CCE that turned out to be reliable in all regards. There was never a need for anything else than the select range statement above.

Whether that holds true for x264 crf mode I really don't know yet. I do remember jonny and I having long communications several years ago about xvid CQ and the same principles did not hold up well at all. The graphs were all over the place. He an I both dropped out about that time and I don't know if he ever pursued that aspect of xvid compression again. I did send him a pm asking him to check in to this thread as this stuff should be second nature to him.

Primarily I am only attempting to do some grunt level labor in the hopes it might provide some of you braniacs with usable date and stimulate conversation and ideas.

If you would like a set of tests run, please provide the commandline you would like used, a base level sampling script/method preferred and your specifications for the test. I'll get it done and collate the results back to you.

Doing some of the grunt labor for some of you is little enough to repay the work you have done with x264.

Btw, none of this has any validity until I do multiple sources and see if the curves track in parallel. Even with a Wcooled D930 running at 4.5GHz the time required is scary.
Also the type of source plays an important role. I've seen several cases where the "-6 QP -> 2x rate" rule of thumb holds decently (say, within 10-15%), but also a case where going from QP 16 to QP 10 brought a 3.8x increase in bitrate. I have no idea whether CRF's rates would differ significantly from constant QP in that specific case, though. That was always the beauty of Using 1P Q in CCE, or I am assuming any constant quality mode in any encoder, as the source complexity completely dictated the bitrate required to "hold" the complexity of the source (within certain caps). That was the key for us back then as the sample size, translated to DABR, allowed a fingerprinting of the source complexity and thus an extrapolation of the size/bitrate needed in advance of actually doing the encode. As well, it allowed one to estimate the effect on bitrate demand by various matrices, filters, and would clearly show even something as simple as the minute effects of DC precision. So incredibly useful. Guess that is why I am hoping some of this might pan out for x264.

[IgorC]

Sorry, I don't completely follow your question. Presently I am just using the method of sampling we always used for CCE that turned out to be reliable in all regards. There was never a need for anything else than the select range statement above.
For different 2% pieces of movie filesize can variate siginifically. i.e 2% of chapter 10th may have different filesize from 2% of chapter 20th due to different motion and another factors. It's a probalistic task. Maybe it would be better to imply estadistic tools for better estimation of the results. I.e. confidence intervals 0.9-0.95. It should be realible.

[DDogg]

For different 2% pieces of movie filesize can variate siginifically. i.e 2% of chapter 10th may have different filesize from 2% of chapter 20th due to different motion and another factors. It's a probalistic task. Maybe it would be better to imply estadistic tools for better estimation of the results. I.e. confidence intervals 0.9-0.95. It should be realible.Yes, certainly various 2% slices would demand different bitrates as the complexity of that 2% slice would dictate the bitrate, but we are speaking of the SelectRangeEvery(600,12) creating an average of all the frames in the source.

I remember huge amounts of discussion on this topic because any intelligent person might doubt the ability of a 1 or 2% sampling of all the frames to accurately average the bitrate. However, after literally 10s of thousands of test on various sources we found it accurate to +-2% consistently and popular programs like DVDRebuilder use these techniques every day now. That's just one of those things that ended up working that people swore would not work. It did and nobody argues about it anymore as it relates to the process with CCE.

Whether it works at all with x264 crf mode is a whole other question. All we can do is try and see. If you have a method of sampling you think would work well, please share it. I can do a controlled set of tests. BTW, I don't have any idea what estadistic tools are.

[akupenguin]

If that's how you sampled it, then I have no trouble believing that it could make a good predictor once you compensate for the systematic errors.
A trivial explanation for why it overestimated the filesize: the test pass is limited to GOP size 12, whereas the final pass uses much larger GOPs. Solutions: (1) Use larger ranges (but then your test wouldn't sample the input as evenly). (2) Estimate the final average GOP size and downweight the I-frames when counting the bitrate of the test pass. In my experience the average GOP is around 100 when unrestricted, or you could try to measure it based on the number of I-frames that are not on the border between ranges (though x264's biased scenecut detection might interfere with that).
Several other options are affected similarly, though the magnitude of error introduced by those are smaller: B-adapt's decisions are also constrained by the GOP size, and multiref is less effective for the first few frames of a GOP.

Edit: I'm making this more complicated than it needs to be. To compensate for the extra I-frames, just delete the first frame in each range. The actual number of scenecuts in the movie will be correctly represented by the remainder of the frames. (This doesn't fix B-frames and refs, though.)

I assume the 5% sample was SelectRangeEvery(240,12)? Then the next test to run is SelectRangeEvery(600,30) and so on.

[DDogg]

If that's how you sampled it, then I have no trouble believing that it could make a good predictor once you compensate for the systematic errors.
A trivial explanation for why it overestimated the filesize: the test pass is limited to GOP size 12, whereas the final pass uses much larger GOPs. Solutions: (1) Use larger ranges (but then your test wouldn't sample the input as evenly). (2) Estimate the final average GOP size and downweight the I-frames when counting the bitrate of the test pass. In my experience the average GOP is around 100 when unrestricted, or you could try to measure it based on the number of I-frames that are not on the border between ranges (though x264's biased scenecut detection might interfere with that).
Several other options are affected similarly, though the magnitude of error introduced by those are smaller: B-adapt's decisions are also constrained by the GOP size, and multiref is less effective for the first few frames of a GOP.

I assume the 5% sample was SelectRangeEvery(240,12)? Then the next test to run is SelectRangeEvery(600,30) and so on. Here is the derived bitrate spreadsheet (http://forum.doom9.org/attachment.php?attachmentid=2924&d=1081970852) I use and the select lines are listed at the bottom which brings up a math question (mine are 1970s US high School=poor). Just for conversation's sake, let's assume the above curve holds true (big stupid assumption now). Would you, or others with good math skills, have a suggestion on how one would take the formula (((Sample_Size_In_Bytes*(100/Sample_Size_Percentage))*8)/1000)/(Total_Frames/Frame_Rate) and add to it a curve correction factor based upon the above chart or data points?

As for your point # 2 above, seems to me that would take a small program to accomplish?

Graph and data points updated above

[foxyshadis]

Are you keeping logs of the x264 output? Try comparing the I-frame size as a % of the total size in the test vs the final, and subtract the difference from the sampled size.

So if in the sample, there I frames are 478632 bytes and 40%, but in the final they're 9042673 and 25%, you subtract 179478 and predict from what's left. ([average full %]/[obtained %]*size will net you the amount, or close enough to it for this.) You'll probably have to make tests to figure out what the average I-frame size in % is in movies. Or you might just be able to find an average obtained % as well and not need to look at logs at all.

Since you're not writing software that can analyse the stream and chop the first two and last two frames of each gop off, which is how the most accurate mpeg-2 and mpeg-4 results seem to come about, this should be the next-best way.

[DDogg]

foxyshadis, that sounds like a very intelligent thing to try. Problem is I am not smart enough, and I did not keep logs. Hopefully you might feel like dipping a toe in this water as I could certainly use some help from somebody like you.

I'm going to take the simple and dumb route with another experiment. I have another source of 150,804 frames with much lower complexity (to my eye) than the previous more complex source of 141,241 frames. I've run the 2% tests and using the same method above, size predicts out at 1,246,861,350 for crf 15 and 160,868,500 for crf 30. Btw, I was pleased to see this as it has more frames but clearly the 2% sample is getting a good fingerprint of the complexity and indicates it will come in smaller in size, which seems right to me.

If the curve holds and I apply the correction of -4.41% and -16.23% respectively they should come in at 1,191,846,279 and 134,752,562 (using identical script and commandline). I would be surprised if they do hit the size, but it is an experiment that has to be done to see.

Will post back whatever the results are.

/add:

Results are encouraging -
CRF 30 - expected 134,752.562 got 128,583,054 which is -4.6% shy of the target
CRF 15 - expected 1,191,846,279 got 1,194,073,460 WOW! I'll take that anyday.

Will run several more toward the belly of the curve:

/add:
These did not work out as well [+-2% is desired]-
CRF 18 - expected 703,308,561 got 684,558,029 - 97.3% of predicted
CRF 20 - exxected 509,203,536 got 489,826,680 - 96.2% of predicted
CRF 22 - expected 377,038,516 got 360,203,986 - 95.5% of predicted

[xyloy]

If you use x264 via MeGUI, logs are usually kept in the program files\megui\logs subdirectory.

BTW, this is a very interesting topic. ;)

[DDogg]

If you use x264 via MeGUI, logs are usually keeped in the program files\megui\logs subdirectory.

BTW, this is a very interesting topic. ;) Thanks, xyloy. I think there is a fair amount of potential in the subject, but I also think the advice of akupenguin and foxyshadis is going to have to be incorporated to take it anywhere. I don't think I can get there on my own.

Still, although super accurate prediction has not been demonstrated (so far) with these very simple methods used, there certainly seems a potential to use them as a reality check before doing a bitrate based encode as well as generally checking the compressibility of a source. Given the slower speed of x264, this might be helpful to some. My hope is this thread will stimulate more advanced members of this forum to take the subject farther.

@akupenguin - just noticed your edit - "...To compensate for the extra I-frames, just delete the first frame in each range. The actual number of scenecuts in the movie will be correctly represented by the remainder of the frames. (This doesn't fix B-frames and refs, though.).." My brain is fried, please suggest the modified SelectRangeEvery statement :)

Hmmm, wondering if a PredictCRFsizefunction.avsi might accomplish this task better? Function writers out there?

[jonny]

please suggest the modified SelectRangeEvery statement

the problem are bad (statistically) frames introduced with SelectRangeEvery
you should think that you are serving to the encoder little snips of the original source (it's full of jumps in different parts of the movie), this introduce abnormal sized frames (at the start and at the end of the snip)

the only way to handle this is using the log with all the frame sizes, summing sizes but discarding bad frames (this will give you a better value, compared to raw output size)

with mp4-asp a good way to handle bframes is discarding 3 frames at the start and 3 frames at the end of the snip

i haven't done testing with avc, i've found a discussion here:

http://forum.doom9.org/showthread.php?p=598896&highlight=x264+prediction#post598896
(i'm trying to see where it ends : )

[foxyshadis]

I don't think he means modifying the avisynth, I'm pretty sure that means chopping the first frame of each segment off in the output. To do that you can modify x264's output module or use a stream parser to ignore every 12 frames, but it's definitely not something you can gather from the basic info you're using now.

Oh, I just noticed that min-gop is still 25, so it's not just every 12 frames... hmm, interesting. Might still be I frames though. I guess I'll have to run a similar test and parse it to check.

[akupenguin]

No need to actually modify the compressed stream. Just ignore the appropriate lines from `x264 -v` output.
And it doesn't matter what min-gop is set to. An I-frame takes almost the same number of bits as a P-frame full of I-blocks. Except for the keyframe qp bonus, but that won't really be correct either way.


#!/usr/bin/perl
$range_length = 12;
$fps = 24000/1001;
while(<>){
/frame= *(\d+).*size=(\d+)/ or next;
$1 % $range_length or next;
$size += $2;
$frames++;
}
printf "%.3f\n", $size*$fps*.008/$frames;

...but this tends to overcompensate a bit. Maybe because the keyframes still got their qp bonus and thus made better references but their bit cost was ignored. Or maybe the converse: the real scenecuts didn't get detected as such and didn't get the qp bonus, so didn't increase the bitrate as much as in the final encode.

[Hellworm]

@akupenguin - just noticed your edit - "...To compensate for the extra I-frames, just delete the first frame in each range. The actual number of scenecuts in the movie will be correctly represented by the remainder of the frames. (This doesn't fix B-frames and refs, though.).." My brain is fried, please suggest the modified SelectRangeEvery statement :)

I think he meant that you should ignore the size of the I-Frames of every cut (analysing the log) and completely ignore those frames as the scenecut was introduced by the selectrange anyway

damn beaten to it :D

[DDogg]

jonny, thanks for posting that link. A general comment, somewhat on topic with the recent VFW thing, is how they are trying to work around all the problems of vfw, vdubmod, etc whereas the modern cli version, if the author chose, could probably generate a specialized log, or for that matter, even have a predictive method built in without all the nonsense of the workarounds. At least, I think this sounds like it may be true. akupenguin, is that theoretically possible, or am I misunderstanding some of your comments?

[Hellworm]

But x264 does no decoding. It could be possible to implement an ultra fast first pass, but the decoding of the remainig 98% had to be done.
So the cuts have to be done one level higher.

[akupenguin]

x264cli with avisynth input can choose which frames to request from avisynth.
piped input (e.g. from mencoder) would still have to decode everything.

[DDogg]

... hmm, interesting. Might still be I frames though. I guess I'll have to run a similar test and parse it to check. I hope you have the time and interest to do that.

Which brings up a question, Is it effective to substitute --crf and a script containing the selectrangeevery statement when creating the stats file? Or, put another way, what is the method you folks would suggest for creating a log that would provide the specific information for the specific purpose we have been discussing? x264cli with avisynth input can choose which frames to request from avisynth.
piped input (e.g. from mencoder) would still have to decode everything.Could you elaborate a little more? I'm trying to understand whether you are saying something additional should be/could be added to x264 cli to better accomplish what we are discussing, or if you are suggesting that is a dead end and using the log information is the direction you suggest be pursued.

Probably best for me to go silent on this stuff as I leaped from mpeg2 to x264 without all the intermediate xvid log stuff and some of the things mentioned here are clearly over my head at this point. I can do some of these long methodical test fairly well, so let me know how I can help in that regard.

[emmel]

I think the answer was pretty clear: the encoder can ask avisynth to seek, decode, and deliver any particular frame it needs, e.g., every 50th frame (check muxers.c for details). It is then up to Avisynth to do what it has got to do - probably it will quickly skip some gops before starting to decode the frame the encoder is looking for. No need to always decode all frames. In theory, that is.

[DDogg]

I thought I would close my part of this discussion with a short "How to" for people that want a quick reality check of potential quality before doing the full 2 pass bitrate based encode. Certainly this is just a rule of thumb method, but it might be useful for some.

1> Temporarily add to the bottom of your script - SelectRangeEvery(600,12) (remove before doing final encode)
2> Use a crf commandline based upon the CQ-ASP_Q2_eq(crf) profile and encode the sample
3> Select properties of the sample rendered and get the size (top one, not "size on disk")
4> multiply (size X50) X .9275
5> Not an exact science, but If that size is more than, say 5% larger than your planned size you most probably will need to take some action before doing the full encode. That might be increasing size allocated, adding a light filter to increase compression if that is acceptable to you (maybe fluxsmoothT), lowering resolution, using a different matrix, etc. Just about any change you make to the script will show a reaction in the sample filesize.

If you want to translate the sample size into a projected bitrate to compare to the one you plan, then use the spreadsheet linked above. You will need your sample filesize, framerate, and the number of total frames. That will translate the sample size into a bitrate you can use for comparison.

Another way to sort of accomplish the same thing is to use a newer version of AutoMK - choose automatic resolution, check allow multiple resize, and input the desired crf (advanced tab - suggest 18, 19, or 20). If AutoMK reduces the resolution, it indicates you might need to take some of the steps in #5 above. You can change the script, matrix, filter, etc and tell AutoMK to redo the prediction until it shows no resolution decrease - then go ahead with the encode. This is a new feature in AutoMK and I imagine it is still a work in progress and will continue to be tweaked, so keep that in mind.

Again, I am not suggesting this is anything more that a quick reality check method. Given the many hours a 2 pass x264 encode can take, having some idea of where you will end up might be useful for some of you, especially if your machine is not a dual core.

Still hoping members with more advanced skills might move this discussion forward.

[DDogg]

Hmmm, may have stumbled upon something useful -

1> The sample size and final filesize from a previous encode (-crf 18) were known. 15,165,556 bytes and 684,558,029 bytes
2> A new sample was run with with same cmdline and script except FFT3DFilter(sigma=3) was added to the script after the resize.
3> New sample size = 12,649,854
4> Divided smaller sample/larger sample = 0.834117391
5> Took previous encode size, 684,558,029 x 0.834117391 = predicted size of 571,001,757 bytes
6> Ran full encode and got 566,691,876 bytes which is 99.25% of the predicted filesize

So, while the ability to predict the final final size of a first encode may not be all that compelling [-+6%] [given the Iframe thing], the ability to predict the effect of a filter on final filesize is proving to be accurate using a simple 2% sample created with SelectRangeEvery(600,12). Many more tests using various matrices and filters on several different type of sources will have to be run to have any sense of confidence. Certainly, it seems worth pursuing.

/add1:
Replicating the above, substituting the crf 20 data (instead of the crf 18 data above), shows a similar result:
Predicted filesize = 430,242,978 b, actual 427,270,410 b, 99.3% on target

/add2: Looking beter and better. I used the more complex source the graph data was made from.

1> Original crf 18 sample and encode was 20,662,192 b, 958,217,194 b respectively
2> New sample using FFT3DFilter(sigma=3) was 17,631,504
3> 17,631,504 / 20,662,192 = 0.853322048
4> 958,217,194 x 0.853322048 = 817,667,859 b predicted
5> Actual re-encode was 818,251,893 which is 99.92% on target

/add3: the crf 20 and 22 filter tests came in as 99.1% 98.1% of the predicted final filesize

[708145]

1> Original crf 18 sample and encode was 20,662,192 b, 958,217,194 b respectively
2> New sample using FFT3DFilter(sigma=3) was 17,631,504
3> 17,631,504 / 20,662,192 = 0.853322048
4> 958,217,194 x 0.853322048 = 817,667,859 b predicted
5> Actual re-encode was 818,251,893 which is 99.92% on target

Very good results indeed.
Now another interesting aspect would be how well you can predict the final size when varying crf and the filter(s).
This would allow for an encode mode where the analysis pass finds the crf parameter together with filters (which filters and parameters) which yield the final size (or very close to it) in just one pass.

This allows for heavy filtering being done only one time (*) for encoding without the need to filter and encode to lossless in advance. Encoding from HDTV lossless is harddrive limited for me at least so it could give some significant speedup to the encode chain.

If the analysis amounts to less than 10% of the final encode the name "1.1 pass encoding" seems appropriate.

bis besser,
T0B1A5

(*) one time + the x% for the analysis pass (pass 0?)

[check]

I have to say, this is turning out to be an extremely interesting thread. One query, has anyone tried using a different min GOP size? As someone (foxyshadis?) said before, the default is 25, and the estimates are coming out slightly too large. Would increasing the min GOP size affect this?

[DDogg]

Very good results indeed.
Now another interesting aspect would be how well you can predict the final size when varying crf and the filter(s).
This would allow for an encode mode where the analysis pass finds the crf parameter together with filters (which filters and parameters) which yield the final size (or very close to it) in just one pass. ...There are separate sections to this stuff -

1> Convergence of a crf value, via sampling, to yield a final filesize, and actually using crf mode for the final encoding does not seem viable as there is no decimal available in crf mode. Note the difference between crf 18 and crf 19 in the graph data in the first posts. It is 138 megs. Not that useful without smaller decimal values like crf 18.5, 18.6, etc.

2> That brings us to some type of bitrate extrapolation using the DABR fingerprint to gauge the compressibility of a source and estimate if a bitrate calculated to hit a size target will allow enough headroom to contain the source complexity and yield some equivalent of the known quality of the crf samples.

That seems viable only if prediction software is written to properly parse the log information into the calculations. My gut tells me the size prediction could get close to the -+ 2% area. Without the log information being used, the simple methods used here would be more like a +- 6% (not even close to being sure of that, yet) [Still, +- 6% may be ok for a quality reality check since we are going to actually use a bitrate based encode for the final encode which will guarantee size accuracy]

3> On the positive side, even if we declare final filesize as an unknown x, the initial filter tests show we can accurately predict that b = x* (sample1/sample2). To put that another way, we know the filtered final filesize is going to be a known percentage [different] than the unknown unfiltered filesize (sounds kind of funny). We can easily arrive at that percentage by using only two quick 2% samples (using the same crf cmdline). This seems useful and could provide GUI programs, or a user, with the logic needed to make more accurate filtering decisions and save a hell of a lot of wasted encoding time.... One query, has anyone tried using a different min GOP size? As someone (foxyshadis?) said before, the default is 25, and the estimates are coming out slightly too large. Would increasing the min GOP size affect this? Good question and hope we can get an answer to it. [I should add the fact that once I started using the cmdline mentioned way above, I was locked in to it, as changing would have invalidated the entire data set]

/add1: I need to do the #3 tests with and without a custom matrix to see if that would show the same accuracy of prediction.
/add2: Slight edits to #3 because we want a multiplier. If the filtered sample is larger we would want to divide larger/smaller to give us a 1.x multiplier. If filtered sample is smaller we would want smaller/larger to get a 0.x multiplier. I think I got that right.

[708145]

1> Convergence of a crf value, via sampling, to yield a final filesize, and actually using crf mode for the final encoding does not seem viable as there is no decimal available in crf mode. Note the difference between crf 18 and crf 19 in the graph data in the first posts. It is 138 megs. Not that useful without smaller decimal values like crf 18.5, 18.6, etc.


1> fractional crf is not impossible. Maybe Loren can comment.

bis besser,
T0B1A5

[DDogg]

> fractional crf is not impossible
We can hope.

Looks like the prediction holds up well for matrices, in this example the sample size is larger due to the effects of eqm_avc_hq -

1> Original crf 18 sample and final was 20,662,192 b and 958,217,194 b
2> New crf 18 sample with eqm_avc_hq was 24,221,744 b
3> 24,221,744 / 20,662,192 = 1.172273687
4> 958,217,194 x 1.172273687 = 1,123,292,803 b
5> Re-encode with matrix was 1,109,699,135 b
6> 1,109,699,135 / 1,123,292,803 = 0.987898375
7> 98.8 % on target

/add: crf 20 w/matrix - 99.1% of predicted target (832,261,957 b actual / 839,735,870 b predicted)

[ChronoCross]

he added fractional crf in the latest revision =D

r591 | pengvado | 2006-10-13 22:04:58 +0200 (Fri, 13 Oct 2006) | 2 lines
Changed paths:
M /trunk/common/common.c
M /trunk/encoder/encoder.c
M /trunk/encoder/ratecontrol.c
M /trunk/x264.c
M /trunk/x264.h

allow fractional values of crf

[DDogg]

he added fractional crf in the latest revision =D

r591 | pengvado | 2006-10-13 22:04:58 +0200 (Fri, 13 Oct 2006) | 2 lines
Changed paths:
M /trunk/common/common.c
M /trunk/encoder/encoder.c
M /trunk/encoder/ratecontrol.c
M /trunk/x264.c
M /trunk/x264.h

allow fractional values of crfThanks! Looking forward to playing with it when Sharktooth gets the build up. [Found it on x264.nl]

[DDogg]

First plot using fractional values of crf in x264 r591 - Same source, x264 cmdline and script used in the original data post. Huge thanks go to the x264 team for implementing fractional values of crf!

http://img214.imageshack.us/img214/3256/source1wj9.jpg (http://imageshack.us)
CRF Sample Size in bytes Dabr Projected filesize

CRF17.0.mp4 24,013,258 1,631 1,200,662,900
CRF17.1.mp4 23,659,721 1,607 1,182,986,050
CRF17.2.mp4 23,316,194 1,583 1,165,809,700
CRF17.3.mp4 22,971,897 1,560 1,148,594,850
CRF17.4.mp4 22,622,602 1,536 1,131,130,100
CRF17.5.mp4 22,246,102 1,511 1,112,305,100
CRF17.6.mp4 21,896,714 1,487 1,094,835,700
CRF17.7.mp4 21,597,862 1,467 1,079,893,100
CRF17.8.mp4 21,274,419 1,445 1,063,720,950
CRF17.9.mp4 20,983,766 1,425 1,049,188,300
CRF18.0.mp4 20,662,193 1,403 1,033,109,650
CRF18.1.mp4 20,367,235 1,383 1,018,361,750
CRF18.2.mp4 20,083,724 1,364 1,004,186,200
CRF18.3.mp4 19,786,368 1,344 989,318,400
CRF18.4.mp4 19,490,543 1,323 974,527,150
CRF18.5.mp4 19,181,005 1,302 959,050,250
CRF18.6.mp4 18,900,673 1,283 945,033,650
CRF18.7.mp4 18,633,764 1,265 931,688,200
CRF18.8.mp4 18,368,735 1,247 918,436,750
CRF18.9.mp4 18,112,083 1,230 905,604,150
CRF19.0.mp4 17,846,929 1,212 892,346,450
CRF19.1.mp4 17,593,500 1,195 879,675,000
CRF19.2.mp4 17,337,434 1,177 866,871,700
CRF19.3.mp4 17,098,223 1,161 854,911,150
CRF19.4.mp4 16,864,579 1,145 843,228,950
CRF19.5.mp4 16,595,722 1,127 829,786,100
CRF19.6.mp4 16,372,733 1,112 818,636,650
CRF19.7.mp4 16,134,694 1,096 806,734,700
CRF19.8.mp4 15,935,226 1,082 796,761,300
CRF19.9.mp4 15,730,880 1,068 786,544,000
CRF20.0.mp4 15,469,190 1,050 773,459,500
CRF20.1.mp4 15,287,690 1,038 764,384,500
CRF20.2.mp4 15,069,438 1,023 753,471,900
CRF20.3.mp4 14,866,851 1,009 743,342,550
CRF20.4.mp4 14,656,212 995 732,810,600
CRF20.5.mp4 14,425,248 979 721,262,400

[Didée]

"1.1 pass encoding"
Yeah! :)

It's already years since I started thinking about the possibility to do something like this with XviD (by encoding & logging an "x% Snip", then doing one full pass by feeding a modified 'CBR'-module with local bitrates interpolated from the x% stats.)

So we had to wait for x.264 to get it, okay. ;)

When more tests have been done and a sufficient formula formulated, getting pretty close to a desired size should be possible with two+one steps:

1a) one initial 2% encoding with some starting CRF value (educated guess) --> predict an CRF value for destination size from this

2) 3 (or 5 or 7) % encoding with CRF from 1) --> predict new CRF

3) full encoding with CRF from 2).

Would estimate that this should come in pretty close.

[DDogg]

Hi, Didée - good to see the Wizard of Scripts is interested with this topic and apologies to 708145 as I realize I did not understand his 1.1 reference.

Having a little trouble understanding your point #2 - I think you are suggesting we would predict the size of the 5% sample from the size of the 2% sample and apply any error as a correction factor? Would appreciate you expanding.

Just had a funny thought - It would not surprise me if you write one of your crazy complex scripts to parse the log and predict final crf size :) Hell, I think you could create a word processor from an avisynth script!

Oh, btw, happy to grind out some data for you. Let me know what would be helpful.

[buzzqw]

and i am very interested on this 1.1 encoding...

could a simple binary iteration be ok ? or must be used the obfusched RoBa method ?

BHH

[Didée]

It's kind of a correction factor, yes. Doing the final pass directly with the CRF value predicted from one 2% encoding surely is not that bad, but will leave quite some error margin on the final size ... depending on how representative this small snip has been in fact, on how far the initial chosen CRF value has been away from the really needed CRF value, and also on how exactly different quantization will react to the characteristics of the particular source.

Mathematicians are used to the "nested intervals" procedure, and that's what I was suggesting, with a nesting depth of "2". :)
The small snip encoding is used to get a closer-to-the-truth CRF value. When now a longer snip is encoded with this refined CRF value, the size prediction should be good.

BTW, Avisynth scripting is easy. In you get values in range [0,255], out you put values in range [0,255]. Nothing spectacular. :D

[DDogg]

Some clarification notes for the casual reader of this thread. The new capability of fractional crf has considerably altered the direction of the original post, and added new capabilities. There is a potential for confusion as two very different methods, sharing a common sampling process are being discussed. To rehash:

1> The addition of fractional crf has presented the possibility to use some of the known convergence techniques to potentially calculate a crf to achieve a size target. This was not practical before, because without fractional crf, the final size between, say crf 18 and 19, was too large to be useful (138 mb in my test source). The addition of fractional crf (18.1,18.2,etc.) has changed this. The crf encoding method would be used for the final encode.

2> A hybrid method using a constant quality crf sample to fingerprint the complexity of a source by deriving the bitrate from the byte size of the resulting sample. This derived bitrate is used to suggest a bitrate appropriate for bitrate based encoding methods tailored to the source complexity. A bitrate based encoding method, 1P or 2P, would be used to do the final encode.

Two seemingly similar, but quite different approaches, each with a potential for being useful depending on your preferred method of encoding.

---------------------------------------------------------------------------------------------------------------------------

... Mathematicians are used to the "nested intervals" procedure, and that's what I was suggesting, with a nesting depth of "2". :) The small snip encoding is used to get a closer-to-the-truth CRF value. When now a longer snip is encoded with this refined CRF value, the size prediction should be good. ... Hmmm, clearly I ain't a Mathematician :) Perhaps consider showing us a more 1,2,3 approach to replicating your thinking with the computations involved? I can normally get that. Then, I could do some methodical tests and provide you with the data. Maybe that would be useful to you? Let me know.

[Didée]

Okay, 3rd try, then.

Assume we want a certain destination size, and the closest to this we could ever get by CRF encoding would be with CRF 25.7. So, 25.7 is the target value that we want, but that we don't know yet.

How close the prediction of a snip encoding will be to the final size, depends on:

- difference between snip's CRF value and final CRF value
(if target is 25.7, prediction from a CRF20 snip encoding will be worse, prediction from a CRF24 snip encoding will be better.)

- prediction from a bigger snip will be better than that from a smaller snip, because more content is encoded, hence the longer snip is more representative.

So. If you just do a 2% snip with, say, CRF22, you might get somewhere in the destination region, but still be off a good bit. Prediction might be like eg. CRF 24.9.

You could get somewhat closer by just doing a bigger snip with CRF22, but it might still be off by too much, because the relatively big step 22.0 -> 25.7.
(Basically, the final destination value is completely unknown, but you have to use something for the [first] snip encoding.)

So, my suggestion simply is to
- first do the short snip with CRF22
- get e.g. CRF 24.9 as predicted result
- do a longer snip, like 5% or 7%, with CRF24.9
- get a prediction like 25.6 ~ 25.8 (hopefully)
- do the final CRF encoding with this CRF value

Much words for explaining a simple thing. :D
(For an even more verbose explanation, I'd have to use German language, I fear... ;) )

Probably the "RoBa method" is not too far from this (or vice versa) ... I can't tell because, you may laugh about me, the only thing I know about "the RoBa method" is its name ... I never have been deeply into Mpeg-2 encoding.

[akupenguin]

Another possibility:
1st pass (2%?) with ABR
2nd pass (5%?) with CRF, using the value of "final ratefactor" from the 1st (adjusted for I-frame bias).
3rd pass adjust the CRF value and do the real encode.

And another:
1st pass runs the analysis part of ABR (SATD complexity measurement and B-adapt) on the whole movie, but only encodes a small fraction.
You now know the exact SATD distribution over the movie, but have only an estimate of the mapping from SATD to bitrate at any given QP. Use this estimate to create a fake 1st pass statsfile.
2nd pass is a normal --pass=2 encode, except using the SATD data instead of a real 1st pass. It will adjust the SATD->bits estimate as it goes, and can compensate for any mispredictions just like it does with the existing fast-1st-pass.
Note: despite the 100% coverage of the 1st pass, this method won't spend any more cpu-time in x264 than the 2% method, since it just moves part of the CRF computation to the 1st pass. But it will have to decode the input twice, instead of 1.02x.

[akupenguin]

could a simple binary iteration be ok ? or must be used the obfusched RoBa method ?
afaict from a few minutes search, the RoBa method is equivalent to 1st pass CRF+VBV, then 2nd pass at the same bitrate? No, that's already possible but doesn't provide the speed of the methods discussed here.

[DDogg]

@Didée - Oh, I think I understand you now. You were simply speaking of the basic convergence method used to calculate the crf to hit a size?. If so, a derivative of this base method (Newton-Raphson) was used for cce: http://www.sosmath.com/calculus/diff/der07/der07.html

But that will not work for x264 because of the I-Frame thing. The logged frame information has to be incorporated. Like jonny said, "... sadly this stuff require programming skills in order to parse the log ..."

There is some more information about that on page 1. /add:

This is the major reason I shifted to the hybrid approach for x264. Sure, we all know a 1P constant quality mode is the fastest method of encoding, but with xvid and x264 the frame structure throws a monkey wrench into accurately predicting the crf within a tight +-2% range.

Another possibility:
1st pass (2%?) with ABR
2nd pass (5%?) with CRF, using the value of "final ratefactor" from the 1st (adjusted for I-frame bias).
3rd pass adjust the CRF value and do the real encode.

And another:
1st pass runs the analysis part of ABR (SATD complexity measurement and B-adapt) on the whole movie, but only encodes a small fraction.
You now know the exact SATD distribution over the movie, but have only an estimate of the mapping from SATD to bitrate at any given QP. Use this estimate to create a fake 1st pass statsfile.
2nd pass is a normal --pass=2 encode, except using the SATD data instead of a real 1st pass. It will adjust the SATD->bits estimate as it goes, and can compensate for any mispredictions just like it does with the existing fast-1st-pass.
Note: despite the 100% coverage of the 1st pass, this method won't spend any more cpu-time in x264 than the 2% method, since it just moves part of the CRF computation to the 1st pass. But it will have to decode the input twice, instead of 1.02x.

The first was what I sort of thought Didée was speaking of. The second sounds very intriguing [in fact, it sounds brilliant!]

afaict from a few minutes search, the RoBa method is equivalent to 1st pass CRF+VBV, then 2nd pass at the same bitrate? No, that's already possible but doesn't provide the speed of the methods discussed here.

Not meaning to act as class historian, but I do have some background. We might want to forget the term Roba as it is misused and means different things to different people (camps).

Instead, think of it as Conditional 2-pass Bach. The first pass was simply what we are speaking of here. 1 or 2% samples were encoded with a constant quality method (cce opv) and Newton-Raphson was used to calculate the appropriate "Q", in this case crf, to hit the size target. It could do that with only 2 -4 samples normally. The constant quality encode was run with this calculated Q while keeping a log of the frames (VAF file). [B]IF the file was on target then the encode was finished - ELSE, a second bitrate based pass was run, using the previously generated VAF log, to hit the size target. Very seldom was this needed - in fact it was rare.

[buzzqw]

@akupenguin
my bad, i mean the Newton-Raphson iteration ( as build, for example by Tylo in ccefront)

BHH

[check]

I've heard a few times that a crf which comes out to bitrate x will not be as high quality as a full 2 pass to the same bitrate (assuming all other settings are equal). Is there any truth to this?

[buzzqw]

if CRF to be used is about 18 (or less) then quality is near maximized. If CRF used is more than 22 than a 2pass should be better, because of better distributing bitrate (where needed 18 when not needed 22 or less.. for example)

BHH

[akupenguin]

I've heard a few times that a crf which comes out to bitrate x will not be as high quality as a full 2 pass to the same bitrate (assuming all other settings are equal). Is there any truth to this?
Yes, but the difference is pretty small. (details tomorrow)

if CRF to be used is about 18 (or less) then quality is near maximized.
No. You just mean that once quality is high enough, you can't see the further improvement. But 2pass still lets you achieve transparency at a slightly lower bitrate.

[DDogg]

Change/: Removed data because of error - re-running tests

[DDogg]

Yes, but the difference is pretty small. (details tomorrow)
Sorry, curiosity is getting the better of me. Does this mean you are tinkering around with some of the ideas you mentioned above, or that you simply did not have the extra time available to get into more detail?

Btw, I'm convinced all developers must take a secret handshake course called "How to speak Cryptic-101" :) Or, maybe it is "How to post short and succinct thoughts-101" - Clearly, the second was not offered in my school.

Jokes aside, let me ask you directly if you foresee any additional hybrid encoding modes in x264? Or do you feel any and all of that type of work should be left to 3rd party developers (if any are interested)? Sorry if I am just not getting it. Hey, I'm old and just need some more information to better understand :)

[akupenguin]

Not tinkering, just benchmarking the existing implementations of crf vs 2pass on several movies. Result: 2pass was better by anywhere from .01 to .12 dB (equivalently, up to 2.5% bitrate at the same psnr). This includes both the effects of ratecontrol (smarter I-frame qp decision) and the fact that direct=auto only really works in 2pass.

Jokes aside, let me ask you directly if you foresee any additional hybrid encoding modes in x264? Or do you feel any and all of that type of work should be left to 3rd party developers (if any are interested)?
I subscribe to the Strategy of Least Entanglement (aka The Unix Way): Do not integrate functionality into one program if it can be achieved through combining simpler programs. So, a hybrid method which consists of a sampling pass and then a standard crf could and should be implemented as a separate script. But an satd-only first pass can't currently be selected externally, so if that's desirable it would need to be in x264.