CRF translated to DABR graph and discussion [Archive]

DDogg

7th October 2006, 05:16

I doubt this is anything new, but I'll flesh it out a little and see if anybody is interested in the topic. If this has been covered before please provide a link or two. I should warn that this stuff is all very tentative at the moment and needs much more testing. It is primarily just my old CCE stuff translated to x264 crf mode.

I spent some amount of time today translating a span of CRF 2% sample encodes into derived bitrate numbers (DABR). Much to my pleasure I found there seems to be a near perfect curve generated, and there also seems to be a strong possibility to use the Newton Ralphson convergence methods that are used to project CCE Q numbers into a predicted filesize.

Perhaps more importantly, it may provide a way to create branching logic in a program to allow the quality of a size constrained encode to be predicted before the actual encode is committed to.

As well this provides a replicable way to measure, quantify and predict the effect of various filters, matrices, etc. on final filesize.

Further, the accuracy of a 1% sample seems to be as effective as a 5% sample in my early tests. The DABR from the 1%, 2% and 5% were very close to each other.

There is a bitrate calculator spreadsheet attached to the post in my sig if anybody wants to replicate the numbers
and method used to create the graph. There may be some more information there that may be translatable for x264. Frankly I am just too new to x264 to know yet. Also, I have attached the spreadsheet data to the data post below if somebody wants to regraph it differently.

http://img96.imageshack.us/img96/734/graphda9.jpg (http://imageshack.us)

The cmdline used was:

C:\apps\x264.exe --crf 19 --ref 3
--mixed-refs --bframes 3 --b-pyramid --b-rdo
--bime --weightb --filter -2,-1 --subme 6 --trellis
1 --analyse all --8x8dct --vbv-maxrate 25000 --me
umh --threads 2 --thread-input --progress --no-psnr
--output "D:\temp\auto3\movienohrQ(xx).mp4"
"d:\temp\auto3\movie.avs"

The 2% range line was: SelectRangeEvery(600,12)

DABR conversion was done with - (((Sample_Size_In_Bytes*(100/Sample_Size_Percentage))*8)/1000)/(Total_Frames/Frame_Rate)

My plan is to set several encodes going tonight and see if the actual finished filesize ends up as predicted.

http://img96.imageshack.us/img96/734/graphda9.th.jpg (http://img96.imageshack.us/my.php?image=graphda9.jpg)

foxyshadis

7th October 2006, 08:46

akupenguin

7th October 2006, 08:55

x264's bitrate predictor for 2pass assumes the step is 5.5, based on a similar measurement a long time ago.
My hypothesis is: The threshold for the magnitude of information to keep does double exactly every 6 qps. But the information in the source is not quite evently (well, laplacianly) distributed over the range of magnitudes. If there's a little more information in the noise end of the spectrum, which gets lost completely at high qp, then that increases the rate change of bitrate as a function of qp.

DDogg

7th October 2006, 16:49

All data below and attached in excel spreadsheet.

CRF 2% Size (BTs) DABR in KBPS Projected file size Actual Filesize dif1% dif2%

CRF15.mp4 32,951,805 2237 1,647,590,250 1,574,893,880 0.9559 4.41%
CRF16.mp4 28,046,445 1904 1,402,322,250 1,327,162,625 0.9464 5.36%
CRF17.mp4 24,013,257 1631 1,200,662,850 1,126,649,226 0.9384 6.16%
CRF18.mp4 20,662,192 1403 1,033,109,600 958,217,194 0.9275 7.25%
CRF19.mp4 17,846,928 1212 892,346,400 819,491,843 0.9184 8.16%
CRF20.mp4 15,469,190 1050 773,459,500 702,993,061 0.9089 9.11%
CRF21.mp4 13,472,796 915 673,639,800 605,788,784 0.8993 10.07%
CRF22.mp4 11,803,302 801 590,165,100 524,643,249 0.8890 11.10%
CRF23.mp4 10,338,978 702 516,948,900 456,028,249 0.8822 11.78%
CRF24.mp4 9,109,543 619 455,477,150 396,922,397 0.8714 12.86%
CRF25.mp4 8,012,111 544 400,605,550 346,629,200 0.8653 13.47%
CRF26.mp4 7,091,370 482 354,568,500 303,524,399 0.8560 14.40%
CRF27.mp4 6,263,265 425 313,163,250 266,540,478 0.8511 14.89%
CRF28.mp4 5,557,481 377 277,874,050 234,993,050 0.8457 15.43%
CRF29.mp4 4,923,517 334 246,175,850 207,760,736 0.8440 15.60%
CRF30.mp4 4,391,764 298 219,588,200 183,939,507 0.8377 16.23%

http://img504.imageshack.us/img504/6796/predrea5od0.jpg (http://imageshack.us)

akupenguin

7th October 2006, 22:49

Did all your tests use the same 2% of the movie? What's the distribution of projected sizes if you run one Q lots of times with different random 2%s?

virus

7th October 2006, 22:57

Interestingly, although AVC is theoretically supposed to double in q steps of 6, this roughly doubles in q steps of 5, nearing 6 as it rises. I wonder if that has to do with crf, the options used, or just x264's implementation.
Also the type of source plays an important role. I've seen several cases where the "-6 QP -> 2x rate" rule of thumb holds decently (say, within 10-15%), but also a case where going from QP 16 to QP 10 brought a 3.8x increase in bitrate. I have no idea whether CRF's rates would differ significantly from constant QP in that specific case, though.

DDogg

7th October 2006, 23:27

Did all your tests use the same 2% of the movie? What's the distribution of projected sizes if you run one Q lots of times with different random 2%s? Sorry, I don't completely follow your question. Presently I am just using the method of sampling we always used for CCE that turned out to be reliable in all regards. There was never a need for anything else than the select range statement above.

Whether that holds true for x264 crf mode I really don't know yet. I do remember jonny and I having long communications several years ago about xvid CQ and the same principles did not hold up well at all. The graphs were all over the place. He an I both dropped out about that time and I don't know if he ever pursued that aspect of xvid compression again. I did send him a pm asking him to check in to this thread as this stuff should be second nature to him.

Primarily I am only attempting to do some grunt level labor in the hopes it might provide some of you braniacs with usable date and stimulate conversation and ideas.

If you would like a set of tests run, please provide the commandline you would like used, a base level sampling script/method preferred and your specifications for the test. I'll get it done and collate the results back to you.

Doing some of the grunt labor for some of you is little enough to repay the work you have done with x264.

Btw, none of this has any validity until I do multiple sources and see if the curves track in parallel. Even with a Wcooled D930 running at 4.5GHz the time required is scary.
Also the type of source plays an important role. I've seen several cases where the "-6 QP -> 2x rate" rule of thumb holds decently (say, within 10-15%), but also a case where going from QP 16 to QP 10 brought a 3.8x increase in bitrate. I have no idea whether CRF's rates would differ significantly from constant QP in that specific case, though. That was always the beauty of Using 1P Q in CCE, or I am assuming any constant quality mode in any encoder, as the source complexity completely dictated the bitrate required to "hold" the complexity of the source (within certain caps). That was the key for us back then as the sample size, translated to DABR, allowed a fingerprinting of the source complexity and thus an extrapolation of the size/bitrate needed in advance of actually doing the encode. As well, it allowed one to estimate the effect on bitrate demand by various matrices, filters, and would clearly show even something as simple as the minute effects of DC precision. So incredibly useful. Guess that is why I am hoping some of this might pan out for x264.

IgorC

8th October 2006, 00:56

Sorry, I don't completely follow your question. Presently I am just using the method of sampling we always used for CCE that turned out to be reliable in all regards. There was never a need for anything else than the select range statement above.
For different 2% pieces of movie filesize can variate siginifically. i.e 2% of chapter 10th may have different filesize from 2% of chapter 20th due to different motion and another factors. It's a probalistic task. Maybe it would be better to imply estadistic tools for better estimation of the results. I.e. confidence intervals 0.9-0.95. It should be realible.

DDogg

8th October 2006, 04:23

For different 2% pieces of movie filesize can variate siginifically. i.e 2% of chapter 10th may have different filesize from 2% of chapter 20th due to different motion and another factors. It's a probalistic task. Maybe it would be better to imply estadistic tools for better estimation of the results. I.e. confidence intervals 0.9-0.95. It should be realible.Yes, certainly various 2% slices would demand different bitrates as the complexity of that 2% slice would dictate the bitrate, but we are speaking of the SelectRangeEvery(600,12) creating an average of all the frames in the source.

I remember huge amounts of discussion on this topic because any intelligent person might doubt the ability of a 1 or 2% sampling of all the frames to accurately average the bitrate. However, after literally 10s of thousands of test on various sources we found it accurate to +-2% consistently and popular programs like DVDRebuilder use these techniques every day now. That's just one of those things that ended up working that people swore would not work. It did and nobody argues about it anymore as it relates to the process with CCE.

Whether it works at all with x264 crf mode is a whole other question. All we can do is try and see. If you have a method of sampling you think would work well, please share it. I can do a controlled set of tests. BTW, I don't have any idea what estadistic tools are.

akupenguin

8th October 2006, 06:07

If that's how you sampled it, then I have no trouble believing that it could make a good predictor once you compensate for the systematic errors.
A trivial explanation for why it overestimated the filesize: the test pass is limited to GOP size 12, whereas the final pass uses much larger GOPs. Solutions: (1) Use larger ranges (but then your test wouldn't sample the input as evenly). (2) Estimate the final average GOP size and downweight the I-frames when counting the bitrate of the test pass. In my experience the average GOP is around 100 when unrestricted, or you could try to measure it based on the number of I-frames that are not on the border between ranges (though x264's biased scenecut detection might interfere with that).
Several other options are affected similarly, though the magnitude of error introduced by those are smaller: B-adapt's decisions are also constrained by the GOP size, and multiref is less effective for the first few frames of a GOP.

Edit: I'm making this more complicated than it needs to be. To compensate for the extra I-frames, just delete the first frame in each range. The actual number of scenecuts in the movie will be correctly represented by the remainder of the frames. (This doesn't fix B-frames and refs, though.)

I assume the 5% sample was SelectRangeEvery(240,12)? Then the next test to run is SelectRangeEvery(600,30) and so on.

DDogg

8th October 2006, 13:29

If that's how you sampled it, then I have no trouble believing that it could make a good predictor once you compensate for the systematic errors.
A trivial explanation for why it overestimated the filesize: the test pass is limited to GOP size 12, whereas the final pass uses much larger GOPs. Solutions: (1) Use larger ranges (but then your test wouldn't sample the input as evenly). (2) Estimate the final average GOP size and downweight the I-frames when counting the bitrate of the test pass. In my experience the average GOP is around 100 when unrestricted, or you could try to measure it based on the number of I-frames that are not on the border between ranges (though x264's biased scenecut detection might interfere with that).
Several other options are affected similarly, though the magnitude of error introduced by those are smaller: B-adapt's decisions are also constrained by the GOP size, and multiref is less effective for the first few frames of a GOP.

I assume the 5% sample was SelectRangeEvery(240,12)? Then the next test to run is SelectRangeEvery(600,30) and so on. Here is the derived bitrate spreadsheet (http://forum.doom9.org/attachment.php?attachmentid=2924&d=1081970852) I use and the select lines are listed at the bottom which brings up a math question (mine are 1970s US high School=poor). Just for conversation's sake, let's assume the above curve holds true (big stupid assumption now). Would you, or others with good math skills, have a suggestion on how one would take the formula (((Sample_Size_In_Bytes*(100/Sample_Size_Percentage))*8)/1000)/(Total_Frames/Frame_Rate) and add to it a curve correction factor based upon the above chart or data points?

As for your point # 2 above, seems to me that would take a small program to accomplish?

Graph and data points updated above

foxyshadis

8th October 2006, 21:35

Are you keeping logs of the x264 output? Try comparing the I-frame size as a % of the total size in the test vs the final, and subtract the difference from the sampled size.

So if in the sample, there I frames are 478632 bytes and 40%, but in the final they're 9042673 and 25%, you subtract 179478 and predict from what's left. ([average full %]/[obtained %]*size will net you the amount, or close enough to it for this.) You'll probably have to make tests to figure out what the average I-frame size in % is in movies. Or you might just be able to find an average obtained % as well and not need to look at logs at all.

Since you're not writing software that can analyse the stream and chop the first two and last two frames of each gop off, which is how the most accurate mpeg-2 and mpeg-4 results seem to come about, this should be the next-best way.

DDogg

9th October 2006, 00:48

foxyshadis, that sounds like a very intelligent thing to try. Problem is I am not smart enough, and I did not keep logs. Hopefully you might feel like dipping a toe in this water as I could certainly use some help from somebody like you.

I'm going to take the simple and dumb route with another experiment. I have another source of 150,804 frames with much lower complexity (to my eye) than the previous more complex source of 141,241 frames. I've run the 2% tests and using the same method above, size predicts out at 1,246,861,350 for crf 15 and 160,868,500 for crf 30. Btw, I was pleased to see this as it has more frames but clearly the 2% sample is getting a good fingerprint of the complexity and indicates it will come in smaller in size, which seems right to me.

If the curve holds and I apply the correction of -4.41% and -16.23% respectively they should come in at 1,191,846,279 and 134,752,562 (using identical script and commandline). I would be surprised if they do hit the size, but it is an experiment that has to be done to see.

Will post back whatever the results are.

/add:

Results are encouraging -
CRF 30 - expected 134,752.562 got 128,583,054 which is -4.6% shy of the target
CRF 15 - expected 1,191,846,279 got 1,194,073,460 WOW! I'll take that anyday.

Will run several more toward the belly of the curve:

/add:
These did not work out as well [+-2% is desired]-
CRF 18 - expected 703,308,561 got 684,558,029 - 97.3% of predicted
CRF 20 - exxected 509,203,536 got 489,826,680 - 96.2% of predicted
CRF 22 - expected 377,038,516 got 360,203,986 - 95.5% of predicted

xyloy

9th October 2006, 16:25

If you use x264 via MeGUI, logs are usually kept in the program files\megui\logs subdirectory.

BTW, this is a very interesting topic. ;)

DDogg

9th October 2006, 17:20

If you use x264 via MeGUI, logs are usually keeped in the program files\megui\logs subdirectory.

BTW, this is a very interesting topic. ;) Thanks, xyloy. I think there is a fair amount of potential in the subject, but I also think the advice of akupenguin and foxyshadis is going to have to be incorporated to take it anywhere. I don't think I can get there on my own.

Still, although super accurate prediction has not been demonstrated (so far) with these very simple methods used, there certainly seems a potential to use them as a reality check before doing a bitrate based encode as well as generally checking the compressibility of a source. Given the slower speed of x264, this might be helpful to some. My hope is this thread will stimulate more advanced members of this forum to take the subject farther.

@akupenguin - just noticed your edit - "...To compensate for the extra I-frames, just delete the first frame in each range. The actual number of scenecuts in the movie will be correctly represented by the remainder of the frames. (This doesn't fix B-frames and refs, though.).." My brain is fried, please suggest the modified SelectRangeEvery statement :)

Hmmm, wondering if a PredictCRFsizefunction.avsi might accomplish this task better? Function writers out there?

jonny

9th October 2006, 18:51

please suggest the modified SelectRangeEvery statement

the problem are bad (statistically) frames introduced with SelectRangeEvery
you should think that you are serving to the encoder little snips of the original source (it's full of jumps in different parts of the movie), this introduce abnormal sized frames (at the start and at the end of the snip)

the only way to handle this is using the log with all the frame sizes, summing sizes but discarding bad frames (this will give you a better value, compared to raw output size)

with mp4-asp a good way to handle bframes is discarding 3 frames at the start and 3 frames at the end of the snip

i haven't done testing with avc, i've found a discussion here:

http://forum.doom9.org/showthread.php?p=598896&highlight=x264+prediction#post598896
(i'm trying to see where it ends : )

foxyshadis

9th October 2006, 19:12

I don't think he means modifying the avisynth, I'm pretty sure that means chopping the first frame of each segment off in the output. To do that you can modify x264's output module or use a stream parser to ignore every 12 frames, but it's definitely not something you can gather from the basic info you're using now.

Oh, I just noticed that min-gop is still 25, so it's not just every 12 frames... hmm, interesting. Might still be I frames though. I guess I'll have to run a similar test and parse it to check.

akupenguin

9th October 2006, 19:20

No need to actually modify the compressed stream. Just ignore the appropriate lines from `x264 -v` output.
And it doesn't matter what min-gop is set to. An I-frame takes almost the same number of bits as a P-frame full of I-blocks. Except for the keyframe qp bonus, but that won't really be correct either way.

#!/usr/bin/perl
$range_length = 12;
$fps = 24000/1001;
while(<>){
/frame= *(\d+).*size=(\d+)/ or next;
$1 % $range_length or next;
$size += $2;
$frames++;
}
printf "%.3f\n", $size*$fps*.008/$frames;

...but this tends to overcompensate a bit. Maybe because the keyframes still got their qp bonus and thus made better references but their bit cost was ignored. Or maybe the converse: the real scenecuts didn't get detected as such and didn't get the qp bonus, so didn't increase the bitrate as much as in the final encode.

Hellworm

9th October 2006, 20:50

@akupenguin - just noticed your edit - "...To compensate for the extra I-frames, just delete the first frame in each range. The actual number of scenecuts in the movie will be correctly represented by the remainder of the frames. (This doesn't fix B-frames and refs, though.).." My brain is fried, please suggest the modified SelectRangeEvery statement :)

I think he meant that you should ignore the size of the I-Frames of every cut (analysing the log) and completely ignore those frames as the scenecut was introduced by the selectrange anyway

damn beaten to it :D

DDogg

9th October 2006, 21:19

jonny, thanks for posting that link. A general comment, somewhat on topic with the recent VFW thing, is how they are trying to work around all the problems of vfw, vdubmod, etc whereas the modern cli version, if the author chose, could probably generate a specialized log, or for that matter, even have a predictive method built in without all the nonsense of the workarounds. At least, I think this sounds like it may be true. akupenguin, is that theoretically possible, or am I misunderstanding some of your comments?

Hellworm

9th October 2006, 21:30

But x264 does no decoding. It could be possible to implement an ultra fast first pass, but the decoding of the remainig 98% had to be done.
So the cuts have to be done one level higher.

akupenguin

9th October 2006, 22:04

x264cli with avisynth input can choose which frames to request from avisynth.
piped input (e.g. from mencoder) would still have to decode everything.

DDogg

10th October 2006, 15:45

... hmm, interesting. Might still be I frames though. I guess I'll have to run a similar test and parse it to check. I hope you have the time and interest to do that.

Which brings up a question, Is it effective to substitute --crf and a script containing the selectrangeevery statement when creating the stats file? Or, put another way, what is the method you folks would suggest for creating a log that would provide the specific information for the specific purpose we have been discussing? x264cli with avisynth input can choose which frames to request from avisynth.
piped input (e.g. from mencoder) would still have to decode everything.Could you elaborate a little more? I'm trying to understand whether you are saying something additional should be/could be added to x264 cli to better accomplish what we are discussing, or if you are suggesting that is a dead end and using the log information is the direction you suggest be pursued.

Probably best for me to go silent on this stuff as I leaped from mpeg2 to x264 without all the intermediate xvid log stuff and some of the things mentioned here are clearly over my head at this point. I can do some of these long methodical test fairly well, so let me know how I can help in that regard.

emmel

10th October 2006, 19:49

I think the answer was pretty clear: the encoder can ask avisynth to seek, decode, and deliver any particular frame it needs, e.g., every 50th frame (check muxers.c for details). It is then up to Avisynth to do what it has got to do - probably it will quickly skip some gops before starting to decode the frame the encoder is looking for. No need to always decode all frames. In theory, that is.

DDogg

11th October 2006, 15:05

I thought I would close my part of this discussion with a short "How to" for people that want a quick reality check of potential quality before doing the full 2 pass bitrate based encode. Certainly this is just a rule of thumb method, but it might be useful for some.

1> Temporarily add to the bottom of your script - SelectRangeEvery(600,12) (remove before doing final encode)
2> Use a crf commandline based upon the CQ-ASP_Q2_eq(crf) profile and encode the sample
3> Select properties of the sample rendered and get the size (top one, not "size on disk")
4> multiply (size X50) X .9275
5> Not an exact science, but If that size is more than, say 5% larger than your planned size you most probably will need to take some action before doing the full encode. That might be increasing size allocated, adding a light filter to increase compression if that is acceptable to you (maybe fluxsmoothT), lowering resolution, using a different matrix, etc. Just about any change you make to the script will show a reaction in the sample filesize.

If you want to translate the sample size into a projected bitrate to compare to the one you plan, then use the spreadsheet linked above. You will need your sample filesize, framerate, and the number of total frames. That will translate the sample size into a bitrate you can use for comparison.

Another way to sort of accomplish the same thing is to use a newer version of AutoMK - choose automatic resolution, check allow multiple resize, and input the desired crf (advanced tab - suggest 18, 19, or 20). If AutoMK reduces the resolution, it indicates you might need to take some of the steps in #5 above. You can change the script, matrix, filter, etc and tell AutoMK to redo the prediction until it shows no resolution decrease - then go ahead with the encode. This is a new feature in AutoMK and I imagine it is still a work in progress and will continue to be tweaked, so keep that in mind.

Again, I am not suggesting this is anything more that a quick reality check method. Given the many hours a 2 pass x264 encode can take, having some idea of where you will end up might be useful for some of you, especially if your machine is not a dual core.

Still hoping members with more advanced skills might move this discussion forward.

DDogg

12th October 2006, 20:43

Hmmm, may have stumbled upon something useful -

1> The sample size and final filesize from a previous encode (-crf 18) were known. 15,165,556 bytes and 684,558,029 bytes
2> A new sample was run with with same cmdline and script except FFT3DFilter(sigma=3) was added to the script after the resize.
3> New sample size = 12,649,854
4> Divided smaller sample/larger sample = 0.834117391
5> Took previous encode size, 684,558,029 x 0.834117391 = predicted size of 571,001,757 bytes
6> Ran full encode and got 566,691,876 bytes which is 99.25% of the predicted filesize

So, while the ability to predict the final final size of a first encode may not be all that compelling [-+6%] [given the Iframe thing], the ability to predict the effect of a filter on final filesize is proving to be accurate using a simple 2% sample created with SelectRangeEvery(600,12). Many more tests using various matrices and filters on several different type of sources will have to be run to have any sense of confidence. Certainly, it seems worth pursuing.

/add1:
Replicating the above, substituting the crf 20 data (instead of the crf 18 data above), shows a similar result:
Predicted filesize = 430,242,978 b, actual 427,270,410 b, 99.3% on target

/add2: Looking beter and better. I used the more complex source the graph data was made from.

1> Original crf 18 sample and encode was 20,662,192 b, 958,217,194 b respectively
2> New sample using FFT3DFilter(sigma=3) was 17,631,504
3> 17,631,504 / 20,662,192 = 0.853322048
4> 958,217,194 x 0.853322048 = 817,667,859 b predicted
5> Actual re-encode was 818,251,893 which is 99.92% on target

/add3: the crf 20 and 22 filter tests came in as 99.1% 98.1% of the predicted final filesize

708145

13th October 2006, 09:38

1> Original crf 18 sample and encode was 20,662,192 b, 958,217,194 b respectively
2> New sample using FFT3DFilter(sigma=3) was 17,631,504
3> 17,631,504 / 20,662,192 = 0.853322048
4> 958,217,194 x 0.853322048 = 817,667,859 b predicted
5> Actual re-encode was 818,251,893 which is 99.92% on target

Very good results indeed.
Now another interesting aspect would be how well you can predict the final size when varying crf and the filter(s).
This would allow for an encode mode where the analysis pass finds the crf parameter together with filters (which filters and parameters) which yield the final size (or very close to it) in just one pass.

This allows for heavy filtering being done only one time (*) for encoding without the need to filter and encode to lossless in advance. Encoding from HDTV lossless is harddrive limited for me at least so it could give some significant speedup to the encode chain.

If the analysis amounts to less than 10% of the final encode the name "1.1 pass encoding" seems appropriate.

bis besser,
T0B1A5

(*) one time + the x% for the analysis pass (pass 0?)

check

13th October 2006, 11:00

I have to say, this is turning out to be an extremely interesting thread. One query, has anyone tried using a different min GOP size? As someone (foxyshadis?) said before, the default is 25, and the estimates are coming out slightly too large. Would increasing the min GOP size affect this?

DDogg

13th October 2006, 15:44

Very good results indeed.
Now another interesting aspect would be how well you can predict the final size when varying crf and the filter(s).
This would allow for an encode mode where the analysis pass finds the crf parameter together with filters (which filters and parameters) which yield the final size (or very close to it) in just one pass. ...There are separate sections to this stuff -

1> Convergence of a crf value, via sampling, to yield a final filesize, and actually using crf mode for the final encoding does not seem viable as there is no decimal available in crf mode. Note the difference between crf 18 and crf 19 in the graph data in the first posts. It is 138 megs. Not that useful without smaller decimal values like crf 18.5, 18.6, etc.

2> That brings us to some type of bitrate extrapolation using the DABR fingerprint to gauge the compressibility of a source and estimate if a bitrate calculated to hit a size target will allow enough headroom to contain the source complexity and yield some equivalent of the known quality of the crf samples.

That seems viable only if prediction software is written to properly parse the log information into the calculations. My gut tells me the size prediction could get close to the -+ 2% area. Without the log information being used, the simple methods used here would be more like a +- 6% (not even close to being sure of that, yet) [Still, +- 6% may be ok for a quality reality check since we are going to actually use a bitrate based encode for the final encode which will guarantee size accuracy]

3> On the positive side, even if we declare final filesize as an unknown x, the initial filter tests show we can accurately predict that b = x* (sample1/sample2). To put that another way, we know the filtered final filesize is going to be a known percentage [different] than the unknown unfiltered filesize (sounds kind of funny). We can easily arrive at that percentage by using only two quick 2% samples (using the same crf cmdline). This seems useful and could provide GUI programs, or a user, with the logic needed to make more accurate filtering decisions and save a hell of a lot of wasted encoding time.... One query, has anyone tried using a different min GOP size? As someone (foxyshadis?) said before, the default is 25, and the estimates are coming out slightly too large. Would increasing the min GOP size affect this? Good question and hope we can get an answer to it. [I should add the fact that once I started using the cmdline mentioned way above, I was locked in to it, as changing would have invalidated the entire data set]

/add1: I need to do the #3 tests with and without a custom matrix to see if that would show the same accuracy of prediction.
/add2: Slight edits to #3 because we want a multiplier. If the filtered sample is larger we would want to divide larger/smaller to give us a 1.x multiplier. If filtered sample is smaller we would want smaller/larger to get a 0.x multiplier. I think I got that right.

708145

13th October 2006, 17:04

1> Convergence of a crf value, via sampling, to yield a final filesize, and actually using crf mode for the final encoding does not seem viable as there is no decimal available in crf mode. Note the difference between crf 18 and crf 19 in the graph data in the first posts. It is 138 megs. Not that useful without smaller decimal values like crf 18.5, 18.6, etc.

1> fractional crf is not impossible. Maybe Loren can comment.

bis besser,
T0B1A5

DDogg

13th October 2006, 20:42

> fractional crf is not impossible
We can hope.

Looks like the prediction holds up well for matrices, in this example the sample size is larger due to the effects of eqm_avc_hq -

1> Original crf 18 sample and final was 20,662,192 b and 958,217,194 b
2> New crf 18 sample with eqm_avc_hq was 24,221,744 b
3> 24,221,744 / 20,662,192 = 1.172273687
4> 958,217,194 x 1.172273687 = 1,123,292,803 b
5> Re-encode with matrix was 1,109,699,135 b
6> 1,109,699,135 / 1,123,292,803 = 0.987898375
7> 98.8 % on target

/add: crf 20 w/matrix - 99.1% of predicted target (832,261,957 b actual / 839,735,870 b predicted)

ChronoCross

13th October 2006, 21:43

he added fractional crf in the latest revision =D

r591 | pengvado | 2006-10-13 22:04:58 +0200 (Fri, 13 Oct 2006) | 2 lines
Changed paths:
M /trunk/common/common.c
M /trunk/encoder/encoder.c
M /trunk/encoder/ratecontrol.c
M /trunk/x264.c
M /trunk/x264.h

allow fractional values of crf

DDogg

13th October 2006, 22:38

DDogg

14th October 2006, 01:17

First plot using fractional values of crf in x264 r591 - Same source, x264 cmdline and script used in the original data post. Huge thanks go to the x264 team for implementing fractional values of crf!

http://img214.imageshack.us/img214/3256/source1wj9.jpg (http://imageshack.us)
CRF Sample Size in bytes Dabr Projected filesize

CRF17.0.mp4 24,013,258 1,631 1,200,662,900
CRF17.1.mp4 23,659,721 1,607 1,182,986,050
CRF17.2.mp4 23,316,194 1,583 1,165,809,700
CRF17.3.mp4 22,971,897 1,560 1,148,594,850
CRF17.4.mp4 22,622,602 1,536 1,131,130,100
CRF17.5.mp4 22,246,102 1,511 1,112,305,100
CRF17.6.mp4 21,896,714 1,487 1,094,835,700
CRF17.7.mp4 21,597,862 1,467 1,079,893,100
CRF17.8.mp4 21,274,419 1,445 1,063,720,950
CRF17.9.mp4 20,983,766 1,425 1,049,188,300
CRF18.0.mp4 20,662,193 1,403 1,033,109,650
CRF18.1.mp4 20,367,235 1,383 1,018,361,750
CRF18.2.mp4 20,083,724 1,364 1,004,186,200
CRF18.3.mp4 19,786,368 1,344 989,318,400
CRF18.4.mp4 19,490,543 1,323 974,527,150
CRF18.5.mp4 19,181,005 1,302 959,050,250
CRF18.6.mp4 18,900,673 1,283 945,033,650
CRF18.7.mp4 18,633,764 1,265 931,688,200
CRF18.8.mp4 18,368,735 1,247 918,436,750
CRF18.9.mp4 18,112,083 1,230 905,604,150
CRF19.0.mp4 17,846,929 1,212 892,346,450
CRF19.1.mp4 17,593,500 1,195 879,675,000
CRF19.2.mp4 17,337,434 1,177 866,871,700
CRF19.3.mp4 17,098,223 1,161 854,911,150
CRF19.4.mp4 16,864,579 1,145 843,228,950
CRF19.5.mp4 16,595,722 1,127 829,786,100
CRF19.6.mp4 16,372,733 1,112 818,636,650
CRF19.7.mp4 16,134,694 1,096 806,734,700
CRF19.8.mp4 15,935,226 1,082 796,761,300
CRF19.9.mp4 15,730,880 1,068 786,544,000
CRF20.0.mp4 15,469,190 1,050 773,459,500
CRF20.1.mp4 15,287,690 1,038 764,384,500
CRF20.2.mp4 15,069,438 1,023 753,471,900
CRF20.3.mp4 14,866,851 1,009 743,342,550
CRF20.4.mp4 14,656,212 995 732,810,600
CRF20.5.mp4 14,425,248 979 721,262,400

Didée

14th October 2006, 15:11

"1.1 pass encoding"
Yeah! :)

It's already years since I started thinking about the possibility to do something like this with XviD (by encoding & logging an "x% Snip", then doing one full pass by feeding a modified 'CBR'-module with local bitrates interpolated from the x% stats.)

So we had to wait for x.264 to get it, okay. ;)

When more tests have been done and a sufficient formula formulated, getting pretty close to a desired size should be possible with two+one steps:

1a) one initial 2% encoding with some starting CRF value (educated guess) --> predict an CRF value for destination size from this

2) 3 (or 5 or 7) % encoding with CRF from 1) --> predict new CRF

3) full encoding with CRF from 2).

Would estimate that this should come in pretty close.

DDogg

14th October 2006, 15:48

Hi, Didée - good to see the Wizard of Scripts is interested with this topic and apologies to 708145 as I realize I did not understand his 1.1 reference.

Having a little trouble understanding your point #2 - I think you are suggesting we would predict the size of the 5% sample from the size of the 2% sample and apply any error as a correction factor? Would appreciate you expanding.

Just had a funny thought - It would not surprise me if you write one of your crazy complex scripts to parse the log and predict final crf size :) Hell, I think you could create a word processor from an avisynth script!

Oh, btw, happy to grind out some data for you. Let me know what would be helpful.

buzzqw

14th October 2006, 16:21

and i am very interested on this 1.1 encoding...

could a simple binary iteration be ok ? or must be used the obfusched RoBa method ?

BHH

Didée

14th October 2006, 16:27

It's kind of a correction factor, yes. Doing the final pass directly with the CRF value predicted from one 2% encoding surely is not that bad, but will leave quite some error margin on the final size ... depending on how representative this small snip has been in fact, on how far the initial chosen CRF value has been away from the really needed CRF value, and also on how exactly different quantization will react to the characteristics of the particular source.

Mathematicians are used to the "nested intervals" procedure, and that's what I was suggesting, with a nesting depth of "2". :)
The small snip encoding is used to get a closer-to-the-truth CRF value. When now a longer snip is encoded with this refined CRF value, the size prediction should be good.

BTW, Avisynth scripting is easy. In you get values in range [0,255], out you put values in range [0,255]. Nothing spectacular. :D

DDogg

14th October 2006, 18:08

Some clarification notes for the casual reader of this thread. The new capability of fractional crf has considerably altered the direction of the original post, and added new capabilities. There is a potential for confusion as two very different methods, sharing a common sampling process are being discussed. To rehash:

1> The addition of fractional crf has presented the possibility to use some of the known convergence techniques to potentially calculate a crf to achieve a size target. This was not practical before, because without fractional crf, the final size between, say crf 18 and 19, was too large to be useful (138 mb in my test source). The addition of fractional crf (18.1,18.2,etc.) has changed this. The crf encoding method would be used for the final encode.

2> A hybrid method using a constant quality crf sample to fingerprint the complexity of a source by deriving the bitrate from the byte size of the resulting sample. This derived bitrate is used to suggest a bitrate appropriate for bitrate based encoding methods tailored to the source complexity. A bitrate based encoding method, 1P or 2P, would be used to do the final encode.

Two seemingly similar, but quite different approaches, each with a potential for being useful depending on your preferred method of encoding.

---------------------------------------------------------------------------------------------------------------------------

... Mathematicians are used to the "nested intervals" procedure, and that's what I was suggesting, with a nesting depth of "2". :) The small snip encoding is used to get a closer-to-the-truth CRF value. When now a longer snip is encoded with this refined CRF value, the size prediction should be good. ... Hmmm, clearly I ain't a Mathematician :) Perhaps consider showing us a more 1,2,3 approach to replicating your thinking with the computations involved? I can normally get that. Then, I could do some methodical tests and provide you with the data. Maybe that would be useful to you? Let me know.

Didée

14th October 2006, 19:35

Okay, 3rd try, then.

Assume we want a certain destination size, and the closest to this we could ever get by CRF encoding would be with CRF 25.7. So, 25.7 is the target value that we want, but that we don't know yet.

How close the prediction of a snip encoding will be to the final size, depends on:

- difference between snip's CRF value and final CRF value
(if target is 25.7, prediction from a CRF20 snip encoding will be worse, prediction from a CRF24 snip encoding will be better.)

- prediction from a bigger snip will be better than that from a smaller snip, because more content is encoded, hence the longer snip is more representative.

So. If you just do a 2% snip with, say, CRF22, you might get somewhere in the destination region, but still be off a good bit. Prediction might be like eg. CRF 24.9.

You could get somewhat closer by just doing a bigger snip with CRF22, but it might still be off by too much, because the relatively big step 22.0 -> 25.7.
(Basically, the final destination value is completely unknown, but you have to use something for the [first] snip encoding.)

So, my suggestion simply is to
- first do the short snip with CRF22
- get e.g. CRF 24.9 as predicted result
- do a longer snip, like 5% or 7%, with CRF24.9
- get a prediction like 25.6 ~ 25.8 (hopefully)
- do the final CRF encoding with this CRF value

Much words for explaining a simple thing. :D
(For an even more verbose explanation, I'd have to use German language, I fear... ;) )

Probably the "RoBa method" is not too far from this (or vice versa) ... I can't tell because, you may laugh about me, the only thing I know about "the RoBa method" is its name ... I never have been deeply into Mpeg-2 encoding.

akupenguin

14th October 2006, 20:09

Another possibility:
1st pass (2%?) with ABR
2nd pass (5%?) with CRF, using the value of "final ratefactor" from the 1st (adjusted for I-frame bias).
3rd pass adjust the CRF value and do the real encode.

And another:
1st pass runs the analysis part of ABR (SATD complexity measurement and B-adapt) on the whole movie, but only encodes a small fraction.
You now know the exact SATD distribution over the movie, but have only an estimate of the mapping from SATD to bitrate at any given QP. Use this estimate to create a fake 1st pass statsfile.
2nd pass is a normal --pass=2 encode, except using the SATD data instead of a real 1st pass. It will adjust the SATD->bits estimate as it goes, and can compensate for any mispredictions just like it does with the existing fast-1st-pass.
Note: despite the 100% coverage of the 1st pass, this method won't spend any more cpu-time in x264 than the 2% method, since it just moves part of the CRF computation to the 1st pass. But it will have to decode the input twice, instead of 1.02x.

akupenguin

14th October 2006, 20:11

could a simple binary iteration be ok ? or must be used the obfusched RoBa method ?
afaict from a few minutes search, the RoBa method is equivalent to 1st pass CRF+VBV, then 2nd pass at the same bitrate? No, that's already possible but doesn't provide the speed of the methods discussed here.

DDogg

14th October 2006, 20:16

@Didée - Oh, I think I understand you now. You were simply speaking of the basic convergence method used to calculate the crf to hit a size?. If so, a derivative of this base method (Newton-Raphson) was used for cce: http://www.sosmath.com/calculus/diff/der07/der07.html

But that will not work for x264 because of the I-Frame thing. The logged frame information has to be incorporated. Like jonny said, "... sadly this stuff require programming skills in order to parse the log ..."

There is some more information about that on page 1. /add:

This is the major reason I shifted to the hybrid approach for x264. Sure, we all know a 1P constant quality mode is the fastest method of encoding, but with xvid and x264 the frame structure throws a monkey wrench into accurately predicting the crf within a tight +-2% range.

Another possibility:
1st pass (2%?) with ABR
2nd pass (5%?) with CRF, using the value of "final ratefactor" from the 1st (adjusted for I-frame bias).
3rd pass adjust the CRF value and do the real encode.

And another:
1st pass runs the analysis part of ABR (SATD complexity measurement and B-adapt) on the whole movie, but only encodes a small fraction.
You now know the exact SATD distribution over the movie, but have only an estimate of the mapping from SATD to bitrate at any given QP. Use this estimate to create a fake 1st pass statsfile.
2nd pass is a normal --pass=2 encode, except using the SATD data instead of a real 1st pass. It will adjust the SATD->bits estimate as it goes, and can compensate for any mispredictions just like it does with the existing fast-1st-pass.
Note: despite the 100% coverage of the 1st pass, this method won't spend any more cpu-time in x264 than the 2% method, since it just moves part of the CRF computation to the 1st pass. But it will have to decode the input twice, instead of 1.02x.

The first was what I sort of thought Didée was speaking of. The second sounds very intriguing [in fact, it sounds brilliant!]

afaict from a few minutes search, the RoBa method is equivalent to 1st pass CRF+VBV, then 2nd pass at the same bitrate? No, that's already possible but doesn't provide the speed of the methods discussed here.

Not meaning to act as class historian, but I do have some background. We might want to forget the term Roba as it is misused and means different things to different people (camps).

Instead, think of it as Conditional 2-pass Bach. The first pass was simply what we are speaking of here. 1 or 2% samples were encoded with a constant quality method (cce opv) and Newton-Raphson was used to calculate the appropriate "Q", in this case crf, to hit the size target. It could do that with only 2 -4 samples normally. The constant quality encode was run with this calculated Q while keeping a log of the frames (VAF file). [B]IF the file was on target then the encode was finished - ELSE, a second bitrate based pass was run, using the previously generated VAF log, to hit the size target. Very seldom was this needed - in fact it was rare.

buzzqw

15th October 2006, 06:52

@akupenguin
my bad, i mean the Newton-Raphson iteration ( as build, for example by Tylo in ccefront)

BHH

check

15th October 2006, 08:00

I've heard a few times that a crf which comes out to bitrate x will not be as high quality as a full 2 pass to the same bitrate (assuming all other settings are equal). Is there any truth to this?

buzzqw

15th October 2006, 09:23

if CRF to be used is about 18 (or less) then quality is near maximized. If CRF used is more than 22 than a 2pass should be better, because of better distributing bitrate (where needed 18 when not needed 22 or less.. for example)

BHH

akupenguin

15th October 2006, 09:36

I've heard a few times that a crf which comes out to bitrate x will not be as high quality as a full 2 pass to the same bitrate (assuming all other settings are equal). Is there any truth to this?
Yes, but the difference is pretty small. (details tomorrow)

if CRF to be used is about 18 (or less) then quality is near maximized.
No. You just mean that once quality is high enough, you can't see the further improvement. But 2pass still lets you achieve transparency at a slightly lower bitrate.

DDogg

15th October 2006, 14:36

Change/: Removed data because of error - re-running tests

DDogg

16th October 2006, 18:17

Yes, but the difference is pretty small. (details tomorrow)
Sorry, curiosity is getting the better of me. Does this mean you are tinkering around with some of the ideas you mentioned above, or that you simply did not have the extra time available to get into more detail?

Btw, I'm convinced all developers must take a secret handshake course called "How to speak Cryptic-101" :) Or, maybe it is "How to post short and succinct thoughts-101" - Clearly, the second was not offered in my school.

Jokes aside, let me ask you directly if you foresee any additional hybrid encoding modes in x264? Or do you feel any and all of that type of work should be left to 3rd party developers (if any are interested)? Sorry if I am just not getting it. Hey, I'm old and just need some more information to better understand :)

akupenguin

17th October 2006, 19:41

Not tinkering, just benchmarking the existing implementations of crf vs 2pass on several movies. Result: 2pass was better by anywhere from .01 to .12 dB (equivalently, up to 2.5% bitrate at the same psnr). This includes both the effects of ratecontrol (smarter I-frame qp decision) and the fact that direct=auto only really works in 2pass.

Jokes aside, let me ask you directly if you foresee any additional hybrid encoding modes in x264? Or do you feel any and all of that type of work should be left to 3rd party developers (if any are interested)?
I subscribe to the Strategy of Least Entanglement (aka The Unix Way): Do not integrate functionality into one program if it can be achieved through combining simpler programs. So, a hybrid method which consists of a sampling pass and then a standard crf could and should be implemented as a separate script. But an satd-only first pass can't currently be selected externally, so if that's desirable it would need to be in x264.

DDogg

17th October 2006, 20:40

... Not tinkering, just benchmarking the existing implementations of crf vs 2pass on several movies. Result: 2pass was better by anywhere from .01 to .12 dB (equivalently, up to 2.5% bitrate at the same psnr). This includes both the effects of ratecontrol (smarter I-frame qp decision) . ... Yeah, I read your post where you laid out very clearly how a log allowed you a span of frame information to work with on a 2nd pass, and how you were somewhat limited by the "on the fly" decisions needed in crf mode. That makes a lot of sense.

... and the fact that direct=auto only really works in 2pass. Ug, that's a stinker - did not know that. Is a permanent thing? [So do you think best to set direct=none in crf modes, or leave it as auto?]... But an satd-only first pass can't currently be selected externally, so if that's desirable it would need to be in x264. ... Seems to me a proof of concept would have to be done by someone who is intimately aware of the guts of x264 before anybody could suggest if it was desirable. Chicken and egg kind of thing as the proof could not be done without a test version allowing satd-only first pass? Do you have any room on your plate to do a quick down and dirty proof?

akupenguin

17th October 2006, 21:11

Ug, that's a stinker - did not know that. Is a permanent thing?
It's permanent. The recommendation for crf is to leave it at the default, which is direct=spatial.

Chicken and egg kind of thing as the proof could not be done without a test version allowing satd-only first pass? Do you have any room on your plate to do a quick down and dirty proof?
If you just want to output the satd stats to test the quality (though not saving any cpu-time yet), that's simple enough.

DDogg

18th October 2006, 02:36

If you just want to output the satd stats to test the quality (though not saving any cpu-time yet), that's simple enough. <change> Going to leave that to somebody else with a better understanding and instead concentrate on documenting the accuracy (or not) of using DABR to properly identify and convert source complexity into a minimum recommended bitrate. I think this is the area where my time would be most usefully spent.

DDogg

19th October 2006, 08:09

Bumbling along, I seem to have again stumbled upon something that might have some value.

Very preliminary, but it seems each increment of .1 crf translates to around 1.014823039 (or decrement by 0.985393474) when using the stock command line mentioned in the first posts.

In other words, with just one 2% sample using any empirically chosen crf, it seems it might be possible to approximate the ~ sample size, Dabr, and predicted filesize of a range of cfr's.

Full data later after some sleep, but here is a short example.

The data below includes real data from a set of 2% sample encodes and is included here as comparison data. The DABR was calculated as normal from the the sample using the byte size, total frames, and frame-rate. The "extrapolated DABR" has simply been created by decrementing each previous value by X 0.985393474. The seed value of 1,917 was the starting point. The extrapolated Dabr tracks the real Dabr with a pretty decent level of accuracy.

CRF Sample size Dabr Extrapolated Dabr
Q16.0.mp4 28,233,339 1,917
Q16.1.mp4 27,820,948 1,889 1,889
Q16.2.mp4 27,354,937 1,857 1,861
Q16.3.mp4 26,930,004 1,829 1,834
Q16.4.mp4 26,528,796 1,801 1,807
Q16.5.mp4 26,119,096 1,774 1,781
Q16.6.mp4 25,716,708 1,746 1,755
Q16.7.mp4 25,283,566 1,717 1,729
Q16.8.mp4 24,921,636 1,692 1,704
Q16.9.mp4 24,514,116 1,665 1,679
Q17.0.mp4 24,123,684 1,638 1,655
Q17.1.mp4 23,806,646 1,616 1,631
Q17.2.mp4 23,387,542 1,588 1,607
Q17.3.mp4 23,032,212 1,564 1,583
Q17.4.mp4 22,687,520 1,541 1,560
Q17.5.mp4 22,350,196 1,518 1,537
Q17.6.mp4 22,023,653 1,495 1,515
Q17.7.mp4 21,689,510 1,473 1,493
Q17.8.mp4 21,355,400 1,450 1,471
Q17.9.mp4 21,060,837 1,430 1,450
Q18.0.mp4 20,705,685 1,406 1,428
Q18.1.mp4 20,399,122 1,385 1,407
Q18.2.mp4 20,112,539 1,366 1,387
Q18.3.mp4 19,818,035 1,346 1,367
Q18.4.mp4 19,528,018 1,326 1,347
Q18.5.mp4 19,243,181 1,307 1,327
Q18.6.mp4 18,969,913 1,288 1,308
Q18.7.mp4 18,679,249 1,268 1,289
Q18.8.mp4 18,389,029 1,249 1,270
Q18.9.mp4 18,122,898 1,231 1,251
Q19.0.mp4 17,835,529 1,211 1,233
Q19.1.mp4 17,602,509 1,195 1,215
Q19.2.mp4 17,301,155 1,175 1,197
Q19.3.mp4 17,086,487 1,160 1,180
Q19.4.mp4 16,858,627 1,145 1,162
Q19.5.mp4 16,632,237 1,129 1,145
Q19.6.mp4 16,399,338 1,114 1,129
Q19.7.mp4 16,136,495 1,096 1,112
Q19.8.mp4 15,914,180 1,081 1,096
Q19.9.mp4 15,664,496 1,064 1,080
Q20.0.mp4 15,439,366 1,048 1,064
Q20.1.mp4 15,264,966 1,037 1,049
Q20.2.mp4 15,013,503 1,019 1,033
Q20.3.mp4 14,809,236 1,006 1,018
Q20.4.mp4 14,602,554 992 1,003
Q20.5.mp4 14,410,499 978 989
Q20.6.mp4 14,216,400 965 974
Q20.7.mp4 13,998,508 951 960
Q20.8.mp4 13,799,629 937 946
Q20.9.mp4 13,621,746 925 932
Q21.0.mp4 13,413,582 911 919
Q21.1.mp4 13,234,527 899 905
Q21.2.mp4 13,057,810 887 892
Q21.3.mp4 12,881,811 875 879
Q21.4.mp4 12,729,880 864 866
Q21.5.mp4 12,558,010 853 853
Q21.6.mp4 12,414,334 843 841
Q21.7.mp4 12,212,705 829 829
Q21.8.mp4 12,051,610 818 817
Q21.9.mp4 11,889,901 807 805
Q22.0.mp4 11,720,845 796 793
Q22.1.mp4 11,578,842 786 781
Q22.2.mp4 11,415,808 775 770
Q22.3.mp4 11,260,005 765 759
Q22.4.mp4 11,121,700 755 748
Q22.5.mp4 10,983,788 746 737
Q22.6.mp4 10,830,070 735 726
Q22.7.mp4 10,685,504 726 715
Q22.8.mp4 10,548,772 716 705
Q22.9.mp4 10,394,736 706 695

akupenguin

19th October 2006, 09:23

Very preliminary, but it seems each increment of .1 crf translates to around 1.014823039
For comparison, the trivial theoretical bitrate factor is 2^(1/60) = 1.011619 per .1 qp (and crf is the same scale as qp). While the ratecontrol currently uses 1.0127887.

DDogg

21st October 2006, 05:05

For comparison, the trivial theoretical bitrate factor is 2^(1/60) = 1.011619 per .1 qp (and crf is the same scale as qp). While the ratecontrol currently uses 1.0127887. Thanks, I've been pondering this for a couple of days and attempting to incorporate this into a spreadsheet for folks to use. Do you think I should simply do two samples (say 18.1, 18.2) and divide for the difference to get the factor, or use your 1.0127887? Wondering if their is some kind of overhead or something because I seem to get 1.014-5-6xx a lot, but never as low as 1.012xx.

btw, I should be putting up some data soon. Don't know if it will be useful to you or others, but hope it might be. Five sources spanning 15.0 to 24.0 plus a worksheet for people to use if they wish.

akupenguin

21st October 2006, 06:45

DDogg

21st October 2006, 08:26

The factor definitely depends on the content, my 1.012 was just an average of the movies I had around when I wrote that code. So a measurement of the movie in question can easily be more accurate than a general average.
But there's no reason to do a separate measurement of the qp vs bitrate exponent: You can derive it from the "spanning 15.0 to 24.0" data, and get a more precise measurement than a single qp pair would, along with an estimate of how well the exponential model fits.That probably would be the better method. I've put some time in already and used 18.0 and 18.1. It worked out pretty good. I think well enough to let folks mess with it and hopefully make it a better tool. Here is a screenshot. I'll put it up tomorrow after some sleep.

The nice thing about this tool is people only have to run two 2% samples. That's it. When the byte size of those samples is input, the sheet will calculate a span of values from 15.0 to 24.0. This includes projected average bitrate and a rough projected file size.

If I can figure out how to do it the way you suggested, I will, because spanning farther ought to do a better job. I thought about it earlier today but could not figure out how to do it. This was hard enough for me :)

DDogg

21st October 2006, 15:32

Ok, I've attached a small zip file with the five source data spanning crf 15.0 to 24.0. It represents a lot of time and effort and I hope it might be useful to some of you. Please look it over.

Also included in the zip is a worksheet/tool for extrapolating a span of 15.0-24.0 data from two small 2% samples. It also has a workspace for doing your own work. This should provide you with enough information to allow good bitrate choices to be made before doing an encode. Should be simple to use. Let me know if not. Also, when you add filters, matrices, etc., running the two samples again will show you how much your bitrate needs and filesizes will be changed.

My hope is those of you with more advanced numeric analysis and spreadsheet skills can use this as a building block to create a better tool for us all.

http://img171.imageshack.us/img171/9813/work1lw5.th.jpg (http://img171.imageshack.us/my.php?image=work1lw5.jpg)http://img171.imageshack.us/img171/7126/work2mu6.th.jpg (http://img171.imageshack.us/my.php?image=work2mu6.jpg)

Had to update the large worksheet. Sorry. Temporarily HERE (http://files.project357.com/uploads/index.php?dir=&file=Worksheet_plus_data_03.zip/).

check

22nd October 2006, 06:35

Very interesting! Thanks for uploading :)

Dayvon

24th October 2006, 17:03

Excellent work guys. I'm very interested in this thread.

I don't know about you guys, but the reason this "CRF estimate" idea is so amazing to me is that it makes it so that you can finally choose your overall quality and know an estimate file size. I was very interested in CRF when it first came out because 1-pass with variable bitrate is awesome. I never was one who looked for a specific size so CRF would seem to be just the ticket for fast, good quality encodes. I could never get into it though, because the file size outputs for CRF were to erratic.

This thread is solving a host of issues however. Having a ball-park figure for filesize/bitrate is AMAZING. Adding decimal values to CRF also adds more wiggle room for choosing your quality while trying to relatively hit a file size. I've always wanted to be able to decide what quality I want for each source, while knowing approximately what the filesize is going to be. From each source, the quality I desire from it varies BUT knowing a general filesize is essential for me to know so I can choose how much quality I want in the file.

What I'm seeing from all of this is a stand alone tool or tool integration into a GUI (MeGUI, etc.). A simple tool, a "Quality Calculator" with file input (d2v), resolution output size (with DAR,SAR options), x264 settings, a quality setting (either in CRF numbers or in high-low options), and a test button. The high-low scheme would translate to different CRF values. Eg: highest (CRF 15), higher (CRF 16.5), high (CRF 18), Medium (CRF 19.5), Low (CRF 21), Lower (CRF 22.5), Lowest (CRF 24). This would give easy choices to people. And essentially, the tool would state back, "If you want a _____ quality file at ___x___, using _____ x264 settings, then you will need ____ average kbps and output will be approximately ____ MBs" where the user would choose the quality (high-low or CRF values), resolution, and x264 settings.

This would be amazing at giving users the choice of quality, and showing them the repercussions of that choice. Until now, most everything is based on hitting specific filesizes to backup to removable media. This tool/method allows users to choose quality, resolution, and x264 settings, based on filesize instead of choosing filesize based on size constraints and hoping quality is good.

DDogg

24th October 2006, 18:26

To be perfectly honest, I had hoped the discussion would have stimulated more discussion, input, and assistance to prove, or disprove some of the thoughts presented.

Also, remember one of the base assumptions is completely unproven with x264. Unproven is whether the bitrate value derived from the size of the crf sample will generate a bitrate based encode of the approximate quality of the crf encode. I think it will generally hold up, but am not willing to put the time into boning back up on how to test the assumption in a numerically comparative way that would hold some validity to the group. This is especially so given the lack of contributory feedback from those with heavy specific X264 knowledge I needed to tap to move forward.

I suppose the bitrate translation is not all that important to you as you mentioned you will be using crf mode for the final encode and you made it clear that you understand the bitrate and final filesize predictions are only approximants.

What is proven is the ability of a simple 2% sample to clearly measure the effects of matrices, filters, resolutions, etc.,etc., etc., on final filesize as a percentage of final filesize.

Being able to personally quantify does help one make better decisions. A good example is how one can know in advance one of the so called "HQ" matrices can increase filesize by 50% on a grainy source like SPR. This method does seem to provide a more informed personal decision on whether the small increase in the amount of perceived visual quality is worth the huge hit on bitrate when encoding that specific source.

Dayvon

25th October 2006, 06:45

This method does seem to provide a more informed personal decision on whether the small increase in the amount of perceived visual quality is worth the huge hit on bitrate when encoding that specific source.
Exactly, and this is something that I find EXTREMELY valuable. You can find out the "quality" needed for your source. Using the sampling, you can identify the quality needed and what filesize you will end with.

I would totally help with your research, but I'm horrible at putting sheets together and such. I do better at encoding. Right now, I'm trying some different sources cartoon and action right now. The cartoon required a much higher bitrate/quality, and running the tests made a huge difference in what I would have chosen as my encode choices.

Anyway, I'd really like to see this integrated into MeGUI...

check

25th October 2006, 12:41

DDogg

25th October 2006, 18:05

Looking interesting. I'm playing around with python at the moment so I'll see if I can cook up a script to do a full 2 step estimation (with adaptive second step quantization and all the bells & whistles). Sounds great!
A few questions:
o From what I gather above, the best method workable so far is to test once, then again using the gathered prediction, and then run the final encode.. right?

o Would it be better to start with a lower or higher quantizer than what you expect the result to be?
Feeling fuzzy today so I need more info to sync up with your brain :) Can you break that down some more? Especially what is in your head when you say ".. the best method workable ...". Which method? I ask because there are several different 'sub-methods' in this whole thing and I don't want to waste your time with an out of sync reply.

[removed data - available from link]

check, HERE (http://files.project357.com/uploads/index.php?dir=&file=Worksheet_plus_5_source_data_crf15-24_5_temp_for_check2.xls) is a spreadsheet with the above data [plus the source 5 data]. Note four 5% samples were done on Source4 - 18.0,18.1,15.0,24.0. You will find these in the far right worksheet area. 18.0 and 18.1 were used for the extrapolation and 15.0 and 24.0 are there for comparison with the extrapolated values.
[updated with source 5 values]

From what I'm seeing, you may need to stick with 2% if the source is shorter. Maybe even 5% if real short, like 30 minutes.

Btw, you used the word "accurate" :) - I guess I would have to say it looks as if .5% is no more inaccurate than 2% is :)

Dayvon

25th October 2006, 20:21

I made a quick mockup of what it might look like to have a "testing method" inserted into MeGUI. This would add a "test x264" button to the bottom of the AVScript Creator tool. Pressing the test button would then pop up a window with choice for Quality as either CRF number or high-low values. Below this would be a selector for x264 profiles, and/or a config button. Below that would be a Test button. Upon pressing test the tool then creates the AVScript, loads it, encodes, and then displays a log stating the average bitrate, and the resulting video filesize. By putting this "quality calculator" into the AVScript Creator, the calculator actually becomes more versatile, because then you can adjust your AVScript to the quality that you need (eg: resolutions in particular). This would probably be a better option as opposed to creating a whole new tool that would, in effect, need to be a AVScript creator as well.

http://img224.imageshack.us/img224/6912/avscriptcreatornewcp3.th.jpg (http://img224.imageshack.us/my.php?image=avscriptcreatornewcp3.jpg)http://img224.imageshack.us/img224/4118/qualitytestqu1.th.png (http://img224.imageshack.us/my.php?image=qualitytestqu1.png)

This tool/method allows users to choose quality, resolution, and x264 settings, based on filesize instead of choosing filesize based on size constraints and hoping quality is good.

DDogg

25th October 2006, 20:49

This is (kind of) already in AutoMKV via the autoresizing when "allow multiple autoresize" is checked. You specify a crf to use as a target (kind of). Instead of accepting the resizing, you change whatever you need to change (matrix, slice size, filters, etc.) and press redo. Do this until resolution is not reduced under what you want. Might work for you now, and who knows, it might be expanded in the future with coherent feedback from folks like you.

check

26th October 2006, 10:26

Sorry, I'm referring to the various methods proposed for estimating required bitrate for a certain quality, ie:
o Do a single 5% sample and estimate from that
o Do a 2%, then estimate a 5% from that, and use the 5%'s result
o more?
Hope this explains it, if not don't worry, I'll just implement a basic method (but don't look forward for anything for at least a week I'm afraid, I've taken on more than I can chew for a while).

DDogg

26th October 2006, 20:43

To estimate a bitrate for a specific crf needs just a 2% sample made with the specific crf. Take the size in bytes, total framecount, and framerate - plug them into this to translate the sample size into a bitrate.(((Sample_Size_In_Bytes*(100/Sample_Size_Percentage))*8)/1000)/(Total_Frames/Frame_Rate)

If you want to project a range of crf's translated to bitrate try this (what I did in the worksheet (http://files.project357.com/uploads/index.php?dir=&file=Worksheet_plus_data_03.zip)):

Create a 2% sample for crf 18.0 and 18.1 - calc the difference that can be used for extrapolating the other crf values. For instance, if we have:

18.0 = 37,679,180 and 18.1 = 37,160,945
then a multiplier can be created:
37,679,180/37,160,945 = 1.013945689 (to guess next smaller crf value)
and
37,160,945/37,679,180 = 0.986246118 (to guess next larger crf value)
so
if we take the 18.1 value: 37,160,945 x 0.986246118 = extrapolated value for 18.2 of 36,649,837
if we take the 18.0 value: 37,679,180 x 1.013945689 = extrapolated value for 17.9 of 38,204,642

You just repeat the process up and down.

A better way might be to use 2 samples with a larger spread like 16 and 22, and use the average difference as the factor. Just thinking out loud on that as I have not had tme to do it.

Those guesstimated sample sizes can be translated into guesstimated bitrates. Important to note this is using data with a less than perfect accuracy to then create an extrapolation. Values are useful for rule of thumb purposes only.

I do think data from a general curve (http://forum.doom9.org/showthread.php?t=116773) (error% down a few posts) could be used to further correct the prediction, but at this time I don't know the full variance of the curve.

check

27th October 2006, 13:54

Thanks for the walkthough ddogg, it's all in place now :) One question though. In your first post the relation between nominal ratefactor & bitrate is clearly exponential (both ae^bx and ax^b model it very closely, correlation coefficienct is nearly 5 nines(!!)) - is there any reason why you suggested I use a plain linear relation instead of an exponential/power relation?

More thinking:
Your results show that we can essentially encode a video with a known ratefactor and a known bitrate. I can see two main applications:

Encode two test samples and use this to generate a list of required bitrate for given quantizer (what your current xls does).
Encode a video to a known size in one pass via -crf, because you can calculate the target bitrate and use your method to derive the equivalent ratefactor. This is essentially a saving in time (assuming no turbo, a 50% reduction), at the cost of: slightly reduced quality & bitrate still not fully known.
Encode a 2/5% sample with a certain crf (your target quality for the final movie) and use your result to see the required bitrate required. Adjust filters and retest until you are satisfied, and then encode in 2pass at the target bitrate. This gives the advantage both quality & bitrate are fully known, but requires 0.02*full_length_encoding_time*number_of_tests + working_time more time than a normal encode.

I have been using the third method for my current encoding project to great effect. Large savings in time and I can essentially 'preview' sections of the film at the same bitrate that they will have in my final encode.

Anyway, what I'm taking the long way to ask is - is any script really neccessary? The third method is all done by hand, the second I am guessing would be rare, as those who care that much about the quality are happy to spend the extra time (which is even less of an increase with a turbo pass1), only the first method remains - and we have your perfectly capable spreadsheet for that.

.. Or, am I missing something?

A trivial explanation for why it overestimated the filesize: the test pass is limited to GOP size 12, whereas the final pass uses much larger GOPs
A question: do you have any idea/hunches about how reducing the IP ratio would do? I can't see any problem with that approach in cq mode, but I'm unsure about what it would do in crf mode.

Dayvon

27th October 2006, 15:36

I've been doing method 2 on a number of sources... Scarface, Se7en, Platoon. They all came out very well with 1-Pass CRF. I used either 19or20 and am thinking I've got better quality then most of my 2-pass VBR rips. Of course, I probably didn't give the 2-passes enough bitrate, which is why this "test" method is so freaking great.

Any info/testing/encoding you guys would like help with?

DDogg

27th October 2006, 18:15

In your first post the relation between nominal ratefactor & bitrate is clearly exponential (both ae^bx and ax^b model it very closely, correlation coefficienct is nearly 5 nines(!!)) - is there any reason why you suggested I use a plain linear relation instead of an exponential/power relation?
Nope, except keep in mind most of the stuff I have put forward is based on my very limited 'country boy' math skills (as in I have no idea what you just said above). The posts were more designed to stimulate and promote discussion from folks with higher level skills in both math and x264. What I do hope is the data I supplied via grunt level labor provides an efficient starting point so people with more expertise don't have to get bogged down in what, I promise you, is a time consuming and mindlessly boring data creation exercise.

The huge area that needs to be addressed to move this forward is increased accuracy via log integration. This may be the best use of your scripting skills?

A question that needs to be answered is whether it makes sense to request akupenguin consider adding some type of stats file capability in crf mode. I say this because it is going to complicate things to use alternating modes just to have a log to work with. I don't understand enough about the inner workings of x264 to know if that request would be constructive or just plain silly.

I also like the idea of a log in crf mode to allow a conditional 2nd pass where/when needed to correct size overruns as was so useful in the conditional 2 Pass Bach method for CCE (even though it was rarely needed).

So, until that is resolved I don't think there can be much improvement on the guesstimated methods (although I am reasonably sure the extrapolation method could be improved with a broader span method). [Note updated sheet below with real and extrapolated data has a fair amount of error on end of crf ranges]

As an ultimately goal we would want to work toward an accurate method of convergence whereby a crf is calculated to hit a size target within a +- 2% range. The byproducts include the natural improvements in accuracy to the derivative methods you mentioned.

As you touched upon, another area to think about is the actual commandline used to create the crf samples. I still don't have a proper handle on some of the options in the commandline I used. Generally I got tied down to it as it would have invalidated the data had it been changed in mid-stream.

Throwing in a general thought, It is interesting that once a sample has been made, any comparison sample is dead on accurate. I assume this is because what ever "I-frame" errors are made in the first sample also occur in the comparative sample and cancel themselves out (so to speak).

As you can see, I've pretty well maxed out my level of expertise. Most of it comes directly from mpeg2 without the intermediate XVid experience so many of you have and I find I am floundering with the concept of parsing logs and somehow including that information in the equations. Especially as there is no log, or other specialized information available from crf mode.

I hope you, or others may get interested enough to move this thread forward.

Note: What might make this stuff easier to use is a small purebasic or delphi mini app that would read the avs, automatically create the samples and present the extrapolation is an easy to understand manner. The basic worksheet (http://files.project357.com/uploads/index.php?dir=&file=Extrap01b.xls)[updated] is brain-dead easy, but it does requires Excel and this may be a problem for some.

@Dayvon - Would you consider making a small contribution of time to write out a short and simple explanation of how you use the sheet? (I like putting people on the spot) :) [Changed it slightly so workspace and extrapolation can be independent of each other]

akupenguin

27th October 2006, 19:09

Dayvon

27th October 2006, 19:35

@Dayvon - Would you consider making a small contribution of time to write out a short and simple explanation of how you use the sheet? (I like putting people on the spot) :)
Right now I'm not using the sheet. I've ripped the above mentioned DVD's to my hard drive, ran a CRF 19 test clip using the 2% script (the first one you gave me), checked that the file size is something I liked, readjusted the resolution/etc, ran a second test clip, and so on. Then when I was happy with the resolution/filesize/quality, then I would run a MeGUI auto-encode of the chosen CRF level. And *poof* great quality encodes in 1pass. Much better than 2-pass shot-in-the-dark-quality encodes I've been doing. I like knowing what quality I'm getting, knowing about how big the file is going to be, and being able to adjust resolution/other factors to better suit my quality needs.

DDogg

28th October 2006, 00:19

... and then choose whether or not to run the 2nd pass.<snip> A 2nd pass what? As in -pass2 with no stats file? How could that work? Perhaps an example might help.

Is the exact crf cmdline I have been using for all these tests ok for this first pass, or will it require additional parameters? If so, an example of that would also be appreciated. Help me out a little here, akupenguin.

akupenguin

28th October 2006, 00:35

DDogg

28th October 2006, 00:55

Assume your target bitrate is 1000, and you estimate that crf=24 will achieve that rate. Run:
x264 in.avs -o out.264 --pass=1 --crf=24
And if that produced a bitrate too far for the target, then run:
x264 in.avs -o out.264 --pass=2 --bitrate=1000Isn't that just the same as deleting the first encode and replacing it with a bitrate based encode since there was no stats file created in the first crf pass? or does the --pass=2 --bitrate=1000 not need a stats file and somehow looks at information in the file it is replacing? I guess I should know that. If so, apologies.

akupenguin

28th October 2006, 00:59

--pass=1 makes a statsfile. That's the entire point of that option.
--pass=2 uses a statsfile. It will complain loudly and refuse to encode if there isn't one.

DDogg

28th October 2006, 01:08

Hmm, I would have swore I tried using a -pass1 with crf and none was created. Maybe that is why I have not been following you throughout this entire thread. I thought you could not have a log file while in crf mode. Off to try it with your examples. Thanks.

/later:

Damnation! It is creating a log. I must have made some syntax mistake when I tried. One of those wrong assumption things that makes you want to kick your cat. As you can imagine, it has caused me endless confusion and a huge amount of frustration when reading some of the replies here.

Well, at least now I might be able to go back and understand a few more things you and others were saying.

akupenguin

28th October 2006, 01:38

Is that literally what your put on the commandline?

The valid names for the option are: `--pass=1', `--pass 1', `-p=1', `-p 1', `-p1'. (i.e. all the variants supported by GNU getopt).
Note that `-pass1' is not on that list. It is instead parsed as equivalent to `-p=ass1', and `ass1' was numerically equal to 0, up until r577 when it became a parse error.

DDogg

28th October 2006, 02:06

"ass1" -- thanks for that in cryptic :) It is about what I am feeling like :o

I started doing this exercise with MeGUI and it will not allow a log name when in crf mode so I may have got it in my head then. Still, I remember trying --pass=1 along with --stats "logname" several times, but I believe I was looking for the results it in my sample output directory instead of my originating batchfile directory. Just a dumb mistake that got compounded. The result was we have been speaking 'past' each other this whole thread.

/add: Getting past the ass1 part, it seems we do then have a conditional 2 pass method.

Q: The quality from the second pass, if used, would be identical to a normal 2nd pass quality, correct? The only reason I ask is because some of your posts talk about how you handle some things differently while in crf mode. However, I'm thinking that would not apply to the log, but ask to make sure.

foxyshadis

28th October 2006, 02:56

The next step is segment recoding. If it comes out too large, instead of a full second pass, re-encode only the largest gops, or the lowest average quant, or whatever. By hand it's not particularly simple, but I made basic tools to massage stats files for manual segment recoding, and it wouldn't be hard to extend them (depending on what language you'd like) for that. Given that the accuracy of this method should be pretty good, you wouldn't have to re-encode much of it.

DDogg

28th October 2006, 03:15

instead of a full second pass, re-encode only the largest gops, or the lowest average quant, or whatever.wow...WOW .. That's a great idea! So ... this would be doing the resized snip as a separate segment, then reassembling it with other parts somehow broken out of the main crf encode? Pretty sure that is what you said, but I am going to be gun-shy about assumptions for a while.

/add: what about --seek and --frame somehow allowed to replace the specified range within the main encoding? Is that farfetched?

/add: Btw, does anybody know if it is hypothetically possible to use compressed-domain transcoding techniques on an mp4 file? I ask because we used to do that with slightly oversized mpg2 files to reduce the size by 2 to 5% in just a few minutes without any appreciable quality loss.

foxyshadis

28th October 2006, 03:40

No, that's exactly how I did it, easier than modifying the avisynth every time. You have to chop up the stats file though, because it starts at the beginning no matter what you seek to. It's fairly simple to merge through mp4box or mkvmerge. (which I'm thinking of modifying to support faster merges, currently you have to split the source into component gops, then reappend them all on the command line; error-prone in gui, painful on commandline.)

There are some considerations: You can't mix two crfs currently, or a crf with a bitrate, because of default quantizer. x264 would probably have to be modified to allow passing a different assumed quant than the crf used to encode.

DDogg

30th October 2006, 01:52

Going back through the posts:

... And another:
1st pass runs the analysis part of ABR (SATD complexity measurement and B-adapt) on the whole movie, but only encodes a small fraction.
You now know the exact SATD distribution over the movie, but have only an estimate of the mapping from SATD to bitrate at any given QP. Use this estimate to create a fake 1st pass statsfile.
2nd pass is a normal --pass=2 encode, except using the SATD data instead of a real 1st pass. It will adjust the SATD->bits estimate as it goes, and can compensate for any mispredictions just like it does with the existing fast-1st-pass.
Note: despite the 100% coverage of the 1st pass, this method won't spend any more cpu-time in x264 than the 2% method, since it just moves part of the CRF computation to the 1st pass. But it will have to decode the input twice, instead of 1.02x.

... If you just want to output the satd stats to test the quality (though not saving any cpu-time yet), that's simple enough.OK, I give up after searching for SATD and x264 longhelp for nearly a full day. Would very much like to try the above and report back, but can not figure out the details of actually how to do it. Some example cmdline(s) to head me in the right direction would be appreciated.

Another possibility:
1st pass (2%?) with ABR
2nd pass (5%?) with CRF, using the value of "final ratefactor" from the 1st (adjusted for I-frame bias).
3rd pass adjust the CRF value and do the real encode. Sorry, same request here if you still think tests are worth doing. I don't understand how you are meaning "final rate factor", especially since you put it in quotes.

akupenguin

30th October 2006, 02:25

... If you just want to output the satd stats to test the quality (though not saving any cpu-time yet), that's simple enough.
Here I meant "simple enough to write the required code". There is no existing command to do it.

I don't understand how you are meaning "final rate factor", especially since you put it in quotes.
I mean literally what x264 prints at the end of an ABR encode. e.g.
x264 [info]: final ratefactor: 24.05
(and this one is with the existing code)

DDogg

30th October 2006, 04:11

Here I meant "simple enough to write the required code". There is no existing command to do it.Ah, erm, ok. Thus my jokes re cryptic and mindreading :)
Hope you may have time to do the "simple enough" test sometime in the future then.

Moving on, and as advised, I was able to get the crf final ratefactor to show in an ABR line by removing --vbv-maxrate 25000 which is included in the MeGUI ABR line as well as my test crf line. Btw, is it there for a reason for it (?) as it causes the warning about buf size.

So, the next part you suggested included "...(adjusted for I-frame bias)". Am I correct in assuming this will require parsing the log with some uninvented software tool so as to provide the data to adjust for I-frame bias?

fyi, the abr(--b 1000) line showed final ratefactor 23.6 and 1000 KB/s. The crf line using crf 23.6 showed 1160 KB/s. Don't know if there is any relationship there yet, but will keep exploring.

Hey, how to get -v output to a text file? I tried > and |on the end, but nada.

foxyshadis

30th October 2006, 07:50

2>&1.

Use it like:
x264 -v -b 1000 .... > log.log 2>&1

You might even want to throw out the stdout stuff and just keep the stderr (which is all the stuff you can't redirect with a plain >), you do that with "2> log.log"

708145

30th October 2006, 08:17

So, the next part you suggested included "...(adjusted for I-frame bias)". Am I correct in assuming this will require parsing the log with some uninvented software tool so as to provide the data to adjust for I-frame bias?

Just take the overhead of short GOP encode (like the samples use) with a normal GOP (like a regular backup does) at the same crf.

IOW: encode the same movie with two different --keyint values with the same crf quant and note the size difference in percent.

edit: A short test of mine showed an overhead of 32% from --keyint 25 to --keyint 250. :eek:

bis besser,
T0B1A5

PS: hopefully not being cryptic ;)

check

30th October 2006, 14:17

Just an update on my scripting adventure. I was going to post earlier but winupdate decided to restart in the middle of me writing out the post, and of course I was writing it in the browser >_>
Anyway, it's morphed into a general compressibility checker for any CLI encoder that accepts avs input because I haven't found one that does anything more than linear interpretation. It will run two (or more) sample encodes at user input quantizers, and then build a quantizer vs. filesize(bitrate) table out of the results.
Describing it is the easy part, the harder part will be actually writing the thing, as I'm a programming newbie. The basic interface is in place now, I just need to write up the extrapolation logic and try to unconvolute the strange mess of code a little so it's readable :)

ToS_Maverick

30th October 2006, 15:57

just did a little test with DDogg's spreadsheet at 24 season 4, pal dvd version.

i coded this season at CRF 20 with the following commanline some time ago:
x264.exe --crf 20 --level 3 --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct none --filter -2,-1 --subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-maxrate 10000 --me umh --threads 2 --thread-input --cqmfile "D:\Program Files\meGUI\extra\eqm_avc_hr.cfg" --progress --no-psnr

i followed this thread since the beginning, so i'm quite interested in crf-predicton. as i still have the coded files on my disk, i could test how accurate the spreadsheet is.

the prediction was made with the instructions from DDogg.
"Pred." is the predicted filesize at crf 20
"Final" is the final filesize at 100% coded crf 20
"Pred/Final" is the difference in %
"Pred. CRF" is the CRF in the spreadsheet which hits the final size (hope you know what i mean)

E02 E03 E04 E13
Pred. 345.430.222 352.903.389 357.912.015 328.542.086
Final 303.399.273 316.183.326 321.705.452 281.222.968
Pred./Final 0,878322896 0,895948681 0,898839487 0,855972431
Pred. CRF 21,2 20,8 21 21,1

conclusion:
the predicted filesize is always a bit bigger than the final result. the error is always aprox. 1 CRF which means for example, if you want to code at CRF 20, take the predicted filesize of CRF 21 in the spreadsheet.

DDogg

30th October 2006, 18:07

@foxyshadis - a big thanks - you are the cat's meow! :)
@708145 - Not cryptic, thanks!
@check - AFLO :)
@ToS_Maverick Excellent! Very much appreciate your coherent post with data

Q: You were using the real data worksheet, or the extrapolation section?

It would be very helpful if you can repeat a similar test using another source - needs to be something with a different level of complexity - Perhaps just use one vob of any movie to keep the time requirements down?

Since it is clear you have a good understanding, I am hoping I can enlist your help with something else we are tinkering with - but understand time may not allow. If you can make the time, grab this (http://files.project357.com/uploads/index.php?dir=&file=Crf_test.zip) little bat file. If the notes are too cryptic, let me know :)

There seems to be some kind of replicable relationship between the final ratefactor crf reported by the ABR sample and a crf final encode using that crf final ratefactor, and/or a converged crf sample that resolves to the KB/s reported in the ABR sample . One can hope ...

Way, way, too soon to say yet, but this stuff may end up being fairly simple. I have 5 machines running tests now, but could use the data from the test bat with one of your 24 eps.

ToS_Maverick

30th October 2006, 21:53

@DDogg i used your "Worksheet plus 5 source data crf15-24 5.xls", only the extrapolation section (18 and 18.1)

ok here are the results from your 1st and 2nd test.

"D:\Program Files\meGUI\tools\x264\x264.exe" --bitrate=1006 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --me umh --threads 2 --thread-input --no-psnr --output "F:\CD-Images\24 - Season 4\E13\E13 comp ABR_1006_2pct.mp4" "F:\CD-Images\24 - Season 4\E13\E13 comp.avs" > "F:\CD-Images\24 - Season 4\E13\DDogg.txt" 2>&1

avis [info]: 640x352 @ 25.00 fps (1272 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
mp4 [info]: initial delay 2 (scale 25)
x264 [info]: slice I:89 Avg QP:15.88 size: 15969
x264 [info]: slice P:729 Avg QP:17.89 size: 6128
x264 [info]: slice B:454 Avg QP:19.83 size: 1550
x264 [info]: mb I I16..4: 35.2% 51.3% 13.4%
x264 [info]: mb P I16..4: 14.7% 18.0% 2.5% P16..4: 36.7% 11.8% 3.4% 0.2% 0.1% skip:12.7%
x264 [info]: mb B I16..4: 0.7% 0.8% 0.1% B16..8: 32.2% 0.9% 1.9% direct: 2.3% skip:61.2%
x264 [info]: final ratefactor: 17.20
x264 [info]: 8x8 transform intra:51.1% inter:75.1%
x264 [info]: ref P 69.9% 19.3% 10.8%
x264 [info]: ref B 85.0% 11.0% 4.0%
x264 [info]: SSIM Mean Y:0.9865291
x264 [info]: kb/s:1036.5

encoded 1272 frames, 15.42 fps, 1037.85 kb/s

"D:\Program Files\meGUI\tools\x264\x264.exe" --crf 17.20 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --me umh --threads 2 --thread-input --no-psnr --output "F:\CD-Images\24 - Season 4\E13\E13 CRF 17.20_hope1006_full.mp4" "F:\CD-Images\24 - Season 4\E13\E13.avs" >> "F:\CD-Images\24 - Season 4\E13\DDogg.txt" 2>&1

avis [info]: 640x352 @ 25.00 fps (63350 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
mp4 [info]: initial delay 2 (scale 25)
x264 [info]: slice I:936 Avg QP:14.82 size: 17794
x264 [info]: slice P:36513 Avg QP:16.95 size: 7172
x264 [info]: slice B:25901 Avg QP:18.63 size: 2007
x264 [info]: mb I I16..4: 35.2% 50.4% 14.4%
x264 [info]: mb P I16..4: 13.2% 17.3% 3.0% P16..4: 37.6% 14.1% 5.1% 0.2% 0.1% skip: 9.4%
x264 [info]: mb B I16..4: 1.0% 1.1% 0.1% B16..8: 35.5% 1.1% 2.4% direct: 3.7% skip:55.2%
x264 [info]: 8x8 transform intra:51.4% inter:71.6%
x264 [info]: ref P 64.1% 21.9% 14.1%
x264 [info]: ref B 76.7% 16.3% 7.0%
x264 [info]: SSIM Mean Y:0.9870292
x264 [info]: kb/s:1043.4

encoded 63350 frames, 16.09 fps, 1044.50 kb/s

until the 3rd test, your instructions were pretty clear. i don't really understand what you want me to do in test 3?
please be so kind and explain it to me, i think i just need a hint to get it ;)

DDogg

30th October 2006, 21:59

While only 2 [3] sources finished, it appears we may have some good news.Results:

Source 1
frames = 141,241
Framerate = 23.976
Complexity judged medium high (subjective)
MeGUI Bitrate Calc'ed - 1116 average KB/s - 1100 MB muxed [1,126,400 KB] (1/4dvd)
Actual video = 764,430,754 (right click properties - size - top value)
Actual Audio = 329,892,864 " "
Actual Muxed = 1,095,037,143 " "
Accuracy = 1,095,037,143 / 1,126,400,000 = 97.215% [changed to use all KB]

Source 4
frames = 243,770
Framerate = 23.976
Complexity very high (subjective)
MeGUI Bitrate Calc'ed - 3247 average KB/s - 4479 MB muxed [4,586,496,000 KB](1dvd5)
video = 3,925,129,249 (right click properties - size - top value)
audio = 569,329,152 b " "
Actual Muxed = 4,495,555,266 " "
Accuracy = 4,495,555,266 / 4,586,496,000 = 98.02% [changed to use all KB]

/add:

Source 3
frames = 196,159
Framerate = 23.976
Complexity medium low (subjective)
MeGUI Bitrate Calc'ed - 741 average KB/s - 1100 MB muxed [1,126,400 KB] (1/4dvd)
video = 688,217,404 (right click properties - size - top value)
audio = 392,629,248 b " " "
Actual Muxed = 1,081,701,144 " "
Accuracy = 1,081,701,144 / 1,126,400,000 = 96.03%

The method used seems too simple to believe:

1> Loaded AVS and audio into MeGUI bitrate calc
2> Used ABR cmdline with the average bitrate from MeGUI to do a 1% sample
3> Noted the KB/s, NOT the crf reported in the ABR cmd window
4> Did 1% samples using a CRF cmdline to converge upon an equal KB/s to that shown in the ABR window. The .1 and .01 values allowed a near exact KB/s convergence. So, for Source 1 it was crf 18.28 that equaled the 1128 KB/s reported in the ABR window. For the long and difficult to compress source 4, it was crf 16.8 to equal the 3247 KB/s from the ABR sample. [Source 3 was crf 19.26 for 779 KB/s]
5> Did the full encodes with the crf value obtained from the sample results in #4
6> Muxed them. Results above.

So far I am having trouble believing this and wonder if the results are just some kind of strange fluke. Going to do some more sources and see if the method holds up. Others that are interested please do some encodes trying the described method and report results back using the same template from above.

Notes:

I used these command lines which may not be optimum:
x264.exe --crf 18.28 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 6
--trellis 1 --analyse all --8x8dct --me umh --threads 2 --thread-input --progress --no-psnr --output "" ""

x264.exe --bitrate=1116 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 6
--trellis 1 --analyse all --8x8dct --me umh --threads 2 --thread-input --progress --no-psnr --output "" ""
Note: Sizes of the muxed files when only the crf ratefactor from the ABR sample was used directly to encode:
Source 1 - Target was as above - 1,126,400,000. Final muxed size was 1,186,472,026 which is 105.33% of target
Source 2 - 4,586,496,000 4,984,984,695 108.69%

@ ToS_Maverick - I sent you a PM earlier about your question

ToS_Maverick

30th October 2006, 22:32

well it seems with 24 it's pretty accurate too. although with the input bitrate of 1006 i got 1037, wich stayed about the same with the crf (1044).

DDogg

31st October 2006, 15:00

With 5 sources completed with an average accuracy of 97.49% I think we can say we have a winner.

Source 2
frames = 150,804
Framerate = 23.976
Complexity low (subjective)
MeGUI Bitrate Calc'ed - 1017 average KB/s - 1100 MB muxed [1,126,400 KB] (1/4dvd)
Actual video = 753,069,034 (right click properties - size - top value)
Actual Audio = 352,228,352 " "
Actual Muxed = 1,105,847,287 " "
Accuracy = 1,105,847,287 / 1,126,400,000 = 98.17%

Source 5
frames = 174,261
Framerate = 23.976
Complexity medium (subjective)
MeGUI Bitrate Calc'ed - 818 average KB/s - 1,126,400 KB] (1/4dvd)
video = 685,505,901 (right click properties - size - top value)
audio = 407,018,752 b " "
Actual Muxed = 1,093,132,035 " "
Accuracy = 1,093,132,035 / 1,126,400,000 = 97.05%
Now comes the question of why this is working.

@akupenguin - On source 5 the abr sample was run with a bitrate value of 818. The values returned were:avis [info]: 640x272 @ 23.98 fps (1752 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
mp4 [info]: initial delay 2002 (scale 24000)
x264 [info]: slice I:133 Avg QP:19.65 size: 18191:00:00
x264 [info]: slice P:750 Avg QP:21.03 size: 5776
x264 [info]: slice B:869 Avg QP:23.06 size: 1477
x264 [info]: mb I I16..4: 18.4% 62.0% 19.7%
x264 [info]: mb P I16..4: 5.6% 12.0% 2.1% P16..4: 37.0% 15.2% 6.7% 0.7% 0
.3% skip:20.4%
x264 [info]: mb B I16..4: 0.3% 0.8% 0.3% B16..8: 30.9% 1.5% 3.2% direct:
2.2% skip:60.8%
x264 [info]: final ratefactor: 18.56
x264 [info]: 8x8 transform intra:61.4% inter:63.6%
x264 [info]: ref P 80.7% 13.2% 6.1%
x264 [info]: ref B 87.6% 9.0% 3.4%
x264 [info]: SSIM Mean Y:0.9825506
x264 [info]: kb/s:879.7

encoded 1752 frames, 17.89 fps, 880.93 kb/sThe 880.98 kb/s is the kb/s value I converged the crf sample to equal. The crf used was 19.04.

So one question is why 880.98 is shown on an ABR encode using --bitrate 818? It is a good thing, just would like to understand better. Here is some more data showing -bitrate and the reported br from the cli and other related values. Frames 141241 150804 196159 243770 174261
FrameRate 23.976 23.976 23.976 23.976 23.976
Source 1 Source 2 Source 3 Source 4 Source 5

MeGUI BR calc ABR 1116 1017 741 3247 818
ABR sample size 8,344,209 8,516,423 8,019,023 41,362,145 8,065,785
Final Ratefactor 17.56 15.94 18.75 16.05 18.56
KB in ABR cmd window 1128.38 1078.19 779.59 3238.79 880.79

Converged crf 18.29 16.39 19.26 16.8 19.04
Converged crf size 8,344,591 8,524,873 8,021,910 41,425,181 8,064,486

Difference 0.73 0.45 0.51 0.75 0.48Temp note: Thinking out loud - There might be some type of simple relationship "factor" to apply to the final ratefactor so that none of the convergence needs to be done. Anybody "see" it? Maybe something to do with the relationship of -bitrate xxx and the KB/s reported by the ABR line?
Hmmm, if you divide the reported kb from the abr cmd window / the bitrate actually used (--bitrate) it gets interesting - Notice the 2 values 0.73 and 0.75 show close to 1, whereas the others tracking around ~ .50 show ~1.05. If that were to prove out, one 1% ABR sample would provide all the information one needs to identify the crf needed. Wouldn't that be a hoot!Difference CVRg'ed crf - RTFactor 0.73 0.45 0.51 0.75 0.48
% diff ABR reported 1.01 1.06 1.05 1.00 1.08
A rule of thumb if one does not care to do the convergence might be to simply take the crf rangefactor from the ABR sample window, add .50-.65 to it, and use that crf value for the encode.

@lexor \/ When we get it all figured out, there will be a how to do it post condensing all this bumbling around in the dark

lexor

31st October 2006, 15:24

Now comes the question of why this is working.

I vote Voodoo magiks.
Also is there a thread explaining this thread for the rest of us (all that stuff about what crf and others are and what they do etc etc)?

akupenguin

31st October 2006, 18:21

So one question is why 880.98 is shown on an ABR encode using --bitrate 818? It is a good thing, just would like to understand better. Here is some more data showing -bitrate and the reported br from the cli and other related values.
Because ABR means Average BitRate, not Constant BitRate. At any given point in the movie, the local bitrate will not be equal to the target, only the average is. But when the movie length is only 1752 frames, there's a lot of room for random variation.

There might be some type of simple relationship "factor" to apply to the final ratefactor so that none of the convergence needs to be done. Anybody "see" it? Maybe something to do with the relationship of -bitrate xxx and the KB/s reported by the ABR line?final ratefactor is supposed to be the crf value which would have produced the target bitrate. So theoretically you shouldn't need any convergence to match the bitrate of the sample you ran ABR on (not counting any difference between the sample and the full movie). But I have noticed before that it's not quite correct.

Edit: Also note that the final ratefactor is estimated for the target bitrate, not for the bitrate that ABR actually ended up with.

DDogg

31st October 2006, 19:27

Because ABR means Average BitRate, not Constant BitRate. At any given point in the movie, the local bitrate will not be equal to the target, only the average is. But when the movie length is only 1752 frames, there's a lot of room for random variation..
Yeah, I get that, but I wonder if it is simply random, or if there is significance in the fact that the sources with the highest level of complexity are showing the highest variation.
final ratefactor is supposed to be the crf value which would have produced the target bitrate. So theoretically you shouldn't need any convergence to match the bitrate of the sample you ran ABR on (not counting any difference between the sample and the full movie). But I have noticed before that it's not quite correct.Well don't change it now :)

akupenguin

31st October 2006, 20:35

Sources with the highest total complexity may also have the highest variation in complexity as a function of time. Even if a movie looks like non-stop action, some frames will be easier to encode than others. But if it looks like nothing is happening, then looks are probably correct.

DDogg

1st November 2006, 00:56

It seems safe to say we now have the ability to use crf to accurately hit a filesize.

To test accuracy with a completely different source than the five above, I wrapped an hourly series episode avi in directshow. Then did the same procedure mentioned above. MeGUI showed the average bitrate of 740 would be needed and the muxed final filesize at 358,400,000 b (1/2 cd).

A 1% ABR sample was done at 740 and showed a final rate factor of 17.99 and a kb/s of 709.43. Several 1% crf sample were run until a value was found to create the same filesize/same kb/s as the abr sample. Episode was then encoded at crf 19.0 (the converged crf).

Resulting muxed filesize was 356,751,738 - so, 356,751,738 / 358,400,000 = 99.54% accuracy in size prediction using only a 1% sample on a short source.

ToS_Maverick

1st November 2006, 14:02

test 3 is now complete, here are my results :cool:

D:\Program Files\meGUI\tools\x264\x264.exe" --crf 17.73 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --me umh --threads 2 --thread-input --no-psnr --output "F:\CD-Images\24 - Season 4\E13\E13 comp CRF 17.73_hope1006_2pct.mp4" "F:\CD-Images\24 - Season 4\E13\E13 comp.avs" >> "F:\CD-Images\24 - Season 4\E13\DDogg.txt" 2>&1

avis [info]: 640x352 @ 25.00 fps (1272 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
mp4 [info]: initial delay 2 (scale 25)
x264 [info]: slice I:89 Avg QP:16.00 size: 15630
x264 [info]: slice P:729 Avg QP:17.77 size: 6229
x264 [info]: slice B:454 Avg QP:19.94 size: 1455
x264 [info]: mb I I16..4: 34.3% 52.5% 13.3%
x264 [info]: mb P I16..4: 15.1% 18.2% 2.5% P16..4: 36.5% 12.0% 3.2% 0.2% 0.1% skip:12.3%
x264 [info]: mb B I16..4: 0.7% 0.8% 0.1% B16..8: 31.9% 0.8% 1.8% direct: 2.0% skip:62.0%
x264 [info]: 8x8 transform intra:51.2% inter:75.4%
x264 [info]: ref P 69.9% 19.3% 10.8%
x264 [info]: ref B 85.2% 10.8% 4.0%
x264 [info]: SSIM Mean Y:0.9866294
x264 [info]: kb/s:1036.5

encoded 1272 frames, 15.54 fps, 1037.84 kb/s

"D:\Program Files\meGUI\tools\x264\x264.exe" --crf 17.73 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --me umh --threads 2 --thread-input --no-psnr --output "F:\CD-Images\24 - Season 4\E13\E13 comp CRF 17.73_hope1006_full.mp4" "F:\CD-Images\24 - Season 4\E13\E13.avs" >> "F:\CD-Images\24 - Season 4\E13\DDogg.txt" 2>&1

avis [info]: 640x352 @ 25.00 fps (63350 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
mp4 [info]: initial delay 2 (scale 25)
x264 [info]: slice I:914 Avg QP:15.36 size: 16490
x264 [info]: slice P:36536 Avg QP:17.47 size: 6404
x264 [info]: slice B:25900 Avg QP:19.16 size: 1750
x264 [info]: mb I I16..4: 36.6% 50.2% 13.2%
x264 [info]: mb P I16..4: 12.9% 16.0% 2.3% P16..4: 39.2% 13.8% 4.2% 0.2% 0.1% skip:11.3%
x264 [info]: mb B I16..4: 0.9% 1.0% 0.1% B16..8: 34.0% 1.0% 2.1% direct: 2.7% skip:58.2%
x264 [info]: 8x8 transform intra:51.2% inter:74.1%
x264 [info]: ref P 64.2% 21.8% 14.0%
x264 [info]: ref B 77.7% 15.6% 6.7%
x264 [info]: SSIM Mean Y:0.9864954
x264 [info]: kb/s:929.4

encoded 63350 frames, 16.81 fps, 930.51 kb/s

as expected the crf-method produced a smaller file. it didn't suprise me, as i had experience from my prediction test.

my next test will be a crf-prediction for the whole dvd (12 eps at once).

usually i backup a season on 2 dvds (in this case ~370 mb/ep). using always the same size for an episode isn't really constant quality. with season 4 i tried it with with crf which resulted in constant quality, but the dvd wasn't fully used.

buzzqw

1st November 2006, 15:20

i am using a little different approach

i encode a bigger sample (5/7%) (in abr )and look at final ratefactor

if in short test this value isn't accurate on bigger sample is near perfect.

I usually subtract a 0.1 to this value and from (small) test i made this value the correct one

BHH

DDogg

1st November 2006, 15:31

Thanks, ToS_Maverick - This new method mentioned in the last few posts seems to be proving out, but I did notice one oddity last night on a low low complexity AVI source.

It seems if the kb/s reported back by the ABR sample is LESS than the --bitrate xxx used to make the sample you would not need to do the convergence and simply use the final rangefactor for your full crf encode.

I still think there is a a key relationship between the ABR --bitrate used and the kb/s reported back by the abr sample window. We need to figure out how to incorporate that relationship into the method. Keep your eyes open when testing to see if something jumps out at you. My eyes are crossed from messing with this stuff so long now and I may not be seeing the forest for the trees.

@buzzqw - you may be right - that would tend to agree with what akupenguin said: final ratefactor is supposed to be the crf value which would have produced the target bitrate. So theoretically you shouldn't need any convergence to match the bitrate of the sample you ran ABR on (not counting any difference between the sample and the full movie). But I have noticed before that it's not quite correct.

Edit: Also note that the final ratefactor is estimated for the target bitrate, not for the bitrate that ABR actually ended up with.
A longer ABR sample might get a better 'taste' of the source. If the convergence samples were not needed, the time saved could pay for the longer 5% sample. [nope, this did not prove out]

/add: via pm, Buzzqw has made a very good suggestion - by adding --ratetol inf to the 1% abr sample line we may get a more stabilized final range factor from the abr sample that can be used directly for the crf encode - without any of the convergence hassle. To prove this out will require all sources to be re-sampled and re-encoded. Will take some time to do, but the initial results look promising. More later, way later :)

@akupenguin (or others) - would you suggest any other additions to the ABR sample line to better "stabilize" the FRF?

DDogg

1st November 2006, 21:48

Using just the final ratefactor from a 1% sample using --ratetol inf in the sample ABR command line. [NOTE: --ratetol inf not used in the crf cmdline for full encode]

Source 1:
Target Muxed = 1,126,400,000 bytes
Actual Muxed = 1,160,902,139 bytes
Accuracy = 103.06% of target size

Source 2
Target Muxed= 1,126,400,000 bytes
Actual Muxed = 1,132,365,338 bytes
Accuracy = 100.53% of target size

Source 3:
Target Muxed = 1,126,400,000 bytes
Actual Muxed = 1,098,796,941 bytes
Accuracy = 97.03% of target size

Source 4: (512x272)
Target Muxed = 2,150,400,000 bytes
Actual Muxed = 2,148,083,551 bytes
Accuracy = 99.89% of target size

Source 5:
Target Muxed = 1,126,400,000 bytes
Actual Muxed = 1,109,990,781 bytes
Accuracy = 98.54% of target size

Shinigami-Sama

1st November 2006, 23:20

ohhh
I just found this thread today
great work guys
if you need more tests I can drop some tests for you
maybe not very timely but I'd like to help out

DDogg

2nd November 2006, 14:42

With all 5 sources re-encoded the average accuracy was 99.81% Here are the basic instructions (http://files.project357.com/uploads/index.php?dir=&file=Simplified_test_instructions_2.txt)so you can test this method yourself.

add: It is interesting to note only one source of the five would have triggered the need for a conditional 2nd pass if absolute oversize control was needed.

Shinigami-Sama

2nd November 2006, 18:09

ohh
that saves me some time

akupenguin

2nd November 2006, 19:02

With all 5 sources re-encoded the average accuracy was 99.81%
No, the average filesize was 99.81% of the target, but that says nothing about the accuracy. If you encoded 2 files and one came out 0kbps and the other 2x the target, would you call that 100% accuracy?
The 99.81% just says that there's little systematic error. What matters for the accuracy is the standard deviation, which is +/- 2.02% in your data.

Shinigami-Sama

2nd November 2006, 19:06

eitherway its still very impressive for a what looks like it turned out to be a hunch doncha think?

but lets see if we can get that SD down to 1% eh?

DDogg

2nd November 2006, 19:38

No, the average filesize was 99.81% of the target, but that says nothing about the accuracy. If you encoded 2 files and one came out 0kbps and the other 2x the target, would you call that 100% accuracy?
The 99.81% just says that there's little systematic error. What matters for the accuracy is the standard deviation, which is +/- 2.02% in your data.Yep, thanks for posting this. The standard deviation was what I wanted, but was math impaired - 2% was the goal, so we are close on these ntsc standard res tests. Hoping another little tweak here and there will give us that. Also hope it will hold up for additional source types. Given the ability of a conditional second pass, if there is an unacceptable deviation on the occasional encode, we get the best of both worlds. CRF one pass speed on many and absolute oversize/undersize control where needed.

DDogg

3rd November 2006, 18:18

For those interested in playing around with this method, but wanting it GUI easy, keep an eye on the AutoMKV thread in the MPEG-4 Encoder GUIs sub forum.

Buzzqw has implemented this method plus more, including the conditional 2nd pass, in a private test version. It is looking very good. I can't speak for Buzzqw on when he will release a test version, but he indicated it would be soon, possibly today or tomorrow, so keep an eye open if you are interested in testing.

Kurth

3rd November 2006, 22:22

Encode_full.avs
AVISource("E:\Encode\video.avi")
Lanczos4Resize(720,400)
Levels(16,1,240,0,255)

Encode_1P.avs
AVISource("E:\Encode\video.avi")
Lanczos4Resize(720,400)
Levels(16,1,240,0,255)
SelectRangeEvery(1200,12)

Calculator
http://img518.imageshack.us/img518/5464/calculatorxy2.jpg

E:\Encode>E:\Encode\x264.exe --bitrate=703 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 6 --trellis 1 --analyseall --8x8dct --me umh --threads 2 --thread-input --progress --no-psnr --ratetol inf --output "E:\Encode\Encode_ABR_703_1p.mp4" "E:\Encode\Encode_1p.avs"
avis [info]: 720x400 @ 23.98 fps (348 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE 3DNow!
mp4 [info]: initial delay 250 (scale 2997)
x264 [info]: slice I:29 Avg QP:23.41 size: 19701:00
x264 [info]: slice P:119 Avg QP:25.71 size: 4139
x264 [info]: slice B:200 Avg QP:27.41 size: 642
x264 [info]: mb I I16..4: 25.6% 48.4% 26.0%
x264 [info]: mb P I16..4: 4.0% 6.6% 2.6% P16..4: 31.7% 5.7% 2.5% 0.2% 0.1% skip:46.7%
x264 [info]: mb B I16..4: 0.3% 0.5% 0.1% B16..8: 13.6% 0.5% 0.8% direct: 0.7% skip:83.5%
x264 [info]: final ratefactor: 21.03
x264 [info]: 8x8 transform intra:49.1% inter:68.4%
x264 [info]: ref P 78.7% 12.8% 8.5%
x264 [info]: ref B 82.3% 14.4% 3.3%
x264 [info]: SSIM Mean Y:0.9788516
x264 [info]: kb/s:657.1

encoded 348 frames, 5.82 fps, 658.58 kb/s

E:\Encode>E:\Encode\x264.exe --crf 21.03 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --me umh --threads 2 --thread-input --progress --no-psnr --output "E:\Encode\Encode_CRF21.03_Full.mp4" "E:\Encode\Encode_full.avs"
avis [info]: 720x400 @ 23.98 fps (34518 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE 3DNow!
mp4 [info]: initial delay 250 (scale 2997)
x264 [info]: slice I:391 Avg QP:19.53 size: 265340:00:00
x264 [info]: slice P:13607 Avg QP:21.86 size: 7242
x264 [info]: slice B:20520 Avg QP:22.49 size: 1279
x264 [info]: mb I I16..4: 20.9% 51.2% 27.9%
x264 [info]: mb P I16..4: 7.2% 10.5% 3.5% P16..4: 37.5% 9.1% 4.4% 0.3% 0.2% skip:27.2%
x264 [info]: mb B I16..4: 0.2% 0.4% 0.1% B16..8: 23.3% 0.6% 1.4% direct: 2.0% skip:72.0%
x264 [info]: 8x8 transform intra:49.7% inter:65.0%
x264 [info]: ref P 74.5% 15.9% 9.6%
x264 [info]: ref B 82.5% 13.4% 4.1%
x264 [info]: SSIM Mean Y:0.9848128
x264 [info]: kb/s:751.0

encoded 34518 frames, 5.70 fps, 752.08 kb/s

Results
http://img54.imageshack.us/img54/9066/resultsbc4.jpg

Well this encode made a file size of 148MB and I calculated 140MB soo I got 8MB more than I wanted.

Your way of CRF file size prediction dont work for my Anime encode.

Maybe I did something wrong or this way of encoding dont fit for me but that was a nice test :D

buzzqw

3rd November 2006, 23:04

@Kurth

for a quality encoding the result isn't so bad !

you can try the new automkv build (0.50), you can trigger a 2 pass encoding if the quality pass goes oversized or undersized !

BHH

DDogg

4th November 2006, 05:26

Kurth, you go down in the tester hall of fame for excellent reportage! Thanks for doing such a great job.

As for you results on that very short source - Actually I am pretty happy to see it. You were 105.71% of target on a 24 minute anim source using a 1% sample to predict which was only 345 frames! Also, putting the final size aside, you knew before starting the encode your crf would be around 21.03 which is very useful information to have before doing the encode.

On these short sources, or actually any of these tests, remember to add the --pass=1 stats ".stats" to the crf line - if you do miss the target, you simply run the second pass with the same cmdline changing --crf to --bitrate xxx-(what u used in the abr line) and change pass=1 to pass=2

That way you have wasted no time over a normal 2 pass encode.

It would be great if you could test some longer anim sources. btw, on that short a source I just can't see how a 1% sample could work well. I wonder if you would get a more accurate predicted crf with a 2% or 5% sample? Would be useful information for us if you feel like messing with it.

For 2% use - SelectRangeEvery(600,12)
For 5% use - SelectRangeEvery(240,12)

akupenguin

4th November 2006, 06:57

DDogg

4th November 2006, 07:26

@Kurth
You skipped the step where you run another 1% sample using the final ratefactor. ABR's FRF is an estimate. I'm not sure if it's inherently innacurate or if that's a bug, but it usually doesn't exactly match the appropriate CRF value even for the exact sample ABR ran on.Loren, you might have not caught the abrupt change where we dropped the convergence step. We added --ratetol inf in an effort to force a more accurate FRF and have been using the FRF generated by the ABR sample directly for the full encode.

I hope you might have the time to suggest ways to get a more accurate FRF from the ABR line, either through suggested mods to the test abr cmdline, or maybe looking at the code to see if there is something that pops out at you.

Kurth, prior to this set of tests, we were running the crf line with the 1P.avs and noting the KB/s shown in the cmd window. Tweaks were made to the crf and the sample was re-run repeatedly until the kb/s in both windows agreed. That crf was then used for the encode using _full.avs. It still may be the way we have to go.

But it wastes quite a bit of time over a turbo 2pass encode. So this technique is good, but only as long as you usually don't have to add a 2nd pass.Yep, will not be much use if size control is the main reason this is used and say, 4 out of 5 don't require the second pass. Still, just the sample gives a lot of useful info before taking the time to do the full encode.

foxyshadis

4th November 2006, 07:40

Maybe it's because it was only 12 seconds (350 frames) of analysis, and 15 minutes of full encode. So far it's only been tested on full-length movies, correct? We already know that the shorter it is, the less confidence there is in the initial result, so maybe a rule of thumb that with less than x frames, a second first pass is run.

Shinigami-Sama

4th November 2006, 07:45

mk
so tomorrow I'll beable to set up a few encodes
do you want any specific size?
I think I can chop up a DVD source into a some smaller pieces if ya think it'll be a good idea

akupenguin

4th November 2006, 08:11

Loren, you might have not caught the abrupt change where we dropped the convergence step. We added --ratetol inf in an effort to force a more accurate FRF and have been using the FRF generated by the ABR sample directly for the full encode.
ratetol=inf does not eliminate the effect, just reduces it. And I can't think of any other options to add. Further improvement will have to modify how FRF is computed.

Kurth

4th November 2006, 10:32

I did the ABR encode again now with 2% and 5%.

SelectRangeEvery(600,12) #2% 696 Frames
x264 [info]: final ratefactor: 21.66

SelectRangeEvery(240,12) #5% 1728 Frames
x264 [info]: final ratefactor: 21.58

What exactly Im supposed to do now to get better FRF results ?

Did I have to do now an encode with 21.66 and another encode with 21.58 to see which one get more near the 140MB that I want to decide which prediction is better 2% or 5% ?

buzzqw

4th November 2006, 10:51

Did I have to do now an encode with 21.66 and another encode with 21.58 to see which one get more near the 140MB that I want to decide which prediction is better 2% or 5% ?

no need to do 2 other encode, the bigger value of frf will produce a smaller file

BHH

Kurth

4th November 2006, 20:15

Encode_full.avs
AVISource("E:\Encode\video.avi")
Lanczos4Resize(720,400)
Levels(16,1,240,0,255)

Encode_1P.avs
AVISource("E:\Encode\video.avi")
Lanczos4Resize(720,400)
Levels(16,1,240,0,255)
SelectRangeEvery(600,12) #2%

Calculator
http://img518.imageshack.us/img518/5...culatorxy2.jpg

E:\Encode>E:\Encode\x264.exe --bitrate=703 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 6 --trellis 1 --analyseall --8x8dct --me umh --threads 2 --thread-input --progress --no-psnr --ratetol inf --output "E:\Encode\Encode_ABR_703_1p.mp4" "E:\Encode\Encode_1p.avs"
avis [info]: 720x400 @ 23.98 fps (696 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE 3DNow!
mp4 [info]: initial delay 250 (scale 2997)
x264 [info]: slice I:50 Avg QP:23.32 size: 18688:00
x264 [info]: slice P:280 Avg QP:25.48 size: 4641
x264 [info]: slice B:366 Avg QP:27.45 size: 642
x264 [info]: mb I I16..4: 26.5% 48.3% 25.2%
x264 [info]: mb P I16..4: 9.7% 10.5% 3.6% P16..4: 27.9% 5.4% 2.2% 0.2% 0.1% skip:40.4%
x264 [info]: mb B I16..4: 0.2% 0.3% 0.1% B16..8: 14.3% 0.5% 0.8% direct: 0.7% skip:83.0%
x264 [info]: final ratefactor: 21.66
x264 [info]: 8x8 transform intra:46.1% inter:68.5%
x264 [info]: ref P 81.3% 12.1% 6.6%
x264 [info]: ref B 84.5% 12.6% 2.9%
x264 [info]: SSIM Mean Y:0.9807621
x264 [info]: kb/s:680.4

encoded 696 frames, 5.25 fps, 681.71 kb/s

E:\Encode>E:\Encode\x264.exe --crf 21.66 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --me umh --threads 2 --thread-input --progress --no-psnr --output "E:\Encode\Encode_CRF21.66_Full.mp4" "E:\Encode\Encode_full.avs"
avis [info]: 720x400 @ 23.98 fps (34518 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE 3DNow!
mp4 [info]: initial delay 250 (scale 2997)
x264 [info]: slice I:394 Avg QP:20.15 size: 247950:00:00
x264 [info]: slice P:13608 Avg QP:22.48 size: 6581
x264 [info]: slice B:20516 Avg QP:23.10 size: 1143
x264 [info]: mb I I16..4: 21.7% 51.5% 26.8%
x264 [info]: mb P I16..4: 7.5% 10.2% 3.3% P16..4: 36.4% 8.6% 4.0% 0.3% 0.1% skip:29.6%
x264 [info]: mb B I16..4: 0.2% 0.3% 0.1% B16..8: 22.3% 0.5% 1.3% direct: 1.7% skip:73.5%
x264 [info]: 8x8 transform intra:48.9% inter:66.0%
x264 [info]: ref P 74.4% 15.7% 9.8%
x264 [info]: ref B 82.5% 13.3% 4.2%
x264 [info]: SSIM Mean Y:0.9840405
x264 [info]: kb/s:682.2

encoded 34518 frames, 4.58 fps, 683.20 kb/s

Results
http://img218.imageshack.us/img218/1853/resultsky1.jpg

This time using ABR 2% prediction I got a final file size of 137MB soo this time I got a file 3MB less than the 140MB that I wanted.

I think this is good because it is more near the 140MB and it is not oversized. The imagem quality is very good and it was just a 1-Pass encode :)

DDogg

6th November 2006, 18:01

ratetol=inf does not eliminate the effect, just reduces it. And I can't think of any other options to add. Further improvement will have to modify how FRF is computed. Maybe it is an area you can look at one day. I'm thinking it might be impossible, even for you.

@Kurth, thanks, those results were helpful and I am glad it worked for you.

tyee

6th November 2006, 19:25

DDogg

6th November 2006, 21:31

This is a great topic, so keep it going guys!

I tried this method yesterday using DDogg's procedure and I got a resultant muxed file about 7% too large. The crf that the 2% test encode told me to use was 15.08. I'll recheck my figures tonight just to make sure. I might try it with a 1% sample instead of 2% also since it is a movie.tyee, I assume by "procedure" you meant this last interation of using the FRF in the ABR sample for the full crf encode?

If so, you could do us a big favor and use the 1 or 2P.avs in your crf line. Run it and keep tweaking the crf and rerunning the crf sample until the kb/s in the abr window from your first sample, and the kb/s from your crf sample are very close in value. Then use that crf for the full encode. It only takes a few crf samples to converge down on the kb/s that was at the bottom of the abr sample window.

It is simple to do and would give us more data. Ask back if not clear. 1% should be fine on a longer source like a movie and would save you some sample time.

tyee

6th November 2006, 23:30

tyee, I assume by "procedure" you meant this last interation of using the FRF in the ABR sample for the full crf encode?

Yes DDogg that's what I did. I will try out your suggestion tonight.

tyee

7th November 2006, 06:52

Here's my first figures from testing the 1% avs, running a bitrate average first, then running the crf passes to get the same bitrate as shown on the first average run.

1st average run gave 1697.5 kbps/crf 15.12 using a MeGUI suggested average bitrate of 1541 kbps
2nd run using crf of 15.12 gave 1768.28 kbps
5th run gave 1700.41 kbps/crf 15.40
7th run gave 1695.40/crf 15.42
8th run gave 1696.75 kbps/crf 15.41

This is closest I can get. I will now run the full.avs overnight using a crf = 15.41.

Whoops. Just discovered I used the command line without the --ratetol inf when I was doing the 1% passes. Oh well, I just started to run the full.avs anyway. We'll see.

Update -- Final Results--Excellent if you ask me!! ----

Final Encoding window of full.avs -> bitrate = 1538.5 kbps (MeGUI said to use 1541 kbps)

Actual (Video only) encoded file size = 1136005392 bytes
MeGUI (video only) calculated filesize = 1135087616 bytes
Error = +0.08%

Actual Muxed filesize (.mkv) = 1466828079 bytes
MeGUI calculated muxed filesize = 1468006400 bytes
Error = -0.08%

Well, I don't think we could get better than that!

buzzqw

7th November 2006, 08:20

using automkv (that automates all process) on Poseidon

with 2% sampled tested (aiming to 1 CD conversion) got a crf of 20.01

the final muxed size was 687 MB (audio+video), not too bad ! for a 1 pass CRF conversion !

BHH

DDogg

8th November 2006, 20:16

Final Encoding window of full.avs -> bitrate = 1538.5 kbps (MeGUI said to use 1541 kbps)

Actual (Video only) encoded file size = 1136005392 bytes
MeGUI (video only) calculated filesize = 1135087616 bytes
Error = +0.08%

Actual Muxed filesize (.mkv) = 1466828079 bytes
MeGUI calculated muxed filesize = 1468006400 bytes
Error = -0.08%

Well, I don't think we could get better than that!tyee, just caught your added results. Excellent. Thanks much for taking the time to do this extra test and post the results so clearly.

tyee

8th November 2006, 21:17

DDogg

9th November 2006, 01:03

DDogg
Yes, this method is pretty slick. Is AutoMKV doing this multipass method in the current version or is it doing just the first 1pass average, getting the crf and then using that value.

How can we automate this better by narrowing in on the nearest crf faster? Can we use any math equation from the graphs you made earlier in this post and just use maybe two crf passes then calculate?AutoMKV is currently using the FRF from the ABR sample to do the CRF encode. I don't know if buzzqw plans any changes.

As for method, a similar convergence method is used in several mpg2 programs and is well documented. Normally the convergence can be done in 3, sometimes 2, and sometime 4 samples. If buzzqw does decide to investigate this method, he will find plenty of information on how to do it.

tyee

17th November 2006, 16:36

I did some prelim investigation with a curve fitting program and used some of DDogg's figures from one of the excel tables and found a good equation to get crf. It is --

crf = a + b (ln bytes)

where a and b are constants, which we get by taking two samples of whatever percent you want. I used crf's of 15 and 24 from the excel tables.

Then using simultaneous equations, solve to get a and b.

Then pick a final "bytes" size for the final full encode and calculate your crf to get that final size. It worked exact so far.

Note - "ln bytes" is the natural log of the bytes value.

I have more testing to do.

tyee

22nd November 2006, 07:22

I have now finished my GUI for automating this procedure and am testing it right now. I'll let you know how accurate it is in a day or so.

DDogg

24th November 2006, 04:50

I have now finished my GUI for automating this procedure and am testing it right now. I'll let you know how accurate it is in a day or so.I have not had much time for this stuff lately, but look forward to seeing your GUI and any data on accuracy.

Dayvon

24th November 2006, 13:39

I have now finished my GUI for automating this procedure and am testing it right now. I'll let you know how accurate it is in a day or so.
Kick a$$ Tyee! I've tested this method out a few times, and I like it alot. But I have been waiting and hoping some wiz would come in here and think up a way to bring this to the masses. Looking forward to checking out your work!!

tyee

24th November 2006, 16:25

The accuracy is not quite what we get when doing it manually even though the program uses basically the same method. Give me another couple of days to dig a little deeper then I'll post it.

DDogg

25th November 2006, 15:04

The accuracy is not quite what we get when doing it manually even though the program uses basically the same method. Give me another couple of days to dig a little deeper then I'll post it.My intuition tells me there is something important about the level of differential between the size target bitrate and the bitrate reported by the sample cli. Sources like SPR tend to come in off size and SPR also shows a larger difference (higher if I remember correctly). Might be an area you could drill down on.

tyee

26th November 2006, 01:35

OK, here is my GUI. Please test it out. This GUI does basically what we were doing manually. That is it first runs an average pass command line, then a "crf low" command line, then a "crf high" command line. It then calculates the crf required to get the same bitrate that was shown in the average pass first run command line window. Then it uses this value to encode the full movie using the "Full AVS" file [without the SelectRangeEvery(1200,12)] line.

Instructions --

-Unpack all the files to the same directory. Do not rename any of the batch files. I have left the x264.exe command line in the two batch files so anyone can change the option switches if we want to get better quality in the future without changing the gui.

-Copy your x264.exe into this same directory too. It's not included!

-Run the GUI and enter your chosen values for CRF Low, CRF High, 1% AVS File and Full AVS File. I have default values of CRF Low and High of 18 and 22. Previously I was using 15 and 25. The net result difference was less than 0.1 crf, but you can change these if you want by just entering in your chosen values.

-Enter your chosen average bitrate from MeGUI's calculations or using your favourite bitrate calculator.

-Press START!

You will be able to see the final chosen crf value in the DOS window command line during the final encode. If you just want to test without doing the whole movie, just add the "SelectRangeEvery(1200,12)" to the end of the Full AVS file. This will allow you check the final byte size of the 1% sample against the average size of the 1% sample. These 3 samples are not deleted by the GUI at the end of the encoding.

download is on next page of this thread.

Shinigami-Sama

26th November 2006, 01:39

I have a question here about this
I've not had the time all together to be doing these tests sady :(

but I'll be going away for a while and was wondering if anyone knew how I set this up to I put the scripts into a schedule
like run one one day
the next one the next day
so forth and so on?

I'd like to beable to help but I'm not good with windows scripting

foxyshadis

26th November 2006, 14:01

DDogg

26th November 2006, 16:26

tyee, nice job. Simple, clean and it works. I have not had time to measure the accuracy. I hope to find some time later in the week.

However, even if the prediction is not perfect, this tool is something nearly everybody can use before they do an encode, even if they are doing a conventional 2 pass.

In just a few minutes, it will point out in advance of the full encode, whether enough bitrate has been allocated by indicating the approximate quality/crf value the encode will produce within the allocated bitrate. This will save folks a lot of wasted time.

To that end you might want to consider adding a checkbox controlling whether to do the full encode or just stop and show the CRF information. People could then also use it as a simple quality prediction tool. It would be great if somebody could just select their full avs and your program would make a copy and add the 1% sample line to that script automatically.

Shinigami-Sama

26th November 2006, 21:29

Look at windows task manager (aka scheduled tasks). If you're confident in the stability of your system and the power grid, though, you can just put all the x264 commands in a single batch file, which will run them one by one. They might still be running when you return, if it's long enough!

thats what I was figuring. Thanks foxy
its got about two weeks, I think it'll be long enough to do five or six tests untill I run out of diskspace

tyee

27th November 2006, 01:34

Thanks DDogg. I'll work on adding your suggestions to it!

tyee

27th November 2006, 07:17

Updated with DDogg's suggestions plus a couple of other options too!

http://tplay.lunarpages.net/tiny/eFLiN

quake74

28th November 2006, 10:42

Updated with DDogg's suggestions plus a couple of other options too!

http://tplay.lunarpages.net/tiny/eFLiN

Is it just me, or there is a "--threads 2" in the bat files? A while ago, running 2 threads on a single core machine was producing unplayable files, is it still the case? Or does "--thread-input" mean that it is 1 thread for avs decoding and 1 for x264 encoding? And I think that if I input only three digits for the bitrate (like 700) then it says something about a missing file. (Will try again later, doing and encode now.)

Edit: Apparently it is not related to bitrate, but (I guess) to the cap program. I can see it creates avgbitrateline.txt pass1avgbitrateline.txt pass2avgbitrateline.txt but they are empty. Weird, because in a different directory it used to work.

CruNcher

28th November 2006, 16:18

jep same here "File Not Found" after each encode

DDogg

28th November 2006, 17:18

Worked perfectly for me ... you guys did make sure x264.exe was in the folder with tyee's program?

quake74

28th November 2006, 17:27

Worked perfectly for me ... you guys did make sure x264.exe was in the folder with tyee's program?

Yep, and I can say that because it does encode the avs to mp4 ;) So in the end I have my 3 mp4 but not the final one, and a bunch of "file not found" during the encoding process.

tyee

28th November 2006, 18:00

When I get home tonight I'll see if I forgot to add a check to prevent this. I don't have the code in front of me right now.

How about trying this --
Open a dos window in the directory that cap.com is in and do a "dir" command, then run the cap.com program from the same dos window (cap T) and see if it writes a file called "text.vid" to the same directory. Then take a look at "text.vid" and see if it captured the dos screen properly.

If that works, the next command is an internal dos "for" command that searches for the bitrate line in that text.vid file. Command extensions must be enabled on your computer for this to work. Try running this from the same dos window --> cmd /e:on

Then try running the GUI from the beginning again.

Let me know.

PS. I'll take that dos "for" command out of the GUI tonight and let the code extract the bitrate. That should fix this problem, but "cap T" still has to work!

quake74

29th November 2006, 11:02

When I get home tonight I'll see if I forgot to add a check to prevent this. I don't have the code in front of me right now.

How about trying this --
Open a dos window in the directory that cap.com is in and do a "dir" command, then run the cap.com program from the same dos window (cap T) and see if it writes a file called "text.vid" to the same directory. Then take a look at "text.vid" and see if it captured the dos screen properly.

It works correctly for me

If that works, the next command is an internal dos "for" command that searches for the bitrate line in that text.vid file. Command extensions must be enabled on your computer for this to work. Try running this from the same dos window --> cmd /e:on

Then try running the GUI from the beginning again.

Ah ah! It says (screencap made using cap ;) )

D:\dvd\raw\coupling>PATH
PATH=D:\download\ELDER_beta5e\parrot\bin;D:\download\ELDER_beta5e\bin;%SystemRo
t%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;C:\Program Files\Common Fil
s\GTK\2.0\bin

D:\dvd\raw\coupling>cmd
'cmd' is not recognized as an internal or external command,
operable program or batch file.

Funny, because cmd.exe is in c:\windows\system32. I'll check my environmental variables next...

tyee

30th November 2006, 05:30

quake74

30th November 2006, 08:28

OK, the new version is up at the same download link. It has no calls to any DOS commands (except the batch files) on your computer. It should therefore work for everyone!

Let me know how the accuracy is when you've had time to test.

Thanks,
tyee

OK, it works for me, although I just discovered it doesn't like names with spaces in it. I will use it a while and report back.

quake74

30th November 2006, 11:47

Weird... I did a test using a sitcom (29m 18s, 25fps) 20/24 max/min crf, 767 bitrate. The gui chose 23.95 crf but the final file came out 129256823 instead of (using megui bitrate calculator) 168581120. A smaller test (encoding just the 1% part as final avs) was right on spot with 1864458 (and the AVG767 file is 1857258). What did I do wrong?

foxyshadis

30th November 2006, 13:30

tyee,

Cap.com simply doesn't work on my system. I assume you're using it to grab that contents of the command window; why not just use "2>file.txt" if you need everything x264 outputs? Just skip every line starting with "encoded frames:".

Is it necessary to modify the actual avisynth file, instead of making a copy and making your changes to that? It makes me quite leery, but it hasn't caused a problem yet.

And you really need quotes around %2 and %3, or if you're not using the batch file now, in the equivalent syscall.

Still, I look forward to using this tool in the future, especially since my avs is usually the bottleneck.

tyee

30th November 2006, 16:03

quake74
Thanks for the info. Not sure what's happening there. I'll give another movie a try here and see if I get the same thing.

foxyshadis
Yes, I would like to use something other than cap.com. I'll change that with your suggestion.
I can make a copy of the avs. That's a better idea, and I'll add the quotes too.

Thanks for the tips.

tyee

quake74

30th November 2006, 18:21

Thanks for the info. Not sure what's happening there. I'll give another movie a try here and see if I get the same thing.

If you could make some kind of debug build which saves a log, so that I would try it, and have something to send back to you, and you would have something to work on.

tyee

1st December 2006, 07:23

I updated the gui again --

-make a copy of the avs file and work with that instead of original

- added quotes in batch files for filenames with spaces

- tried to use redirect (>) in batch files but didn't work, no matter what I tried. If anyone knows how to redirect to a file please let me know. I think the ouput of x264.exe does not sent it to stdout, and that's why dos redirection works and this doesn't. I'm still looking for a better way for this.

http://tplay.lunarpages.net/tiny/eFLiN

Shinigami-Sama

1st December 2006, 07:25

force it stdout with 1>?

foxyshadis

1st December 2006, 11:39

x264 puts everything on stderr, which is 2. x264 2>file.txt will capture it. (I thought you might need ">file 2>&1", but since there's nothing on stdout to capture, it's unnecessary.)

tyee

1st December 2006, 17:32

Thanks foxyshadis. I'll add it tonight!

tyee

2nd December 2006, 22:14

Ok, new updated version again with all the suggestions included. Cap.com is now gone! (was used for screen capture). Now the batch file output is written to "text.vid", which is then searched for the bitrate written by x264.exe.

One problem was encountered when I redirected stderr to a file. Yes, it did write to the file but the console screen window did not show any activity during the encoding, only a flashing cursor. I wanted to see the % encoded also, so I had to route the stderr output to a program included in the zipfile called "mtee.exe". This split the output to the screen and to the "text.vid" file so we have the best of both worlds. Thanks to foxyshadis for the knowledge of how to do the redirection. I hadn't done this before.

"mtee.exe" only works on WinNT,XP,2000,2003. I hope we all are using one of these operating systems.

http://tplay.lunarpages.net/tiny/eFLiN

tyee

DDogg

2nd December 2006, 22:42

Hmm .. Can anybody translate?

http://img131.imageshack.us/img131/5752/tyree01yx6.jpg (http://imageshack.us)

/Add: sorry, forgot to put x264.exe in the directory. You might want to do a check for that?

tyee

2nd December 2006, 22:47

DDogg
That's the error that occurs when the math calculator doesn't like what it's been given in the equation, or I should say that's when I've only seen it when I'm debugging. What version was that from and did it show up on your first use of it?

Ok, I see your addition. That could do it.

JeffAlso

17th December 2006, 22:44

Hey all.

This thread recently caught my attention, and I'd like to assist if you guys are still persuing this idea. The app posted above doesn't work for me, and gives the same error listed above.

Is this still being worked on??

tyee

18th December 2006, 03:17

Hi JeffAlso
Did you see DDogg's solution at the bottom of his post about adding the x264.exe to the folder. I didn't include it in the zipfile you downloaded.

tyee

JeffAlso

19th December 2006, 03:41

Thanks for the response

I found my problem, the path to my avs file contained spaces which goofed with the argument parsing in the batch file. I've got it working now.

I was wondering if you would share the algorithm that you're using in your prediction. I'm afraid I'm not as sharp as I'd like to be with the math involved, so I really need to see it laid out in order to understand it, and I'd very much would like to understand this technique - it would save me considerable amounts of time.

Thanks much in advance!!

tyee

19th December 2006, 05:06

Hi JeffAlso
I'm solving two simultaneous equations for two variables then using them to calculate the value of crf to use for the same bitrate as the 1st average pass. The program can't do anything more than that. We are assuming that the bitrate will average this value for the full movie, which may not be true but it should be fairly close.

The equation is --

square root (crf) = a + b (ln bitrate)

a and b are the two variables to solve for using the two crf passes of 18 and 25. For each pass the bitrate is given at the bottom of the dos window.

tyee

akupenguin

19th December 2006, 09:26

Where'd the square root come from? The appropriate equation is
crf = a + b * ln(bitrate)

tyee

19th December 2006, 16:50

Where'd the square root come from? The appropriate equation is
crf = a + b * ln(bitrate)

I found the square root equation gave slightly more accuracy over the 18 to 25 range when I was checking DDogg's results from one of his excel tables. Have you compared both to see how accurate they are?

DarkZell666

9th January 2007, 17:56

I also get the last error DDog posted. The 3 encoding operations run well (x264 is in the GUI's directory), and then the GUI pops that message that DDog shows.

For who does it work ? I tried V6 and had no luck (btw, the GUI's background is green, contrary to DDog's who's white).

tyee

10th January 2007, 16:04

DarkZell666
The only thing that comes to mind is the math calc routine came up with an invalid result. Try narrowing the range of CRF low and high values and try again. Try 20 for low and 22 for high just to see if the error doesn't occur.

DarkZell666

10th January 2007, 16:47

Changing the range from 18-22 to 20-22 worked. The question is: why ? Was it a floating-point overflow by any chance ?

Anyhow, I asked for 333kbps, and it resulted in a 26.6 crf for the anime source I tried (which is indeed somewhere around what I expected). Doing the full encode right now.

Thx for your contribution, it's very appreciable ;)

Edit: final encode wouldn't run. More precisely: x264 launched and quitted straight away (I saw the window go pop ^^). I didn't even have time to catch the logfile before it got deleted.
(the 3 test runs worked out great, and the crf calculation worked).

What could have gone wrong ? Could you update the GUI to keep the logfiles at the end, if it isn't too much work ?

Edit2: I also noticed two wierd files named "77" and "86" were created on the two occasions I tried. Are those error codes or something ?

DarkZell666

10th January 2007, 19:39

New observations:

I added a "pause" instruction at the end of the crf.bat file (stupid me for not thinking about this before :o), and noticed this:

D:\DLz\Encoding tools\CRF-OnePass-GUI>x264.exe --crf 28 --ref 3 --mixed-refs --bframes 3 --weightb -
-subme 6 --trellis 1 --analyse all --8x8dct --me dia --thread-input --progress --no-psnr --output "9
9" "D:\Vidz\HackRoots\hack01.avs.copy.avs_CRF28"
x264 [error]: could not open input file 'D:\Vidz\HackRoots\hack01.avs.copy.avs_CRF28'

D:\DLz\Encoding tools\CRF-OnePass-GUI>pause
Appuyez sur une touche pour continuer...
Dunno what the "_CRF28" is doing there but it seems to have invited itself anyway ^^

And the quotes aren't enough to protect from spaces in the path to the AVS file (I just tried and it won't work, but that's a minor problem once you know about it).

I also tried the GUI on another source, using the 18-22 range, and this time it computed the crf correctly :)

toofas

12th February 2007, 02:49

Hello,

Well, after years of lurking, I have finally signed up to make a post. You can attribute that to the vast knowledge and expertise displayed by all the active members here. I have never registered as I never had anything to contribute beyond what is found here (I.E. I am a student, not a teacher in this field) and, with a lot of research, I have been able to find an answer to every question I've had. In fact I registered five days ago to ask one, but was unable to post... yet I have been able to piece together the answer of my the question I originally intended to ask. I suppose that is the purpose behind the waiting period. :thanks:

Anyway, on to the point of this one...

This is a pretty cool tool. Thanks for putting forth the effort to make it. I have a couple of questions in reference to the results it gives. I set the tool up for crf 18, 20, and a bitrate of 2048. The result for this clip was to encode at crf 13.36. I ran several other tests with similar results. My questions are this:

01 Given the nature of crf, is this saying that if I were to 2pass encode at 2048 it would be overkill?

From what I understand, 18 is pretty much (for lack of a better description) as 'good' as the quality will get without just going lossless. In other words, using a lower crf number will not result in a noticable visual difference vs. 18.

02 I intend to run a couple of tests to compare, but theoretically, would a crf encode at 13.36 render enough of a visual difference vs. 18 to notice?

I used to just encode with XviD at 2000 and take the filsize and quality it gave me, but now I am really getting into seeing what x264 can do here and would like to try to maximize quality while minimizing filesize (who isn't, eh? ;) ) for storage on a media server.

TIA!

toofas

Shinigami-Sama

12th February 2007, 03:17

oh right, I'm still having troubles with my PC, mostly the shell desided it wanted to corrupt after a power loss, so I'm still in the process of hunting down some drivers for it before I start testing again

check

12th February 2007, 04:36

01 Given the nature of crf, is this saying that if I were to 2pass encode at 2048 it would be overkill?
From what I understand, 18 is pretty much (for lack of a better description) as 'good' as the quality will get without just going lossless. In other words, using a lower crf number will not result in a noticable visual difference vs. 18.
02 I intend to run a couple of tests to compare, but theoretically, would a crf encode at 13.36 render enough of a visual difference vs. 18 to notice?
Depends on your eyes, use them to decide.

toofas

16th February 2007, 19:06

Depends on your eyes, use them to decide.

Right, perceptually this is what counts, but I'm asking more about the technical/theoretical aspects. In other words, what it is supposed to do mathematically.

Reason being is this: I ran the CRF OnePass GUI on several different sources with CRF values of 18/20 and two different bitrates of 1024 and 2048.

Look at two in particular:
Source A: Bitrate of 2048 said to run at CRF 13.36 and at 1024 to run at 17.26.
Final encode at 17.26 looked very, very good with an average final bitrate in the high 900's... so it came close to the 1024 estimate.

Source B: Bitrate of 2048 said to run at CRF 20.34 and at 1024 to run at 25.24.
Final encode at 25.24 looked awful with an average final bitrate in the mid 600's... not very close to the target of 1024.

In fact, it takes encoding Source B at CRF 18 to come close to looking decent and that creates a final bitrate at around 2000 when the GUI estimated 20ish.

Given the drastic difference, I am interested in the technical aspects to know what is causing the differences. I don't know if I've explained this clearly enough, so if there are specifics you need to help answer this, please let me know and I will post them.

Just FYI, Source A was The DaVinci Code and Source B was House of Flying Daggers. Both are NTSC Anamorphic DVDs (16:9) with the PAR(?) being 2.40:1 and 2.35:1, respectively. The AviSynth scripts are nearly identical (only the cropping values are different) and look like this (psuedo code as I don't have them with me):

DGDecode_mpeg2source(whatever.d2v,info=3)
ColorMatrix(hints)
Crop(only top and bottom to remove black bars)
Lanzcos4Resize (to 720 x 352)
Undot()

there are also two lines MeGUI adds for the aspect ratio for x264.

TIA!

DarkZell666

16th February 2007, 20:46

The method used in tyee's CRF gui isn't as accurate as the one experimented by DDog a bit earlier. Try DDog's (very manual) method and compare the results, you might be luckier :)

tyee's GUI tries to solve the crf value with sophisticated maths (which was a very nice try indeed imho), and DDog used a trial-and-error iterative method (which does it's job fairly well if you're patient enough).

toofas

17th February 2007, 04:10

tyee, would you be willing to share the source? I'm no genius, but I'd like to take a look if you don't mind. If not, that's cool.

DDogg

17th February 2007, 16:28

toofas, one of the things I have noticed is how hard it is for the casual encoder to understand a very simple fact. That is, bitrate is simply the envelope that carries or holds the source complexity.

By and for itself, bitrate has nothing to do with the quality of the encode. To put that a slightly different way, every source has a unique fingerprint that dictates what bitrate will be required to contain the complexity of the source. If the bitrate allocated is not high enough, the source complexity will 'overflow' the container and the encode will not be acceptable. (Stretching a bit to make that word picture/metaphor)

Luckily, it is a non issue when the encoder has a constant quality mode that we can use to do a small sample. The actual filesize of a sample can be input into a formula that will give us the derived average bitrate. That DABR IS the approximate bitrate that will be required to achieve an approximate quality to the sample CRF that was used.

You might want to take your two sources mentioned above and run a 2% sample at crf 18 on each. Derive the bitrate of the sample. I think you will find one will take a much higher bitrate to achieve the crf 18 equivalent.

This will let you know in advance of the full bitrate based encode if the bitrate allocated will contain the source complexity. If it will not, you can add certain remedies such as light filtering to the script and rerun the sample until the DABR is inline with your size requirement.

This assumes you do have a size requirement. If not, just encode all your stuff for the server at crf 18 and don't look back. It will use the bitrate it needs to achieve the specified approximate quality.

I don't know if this even starts to answer any of your questions. If not, ask back.

tyee

19th February 2007, 04:23

Hi toofas

Well I'm not a real coder! There is no real source code that I personally have. All I have is my script written in this media authoring program! I think it's delphi based. I probably should have used a real coding language but I at this point in time I don't know any. I did download the latest C# recently but haven't had time to learn it yet. For this app I used, now don't laugh, c'mon, now....Neobook 5! Google it to find out more.

tyee

toofas

21st February 2007, 06:05

DDogg,

Thanks bro, that actually does help clarify it a lot. I feel like I was on the right track, but that helped pinpoint it for me... only leaving me with one question really.

Given that crf 18 is somewhat the de facto standard, how do the other figures compare with regards to the source? In other words, would crf 0 be the equivalent of a Lossless encode and crf 18 be the threshold of where most people 'won't notice,' (generalized, of course ;-)?

toofas

toofas

21st February 2007, 06:12

Oh man, tyee... Neobook 5.

It is crazy how 'dummy proof' some of these programs can try to make something for the everyday person.

I think I got to the point where everyone else did and why the thread just stopped... moved on, so to speak. It was a hell of an interesting read and test period. I enjoyed it and learned a lot.

I think what through me off was using The DaVinci Code as one of my test sources. I just couldn't believe that it didn't need much bitrate to encode when compared to others. I guess that is what a slow, generally dark film will give you.

I went back and re-read where you had posted the formula you were using and was able to derive the information I was looking for, so that's cool. Thanks for putting the time into it.

toofas