HCEnc^n - ideas for dynamic Bitrate assignment for segments ?

[Darksoul71]

Hi all,

my question is basically in the topic of this thread. As some of you might be aware I´ve released a Multi-Core / Multi-CPU for QuEnc called QuEnc^n.

Since the concept is pretty easy I was planing to adapt this for HCEnc. Hence the topic "HCEnc^n"

Ligh has raised a concern that independently encoded segments could cause a drop in quality compare to the same movie encoded as one movie. This is mainly because the "n" segments could have very different demands in bitrate, e.g. you encode two segments and the first part of the movie shows no big action while you have a big showdown with a lot of explosions and gun fire in the end.

This scenario is not that uncommon :)

Since DVD-RB uses some distribution algorithm to assign different bitrates for each segment, I´m trying to figure out a quick working and simple to implement approach for "HCEnc^n".

My initial idea would be a CQ run with 1% of each segment.
Using CQ you´ll get different filesizes for each segment. The differences in filesize are (more or less) an indicator for the complexity of each segment. High complexity is indicated by a higher filesize which simply means that such a segment need a bit more "bits per seconds" to encode than a segment with smaller filesize which indicates less complexity.

Of course the dependency between filesize in CQ and the complexity might not be linear but to my understanding it would at least give you a rough indicator about complexity.

I´ve done a simple table in Excel to show what I´m talking about.
Target bitrate for the complete movie is 4 MBit


Segment# Size (MB) Deviation Bitrateadjust Factor Adjusted Bitrate
1 18,30 -10,93% 0,89 3562,91
2 24,70 20,22% 1,20 4808,96
3 20,15 -1,92% 0,98 3923,10
4 19,03 -7,37% 0,93 3705,04


Size is the value of the encoded 1% sample of the segment.
Deviation shows the size difference of the sample to the average size of all segments
Bitrateadjust factor and adjusted bitrate should be obvious.

Of course the distribution is not that simple. It must include max-bitrate checks plus a re-distribution of bitrate if a re-calculated bitrate exceeded the max bitrate.

What do you think ?

Best regards,
D$

[Revgen]

First of all, I think you should release a regular non-adaptive segment version first before you start working on the algorithim. An possible 82% improvement for HCEnc would still be great.

As far as algorithim goes, is there a way to get the code from JDobbs or is it a totally a closed source?

Also I believe ELDER uses Pearl script to assign bitrate to each segment. You could download ELDER and see if the Pearl script reveals any ideas.

Hopefully everything works out.:thanks:

[danpos]

@Darksoul71

Hi friend! :)

I think this is nice for a ZERO-ORDER approximation. Prodater64 (here and in others boards like vm.com ;) with same nick) did a tool called ASPA which it used something else similar to your approach.

I'm agreed with Revgen's idea: it would be good if the app to have a setting which the user could to choose between "non-adaptative bitrate" and "adaptive bitrate". In this way the user will choose which is more adequate to fit yours needs.

Just my poor R$ 0,02 ...

See ya,

[foxyshadis]

Since a lot of people run two passes anyway, it's moot, you just split the first pass on all the cpus, then parse the stats and divide the bitrate in some sane way (ripping the stats parser right out of the encoder is one way, hank might be willing to share).

For one-pass, Amnon is the experienced one, you can follow his formula (http://forum.doom9.org/showthread.php?t=108414) to do the compressibility check in a reasonable time, except ignoring the part about setting it to a constant quant. ;) I'm sure he might have other ideas too.

[Amnon82]

I was a long time out of the CQ-Encoding of HCEnc. D$ knows my formula already. He got it before I posted it here at doom9.

QuEnc's CQ-Pass is more stable atm. If you have thougts about quality losts try to get a solution for QuEnc^n first, or do your experinence parallel with both tools.

What I will do at your position is followed: Release an adapted version of your actual QuEnc^n for HCEnc and let the guys and girls of this forum test the basics, like how the processor react on this. Does the join of the parts work. Is the joined m2v playable on the dvd, etc.

When yes you can go on, when not, drop the project and keep going for QuEnc. You've all the time of the world for this. Take your time and HCEnc will be better than it is atm.

Thats only what I'm thinking about this.

Btw. I wrote Prodater64 a PM. Maybe he'll give me some sourcecode of ASPA.

[Darksoul71]

QuEnc's CQ-Pass is more stable atm. If you have thougts about quality losts try to get a solution for QuEnc^n first, or do your experinence parallel with both tools.
Err, QuEnc and CQ ?

Do you mean the QuEnc from Dragongodz or the modded version by PeterCheat ?

I didn´t find anything about CQ in the docs of QuEnc. I know that some of the libva based encoders out there, e.g. FreeEnc can do CQ but never knew that QuEnc can work with CQ.

Btw. I wrote Prodater64 a PM. Maybe he'll give me some sourcecode of ASPA. .
Sounds good !

:thanks:

[Amnon82]

I wrote you a PM with the core of ASPA and the old Slicer-Package by Inc.

For the QuEnc thing. In my OldXoolToolZ package (http://forum.doom9.org/showthread.php?t=110495) you may find the answer I found in the year 2004. It is a little old technic but a good start for you. If you need some sourcecodes of the old tools drop me a pm.

[danpos]

Err, QuEnc and CQ ?

Do you mean the QuEnc from Dragongodz or the modded version by PeterCheat ?

I didn´t find anything about CQ in the docs of QuEnc. I know that some of the libva based encoders out there, e.g. FreeEnc can do CQ but never knew that QuEnc can work with CQ.



Yep, it does. You just set the "average bitrate" into range [1-31] and so the CQ (Constant Quantization) will be engaged (i.e., if you set it up as "5" the CQ mode will be engaged with Q=5 ). As a matter of fact this isn't well documented but already answered by dragongodz here a couple of time (I myself already ask this to him once time ago :D)

Take care,

[Rumbah]

jdobbs has the advantage that he knows the original bitrates of the segments. I think he just scales down the original bitrate distribution of the segments in some way (perhaps even direct linear with the given compression factor) so that does not seem to help with your problem.

[Darksoul71]

Hi Rumbah,

thnx for posting ! Yes, indeed, jdobbs "knows" the bitrate. Either by analyzing the M2V itself or using the D2V info ? I dunno...

I think that I´ve already found a working approach for bitrate distribution over the segment.as described above. While it may not be wise to scale the target bitrate by the filesize deviation in direct linear dependency, I can always add a simple factor to lower the impact, e.g.

"Impactfactor" = 0.7
Filesize Deviation = -20% compared to average sample size.
Bitrate Correction Value would be normally = 1-0,2= 0.8
With the "Impactfactor" this would be 1-(0.2*0.7) = 1-0,14=0.86
Target bitrate = 4000
Corrected target bitrate would be 3440 with corrected and 3200 without corrected factor.

Do you understand what I talk about ? :D

All in all this is still in early stages and I´m shure it will need fine tuning :)

-D$

[Inc]

I just overflew this thread and I do hope I catched your approach well D$.

Use a Q based "sliced" prediction on the sum of your aligned sources/segments which will have to fit on the given Media or the wanted final size.
Do calculate from each result the used %age of the whole, lets say a CQ based encoding to match in here a DVD on all sources did result in 4.3Gb .... which almost fits.

Source1: 1.761,28 = 40%
Source2: 1.320,96 = 30%
Source3: 1.320.96 = 30%
----------------------------
_______ =4.403,2 = 100%

Now in 2pass do assign to source1 40% of the avg bitrate, Source2 30% and Source3 30%. Thus this lets result all Streams in the same Quality (imho).

--> Means: Same approach as your first posting.

I do recomend a ping-pong basd prediction for 1% as this kind of prediction lets you predict 2% in effective using the same time as if 1% would have been used. Ping Pong means. First run using offset of 0, second run using an offset between the slices, third again 0 and second again between the slices and so on ...
My function "slicer" does support that offset based prediction approach.

From an old posting of mine at Kvcd 2 years ago:

Here you can see how the Offset-way within my routine works:

http://mitglied.lycos.de/incredibleweb/offset.gif

[Darksoul71]

Hi Inc,

thnx for posting !
Yes, you have understood my approach perfectly right. The only thing I´m not really shure is wether the dependency between complexity / "Action Level" of a movie and the filesize in CQ is linear. Thus I´ve implemented an additional multiplier which lets the user adjust the impact of the "BitrateAdjustValue".

Also thanks for posting and explaining the "PingPong" approach. Now I´m beginning to understand the purpose of your slicer script.

What do you think is a good offset value ? 25% of your sample segment ?

If I understood everything correctly I could also use a simpler approach to do "Offset sampling".


Offset=250
Trim (OffSet, Frames)
SelectRangeEvery(2600,26)


Right ?
Of course the sample length should equal your GOP length if you do MPEG encoding.

When I do two CQ prediction runs with offset sampling and one results in a bigger filesize, I guess that
I choose the bigger one as estimated filesize, right ?

TIA,
D$

[foxyshadis]

SelectRangeEvery has a third parameter which is offset. :p

GOP length is fine for mpeg-2 but too high for mpeg-4, where for PC backup it's usually 250-400. I just stick to 100 max. Good idea though.

I've started testing x264 predictions, haven't gone as far as I want but preliminary results are good. Basically even at 5% sampling at 50 frames a sample, it delivers bitrate close to a 100% sample, but always high. The smaller the chunks, the higher it is, but I don't have enough information to try to find a formula to reduce it to the 100% rate, and it probably wouldn't apply to mpeg-2 codecs anyway.

I think I'll have to take a look at "ping-pong" more closely.

[Inc]

SelectRangeEvery has a third parameter which is offset.
Jup, Slicer uses SelectRange every except that you can set the %ange of the movielenght to be predicted directly.

When I do two CQ prediction runs with offset sampling and one results in a bigger filesize, I guess that
I choose the bigger one as estimated filesize, right ?
Nope, its the average between both. Do look at the graphs. Both ones (ping and pong) do have to be taken into account.
I gonna give a search on my postings at kvcd where I did explained the approach incl. Filesize graphs etc.

The pong should be in its offset set to the half of the range between pings as you can see in the graph.

Normaly you can even speed it up more by doing ...

1. Ping = size x
2. Pong = size y (here do keep in mind the diff. to the ping resulted size x)
3. Ping
4. Ping
5. Ping
6. Ping
7. Ping ... as the proportional diff from the second turn (the pong) is still in mind we can see if we matched the desired final sample size.
8. Pong ... safe check.
9 ... if all is ok .... do encode :)

means if you set 2% of the movie to be sampled within the prediction, in real its 4% but in the same time as if you use 2% in a common prediction way. So thats why its double precise.

By using this I got almost all my encodes exactly matched to the wanted final size when using TmpgEnc two years ago. CCE encodes also did result nicely matched.
Shure TmpgEncs CQ isnt constant but thats why you do the final check-pong.

[Darksoul71]

Hi Inc,

doooooh, I guess I need some coaching :confused:

1st, I would be happy if you have some more background info / discussion on PingPong prediction.

OK, lets first talk about my use of Ping Pong:
From what I understood so far I need to do the following things to calculate the scale factor for the bitrate for a segment like this:

Run a CQ pass with CQ=5 and "Ping". Lets say the size was 40 MB. Now run a CQ pass with CQ=5 and Pong. Lets say the pass was 10 MB. So the average size calculated is 25 MB and I would use this to correct the bitrate value, right ?

Note: In my usage I´m not talking about size prediction for CQ encoding. It´s more the other way round: Doing size evaluation based on CQ for each segment to determine the "Action Level" of each segment.

BTW: I already did such a bitrate adjusting algorithm for QuEnc^n (see the other thread).

The pong should be in its offset set to the half of the range between pings as you can see in the graph.
I´m more a man of examples :)
If "Ping" was a SelectRangeEvery(1200,12) from the beginning of the movie, I guess "Pong" should start somewhere in the middle of 1200-12 frames = 1188. So the offset for pong could be 594, right ?


1. Ping = size x
2. Pong = size y (here do keep in mind the diff. to the ping resulted size x)
3. Ping
4. Ping
5. Ping
6. Ping
7. Ping ... as the proportional diff from the second turn (the pong) is still in mind we can see if we matched the desired final sample size.
8. Pong ... safe check.
9 ... if all is ok .... do encode

Oh mate, if you are talking about OPV size prediction (as it seems) I don´t understand the approach is the offset of ping and pong adjusted ? How is CQ modified ?

I would be glad if you have the time to provide some background. I´m guess I see light and the end of the tunnel. Hope it´s not the train heading towards my position :cool:

BTW: Have you stopped using NuEnc ? It´s a cool "mod" of QuEnc but unfortunately with problems to keep maxbitrate. Very sad since size prediction works very well with it. At least that was the impression I got from my manual prediction tests.

TIA,
D$

[Inc]

Its kinda late already and I gotta wake up tomorrow very early so sorry for only one statement now ;)

I would be glad if you have the time to provide some background.You'll get it.
And yes the approach in the last 2 paosts we are talking about is a simple prediction which is NOT your purpose as you want to get the average complexitivity or Q consumption of all your segments. Thats beeing one via the approach you already mentioned and which was duplicated by my first post in this thread.

Have you stopped using NuEnc ? It´s a cool "mod" of QuEnc but unfortunately with problems to keep maxbitrate. Very sad since size prediction works very well with it.
NuEnc already is one? year old and is very behind the advantages which have been made to Qenc i.E. since then.
NuEnc gots one verrry nice behavior in its CQ Mode. On the one hand the CQ is linear and can be forced to floats! On the other side that "linear" behaviour does kill the real advantage on CQ individual behaviour based on the complexitivity of the actual frame/farmes. DO look at a CQ encode or a Q encode in TmpgEnc or CCE and that CQ curve isnt flat at all but "smart" ;)
And NuEnc maybe still gots that (maybe) non-DVD specs compilant output. (Pioneer SAPs?? Im not shure).
Unfortunately NuEnc in the state where it supports flat float CQ encoding doesnt come with ist sources (AFAIK) and PeterCheat hasnt been seen here around anymore (?).

[Darksoul71]

Hi Inc,

I didn´t have the time to do more tests on NuEnc and of course I can imagine that a lot of enhancements / changes took place to both QuEnc and AVCodec.dll since then. All I can say is that I did a few sample encodes (shorter clips) with manual prediction and this worked very well. All samples hit their predicted size very well. ;)

...but I´m also aware of the "too much max bit" problems that older versions of most AVCodec.dll based encoders have.

From my point of view I rather like to have an encoder with "Q" or "Quality" mode where the quantizer are scaled based on the complexity with the actual frame, if the encoder behaves more "predictable" -> read: it´s behaviour with size prediction is reproducable. CCE is very predictable and offers great quality. AutoMatQEnc is also quite predictable, also based on AVCodec.dll and offers very good quality (esp. when using QMatOp).

So no matter if HCEnc will be predictable for me in CQ mode, I still have interest in your Ping Pong Prediction method for AQE in CQ mode.

In regard to QuEnc I don´t want to be ungrateful. dragongodz has done a great job on QuEnc and I can perfectly understand that he don´t want to "mess around" with a finer granulated CQ mode. Mainly because this requires modification within AVCodec.dll as far as I understood.

Cheers,
D$

[Darksoul71]

@foxyshadis:

Sorry, I simply didn´t pay attention to your post:

I've started testing x264 predictions, haven't gone as far as I want but preliminary results are good. Basically even at 5% sampling at 50 frames a sample, it delivers bitrate close to a 100% sample, but always high. The smaller the chunks, the higher it is, but I don't have enough information to try to find a formula to reduce it to the 100% rate, and it probably wouldn't apply to mpeg-2 codecs anyway.

It´s interesting to hear that prediction also works for x264.
From what I´ve seen discussed here at doom9 in regard to size prediction sometimes an encoder requires a "correction factor", e.g. your formula works like this:
Q(n+1) = Q(n) *( (Estimated sample size) / (Targetsize))

a better working formula could look like this:
Q(n+1) = MyFactor * Q(n) *( (Estimated sample size) / (Targetsize))

For CCE there was once a factor of 1.35 mentioned which could be recalculated during prediction for faster convergence of the CQ algorithm. I´m no math guy...so don´t ask me :D

It should be possible to find out this factor by doing automated multiple encodes with altering values for sample size and video length. If you have a x264 encoder with CLI this can be pretty easy done with AutoIt or AHK.

[Inc]

From my point of view I rather like to have an encoder with "Q" or "Quality" mode where the quantizer are scaled based on the complexity with the actual frame, if the encoder behaves more "predictable" -> read: it´s behaviour with size prediction is reproducable.Thats paradox.
For CCE there was once a factor of 1.35 mentionedProblem here is that this value is a constant. And if I do have a look at the "curve" of the resulted filesizes during predictions such a value should result out of a function where the curve will be calculated. This will NOT guarentee a direct match but you could be verrrry close and so spending the prediction time gained by that faster enclosing into bigger prediction chunks = more accurate in the effective same time.

[Darksoul71]

@Inc: What is paradox ?
That I like rather like an encoder with "scaled" quantizer but predictable filesize over an encoder with unchanged quantizer based encoding and unpredictable / less predictable filesize ?

The factor 1.35 mentioned for CCE was just for a simplified version of the Q-calcutation discussed in the CCE forum a long time ago IIRC.

Of course it would be perfect to know th formula / function used to scale the quantizer based on the "Q"-Value for CCE if you like to do size prediction for CCE.