HC encoder

[JasonFly]

Thank you for your response Boulder but that didn't helped. Same message. I 've found some method in the avisynth forum. It uses batch and Vdub frameserving to append all the files and to use AviSource only once, but it's complicated and I don't know if I can adpt this method to my capture files which haven't exactly the same framerate.

Other ideas are welcome

[jdobbs]

Have you tried SegmentedAVISource? Your files look like they were meant for it...

Maybe like this:

Clip=SegmentedAVISource(("I:\1993\1993.avi").ChangeFPS(25000, 1000)

[JasonFly]

Yes, I have tried but I don't know why, that only load the first clip. Maybe because of the fps which isn't quite the same.(In capture, fps isn't always perfectly 25)

Besides, SegmentedAVI uses UnalignedSplice whereas I need Alignedsplice(for audio synchronisation)

[Boulder]

You could open the script in VDub and save as AVI using HuffYUV for example (ffdshow has a YV12-capable HuffYUV) and then loading that clip in HC with another script.

[JasonFly]

I think I'll do that for these files. Thanks

Besides, I didn't remember AVI filesize wasn't limited in NTFS, so I'll capture to one big file for the next capture I'll do.

Sorry for this discussion which wasn't really related to HCEenc.

[boombastic]

I use DVD-RB PRO to backup my DVDs using HC as the encoder;i've got a question:suppose i do an OPV encode and get a final size of 4,37 GB(in a real case matbe the final size is not so exact...) and let's say i do a 2-pass encode with the same final size:is it always true that the 2pass encode results in a superior quality?Or does it depend on the bitrate range used or on some other factor?Apart from using every singòe byte of the dvd,is always better in terms of quality to use a 2pass encoding?

[Boulder]

A 2-pass VBR may or may not look better than OPV. Depends on the situation.. I'd say usually the difference cannot be seen if the average bitrates are close enough to each other. That is, if we are talking about constant quality type OPV.

I use CQ mode (@q2 and a hi-bitrate matrix) with max bitrate check in HC if I don't have to worry about the filesize.

[boombastic]

I do care about filesize because the movie must fit into a dvd,but with the prediction function of DVD-RB,this is not a problem,so when i want backup a dvd with a final bitrate around 3000 Kbps i need to choose between a 1pass opv and a 2 pass encoding and if you consider that i always use the best profile in HC it means that choosing a 2pass means 4 hours more in the encoding.That's why i'm asking that.

[Boulder]

I'd say the only way to know what your eyes think is to try.

I've often usen CCE's OPV and been happy with the results. Then again, I've been happy with a 2-pass encode by CCE as well;) You'll have to remember that OPV is never dead accurate so you might end up with serious over/undersizing which then requires a re-encode. The same goes for HC's CQ mode.

[dragongodz]

please note that HC's CQ is constant quant and should not really be called OPV. it is not OPV in the sense of variable bitrate with variable quants. the only time quants will vary is with the check vbv/max bitrate constraint.

CQ mode (@q2 and a hi-bitrate matrix)
of course at a very low quant such as that it would be hard to tell any difference. however if you are encoding something much larger where the average quant was more like 10(just as an example) then it would be much easier to see where varying the quants, such as the ability to use lower quants, has clear advantages.

You'll have to remember that OPV is never dead accurate
real One Pass VBR can actually be accurate. the RC does need to keep it fairly well under control of course which some do not. the problem is so many think of Constant Quant as being One Pass VBR because of course the bitrate itself does vary with CQ because of the variance of the footage. however it is better to keep clear that these are not the same thing. real OPV you can set a target bitrate for while CQ you can not.

[Boulder]

Yep, actually the term OPV points to CCE.. although lately it seems they have changed it to 1-pass VBR ;)

Since HC doesn't offer CBR, I've used CQ mode for all encodes that are not size dependant. One option would be to use the same value for max and avg bitrate, but I thought I'd give the standalone some breathing space by using CQ with a very low quant.

[Boulder]

I just noticed that with Avisynth 2.5.7alpha, you need to add the line SetPlanarLegacyAlignment(true) to the avs script.

[IanB]

@Hank315,

I thought you were going to fix the pitch assumptions, slipped thru the crack? ;) PM me if you need any help or hints. In 2.6 the pitch and alignment of each plane will be independant, possibly even in different buffers. Always use the GetPitch method.

Regards
IanB

[dragongodz]

hank315 has been busy with work among other things. however he has been working on things(you will have to wait to find out so dont ask). ;)

he is aware of the reported problem however.

amazing how i know all these things isnt it. :D

[IanB]

@dragongodz,

Has Hank released the source for HC to anyone?

[dragongodz]

not that i am aware of. though that doesnt mean he hasnt of course, just that i do not know. :)

i for example have only seen small sections/snippets of code when hank315 has asked my opinion on how to fix certain problems etc.

[techmule]

please note that HC's CQ is constant quant and should not really be called OPV. it is not OPV in the sense of variable bitrate with variable quants. the only time quants will vary is with the check vbv/max bitrate constraint.


of course at a very low quant such as that it would be hard to tell any difference. however if you are encoding something much larger where the average quant was more like 10(just as an example) then it would be much easier to see where varying the quants, such as the ability to use lower quants, has clear advantages.


real One Pass VBR can actually be accurate. the RC does need to keep it fairly well under control of course which some do not. the problem is so many think of Constant Quant as being One Pass VBR because of course the bitrate itself does vary with CQ because of the variance of the footage. however it is better to keep clear that these are not the same thing. real OPV you can set a target bitrate for while CQ you can not.

How to get CQ work to your expected final file size, is still a big questionmark. I used GetCQ tool also and the size is always way out. I just tried it again yesterday and it came out to be 4.87GB. If its few MB less, its acceptable, but over is a waste of time.

Unless someone can give some good recommendations on how to get the CQ feature of HC working to your expectations, I would stay clear of it.

Why was this feature developed in first place itself ???, if it cannot be tamed.

[Boulder]

The prediction can never be dead accurate as the sample may or may not represent the whole clip well. That is the problem with CQ (be it constant quant or quality) based methods. The only advice I can give is to increase the sample size.

[techmule]

The prediction can never be dead accurate as the sample may or may not represent the whole clip well. That is the problem with CQ (be it constant quant or quality) based methods. The only advice I can give is to increase the sample size.


I understand that and i am okay with few MB+/-, as long as they fit the DVD :)

How much sample % in your opinion should be good enough and reliable ???

[Boulder]

I've used 1-5% myself but that cannot ensure an accurate result.

[Inc]

The only advice I can give is to increase the sample size.Or use that ping pong technique using offsets on every turn so you end up in the same prediction time but with more samples in a whole used.

[techmule]

Or use that ping pong technique using offsets on every turn so you end up in the same prediction time but with more samples in a whole used.

I guess, GetCQ uses that script only for its predicted Q value.

[dragongodz]

Why was this feature developed in first place itself ???, if it cannot be tamed.
because it was basically meant to be a more or less consistant/quick 1 pass encode where file size, max bitrate etc were not really important. no it wasnt really designed for constrained formats such as svcd/dvd etc or when you are actually going for target size/bitrate accuracy. this doesnt mean you can not use it for such just that you should be aware of its limitations when doing so.

[DarkNite]

Just wanted to chime in and say thank you for the excellent work. I've finally gotten around to testing this release out, and was quite pleased with the overall results in comparison to the versions I have used previously.

Now if I just had more boxes with SSE2 support... :)

[hank315]

I thought you were going to fix the pitch assumptions, slipped thru the crack? It just didn't made it into HC017 but it will be in the next release.

Has Hank released the source for HC to anyone?No, but dragongodz and I mail a lot to discuss all kind of MPEG related stuff.

[techmule]

because it was basically meant to be a more or less consistant/quick 1 pass encode where file size, max bitrate etc were not really important. no it wasnt really designed for constrained formats such as svcd/dvd etc or when you are actually going for target size/bitrate accuracy.


What is the applicability of this feature then, in real life situations, can you quote some examples. Just trying to understand for what can I use CQ.


this doesnt mean you can not use it for such just that you should be aware of its limitations when doing so.

Yes, I am aware of the limitations, but was hoping that they might either go away or diminish with newer versions of HC.

Although its a great product, no doubt.

[dragongodz]

dragongodz and I mail a lot to discuss all kind of MPEG related stuff.
and some facinating discussions have been had. ;)

What is the applicability of this feature then, in real life situations, can you quote some examples.
fast archival purposes would be a good example. also there are business related possabilities, such as licensed footage being transfered to be re-encoded by the licensee, in such a case they would want high quality while not ridiculously big as a raw copy would be. so a constant quant encode at a low quant could provide such and be relativly fast to do.

was hoping that they might either go away or diminish with newer versions of HC.
the unpreictable file size is inherent in the method and not the program. there is no way to remove the limitations and still have it be constant quant. hank315 already made 1 change, possible to limit max bitrate/vbv, which already has changed it to no longer being fully constant. to set it to hit a specific target size ot bitrate would require much more and turn it in to more of a 1 pass vbr mode. if you start going that route then its best to do it fully and not just mess with constant quant to try and get it.

[LigH]

use that ping pong technique
You probably mean the Bisection method (http://en.wikipedia.org/wiki/Bisection_method) from the numerical analysis?

[Darksoul71]

Hi all,

I´m currently "developing" a size prediction frontend for HC Enc based on the same approach that the
most size-prediction tools for CCE or TMPEGEnc use. I do a "Q-based" encoding for a 2% sample
of the movie using AVISynth. Based on the estimated size for the encoded sample, I vary the Q-value
unless the estimated size is within a +/- 10% of the target size.

The Q-value for the next pass can be calculated on various algorithms:
* The simplest but not the fastest algorithm is bi-section
* Another approach is described by Amnon82
* A simplified version which can be used instead of Newton-Raphson (-> D2SRoBa) is this formula:
Q(n+1) = Q(n) * (Estimated_Size / Target_Size)

My manual tries (calculating the new Q-Value by using a pocket calculator) unfortunately confirmed what Hank already pointed out in his last PM: HC Enc has the tendency to be not very predictable for analogue captures. Pre-quantized material (e.g. DVB-T broadcast, Hardware MPEG2 encoded material, DVD) seems to be less vulnerable and provide a better prediction precision but my MJPEG captures (480x576 filtered and resized to 352x576 via AVISynth) were up to 40% oversized as well as 20% undersized.

To quote Hank:
A while ago I did an implementation for OPV, it did pretty well but it only worked OK for DVD backup's, for plain AVI input it was just bad.

Based on this and my personal
I would like to comment a few things dragongodz posted above:

the unpreictable file size is inherent in the method and not the program.

I would disagree on this one:
The final file size might not be exactly predictable there is a valid approach (using a 1-5% sample of a movie) to get a pretty good aproximation (mostly +/- 5% for CCE which I use currently). D2SRoBa (in combination with DVD2SVCD) has proven that the concept works very well at least for CCE. The variety of tools out there for TMPEGEnc show that the concept works also for other encoders. So IMHO the unpredictable size (for analogue captures) in contrary is inherent to the program (= HC Enc).

As my time is currently quite limited I can not do further experiments but there might be simple aproaches to "stabilize" the prediction for analogue material, e.g. limiting the upper CQ bitrate to 6 MBit, using good denoisers, etc.

All this worked quite fine during my low-bitrate encoding experiments with CCE.

Unfortunately I don´t know enough about how MPEG2 encoder(s) work internally. Thus I can and will not comment weather "CQ" encoding or 1 pass VBR mode would do the job better. All I can say is that two other MPEG2 encoders have "VBR"-modes (be it either named Q-Value or Quality) and both are able to hit the targeted size even with analogue material (-> lot´s of noise) whereas one encoder fails.

For me a "OPV"-frontend for HC Enc is a "no no" as long as the encoder has this "problem" with analogue material.
From a time perspective I guess even an implementation based on bi-section with something like 10 estimation runs max mean 90% time saving compared to a two pass mode. Judge yourself wether this is worth the implementation effort.

Just my two cents,
D$

[dragongodz]

I would disagree on this one:
hmm thats interesting considering your own words say a completely different thing. to quote
The final file size might not be exactly predictable
to get a pretty good aproximation (mostly +/- 5% for CCE which I use currently)
hmm looks to me like you are agreeing you can not predict the exact size.

an important point is
All this worked quite fine during my low-bitrate encoding experiments with CCE.
its nice that this has all been expeirimented on with CCE for so long and the best methods for getting close with CCE have been found. however to criticise another encoder because it doesnt work the same way as CCE is ridiculous.

also about this quote
A while ago I did an implementation for OPV, it did pretty well but it only worked OK for DVD backup's, for plain AVI input it was just bad.
sorry but you have no idea what hank315 was talking about with that. i do because i saw and tested it. it was a near real 1 pass VBR(near because of methodology to determine other facters etc i wont go in to here) and didnt need any size prediction by anything else, you set a target average bitrate. the problems hank315 was talking about are not related to CQ encoding and shouldnt try and be used as any proof of what you consider a problem with HC, because it isnt.

[Darksoul71]

No need to nitpick my posting to death !

I am not starting a flame war here (e.g. my encoder is "bigger than yours").

Everything posted by me is based on the observations I made.

Saying "not exactly predictable" versus your "the unpreictable file size", sounds quite different to me.

Getting a 95% - 105% of my final target size is pretty good to me if it only takes half the time and gives
me better quality compared to CBR encoding. Not matter which encoder is used.


hmm looks to me like you are agreeing you can not predict the exact size.

Good estimation <> exact size (but still "good enough")


its nice that this has all been expeirimented on with CCE for so long and the best methods for getting close with CCE have been found. however to criticise another encoder because it doesnt work the same way as CCE is ridiculous.

Claiming I am criticising HC is indeed ridiculous. The two examples I named (better filtering & limiting the upper bitrate) are not bound to any specific encoder:
1) Filtered / denoised / smoothed material puts less stress to any encoder (be it DivX, HC Enc, CCE or whatever) which results in "less risk to get macroblocks during complex scenes". The same goes for "pre-compressed" material which provides less details for the encoder to compress (-> DVD backup, DVB broadcast).

2) Limiting the upper bitrate limit (e.g. to 6 MBit) leaves the encoder less space to "waste" bitrate. No matter what encoder you use.

All in all it where just examples and I would be happy to proof them right by doing further testing (for which I don´t have time). As pointed out I don´t know the difference between a constant quantizer (HC Enc ?), a Q-Value (CCE) and quality-based encoding (TMPEG Enc) method is. All I know is the following:
Varying the corresponding Value "X" (be it named Q, Quality, whatever) as described above leads to successful results when using TMPEG Enc (try TOK or CQ Tester by Borad) and CCE (D2SRoBa / my implementation with Komprezz2MPEG). When I say successful results this translates to "the encoder was runing using value X and the final filesize was within the estimated range". It does not mean encoder "a" is better compared to encoder "b".

May be I should have written it that way:
TMPEG Enc and CCE have a less fluctuating bitrate graph when using "Q-based" encoding compared to HC Enc...

Better ?


sorry but you have no idea what hank315 was talking about with that. i do because i saw and tested it. it was a near real 1 pass VBR(near because of methodology to determine other facters etc i wont go in to here) and didnt need any size prediction by anything else, you set a target average bitrate. the problems hank315 was talking about are not related to CQ encoding and shouldnt try and be used as any proof of what you consider a problem with HC, because it isnt.

This posting was directly an answer of hank315 to me when I asked him why he didn´t implement OPV encoding in first place. If hank was talking about some sort of VBR encoding where you specify the ABR, well then sorry....but to me OPV refers to the approach used by D2SRoBa and all the other tools out there.

Nevertheless it covers the same experiences I had playing around with HC Enc and "OPV".....

Later,
D$

[dragongodz]

No need to nitpick my posting to death !
hmmm
Saying "not exactly predictable" versus your "the unpreictable file size", sounds quite different to me.
funny that sounds exactly like nitpicking to me as both statements mean basically the same thing in the end.
you cant expect to say you disagree with me and then not have me say where you contradict yourself.

Claiming I am criticising HC is indeed ridiculous.
your words
So IMHO the unpredictable size (for analogue captures) in contrary is inherent to the program (= HC Enc).
and you say this because you dont get the same results as you do using CCE or Tmpgenc, neither of which do CQ but other methods.

I asked him why he didn´t implement OPV encoding in first place. If hank was talking about some sort of VBR encoding where you specify the ABR, well then sorry....but to me OPV refers to the approach used by D2SRoBa and all the other tools out there.
go back to page 31 of this thread and read boulders and mine posts. real One Pass Variable is not CQ or some sort of quality setting(though this is closer) etc. it is what it says it is. the fact that CCE uses the term OPV has done more damage in some ways i think for people understanding what these terms really are meant to mean. CCE appears to do a type of OPV in that it does vary the quants etc but for whatever reason they chose not to have targeting average bitrate. so people seem to think that OPV means you MUST use these predictions to estimate the size because there is no other way. thats simply false.

I am not starting a flame war here (e.g. my encoder is "bigger than yours").
hmm did i say you were ? did i call you names ? did i say you were full of it ? no. i disagreed with what you said and explained why. so please dont get upset because you dont like the blunt way i say things.
oh and of course HC isnt "my encoder" at all. incase you missed it i actually work on a different encoder. :)

[techmule]

@darksoul,

I guess you should discuss this in detail with Hank or better wait for his comments on your post above, since he is the developer of HCEnc and can guide you better on your concerns.

After that its your decision (since its your time) whether to develop this tool for CQ value prediction or not.

IMO, it does save a considerable amount of time using (CQ vs 2-pass) and would help the the users of HCenc, moreso since apart from GetCQ, I dont think many tools are present to do the CQ prediction for HCenc.

And if you can come out with a better and more accurate prediction tool, it would be good effort spent.

Just my 10 cents.

[Darksoul71]

@techmule:
Agreed, I´m just a bit disappointed that my "manual" experiments with analogue material turned out to be a bit unprecise in terms of prediction.

@dragongodz:
Nothing more to add....

[techmule]

@techmule:
Agreed, I´m just a bit disappointed that my "manual" experiments with analogue material turned out to be a bit unprecise in terms of prediction.


But on the brighter side of it, atleast it gave you some perspective about the other sources (DVD etc). Without your "manual" experiments, it couldnt have been possible.

Even if you decide not to pursue this project, it will be a lead for others.

Well done !!! :)

[Darksoul71]

@techmule:
Well, I don´t know how well reproducible my experiments are. My analogue cable signal is quite stable and my Cinergy 400 captures nice quality. Someone else with a BT 8X8 based card and noisy signal might get different results ?!?

[hank315]

A while ago I also tried to implement OPV based on Constant Quantization, well not completely constant, it could vary a bit.
This sounds a lot like the method used by Darksoul and I couldn't get it to work properly either...
With properly I mean the prediction size was sometimes *very* wrong even if 5 - 10 % samples were taken.

Method used:
- start with first estimate: Q=8
- second estimate: Q(n+1) = Q(n) * (Estimated_Size / Target_Size)
- next estimate(s): secant or Ridders' method, converges faster than bisection if close to the root

Normally this converges in 3-4 steps.

The final file size might not be exactly predictable there is a valid approach (using a 1-5% sample of a movie) to get a pretty good aproximation (mostly +/- 5% for CCE which I use currently). D2SRoBa (in combination with DVD2SVCD) has proven that the concept works very well at least for CCE. The variety of tools out there for TMPEGEnc show that the concept works also for other encoders. So IMHO the unpredictable size (for analogue captures) in contrary is inherent to the program (= HC Enc).
I just can't see why HCenc does something really different than CCE or TMPEG.
If I look at bitrate curves and Q-curves in BRV produced by CCE or HCenc it shows they are almost identical.
CCE bitrate curves tend to be a bit more flat than HCenc curves.

Because it seems you need 10% or more sample size with 3-4 steps to converge the time gain was just too small so I skipped the whole thing (for now).

As pointed out I don´t know the difference between a constant quantizer (HC Enc ?), a Q-Value (CCE) and quality-based encoding (TMPEG Enc) method is.
I don't know the difference either :)
A Constant Quantizer as used by HCenc is decribed in the ISO document, Q-value and quality-based are "invented" by CCE and TMPEG, I think there's nobody who really knows how it works.

[Amnon82]

My self I didn't tried your latest version. I think, I should play a little with Darksouls simplified formula. Also I'll add it to AutoQ2 to see if it work with CCE.

[Darksoul71]

Hi hank,

thanks for taking the time to reply. Hopefully my postings didn´t sound like an offence against HC Enc.

Your observations are indeed interesting. Since my time to experiment was quite limited I didn´t play
around too much with sample size and I guess 5% were my maximum. For CCE even 0.5-2% are "good enough".

The simplified formula you´ve also used to calculate the second Q-value normally converges after 3-4 steps for CCE.
At least that´s my observation using CCE 2.5 SP for my analogue captures.

Here is a little more background on my experiments....

My source material was:
* DVB-T broadcast (-> MPEG2 already compressed with a very low bitrate)
* PIC MJPEG AVI captured in PAL SVCD Res from analogue cable using my Terratec Cinergy 400
* MPEG2 files captured in PAL D1 Res from analogue cable using my Haupauge WinTV PVR USB2

I´ve choosen not to use DVD material since it was already pointed out that HCenc is pretty much able to hit

The results were, errr, "mixed":
DVB-T broadcasts had the tendency to converge towards Q=1 since they were already highly compressed (~ 2.5 MBit/s mostly). For those with Q-Values above 1 the resulting files were mostly a bit undersized. The MPEG2 captures had mostly ~ 9MBit/s and the resulting encodings met the target size very well (~ 5-10%).
The MJPEG AVI´s showed the biggest deviations in terms of size. I encoded 8 movies and 5 of them met the target size quite ok ( max 15%) whereas 3 of them did go wild (up to 40% oversize and 60% undersize).

My conclusion:
I wasn´t suprised to see the biggest deviation on the source material with the highest details and I wouldn´t be suprised if HCenc shows even higher variantions when using Huffyuv captures or DV material captured with my Canopus ADVC100 as it provides even more details (also noise) compared to my Cinergy 400.

I just can't see why HCenc does something really different than CCE or TMPEG.
If I look at bitrate curves and Q-curves in BRV produced by CCE or HCenc it shows they are almost identical.
CCE bitrate curves tend to be a bit more flat than HCenc curves.
Hmmmm, this is really strange. May be the rate control (is this the right word ?) of CCE is a bit slower / less dynamic compared to HCenc ?
May be you could play a little bit with the rate control and add a little more "damping" to it (yes, I know. That´s the wrong term :D)

I can only guess.....

At least HCenc behaves different compared to CCE and TMPEG Enc since one is able to get a valid size estimation by using 2-5% of the movie, wouldn´t you agree ?

I have never seen deviations higher than 10% for the last 80 movies I encoded from my analogue captures over the last month. Mostly I don´t care about a few percent oversize since I normally do not fill the DVD to the max and are not afraid to run the finally mastered DVD (normally with 2 movies) through DVD Shrink or DVD-RB if it is oversized.

Because it seems you need 10% or more sample size with 3-4 steps to converge the time gain was just too small so I skipped the whole thing (for now).
Hm, this depends on. Even if it would take a 10% sample with 5 steps to converge you would still have a speed gain of 25% compared to 2 pass encoding.
At least I would be happy to get such a speed gain. Example: You would be able to encode 5 movies in "OPV" mode in the same time as 4 movies would take


A Constant Quantizer as used by HCenc is decribed in the ISO document, Q-value and quality-based are "invented" by CCE and TMPEG, I think there's nobody who really knows how it works.
So they are mainly marketing gags I guess ;)

@Amnon:
My self I didn't tried your latest version. I think, I should play a little with Darksouls simplified formula. Also I'll add it to AutoQ2 to see if it work with CCE
The formula is not "mine". I can´t remember exactly who provided it but IIRC it was tylo somewhere over in the DVD-RB forum when they talked about OPV support in DVD-RB. You´ll also find a modified version for CCE of this using a multiplicator ("f") with a start value of 1.35 and re-calculate f during the aproximation process. I used this formula also for FitDVD (I gave you the sources, didn´t I ?). Nevertheless the "f" factor stuff is pretty much CCE specific.

The prediction algorithm itself has no impact on the size problem. It just converges faster or slower.

I think I´ll concentrate on my "multi CPU" frontend for HCenc and come back to size prediction once I have new ideas...

Later,
D$

[Amnon82]

Everybody knows DVDShrink. When you open a VOB-file it anaylses it first. Why not adding such a feature to HCEnc. Now we know the compression rates of the source and we can tweak the CQ-value a little bit. I'm thinking of a factor. Where you can compress more you higher the CQ-Value, otherwise you lower it using the factor.

Exp.: Our CQ-Value is 8, our factor is 1.08, so in more compressable scenes we are using 8.64 and in less compressable scenes 7.40

I think this might be the "magic" behind CCE and TMPGEnc.

[Darksoul71]

Hm, the compression rates / complexity of our source is something we do not know. In order to find them out, the encoder has to analyze the complete video stream (which is done in the first pass of two-pass mode). Depending on the filters used on source (De-Interlacer, Denoise, Resize, etc) the video decoding & filtering will mostly be below real-time. This leads to two options:
1) Doing a complexity analyse based on destributed short samples (e.g. SelectRangeEvery)
or
2) Doing a 100% analyse of the movie (a.k.a. 2 pass mode :D)

Using samples for a movie is more or less representative depending on sample size and the variation of the "action level" / complexity of the movie. For CCE and TMPEG normally a maximum of 5% is "enough". HC Enc might need 10 to 20%.

I can only guess but I wouldn´t be suprised if the size aproximation is proportional to the sample size, e.g. you hit the target size more precisely using a 20% sample of the movie instead of using only 2%. When using 100% sample (= full movie) you will of course get perfect size but running 8 * 100% of a movie to get the right CQ value is not really a time saving compare to 2-pass mode (2* 100 % encoding :D).

I think it would need intensive testing with a critical source (something that showed 40% oversize), the usage of a bitrate viewer and playing around a lot with filters, upper bitrate limit, etc. to nail down the hot spot which is actually causing the deviation. As posted above I think the "rate control" of HCenc might be a good starting point as well as comparing the bitrate curve for a critical source encoded via CCE with OPV and HCenc with CQ.

[Amnon82]

There are two differnce of the 100% analyse DVDShrink does. There is the fast one which only scans the bitrate or stuff like that and the DeepSearch analyse which can be taken as the first pass of an 2pass encode.

RateControl might be the problem, but adding a tweak factor for the CQ-value might also help using the fast analyse as base.

[Darksoul71]

Short note:
I´ve just thought about it. Rate Control might not be the right word. If HCenc encodes at a constant quantizer the encoder "doesn´t care about the bits spend" (I don´t know to explain this better with my own words). So basically one approach that might help to "stabilize" HC enc in terms of the oversized files could be a "bitrate control", e.g. HCenc monitors the bitrate delta over a certain frame range and limits the "amount of bits spend" to a fixed value (Delta Bitrate Max = 3000 kBit / 300 Frames) -> modifying the quantizer value as suggested by Amnon ?

But I guess this would be more of the approach a one pass VBR with fixed average bitrate would work.

BTW: If I haven´t said this before: Thank you hank for this great free encoder :)

[techmule]

There are two differnce of the 100% analyse DVDShrink does. There is the fast one which only scans the bitrate or stuff like that and the DeepSearch analyse which can be taken as the first pass of an 2pass encode.


Thats what I was thinking, how does Deep analysis gives Shrink all the info it need to compress and reach a target size ???

Deep analysis gives you your desired target size without fail, anytime and all the time!!!! and does not take more than 10-15 min for a normal 2 hrs DVD.(depends on PC conf also)

[techmule]

Everybody knows DVDShrink. When you open a VOB-file it anaylses it first. Why not adding such a feature to HCEnc. Now we know the compression rates of the source and we can tweak the CQ-value a little bit. I'm thinking of a factor. Where you can compress more you higher the CQ-Value, otherwise you lower it using the factor.

Exp.: Our CQ-Value is 8, our factor is 1.08, so in more compressable scenes we are using 8.64 and in less compressable scenes 7.40


Good thinking Amnon, and different values can be applied to the different segments of the video too, based on the analysis.

[dragongodz]

But I guess this would be more of the approach a one pass VBR with fixed average bitrate would work.
yes.

DeepSearch analyse which can be taken as the first pass of an 2pass encode.
sorry but no it can not. compressed domain transcoders do not work the same as full encoders. yes some of the information such as original curve etc could be obtained and used from a similar analysis and hank315 already looked at that. to say it could be taken or used as a real first pass though is wrong. there is information that has to come from the encoder encoding frames which isnt got from analysing the source in such a way. if you tried to just use information from such an analysis you would get suboptimal results because the encoder would be having to adjust itself for inaccuracies and information it doesnt get until the second pass all the time.

Deep analysis gives you your desired target size without fail, anytime and all the time!!!! and does not take more than 10-15 min for a normal 2 hrs DVD.
dvdshrinks deep analysis is actually just seeing which frames have more coefficients etc so it can remove more from frames that have them instead of having to play catchup with later frames that may be hit harder doing so. the reason its fast is because nothing is being decompressed, simply analysed.

[manolito]

Just a few thoughts:

According to the CCE manual CCE's so called OPV method really is a Constant Quant method with the exception that max bitrate and min bitrate will be observed. The quantizer is only changed when the actual bitrate goes over the specified max bitrate or under the specified min bitrate.

So CEE's OPV method should indeed be very similar to HC's CQ_Maxbitrate method (with the exception that with HC you cannot set the min bitrate).

From all the stuff I learned from D2SRoBa it is necessary to introduce a "correction factor" to the prediction. The sample passes will have tons of scene changes which will lead to heavy undersizing without correction. It also makes sense to turn off scene change detection for the analyzing passes. D2SRoBa calles this "estimation accuracy tweak". I remember that this idea originally came from forum member "jonny".

I also remember hot discussions in the D2SRoBa thread whether a larger sample size really leads to a more accurate prediction. Experience tells me that it does, but this was heavily disputed by r6d2 (and this guy knows a lot more about math than I ever will...).

Anyways, with a 2% sample size D2SRoBa always gets within +/- 3% of the target size. And if the encode is just a few percentage points oversized, I found that using a transcoder does not visibly hurt quality. It would be really nice if the same accuracy could be achieved for HC.


Cheers
manolito

[jdobbs]

There are definitely a few differences between CCE/OPV and true constant quant... if you analyze the output you'll see that the quantizer values for I, P, and B frames vary considerably from frame to frame. In constant quant they would have to always be the same. Hank315 has given the ability to adjust the quantizer values on P and B frame frames using a constant modifier, but in CCE there is some other decision mechanism at work for distribution of quantizer values in OPV... and I, for one, don't know what it is.

[Amnon82]

Maybe a mix of my thougths and ABR will do the magic for HCEnc. Maybe a fast analysis will help for DVD-Sources, but what is if the source is an uncompressed avi-source? What do you guys think the fast analysis should analyse?

[dragongodz]

Maybe a fast analysis will help for DVD-Sources, but what is if the source is an uncompressed avi-source? What do you guys think the fast analysis should analyse?
well i am not going to paste all the conversations hank315 and i have had by email but let me just say we have already discussed all this some time ago. so hank315 already has a good idea of a way to do these things.

i mean go ahead and discuss it if you want, maybe you will even come up with something we didnt. just dont be surprised if we are not amazed or something while you are going over old ground for us. ;)