HC encoder

[hank315]

Hank, am I right the maximal GOP length in my sample equals 18 ? HC displayed 18, too, but I wasn't sure it was the correct value, since it's beta after all. If I'm not mistaken, HC beta makes GOPs one frame longer than specified.You're absolutely right, it was also the first thing I noticed, and I just fixed it.

Sorry for finding a different problem to bug you!No, thank you for reporting it :)

However, I need a statement from you. Could you, please, tell me, whether I should file a bug report to libavcodec developers or wait until you check HC further ?Uh, a statement from me?
I can only repeat what I saw using different decoders, some decoded it OK, some didn't.
The error can be in HC or in one of the decoders, I just don't know ATM.
Better just encode that part again with a higher quant to be sure.

[Marius-the-Mad]

Hah, OK. :) I'll try to make CQ 4 encode and see what happens. Thanks for the reply! :)

[hank315]

I just removed the HC016 test release from my server.
The issue Marius-the-Mad reported, it is a HC error :mad:
Will report back when it is solved.

[DK]

the latest016 TR version you sent me, works fne with dvd-rb and option *run encoder minimized* set

i must correct myself

after ~65% HC did hang again and again with the rebuilder options active *run encoder minimized*

amazing though that numerous segments worked fine

[jdobbs]

I don't think the hang is coming from the DVD-RB end... it just starts the process and then waits for the end status to be something other than STILL_ACTIVE.

[DK]

agreed; it is not resulting from rb

disabling the option *run encoder minimized*, however solves this "bug" for the time being

[2COOL]

@hank315

Seeing how this official thread used for all versions of HC, perhaps a annoucement post list is suggested? Your thread is up to 455 posts now. :eek: It's better for people who want to peruse the whole thread for posts on a certain version. ;)

Here are some examples of threads I've suggested this.

BeLight (http://forum.doom9.org/showthread.php?p=571527#post571527)
VobBlanker (http://forum.doom9.org/showthread.php?t=92481)
PgcEdit (http://forum.doom9.org/showthread.php?p=683483#post683483)

[Boulder]

I have a small feature request:

Would it be possible to add a prediction engine to the CQ mode? It would make using CQ much easier as one wouldn't need to encode the sample, check the log for the avg bitrate, adjust CQ, encode sample etc.

A simple workflow could be
1) take the desired avg bitrate from the ini file
2) encode a 1-3% sample clip
3) using a simple binary method, calculate the new CQ value
4) encode the sample with a new CQ until the avg bitrate is close enough to the target
5) encode the whole clip
6) if the final result was off the target (a user specified percentage), redo the prediction using the predicted and final avg bitrate for calculating a correction factor (since it's constant quantizer, it should work)

I understand you want to concentrate on the quality and speed for the time being, but it would be nice if you could add the possibility to your to-do list ;)

[Darksoul71]

@Boulder:
Why should OPV Encoding à la D2SBoBa be integrated in the encoder itself ?
IMO thats something frontends such as CCEFront, AutoQ2, etc are for.

I haven´t played around with HC but my AutoIt Implementation for CCE doing Q-Search is only 5 lines of code. This should not be to hard to modify for HC.

-D$

[Boulder]

@Boulder:
Why should OPV Encoding à la D2SBoBa be integrated in the encoder itself ?
IMO thats something frontends such as CCEFront, AutoQ2, etc are for.

I haven´t played around with HC but my AutoIt Implementation for CCE doing Q-Search is only 5 lines of code. This should not be to hard to modify for HC.

-D$
CCEFront does the job correctly, unfortunately for CCE only. CCE doesn't like Didée's LTSMC function at all but usually ends up crashing Avisynth during the prediction phase and also during a 2-pass encode. So far HC hasn't done the same so I figured that LTSMC being a very slow function, it would be useful to have a working CQ prediction, preferably inside the encoder itself as Hank knows the program best.

If you could modify the code and get it working, I'd be more than happy to test it as I encode my captures weekly. The thing I'm missing most is the automated process of correcting the CQ value if the final filesize hasn't hit close enough to the target. I'd be glad to code it myself if I just had the time to start brushing up my old C++ skills again..unfortunately it'll be next year till I get to that :(

[john3voltas]

I haven´t played around with HC but my AutoIt Implementation for CCE doing Q-Search is only 5 lines of code. This should not be to hard to modify for HC.

-D$
Hi Darksoul71,
I am just trying to grasp the basics on AutoIT v3 and I am a big fan of HCEnc.
And I would really like to have an OPV prediction FrontEnd for HC.
So if you could post the source code from your tool maybe I could adapt it to HC.
That is, if you're willing to show the sources ;).
Oh, of course that you might also be interested in doing the HC frontend yourself which would save us all a lot of headaches LOL.
Thanks

[Darksoul71]

@john3voltas:
If you are willing to code, this would "save" me from coding some HCEfront in Delphi.
Here is the code snippet from my tool Komprezz2MPEG which does the prediction.
$CQ = 40

; Do Q-Estimation if OPV encoding selected
if ($EncMode = "opv") Then
$compAVS = $TempPath & "\comp_movie.avs"
$compMPV = StringTrimRight($CompAVS,3) & "m2v"
FileCopy ($FinalAVS, $compAVS)
AddLine ($compAVS, "SelectRangeEvery(1500,15)")

$cWidth = 0
$cHeight = 0
$cFrames = 0

AVI_Info($compAVS, $cWidth, $cHeight, $cFrames)

$CQ = 40
$CQ_Alt = $CQ + 2

While (Abs ($CQ-$CQ_Alt) > 1)
$CQ_Alt = $CQ
GenerateECL ($CompAVS, $CompMPV, $TempECL, $cFrames, $EncBitrate, $CQ, $EncMode, $SourceWidth, $SourceHeight)
RunEclCCE ($TempECL)
Sleep (1000)

$EstSize = (FileGetSize ($CompMPV) * 100) / 1024
$TSize = ($EncBitrate / 8) * $Duration
$CQ = Round ($CQ*($EstSize / $TSize))
WEnd
Endif

Ok, agreed ! I´ve lied. It´s a bit more than five lines <ggg>

But the "core" ist the the while loop. If the quality values for HC Enc are different from CCE, you´ll have to modify the start value. Also CCE´s Q-Value is different from e.g. the Qualitylevel of TMPEGEnc.

Reason:
For CCE a low Q-value means higher quality versus a high Q-value means lower quality. For TMPEGEnc it´s vice versa: A high percent value (e.g. 70%) means higher quality compared to a lower percent value (e.g. 30%).

So if the Q-value of HC Enc would be proportional to the quality (as with TMPEGEnc), you would have to modify the formula:
$CQ = Round ($CQ*($EstSize / $TSize))
needs to be modified to
$CQ = Round ($CQ*($TSize / $EstSize))

The approach of the formula is really simple (Note: I didn´t invent the formula. I´ve just used it. IIRC it was posted over at the DVD-Rebuilder forum when they discussed how to incorporate the D2SBoBa OPV encoding approach into DVD-Rebuilder). The ratio between the estimated filesize (as we only encode a small snippet of the final movie) and the targeted filesize is used as factor for the new q-value. The closer the filesize for the encoding with the calculated q-value gets to your target filesize, the smaller the changes to the q-value will be. Normally this formula converges within 3-4 passes with CCE. Of course not as fast as Newton Raphson, but the sources of D2SRoBa are available. So feel free <ggg>

Oh, there´s one thing I have forgotten: If you do OPV encoding based on your target bitrate rather than your target filesize, you´ll also need to obtain the duration of the movie in second (see my calculation for $T_Size). I use a small Delphi app which I call from my AutoIt script for extracting the relevant informations of the AVISynth files. Not shure how tylo solved this within CCEFront but as Newton-Raphson requires two values to calculate the gradient, my first guess would be that he reads the frame number from the ECL file which CCE generates. If you can live with specifying targetsize only, you don´t need to do this.

Have fun,
D$
Below is the function I use to generate a CCE2.5 compatible ECL-file for encoding.

;++++++++++++++++++++++++++++++++++++++++++++++++++
Func GenerateECL ($Source, $Target, $ECLFile, $Frames, $Bitrate, $QValue, $Mode, $Width, $Height)

$Target = StringTrimRight($Target,4)

If ($Mode = "opv") Then $VMode = 0
If ($Mode = "cbr") Then $VMode = 1
If ($Mode = "mpcbr") Then $VMode = 3

$file = FileOpen($ECLFile, 2)

FileWriteLine($file, "; Cinema Craft Encoder SP -- Encoder Control List")
FileWriteLine($file, "")
FileWriteLine($file, "[item]")
FileWriteLine($file, "title=Generated by Komprezz2MPEG")
FileWriteLine($file, "vid_out=1")
FileWriteLine($file, "vaf_out=1")
FileWriteLine($file, "aud_out=0")
FileWriteLine($file, "vid_file0=" & $Target & ".m2v")
FileWriteLine($file, "vid_file1=" & $Target & ".m2v")
FileWriteLine($file, "vaf_file=" & $Target & ".vaf")
FileWriteLine($file, "aud_file=" & $Target & ".mpa")
FileWriteLine($file, "file_focused=0")
FileWriteLine($file, "encode_mode=0")
FileWriteLine($file, "packet_size=4096")
FileWriteLine($file, "timecode=0x1000000")
FileWriteLine($file, "width=" & $Width)
FileWriteLine($file, "height=" & $Height)
FileWriteLine($file, "vmode=" & $VMode)
FileWriteLine($file, "frame_rate_idx=3")
FileWriteLine($file, "cbr_brate=" & $Bitrate)
FileWriteLine($file, "opv_q_factor=" & $QValue)
FileWriteLine($file, "opv_brate_min=300")
FileWriteLine($file, "opv_brate_max=9000")
FileWriteLine($file, "vbr_brate_avg="& $Bitrate)
FileWriteLine($file, "vbr_brate_min=1200")
FileWriteLine($file, "vbr_brate_max=9000")
FileWriteLine($file, "vbr_bias=30")
FileWriteLine($file, "vbr_pass=2")
FileWriteLine($file, "quality_prec=30")
FileWriteLine($file, "use_filter=0")
FileWriteLine($file, "filter_val=6")
FileWriteLine($file, "seq_endcode=1")
FileWriteLine($file, "dvd=0")
FileWriteLine($file, "half_width=0")
FileWriteLine($file, "half_height=0")
FileWriteLine($file, "fast_mode=0")
FileWriteLine($file, "progressive=1")
FileWriteLine($file, "alternate_scan=0")
FileWriteLine($file, "non_linear=1")
FileWriteLine($file, "top_first=1")
FileWriteLine($file, "lum_level=0")
FileWriteLine($file, "intra_dc_prec=2")
FileWriteLine($file, "aspect_ratio=2")
FileWriteLine($file, "gop_m=3")
FileWriteLine($file, "gop_nm=5")
FileWriteLine($file, "gop_hdr=15")
FileWriteLine($file, "seq_hdr=1")
FileWriteLine($file, "all_closed_gop=0")
FileWriteLine($file, "fix_gop_length=0")
FileWriteLine($file, "mpeg1=0")
FileWriteLine($file, "mpeg1_cps=0")
FileWriteLine($file, "samples_per_sec=48000")
FileWriteLine($file, "stereo=2")
FileWriteLine($file, "brate_idx=7")
FileWriteLine($file, "crc=1")
FileWriteLine($file, "[file]")
FileWriteLine($file, "name=" & $Source)
FileWriteLine($file, "type=0")
FileWriteLine($file, "frame_first=0")
FileWriteLine($file, "frame_last=" & $Frames)
FileWriteLine($file, "encode_first=0")
FileWriteLine($file, "encode_last=" & $Frames)
FileClose($file)
EndFunc
;++++++++++++++++++++++++++++++++++++++++++++++++++

[john3voltas]

Gee, thanks for the sources Darksoul71 ;).
That was a very nice gesture from you :).
I will look into these sources in more detail during the WE.
Nevertheless, I'm affraid it will take me one week or so to have something ready for testing.
I will post it afterwards.
C u.

[Darksoul71]

@john3voltas:
You´re welcome ! I´m always pleased to help und normally always release my sources. I could do some modifications to my AVIInfo Tool, so it can be controlled solely via commandline interface and be run hidden. Currently my AVIInfo is completely GUI-driven and I use the following code to obtain resolution and frame count of the opened window.

;++++++++++++++++++++++++++++++++++++++++++++++++++
Func AVI_Info($AVIFile, ByRef $W, ByRef $H, ByRef $Dur)
BlockInput(1)

Run($aviinfo, "", @SW_SHOW)
WinActivate ("AVI Info")
WinWaitActive("AVI Info")

Sleep (500)
Send("{TAB}")
Sleep (500)
Send("{ENTER}")
Sleep (500)
Send($AVIFile)
Send("{ENTER}")

WinWaitActive("AVI Info")

Sleep (500)

$W = ControlGetText("AVI Info", "", "TEdit3")
$H = ControlGetText("AVI Info", "", "TEdit2")
$Dur = ControlGetText("AVI Info", "", "TEdit1")

Send("!{F4}")

BlockInput(0)
EndFunc
;++++++++++++++++++++++++++++++++++++++++++++++++++


I could store the informations either inside an ini-file (which is easy to read from within AutoIt) or some environmental variables which could be read via "EnvGet". Let me know via PM if you have interest in this. May be we should ask over at the DVDRebuilder forum for the source code which is used within DVDRebuilder for the OPV encoding with HC Enc ?

Just an idea...

-D$

[jdobbs]

Just as a suggestion, you may be interested to know that in HC you can use fractional CQ value for more accuracy. So you could use CQ=2.65, for example, as opposed to having to select between 2 and 3.

Also, the sizing result and the Q are not linear, so you can reduce the number of prediction passes required by not making linear decisions.

So the calculation:

($CQ*($EstSize / $TSize))

could be:

($CQ*($F * ($EstSize / $TSize)))

Based on suggestions I got from tylo, I've found the $F=1.35 works well. But you may want to play with it.

I usually loop through until the predicted size is within 1% of the target size or the value doesn't significantly change.

[Boulder]

Just as a suggestion, you may be interested to know that in HC you can use fractional CQ value for more accuracy. So you could use CQ=2.65, for example, as opposed to having to select between 2 and 3.

Also, the sizing result and the Q are not linear, so you can reduce the number of prediction passes required by not making linear decisions.

So the calculation:

($CQ*($EstSize / $TSize))

could be:

($CQ*($F * ($EstSize / $TSize)))

Based on suggestions I got from tylo, I've found the $F=1.35 works well. But you may want to play with it.

I usually loop through until the predicted size is within 1% of the target size or the value doesn't significantly change.
And then the "sizing pass" which should be done if the final filesize of the encoded video is not within specified percentage of the target filesize, the correction factor ($CorFac=$target/$final) should be used in the formula: ($CQ*($F * $CorFac *($EstSize / $TSize))). That should ensure that the second CQ encode will be close enough to the target. Or am I totally wrong? :D

[jdobbs]

With CCE is would be better just to a normal multi-pass encode using an average bitrate, etc... that way it would use the .VAF from the OPV pass and you will get exactly the size you requested. The purpose of the prediction passes is to save time -- if it failed to be accurate enough, you might as well make sure it won't happen again.

[Boulder]

With CCE is would be better just to a normal multi-pass encode using an average bitrate, etc... that way it would use the .VAF from the OPV pass and you will get exactly the size you requested. The purpose of the prediction passes is to save time -- if it failed to be accurate enough, you might as well make sure it won't happen again.
That's what D2SRoBa/CCEFront does, and that's what would be nice to have in HC - it would create the dbs file in CQ mode which could be used for a sizing pass if necessary but I'm afraid it would slow the CQ encoding down a bit.

[jdobbs]

Hmmm... that's a coincidence. I recently asked Hank315 to look at that possibility. Not sure if it's feasible. He'd have to save all his collected data from the CQ pass into a file -- and that's assuming a CQ pass is equivalent to the first of a two pass encode in his algorithm...

[Darksoul71]

Hi jdobbs,

thanks for your infos / suggestions. You´re absolutely right. As HC_Enc uses a floating point format with three decimal places the (modified) formula needs to be changed to somewhat like this: $CQ = Round ($CQ*$F*($EstSize / $TSize),3)
Not quite shure if you really need all three decimal places to hit the exact target. That is something I would have to evaluate. I can somewhat remember the "f" value from the discussion over at the DVDRebuilder thread on RoBa encoding.

Most of my OPV-encoding tend out to be smaller than my targeted size and oversized file are very rare. To avoid oversizing a simple approach for CCE is to increase the estimated Q-value by 1. You will mostly not notice any drop in quality and definitely not get oversized files (unless your 1% samples were not very representative for your movie). The other approach is to aim for a smaller size than your media has storage capacity, e.g. using 4300 MB as target size for a DVD-R.

Using rejig if the file is oversized might also be an option as you will mostly not notice a decrease in quality if you shrink the file for let´s say 2%.

-D$


BTW: This was the posting of tylo over there (hopefully no-one "smashes" me for posting such a long text :)

Oh, my. This thread was really about RoBa, wasn't it?

jdobbs, I don't know much about how you implemented your Q-estimation routine. I know you use 0.5% sample, which in my experience gives a too high error variance. I use 1% sample which is barely OK. (0-2% estimation error most of the time, but sometimes up to 3%).

If you want, I could give you a simple condensed algorithm how to search for Q. It's not very complicated.

The way D2SRoBa does it is somewhat more advanced because it must deal with the max-bitrate limitation for SVCD. The Q-function normally follows an exponential (almost straight) slope, but near max-bitrate it suddenly bends a lot. In that case I switch from newton-raphson to binary search for Q.

For DVD bitrates, that shouldn't be an issue, and it should be fast and easy to find the closest Q that match the target bitrate. Btw. you have to subtract about 1.2% from the estimated size, when determining the Q. Also check 'Restrict auto I-frame insertion' in CCE (fix_gop_length=1) - it improves the estimation accuracy a little (tips from jonny).

/More technical talk: Another note about the accuracy of the final mpv bitrate (from an OPV encode) versus the original target bitrate:

In D2SRoBa, I compute the current estimated Q-step size in percentage, i.e. 100*(estBR(Q) - estBR(Q+1))/targetBR. This varies from 0.8% to 4%. Example: if this number is 3%, it means that it is 3% between each Q (in that Q-area), which again means that, in worst case you cannot hit better than 1.5% on target, even if your estimates are absolutely correct. What I'm saying is that this error comes on top of the possible estimation error (e.g. if test samples are not representable).

Anyway, normally this all equals out, and D2SRoBa has proven that the method works excellent.

when I use undot and deen on a source it yields about 12Q lower than without them. At least with about Q40-50, cause I dont really remember how much improvement the lower ones yielded, but it was definately an improvement.

The material below covers two areas.

1> How the complexity, and the resulting compressibility of a video source, can be quantified via the numerical bitrate derived from the physical filesize of a 1% sample where the sample has been created at a known Q value with CCE OnePassVbr (note 3).

Q values normally in the range of 10-40 would be considered the working range to provide excellent to acceptable quality. The goal is for the user to develop a personal quality constant that is always appropriate for the users display methods, and to be able to know in advance of encoding if that level of quality can be achieved. My personal 'quality constant' is Q28

2> How to provide a simple logic so that a program or user can know in advance the approximate quality that will be delivered from a CCE multipass encode.

These goals, and the suggested method to achieve it, is based upon certain educated guesses Bach made since he did not having access to CCE source code (note 2 below). From a practical standpoint, and at the very least, it should provide the user with a very good quality indication before the full encoding process.

Explanation -

This average bitrate number derived from the filesize of the 1% OPV sample is the minimum needed to contain the complexity of the sampled video source and closely deliver the same quality of the OPV sample. This may seem obvious, yet the important fact is that the complexity, and thus the resulting compressibility of the sampled video source has been quantified within a number, the size. This size can then be translated to an average bitrate in Kbps which is compatible with CCE VBR.

This number, the derived average bitrate, can be used as a bridge between the two different encoding methods (OPV and VBR), each of which has distinct strengths and weaknesses when standing alone. By bridging these methods we can combine many of the strengths of both.

As a proof exercise, CCE Multipass VBR when using this derived Kbps number and the same MIN/MAX very closely mirrors the encode that is produced by OPV using a Q number. This is shown in the attached spreadsheet, linked here and attached at the bottom of this post. The accuracy of the translation between Q and KBPS seems clear.

With a working method of translation, we can combine the strengths of both processes for certain situations.

One situation is where multipass is specified as the encoder tool because of its ease of use, and/or the need for its main strength of creating a final file size exactly on target. While multipass VBR is brilliant in achieving a size target by the nature of its size based process, it has an inherent weakness. It does not possess any internal process that allows it to understand when the ABR, specified by the user, will not be sufficiently high enough to yield a acceptable encode quality.Thus, the final quality of the encode is unknown until after the encode is finished (many hours later).

OPV has opposite characteristics in that quality, via a controlled quantizer approach, is achieved at the cost of non existent size control. OPV is one of the kings of quality and also the fastest software Mpeg2 encoding method available, but without accurate filesize prediction, OPV was only useful for encodes to be held on hard drive as the source complexity solely dictated the bitrate. The final encode size could fluctuate radically, depending on the unique compression characteristics of the individual source, between two sources of the same time length. Thus, the final size of the encode is unknown until after the encode is finished.

Predictive sampling, sometimes called 1-PassBach or its iteration, 2-PassBach and best exampled in the D2SRoBa plugin used in combination with Dvd2Svcd/Dvd, was introduced to deal with this weakness in OPV. It uses multiple small samples and solves for the Q which will achieve a target size. Because it solves for Q, which indirectly manipulates bitrate, it effectively provided size control as well as the ability to predict the final quality before the actual encoding process. It does this brilliantly and yields a HQ encode in the shortest time possible.

However, Conditional 2Pass Bach as implemented by D2SRoBa took a long time to develop and quietly does many complex actions not readily apparent to the user. The series of prediction samples, and the Newton-Raphson Method calculus (adapted by R6D2) as used by D2SRoBa, are complicated to implement. This can be beyond the scope of many casual hobby programmers, or manual encoding techniques.

In this situation, the derived average bitrate of a known Q sample as described above, can be used to translate between the two processes. Because the OPV sampling process has actually encoded 1% of the video source as small equally spaced sections, the derived bitrate generated from the sample size inherently contains an awareness of the compressibility of the source. Thus, the D-ABR of the known Q sample is the minimum required bitrate that will be needed to achieve a similar quality in the bitrate based multipass VBR encode. In effect, the process matches the distinct compression characteristics, which are unique to the source complexity, to the appropriate Average Bitrate needed to hold and contain our desired quality. In addition, the identified Derived-ABR (as Kbps) will also provide the framework for creating branching program logic.

The derived Kbps value is used to establish a quality 'GO/NO-GO' conditional.

To do this, an extra step is added when using conventional multipass.

Normally an average bitrate would have been calculated to fill the space available and the encode started, whether the bitrate was adequate, or inadequate, to achieve an acceptable quality level for the particular compression needs of the source. There is not much difference in this method, except before starting the encode, a comparison is made between the Kbps calculated to fill the size available and the derived bitrate from the known quality sample.

GO - When the size calculated bitrate is equal or larger than the derived bitrate from the sample -

The encode is started as normal. This will yield a minimum guaranteed level of known quality within a known filesize. Note: if more space is available for the encode than the minimum requirement predicted by the sample D-ABR, the size calculated ABR will of course be larger than that of the sample's D-ABR. The quality will therefore also increase over that of the known quality sample.

NO-GO - When the size calculated bitrate is less that the derived bitrate from the sample -

Allows the programmer or manual user to detect, in advance of the encode, when a known quality
cannot be achieved within a known filesize. Additional logic can then be triggered to deal with
this situation such as resolution adjustment, compression filtering, downsizing of audio, etc., to
make more space available for the encode.
---------------------------------------------------

Note 1: This is a work in progress as a self learning exercise and is based upon the original propositions posted by Bach and many subsequent revisions by the members of this community, especially those that relate to what we now call 1-Pass and 2-Pass Bach as used in D2SRoBa. This post targets how to apply some of these OPV techniques to CCE Multipass VBR. I highly doubt any of the information here is original, but I do hope this compilation may provide a useful method for those who wish to use CCE multipass and would like to know general quality information in advance of the actual encoding process. Also, I may have not been precise in the use of some of the technical terms, but they are accurate enough for now.

Note 2: This line from the original post is an important part of the assumptions here. However, it was never independently verified as correct, hence the word assumption [all emphasis is mine] - : Originally posted by Bach - "Attention that the Q.factor is an exclusive concept of CCE, directly related with the quantisation, but it does not mean QM nor S, and it is not very clear the way that it is calculated. Based in the results of my tests, I believe that Q.factor is in some way proportional to the average value of the product QM(i, j)*S. Anyway, the Q.factor is the direct measure of the quality to be gotten so that, if two different films are recompressed using the same settings and the same Q.factor, can be guaranteed that they have the same quality."

Note 3: The sampling method is only available when using AviSynth to frameserve video to CCE OnePassVbr (OPV). It assumes the reader understands the use of avisynth and is capable of editing an AVS script, loading the script into CCE, choosing OPV mode and a Q value. To produce an AVS for creating the sample, simple add this line at the bottom of your working script: SelectRangeEvery(1200,12). Obviously, this line must be removed after the sample has been created.

You will need the derived bitrate spreadsheet calculator, linked here, or attached to the NEXT post below, OR use this formula - (((Sample_Size_In_Bytes*(100/Sample_Size_Percentage))*8)/1000)/(Total_Frames/Frame_Rate)

[jdobbs]

Not quite shure if you really need all three decimal places to hit the exact target. That is something I would have to evaluate. I can somewhat remember the "f" value from the discussion over at the DVDRebuilder thread on RoBa encoding. I use two decimal places myself. In fact I use the same prediction routine for CCE and for HC by just multiplying HC's CQ value by 10 before entering the analysis routine, and dividing by 10 on exit.

Also, the formula below works better if you want to calculate F within the passes. It will reduce by half the number of prediction passes needed on larger Q (or CQ) values.

F = 1.35 + (Q * (Q / 20) / 1000)

[Darksoul71]

@jdobbs:
Thnx for your explanations.

@john3voltas:
I´ve been playing around a bit with autoit but I´m too tired to find out why this thingie doesn´t converge :)

Feel free to use my sources but please also share your sources with other peoples to let them know how you did this. If you require my AVIInfo proggie, drop me a PM and I´ll mail it to you.

Good night,
D$

; ----------------------------------------------------------------------------
;
; AutoIt Version: 3.1.0
; Script Title: HCpred V1.0
; Author: Dark$oul71
;
; Script Function:
; Provide OPV encoding similar to D2SRoBa for HC enc
;
; ----------------------------------------------------------------------------
$aviinfo = @ScriptDir & "\AVIInfo\AVIInfo.exe"
$hc_enc = @ScriptDir & "\HCbatch_016_testrelease_08-09-2005.exe"
;++++++++++++++++++++++++++++++++++++++++++++++++++
Func AVI_Info($AVIFile, ByRef $W, ByRef $H, ByRef $Dur)
BlockInput(1)

Run($aviinfo, "", @SW_SHOW)
WinActivate ("AVI Info")
WinWaitActive("AVI Info")

Sleep (500)
Send("{TAB}")
Sleep (500)
Send("{ENTER}")
Sleep (500)
Send($AVIFile)
Send("{ENTER}")

WinWaitActive("AVI Info")

Sleep (500)

$W = ControlGetText("AVI Info", "", "TEdit3")
$H = ControlGetText("AVI Info", "", "TEdit2")
$Dur = ControlGetText("AVI Info", "", "TEdit1")

Send("!{F4}")

BlockInput(0)
EndFunc
;++++++++++++++++++++++++++++++++++++++++++++++++++
Func GenerateHCL ($HCL, $Input, $Output, $Q)
$file = FileOpen($HCL, 2)
FileWriteLine($file, "*infile " & $Input)
FileWriteLine($file, "*outfile " & $Output)
FileWriteLine($file, "*bitrate 4000")
FileWriteLine($file, "*maxbitrate 9000")
FileWriteLine($file, "*cq " & $Q)

$myFile = FileOpen(@ScriptDir & "\myHC settings.txt",0)

While 1
$line = FileReadLine($myFile)
If @error = -1 Then ExitLoop
FileWriteLine($file,$line)
Wend

FileClose($file)
FileClose($myFile)
EndFunc
;++++++++++++++++++++++++++++++++++++++++++++++++++
Func AddLine ($TextFile, $Line)
$file = FileOpen($TextFile, 1)

; Check if file opened for reading OK
If $file = -1 Then
MsgBox(0, "Error", "Unable to open file.")
Exit
EndIf

FileWriteLine($file, $Line)

FileClose($file)

EndFunc
;++++++++++++++++++++++++++++++++++++++++++++++++++
Func GetPAR ($Width, $Height, ByRef $FPS)
; Determin FPS by vertical resolution
If (Mod($Height, 240) = 0) Then $FPS = 29.97
If (Mod($Height, 288) = 0) Then $FPS = 25

; Note: Anamorphic material is not supported
; PAL AR´s
If ($Width <= 352) And ($Height = 576) Then Return (2.188)
If ($Width <= 480) And ($Height = 576) Then Return (1.641)
If ($Width <= 352) And ($Height = 288) Then Return (1.094)
If ($Width <= 704) And ($Height = 576) Then Return (1.094)
If ($Width <= 720) And ($Height = 576) Then Return (1.094)

; NTSC AR´s
If ($Width <= 352) And ($Height = 480) Then Return (1.818)
If ($Width <= 480) And ($Height = 480) Then Return (1.364)
If ($Width <= 352) And ($Height = 240) Then Return (0.909)
If ($Width <= 704) And ($Height = 480) Then Return (0.909)
If ($Width <= 720) And ($Height = 480) Then Return (0.909)

Return(1)
EndFunc
;++++++++++++++++++++++++++++++++++++++++++++++++++
; Processing of CLI. Open file dialog if no parameter (= source file) was given.
If ($CmdLine[0] >= 1) Then
$SourceFile = $CmdLine[1]
Else
$SourceFile = FileOpenDialog("Choose Videofile to process" , "C:\", "Videofiles (*.AVS)", 2)
EndIf

If ($CmdLine[0] = 2) Then
$BitRate = $CmdLine[2]
Else
$BitRate = InputBox("Bitrate", "Enter your bitrate in kBit/s:")
EndIf

; Initialize values for video cropping
$Left = 0
$Right = 0
$Top = 0
$Bottom = 0

$HasCropInfo = 0

; Quellauflösung auslesen
$SourceWidth = 0
$SourceHeight = 0
$Frames = 0
$FPS = 0

AVI_Info($SourceFile, $SourceWidth, $SourceHeight, $Frames)

; Determine Pixel Aspect Ratio of Source Material
; Note: Anamorphic material is not supported -> 4:3 max
; Supported resolutions are
; PAL: 352x288, 352x576, 480x576, 704x576, 720x576
; NTSC: 352x240, 352x480, 480x480, 704x480, 720x480
$SourcePAR = GetPAR ($SourceWidth, $SourceHeight, $FPS)

$Duration = Round ($Frames / $FPS)

; Do Q-Estimation if OPV encoding selected
$compAVS = @ScriptDir & "\comp_movie.avs"
$compMPV = StringTrimRight($CompAVS,3) & "mpv"
$HCFile = @ScriptDir & "\hc_temp.ini"

FileCopy ($SourceFile, $compAVS)
AddLine ($compAVS, "SelectRangeEvery(1500,15)")

$cWidth = 0
$cHeight = 0
$cFrames = 0

AVI_Info($compAVS, $cWidth, $cHeight, $cFrames)

; HC Q-Faktor: 1 to 31
$CQ = 5
$CQ_Alt = $CQ + 2

$F = 1.35

$EstSize = 1
$TSize = 3

While (Abs ($EstSize - $TSize) > 0.01)
;While (Abs ($Es-$CQ_Alt) > 0.1)
$CQ_Alt = $CQ

GenerateHCL ($HCFile, $compAVS, $compMPV, $CQ)
RunWait ($hc_enc & " -ini " & Chr(34) & $HCFile & Chr(34), @ScriptDir, @SW_SHOW)
$EstSize = (FileGetSize ($CompMPV) * 100) / 1024
$TSize = ($Bitrate / 8) * $Duration
MsgBox(0,"Q-Wert", $CQ & " " & $EstSize)
;$CQ = Round ($CQ*$F*($TSize / $EstSize ),3)
$CQ = Round ($CQ*$F*($EstSize / $TSize),3)
;$F = 1.35 + ($CQ * ($CQ / 20) / 1000)
WEnd


$FinalMPV = StringTrimRight($SourceFile,3) & "mpv"

GenerateHCL ($HCFile, $SourceFile, $FinalMPV, $CQ)
RunWait ($hc_enc & " -ini " & Chr(34) & $HCFile & Chr(34), @ScriptDir, @SW_SHOW)

; Clean up temporary files
FileDelete ( $compAVS )
FileDelete ( $compMPV )
;FileDelete ( $HCFile )

[john3voltas]

@Darksoul71,
Thanks again for posting all your goodies on AutoIT and OPV prediction.
And be sure that when I finish the tool (if I ever do finish it :rolleyes: ) I will certainly release my sources too ;).
I guess you will all have to be patient with me because last time I coded anything, Turbo Pascal still ruled.
That is, roughly 12 years ago :D.
Cheers

[Boulder]

I guess you will all have to be patient with me because last time I coded anything, Turbo Pascal still ruled.

Uh oh, I guess I opened a can of worms :D I can do some testing for you if my time just allows any extensive tests, I'm about to become a father of two so time might be heavily restricted ;) That's one reason why I asked for the automatic CQ mode.

Thank you goes to both you and Darksoul71, take your time:)

[EpheMeroN]

I use MuxMan to do simple authoring for many projects I do. I just encoded a video with the new HC Encoder v0.15A and when I tried to author it in MuxMan I received this error:

Largest GOP of 38 fields exceeds DVD specification of 36. Resulting DVD is non-standard.

Is there ANY way to correct this without re-encoding? I know there probably isn't, but I spent 64 hours encoding this thing (heavy filtering).

A copy of the log from the encode can be found here (http://users.adelphia.net/~l337/movie.log).

[Marius-the-Mad]

You used DGPulldown afterwards, am I right ? Performing a pulldown from 23.976 to 29.97 (or any other higher than source framerate) means using flags to duplicate fields on playback. And from the decoder's point of view, the GOPs get longer. I'm afraid you will have to reencode. :(

I did some similar encodings and from my experience: maximum GOP length for 23.976 -> 29.97 = (23.976 / 29.97) * 18 - 1.

Or 2 instead of 1 - I don't remember whether I had to do it the former or the latter way. :(

It may look weird, but setting GOP length to 14 didn't help me and I had to reencode.

[EpheMeroN]

Correct. I did use DGPulldown. So when I re-encode in HC, what do I do differently? I uncheck "autogop" and what settings will I use? I have no knowledge of GOP length or anything GOP-related whatsoever.

[Marius-the-Mad]

Uncheck autogop, set GOP length to 13. I hope this will be low enough and you won't have to reencode once again. I'm sorry, but my memory fails me in this case. :( Leave the number of B-frames at 2. And this should do it.

Regards
MTM


<edit> Try to encode 2000 frames, pulldown the file, feed it to muxman and see what happens. If it's OK, try the same with autogop. If it fails, I think there is a reasonable chance everything will be all right with GOP legth at 13.

Please, may someone with a higher amount of knowledge than mine correct me if I'm wrong. :)

[Sir Didymus]

Largest GOP of 38 fields exceeds DVD specification of 36. Resulting DVD is non-standard.

Is there ANY way to correct this without re-encoding? I know there probably isn't, but I spent 64 hours encoding this thing (heavy filtering).

A copy of the log from the encode can be found here (http://users.adelphia.net/~l337/movie.log).

There is...
Or better, there are some...
I know a couple:

One is Womble MPEG Video Wizard
http://www.womble.com/products/

A second possibility is based on Cuttermaran.
Settings --> Encoding --> Create DVD Compliant Stream

Both are extremely fast, since just the out-of-dvd-spec segments are re-encoded...

At the present Cuttermaran is supporting whatever encoder (but needs to write a little xml configuration file in order to make the encoder available to the application). The default encoder is TmpgEnc. Don't ask me more since I've never been in a situation like the yours...


Edit: but please report your experience, if you intend to try with one of the above applications...

Edit2; OOPS!!! SORRY!!! IT SEEMS I WROTE SOMETHING TOTALLY WRONG, due to the quick reading of the last posts in the thread... Hem... Well, after some reconsideration, and especially after reading:

http://forum.doom9.org/showthread.php?t=98569&highlight=dgpulldown

and

http://forum.doom9.org/showthread.php?t=93842

It seems it is highly possible that the out of specs GOP size problem you face is present in the whole encoded stream you have...

So please discard all of the suggestions based on Womble and Cuttermaran: they may work efficiently just if the size of out-of-spec Gop is little compared to the whole stream size... And this seems is not your case... So re-encoding the whole stream, with a proper GOP structure is necessary if you know you will need DGPulldown in your process...

Cheers,
SD

[hank315]

Would it be possible to add a prediction engine to the CQ mode?Yes, will try to implement some kind of 1-pass VBR with prediction in the next release.

@Boulder
Looking at all the posts after yours it seems more people are interested in it :D

@2COOL
You're right, this thread is getting pretty large.
Will open a new one for HC016 (very soon)

[EpheMeroN]

@Sir Didymus: I will need DGPulldown because the source is 23.976fps. I will do what Marius-the-Mad recommended by using a lower GOP. But I have a question. If HC's default GOP is "automax", is there a way, from viewing the log, to see what the actual max of the GOP was? The error said that it was over the GOP limit by 2. Now correct me if I am wrong (which I probably am) but wouldn't I just have to limit that number by 2? I heard the NTSC MAX was 18 and that GOP size affects quality. Switching it to 13 seems like a drastic measure. Last time I used DGPulldown, it was on a 25fps source that I did a 25 > 29.97 on so I could view it on my NTSC TV and it never reported that error.

[Trahald]

gop sizes are usually 12 15 and 18.. (for a regular mpeg2, bframes = 2, gop)
23.976 fps -> 29.97fps pulldown is generally best done at gop size of 12. (12x2.5frames = 30 fields).. since 15x2.5frames = 37.5 which is higher than spec... you can do oddly configurated gops to get to 13 or 14 but they would tend to hurt things more than having a traditional gop format (2 bframes) at a size of 12.

25->29.97 conversion works at 15 GOP because it brings you to right around 36 so your ok.

[Sir Didymus]

@EpheMeroN

The reasons of your troubles are perfectly described by the last post of Trahald.

Just want to add that the "traditional" GOP sizes of 15 and 18 (limits for PAL and NTSC) comes from some timing limits in the DVD specs (from 0,4 to 1.2 second max per VOBU, except the last VOBU in the cells, where it is allowed to go up to 1.4, allowing the authoring package the possibility of mapping up to two full GOPs per VOBU). For other reasons, as already indicated, it is almost necessary to have GOP sizes multiple of three, so the indicated mac GOP of 12 should by the right chioce in your situation.

From the compressibility/quality point of view, the situation is as follow: reducing the GOP size means increasing the I frame density in the mpeg2 stream, which improves the overall stream quality but decreases its compressibility...

Reading the log of your encoding I see your average video bitrate is beyond 4500 Kbit/s, so you have a lot of margin...

I am pretty sure with this bitrate you are going to obtain an encoded stream, imposing a GOP size of 12, almost undistinguishable from the original...

Cheers,
SD

[john3voltas]

Yes, will try to implement some kind of 1-pass VBR with prediction in the next release.
Not to get you off your track or anything Hank315, but I fully agree on what DarkSoul71 said about prediction being the job for a frontend tool.
Looking at other free or commercial encoders I can't find any with 1-pass prediction.
Just hope that Boulder won't try to kill me for this :D.
If I were in your shoes I would add that to my to-do list, but I would make sure that bugs would come in 1st place ;).
Even because with the sources that DarkSoul71 posted here, I am right now predicting with very good results.
See you

[Amnon82]

btw. Hank released his new Version 0.16 (http://forum.doom9.org/showthread.php?t=100133)

[Darksoul71]

@john3voltas:
Even because with the sources that DarkSoul71 posted here, I am right now predicting with very good results.
Good to hear ! Please keep me "posted" on your progress. May be I can integrate "our" code for CQ encoding with HC Enc into my MPEG2DVD tool.

Cheers,
D$

[Boulder]

Looking at other free or commercial encoders I can't find any with 1-pass prediction.
Just hope that Boulder won't try to kill me for this :D.

Hey, no problem :) I just thought it might be easiest if it was in the encoder itself. That way it would work for any other program that uses HC without the need of any extra programs.

[Darksoul71]

@Boulder: I agree that OPV encoding integrated into HC would be less of a PITA for other pograms calling HC enc. On the other hand doing OPV encoding "outside" the encoder gives you increased flexibiliy for things such as:
* Conditional filtering (à la KISS for D2SRoBa)
* Using different approaches for Q-Value calculation
etc.

[Marius-the-Mad]

you can do oddly configurated gops to get to 13 or 14 but they would tend to hurt things more than having a traditional gop format (2 bframes) at a size of 12.Thank you for this explanation, Trahald. I didn't know that.

I have a question, if I may. Do you know why using a GOP length of 14 resulted in a too long GOPs after using DGPulldown to go from 23.976 to 29.97 ? I tried it once and that was the result. I'm curious why - theoretically it should work I think, as 18 * 23.976 / 29.97 = 14.4...


Regards,
MTM

[LigH]

btw. Hank released his new Version 0.16 (http://forum.doom9.org/showthread.php?t=100133)
:angry:

If he did, I'd know before you...

This is a "test release" for hunting down specific bugs. Not a production release. Learn to tell those apart.

[Sir Didymus]

@Marius

I am not sure about the HC behaviour, but other encoders simply disobey the input GOP sizes that are not acceptable. If M (parameter defining the b frame rate into the GOP structure) is set to 3, the smallest acceptable GOP size below 15 is 12...

Here is what the CCE manual say on the matter:

http://img51.imageshack.us/img51/5088/image13tm.th.gif (http://img51.imageshack.us/my.php?image=image13tm.gif)

[john3voltas]

Good to hear ! Please keep me "posted" on your progress.
Sorry Darksoul71 but unfortunately last night my Seagate 80Gb SATA HDD died.
It just won't spin or be recognized by the BIOS.
It's dead alright! :devil:
I spent the whole morning reinstalling everything on an old Seagate 30GB IDE HDD that I had around.
But I lost the changes I had done to your script :(.
I am planning on getting back at this project but I'm going away for training in Hungary today and I'll only get back within two weeks...
Also, I am thinking about dropping AutoIt and move on to Delphi or VB because AutoIt is just very bad to build GUIs.
I'll let you know when I get back home.
Thanks for the support you've given me ;)

[john3voltas]

If he did, I'd know before you...
This is a "test release" for hunting down specific bugs.
Not a production release. Learn to tell those apart.
Always the good old Avalon! :D

[Amnon82]

This is a "test release" for hunting down specific bugs. Not a production release. Learn to tell those apart.

@LigH: Sorry, I only wanted to mention it, that a "new" version is out. ... and I know what a test release is.

Always the good old Avalon!

@john3voltas: Ah, You still now my old nicname ;)

Will open a new one for HC016 (very soon)

@all: It is realy a large thread. Lets see what the endcount will be :D

[hank315]

I am not sure about the HC behaviour, but other encoders simply disobey the input GOP sizes that are not acceptable. If M (parameter defining the b frame rate into the GOP structure) is set to 3, the smallest acceptable GOP size below 15 is 12...All GOP sizes are acceptable but some are better than others.
With 2 B-frames the optimal GOP lengths are 3, 6, 9, 12, 15 etc. for open GOPs.
With 1 B-frame the optimal lengths are 2, 4, 6, 8, 10, 12, 14 etc. for open GOPs but if the GOP is closed it's better to use 3, 5, 7, 9, 11, 13, 15...
The autogop option in HC016 tries to optimize these GOP sizes and will also use 3 consecutive B-frames if there's very little motion.
You can also encode with only I-frames using *GOP 1 0

[Darksoul71]

@hank:
would it be possible that you switch away from your
current "ini"-File format to a true ini-Format ?

Example:

*infile d:\....\test.avs
*outfile d:\....\test.m2v
*bitrate 4000
*maxbitrate 9000


This could look in "true" ini-format something
like this:

[HC Enc]
infile=d:\....\test.avs
outfile=d:\....\test.m2v
bitrate=4000
maxbitrate=9000


It would make it much easier to read / write
the config files for HC Enc as most of the
languages provide pre-built functions for
reading ini-files. Commenting out a setting
is still easy since your can add a simple _
or blank in front of the setting you want to
disable, e.g. "maxbitrate=9000" will not be
found anymore if you make it look like this
"_maxbitrate=9000"

Just an idea rather than a feature request.

Keep up the good work !

-D$

[Sir Didymus]

@hank315.

Sorry for the little digression in the thread... If I understand correctly you mean that even knowing it is sub-optimal, it is possible to set 2 b frames and GOP sizes of 13 and 14.

That's nice to know. But is the encoder (and more specifically, is your excellent encoder), under these conditions, capable of strictly obeying the GOP limits of 13 and 14 ?

In case it is, it becomes less clear why, as reported by Marius, by applying a 23.976 --> 29.97 pulldown process, that should present less than 36 fields to the decoder, is not actually working...

Thanks, and keep up your work !!! It's really appreciated by the whole community...

[jdobbs]

I use two decimal places myself. In fact I use the same prediction routine for CCE and for HC by just multiplying HC's CQ value by 10 before entering the analysis routine, and dividing by 10 on exit.

Also, the formula below works better if you want to calculate F within the passes. It will reduce by half the number of prediction passes needed on larger Q (or CQ) values.

F = 1.35 + (Q * (Q / 20) / 1000)Hmmm... just for everyone's sake -- while this formula works very well with CCE, it seems to be off on HC. I'm working on a new formula for HC now.

[Boulder]

Can the scaling between linear and non-linear quantization be calculated somehow, maybe that would be useful as HC's CQ mode means encoding at constant quantizer.

[EpheMeroN]

I am unaware if this issue I'm having now is HC-specific or just in general when changing GOP size. From my posts above, you can see that I had to lower the GOP # for my encode. Now that I am using a GOP or 12, the encode is going at least 2x slower. Is this normal? Before when I had HC on autogop, the 2-pass encode with my script took appx. 65 hours. Right now it's been 43 hours just for 74% of pass #1.