"Real" 1-Pass VBR for HCEnc without Prediction

[manolito]

While everyone else is busy watching soccer games I came up with an idea to teach HC to do 1-pass VBR encodes without the need for prediction encodes. QuEnc, MainConcept and Procoder can do it, why not HC?

The idea is pretty simple:
Divide the encode into a fixed number of segments, encode the first segment with a "safe" initial CQ (maybe 3), compare the resulting size (or bitrate) to the desired size (or bitrate), change the CQ value accordingly, encode the next segment using the new CQ value, repeat until the end of the source is reached.

At this time the project is rather more a feasibility study or a test platform than a finished implementation. The reason I go public with it at such an early stage is that this thing could use some tweaking, and any input from you guys is highly appreciated.


I managed to make this thing work using a simple batch file. I did do some basic tests, and my results are very promising so far.

@ECHO off
REM ************************************************************************************
REM "REAL" One-Pass-VBR for HCEnc without the need for prediction
REM
REM ************************************************************************************


%SYSTEMROOT%\system32\FIND "TOTAL_FRAMES" /i HC_OPV.ini >temp.ini
FOR /f "tokens=1,2 delims==" %%a IN (temp.ini) DO SET TOTAL_FRAMES=%%b
%SYSTEMROOT%\system32\FIND "TARGET_SIZE" /i HC_OPV.ini >temp.ini
FOR /f "tokens=1,2 delims==" %%a IN (temp.ini) DO SET TARGET_SIZE=%%b
%SYSTEMROOT%\system32\FIND "AUDIO_SIZE" /i HC_OPV.ini >temp.ini
FOR /f "tokens=1,2 delims==" %%a IN (temp.ini) DO SET AUDIO_SIZE=%%b
%SYSTEMROOT%\system32\FIND "VERY_LOW_BR" /i HC_OPV.ini >temp.ini
FOR /f "tokens=1,2 delims==" %%a IN (temp.ini) DO SET VERY_LOW_BR=%%b
%SYSTEMROOT%\system32\FIND "INFILE" /i HC_OPV.ini >temp.ini
FOR /f "tokens=1,2 delims==" %%a IN (temp.ini) DO SET INFILE=%%b
%SYSTEMROOT%\system32\FIND "OUTFILE" /i HC_OPV.ini >temp.ini
FOR /f "tokens=1,2 delims==" %%a IN (temp.ini) DO SET OUTFILE=%%b
IF EXIST temp.ini DEL temp.ini
CALL :GetPath "%OUTFILE%"
SET WORK_FOLDER=%$PATH%


SET SEGMENTS=200
SET SKIP_PERCENT=3
SET MAX_ADJUST=0.30
SET INITIAL_Q=3.50
IF !%VERY_LOW_BR%==!1 SET INITIAL_Q=6.00
SET CLAMP_UPPER=3.50
SET CLAMP_LOWER=0
SET AGGRESSIVE_THRESHOLD=90


SET SEARCH_TEXT=enter
SET HC_DRIVE=%~d0
SET HC_PATH=%~p0
%HC_DRIVE%
CD "%HC_PATH%"
IF NOT EXIST HCenc.exe COPY /b HCenc*.exe HCenc.exe >NUL
IF NOT EXIST HC.INI COPY NUL HC.INI >NUL


SET /A AGGRESSIVE_THRESHOLD=%SEGMENTS% * %AGGRESSIVE_THRESHOLD% / 100
SET /A TARGET_SIZE=%TARGET_SIZE% * 1024
SET /A TARGET_SIZE=%TARGET_SIZE% - %AUDIO_SIZE%

REM *** Decrease Target Size by 3% ***
SET /A YYY=%TARGET_SIZE% * 30
CALL :Round_DIV %YYY% 1000
SET /A TARGET_SIZE=%TARGET_SIZE% - %RESULT%

CALL :Round_DIV %TARGET_SIZE% %SEGMENTS%
SET DESIRED_SIZE=%RESULT%
SET /A FRAMES=%TOTAL_FRAMES% / %SEGMENTS%
CALL :Round_DIV 1000 %SEGMENTS%
SET ONE_SEG=%RESULT%
SET /A SKIP_PERCENT=%SKIP_PERCENT% >NUL 2>&1
SET /A SKIP_PERCENT=%SKIP_PERCENT% * 10
CALL :Round_DIV %SKIP_PERCENT% %ONE_SEG%
SET SKIP_SEGMENTS=%RESULT%

COPY "%INFILE%" "%WORK_FOLDER%$$Encode$$.avs" >NUL
SET FIRST=0
SET LAST=%FRAMES%
>>"%WORK_FOLDER%$$Encode$$.avs" ECHO Trim(%FIRST%,%LAST%)
COPY HC.ini $$Encode$$.ini >NUL
SET CQ=%INITIAL_Q%
SET CQ=%CQ:.=%
IF %CQ% LSS 100 SET CQ=100
SET INITIAL_Q=%CQ%
SET CQ=%INITIAL_Q:~0,1%.%INITIAL_Q:~1,2%
IF %INITIAL_Q% GTR 999 SET CQ=%INITIAL_Q:~0,2%.%INITIAL_Q:~2,2%
>>$$Encode$$.ini Echo *NOSEQ_ENDCODE
>>$$Encode$$.ini Echo *CQ_MAXBITRATE %CQ%

SET MM=N/A
ECHO. | time | Find /V /I "%SEARCH_TEXT%" >"%WORK_FOLDER%HC_OPV.log"
>>"%WORK_FOLDER%HC_OPV.log" ECHO Start Encoding...
>>"%WORK_FOLDER%HC_OPV.log" ECHO.
>>"%WORK_FOLDER%HC_OPV.log" ECHO Segment #1 Mismatch: %MM% CQ = %CQ%
ECHO.
ECHO Encoding Segment #1 of %SEGMENTS%
ECHO.
ECHO Current Quantizer: %CQ%
ECHO.
ECHO Overall Progress: 0%% done

IF !%MAX_ADJUST% == ! SET MAX_ADJUST=0.00
IF %MAX_ADJUST% == 0 SET MAX_ADJUST=0.00
SET MAX_ADJUST=%MAX_ADJUST:.=%
IF %MAX_ADJUST:~0,1% == 0 SET Max_ADJUST=%MAX_ADJUST:~1,2%
IF %MAX_ADJUST:~0,1% == 0 SET Max_ADJUST=%MAX_ADJUST:~1,1%

IF !%CLAMP_UPPER% == ! SET CLAMP_UPPER=0.00
IF %CLAMP_UPPER% == 0 SET CLAMP_UPPER=0.00
SET CLAMP_UPPER=%CLAMP_UPPER:.=%
IF %CLAMP_UPPER:~0,1% == 0 SET CLAMP_UPPER=%CLAMP_UPPER:~1,2%
IF %CLAMP_UPPER:~0,1% == 0 SET CLAMP_UPPER=%CLAMP_UPPER:~1,1%
IF NOT %CLAMP_UPPER% == 0 SET /A CLAMP_UPPER=%INITIAL_Q% + %CLAMP_UPPER%
IF !%CLAMP_LOWER% == ! SET CLAMP_LOWER=0.00
IF %CLAMP_LOWER% == 0 SET CLAMP_LOWER=0.00
SET CLAMP_LOWER=%CLAMP_LOWER:.=%
IF %CLAMP_LOWER:~0,1% == 0 SET CLAMP_LOWER=%CLAMP_LOWER:~1,2%
IF %CLAMP_LOWER:~0,1% == 0 SET CLAMP_LOWER=%CLAMP_LOWER:~1,1%
IF NOT %CLAMP_LOWER% == 0 SET /A CLAMP_LOWER=%INITIAL_Q% - %CLAMP_LOWER%
IF %CLAMP_LOWER% LSS 100 SET CLAMP_LOWER=100

SET COUNTER=0
SET JOINED_SIZE=0
SET LAST_MM=0
SET MM_DIFF=0


REM ************************************************************************************

:Loop
SET /A COUNTER=%COUNTER% +1
SET /A NAME=1000 + %COUNTER%
HCEnc.exe -i "%WORK_FOLDER%$$Encode$$.avs" -o "%WORK_FOLDER%%NAME%.$$$" -ini $$Encode$$.ini
IF ERRORLEVEL 1 GOTO Abort
IF %LAST%==0 GOTO Join

CALL :GetSize "%WORK_FOLDER%%NAME%.$$$"
CALL :Round_DIV %SIZE% 1024
SET SIZE=%RESULT%
SET /A JOINED_SIZE=%JOINED_SIZE% + %SIZE%
SET /A REF_SIZE=%DESIRED_SIZE% * %COUNTER%
SET /A YYY=%JOINED_SIZE% * 100
CALL :Round_DIV %YYY% %REF_SIZE%
SET MISMATCH=%RESULT%
SET /A MM=%MISMATCH% - 100
IF %LAST_MM% GTR 0 SET /A MM_DIFF=%LAST_MM% - %MM%
IF %LAST_MM% LSS 0 SET /A MM_DIFF=%MM% - %LAST_MM%
IF NOT %LAST_MM% == 0 IF %MM_DIFF% GTR 0 GOTO DISCARD
IF %COUNTER% LSS %SKIP_SEGMENTS% GOTO Discard
SET CQ=%CQ:.=%
SET /A YYY=%CQ% * %MISMATCH%
CALL :ROUND_DIV %YYY% 100
SET NEW_CQ=%RESULT%
SET /A XXX=%SEGMENTS% / 2
IF %AGGRESSIVE_THRESHOLD% LEQ %XXX% SET XXX=1
IF %COUNTER% LSS %XXX% GOTO Normal
SET /A YYY=%NEW_CQ% * %MISMATCH%
CALL :ROUND_DIV %YYY% 100
SET NEW_CQ=%RESULT%
SET /A YYY=%NEW_CQ% * %MISMATCH%
CALL :ROUND_DIV %YYY% 100
SET NEW_CQ=%RESULT%
IF %COUNTER% LSS %AGGRESSIVE_THRESHOLD% GOTO Normal

:Aggressive
CALL :Round_DIV %REF_SIZE% 10
SET XXX=%RESULT%
SET /A YYY=%JOINED_SIZE% * 100
CALL :Round_DIV %YYY% %XXX%
SET MISMATCH=%RESULT%
SET /A MISMATCH=%MISMATCH% - 900
SET /A YYY=%CQ% * %MISMATCH%
CALL :ROUND_DIV %YYY% 100
SET NEW_CQ=%RESULT%
SET MAX_ADJUST=0
SET CLAMP_UPPER=800
IF %INITIAL_Q% GEQ 600 SET CLAMP_UPPER=0
SET CLAMP_LOWER=0

:Normal
IF %NEW_CQ% LSS 100 SET NEW_CQ=100
SET /A MAX_BIGGER=%CQ% + %MAX_ADJUST%
SET /A MAX_SMALLER=%CQ% - %MAX_ADJUST%
IF NOT %CLAMP_UPPER% == 0 IF %NEW_CQ% GTR %CLAMP_UPPER% SET NEW_CQ=%CLAMP_UPPER%
IF NOT %MAX_ADJUST% == 0 IF %NEW_CQ% GTR %MAX_BIGGER% SET NEW_CQ=%MAX_BIGGER%
IF NOT %CLAMP_LOWER% == 0 IF %NEW_CQ% LSS %CLAMP_LOWER% SET NEW_CQ=%CLAMP_LOWER%
IF NOT %MAX_ADJUST% == 0 IF %NEW_CQ% LSS %MAX_SMALLER% SET NEW_CQ=%MAX_SMALLER%
IF %NEW_CQ% LSS 100 SET NEW_CQ=100
SET CQ=%NEW_CQ:~0,1%.%NEW_CQ:~1,2%
IF %NEW_CQ% GTR 999 SET CQ=%NEW_CQ:~0,2%.%NEW_CQ:~2,2%

:Discard
SET LAST_MM=%MM%
COPY "%INFILE%" "%WORK_FOLDER%$$Encode$$.avs" >NUL
SET /A FIRST=%LAST% + 1
SET /A LAST=%FIRST% + %FRAMES%
IF %LAST% GTR %TOTAL_FRAMES% SET LAST=0
>>"%WORK_FOLDER%$$Encode$$.avs" ECHO Trim(%FIRST%,%LAST%)
COPY HC.ini $$Encode$$.ini >NUL
IF NOT %LAST% == 0 Echo *NOSEQ_ENDCODE >>$$Encode$$.ini
>>$$Encode$$.ini Echo *CQ_MAXBITRATE %CQ%
SET /A SEG=%COUNTER% +1

IF %MM% GEQ 1 SET MM=+%MM%
IF %MM% == 0 SET MM= %MM%
>>"%WORK_FOLDER%HC_OPV.log" ECHO Segment #%SEG% Mismatch: %MM%%% CQ = %CQ%
SET /A YYY=(%ONE_SEG% * %COUNTER%) / 10
CLS
ECHO.
ECHO Encoding Segment #%SEG% of %SEGMENTS%
ECHO.
ECHO Accumulated Size Mismatch: %MM%%%
ECHO.
ECHO Current Quantizer: %CQ%
ECHO.
ECHO Overall Progress: %YYY%%% done

GOTO Loop


REM ************************************************************************************

:Abort
>>"%WORK_FOLDER%HC_OPV.log" ECHO.
ECHO. | time | Find /V /I "%SEARCH_TEXT%" >>"%WORK_FOLDER%HC_OPV.log"
>>"%WORK_FOLDER%HC_OPV.log" ECHO User Abort...
>>"%WORK_FOLDER%HC_OPV.log" ECHO.
GOTO Cleanup

:Join
>>"%WORK_FOLDER%HC_OPV.log" ECHO.
ECHO. | time | Find /V /I "%SEARCH_TEXT%" >>"%WORK_FOLDER%HC_OPV.log"
>>"%WORK_FOLDER%HC_OPV.log" ECHO Start Joining Segments...
CLS
ECHO.
ECHO Joining Segments
ECHO.
ECHO Please wait...
IF EXIST "%OUTFILE%" DEL "%OUTFILE%"
COPY /B "%WORK_FOLDER%*.$$$" "%OUTFILE%" >NUL
>>"%WORK_FOLDER%HC_OPV.log" ECHO.
ECHO. | time | Find /V /I "%SEARCH_TEXT%" >>"%WORK_FOLDER%HC_OPV.log"
>>"%WORK_FOLDER%HC_OPV.log" ECHO Finished...

:Cleanup
IF EXIST "%WORK_FOLDER%*.$$$" DEL "%WORK_FOLDER%*.$$$"
IF EXIST $$Encode$$.ini DEL $$Encode$$.ini
IF EXIST "%WORK_FOLDER%$$Encode$$.avs" DEL "%WORK_FOLDER%$$Encode$$.avs"
EXIT


REM ************************************************************************************

:Round_DIV
SET /A RESULT=%1 / %2
SET /A XXX=%1 %% %2 * 10 / %2
IF %XXX% GEQ 5 SET /A RESULT=%RESULT% + 1
GOTO :EOF

:GetSize
SET SIZE=%~z1
GOTO :EOF

:GetPath
SET $PATH=%~dp1




Save this batch file under any name you want and copy it to your HC folder. I call it "AdaptiveOPV.bat" just because I have not found a better name so far.


The "HowTo" is in the next post

Cheers
manolito



//Edit:

The batch file is just there to give you the opportunity to analyze the algorithm. For really using the software please download the complete package under
http://scifi.pages.at/manolito/HC_OPV/HC_OPV.zip

Also please have a look here: http://forum.doom9.org/showthread.php?p=1165481#post1165481

[manolito]

Requirements:

HCEnc 0.23 (earlier versions should work)

The HC executable must be "HCEnc.exe". The batch file will take care of this.

The INI file for HC must be "HC.ini". The batch file only passes input and output files to HC via command line, so all other settings have to be made via HC.ini.

IMPORTANT: HC.ini MUST have an entry for Max Bitrate. But it must NOT have an entry for CQ_MaxBitrate.

The batch file uses sizes for its calculations instead of bitrates. Before encoding you need to know the combined size of your audio tracks.

You also need to know the total number of frames of your source. Load the AVS into VDub, under "File / File Information" you can get the frame count.


********************************

Before you can start encoding you have to edit the batch file. All "SET" entries in the upper section of the file must be edited according to your needs.

The entries for input and output files and the entries for your folders are self explaining. Remember to not use quotes.

The variables which are most interesting are:
SEGMENTS=
SKIP_PERCENT=
MAX_ADJUST=
INITIAL_Q=


The SEGMENTS entry specifies into how many segments the encode should be divided. More segments mean that bitrate distribution will get closer to CBR while fewer segments might lead to not reaching the target size. Default is 200.

SKIP_PERCENT is there due to the fact that for most movies the first couple of minutes are not representative for the whole movie (studio logos, start credits). You can specify a percentage at the start of the movie which will not be used for CQ calculation. Default is 3.

MAX_ADJUST specifies the maximum CQ adjustment between segments. Default is 0.50.

INITIAL_Q specifies the CQ value which will be used for the first segment. Values between 2.00 and 4.00 seem to be sensible. Default is 4.00.

********************************************************************

The batch file creates a simple log file in the work folder which can be helpful to compare CQ jumps.

I do intend to implement this method (if it proves to be useful) for DVD2SVCD. I believe that DVD-Rebuilder would not profit from this method because it already does segmented encodes. And for software like FAVC and AVStoDVD the authors will certainly know how to integrate this method into their software :)


Enjoy
manolito

[45tripp]

While everyone else is busy watching soccer games

you're missing all the action!


SKIP_PERCENT is there due to the fact that for most movies the first couple of minutes are not representative for the whole movie (studio logos, start credits). You can specify a percentage at the start of the movie which will not be used for CQ calculation.

reading that, i'd interpret as: skip percentage sets segment 1,
which it doesn't.


I gave this a trial.
didn't work well for me.
used an animation clip i'd tested recently.
directshowsource().lanczosresize()


SET TOTAL_FRAMES=33680
SET TARGET_SIZE=480
SET AUDIO_SIZE=0
SET SEGMENTS=20
SET SKIP_PERCENT=5
SET MAX_ADJUST=0
SET INITIAL_Q=3.00



Segment #1 CQ = 3.00
Segment #2 CQ = 7.41
Segment #3 CQ = 1.52
Segment #4 CQ = 3.31
Segment #5 CQ = 6.98
Segment #6 CQ = 1.28
Segment #7 CQ = 2.44
Segment #8 CQ = 4.53
Segment #9 CQ = 8.06
Segment #10 CQ = 1.34
Segment #11 CQ = 2.33
Segment #12 CQ = 4.03
Segment #13 CQ = 6.81
Segment #14 CQ = 1.10
Segment #15 CQ = 1.83
Segment #16 CQ = 3.12
Segment #17 CQ = 5.30
Segment #18 CQ = 8.79
Segment #19 CQ = 1.42
Segment #20 CQ = 2.34


output zize: 772MB

[manolito]

Hi Immersion,

thanks for testing. For such a small size 20 segments will not be enough. Use 100 or more. And the method is really meant for DVD conversions.

SKIP_PERCENT=5 means that for the first 5% of the movie no calculation for a new CQ value will be done. In your case 20 segments mean that each segment will be equal to 5% of the movie. So the setting has no effect. If you had entered a value of 10 for SKIP_PERCENT then the first two segments would have been encoded with your initial CQ of 3.00, the third segment would have a different CQ.


I just edited the first post for an update of the batch file. I added correct rounding for the integer divisions (Batch files only do integer math, so a result of 1.9 would come out as 1). I am not sure if it helps at all since all the calculations are done in MB (instead of KB or Bytes). I could switch to KB, but using Bytes does not work, I get wraparounds.

Cheers
manolito

[MrC]

Thanks for the idea and for its development. Speed up mpeg2 encoding using HCenc (which is fast by itself) is for sure a good improvement.

What are exactly the goals of progressive VBR vs predictive VBR encoding? Size targeting? Quality? Both?

;)

MrC

[manolito]

Size Targeting !

Prediction is never a very accurate thing, especially for HC. For a high bitrate encode (where max bitrate and average bitrate are not too far apart) HCEnc is constantly raising the specified CQ value to avoid overshooting the max bitrate. This can make prediction a real gamble. More than once it happened to me that a prediction run with a higher CQ than the previous one resulted in a higher file size. It should have been the opposite, though.

Hopefully my new method will be able to always reach the target size while maintaining a good bitrate distribution at the same time. Needs more tweaking, though...

Cheers
manolito

[45tripp]

SKIP_PERCENT=5 means that for the first 5% of the movie no calculation for a new CQ value will be done. In your case 20 segments mean that each segment will be equal to 5% of the movie. So the setting has no effect.

i realised.
i was saying that reading the description, i'd intrerpret differently.


thanks for testing. For such a small size 20 segments will not be enough. Use 100 or more. And the method is really meant for DVD conversions.

meaning?
you want durations of 80min and upwards? :)

anyway,
i tried again with the same clip,
cause i'm not really encoding anything,
and this is the longest sample i have.
and i'm not going to pull a dvd, for the sake of it.


SET TOTAL_FRAMES=33680
SET TARGET_SIZE=480
SET AUDIO_SIZE=0
SET SEGMENTS=100
SET SKIP_PERCENT=9.9
SET MAX_ADJUST=0.00
SET INITIAL_Q=3.00



Segment #1 CQ = 3.00
Segment #2 CQ = 3.00
Segment #3 CQ = 3.00
Segment #4 CQ = 3.00
Segment #5 CQ = 3.00
Segment #6 CQ = 3.00
Segment #7 CQ = 3.00
Segment #8 CQ = 3.00
Segment #9 CQ = 3.00
Segment #10 CQ = 3.00
Segment #11 CQ = 7.02
Segment #12 CQ = 1.60
Segment #13 CQ = 3.68
Segment #14 CQ = 8.32
Segment #15 CQ = 1.80
Segment #16 CQ = 3.87
Segment #17 CQ = 7.97
Segment #18 CQ = 1.59
Segment #19 CQ = 3.21
Segment #20 CQ = 6.48
Segment #21 CQ = 1.27
Segment #22 CQ = 2.50
Segment #23 CQ = 4.95
Segment #24 CQ = 9.55
Segment #25 CQ = 1.78
Segment #26 CQ = 3.33
Segment #27 CQ = 6.13
Segment #28 CQ = 1.09
Segment #29 CQ = 1.97
Segment #30 CQ = 3.57
Segment #31 CQ = 6.39
Segment #32 CQ = 1.11
Segment #33 CQ = 1.95
Segment #34 CQ = 3.47
Segment #35 CQ = 6.11
Segment #36 CQ = 1.05
Segment #37 CQ = 1.84
Segment #38 CQ = 3.24
Segment #39 CQ = 5.64
Segment #40 CQ = 9.76
Segment #41 CQ = 1.66
Segment #42 CQ = 2.86
Segment #43 CQ = 4.89
Segment #44 CQ = 8.26
Segment #45 CQ = 1.38
Segment #46 CQ = 2.33
Segment #47 CQ = 3.94
Segment #48 CQ = 6.62
Segment #49 CQ = 1.09
Segment #50 CQ = 1.82
Segment #51 CQ = 3.06
Segment #52 CQ = 5.11
Segment #53 CQ = 8.43
Segment #54 CQ = 1.38
Segment #55 CQ = 2.28
Segment #56 CQ = 3.76
Segment #57 CQ = 6.17
Segment #58 CQ = 1.00
Segment #59 CQ = 1.64
Segment #60 CQ = 2.71
Segment #61 CQ = 4.47
Segment #62 CQ = 7.33
Segment #63 CQ = 1.18
Segment #64 CQ = 1.92
Segment #65 CQ = 3.15
Segment #66 CQ = 5.13
Segment #67 CQ = 8.31
Segment #68 CQ = 1.33
Segment #69 CQ = 2.13
Segment #70 CQ = 3.43
Segment #71 CQ = 5.52
Segment #72 CQ = 8.94
Segment #73 CQ = 1.43
Segment #74 CQ = 2.30
Segment #75 CQ = 3.73
Segment #76 CQ = 6.01
Segment #77 CQ = 9.62
Segment #78 CQ = 1.53
Segment #79 CQ = 2.45
Segment #80 CQ = 3.92
Segment #81 CQ = 6.23
Segment #82 CQ = 9.78
Segment #83 CQ = 1.52
Segment #84 CQ = 2.39
Segment #85 CQ = 3.73
Segment #86 CQ = 5.82
Segment #87 CQ = 9.08
Segment #88 CQ = 1.41
Segment #89 CQ = 2.21
Segment #90 CQ = 3.47
Segment #91 CQ = 5.45
Segment #92 CQ = 8.56
Segment #93 CQ = 1.33
Segment #94 CQ = 2.07
Segment #95 CQ = 3.25
Segment #96 CQ = 5.07
Segment #97 CQ = 7.86
Segment #98 CQ = 1.21
Segment #99 CQ = 1.88
Segment #100 CQ = 2.91


output= 774MB

and..


SET TOTAL_FRAMES=33680
SET TARGET_SIZE=480
SET AUDIO_SIZE=0
SET SEGMENTS=20
SET SKIP_PERCENT=8
SET MAX_ADJUST=0.50
SET INITIAL_Q=3.50



Segment #1 CQ = 3.50
Segment #2 CQ = 3.50
Segment #3 CQ = 4.00
Segment #4 CQ = 4.50
Segment #5 CQ = 5.00
Segment #6 CQ = 5.50
Segment #7 CQ = 6.00
Segment #8 CQ = 6.50
Segment #9 CQ = 7.00
Segment #10 CQ = 7.50
Segment #11 CQ = 8.00
Segment #12 CQ = 8.50
Segment #13 CQ = 9.00
Segment #14 CQ = 9.50
Segment #15 CQ = 1.00
Segment #16 CQ = 1.18
Segment #17 CQ = 1.48
Segment #18 CQ = 1.92
Segment #19 CQ = 2.42
Segment #20 CQ = 2.92


output= 659MB

[MrC]

[...] HCEnc is constantly raising the specified CQ value to avoid overshooting the max bitrate. This can make prediction a real gamble. [...]

You are right. In addition, when raising the Q value near maxbirate, HCenc becomes really slow, even if quality is assured.
That's why, in AVStoDVD, I suggest to use QuEnc's CBR mode when output DVD bitrate is more than 6000 kbps.

Lookin' forward to see your routine development.

:)

Bye

[manolito]

I just updated the batch file and the HowTo. Added new variables "CLAMP_UPPER" and "CLAMP_LOWER" which can restrict the CQ range plus a lot of bug fixing.

So far I get good results for DVD to DVD conversions using the following settings:

SEGMENTS=100
MAX_ADJUST=0.20
SKIP_PERCENT=5
INITIAL_Q between 2 and 4 depending on the movie characteristics


My latest idea is to combine this method with prediction to get the best of two worlds. ATM I can only implement this for DVD2SVCD because I already have a working prediction routine for DVD2SVCD (based on GetCQ by Amnon).


Here is the procedure I am testing right now:

1. Make a quick and dirty CQ prediction using 0.5% sample size and limiting the number of sample encodes to 4 or 5.

2. Use this CQ value as the INITIAL_Q in my routine. Set SKIP_PERCENT to 40 or 50 and set SEGMENTS to 100 or even 200. MAX_ADJUST is 0.10, CLAMP_UPPER and CLAMP_LOWER are both set to 0.50.

This way a big portion of the movie is encoded with a constant CQ, and for the other portion of the movie the CQ range is highly restricted while still (hopefully) reaching the exact target size.


I will keep you posted about my test results...

Cheers
manolito

[Fluffbutt]

Sorry if I'm missing something here, but wouldn't this fluctuation of Q make the movie have fluctuating quality when watched - you could end up with very blocky scenes on a car chase, with near perfect video of the two heroes looking out over a sunset near the end..??

Also it'll mean that the movie starts off with poor quality, slowly getting better and better (with jumps between 'qualitiys').

It's a pity we can't do a sort of 'pre-analysis' of the next, say, 1000 frames for action amounts (like 2 pass does, but in one pass), and increase the bitrate for them, then lower it for the sooky chick-flick bits.

[GodofaGap]

Sorry if I'm missing something here, but wouldn't this fluctuation of Q make the movie have fluctuating quality when watched - you could end up with very blocky scenes on a car chase, with near perfect video of the two heroes looking out over a sunset near the end..??
This is inherent to 1-pass ABR, yes. If you want constant quality with 1-pass you can't have a predictable filesize at the same time.


Also it'll mean that the movie starts off with poor quality, slowly getting better and better (with jumps between 'qualitiys').
It doesn't really mean that. The movie starts with whatever quality you want it and the mechanism will correct to worse or better quality according to what bitrate that quality requires.

[manolito]

@Fluffbutt

Sorry I cannot change the laws of physics, so 1-pass VBR without prediction will always be a tradeoff between two extremes:

1. Use many segments and do not restrict CQ changes between segments.

This way you will hit the target size with high precision, but bitrate distribution will look a lot like CBR. And you will also get noticeable jumps in quality between segments.


2. Use only few segments and restrict the CQ range by specifying a maximum value by which CQ can change between segments. Plus put a clamp around CQ so it can never go above and below a specified range.

This way you will get almost constant quality, but chances are you will not be able to hit the target size.


I want to find the best (sorry, forbidden here, replace by "optimal") compromise between these two extremes. I use 1-pass VBR with prediction a lot, but it just happens to me too many times that final size is not where I want it to be.


Cheers
manolito

[manolito]

Some stupid bugs fixed, please redownload...

Cheers
manolito

[45tripp]

still using the same clip.


SET TOTAL_FRAMES=33680
SET TARGET_SIZE=480
SET AUDIO_SIZE=10
SET SEGMENTS=50
SET SKIP_PERCENT=5
SET MAX_ADJUST=0.50
SET INITIAL_Q=3.50
SET CLAMP_UPPER=0
SET CLAMP_LOWER=0


Segment #1 CQ = 3.50
Segment #2 CQ = 3.50
Segment #3 CQ = 3.50
Segment #4 CQ = 4.00
Segment #5 CQ = 4.50
Segment #6 CQ = 5.00
Segment #7 CQ = 5.50
Segment #8 CQ = 6.00
Segment #9 CQ = 6.50
Segment #10 CQ = 7.00
Segment #11 CQ = 7.50
Segment #12 CQ = 8.00
Segment #13 CQ = 8.50
Segment #14 CQ = 9.00
Segment #15 CQ = 9.50
Segment #16 CQ = 10.00
Segment #17 CQ = 10.50
Segment #18 CQ = 11.00
Segment #19 CQ = 11.50
Segment #20 CQ = 12.00
Segment #21 CQ = 12.50
Segment #22 CQ = 13.00
Segment #23 CQ = 13.50
Segment #24 CQ = 14.00
Segment #25 CQ = 14.50
Segment #26 CQ = 14.79
Segment #27 CQ = 14.79
Segment #28 CQ = 14.35
Segment #29 CQ = 13.85
Segment #30 CQ = 13.35
Segment #31 CQ = 12.85
Segment #32 CQ = 12.35
Segment #33 CQ = 11.85
Segment #34 CQ = 11.35
Segment #35 CQ = 10.85
Segment #36 CQ = 10.35
Segment #37 CQ = 9.85
Segment #38 CQ = 9.35
Segment #39 CQ = 8.85
Segment #40 CQ = 8.35
Segment #41 CQ = 7.85
Segment #42 CQ = 7.35
Segment #43 CQ = 6.85
Segment #44 CQ = 6.35
Segment #45 CQ = 5.85
Segment #46 CQ = 5.35
Segment #47 CQ = 4.85
Segment #48 CQ = 4.35
Segment #49 CQ = 3.85
Segment #50 CQ = 3.39


result 416 MB
undersized


also tried again with 20 segments and no max_adjust


SET TOTAL_FRAMES=33680
SET TARGET_SIZE=480
SET AUDIO_SIZE=0
SET SEGMENTS=20
SET SKIP_PERCENT=5
SET MAX_ADJUST=0.00
SET INITIAL_Q=3.50
SET CLAMP_UPPER=0
SET CLAMP_LOWER=0


Segment #1 CQ = 3.50
Segment #2 CQ = 8.19
Segment #3 CQ = 15.72
Segment #4 CQ = 22.64
Segment #5 CQ = 26.04
Segment #6 CQ = 25.26
Segment #7 CQ = 21.22
Segment #8 CQ = 15.92
Segment #9 CQ = 11.30
Segment #10 CQ = 7.80
Segment #11 CQ = 5.46
Segment #12 CQ = 3.93
Segment #13 CQ = 3.03
Segment #14 CQ = 2.45
Segment #15 CQ = 2.11
Segment #16 CQ = 1.98
Segment #17 CQ = 2.00
Segment #18 CQ = 2.14
Segment #19 CQ = 2.42
Segment #20 CQ = 2.78


result 549 MB
oversized
and i don;t like at all the high CQ spikes and the way 1/3 of the encode is with CQs lower than initial CQ.

[Fluffbutt]

@Fluffbutt

Sorry I cannot change the laws of physics, so 1-pass VBR without prediction will always be a tradeoff between two extremes:

<snip>

manolito

Sorry if that came across as a complaint - it wasn't meant to be at all.

I love any form of prediction type workings - very helpful.

Thanks for the tool, and sorry again if it came out wrong..

[manolito]

OK guys, this is the final version for this thread.

After more than a week of testing the default values in the batch file represent the optimal settings I could find for DVD to DVD or AVI to DVD conversions.

SEGMENTS=200 works best for me, but target size has to be decreased by 5% in this case (Probably accumulation of rounding errors).

MAX_ADJUST is 0.50.

INITIAL_Q is 4.00, SKIP_PERCENT is 3.

I removed the CLAMP parameter since it is only useful if your INITIAL_Q is pretty close to the final average Q, and without prediction this is almost impossible.


With these settings I always reach a final size which ImgBurn reports as 99% fill rate. And most importantly I really like the visual quality of these encodes.

My next project will be to transparently integrate this method into DVD2SVCD. So I am outta here for now...


Thanks everybody for the comments and for testing.

Cheers
manolito

[MrC]

Hi manolito,

why not creating a ActiveX DLL to be used directly with Visual Studio's languages (VB, VC, etc.)?

Programs like AVStoDVD or FAVC or DVDForger could implement it with great ease.

:)

Bye

[Darksoul71]

@Manolito & MrC:
Hi there,

haven´t had a chance to take a closer look at this but may be I could help out with some AutoIt here ?!?
Currently I have no time to "analyze" the algorithms this batch file use.

@Manolito:
Can you enlighten me how you calculate the CQ value for the next segment based on the size of the current
segment ? The rest would be simple since all this segment stuff is already done by HCEnc^n.

Later,
D$

[45tripp]

200 segments?
that's rather slow.


Here is the procedure I am testing right now:

1. Make a quick and dirty CQ prediction using 0.5% sample size and limiting the number of sample encodes to 4 or 5.

2. Use this CQ value as the INITIAL_Q in my routine. Set SKIP_PERCENT to 40 or 50 and set SEGMENTS to 100 or even 200. MAX_ADJUST is 0.10, CLAMP_UPPER and CLAMP_LOWER are both set to 0.50.

This way a big portion of the movie is encoded with a constant CQ, and for the other portion of the movie the CQ range is highly restricted while still (hopefully) reaching the exact target size.


i liked this idea.
woudn;t it in fact be faster than 200 segments and result in better or at least more evenly distributed quality?

anyway,
as i tested the other batches,
same clip:


SET TOTAL_FRAMES=33680
SET TARGET_SIZE=480
SET AUDIO_SIZE=0
SET SEGMENTS=200
SET SKIP_PERCENT=3
SET MAX_ADJUST=1
SET INITIAL_Q=4.00



Segment #1 CQ = 4.00
Segment #2 CQ = 4.00
Segment #3 CQ = 4.00
Segment #4 CQ = 4.00
Segment #5 CQ = 4.00
Segment #6 CQ = 4.00
Segment #7 CQ = 4.01
Segment #8 CQ = 4.02
Segment #9 CQ = 4.03
Segment #10 CQ = 4.04
Segment #11 CQ = 4.05
Segment #12 CQ = 4.06
Segment #13 CQ = 4.07
Segment #14 CQ = 4.08
Segment #15 CQ = 4.09
Segment #16 CQ = 4.10
Segment #17 CQ = 4.11
Segment #18 CQ = 4.12
Segment #19 CQ = 4.13
Segment #20 CQ = 4.14
Segment #21 CQ = 4.15
Segment #22 CQ = 4.16
Segment #23 CQ = 4.17
Segment #24 CQ = 4.18
Segment #25 CQ = 4.19
Segment #26 CQ = 4.20
Segment #27 CQ = 4.21
Segment #28 CQ = 4.22
Segment #29 CQ = 4.23
Segment #30 CQ = 4.24
Segment #31 CQ = 4.25
Segment #32 CQ = 4.26
Segment #33 CQ = 4.27
Segment #34 CQ = 4.28
Segment #35 CQ = 4.29
Segment #36 CQ = 4.30
Segment #37 CQ = 4.31
Segment #38 CQ = 4.32
Segment #39 CQ = 4.33
Segment #40 CQ = 4.34
Segment #41 CQ = 4.35
Segment #42 CQ = 4.55
Segment #43 CQ = 4.75
Segment #44 CQ = 4.95
Segment #45 CQ = 5.15
Segment #46 CQ = 5.35
Segment #47 CQ = 5.55
Segment #48 CQ = 5.75
Segment #49 CQ = 5.95
Segment #50 CQ = 6.15
Segment #51 CQ = 6.35
Segment #52 CQ = 6.55
Segment #53 CQ = 6.75
Segment #54 CQ = 6.95
Segment #55 CQ = 7.15
Segment #56 CQ = 7.35
Segment #57 CQ = 7.55
Segment #58 CQ = 7.75
Segment #59 CQ = 7.95
Segment #60 CQ = 8.15
Segment #61 CQ = 8.35
Segment #62 CQ = 8.55
Segment #63 CQ = 8.75
Segment #64 CQ = 8.95
Segment #65 CQ = 9.15
Segment #66 CQ = 9.35
Segment #67 CQ = 9.55
Segment #68 CQ = 9.75
Segment #69 CQ = 9.95
Segment #70 CQ = 10.15
Segment #71 CQ = 10.35
Segment #72 CQ = 10.55
Segment #73 CQ = 10.75
Segment #74 CQ = 10.95
Segment #75 CQ = 11.15
Segment #76 CQ = 11.35
Segment #77 CQ = 11.55
Segment #78 CQ = 11.75
Segment #79 CQ = 11.95
Segment #80 CQ = 12.15
Segment #81 CQ = 12.35
Segment #82 CQ = 12.55
Segment #83 CQ = 12.75
Segment #84 CQ = 12.95
Segment #85 CQ = 13.15
Segment #86 CQ = 13.35
Segment #87 CQ = 13.55
Segment #88 CQ = 13.75
Segment #89 CQ = 13.95
Segment #90 CQ = 14.15
Segment #91 CQ = 14.35
Segment #92 CQ = 14.55
Segment #93 CQ = 14.75
Segment #94 CQ = 14.95
Segment #95 CQ = 15.15
Segment #96 CQ = 15.35
Segment #97 CQ = 15.55
Segment #98 CQ = 15.75
Segment #99 CQ = 15.95
Segment #100 CQ = 16.15
Segment #101 CQ = 16.35
Segment #102 CQ = 16.55
Segment #103 CQ = 16.75
Segment #104 CQ = 16.95
Segment #105 CQ = 17.15
Segment #106 CQ = 17.35
Segment #107 CQ = 17.55
Segment #108 CQ = 17.75
Segment #109 CQ = 17.95
Segment #110 CQ = 18.15
Segment #111 CQ = 18.35
Segment #112 CQ = 18.55
Segment #113 CQ = 18.75
Segment #114 CQ = 18.95
Segment #115 CQ = 19.15
Segment #116 CQ = 19.35
Segment #117 CQ = 19.55
Segment #118 CQ = 19.75
Segment #119 CQ = 19.95
Segment #120 CQ = 20.15
Segment #121 CQ = 20.35
Segment #122 CQ = 20.55
Segment #123 CQ = 20.55
Segment #124 CQ = 20.35
Segment #125 CQ = 20.15
Segment #126 CQ = 19.95
Segment #127 CQ = 19.75
Segment #128 CQ = 19.55
Segment #129 CQ = 19.35
Segment #130 CQ = 19.15
Segment #131 CQ = 18.95
Segment #132 CQ = 18.75
Segment #133 CQ = 18.55
Segment #134 CQ = 18.35
Segment #135 CQ = 18.15
Segment #136 CQ = 17.95
Segment #137 CQ = 17.75
Segment #138 CQ = 17.55
Segment #139 CQ = 17.35
Segment #140 CQ = 17.15
Segment #141 CQ = 16.95
Segment #142 CQ = 16.75
Segment #143 CQ = 16.55
Segment #144 CQ = 16.35
Segment #145 CQ = 16.15
Segment #146 CQ = 15.95
Segment #147 CQ = 15.75
Segment #148 CQ = 15.55
Segment #149 CQ = 15.35
Segment #150 CQ = 15.15
Segment #151 CQ = 14.95
Segment #152 CQ = 14.75
Segment #153 CQ = 14.55
Segment #154 CQ = 14.35
Segment #155 CQ = 14.15
Segment #156 CQ = 13.95
Segment #157 CQ = 13.75
Segment #158 CQ = 13.55
Segment #159 CQ = 13.35
Segment #160 CQ = 13.15
Segment #161 CQ = 12.95
Segment #162 CQ = 12.75
Segment #163 CQ = 12.55
Segment #164 CQ = 12.35
Segment #165 CQ = 12.15
Segment #166 CQ = 11.95
Segment #167 CQ = 11.75
Segment #168 CQ = 11.55
Segment #169 CQ = 11.35
Segment #170 CQ = 11.15
Segment #171 CQ = 10.95
Segment #172 CQ = 10.75
Segment #173 CQ = 10.55
Segment #174 CQ = 10.35
Segment #175 CQ = 10.15
Segment #176 CQ = 9.95
Segment #177 CQ = 9.75
Segment #178 CQ = 9.55
Segment #179 CQ = 9.35
Segment #180 CQ = 9.15
Segment #181 CQ = 8.95
Segment #182 CQ = 8.75
Segment #183 CQ = 8.55
Segment #184 CQ = 8.35
Segment #185 CQ = 8.15
Segment #186 CQ = 7.95
Segment #187 CQ = 7.75
Segment #188 CQ = 7.55
Segment #189 CQ = 7.35
Segment #190 CQ = 7.15
Segment #191 CQ = 6.95
Segment #192 CQ = 6.75
Segment #193 CQ = 6.55
Segment #194 CQ = 6.35
Segment #195 CQ = 6.15
Segment #196 CQ = 5.95
Segment #197 CQ = 5.75
Segment #198 CQ = 5.55
Segment #199 CQ = 5.35
Segment #200 CQ = 5.15


383 MB , way undersized

[manolito]

@Immersion,

you cannot use MAX_ADJUST=1, you have to enter MAX_ADJUST=1.00 instead.

I know the batch file could use a little more exception handling...:). For the MAX_ADJUST parameter I so far only catch it if you leave the entry empty or if you set it to 0. Otherwise the format has to be x.xx.

And still this method will never work well on a 480 MB clip.


i liked this idea.
woudn;t it in fact be faster than 200 segments and result in better or at least more evenly distributed quality?

Oh yes, and this is the reason why I am now mainly working on this method. I call it "Hybrid", and it really combines the best of both worlds: Constant Quality and accurate sizing.

But as I already said, I will only implement this method for DVD2SVCD. (I have no intention to develop a prediction routine using batch language...)


Anyway, I do update the batch file in this thread regularly to reflect the ongoing test results.

Cheers
manolito

[45tripp]

you cannot use MAX_ADJUST=1, you have to enter MAX_ADJUST=1.00 instead.


yeah, i figured. oversight


Oh yes, and this is the reason why I am now mainly working on this method. I call it "Hybrid", and it really combines the best of both worlds: Constant Quality and accurate sizing.


nice.



Segment #1 CQ = 4.00
Segment #2 CQ = 4.00
Segment #3 CQ = 4.00
Segment #4 CQ = 4.00
Segment #5 CQ = 4.00
Segment #6 CQ = 4.00
Segment #7 CQ = 5.00
Segment #8 CQ = 6.00
Segment #9 CQ = 7.00
Segment #10 CQ = 8.00
Segment #11 CQ = 9.00
Segment #12 CQ = 10.00
Segment #13 CQ = 11.00
Segment #14 CQ = 12.00
Segment #15 CQ = 13.00
Segment #16 CQ = 14.00
Segment #17 CQ = 15.00
Segment #18 CQ = 16.00
Segment #19 CQ = 17.00
Segment #20 CQ = 18.00
Segment #21 CQ = 19.00
Segment #22 CQ = 20.00
Segment #23 CQ = 21.00
Segment #24 CQ = 22.00
Segment #25 CQ = 23.00
Segment #26 CQ = 24.00
Segment #27 CQ = 25.00
Segment #28 CQ = 26.00
Segment #29 CQ = 27.00
Segment #30 CQ = 28.00
Segment #31 CQ = 29.00
Segment #32 CQ = 30.00
Segment #33 CQ = 30.60
Segment #34 CQ = 30.60
Segment #35 CQ = 29.99
Segment #36 CQ = 28.99
Segment #37 CQ = 27.99
Segment #38 CQ = 26.99
Segment #39 CQ = 25.99
Segment #40 CQ = 24.99
Segment #41 CQ = 23.99
Segment #42 CQ = 23.79
Segment #43 CQ = 23.59
Segment #44 CQ = 23.39
Segment #45 CQ = 23.19
Segment #46 CQ = 22.99
Segment #47 CQ = 22.79
Segment #48 CQ = 22.59
Segment #49 CQ = 22.39
Segment #50 CQ = 22.19
Segment #51 CQ = 21.99
Segment #52 CQ = 21.79
Segment #53 CQ = 21.59
Segment #54 CQ = 21.39
Segment #55 CQ = 21.19
Segment #56 CQ = 20.99
Segment #57 CQ = 20.79
Segment #58 CQ = 20.59
Segment #59 CQ = 20.39
Segment #60 CQ = 20.19
Segment #61 CQ = 19.99
Segment #62 CQ = 19.79
Segment #63 CQ = 19.59
Segment #64 CQ = 19.39
Segment #65 CQ = 19.19
Segment #66 CQ = 18.99
Segment #67 CQ = 18.79
Segment #68 CQ = 18.59
Segment #69 CQ = 18.39
Segment #70 CQ = 18.19
Segment #71 CQ = 17.99
Segment #72 CQ = 17.79
Segment #73 CQ = 17.59
Segment #74 CQ = 17.39
Segment #75 CQ = 17.19
Segment #76 CQ = 16.99
Segment #77 CQ = 16.79
Segment #78 CQ = 16.59
Segment #79 CQ = 16.39
Segment #80 CQ = 16.19
Segment #81 CQ = 15.99
Segment #82 CQ = 15.79
Segment #83 CQ = 15.59
Segment #84 CQ = 15.39
Segment #85 CQ = 15.19
Segment #86 CQ = 14.99
Segment #87 CQ = 14.79
Segment #88 CQ = 14.59
Segment #89 CQ = 14.39
Segment #90 CQ = 14.19
Segment #91 CQ = 13.99
Segment #92 CQ = 13.79
Segment #93 CQ = 13.59
Segment #94 CQ = 13.39
Segment #95 CQ = 13.19
Segment #96 CQ = 12.99
Segment #97 CQ = 12.79
Segment #98 CQ = 12.59
Segment #99 CQ = 12.39
Segment #100 CQ = 12.19
Segment #101 CQ = 11.99
Segment #102 CQ = 11.79
Segment #103 CQ = 11.59
Segment #104 CQ = 11.39
Segment #105 CQ = 11.19
Segment #106 CQ = 10.99
Segment #107 CQ = 10.79
Segment #108 CQ = 10.59
Segment #109 CQ = 10.39
Segment #110 CQ = 10.19
Segment #111 CQ = 9.99
Segment #112 CQ = 9.79
Segment #113 CQ = 9.59
Segment #114 CQ = 9.39
Segment #115 CQ = 9.19
Segment #116 CQ = 8.99
Segment #117 CQ = 8.79
Segment #118 CQ = 8.59
Segment #119 CQ = 8.39
Segment #120 CQ = 8.19
Segment #121 CQ = 7.99
Segment #122 CQ = 7.79
Segment #123 CQ = 7.59
Segment #124 CQ = 7.39
Segment #125 CQ = 7.19
Segment #126 CQ = 6.99
Segment #127 CQ = 6.79
Segment #128 CQ = 6.59
Segment #129 CQ = 6.39
Segment #130 CQ = 6.19
Segment #131 CQ = 5.99
Segment #132 CQ = 5.79
Segment #133 CQ = 5.59
Segment #134 CQ = 5.39
Segment #135 CQ = 5.19
Segment #136 CQ = 4.99
Segment #137 CQ = 4.79
Segment #138 CQ = 4.59
Segment #139 CQ = 4.39
Segment #140 CQ = 4.19
Segment #141 CQ = 3.99
Segment #142 CQ = 3.79
Segment #143 CQ = 3.59
Segment #144 CQ = 3.39
Segment #145 CQ = 3.19
Segment #146 CQ = 2.99
Segment #147 CQ = 2.79
Segment #148 CQ = 2.59
Segment #149 CQ = 2.39
Segment #150 CQ = 2.19
Segment #151 CQ = 1.99
Segment #152 CQ = 1.79
Segment #153 CQ = 1.59
Segment #154 CQ = 1.39
Segment #155 CQ = 1.19
Segment #156 CQ = 1.00
Segment #157 CQ = 1.00
Segment #158 CQ = 1.00
Segment #159 CQ = 1.00
Segment #160 CQ = 1.00
Segment #161 CQ = 1.00
Segment #162 CQ = 1.00
Segment #163 CQ = 1.00
Segment #164 CQ = 1.00
Segment #165 CQ = 1.00
Segment #166 CQ = 1.00
Segment #167 CQ = 1.00
Segment #168 CQ = 1.00
Segment #169 CQ = 1.00
Segment #170 CQ = 1.00
Segment #171 CQ = 1.00
Segment #172 CQ = 1.00
Segment #173 CQ = 1.00
Segment #174 CQ = 1.00
Segment #175 CQ = 1.00
Segment #176 CQ = 1.00
Segment #177 CQ = 1.00
Segment #178 CQ = 1.00
Segment #179 CQ = 1.00
Segment #180 CQ = 1.00
Segment #181 CQ = 1.00
Segment #182 CQ = 1.00
Segment #183 CQ = 1.00
Segment #184 CQ = 1.01
Segment #185 CQ = 1.03
Segment #186 CQ = 1.05
Segment #187 CQ = 1.08
Segment #188 CQ = 1.12
Segment #189 CQ = 1.16
Segment #190 CQ = 1.22
Segment #191 CQ = 1.29
Segment #192 CQ = 1.37
Segment #193 CQ = 1.47
Segment #194 CQ = 1.57
Segment #195 CQ = 1.70
Segment #196 CQ = 1.84
Segment #197 CQ = 2.01
Segment #198 CQ = 2.19
Segment #199 CQ = 2.39
Segment #200 CQ = 2.59


510 MB
which is the closest it's been to target,
but i really don't like the q distribution pattern.


ty

[manolito]

It looks like this project is finally finished. I did integrate the app transparently into DVD2SVCD, and the new Hybrid Mode I was talking about also came out nicely.

In the meantime the adaptive algorithm changed considerably, and here is the final incarnation of the project for everyone not using DVD2SVCD. Please download the package under:
http://scifi.pages.at/manolito/HC_OPV/HC_OPV.zip

The program parameters have been carefully tuned over more than 100 test encodes, so I removed all options for the user to change parameters. The algorithm works best for "real world" encodes. For small test clips the results will be less than optimal.

Cheers
manolito

[Darksoul71]

Hi manolito,

great to hear you had good results.

...adaptive algorithm changed considerably
May I ask which algorithm you are using to calculate the CQ value for the next segment based on the size deviation of the last segment ?

:thanks:

Cheers,
D$

[manolito]

May I ask which algorithm you are using to calculate the CQ value for the next segment based on the size deviation of the last segment ?

Sure, I'll try my best :)
(You can also have a look at the batch file in the first post of this thread. Everything is within the LOOP section).

First I use size instead of bitrate for all calculations. This is just because it makes my life easier using the limited math capabilities of the batch language.

The basic algorithm is simple: Encode a segment, compare the resulting size to the desired size and change the quantizer accordingly for the next segment. Example: The just encoded segment is 20% too big, so the quantizer gets raised by 20% for the next segment.

But I do not compare each segment for itself, instead after each segment the size is added to the combined size of all previous segments and then this combined size is compared to the desired combined size. This also means that I somehow have to introduce some kind of "Weight Factor" to the quantizer correction the more I get to the end of the movie. (Example: If the first segment is 50% too big then the correction factor is also 50%. But if the last segment is 50% too big then the correction factor will only be a fraction of 50%).

I do not use a gliding weight factor, I just divide the encode into three parts. For the first half of the movie I do not use any weight factor. After this the correction factor is multiplied by 3, and for the last 10% of the movie the correction factor gets multiplied by 10.

But there are more tweaks for this general algorithm. For the first 3% of the movie I just use a CQ of 3.50 without any correction. Then I limit the maximum CQ jump between segments to 0.30, plus I set an upper CQ limit of 7.00. Only for the last 10% of the movie these restrictions are lifted. (IF you specify VERY_LOW_BR then the initial CQ will be 6.00 and the upper CQ limit is 9.50).

And there is another important tweak which should minimize overcorrection: I always compare the current mismatch with the previous mismatch. If the current mismatch is smaller than the previous one then no CQ correction will take place.


Alright, I hope this explanation makes it somewhat clearer what I'm doing.

Cheers
manolito

[Darksoul71]

Hello manolito,

thanks for your detailed explanations !

You´ve choosen a pretty similar approach as I assumed.

I´m currently thinking about mixing your mode and CQ prediction.

I would first do normal size prediction to find out the CQ to meet my target size.
Then segment the movie into n parts (pretty similar to what you are doing and
what HCEnc^n does for multicore encoding). The CQ found could be the intial
CQ. Then I would adjust the CQ value for each segment depending of size variation
of the previous segment and the remaining size available.

What do you think ? Does the first step make more sense to you than always
using the same CQ start value ?

Cheers,
D$

[manolito]

Hi Darksoul,

yes, mixing prediction with this adaptive method does indeed give you the best results. I call this mode "Hybrid Mode", and I already did integrate this mode into DVD2SVCD.

It works like this:
First I do a prediction using Amnon's GetCQ (which uses the ping-pong method). But I do not waste too much time on the prediction, I use 0.5% sample size and I restrict the number of prediction runs to something between 5 and 10.

This CQ value is used as the initial CQ for the adaptive mode. The first 40% of the movie are encoded with this CQ value. From then on I do allow CQ correction, but only in a very limited way. I set a clamp around the initial CQ so it can only get bigger by 0.60 and smaller by 0.50. This is enough to take care of prediction inaccuracies while still keeping an almost constant quantizer. Only for the last 10% of the movie I switch to "Aggressive Mode" where the CQ gets corrected without restrictions (almost: I still enforce an upper CQ limit of 8.00 unless the initial CQ is above 6.00).

I came to these parameters by doing lots (more than 100) of "Real World" encodes. But of course these parameters can be optimized even further.


Cheers
manolito

[Darksoul71]

Hey manolito,

thanks once again for your help and explanations.
It finally resulted in a more independent solution which I call HCShredder for now.

Have a look here:
http://www.vmesquita.com/forum/index.php/topic,8888.msg57355/topicseen.html#new

If you like I could "rip out" my CQ prediction part and put it in a CLI tool. Shouldn´t be
a big deal and would enable you to replace GetCQ plus SmartDialog.

Feel free to contact me if you have interest in this.

Cheers,
D$

P.S.: Clamp values around 0.6 work very nice. Thanks for the hint.

[manolito]

Hi Darksoul,

this looks very interesting indeed! I do follow the discussion at Vmesquita's forum, and it looks like the real development has just begun... And BTW it is very nice for me to see how my little idea is the catalyst for something much bigger.

Thanks for the offer to rip out the predictin code of HCShredder for me, but right now I think I will stick with GetCQ (mainly for all the time I spent finding workarounds for GetCQ's shortcomings).

In the meantime I changed the defaults for Hybrid mode quite a bit, here are the changes:

1. I now use 1% sample size. A size of 0.5% did mostly work, but for some sources the inaccuracy was just too much.

2. The resulting CQ is now good enough to encode the first 70% of the movie (instead of 40%) using this CQ.

3. For the next 25% correction is enabled. Correction is a little more aggressive than before (the correction factor is now multiplied by 5 instead of 3, upper clamp value is 0.70, lower clamp stays at 0.50).

4. Only for the last 5% of the movie I switch to Aggressive mode (was 10% before). Plus Aggressive mode is now more aggressive than it was (correction factor is multiplied by 20 instead of 10).

So far these settings did always result in final sizes between 97% and 99% of the desired size.


Cheers
manolito

[Darksoul71]

Hi manolito,

this looks very interesting indeed! I do follow the discussion at Vmesquita's forum, and it looks like the real development has just begun... And BTW it is very nice for me to see how my little idea is the catalyst for something much bigger.
You´re welcome to comment things over at VM... :)

For HCShredder I think that the development has "stopped" but the intense finetuning phase has just started. *ggg*

Since I´ve choosen to use equally diveded segments I think that the clamp values and the "rate control" are a bit different. Esp. in terms of "weighting" the correction factor for every segment.

May be we´ll see something like this in HCEnc.

I think of HCShredder as some kind of "proof of concept" or experimental software.

Thanks for the offer to rip out the predictin code of HCShredder for me, but right now I think I will stick with GetCQ (mainly for all the time I spent finding workarounds for GetCQ's shortcomings).
OK, I understand this very well...
Personally I wouldn´t want to touch a working solution either :cool:

Just one final comments on your readme document:
AFAIK the prediction algorithm is borrowed from
Prodater64, he calls it the "Ping-Pong" method. It seems to be a purely binary
algorithm, not a Newton based algorithm like D2SRoBa.

The "Ping Pong" method doesn´t discribe the prediction algorithm
but the use of a "shifted sample" approach to enshure that the
1% samples (by coincidence) just capture the areas with low movement.
The 1% samples are shifted by half of the segment length to capture
all areas of movement for CQ prediction.

The pingpong method was quite well explained by prodater:
http://www.vmesquita.com/forum/index.php/topic,3895.0.html

Author of the sampler.avs is/was Incredible.

The algorithm for calculation Q can be bisection, newton raphson
or whatever you can think of.

Edit:
You can grab some more infos on pingpong in this thread:
http://forum.doom9.org/showthread.php?t=111218

Here is Inc´s posting:
http://forum.doom9.org/showpost.php?p=831325&postcount=14

Cheers,
D$