2nd disk too small ... Tmpgenc CQ estimate way too low

[dwayneh]

I'm confused by how the CQ estimates are calculated for Tmpgenc. The result is a first CD at the expected 800 mb, but the second (last) CD is less than 200 mb! I have no intention here of revisiting the Tmpgenc vs CCE debate, or the CQ vs VBR debate. I am trying to use Tmpgenc and let D2S estimate the appropriate CQ.

My first question is to clarify the role of the file prediction factor. My understanding is that it is simply a "fudge factor" that is used to round down a bit to avoid creating disks that are too large, correct? That is, after D2S calculates the optimal CQ, the file prediction factor is multiplied by that CQ before the actual encoding, right? A default factor of .950 seems very conservative, but that is of course up to the user to accept the risk of a higher factor.

My bigger issue seems to be with how D2S calculates the optimal CQ. I can see that it runs a small sample through a series of tests, but I can't see where it "narrows in" much at all on an optimal CQ. In the example below, it bounces back and forth between approximately 95 and approximately 81 FIVE times! It never seems to narrow toward something in the middle of those extremes. Then it settles on the low of 81, and of course multiplies that by the file prediction factor (default .950) and does the encode at 77. The result is a lot of wasted bits, of course.

I'm hoping this is a simple "can't see the forest for the trees" issue. I'll gladly be embarrased by a simple fix, LOL! Any ideas?


WNASPI32.DLL 4.6.0.1021
--------------------------------------------------------
- 5/30/2003 10:59:17 AM
- DVD to SVCD Conversion
- DVD2SVCD ver. 1.1.3 build 2
--------------------------------------------------------
Initializing
Initializing finished.

--------------------------------------------------------
- 5/30/2003 10:59:20 AM
- DVD2AVI
--------------------------------------------------------
Creating DVD2AVI INI file:
- C:\Program Files\DVD2SVCD\DVD2AVI\DVD2AVI.INI

Variable settings:
iDCT_Algorithm: 32-bit SSE MMX

Executing DVD2AVI.
Executing DVD2AVI. Commandline:
"C:\Program Files\DVD2SVCD\DVD2AVI\DVD2AVI.exe" -CS=2 -YR=1 -EXIT -OF=[C:\Program Files\DVD2SVCD\Movie\DVD2AVI_Project_file] -IF=[C:\Program Files\DVD2SVCD\Movie\VTS_01_1.vob,C:\Program Files\DVD2SVCD\Movie\VTS_01_2.vob,C:\Program Files\DVD2SVCD\Movie\VTS_01_3.vob,C:\Program Files\DVD2SVCD\Movie\VTS_01_4.vob,C:\Program Files\DVD2SVCD\Movie\VTS_01_5.vob]
Analyzing DVD2AVI Project file
Force Film activated!
Framerate: 23976
DVD2AVI processing done.

--------------------------------------------------------
- 5/30/2003 11:01:01 AM
- Free on drive C: 31250.34 mb
- AUDIO Extraction
--------------------------------------------------------
Found AC3 stream id: 0x81
Filename: C:\Program Files\DVD2SVCD\Movie\Extracted_audio_1.ac3
Audio1 delay: 0 ms
Audio extraction finished.

--------------------------------------------------------
- 5/30/2003 11:03:49 AM
- Free on drive C: 30875.37 mb
- AUDIO conversion
--------------------------------------------------------

Encoding Audio. Filename: C:\Program Files\DVD2SVCD\Movie\Extracted_audio_1.ac3
Executing BeSweet. Commandline:
"C:\Program Files\DVD2SVCD\BeSweet\BeSweet.exe" -core( -input "C:\Program Files\DVD2SVCD\Movie\Extracted_audio_1.ac3" -output "C:\PROGRA~1\DVD2SVCD\Movie\Encoded_audio_1.mp2" -logfile "C:\PROGRA~1\DVD2SVCD\Movie\Encoded_audio_1.log" ) -azid( -L -3db -c normal -s surround -g max ) -shibatch( --rate 44100 ) -2lame( -e -b 192 -m s )
Audio conversion of C:\Program Files\DVD2SVCD\Movie\Extracted_audio_1.ac3 finished.

Audio conversion finished.

--------------------------------------------------------
- 5/30/2003 11:26:45 AM
- Free on drive C: 30714.66 mb
- Converting Pictures
--------------------------------------------------------
Converting: C:\Program Files\DVD2SVCD\DefaultTitlepicture.bmp
Converting: C:\Program Files\DVD2SVCD\DefaultChangeCD.bmp
Finished converting pictures
--------------------------------------------------------
- 5/30/2003 11:26:54 AM
- Free on drive C: 30713.70 mb
- Video Encoding using TMPGEnc
--------------------------------------------------------
- Encoding C:\Program Files\DVD2SVCD\Movie\TitlePicture.bmp.avs
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding C:\Program Files\DVD2SVCD\Movie\ChangeCDPicture.bmp.avs
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Wanted: 1461209307 (1426962kb) Predicted: 979851811 (956887kb) Using CQ Value: 80.00
Calculated next CQ Value: 95.81
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Wanted: 1461209307 (1426962kb) Predicted: 1830878908 (1787968kb) Using CQ Value: 95.81
Calculated next CQ Value: 81.27
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Wanted: 1461209307 (1426962kb) Predicted: 1040943417 (1016546kb) Using CQ Value: 81.27
Calculated next CQ Value: 95.29
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Wanted: 1461209307 (1426962kb) Predicted: 1815952028 (1773391kb) Using CQ Value: 95.29
Calculated next CQ Value: 81.41
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Wanted: 1461209307 (1426962kb) Predicted: 1050629342 (1026005kb) Using CQ Value: 81.41
Calculated next CQ Value: 95.14
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Wanted: 1461209307 (1426962kb) Predicted: 1809998373 (1767577kb) Using CQ Value: 95.14
Calculated next CQ Value: 81.51
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Wanted: 1461209307 (1426962kb) Predicted: 1056465057 (1031704kb) Using CQ Value: 81.51
Calculated next CQ Value: 95.06
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Wanted: 1461209307 (1426962kb) Predicted: 1807097673 (1764744kb) Using CQ Value: 95.06
Calculated next CQ Value: 81.56
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Wanted: 1461209307 (1426962kb) Predicted: 1059532404 (1034700kb) Using CQ Value: 81.56
Calculated next CQ Value: 95.01
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Wanted: 1461209307 (1426962kb) Predicted: 1805010686 (1762706kb) Using CQ Value: 95.01
Calculated next CQ Value: 81.60
Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

- Encoding Test Clip C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file_CQ_TEST.avs
Wanted: 1461209307 (1426962kb) Predicted: 1061420961 (1036544kb) Using CQ Value: 81.60
Final CQ Value: 81.60
- Encoding C:\Program Files\DVD2SVCD\Movie\AviSynth_Script_file.avs
StreamSectors: 1665085576
AudioSectors: 170951116
VideoPAPO: 23144940
ScanOffsetBytes: 151501
SeqAligningBytes: 9628705
DVDBytes: 0
VideoEndHeader: 8
SubtitleSectors: 0
EmptySectors: 238.00
PictureSectors: 600.00
PureMPEGStream: 1461209306.60
Seconds: 7027.94
CalcMPEGStream: 1461209306.60
Frames: 168334
CDSize: 800.00
Number of CDs: 2
Cut point 795.00
Using CQ Value: 77.52
Aspect Ratio: 16:9 (borders added, encoded as 4:3)

---AVS Begin---
Import("C:\PROGRA~1\DVD2SVCD\Movie\RESAMP~1.AVS")
LoadPlugin("C:\PROGRA~1\DVD2SVCD\AVISYN~1\Mpeg2dec\mpeg2dec.dll")
mpeg2source("C:\PROGRA~1\DVD2SVCD\Movie\DVD2AV~1.D2V")
BicubicResize(480,360,0.0,0.6)
AddBorders(0,60,0,60)
----AVS End----

Executing TMPGEnc. Commandline:
"C:\Program Files\Pegasys Inc\TMPGEnc Plus 2.5\TMPGEnc.exe" "C:\Program Files\DVD2SVCD\Movie\TMPGEnc_Project_file.tpr" /Encode /Close
Video Encoding finished.

--------------------------------------------------------
- 5/30/2003 5:45:13 PM
- Free on drive C: 29889.85 mb
- Converting Pictures from ES to PS
--------------------------------------------------------
Saving bbMPEG settings: C:\Program Files\DVD2SVCD\bbMPEG\default.ini

Executing RunbbMPEG. Commandline:
"C:\Program Files\DVD2SVCD\bbMPEG\RunbbMPEG.exe" "C:\Program Files\DVD2SVCD\Movie\TitlePicture.bmp.mpg"
--------------------------------------------------------
- 5/30/2003 5:45:17 PM
- Free on drive C: 29889.81 mb
- Converting Pictures from ES to PS
--------------------------------------------------------
Saving bbMPEG settings: C:\Program Files\DVD2SVCD\bbMPEG\default.ini

Executing RunbbMPEG. Commandline:
"C:\Program Files\DVD2SVCD\bbMPEG\RunbbMPEG.exe" "C:\Program Files\DVD2SVCD\Movie\ChangeCDPicture.bmp.mpg"
--------------------------------------------------------
- 5/30/2003 5:45:20 PM
- Free on drive C: 29889.77 mb
- Converting Pictures from ES to PS
--------------------------------------------------------
Saving bbMPEG settings: C:\Program Files\DVD2SVCD\bbMPEG\default.ini
--------------------------------------------------------
- 5/30/2003 5:45:23 PM
- Free on drive C: 29889.77 mb
- Multiplexing and cutting
--------------------------------------------------------
Saving bbMPEG settings: C:\Program Files\DVD2SVCD\bbMPEG\default.ini
Offset in Seconds: 2
Executing bbMPEG.
Variable Settings:
Movie offset: 2 seconds
Cut point: 795 mb

Executing RunbbMPEG. Commandline:
"C:\Program Files\DVD2SVCD\bbMPEG\RunbbMPEG.exe" "C:\Program Files\DVD2SVCD\Movie\bbMPEG_Muxed_File.mpg"
Length of C:\Program Files\DVD2SVCD\Movie\bbMPEG_Muxed_File00.mpg
5643 Seconds
Offset in Seconds: 5645
Executing bbMPEG.
Variable Settings:
Movie offset: 2 seconds
Cut point: 795 mb

Executing RunbbMPEG. Commandline:
"C:\Program Files\DVD2SVCD\bbMPEG\RunbbMPEG.exe" "C:\Program Files\DVD2SVCD\Movie\bbMPEG_Muxed_File.mpg"
Multiplexing and cutting finished.

--------------------------------------------------------
- 5/30/2003 5:51:10 PM
- Free on drive C: 28896.08 mb
- Determining length of audio
--------------------------------------------------------
Analyzing: C:\Program Files\DVD2SVCD\Movie\bbMPEG_Muxed_File00.mpg
5645.714 seconds C:\Program Files\DVD2SVCD\Movie\bbMPEG_Muxed_File00.mpg
Analyzing: C:\Program Files\DVD2SVCD\Movie\bbMPEG_Muxed_File01.mpg
1375.347 seconds C:\Program Files\DVD2SVCD\Movie\bbMPEG_Muxed_File01.mpg
Analyzing finished.

--------------------------------------------------------
- 5/30/2003 5:51:38 PM
- Free on drive C: 28896.07 mb
- Creating CD-Images using VCDXBuild
--------------------------------------------------------
- No. of chapters to create: 24

CD image #1: CD_Image_File_CD1.bin

Executing VCDXBuild. Commandline:
"C:\Program Files\DVD2SVCD\VCDImager\VCDXBuild.exe" --verbose --cue-file="C:\PROGRA~1\DVD2SVCD\Movie\CD_Image_File_CD1.cue" --bin-file="C:\PROGRA~1\DVD2SVCD\Movie\CD_Image_File_CD1.bin" "C:\PROGRA~1\DVD2SVCD\Movie\VCDXBU~1.XML"
--------------------------------------------------------
- 5/30/2003 5:53:13 PM
- Free on drive C: 28089.86 mb
- Creating CD-Images using VCDXBuild
--------------------------------------------------------
- No. of chapters to create: 5

CD image #2: CD_Image_File_CD2.bin

Executing VCDXBuild. Commandline:
"C:\Program Files\DVD2SVCD\VCDImager\VCDXBuild.exe" --verbose --cue-file="C:\PROGRA~1\DVD2SVCD\Movie\CD_Image_File_CD2.cue" --bin-file="C:\PROGRA~1\DVD2SVCD\Movie\CD_Image_File_CD2.bin" "C:\PROGRA~1\DVD2SVCD\Movie\VCDXBU~2.XML"
CD-Image creation finished.

--------------------------------------------------------
- 5/30/2003 5:53:25 PM
- Free on drive C: 27887.13 mb
- SVCD Creation finished!
--------------------------------------------------------

[UltimateDBZ]

Do a search. TMPG is notorious for this sort of thing.

[dwayneh]

Originally posted by UltimateDBZ
Do a search. TMPG is notorious for this sort of thing. I should have explained my previous research in the original post. I did do quite a bit of searching, and that was BEFORE I had to wait five days to be able to post. I am a moderator on another public board (not nearly as busy as this one though), and I can appreciate the "cooling off period" and the "did you search first" response. That said, I certainly could have missed the explanation / fix.

In my search, I found several mentions of the last disk being slightly too large. Yes, I understand that D2S is using a sample, and the actual encode may turn out different. The file prediction factor is a great way to "round down" a bit to compensate for that margin of error. I did not however, find any explanation / fix for being grossly too small.

You stated that "TMPG is notorious for this sort of thing." I don't mean to disagree, but I didn't think TMPG was doing the estimating. While I don't pretend to know the inner workings of D2S, my guess is that it is somehow taking the "wanted" and "predicted" numbers (as reported in the log), and running it through some algorithm to make the next estimate. It is that algorithm of which I am suspicious. Even a simple average between tests would move the numbers closer together. As you can tell from the log, the numbers basically switch back and forth between two extremes, and never get closer together.

I am aware that a "bug report" should be a last resort. I would actually have to research how to submit one, and goodness knows our esteemed author is busy enough, but I can't accept the "it's not an exact science" answer quite yet. It's just too far off. I will do a few more tests with other movies (though it's been consistent on the last 3 I've done). I may even try some older builds of D2S, because I know I've actually been impressed by the estimates in the past. I will follow up here with more information. I try my best not to just be a complainer, and I will do as much as I can to help resolve the issue.

In the meantime, I would appreciate any other insight or experiences from other users on this matter.

[mordant]

Take a look over at www.kvcd.net

They have done a lot of work on the CQ prediction with TMPGenc and have come up with a very accurate formula. The program MovieStacker there has one version of the prediction, and ToK (To KVCD) has a faster version. I think you can modify the settings in ToK to do predictions without the kvcd gop/matrix.

[UltimateDBZ]

Glad to hear you've done your homework beforehand, that wins you points in my book :)

[canoner]

for a non-action moive, I use 1.0 instead of 0.95 for prediction factor. But it is not a perfect prediction, and sometimes it gets quite a bit off. An easy fix will be forcing TMPG to resample different clips each time. Statistically it will be more robust, but I don't know how to implement it.

[dwayneh]

Originally posted by canoner
for a non-action moive, I use 1.0 instead of 0.95 for prediction factor. But it is not a perfect prediction, and sometimes it gets quite a bit off. An easy fix will be forcing TMPG to resample different clips each time. Statistically it will be more robust, but I don't know how to implement it. Yes, that would help as well. And I wouldn't mind doing the math myself on the estimating, but I too haven't dug enough to know how to interrupt the process safely.