Undersized File with x264 2pass encoding?

[HOLiC]

I have been using x264/megui for.. some time now. But never had problem with getting undersized file / 2pass encoding before. I'm using rev.502/megui 2125, and here's the log:


Starting job job1-1 at pm 3:28:31
Job is a video job. encoder commandline:
--pass 1 --bitrate 2040 --stats "C:\DVDrip\blood02\new.stats" --bframes 3 --b-pyramid --subme 1 --analyse none --qcomp 0.8 --direct auto --scenecut 80 --me dia --threads 2 --thread-input --cqmfile "C:\Program Files\Codec\custom_matrices_x264\custom_matrices_x264\eqm_avc_hr.cfg" --progress --no-psnr --output NUL "C:\DVDrip\blood02\new.avs"
successfully started encoding
Processing ended at pm 7:30:53
----------------------------------------------------------------------------------------------------------

Log for job job1-1

avis [info]: 704x528 @ 23.98 fps (34569 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
x264 [info]: slice I:333 Avg QP:11.13 size: 50055
x264 [info]: slice P:10799 Avg QP:12.70 size: 19122
x264 [info]: slice B:23437 Avg QP:13.13 size: 6245
x264 [info]: mb I I16..4: 29.5% 0.0% 70.5%
x264 [info]: mb P I16..4: 17.1% 0.0% 0.0% P16..4: 65.6% 0.0% 0.0% 0.0% 0.0% skip:17.3%
x264 [info]: mb B I16..4: 0.7% 0.0% 0.0% B16..8: 20.2% 0.0% 0.0% direct:24.8% skip:54.3%
x264 [info]: final ratefactor: 14.03
x264 [info]: direct mvs spatial:100.0% temporal:0.0%
x264 [info]: kb/s:2050.4

Actual bitrate after encoding without container overhead: 2051.45

----------------------------------------------------------------------------------------------------------
Job completed successfully and deletion of intermediate files is activated
job job1-1 has been processed. This job is linked to the next job: job1-2
Starting job job1-2 at pm 7:30:53
Job is a video job. encoder commandline:
--pass 2 --bitrate 2040 --stats "C:\DVDrip\blood02\new.stats" --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --analyse all --8x8dct --qcomp 0.8 --direct auto --scenecut 80 --me umh --threads 2 --thread-input --cqmfile "C:\Program Files\Codec\custom_matrices_x264\custom_matrices_x264\eqm_avc_hr.cfg" --progress --no-psnr --output "C:\DVDrip\blood02\new.mp4" "C:\DVDrip\blood02\new.avs"
successfully started encoding
Processing ended at am 12:24:06
----------------------------------------------------------------------------------------------------------

Log for job job1-2

avis [info]: 704x528 @ 23.98 fps (34569 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
mp4 [info]: initial delay 2002 (scale 24000)
x264 [info]: slice I:333 Avg QP:10.07 size: 50954
x264 [info]: slice P:10799 Avg QP:10.08 size: 21951
x264 [info]: slice B:23437 Avg QP:11.73 size: 3805
x264 [info]: mb I I16..4: 7.1% 34.6% 58.3%
x264 [info]: mb P I16..4: 1.7% 6.8% 6.1% P16..4: 50.6% 15.7% 11.6% 0.4% 0.3% skip: 6.9%
x264 [info]: mb B I16..4: 0.1% 0.3% 0.2% B16..8: 14.2% 0.7% 1.7% direct: 2.5% skip:80.3%
x264 [info]: 8x8 transform intra:44.9% inter:48.2%
x264 [info]: direct mvs spatial:94.3% temporal:5.7%
x264 [info]: ref P 59.4% 10.6% 8.2% 3.5% 3.6% 2.4% 2.7% 1.3% 1.5% 1.1% 1.3% 0.9% 1.1% 0.9% 1.0% 0.6%
x264 [info]: ref B 63.6% 12.7% 6.5% 3.7% 2.8% 2.2% 1.6% 1.3% 1.0% 1.0% 0.8% 1.0% 0.7% 0.7% 0.5%
x264 [info]: kb/s:1904.2

Actual bitrate after encoding without container overhead: 1905.38
desired video bitrate of this job: 2040 kbit/s - obtained video bitrate: 1907.45214656904 kbit/s


I just want to know if this is normal. Getting undersized/oversized file is really no big deal to me, if this is just part of the encoding process, as explained in Xvid undersized/oversized sticky. But because I never experienced this problem with any other x264 encodings, I would feel better if someone can confirm this...

[akupenguin]

Avg QP:10.08
It's just saturated.

[vanger]

I have a problem with second pass :(
1. The source is 58min progressive 704x560 25fps PAL video AVI (HuffYUV).
AVISource("D:\temp\11.avi")
Starting job job1-1 at 8:51:48
encoder commandline:
--pass 1 --bitrate 1623 --stats "D:\temp\vv.stats" --bframes 3 --b-pyramid --direct auto --filter -2,-2 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --thread-input --progress --no-psnr --output NUL "D:\temp\vv.avs"
successfully started encoding
Processing ended at 9:58:13
----------------------------------------------------------------------------------------------------------

Log for job job1-1

avis [info]: 704x560 @ 25.00 fps (87230 frames)
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!
x264 [info]: slice I:386 Avg QP:22.89 size: 31209
x264 [info]: slice P:33917 Avg QP:25.05 size: 13167
x264 [info]: slice B:52927 Avg QP:26.54 size: 4799
x264 [info]: mb I I16..4: 39.6% 0.0% 60.4%
x264 [info]: mb P I16..4: 23.2% 0.0% 0.0% P16..4: 57.3% 0.0% 0.0% 0.0% 0.0% skip:19.5%
x264 [info]: mb B I16..4: 1.1% 0.0% 0.0% B16..8: 20.4% 0.0% 0.0% direct:18.9% skip:59.7%
x264 [info]: final ratefactor: 24.35
x264 [info]: direct mvs spatial:98.9% temporal:1.1%
x264 [info]: SSIM Mean Y:0.9496316
x264 [info]: kb/s:1633.9

encoded 87230 frames, 21.90 fps, 1634.01 kb/s

----------------------------------------------------------------------------------------------------------
Job completed successfully and deletion of intermediate files is activated
job job1-1 has been processed. This job is linked to the next job: job1-2
Starting job job1-2 at 9:58:13
encoder commandline:
--pass 2 --bitrate 1623 --stats "D:\temp\vv.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-2 --subme 6 --trellis 1 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --thread-input --progress --no-psnr --output "D:\temp\vv.mp4" "D:\temp\vv.avs"
successfully started encoding
Processing ended at 13:52:29
----------------------------------------------------------------------------------------------------------

Log for job job1-2

avis [info]: 704x560 @ 25.00 fps (87230 frames)
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!
mp4 [info]: initial delay 2 (scale 25)
x264 [info]: slice I:386 Avg QP:18.08 size: 57674
x264 [info]: slice P:33917 Avg QP:20.54 size: 7382
x264 [info]: slice B:52927 Avg QP:21.66 size: 8190
x264 [info]: mb I I16..4: 24.6% 50.5% 24.8%
x264 [info]: mb P I16..4: 6.0% 6.8% 1.2% P16..4: 43.9% 7.8% 2.7% 0.2% 0.1% skip:31.4%
x264 [info]: mb B I16..4: 0.6% 1.1% 0.8% B16..8: 25.9% 2.6% 6.6% direct: 2.0% skip:60.3%
x264 [info]: 8x8 transform intra:47.8% inter:59.4%
x264 [info]: direct mvs spatial:92.2% temporal:7.8%
x264 [info]: ref P 48.9% 35.5% 7.8% 4.8% 3.1%
x264 [info]: ref B 84.0% 8.7% 3.2% 2.2% 1.9%
x264 [info]: SSIM Mean Y:0.9699417
x264 [info]: kb/s:1619.0

encoded 87230 frames, 6.21 fps, 1619.22 kb/s

desired video bitrate of this job: 1623 kbit/s - obtained video bitrate (approximate): 733 kbit/s
----------------------------------------------------------------------------------------------------------
Job completed successfully and deletion of intermediate files is activated
Found intermediate output file 'D:\temp\vv.stats', deleting...
Deletion succeeded.


2. The source is 113min 720x576 25fps PAL video in MPEG-2.
DGDecode_mpeg2source("D:\capture\video\last.d2v",info=3)
ColorMatrix(hints=true)
FFT3DFilter(sigma=6)

Starting job job1-1 at 23:53:21
encoder commandline:
--pass 1 --bitrate 1600 --stats "D:\capture\video\zxxxz.stats" --bframes 3 --b-pyramid --direct auto --filter -2,-2 --subme 1 --analyse none --vbv-maxrate 25000 --me dia --thread-input --progress --no-psnr --output NUL "D:\capture\video\zxxxz.avs"
successfully started encoding
Processing ended at 2:13:33
----------------------------------------------------------------------------------------------------------

Log for job job1-1

avis [info]: 720x576 @ 25.00 fps (170359 frames)
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!
x264 [info]: slice I:1154 Avg QP:16.66 size: 27117
x264 [info]: slice P:82672 Avg QP:18.70 size: 11742
x264 [info]: slice B:86533 Avg QP:20.21 size: 4150
x264 [info]: mb I I16..4: 27.8% 0.0% 72.2%
x264 [info]: mb P I16..4: 34.9% 0.0% 0.0% P16..4: 59.1% 0.0% 0.0% 0.0% 0.0% skip: 6.1%
x264 [info]: mb B I16..4: 2.7% 0.0% 0.0% B16..8: 32.5% 0.0% 0.0% direct:30.2% skip:34.6%
x264 [info]: final ratefactor: 20.05
x264 [info]: direct mvs spatial:99.6% temporal:0.4%
x264 [info]: SSIM Mean Y:0.9871427
x264 [info]: kb/s:1598.0

encoded 170359 frames, 20.28 fps, 1598.07 kb/s

----------------------------------------------------------------------------------------------------------
Job completed successfully and deletion of intermediate files is activated
job job1-1 has been processed. This job is linked to the next job: job1-2
Starting job job1-2 at 2:13:33
encoder commandline:
--pass 2 --bitrate 1600 --stats "D:\capture\video\zxxxz.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-2 --subme 6 --trellis 1 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --thread-input --progress --no-psnr --output "D:\capture\video\last.mp4" "D:\capture\video\zxxxz.avs"
successfully started encoding
Processing ended at 4:32:54
----------------------------------------------------------------------------------------------------------

Log for job job1-2

avis [info]: 720x576 @ 25.00 fps (170359 frames)
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!
mp4 [info]: initial delay 2 (scale 25)

----------------------------------------------------------------------------------------------------------
The current job contains errors. Skipping chained jobs


The 1pass encoding of the file
#SelectRangeEvery(280, 14)
with crf=18 works fine...

What is my mistake???

[foxyshadis]

Often when I see that, it means avisynth crashed. Did you try watching your encode, a few seconds each chapter? You might find a glaring red message like "System Exception" starting halfway through.

[vanger]

Did you try watching your encode, a few seconds each chapter? You might find a glaring red message like "System Exception" starting halfway through.
All the movie plays normal. The encoding of 1st pass works normal.
The MainConcept H264 encodes this AVS script with the similar parameters without any problem... But the MeGUI is more useful.

[Sharktooth]

do you have a dualcore or HT CPU?
If so try disabling the "Automatically set the number of threads" option in settings and ensure the Threads option is set to 1 (into the codec config dialog).
After doing that check if it encodes correctly...

[vanger]

do you have a dualcore or HT CPU?
If so try disabling the "Automatically set the number of threads" option in settings and ensure the Threads option is set to 1 (into the codec config dialog).
After doing that check if it encodes correctly...
Yes, I have Dual Core, but threads option is set to 1 and "Automatically set the number of threads" option is disabled...

[vanger]

the new rev.611 works fine!
thnx!

[twazerty]

Couple of days/weeks ago I ran into the undersizing problem. The generated output files are about 2-5% too small!!! I encounter this problem with x264 r1170 until r1184. I use x264 in my converter AVCHDCoder and in previous releases (with older x264 revisions) I never had undersizing .h264 files.

I also updated ffdshow to r3029 but the undersizing problem is still there. As bitrate I used 10778

Bitrate: 10778
Movielenght: 5238 seconds
Expected size: 10778/8 * 5238 /1024/1024 = 6.73GB
Real size: 6.38GB

Difference: 0.35GB

When output is about 18 GB the file is about 1GB undersized.

I use the same avisynth script with all revisions of x264. This is the x264 command line

x264.exe --pass 1 --bitrate 10778 --stats "5.stats" --level 4.1 --keyint 24 --min-keyint 2 --bframes 3 --weightb --direct auto --subme 2 --partitions none --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 9000 --vbv-maxrate 24000 --me dia --threads auto --thread-input --sar 1:1 --output NUL "5.avs"

x264.exe --pass 2 --bitrate 10778 --stats "5.stats" --level 4.1 --keyint 24 --min-keyint 2 --ref 3 --mixed-refs --no-fast-pskip --bframes 3 --weightb --direct auto --subme 7 --trellis 1 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 9000 --vbv-maxrate 24000 --me umh --threads auto --thread-input --sar 1:1 --output "output.264" "5.avs"


What happend???

[nurbs]

Post your log.

[twazerty]

Post your log.

There isn't a log (No support yet) but this is the avisynth Script:
DirectShowsource("input.mkv",fps=23.976,audio=false).AssumeFPS(24000,1001).ConvertToYV12().addborders(0,0,0,0)

and in old versions I used:
DirectShowsource("input.mkv",fps=23.9759856527702,audio=false)
addborders(0,0,0,0)

I tried:
DirectShowsource("input.mkv",fps=23.976,audio=false).ConvertToYV12().addborders(0,0,0,0)
but the undersizing is still there.

[LoRd_MuldeR]

There isn't a log (No support yet)

There is a log. By default x264 will write the log to the console window. You can just copy the x264 output from the console.

Alternatively you can redirect the x264 log to a text file:
x264.exe --crf 22 --output "c:\Test.mkv" "c:\Source.avs" 2> "c:\Test.log"

[kemuri-_9]

Bitrate: 10778
Movielenght: 5238 seconds
Expected size: 10778/8 * 5238 /1024/1024 = 6.73GB
Real size: 6.38GB

your math is also wrong btw:
(10778 kb/s) / (8 bits/byte) * (1000 bits/s / 1 kb/s) * (5238s) / (1024 B/KB) / (1024 KB/MB) / (1024 MB/GB) = 6.57GB

[twazerty]

There is a log. By default x264 will write the log to the console window. You can just copy the x264 output from the console.

Alternatively you can redirect the x264 log to a text file:
x264.exe --crf 22 --output "c:\Test.mkv" "c:\Source.avs" 2> "c:\Test.log"

my latest builds of AVCHDCoder doesn't output x264 logs :D

Within 12 hours I post the x264 log.

@kemuri-_9
You are right. Aargh 1024 vs 1000

but then there is still a 0.19GB difference.

When I calculate the bitrate:
6.57GB = 7054483783,68 bytes
7054483783 * 8 / 1000 / 5238 = 10774
This formula is in use for 6 months so that can't be the problem. 10778 goes into x264 and the output file is 6.38GB and not 6.57GB

[LoRd_MuldeR]

Within 12 hours I post the x264 log.

Put your attention on these lines:
x264 [info]: slice I:19 Avg QP:23.16 size: 21516
x264 [info]: slice P:1964 Avg QP:25.46 size: 6685
x264 [info]: slice B:3019 Avg QP:27.90 size: 2630

If "Avg QP" is close to 10.00, the encoder is simply saturated. In that case try to lower "--qpmin" ;)

[kemuri-_9]

likely indicative messages of LoRd_MuldeR's proposed case would be:

x264 [warning]: Error: 2pass curve failed to converge
x264 [warning]: target: _ kbit/s, expected: _ kbit/s, avg QP: _
x264 [warning]: try reducing target bitrate or reducing qp_min (currently _)

[twazerty]

likely indicative messages of LoRd_MuldeR's proposed case would be:

x264 [warning]: Error: 2pass curve failed to converge
x264 [warning]: target: _ kbit/s, expected: _ kbit/s, avg QP: _
x264 [warning]: try reducing target bitrate or reducing qp_min (currently _)


I have this undersizing problem on 4 computers. (3 alpha testers) We tried with different output sizes and movies. DVD5/DVD9/BD25. Very strange. Maybe the qp_min value is changed in later x264 revisions?

Currently testing with my old version of AVCHDCoder to see if output is correct. But I am almost 100% convinced the output will be correct.

[Dark Shikari]

I have this undersizing problem on 4 computers. (3 alpha testers) We tried with different output sizes and movies. DVD5/DVD9/BD25. Very strange. Maybe the qp_min value is changed in later x264 revisions?

Currently testing with my old version of AVCHDCoder to see if output is correct. But I am almost 100% convinced the output will be correct.qpmin hasn't changed in over four years (http://git.videolan.org/?p=x264.git;a=commit;h=aecc6ab057616f32eb0643b36db2d5b04d7a07ea)

[twazerty]

Compared the old x264 with the new one and output is both correct. I now used identical situations. Currently testing with my new x264 settings. I expect the output will be correct. But to make sure I will test it. If output is correct I have to compare this avisynth script:

DirectShowsource("input.mkv",fps=23.9759856527702,audio=false)
addborders(0,0,0,0)
LanczosResize(1920,1080) #Lanczos(Sharp)

With:

DirectShowsource("input.mkv",fps=23.976,audio=false).AssumeFPS(24000,1001).ConvertToYV12().addborders(0,0,0,0)


I expect the fps setting will cause the undersizing problem. I am going to check that too.

I am informed that BDRebuilder also has the undersizing problem. The output was 20.8 instead of 23.1GB

[Dark Shikari]

I am informed that BDRebuilder also has the undersizing problem. The output was 20.8 instead of 23.1GBIf you actually think there is a problem, can you perform a bisect and find the revision that caused it?

[twazerty]

If you actually think there is a problem, can you perform a bisect and find the revision that caused it?

What is a bisect?

[Dark Shikari]

What is a bisect?

You know revision 1000 is working, and 1100 is broken.

You try 1050. Does it work? If so, try 1075. If it doesn't work, try 1025. And so forth until you find the revision that broke it.

"git bisect" does this automatically.