x264 + Mediacoder + VBV + Maxrate = Bug

[thiagoprado]

I'm trying to encode to streaming using a average bandwidth of 250Kbps.
But I would like to put a cap to limit the bitrate at 512Kbps.
So I used this setting:

# ".\codecs\x264.exe" "$(SourceFile)" --no-psnr --no-ssim --keyint 250 --min-keyint 25 --level 51 --vbv-maxrate 480 --vbv-bufsize 480 --me umh --merange 16 --no-fast-pskip --non-deterministic --subme 1 --ref 1 --partitions all --8x8dct --bframes 5 --direct auto --mixed-refs --trellis 2 --b-pyramid --weightb --deblock -2:-1 --bitrate $(VideoBitrate) --qcomp 0.6 --vbv-bufsize 480 --threads auto --stats "$(PassLogFile)" --pass 1 -o NUL

# ".\codecs\x264.exe" "$(SourceFile)" --no-psnr --no-ssim --keyint 250 --min-keyint 25 --level 51 --vbv-maxrate 480 --vbv-bufsize 480 --me umh --merange 16 --no-fast-pskip --non-deterministic --aq-mode 1 --aq-strength 1.0 --b-adapt 2 --ref 6 --subme 7 --psy-rd 1:0 --partitions all --8x8dct --bframes 5 --direct auto --mixed-refs --trellis 2 --b-pyramid --weightb --deblock -2:-1 --bitrate $(VideoBitrate) --qcomp 0.6 --vbv-bufsize 480 --threads auto --stats "$(PassLogFile)" --pass 2 -o "$(DestFile)"

And got this terrible result when changing from one frame to other (no fade):

http://www.thiagoprado.com.br/vbv_500.jpg


So I thought the problem was the to low cap and removed the vbv-maxrate and vbv-bufsize

And got this perfect transition and a peak bitrate at 604:

http://www.thiagoprado.com.br/vbv_0.jpg


Then I used the vbv-maxrate and vbv-bufsize at 700Kbps (96Kbps above the peak rate).

And got the same problem:

http://www.thiagoprado.com.br/vbv_700.jpg


After I did a lot of tests (cap with 800, 900, 1000Kpbs) and got the same result.

It is a bug? How can I set the vbv-maxrate above the peak bitrate and still get a bad result?
Why the peak bitrate remained at 555Kbps if I used the vbv_maxrate at 700Kbps?

Configuration:

Intel Q8200
Windows Vista Ultimate 64
Medicoder x64


Links to the encoded movies:

http://www.thiagoprado.com.br/10_papeis_lider_500_500.mp4
http://www.thiagoprado.com.br/10_papeis_lider_0_0.mp4
http://www.thiagoprado.com.br/10_papeis_lider_700_700.mp4

[Dark Shikari]

You're using an ancient version of x264, from years before VBV handling was modernized.

What is with this proliferation of MediaCoder users "reporting bugs" with ancient x264 builds anyways? We had one yesterday too.

[LoRd_MuldeR]

Also "rc=abr" is a bad choice, I think. You better go with 2-Pass, especially when VBV is involved.

[Dark Shikari]

Also "rc=abr" is a bad choice, I think. You better go with 2-Pass, especially when VBV is involved.Not with a version of x264 so old that it is from before 2pass VBV existed :p

[LoRd_MuldeR]

Not with a version of x264 so old that it is from before 2pass VBV existed :p

But given he follows the advice and updates to a recent revision of x264, he should also consider using 2-Pass instead of ABR...

[thiagoprado]

Sorry guys, I'm using 2-pass with vbr.

I download the latest x264 build from the x264.nl and replaced the codec from the Mediacoder. I did the test with the original Mediacoder x264 and with the build 1114. Got the same result.

[LoRd_MuldeR]

Sorry guys, I'm using 2-pass with vbr.

Your first post said something different:
... / rc=abr / ...

Also check the line "Writing library" in MediaInfo's report ;)

[Dark Shikari]

Sorry guys, I'm using 2-pass with vbr.Doesn't look like it to me.

[thiagoprado]

Sorry I wrote a wrong settings.

Correct

# ".\codecs\x264.exe" "$(SourceFile)" --no-psnr --no-ssim --keyint 250 --min-keyint 25 --level 51 --vbv-maxrate 480 --vbv-bufsize 480 --me umh --merange 16 --no-fast-pskip --non-deterministic --subme 1 --ref 1 --partitions all --8x8dct --bframes 5 --direct auto --mixed-refs --trellis 2 --b-pyramid --weightb --deblock -2:-1 --bitrate $(VideoBitrate) --qcomp 0.6 --vbv-bufsize 480 --threads auto --stats "$(PassLogFile)" --pass 1 -o NUL

# ".\codecs\x264.exe" "$(SourceFile)" --no-psnr --no-ssim --keyint 250 --min-keyint 25 --level 51 --vbv-maxrate 480 --vbv-bufsize 480 --me umh --merange 16 --no-fast-pskip --non-deterministic --aq-mode 1 --aq-strength 1.0 --b-adapt 2 --ref 6 --subme 7 --psy-rd 1:0 --partitions all --8x8dct --bframes 5 --direct auto --mixed-refs --trellis 2 --b-pyramid --weightb --deblock -2:-1 --bitrate $(VideoBitrate) --qcomp 0.6 --vbv-bufsize 480 --threads auto --stats "$(PassLogFile)" --pass 2 -o "$(DestFile)"

[thiagoprado]

Links to the encoded movies:

http://www.thiagoprado.com.br/10_papeis_lider_500_500.mp4
http://www.thiagoprado.com.br/10_papeis_lider_0_0.mp4
http://www.thiagoprado.com.br/10_papeis_lider_700_700.mp4

[Dark Shikari]

I checked out the first video; it looks fine to me. Is there any particular frame # you're having issues with?

[thiagoprado]

00:01:12;25 to 00:01:12;26
00:01:13;21 to 00:01:13;22
00:01:19;21 to 00:01:19;22

In videos vbv_500_500 and vbv_700_700, vbv_0_0 is perfect.

Thanks

[Dark Shikari]

Does the bug go away if you:

a) Don't use threads?
b) Don't use B-frames?

[lexor]

Does the bug go away if you:

a) Don't use threads?
b) Don't use B-frames?

I would add:
c) decoder is too old/problematic (get latest ffdshow from the try-outs thread)

[Dark Shikari]

I would add:
c) decoder is too old/problematic (get latest ffdshow from the try-outs thread)No, it's an encoder issue.

[shon3i]

Try this build :) http://www.mediafire.com/?3nozdw0ozn1

[Dark Shikari]

Try this build :) http://www.mediafire.com/?3nozdw0ozn1No, the fix in that build was already put into official x264.

[thiagoprado]

I did with no b-frames. Same problem.
I don't know how to disable threads. I can only set 0 (auto) or 1, 2, 3...
I also did with the suggested build and got the same problem.

I read in other post a similar problem with fade transition, but I'm having with dry cuts.

Thanks

[LoRd_MuldeR]

I don't know how to disable threads. I can only set 0 (auto) or 1, 2, 3...

Using "--threads 1" will disable multi-threading...

[thiagoprado]

1) With vbv-maxrate and vbv-bufsize 480 to 900Kbps = Problem
2) With vbv-maxrate and vbv-bufsize 480 to 900Kbps + no b-frames + no threads = Problem
3) With vbv-maxrate and vbv-bufsize 0 = Perfect
4) With vbv-maxrate and vbv-bufsize 3000Kbps (incredible high to this bitrate and out of my bandwidth) = Perfect

[thiagoprado]

I download the build x264.1113M.x64.exe and made a test with vbv-maxrate / vbv-bufsize to 900Kbps.

The same problem.

I can't understand why the Bitrate Viwer is showing a 584Kbps peak if the settings allow 900Kbps.

Does anybody has a soft to check my movies bitrate?

http://www.thiagoprado.com.br/10_papeis_lider_500_500.mp4
http://www.thiagoprado.com.br/10_papeis_lider_0_0.mp4
http://www.thiagoprado.com.br/10_papeis_lider_700_700.mp4

[shon3i]

obvious that no solution for this problem except that developers make some quick fix as in my type :) I can only recommend you to switch to mainconcept or elecard encoder because they encoder have stable VBV system.

[MasterNobody]

obvious that no solution for this problem except that developers make some quick fix as in my type :)And for this they probably would need the source sample to reproduce the bug.

thiagoprado
Can you upload the source sample?

[thiagoprado]

Movie source

http://www.thiagoprado.com.br/10_papeis_lider.mpeg

[poisondeathray]

How accurate are the "bitrate viewers" ?

Testing on the 500/500 download .mp4 (1st link provided) (you have audio included) vbv maxrate 480, vbv buffersize 480

Bitrate Viewer on .mp4 : PEAK 543
Bitrate Viewer on .h264 : PEAK 418
StreamEYE on .mp4 : error on reading
StreamEYE on .h264 : real max 494
Mediainfo on .mp4 : Max Bitrate 546

Mediainfo just reads the header info, but I think Streameye & bitrate viewer actually parse each frame, yet there is a significant difference on the raw avc stream ?

Is using bitrate viewer on the .mp4 reading both audio & video? The results would suggest so...

[shon3i]

StreamEye/Buffer Analyser tools need NAL HRD in stream to show graph

[poisondeathray]

^ Yes, the buffer analyzer wouln't work with that sample, but streameye did.

[thiagoprado]

Can you do a test with the 0_0 version.
This has no vbv-maxrate / bufsize.
I got 604Kbps (probably with audio included) with Bitrate Viewer.
Since this is the bitrate peak that x264 needs to encode with no blocks, the vbv-maxrate / bufsize at 700Kbps should be enough.

But I'm only get smooths encodes with vbv-maxrate / bufsize above 1500Kbps.

Strange...

[poisondeathray]

0_0 sample (second link)

Bitrate Viewer .mp4: max 604
Bitrate Viewer .h264: max 525
Streameye .h264: max 583

[thiagoprado]

I realy can't understand why x264 have problems with vbv-maxrate / bufsize at 700Kbps if the uncaped peak is only 604Kbps with audio.

[Dark Shikari]

I realy can't understand why x264 have problems with vbv-maxrate / bufsize at 700Kbps if the uncaped peak is only 604Kbps with audio.Because the VBV algorithm is complicated and if it makes a mistake, it can result in the encoder thinking that that the VBV may underflow soon--when it actually isn't going to. It relies on predictions of frame size that have to be very accurate.

[LoRd_MuldeR]

...because VBV doesn't only restrict the peak bitrate, but also the buffer size ???

That means you are allowed to exceed the average bitrate (up to the max. peak bitrate), but only for a very short time. Otherwise the buffer would underflow.

Without VBV there's no such buffer restriction at all.

[thiagoprado]

Is there any way to fix this?
I'm worried to leave the VBV uncaped and get problems with streaming bandwidth.

Thanks

[Dark Shikari]

Don't use threads until we fix it?

[thiagoprado]

I did tests with all the encoders I found...

1) ProCoder - Terrible
2) Compressor (Apple H.264) - Terrible
3) Compressor (x264 encoder) - Reasonable
4) Sorenson (MainConcept) - Reasonable
5) Mainconcept - Reasonable
6) Hanbrake (x264) - Excelent but with a terrible audio
7) Mediacoder (x264) The best!!! But with this small problem :confused:

[thiagoprado]

I already set threads to 1 and got the same problem...

[LoRd_MuldeR]

1) ProCoder - Terrible
2) Compressor (Apple H.264) - Terrible
3) Compressor (x264 encoder) - Reasonable
4) Sorenson (MainConcept) - Reasonable
5) Mainconcept - Reasonable
6) Hanbrake (x264) - Excelent but with a terrible audio
7) Mediacoder (x264) The best!!! But with this small problem :confused:

All tools that are based on x264 (3, 6 and 7) will give identical results as long as you are using the same encoder settings.

The result depends on the encoder itself and its settings, not on the GUI wrapped around it...

[Dark Shikari]

I already set threads to 1 and got the same problem...Now that's interesting--this is the first VBV bug I have seen in a long time that occurs even with threads and B-frames off. I will look into debugging this tonight or tomorrow.

If you are using two-pass, you may be able to get away with commenting out line 933 in encoder/ratecontrol.c:

if( h->mb.i_mb_x != h->sps->i_mb_width - 1 || !rc->b_vbv)

This completely disables all row-based ratecontrol and relies solely on 2pass VBV for VBV accuracy. It is not guaranteed to avoid underflow, but it should eliminate your problem for now.

[thiagoprado]

The Compressor has a terrible deinterlace and a nice audio. Handbrake use the same x264 but I can't get good audio with FAAC.

The MediaCoder has everything I need: Barch Encode, excelent deinterlace, Nero AAC and th excelenet x264.

But has this problem with VBV...

[LoRd_MuldeR]

The Compressor has a terrible deinterlace and a nice audio. Handbrake use the same x264 but I can't get good audio with FAAC.

Use the x264 CLI encoder with Avisynth input and you can use whatever deinterlace filter you prefer :)

[thiagoprado]

Now that's interesting--this is the first VBV bug I have seen in a long time that occurs even with threads and B-frames off. I will look into debugging this tonight or tomorrow.

If you are using two-pass, you may be able to get away with commenting out line 933 in encoder/ratecontrol.c:

if( h->mb.i_mb_x != h->sps->i_mb_width - 1 || !rc->b_vbv)

This completely disables all row-based ratecontrol and relies solely on 2pass VBV for VBV accuracy. It is not guaranteed to avoid underflow, but it should eliminate your problem for now.


How can I do this? Get the original code and then build x264?

[Dark Shikari]

How can I do this? Get the original code and then build x264?Sure, or ask someone to build such a modified version for you.

A proper description of this modification would be "disable row-based VBV handling".

[LoRd_MuldeR]

How can I do this? Get the original code and then build x264?

Exactly.

[thiagoprado]

Use the x264 CLI encoder with Avisynth input and you can use whatever deinterlace filter you prefer :)

The problem is that I need bach encode. I have 700 movies to encode :eek:

[LoRd_MuldeR]

The problem is that I need bach encode. I have 700 movies to encode :eek:

Ever heard of batch/script files?

Also you can use a GUI like MeGUI, which uses Avisynth input too, offers job control (batch processing) and offers all options you'll need.

[thiagoprado]

Ever heard of batch/script files?

Also you can use a GUI like MeGUI, which uses Avisynth input too, offers job control (batch processing) and offers all options you'll need.

Since this solution still use x264 I will get same problem? Correct?

[LoRd_MuldeR]

Since this solution still use x264 I will get same problem? Correct?

Yes, as said before: All tools that use the x264 encoder will give identical output for identical input (and identical encoder settings).

However Avisynth allows you to use the pre-processing filters of your choice, including your preferred deinterlacer ;)

[thiagoprado]

Sure, or ask someone to build such a modified version for you.

A proper description of this modification would be "disable row-based VBV handling".

I will try to build myself.

With this patch the vbv-maxrate / bufzie will still work?
I will be able to set the x264 to use VBV at 250Kbps with maximum 500Kbps?

Thanks

[Dark Shikari]

I will try to build myself.

With this patch the vbv-maxrate / bufzie will still work?Yes, but it will only be remotely reliable in 2-pass mode.

[J_Darnley]

If you don't have a build environment already setup, here is a build with Dark Shikari's change: http://users.telenet.be/darnley/x264/x264-thiagoprado.7z