x264 + Mediacoder + VBV + Maxrate = Bug

[thiagoprado]

I'm trying to encode to streaming using a average bandwidth of 250Kbps.
But I would like to put a cap to limit the bitrate at 512Kbps.
So I used this setting:

# ".\codecs\x264.exe" "$(SourceFile)" --no-psnr --no-ssim --keyint 250 --min-keyint 25 --level 51 --vbv-maxrate 480 --vbv-bufsize 480 --me umh --merange 16 --no-fast-pskip --non-deterministic --subme 1 --ref 1 --partitions all --8x8dct --bframes 5 --direct auto --mixed-refs --trellis 2 --b-pyramid --weightb --deblock -2:-1 --bitrate $(VideoBitrate) --qcomp 0.6 --vbv-bufsize 480 --threads auto --stats "$(PassLogFile)" --pass 1 -o NUL

# ".\codecs\x264.exe" "$(SourceFile)" --no-psnr --no-ssim --keyint 250 --min-keyint 25 --level 51 --vbv-maxrate 480 --vbv-bufsize 480 --me umh --merange 16 --no-fast-pskip --non-deterministic --aq-mode 1 --aq-strength 1.0 --b-adapt 2 --ref 6 --subme 7 --psy-rd 1:0 --partitions all --8x8dct --bframes 5 --direct auto --mixed-refs --trellis 2 --b-pyramid --weightb --deblock -2:-1 --bitrate $(VideoBitrate) --qcomp 0.6 --vbv-bufsize 480 --threads auto --stats "$(PassLogFile)" --pass 2 -o "$(DestFile)"

And got this terrible result when changing from one frame to other (no fade):

http://www.thiagoprado.com.br/vbv_500.jpg


So I thought the problem was the to low cap and removed the vbv-maxrate and vbv-bufsize

And got this perfect transition and a peak bitrate at 604:

http://www.thiagoprado.com.br/vbv_0.jpg


Then I used the vbv-maxrate and vbv-bufsize at 700Kbps (96Kbps above the peak rate).

And got the same problem:

http://www.thiagoprado.com.br/vbv_700.jpg


After I did a lot of tests (cap with 800, 900, 1000Kpbs) and got the same result.

It is a bug? How can I set the vbv-maxrate above the peak bitrate and still get a bad result?
Why the peak bitrate remained at 555Kbps if I used the vbv_maxrate at 700Kbps?

Configuration:

Intel Q8200
Windows Vista Ultimate 64
Medicoder x64


Links to the encoded movies:

http://www.thiagoprado.com.br/10_papeis_lider_500_500.mp4
http://www.thiagoprado.com.br/10_papeis_lider_0_0.mp4
http://www.thiagoprado.com.br/10_papeis_lider_700_700.mp4

[Dark Shikari]

You're using an ancient version of x264, from years before VBV handling was modernized.

What is with this proliferation of MediaCoder users "reporting bugs" with ancient x264 builds anyways? We had one yesterday too.

[LoRd_MuldeR]

Also "rc=abr" is a bad choice, I think. You better go with 2-Pass, especially when VBV is involved.

[Dark Shikari]

Also "rc=abr" is a bad choice, I think. You better go with 2-Pass, especially when VBV is involved.Not with a version of x264 so old that it is from before 2pass VBV existed :p

[LoRd_MuldeR]

Not with a version of x264 so old that it is from before 2pass VBV existed :p

But given he follows the advice and updates to a recent revision of x264, he should also consider using 2-Pass instead of ABR...

[thiagoprado]

Sorry guys, I'm using 2-pass with vbr.

I download the latest x264 build from the x264.nl and replaced the codec from the Mediacoder. I did the test with the original Mediacoder x264 and with the build 1114. Got the same result.

[LoRd_MuldeR]

Sorry guys, I'm using 2-pass with vbr.

Your first post said something different:
... / rc=abr / ...

Also check the line "Writing library" in MediaInfo's report ;)

[Dark Shikari]

Sorry guys, I'm using 2-pass with vbr.Doesn't look like it to me.

[thiagoprado]

Sorry I wrote a wrong settings.

Correct

# ".\codecs\x264.exe" "$(SourceFile)" --no-psnr --no-ssim --keyint 250 --min-keyint 25 --level 51 --vbv-maxrate 480 --vbv-bufsize 480 --me umh --merange 16 --no-fast-pskip --non-deterministic --subme 1 --ref 1 --partitions all --8x8dct --bframes 5 --direct auto --mixed-refs --trellis 2 --b-pyramid --weightb --deblock -2:-1 --bitrate $(VideoBitrate) --qcomp 0.6 --vbv-bufsize 480 --threads auto --stats "$(PassLogFile)" --pass 1 -o NUL

# ".\codecs\x264.exe" "$(SourceFile)" --no-psnr --no-ssim --keyint 250 --min-keyint 25 --level 51 --vbv-maxrate 480 --vbv-bufsize 480 --me umh --merange 16 --no-fast-pskip --non-deterministic --aq-mode 1 --aq-strength 1.0 --b-adapt 2 --ref 6 --subme 7 --psy-rd 1:0 --partitions all --8x8dct --bframes 5 --direct auto --mixed-refs --trellis 2 --b-pyramid --weightb --deblock -2:-1 --bitrate $(VideoBitrate) --qcomp 0.6 --vbv-bufsize 480 --threads auto --stats "$(PassLogFile)" --pass 2 -o "$(DestFile)"

[thiagoprado]

Links to the encoded movies:

http://www.thiagoprado.com.br/10_papeis_lider_500_500.mp4
http://www.thiagoprado.com.br/10_papeis_lider_0_0.mp4
http://www.thiagoprado.com.br/10_papeis_lider_700_700.mp4

[Dark Shikari]

I checked out the first video; it looks fine to me. Is there any particular frame # you're having issues with?

[thiagoprado]

00:01:12;25 to 00:01:12;26
00:01:13;21 to 00:01:13;22
00:01:19;21 to 00:01:19;22

In videos vbv_500_500 and vbv_700_700, vbv_0_0 is perfect.

Thanks

[Dark Shikari]

Does the bug go away if you:

a) Don't use threads?
b) Don't use B-frames?

[lexor]

Does the bug go away if you:

a) Don't use threads?
b) Don't use B-frames?

I would add:
c) decoder is too old/problematic (get latest ffdshow from the try-outs thread)

[Dark Shikari]

I would add:
c) decoder is too old/problematic (get latest ffdshow from the try-outs thread)No, it's an encoder issue.

[shon3i]

Try this build :) http://www.mediafire.com/?3nozdw0ozn1

[Dark Shikari]

Try this build :) http://www.mediafire.com/?3nozdw0ozn1No, the fix in that build was already put into official x264.

[thiagoprado]

I did with no b-frames. Same problem.
I don't know how to disable threads. I can only set 0 (auto) or 1, 2, 3...
I also did with the suggested build and got the same problem.

I read in other post a similar problem with fade transition, but I'm having with dry cuts.

Thanks

[LoRd_MuldeR]

I don't know how to disable threads. I can only set 0 (auto) or 1, 2, 3...

Using "--threads 1" will disable multi-threading...

[thiagoprado]

1) With vbv-maxrate and vbv-bufsize 480 to 900Kbps = Problem
2) With vbv-maxrate and vbv-bufsize 480 to 900Kbps + no b-frames + no threads = Problem
3) With vbv-maxrate and vbv-bufsize 0 = Perfect
4) With vbv-maxrate and vbv-bufsize 3000Kbps (incredible high to this bitrate and out of my bandwidth) = Perfect

[thiagoprado]

I download the build x264.1113M.x64.exe and made a test with vbv-maxrate / vbv-bufsize to 900Kbps.

The same problem.

I can't understand why the Bitrate Viwer is showing a 584Kbps peak if the settings allow 900Kbps.

Does anybody has a soft to check my movies bitrate?

http://www.thiagoprado.com.br/10_papeis_lider_500_500.mp4
http://www.thiagoprado.com.br/10_papeis_lider_0_0.mp4
http://www.thiagoprado.com.br/10_papeis_lider_700_700.mp4

[shon3i]

obvious that no solution for this problem except that developers make some quick fix as in my type :) I can only recommend you to switch to mainconcept or elecard encoder because they encoder have stable VBV system.

[MasterNobody]

obvious that no solution for this problem except that developers make some quick fix as in my type :)And for this they probably would need the source sample to reproduce the bug.

thiagoprado
Can you upload the source sample?

[thiagoprado]

Movie source

http://www.thiagoprado.com.br/10_papeis_lider.mpeg

[poisondeathray]

How accurate are the "bitrate viewers" ?

Testing on the 500/500 download .mp4 (1st link provided) (you have audio included) vbv maxrate 480, vbv buffersize 480

Bitrate Viewer on .mp4 : PEAK 543
Bitrate Viewer on .h264 : PEAK 418
StreamEYE on .mp4 : error on reading
StreamEYE on .h264 : real max 494
Mediainfo on .mp4 : Max Bitrate 546

Mediainfo just reads the header info, but I think Streameye & bitrate viewer actually parse each frame, yet there is a significant difference on the raw avc stream ?

Is using bitrate viewer on the .mp4 reading both audio & video? The results would suggest so...

[shon3i]

StreamEye/Buffer Analyser tools need NAL HRD in stream to show graph

[poisondeathray]

^ Yes, the buffer analyzer wouln't work with that sample, but streameye did.

[thiagoprado]

Can you do a test with the 0_0 version.
This has no vbv-maxrate / bufsize.
I got 604Kbps (probably with audio included) with Bitrate Viewer.
Since this is the bitrate peak that x264 needs to encode with no blocks, the vbv-maxrate / bufsize at 700Kbps should be enough.

But I'm only get smooths encodes with vbv-maxrate / bufsize above 1500Kbps.

Strange...

[poisondeathray]

0_0 sample (second link)

Bitrate Viewer .mp4: max 604
Bitrate Viewer .h264: max 525
Streameye .h264: max 583

[thiagoprado]

I realy can't understand why x264 have problems with vbv-maxrate / bufsize at 700Kbps if the uncaped peak is only 604Kbps with audio.

[Dark Shikari]

I realy can't understand why x264 have problems with vbv-maxrate / bufsize at 700Kbps if the uncaped peak is only 604Kbps with audio.Because the VBV algorithm is complicated and if it makes a mistake, it can result in the encoder thinking that that the VBV may underflow soon--when it actually isn't going to. It relies on predictions of frame size that have to be very accurate.

[LoRd_MuldeR]

...because VBV doesn't only restrict the peak bitrate, but also the buffer size ???

That means you are allowed to exceed the average bitrate (up to the max. peak bitrate), but only for a very short time. Otherwise the buffer would underflow.

Without VBV there's no such buffer restriction at all.

[thiagoprado]

Is there any way to fix this?
I'm worried to leave the VBV uncaped and get problems with streaming bandwidth.

Thanks

[Dark Shikari]

Don't use threads until we fix it?

[thiagoprado]

I did tests with all the encoders I found...

1) ProCoder - Terrible
2) Compressor (Apple H.264) - Terrible
3) Compressor (x264 encoder) - Reasonable
4) Sorenson (MainConcept) - Reasonable
5) Mainconcept - Reasonable
6) Hanbrake (x264) - Excelent but with a terrible audio
7) Mediacoder (x264) The best!!! But with this small problem :confused:

[thiagoprado]

I already set threads to 1 and got the same problem...

[LoRd_MuldeR]

1) ProCoder - Terrible
2) Compressor (Apple H.264) - Terrible
3) Compressor (x264 encoder) - Reasonable
4) Sorenson (MainConcept) - Reasonable
5) Mainconcept - Reasonable
6) Hanbrake (x264) - Excelent but with a terrible audio
7) Mediacoder (x264) The best!!! But with this small problem :confused:

All tools that are based on x264 (3, 6 and 7) will give identical results as long as you are using the same encoder settings.

The result depends on the encoder itself and its settings, not on the GUI wrapped around it...

[Dark Shikari]

I already set threads to 1 and got the same problem...Now that's interesting--this is the first VBV bug I have seen in a long time that occurs even with threads and B-frames off. I will look into debugging this tonight or tomorrow.

If you are using two-pass, you may be able to get away with commenting out line 933 in encoder/ratecontrol.c:

if( h->mb.i_mb_x != h->sps->i_mb_width - 1 || !rc->b_vbv)

This completely disables all row-based ratecontrol and relies solely on 2pass VBV for VBV accuracy. It is not guaranteed to avoid underflow, but it should eliminate your problem for now.

[thiagoprado]

The Compressor has a terrible deinterlace and a nice audio. Handbrake use the same x264 but I can't get good audio with FAAC.

The MediaCoder has everything I need: Barch Encode, excelent deinterlace, Nero AAC and th excelenet x264.

But has this problem with VBV...

[LoRd_MuldeR]

The Compressor has a terrible deinterlace and a nice audio. Handbrake use the same x264 but I can't get good audio with FAAC.

Use the x264 CLI encoder with Avisynth input and you can use whatever deinterlace filter you prefer :)

[thiagoprado]

Now that's interesting--this is the first VBV bug I have seen in a long time that occurs even with threads and B-frames off. I will look into debugging this tonight or tomorrow.

If you are using two-pass, you may be able to get away with commenting out line 933 in encoder/ratecontrol.c:

if( h->mb.i_mb_x != h->sps->i_mb_width - 1 || !rc->b_vbv)

This completely disables all row-based ratecontrol and relies solely on 2pass VBV for VBV accuracy. It is not guaranteed to avoid underflow, but it should eliminate your problem for now.


How can I do this? Get the original code and then build x264?

[Dark Shikari]

How can I do this? Get the original code and then build x264?Sure, or ask someone to build such a modified version for you.

A proper description of this modification would be "disable row-based VBV handling".

[LoRd_MuldeR]

How can I do this? Get the original code and then build x264?

Exactly.

[thiagoprado]

Use the x264 CLI encoder with Avisynth input and you can use whatever deinterlace filter you prefer :)

The problem is that I need bach encode. I have 700 movies to encode :eek:

[LoRd_MuldeR]

The problem is that I need bach encode. I have 700 movies to encode :eek:

Ever heard of batch/script files?

Also you can use a GUI like MeGUI, which uses Avisynth input too, offers job control (batch processing) and offers all options you'll need.

[thiagoprado]

Ever heard of batch/script files?

Also you can use a GUI like MeGUI, which uses Avisynth input too, offers job control (batch processing) and offers all options you'll need.

Since this solution still use x264 I will get same problem? Correct?

[LoRd_MuldeR]

Since this solution still use x264 I will get same problem? Correct?

Yes, as said before: All tools that use the x264 encoder will give identical output for identical input (and identical encoder settings).

However Avisynth allows you to use the pre-processing filters of your choice, including your preferred deinterlacer ;)

[thiagoprado]

Sure, or ask someone to build such a modified version for you.

A proper description of this modification would be "disable row-based VBV handling".

I will try to build myself.

With this patch the vbv-maxrate / bufzie will still work?
I will be able to set the x264 to use VBV at 250Kbps with maximum 500Kbps?

Thanks

[Dark Shikari]

I will try to build myself.

With this patch the vbv-maxrate / bufzie will still work?Yes, but it will only be remotely reliable in 2-pass mode.

[J_Darnley]

If you don't have a build environment already setup, here is a build with Dark Shikari's change: http://users.telenet.be/darnley/x264/x264-thiagoprado.7z

[Guest]

The VBV bufsize is ridiculously low. It has no relation to bitrate. It's determined by the HW of your player. Find out what your player has and set it appropriately.

[Dark Shikari]

The VBV bufsize is ridiculously low. It has no relation to bitrate. It's determined by the HW of your player. Find out what your player has and set it appropriately.A 1 second VBV is unreasonable?

[thiagoprado]

Worked!!! Worked!!! :)

Thanks!!!!

[thiagoprado]

I thought bufsize is the time in seconds maxrate can be used.

Ex: maxrate 500 / bufzise 500 - One second of 500Kbps peak
maxrate 500 / bufsize 1000 - Two seconds of 500Kbps peak

I'm using JW Player with Flash Media Streaming 3.5

[thiagoprado]

Thanks Dark Shikari for the help and for the build.

It's incredilble how fast a problem was fixed.

I bought the Flash Media Server ($990) and I still don't get an awnser for a question posted one month ago.

[Dark Shikari]

I thought bufsize is the time in seconds maxrate can be used.

Ex: maxrate 500 / bufzise 500 - One second of 500Kbps peak
maxrate 500 / bufsize 1000 - Two seconds of 500Kbps peak

I'm using JW Player with Flash Media Streaming 3.5Are you actually streaming the H.264, or just using progressive download?

If you're using progressive download, you don't need VBV.

[thiagoprado]

Are you actually streaming the H.264, or just using progressive download?

If you're using progressive download, you don't need VBV.

Streaming

[Guest]

A 1 second VBV is unreasonable? I haven't looked at the particulars of his stream. I was just saying that it is small compared to what typical HW provides.

[Dark Shikari]

StreamingIf you're encoding beforehand and not in real-time, why do you need to stream instead of progressive download?

IMO progressive download avoids most of these annoyances and is much easier for both the user and server.

[thiagoprado]

Security, time seek and bandwidth control.

It's a pay-per-view site.

[Dark Shikari]

Security, time seek and bandwidth control.

It's a pay-per-view site.Oh, in that case, of course it makes sense.

[thiagoprado]

I did a test video using the Fabio Sonnati's settings (the same I posted in the beginning).

The best I could find to stream video.

It's realy amazing the quality we can get with only 250Kbps.

Look at the video:

http://www.thiagoprado.com.br/teste.mp4

[Dark Shikari]

I did a test video using the Fabio Sonnati's settings (the same I posted in the beginning).

The best I could find to stream video.

It's realy amazing the quality we can get with only 250Kbps.

Look at the video:

http://www.thiagoprado.com.br/teste.mp4You can still do better. Raise subme to 9 for starters (no matter what you do, this is worth it), and if you feel like wasting a ton of time, you can use --ref 16 --me tesa as well. Also, given how much your source used 5 bframes, you might benefit from raising --bframes a bit. If instead of wasting tons of time like the previous suggestions and want to save time without too much quality loss, drop trellis from 2 to 1.

x264 is good stuff ;)

[thiagoprado]

You can still do better. Raise subme to 9 for starters (no matter what you do, this is worth it), and if you feel like wasting a ton of time, you can use --ref 16 --me tesa as well. Also, given how much your source used 5 bframes, you might benefit from raising --bframes a bit. If instead of wasting tons of time like the previous suggestions and want to save time without too much quality loss, drop trellis from 2 to 1.

x264 is good stuff ;)

My god! It's better. The image is sharper.

x264 is incredible

[Dark Shikari]

Also, you could try some psy-trellis; --psy-rd 1.0:0.2 will add sharpen things up (but tends to do so at the cost of slightly increased artifacting)

[thiagoprado]

Also, you could try some psy-trellis; --psy-rd 1.0:0.2 will add sharpen things up (but tends to do so at the cost of slightly increased artifacting)

I will try

[thiagoprado]

Also, you could try some psy-trellis; --psy-rd 1.0:0.2 will add sharpen things up (but tends to do so at the cost of slightly increased artifacting)

I did the test and the result is incredible but took a long time to encode.
Since my videos are basically one person talking at low motion scenes the 250Kbps are fine.
But in some motion scenes the image get blocky.

Is there any setting in x264 that can avoid blocky?

Thanks

[Dark Shikari]

I did the test and the result is incredible but took a long time to encode.
Since my videos are basically one person talking at low motion scenes the 250Kbps are fine.
But in some motion scenes the image get blocky.

Is there any setting in x264 that can avoid blocky?What makes it blockier in higher motion scenes is the fact that you're limiting the bitrate using VBV--if those scenes could have higher peak bitrate, they'd get higher quality.

Also, using such a low deblock setting isn't helping either.

[thiagoprado]

What makes it blockier in higher motion scenes is the fact that you're limiting the bitrate using VBV--if those scenes could have higher peak bitrate, they'd get higher quality.

Also, using such a low deblock setting isn't helping either.

Thanks

[Guest]

Because the VBV algorithm is complicated and if it makes a mistake, it can result in the encoder thinking that that the VBV may underflow soon--when it actually isn't going to. It relies on predictions of frame size that have to be very accurate. So if you can use a larger VBV buffer as I suggested (not maxrate), wouldn't this tend to mitigate the problem?

There's no reason to minimize the VBV buffer size. You just want it no bigger than the player's HW is providing.

[Dark Shikari]

So if you can use a larger VBV buffer as I suggested, wouldn't this tend to mitigate the problem?Theoretically, but such bugs (depending on the cause) can be triggered merely by VBV being on regardless of actual VBV state.There's no reason to minimize the VBV buffer size. You just want it no bigger than the player's HW is providing.But he's not encoding for hardware--he's encoding for a software stream.

If he has a buffer of, for example, 10 seconds, that means you have to wait 10 seconds before you can tune in to the stream.

[Manao]

His gop is already 10 second long. I think he can afford a slightly longer VBV. Furthermore, in the OP, why --level 51 ?

[Dark Shikari]

His gop is already 10 second long.Ah you're right. In this case he could certainly get away with more VBV.

[thiagoprado]

Thanks, Guys

I increased the maxrate to 700 and the bufsize two 1400 (two seconds of 700Kbps peak).

Since I'm using a prebuffering of 5 seconds and the flash media server 3.5 now have the dynamic buffering, I think will be fine.

What is GOP?

[qbukhari]

Very impressive, Excellent work all of you

I am a mediacoder user too but a newbie I have learnt a lot from you guys. Hats Off !!!!

I am dyeing to know how to use these settings in mediacoder

# ".\codecs\x264.exe" "$(SourceFile)" --no-psnr --no-ssim --keyint 250 --min-keyint 25 --level 51 --vbv-maxrate 480 --vbv-bufsize 480 --me umh --merange 16 --no-fast-pskip --non-deterministic --subme 1 --ref 1 --partitions all --8x8dct --bframes 5 --direct auto --mixed-refs --trellis 2 --b-pyramid --weightb --deblock -2:-1 --bitrate $(VideoBitrate) --qcomp 0.6 --vbv-bufsize 480 --threads auto --stats "$(PassLogFile)" --pass 1 -o NUL

# ".\codecs\x264.exe" "$(SourceFile)" --no-psnr --no-ssim --keyint 250 --min-keyint 25 --level 51 --vbv-maxrate 480 --vbv-bufsize 480 --me umh --merange 16 --no-fast-pskip --non-deterministic --aq-mode 1 --aq-strength 1.0 --b-adapt 2 --ref 6 --subme 7 --psy-rd 1:0 --partitions all --8x8dct --bframes 5 --direct auto --mixed-refs --trellis 2 --b-pyramid --weightb --deblock -2:-1 --bitrate $(VideoBitrate) --qcomp 0.6 --vbv-bufsize 480 --threads auto --stats "$(PassLogFile)" --pass 2 -o "$(DestFile)"


how to enter these exact settings in my mediacoder ? a stupid question but please guide me.


Thanks

Bukhari
qbukhari@yahoo.com