Ridiculous peaks with mb-tree

[Chengbin]

I've noticed that when encoding videos with mb-tree, there are ridiculous peaks, like a 150Mbps peak for a 1920x816 video with an average bitrate of 10000Kbps for a scene with artificial noise (TV statics). This is Monsters Vs Aliens, 1:05:51

I've seen others, like a 13Mbps peak for a CRF 24 video @ 688x288.

Is that normal?

[thewebchat]

Well, if you don't apply a VBV restriction, then there is no expectation on the bitrate of the stream to stay within any particular bounds.

[LoRd_MuldeR]

Also "artificial noise" basically is random data and thus is extremely hard to compress. So it makes sense that the encoder moves a lot of bits to that sequence.

If you didn't limit the maximum bitrate via VBV and if the target average bitrate was hit correctly, I see no reason for concern...

[Chengbin]

I know artificial noise is very difficult to compress, but I don't see non mbtree encodes with such high peaks...

That's why I'm concerned.

I'm seeing high peaks with mbtree for frames that last for less than a fraction of a second (usually in flashbacks). Why waste so much bits on frames that you're not gonna see?

[LoRd_MuldeR]

Obviously MB-Tree RC does a better job to move bits to locations where they are actually needed ;)

Remember: The goal is always to retain good quality throughout the entire movie. And parts that are "hard to compress" (like random noise) will obviously need a lot of bits to retain good quality!

So if you want to avoid that the "artificial noise" sequence eats that many bits, you must denoise. If you would simply lower bitrate in that sequence, then the noise would be blurred/smeared out anyway.

However I would expect that MB-Tree RC tends to assign a lower bitrate to "flash" frames, because those frames have near-zero impact on future frames :confused:

[Chengbin]

So if you want to avoid that the "artificial noise" sequence eats that many bits, you must denoise. If you would simply lower bitrate in that sequence, then the noise would be blurred/smeared out anyway.

I can't do that. Even if there is a denoiser so strong that would remove that noise, it would just be blanks.

However I would expect that MB-Tree RC tends to assign a lower bitrate to "flash" frames, because those frames have near-zero impact on future frames :confused:

That's what I thought!

[Fr4nz]

I don't know if the problem I encounter is related to what says Chengbin, anyway:

Yesterday I bought a copy of "X-Men Origins: Wolverine" and I was doing some tests with x264, because at the beginning of the film there are scenes with very very strong noise (Wolverine and his brother during the civil, 1st, 2nd and Vietnam war to be precise), and I was interested to see how x264 performed in these.

Well, the problem is that, altough I use Profile "High" and
"Level 4"[1], when I play-back the encoded video in my blu-ray standalone player in those scenes the player skips, as if the bitrate and buffer restrictions are brutally violated.

This shouldn't happen with latest improvements, right? I'm using the last 1271 version.

----
[1] The complete command-line is:
program --profile high --level 4 --preset slow --tune film --pass 2 --bitrate 5090 --stats ".stats" --thread-input --output "output" "input"

[Dark Shikari]

bitrate and buffer restrictions are brutally violated.But you didn't set any bitrate and buffer restrictions.

[Fr4nz]

But you didn't set any bitrate and buffer restrictions.

Shouldn't be set implicitly with "--level 4" parameter? Or is that only a flag?

[Dark Shikari]

Shouldn't be set implicitly with "--level 4" parameter? Or is that only a flag?It's only a flag. If you want to enable VBV, you need to specifically turn it on.

In your case, --vbv-maxrate 25000 --vbv-bufsize 31250.

[Fr4nz]

It's only a flag. If you want to enable VBV, you need to specifically turn it on.

In your case, --vbv-maxrate 25000 --vbv-bufsize 31250.

Got it, thanks DS...damn, the latest version of MeGUI is a little borked with the "advanced settings" :\

[burfadel]

Try the --nr option. Note that PSNR and SSIM will always drop using it since it changes the video at the encoder end, not at the source end for the encoder! Easy thing is to go by how it looks, not how the PSNR/SSIM tells you how it looks :) Try a value of say, 900 for a start and work up from there. Setting it too high is bad, but 900 is pretty 'safe' :) Highest setting I believe is 65536, as was explained to me in a different thread.

[Fr4nz]

Try the --nr option. Note that PSNR and SSIM will always drop using it since it changes the video at the encoder end, not at the source end for the encoder! Easy thing is to go by how it looks, not how the PSNR/SSIM tells you how it looks :) Try a value of say, 900 for a start and work up from there. Setting it too high is bad, but 900 is pretty 'safe' :) Highest setting I believe is 65536, as was explained to me in a different thread.

I'm already using fft3dgpu as noise filter and it does a very very good work :)

[Fr4nz]

It's only a flag. If you want to enable VBV, you need to specifically turn it on.

In your case, --vbv-maxrate 25000 --vbv-bufsize 31250.

Hi DS, I'm still encountering skipping problems with X-Men encoded played on my blu ray player (which is a Samsung bd-1400p) on those scenes in which there are a lot of noise.

The command-line i've used is:

program --profile high --level 4.1 --pass 2 --bitrate 5090 --stats ".stats" --thread-input
--tune film --preset slow --vbv-maxrate 25000 --vbv-bufsize 31250 --output "output" "input"

As you can see I've used the parameters with the values you have suggested.

Do you have any other idea?

[Dark Shikari]

Hi DS, I'm still encountering skipping problems with X-Men encoded on my blu ray player (which is a Samsung bd-1400p) on those scenes in which there are a lot of noise.

The command-line i've used is:

program --profile high --level 4.1 --pass 2 --bitrate 5090 --stats ".stats" --thread-input
--tune film --preset slow --vbv-maxrate 25000 --vbv-bufsize 31250 --output "output" "input"

As you can see I've used the parameters with the values you have suggested.

Do you have any other idea?Then clearly your Blu-ray player isn't up to spec. If x264 didn't say "VBV underflow", then it obeyed the VBV.

And the restrictions I gave you were level 4, not level 4.1.

[Fr4nz]

Then clearly your Blu-ray player isn't up to spec. If x264 didn't say "VBV underflow", then it obeyed the VBV.

And the restrictions I gave you were level 4, not level 4.1.

Samsung...anyway, the original blu-ray is played without problems, that is strange...could you suggest me lower values that are reasonable?

[Dark Shikari]

Samsung...anyway, the original blu-ray is played without problems, that is strange...could you suggest me lower values that are reasonable?Well, remember, you're still not abiding by the Blu-ray spec here, via keyframe interval or NAL-HRD. Maybe lack of NAL-HRD is a problem? (It really shouldn't be...)

But feel free to lower the VBV params as much as you want.

[Fr4nz]

Well, remember, you're still not abiding by the Blu-ray spec here, via keyframe interval or NAL-HRD. Maybe lack of NAL-HRD is a problem? (It really shouldn't be...)

Don't know, you're the expert :)
Anyway, if needed, I could do some tests on my damn player in order to clarify the problem...just tell me :)


But feel free to lower the VBV params as much as you want.

Okay...do you think that 25000/25000 for both parameters should be safe?

EDIT: Just a guess...maybe the standalone player simply isn't able to sustain a read speed of ~30mbps on a DVD?

[Dark Shikari]

EDIT: Just a guess...maybe the standalone player simply isn't able to sustain a read speed of ~30mbps on a DVD?I think this is the first time you've mentioned you were reading this off a DVD ;)

[Fr4nz]

I think this is the first time you've mentioned you were reading this off a DVD ;)

Yes, right, I recorded my test on a DVD because, you know, doing tests on blank BD media would be quite expensive at the moment.

Anyway, keep in mind that I have the original blu-ray, so don't worry I'm not a pirate ;)

In conclusion: do you think the DVD is the culprit? 30mbps DVD read speed shouldn't be hard to achieve on modern standalone players (although out of spec for MPEG2 reading on DVDs...).

[Sharc]

jdobb's BD-Rebuilder sets --vbv-maxrate 17500 and --vbv-bufsize 14500 for 'BD5'. Perhaps you may want to try with these settings.
You might also find the handbrake calculator useful:
http://handbrake.dynaflashtech.net/cgi-bin/vbv_calculator.cgi

[Fr4nz]

jdobb's BD-Rebuilder sets --vbv-maxrate 17500 and --vbv-bufsize 14500 for 'BD5'. Perhaps you may want to try with these settings.
You might also find the handbrake calculator useful:
http://handbrake.dynaflashtech.net/cgi-bin/vbv_calculator.cgi

Thanks for the informations Sharc. I will try your suggestions.

[shon3i]

VBV-bufsize for Blu-Ray compilancy can be max 30000 and less (from BD specs) not 31250 as Dark Shikari suggest, aslo vbv-maxrate should be equal or greater than buffersize to avoid buffer underflows. vbv-bufsize 29999 and less is better to leave room for leak. There is no restrictions for BD 5/9, and --vbv-maxrate 30000 and --vbv-bufsize 30000 work fine with DVD media. It's better to have higher buffer and maxrate than bitrate to avoid overflows/underflows generaly.

[Fr4nz]

VBV-bufsize for Blu-Ray compilancy can be max 30000 and less (from BD specs) not 31250 as Dark Shikari suggest, aslo vbv-maxrate should be equal or greater than buffersize to avoid buffer underflows. vbv-bufsize 29999 and less is better to leave room for leak. There is no restrictions for BD 5/9, and --vbv-maxrate 30000 and --vbv-bufsize 30000 work fine with DVD media. It's better to have higher buffer and maxrate than bitrate to avoid overflows/underflows generaly.

Shon3i, in theory you're right but...begin to read from this (http://forum.doom9.org/showpost.php?p=1331155&postcount=7) post to the last.

Probably the standalone player I use is somewhat speed-limited when reading DVDs.

EDIT: I think I've found the answer. From this (http://forum.doom9.org/showpost.php?p=1167471&postcount=193) jdobbs post:

When you're writing to DVD-R (BD-5/BD-9) you have to follow the AVCHD specifications -- and that means you can't assume anything greater than 2X spin speed. I also think that is where the 3 B-Frame limit came from too, but I'd have to check. With AVC 17,500 Kbs is plenty of bitrate for 1920x1080. As Frank mentioned, 15000 would probably be just as good...

[shon3i]

Shon3i, in theory you're right but...begin to read from this post to the last.Yes, but i don't saw you use compatible switches in your commandline?

Preset slow use --ref 5 which is out of bd specs too. max for 1080p is 4 and for 720p is 6. Aslo you need to use patched x264 buld with nal-hrd support i recomend you to use this last VRF maniac build from here http://forum.doom9.org/showthread.php?p=1329334#post1329334, not to leave tsmuxer or h264info to insert hrd info. After encoding use tool such as Elecard Buffer Analyser to check HRD VBV to see is there over/underflows. And if you can mux with scenarist.

EDIT: i just reread whole thread. if you go with level 4, vbv-maxrate can't exceed 24000 and vbv-buffersize 30000 that is from BD specs. If you go with level 4.1 buffersize max is alslo 3000, max rate can up to 40000, but you need aslo in commandline put --slices 4 for 100% compilant stream

Probably the standalone player I use is somewhat speed-limited when reading DVDs.
Fr4nz is offline Report Post Reply With QuoteAgree but DVD @ 3X is just enoguht for VBV @ 30000

[Fr4nz]

Yes, but i don't saw you use compatible switches in your commandline?

Preset slow use --ref 5 which is out of bd specs too. max for 1080p is 4 and for 720p is 6. Aslo you need to use patched x264 buld with nal-hrd support, not to leave tsmuxer or h264info to insert hrd info. After encoding use tool such as Elecard Buffer Analyser to check HRD VBV to see is there over/underflows. And if you can mux with scenarist.

Agree but DVD @ 3X is just enoguht for VBV @ 30000

Actually the outputted encoding has 4 ref frames because I've forced level and profile. For the rest (nal-hrd), as DS said that shouldn't be the culprit.

[shon3i]

BD 5/9 and AVCHD are not nearly same so that means VBV restrictions are not same, and BD specs are alow to use max vbv-bufsize 30000 for all media.

as DS said that shouldn't be the culprit. Wrong. HRD control decoding proces, if HRD is broken decoding is broken. Tsmuxer cant propertly rewrite HRD info, you need to use pached build.

Aslo you can use for example --vbv-bufsize 15000 and vbv-maxrate 15000.

Then clearly your Blu-ray player isn't up to spec. If x264 didn't say "VBV underflow", then it obeyed the VBV.That is because HRD look here (http://forum.doom9.org/showthread.php?t=149601) i have same problem VBV is fine but HRD VBV represent whole different thing.

[shon3i]

Ok Fr4nz can i suggest you these settnigs to try?

--bframes 3
--level 4.1
--ref 4
--slices 4
--vbv-bufsize 30000
if you use --vbv-maxrate 30000
then use --keyint 24
end if you use --vbv-maxrate 15000
then you can use --keyint 48, and aslo you can down --vbv-bufsize to 15000 to avoid buffer underflow
--sar 1:1
--mvrange 511
--aud
--nal-hrd

and no --b-pyramid at moment until not fixed.

And x264 patched build from here (http://forum.doom9.org/showthread.php?p=1329334#post1329334)

You can change all other settings but keep in mind that not broke settings i wrote.

Aslo when muxing with TSMuxer uncheck option Add Picture Structure to prevent tsmuxer to rewrite HRD.

[Fr4nz]

Wrong. HRD control decoding proces, if HRD is broken decoding is broken. Tsmuxer cant propertly rewrite HRD info, you need to use pached build.

At this point I think you're right Shon3i because i tried with maxrate 17500 and bufsize 30000: my blu-ray player still skips in the "incriminated" X-Men scenes.

Now I'm trying the latest Jeeb 1271 nal-hrd patched build: I'll try to set maxrate to 25000, bufsize to 20000 and use the nal-hrd switch; let's see if something changes now...

PS: I think for my encodings I will not use aud, sar, mvrange and keyint switches, I think they are useless...if not, please, tell me why :)
The command line I'm going to use (under MeGUI) is this:

program --profile high --level 4.1 --pass 2 --bitrate 5090 --stats ".stats" --thread-input
--tune film --preset slow --vbv-maxrate 25000 --vbv-bufsize 20000 --slices 4
--nal-hrd --output "output" "input"

[rack04]

Ok Fr4nz can i suggest you these settnigs to try?

--level 4.1
--ref 4
--slices 4
--vbv-bufsize 30000
if you use --vbv-maxrate 30000
then use --keyint 24
end if you use --vbv-maxrate 15000
then you can use --keyint 48, and aslo you can down --vbv-bufsize to 15000 to avoid buffer underflow
--sar 1:1
--mvrange 511
--aud
--nal-hrd

and no --b-pyramid at moment until not fixed.

And x264 patched build from here (http://forum.doom9.org/showthread.php?p=1329334#post1329334)

You can change all other settings but keep in mind that not broke settings i wrote.

Aslo when muxing with TSMuxer uncheck option Add Picture Structure to prevent tsmuxer to rewrite HRD.

Don't forget --bframes 3. I would suggest leaving --mvrange to the default 511.75.

[prOnorama]

Don't forget --bframes 3.

Isn't bframes=3 already the standard setting for all presets between and including --veryfast and --slower ?

http://forum.doom9.org/showthread.php?t=148149

[rack04]

Isn't bframes=3 already the standard setting for all presets between and including --veryfast and --slower ?

http://forum.doom9.org/showthread.php?t=148149

Yes but if you use --tune animation it will increase the number of bframes.

[shon3i]

think for my encodings I will not use aud, sar, mvrange and keyint switches, I think they are useless...if not,Hell no :) they are not usless. Aud is need for 100% blu-ray compilant stream. SAR is need for singnal aspect ratio, but must be present in stream for full compilancy, mvrange control motion vectors should not be greater than 511.75, but i like to be safe and avoid out of range. And keyint are most important thing. Blu-Ray allow 1 second GOP if maxbitrate is over 15mbps so max keyint should never exceed 24. If maxbitrate is less and equal to 15mbps 2 second GOP is alowed aslo called Long GOP. Default of x264 is 250 which is to high. You can use default x264 settings but that is out of any Blu-Ray specs, and may cause "bad" playback

Don't forget --bframes 3. I would suggest leaving --mvrange to the default 511.75. Yes, --bframes 3 are aslo very important thing. Thanks

[Fr4nz]

Hell no :) they are not usless. Aud is need for 100% blu-ray compilant stream. SAR is need for singnal aspect ratio, but must be present in stream for full compilancy, mvrange control motion vectors which not be greater than 511.75, but i like to be safe and avoid out of range. And keyint are most important thing. Blu-Ray allow 1 second GOP if maxbitrate is over 15mbps so max keyint should never exceed 24. If maxbitrate is less and equal to 15mbps 2 second GOP is alowed aslo called Long GOP. Default of x264 is 250 which is to high. You can use default x264 settings but that is out of any Blu-Ray specs.

Well, at the moment I'm interested only in avoiding the "skipping" issue during the playback, which is caused by buffer problems as we said.

So, I think the most important parameters now are "bufsize", "maxrate" and "nal-hrd" in order to correct this problem...correct me if I'm wrong.

For the full-compliancy we have time :)

[shon3i]

So, I think the most important parameters now are "bufsize", "maxrate" and "nal-hrd" in order to correct this problem...correct me if I'm wrong.Theoretically, keep in mind that aud is need aslo for hrd

[rack04]

Aslo when muxing with TSMuxer uncheck option Add Picture Structure to prevent tsmuxer to rewrite HRD.

Are you referring to contSPS?

[shon3i]

Are you referring to contSPS?
I think both, insertSEI and contSPS. I'm not sure but I think in present versions of tsmuxer these two options are automaticly disabled when stream conatin HRD, but i don't have developer confirmation.