Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
15th August 2018, 21:34 | #1 | Link |
Registered User
Join Date: Jul 2003
Location: Italy
Posts: 1,135
|
Abnormal bitrate and quality drop
Using x264 on blurays (high bitrates and vbv)
Under certain unknown circumstances, there are significant bitrates and very visible quality drop on scene changes. Generally, it appears more often when the previous scene is quite complex with some particles effects (snow falling, fire crackles, and so on...). The bitrate drop is very pronunced, it can be even 5x lower than expected. The problem propagates for no more than a gop (on bluray it translates to 1 second), infact, the *same* scene, after that second, "regain" its bitrate and continue with good quality. I'm not even able to fix it with the use of zones, because with zones I can fix that scene a bit (albeit x264 totally ignore my bitrate multiplier but still it raise the bitrate in a decent way) but the problem is then "shifted", and just after the zone ends, the same bitrate drop appear. This is an example: As you can see here, after a scene change (new scene), the bitrate drops enormously (notice the next six B/P frames with their very low size). Then at the next gop (same scene I frame), the bitrate returns to its right value and so the quality. The quality in that zone is really low, with lots of compression artefacts... and the average bitrate of the stream should be 33 Mbps. Also, I would like to point out that it's not happening with a particular x264 release... I already encoutered this type of problem with elder x264 revisions. Depending on the footage, on a general 2 hrs movie this may happens 3-4 times. With the movie I'm working on now (action anime), it's happening 15 times Last edited by mp3dom; 15th August 2018 at 21:50. |
16th August 2018, 19:58 | #3 | Link |
Registered User
Join Date: Jul 2003
Location: Italy
Posts: 1,135
|
The encoder settings are fairly standard for bluray production:
Code:
--bitrate 33000 --preset veryslow --tune film --bframes 2 --bluray-compat --vbv-maxrate 38000 --vbv-bufsize 30000 --level 4.1 --no-fast-pskip --no-dct-decimate --keyint 24 --open-gop --slices 4 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 |
17th August 2018, 13:45 | #4 | Link |
Registered User
Join Date: Jul 2003
Location: Italy
Posts: 1,135
|
Just a quick update: I already changed lots of settings including qcomp, qpstep and others even in an extreme way just to see if something happens, and I guess the culprit is the VBV management. The *only* way I found so far to fix the problem is just to eliminate the VBV management (which of course is not possible for bluray).
In the example I posted above, the "previous scene" without VBV on crf 9 reaches 100 Mbps (it's a complex scene with lots of added grain). The scene just after that, without VBV is around 35-40 Mbps (less grain and less complexity) and it's encoded perfectly fine. But when everything is under VBV management, the 100 Mbps is then capped to 35-38 Mbps (of course, it can't go beyond that), but the scene just after that is "capped" too, to a 10-15 Mbps for the first gop, then it returns to its 30-35 Mbps. I don't know exactly how it works, but is it possible that something is wrong with vbv, that need to "flush" the buffer for one gop or something similar? Anyway it's too much visible quality wise. As I already wrote in the post, with this source the problem happens 15 times so in the actual state it's impossible for me to use x264, there's too much visible quality drop. Last edited by mp3dom; 17th August 2018 at 14:01. |
17th August 2018, 13:54 | #5 | Link |
Registered User
Join Date: Dec 2002
Posts: 5,565
|
Does the same happen with open-gop disabled, vbv enabled?
I'd probably try to cut a sample and send it to #x264dev (on freenode). The vbv-lookahead with your settings should be like 30 frames so it should see the I frame soon enough. Maybe the devs can improve the algo. |
17th August 2018, 14:06 | #6 | Link |
Registered User
Join Date: Jul 2003
Location: Italy
Posts: 1,135
|
Need to test about closed-gop (not tested yet, I'll try asap and report).
I tried to get in touch with kierank already on freenode, but unable to speak with him directly... that's why I then wrote here. |
17th August 2018, 15:02 | #7 | Link |
Registered User
Join Date: May 2006
Posts: 3,997
|
As a workaround you could try to reduce the resolution to 1440*1080, sar 4:3 (for DAR 16:9) which is still blu-ray compliant. It may reduce the bitrate of the previous scene and stress the buffer control less.
A slightly lower overall quality is less annoying than an excellent quality for the vast majority of the frames but with few visibly bad frames. It surprises me that this probelm has not been discovered before ..... Last edited by Sharc; 17th August 2018 at 15:10. |
17th August 2018, 15:35 | #8 | Link |
Big Bit Savings Now !
Join Date: Feb 2007
Location: close to the wall
Posts: 1,545
|
Back then when those 3 reference movies were encoded with x264 r1538+1 to prove Blu-ray compliance
I found the same behaviour resulting in "wet patches". x264 is trying to save bits when it should not. Since then my remedy is to tickle x264 by adding grain and not looking back. Code:
GrainFactory3(2,2,1,100,100,100,0.9,0.7,0.5,0,0,0,0,0,24,56,128,160)
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain) "Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..." Last edited by Emulgator; 17th August 2018 at 15:37. |
17th August 2018, 15:38 | #9 | Link |
Registered User
Join Date: Dec 2002
Posts: 5,565
|
Btw, how does the scene change look? Is it an abrupt cut or some kind of fade?
IIRC the devs recommended using something like --tune grain for such high bitrates anyways as it would more evenly distribute bits. (--tune grain has lower --qcomp and --aq-strength than default and film.) Maybe you can mitigate the problem a bit that way for the time being. |
17th August 2018, 16:04 | #10 | Link |
Registered User
Join Date: Jul 2003
Location: Italy
Posts: 1,135
|
I'm testing the closed-gop now.
BTW, reducing the resolution and going "anamorphic" (assuming this can solve the problem) is out of question mainly because the movie is supposed to be full 1920x1080 and you can't even justify this commercially above the fact that all over the world this was released as 1920x1080. In the worst case I prefer to simply change the encoder and keep the full resolution. I don't even think it has to do with grain preservation or something "grain-related" mainly because with zones I can fix that part but then the problem is just shifted to when the zone ends regardless of the type of the scene (i.e. it appears with the same problem afterwards). Regarding the scene change, it's a complete scene change, no fades or something similar. x264 correctly puts the I frame there, I also tried to specify an IDR rather than standard I, but the result is the same. Also tried to set qcomp way higher or way lower than standard (as an extreme) but nothing changed. I can't vary AQ-strength too much because I have also lots of smooth gradients that I need to preserve. With such high bitrates and the footage I'm working on, I found 0.8 to be a good compromise. Edit: Closed GOP+VBV didn't change the result Last edited by mp3dom; 17th August 2018 at 16:57. Reason: update |
17th August 2018, 19:04 | #11 | Link | |
Registered User
Join Date: Oct 2002
Location: France
Posts: 2,316
|
Quote:
Also it occurs only when VBV is used, which is probably "only" almost in blu-ray targeting. My guess, is more than 95% of x264 users are only doing one pass crf encode without using VBV. So, finaly, after thinking, maybe a little less surprising (but still a little...).
__________________
My github. |
|
18th August 2018, 02:44 | #14 | Link |
Registered User
Join Date: Feb 2002
Location: San Jose, California
Posts: 4,407
|
It really does seem like a bug or failing in the VBV logic. Somehow it thinks the buffer is empty when it shouldn't?
__________________
madVR options explained |
18th August 2018, 09:30 | #15 | Link | |
Registered User
Join Date: May 2006
Posts: 3,997
|
Quote:
You could also try to PM the developers. |
|
18th August 2018, 10:15 | #16 | Link |
Registered User
Join Date: Jul 2007
Posts: 552
|
1) it is for sure VBV problem;
2) to fix it developers will need source samples (preferably short one for quick problem reproduction); 3) to mitigate this problem you can try to lower encoder threads count (the lower the better; so you can try even --threads 1) to help VBV (less prediction and faster adaptation). |
18th August 2018, 20:36 | #17 | Link | |
Moderator
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,770
|
Quote:
The first thing I'd try is a 2-pass encode, which should fix any rate control oddities from a first pass. With --keyint 24, -rc-lookahead will be capped at 24. One thing to try, on a hunch that --slices are used very rarely and could have weird quality impacts, is to encode as --level 4.0 --bitrate 25000 --slices 1. If you stick to Level 4.0 in BD, you don't need slices. You could even try --level 4.0 --bitrate 15000 --slices 1 --keyint 48. If you have a max bitrate of 15M, you can use 2-second GOPs, which would offer a lot more flexible scene cut/IDR placement. I don't know if either would be a net improvement in quality, but that would help narrow down the source of the issue. |
|
18th August 2018, 21:47 | #18 | Link |
Registered User
Join Date: Jul 2003
Location: Italy
Posts: 1,135
|
2 pass and 1 pass crf have almost the same output, and I'm not saying this from a theoretical point of view, but I've already made both encodes and checked, and the problem is there on both.
Without VBV the output is fine (4 slices included). With VBV the "bad frames" are bad as a whole, there's no discrepancy between each sliced part of the same frame (the quantizers are very similar). I can run an encode with L4.0 and 1 slice for testing purposes, but cannot limit the max bitrate to 25 Mbps on a final encode when blurays allows to go up to 40 Mbps. Also, there's also the possibility that a lower bitrate may "hide" the problem itself. I'm currently running an encode with only 1 thread, but obviously it's very slow (less than 1 fps). I think by tomorrow I should be able to report the result. |
19th August 2018, 12:08 | #19 | Link |
Registered User
Join Date: Oct 2002
Location: France
Posts: 2,316
|
As MasterNobody said, it's seem now obvious that it's a VBV problem. Are you able to provide devs a small sample which produce the problem, allowing them to analyse/check and, hopely, fix it ?
For now, it seems the only thing to do to have a real solution, i'm affraid. And if done "quick enough", and by luck, they have the possibility and time to make a fix within not too long, and they put it on their dev branch, i would be able to make a build with the fix.
__________________
My github. |
19th August 2018, 13:17 | #20 | Link |
Registered User
Join Date: Jul 2003
Location: Italy
Posts: 1,135
|
I'm able to provide only encoded parts that shows the problem. Just a part of the source is not enough, because it doesn't "trigger" the problem if encoded as "standalone", it appears only when the length is quite long but the source itself is very huge in filesize (not considering that, since it comes from a real master, I can't provide it anyway).
Edit: It appears also with threads=1 Last edited by mp3dom; 19th August 2018 at 13:40. Reason: Update |
Thread Tools | Search this Thread |
Display Modes | |
|
|