Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 14th October 2013, 11:21   #1  |  Link
kabelbrand
Compression mode: Lousy
 
kabelbrand's Avatar
 
Join Date: Mar 2009
Location: Hamburg, Germany
Posts: 72
x264 letterbox border detail loss

When encoding videos with a top letterbox size that is not a multiple of 16 there often is some noticable loss of detail within the first few lines.
This happens especially with sources containing grain and weirdly only if the runtime is longer e.g. a trailer encode will be ok but a full feature encode isn't.

For example if you have a 1920x1080 source with a top letterbox of 132 pixels (2.35:1) the first 12 lines of active picture seem blurred or not as detailed as the rest of the picture.
The obvious solution is to shift the picture up or down a few pixels to the next macroblock border. This might be ok with a large letterbox but if you have a small letterbox like 20 pixels (1.85:1) the viewer will notice the picture is not centered.

Can you think of any other solutions to this problem?

Settings: --preset slower --keyint 24 --slices 4 --bitrate 30000 --vbv-maxrate 40000 --vbv-bufsize 30000 --level 4.1 --bluray-compat

Last edited by kabelbrand; 14th October 2013 at 16:16.
kabelbrand is offline   Reply With Quote
Old 14th October 2013, 13:24   #2  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,248
Why not crop away the black borders completely?
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊
LoRd_MuldeR is offline   Reply With Quote
Old 14th October 2013, 13:58   #3  |  Link
kabelbrand
Compression mode: Lousy
 
kabelbrand's Avatar
 
Join Date: Mar 2009
Location: Hamburg, Germany
Posts: 72
Quote:
Originally Posted by LoRd_MuldeR View Post
Why not crop away the black borders completely?
Unfortunately that's not an option since I am aiming for Blu-ray compatibility.
kabelbrand is offline   Reply With Quote
Old 14th October 2013, 14:02   #4  |  Link
detmek
Registered User
 
Join Date: Aug 2009
Posts: 463
Blu-Ray compatibility or Blu-Ray Disc authoring? For compatibility you can use crop.
detmek is offline   Reply With Quote
Old 14th October 2013, 14:06   #5  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,248
Just an idea: Might this be related to Psy-RDO/Psy-Trellis? Those try to retain the same "complexity" in each compressed block as in the original block. Now, if you look at those blocks directly on the border between the content and the "black" area, about 1/2 of the block is just 100% black, which means zero complexity in that part of the block. Overall, these blocks probably are strongly biased to look like "low complexity" blocks to the encoder. So Psy-RDO/Psy-Trellis will try to keep them "low complexity", which might explain the blurring - even in the part of the block that is not 100% black.

Other than that: Have you tried with and without MB-Tree enabled?
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊

Last edited by LoRd_MuldeR; 14th October 2013 at 14:18.
LoRd_MuldeR is offline   Reply With Quote
Old 14th October 2013, 14:23   #6  |  Link
kabelbrand
Compression mode: Lousy
 
kabelbrand's Avatar
 
Join Date: Mar 2009
Location: Hamburg, Germany
Posts: 72
Quote:
Originally Posted by detmek View Post
Blu-Ray compatibility or Blu-Ray Disc authoring? For compatibility you can use crop.
BD Authoring, so after cropping (for clean black borders) the video is padded back to full resolution.

Quote:
Originally Posted by LoRd_MuldeR View Post
Overall, these blocks probably are strongly biased to look like "low complexity" blocks to the encoder.
That makes sense. I was wondering if x264 looks at the whole macroblock to calculate the complexity or just a smaller area which might be completely black.

Quote:
Originally Posted by LoRd_MuldeR View Post
Other than that: Have you tried with and without MB-Tree enabled?
Yes. While it makes fades look a bit better there is no quality gain at the letterbox border.
kabelbrand is offline   Reply With Quote
Old 14th October 2013, 14:48   #7  |  Link
Lyris
Registered User
 
Join Date: Sep 2007
Location: Europe
Posts: 602
kabelbrand, I had this exact same problem - a very textured film with borders top and bottom, and the grain would get stripped away inside the block with the black border.

As you say, with large letterboxing you can just shift the image a little, but with thin strips top and bottom, that'd look odd.

Jason and Kieran pointed me to the solution: AQ. From my experience, the setting you need to adjust is --aq-strength. The default is 1.0. --aq-strength 0.5 or so should resolve this to a satisfactory extent, although I believe the trade-off is coarser compression in dark areas (although you'd have to crank up the brightness and/or gamma on a badly adjusted TV for this to really be a problem).

Last edited by Lyris; 14th October 2013 at 14:57.
Lyris is offline   Reply With Quote
Old 14th October 2013, 14:52   #8  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,248
Shouldn't AQ tend to improve those blocks located directly on the border?

About 1/2 of these blocks is pure black, so those blocks should be biased to appear relatively "flat". And, AFAIK, the VAQ algorithm moves more bits into "flat" blocks, whilst taking away bits from "complex" blocks...
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊

Last edited by LoRd_MuldeR; 15th October 2013 at 00:08.
LoRd_MuldeR is offline   Reply With Quote
Old 14th October 2013, 15:09   #9  |  Link
kabelbrand
Compression mode: Lousy
 
kabelbrand's Avatar
 
Join Date: Mar 2009
Location: Hamburg, Germany
Posts: 72
Quote:
Originally Posted by Lyris View Post
--aq-strength 0.5 or so should resolve this to a satisfactory extent
Thanks. I remember testing with --preset grain (includes --aq-strength 0.5) unsuccessfully before but I'll give it another try.
kabelbrand is offline   Reply With Quote
Old 14th October 2013, 16:12   #10  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,666
Re-generate the borders so that they're on mod-16 pixel boundaries (e.g. slightly crop the film, or move it up/down a few pixels) and the problem will go away.
Dark Shikari is offline   Reply With Quote
Old 14th October 2013, 16:29   #11  |  Link
kabelbrand
Compression mode: Lousy
 
kabelbrand's Avatar
 
Join Date: Mar 2009
Location: Hamburg, Germany
Posts: 72
Quote:
Originally Posted by Dark Shikari View Post
Re-generate the borders so that they're on mod-16 pixel boundaries (e.g. slightly crop the film, or move it up/down a few pixels) and the problem will go away.
I had hoped for a more elegant solution... Cropping/padding an additional 4 pixels on each side for a 2.40:1 feature might be ok but 12 pixel each for a 2.35:1 or 1.85:1 feature just feels wrong.

Anyway... Thank you all!
kabelbrand is offline   Reply With Quote
Old 15th October 2013, 04:57   #12  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,752
Quote:
Originally Posted by kabelbrand View Post
I had hoped for a more elegant solution... Cropping/padding an additional 4 pixels on each side for a 2.40:1 feature might be ok but 12 pixel each for a 2.35:1 or 1.85:1 feature just feels wrong.
Not on the sides, but top/bottom.

For example, a 1920x1080 source with 140 pixels of black on the top and bottom. 140 isn't mod16. So you could move it down slightly to have 144 on the top and 136 on the bottom, both of which are mod16* aligned.

*"1080p" is really encoded as 1920x1088 pixels with the bottom 8 of dummy data that isn't displayed. So that 136 is really 136+8=144 which is mod16. The key thing is that the bottom black bar starts at 944 pixels from the top, which is mod16.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 15th October 2013, 11:45   #13  |  Link
kabelbrand
Compression mode: Lousy
 
kabelbrand's Avatar
 
Join Date: Mar 2009
Location: Hamburg, Germany
Posts: 72
Cropping/Padding to the next multiple of 16 is what I was doing in the past and I just wondered if there was a better way.

I even tried to add an additional 4 black lines to the top and explicitly set the bitstream cropping rectangle to crop these instead of the bottom 8 lines for 1088->1080 to have that 140 letterbox aligned with macroblock boundaries but this did not work as I expected and I ended up with weird resolutions like 1920x1076.

Quote:
Originally Posted by benwaggoner View Post
The key thing is that the bottom black bar starts at 944 pixels from the top, which is mod16.
At least with decent bitrates the bottom border does not seem to be problematic with x264.

Last edited by kabelbrand; 15th October 2013 at 11:52.
kabelbrand is offline   Reply With Quote
Old 16th October 2013, 02:18   #14  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,752
Quote:
Originally Posted by kabelbrand View Post
Cropping/Padding to the next multiple of 16 is what I was doing in the past and I just wondered if there was a better way.
That should work. Just remember to count mod16 from the top.

Quote:
I even tried to add an additional 4 black lines to the top and explicitly set the bitstream cropping rectangle to crop these instead of the bottom 8 lines for 1088->1080 to have that 140 letterbox aligned with macroblock boundaries but this did not work as I expected and I ended up with weird resolutions like 1920x1076.
Your source was never really 1088. It was 1080 with 8 null lines at the bottom that contain no visual data and should be ignored.

Quote:
At least with decent bitrates the bottom border does not seem to be problematic with x264.
As long as you hit it at least mod8 you should be okay, so you are luma block aligned.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 16th October 2013, 13:18   #15  |  Link
kabelbrand
Compression mode: Lousy
 
kabelbrand's Avatar
 
Join Date: Mar 2009
Location: Hamburg, Germany
Posts: 72
Quote:
Originally Posted by benwaggoner View Post
Your source was never really 1088. It was 1080 with 8 null lines at the bottom that contain no visual data and should be ignored.
I am aware of this. If your source is 1080 and you have a 140 pixel top letterbox you could add an extra 4 black lines on top and 4 on the bottom to get 1088 lines. The top letterbox will now be aligned with a macroblock border at 144 pixels.
Then encode using "--crop-rect 0,4,0,4" which should signal the playback software to discard the top and bottom 4 lines to render a 1080 picture. Unfortunately this won't be usable for Blu-ray Disc because the specification allows cropping at the bottom only.

Other software seems to ignore "frame_crop_top_offset" in the SPS as well: PowerDVD, VLC, DGIndexNV

Last edited by kabelbrand; 16th October 2013 at 13:41.
kabelbrand is offline   Reply With Quote
Old 16th October 2013, 20:51   #16  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,752
Quote:
Originally Posted by kabelbrand View Post
I am aware of this. If your source is 1080 and you have a 140 pixel top letterbox you could add an extra 4 black lines on top and 4 on the bottom to get 1088 lines. The top letterbox will now be aligned with a macroblock border at 144 pixels.
Then encode using "--crop-rect 0,4,0,4" which should signal the playback software to discard the top and bottom 4 lines to render a 1080 picture. Unfortunately this won't be usable for Blu-ray Disc because the specification allows cropping at the bottom only.
You're not cropping, but shifting. If you shift up 8 pixels by cropping 8 more at the top, you need to add 8 black pixels to the bottom before you reencode. The input frame to the encoder for BD always needs to be 1080 for 1080p.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 16th October 2013, 21:32   #17  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
I don't see anything wrong with kabelbrand's last post. He's shifting down (not up) by 4 (not 8) pixels. You can certainly manually tell x264 where to put the cropping - otherwise it will automatically do it on the right and bottom side. The only problem would indeed be that the Blu-Ray specs don't allow it (assuming that is correct, I don't know if it's true), i.e. it's no use here.

Last edited by sneaker_ger; 16th October 2013 at 22:39.
sneaker_ger is offline   Reply With Quote
Old 16th October 2013, 21:41   #18  |  Link
kabelbrand
Compression mode: Lousy
 
kabelbrand's Avatar
 
Join Date: Mar 2009
Location: Hamburg, Germany
Posts: 72
Quote:
Originally Posted by benwaggoner View Post
You're not cropping, but shifting. If you shift up 8 pixels by cropping 8 more at the top, you need to add 8 black pixels to the bottom before you reencode. The input frame to the encoder for BD always needs to be 1080 for 1080p.
Shifting or Cropping&Padding - I think we are kinda talking about the same thing
Encoding a 1080 source with x264 is actually as good as encoding a 1088 source with "--crop-rect 0,0,0,8" specified.
Both are ok for BD - x264 (and other encoders) will repeat the last line of the 1080 source 8 times to fill 68 macroblocks vertically (1088 pixels) and specify 8 pixels bottom cropping at bitstream level internally.

But as I said, specifying bitstream cropping other than at the bottom is illegal for BD. Sadly...
kabelbrand is offline   Reply With Quote
Reply

Tags
detail, letterbox, macroblock, mod16, x264

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:28.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.