colorspace - DVD with odd luma range 16 to ca. 215 - how to correct?

[Lariel22]

I'm working with a DVD which has even the brightest luma values a bit low. Bright scenes look toned down, like the material is a little underexposed. I used
ColorYUV(analyze=true).Histogram("levels")
to have a closer look at the numbers. And I tried to fix this issue using
Levels(0,1,235,0,255,coring = false)
because it looks to me like the the lower luma end is fine, but at the upper end someone seems to have forgotten to open up the range one time. 92% of 255 is 235, and 92% of 235 is 216,6. And it appears the upper range is about 216.

Here are some pictures of bright scenes.

Original frame 533 (this frame seems to be an exception, see my explanation below):
https://i.ibb.co/Dt7Ktt5/533orig.png (https://ibb.co/Czn0zzh)

Corrected frame 533:
https://i.ibb.co/3yHNhg5/533corr.png (https://ibb.co/vx8Pz0C)

Original frame 3890:
https://i.ibb.co/6bGpWB7/3890orig.png (https://ibb.co/MpKjC6x)

Corrected frame 3890:
https://i.ibb.co/k1G91TK/3890corr.png (https://ibb.co/L50J5yS)

Original frame 4454:
https://i.ibb.co/xzP2CKb/4454orig.png (https://ibb.co/NWQFVgq)

Corrected frame 4454:
https://i.ibb.co/549D9vP/4454corr.png (https://ibb.co/r0tPtMV)

Original frame 30368:
https://i.ibb.co/Z6ZVybC/30368orig.png (https://ibb.co/VWKtXk8)

Corrected frame 30368:
https://i.ibb.co/Pt9CTpg/30368corr.png (https://ibb.co/6N1svhF)

Frame 533 is a rare exception. After the correction, the upper luma value extends well beyond 235, being 252 max. and 239 loose max. However, if I understand it correctly, the luma range is just a flag and the restricted range from 16 to 235 is used so that there is some overhead. Already the lower luma end does not stop at 16 but can be found at 11, 12 14, etc.

The other bright scenes look alright with this correction, in my opinion. The maximum values are at 230, 231, 232. So still below but close to 235.

Question 1: Is this just some film material with low brightness values deliberately used or am I seeing things correctly and this is a badly mastered DVD that is worthy of correction?

Question 2: Is my way of correcting this the way to go about this or is there some better way to do it?

Quesiton 3: The graphs of the corrected frames show some gaps. Can this be done without such gaps occuring, for example by converting to 16 bit or something like this?

Question 4: Is it okay that the corrected frame 533 has a max. value way beyond 235? Will this be clipped when encoding as MPEG2 and the range set to 16 to 235?

[poisondeathray]

1) +2) Some of it might be subjective, but you could argue some of those shots like explosion should be "brighter".

3) for gaps - levels(dither=true) , or smoothlevels

4) depends on the mpeg2 encoder. Overshoots are often seen on retail DVD's.

You can adjust per scene too; you don't have to use 1 set of values for everything

[Arx1meD]

I use OutRange (http://forum.doom9.org/showthread.php?p=1624992#post1624992) to find the maximum and minimum luma values.
My simplified version of OutRange:
Function OutRange2(clip c, float "thr", bool "Y", bool "U", bool "V", int "borders", string "filename") {
thr = String(Default(thr, 0.0))
Y = Default(Y, true)
U = Default(U, false)
V = Default(V, false)
br = Default(borders, 0)
FN = Default(filename, "OutRange.log")

Assert(Y || U || V, "OutRange: One of channel Y or U or V mast be true")

Yeval = Y ? "ymax=YPlaneMax("+thr+") ymin=YPlaneMin("+thr+")" : """ "" """
Ueval = U ? "umax=UPlaneMax("+thr+") umin=UPlaneMin("+thr+")" : """ "" """
Veval = V ? "vmax=VPlaneMax("+thr+") vmin=VPlaneMin("+thr+")" : """ "" """

wriY = Y ? """ "YMax = " + String(ymax) + "; YMin = " + String(ymin) """ : """ "" """
wriU = U ? """ "UMax = " + String(umax) + "; UMin = " + String(Umin) """ : """ "" """
wriV = V ? """ "VMax = " + String(vmax) + "; VMin = " + String(vmin) """ : """ "" """

expe = Yeval + Chr(10) + Ueval + Chr(10) + Veval

expw = wriY + String(Y && (U || V) ? """ + "; " + """ : """ + "" + """)
\ + wriU + String(U && V ? """ + "; " + """ : """ + "" + """)
\ + wriV

c.Crop(br, br, -br, -br)
FrameEvaluate(expe)
WriteFile(last, FN, "current_frame", """ ": " """, expw)
}

For example:
LWLibavVideoSource("C:\MV02\VideoFile.mkv")
Import(C:\AVS\OutRange.avsi")
OutRange2(thr=0, Y=true, U=false, V=false, borders=10, filename="OutRange.log")
Run it in AVSMeter (https://forum.doom9.org/showthread.php?t=174797) to collect frame statistics. The "OutRange.log" file will be created.

To smooth the levels on the graphs use the SmoothLevels (http://avisynth.nl/index.php/SmoothAdjust) filter, this is the easiest way. Or a combination of filters after Levels(...):
UnDot()
MiniDeen(radiusY=1, radiusUV=1, thrY=10, thrUV=12, y=3, u=3, v=3)
f3kdb(preset="low/nograin", sample_mode=2)

[Sharc]

@Lariel22: Keep in mind that the level shift may drive the RGB components beyond legal values and clip, even though the YUV (YCbCr) values are still sane.

[Richard1485]

The original looks fine to me. There's no reason that the luma has to push the edge of the range, even if there is an explosion. But, of course, it's down to user preference.

[kolak]

Yep, no idea why you assume it has to be pushed to the edge. Final look was established during grading process. You just create your own version. I don't see anything better there (brighter doesn't mean better).
Overshoots are basically always there when you use lossy compression. Even intermediate codecs like ProRes, DNxHR do it. It's not that big deal (unless you sending it to broadcast).
You could have whole 90 minutes movie with max luma at eg 200. It can happen and no need to "fix it".

[Emulgator]

And to add to that, lariel22, Levels made holes in your histogram.
no dither -> unpopulated values -> banding.
If you must stretch you may consider SmoothAdjust, this dithers well.
frame 533 looks already blown out to me after your stretch, well this depends on mapping of the display anyway.

[Lariel22]

Sorry for the late reply and thanks a lot for all your input!

So I guess I will use SmoothLevels. I was pondering why I think that the levels are off. I simply remember them being brighter when seeing this show on TV. I remembered that I also have TV recordings of the show, even from two different countries. Both confirm my assumption. See for yourself. The following is the exact same frame (despite the frame numbers being slightly apart).

https://i.ibb.co/RCVBp67/frame-retail.png
https://i.ibb.co/ThBs00f/frame-tv.png

The first is from DVD, the second from TV capture. In the first you can hardly guess what the color of the barrel on the right is, while in the second you can clearly see that it is blue.

[Asmodian]

I think you are correct, stretching the the range with dithering is probably a good idea. I would be careful to not overdo it though, less is more. :)

[Ghitulescu]

I would rather keep the TV capture, it certainly has a better resolution (details) however, then, one has to be careful when using screenshots to compare various videos (there is first the 16-235 vs 0-255 issue, then, for US NTSC the issue of pedestal).

The "reds" are washed out in the TV capture, though. Probably a hue issue, easily to be corrected.

[Sharc]

It's problematic to make judgements based on a single frame only, but for archiving I would definitely keep the retail (DVD) variant and make any tweaks (brightness, contrast, saturation) during viewing on the PC or TV as you prefer.
The TV version has clipped blues (e.g. the sky), and the black level of the active picture appears to be offset too high. The picture looks washed to me (e.g. the red light on the police car).
The TV variant also tends to producing illegal RGB triplets when decoded to RGB (sky for example), although it's minor for this picture, and it depends on the player how 'smart' these violations will be dealt with for playback.

Edit: The TV variant looks perhaps sharper but it has sharpening artefacts like halos, best seen on vertical edges. I don't see more real details.

[Ghitulescu]

Well, try reading the licence plate :)

[Sharc]

So what can you read? :D