Strange values for Levels

[george84]

video = core.std.Levels(video, min_in=504, max_in=520, gamma=1, min_out=0, max_out=1023, planes=[0,1,2])

Video is YUV420P10.

Above statement will produce values > 1023 for some pixels in Vsscript preview.

Doc says:

The range [min_in, max_in] is remapped into [min_out, max_out].

[StainlessS]

https://forum.doom9.org/showthread.php?p=1479855&highlight=PixelClip#post1479855

Related ??? (Avisynth PixelClip)

[poisondeathray]

Do you have a sample that reproduces those results ? I tried a few random ones and I don't get anything >1023

[george84]

import vapoursynth as vs
from vapoursynth import core
video1 = core.ffms2.Source(source='JPEGUHD709PhotoDirList.xml.smil.avs.mkv')
video2 = core.ffms2.Source(source='TIFFUHD2020PhotoDirList.xml.smil.avs.mkv')
video = core.std.MakeDiff(video1,video2)
video = core.std.Levels(video, min_in=504, max_in=520, gamma=1, min_out=0, max_out=1023, planes=[0,1,2])
video.text.ClipInfo()
video.set_output()

See attachment. All purple colors are > 1023 (all 3 components Y,U,V >1023).

[Myrsloik]

See attachment. All purple colors are > 1023 (all 3 components Y,U,V >1023).

Use an external file host or we'll never see it. I'll try to reproduce it here later.

[george84]

Use an external file host or we'll never see it. I'll try to reproduce it here later.

It seems to happen to pixels which are not in range defined by min_in ... max_in.

What should they display?

[Myrsloik]

It seems to happen to pixels which are not in range defined by min_in ... max_in.

What should they display?

That's a very interesting observation, I think I see the problem in the code already. Those values should be clamped to the min/max in range but the max isn't clamped so it can become too big.

[Myrsloik]

R39-RC2 (https://www.dropbox.com/s/sb1pmrbq4i5iu2r/VapourSynth-R39-RC2.exe?dl=1)

It should properly clamp things. Go ahead and test it.