Help me adjust brightness for a dark video source

[B3-bomber]

preferably without screwing up the other colors. i am not sure what type of brightening would work best. i can PM details about what source and other things i notice with it (like default driver brightness makes it appear fine, while washing out colors on EVERYTHING else on my system, YAY).

[StainlessS]

You are not going to get much help without posting a ~30MB sample, use eg DGIndex to cut VOB and up to MediaFire.

[Mounir]

try this filter: http://strony.aster.pl/paviko/hdragc.htm

[StainlessS]

To do an optional Auto Crop with auto levels


Function RoboLevels(clip c,float "Strength",int "Samples",float "Ignore",Bool "QBC",Float "QBC_Thresh",int "X",int "Y",int "W",int "H", \
int "Matrix",bool "PC",bool "DEBUG") {
c
Frames=FrameCount
Strength=Float(Default(Strength,1.0))
Samples=(!Defined(Samples)) ? 24 : (Samples > Frames) ? Frames : (Samples < 1) ? 1 : Samples
Ignore=Float(Default(Ignore,0.2))
QBC=Default(QBC,True)
QBC_Thresh=Float(Default(QBC_Thresh,-32.0))
QBCropXP = Default(X,0)
QBCropYP = Default(Y,0)
QBCropWP = Default(W,0)
QBCropHP = Default(H,0)
Matrix = Default(Matrix,(width <= 720) ? 2 : 3)
PC=Default(PC,False)
DEBUG=Default(DEBUG,True)
Assert(Strength>0.0 && Strength <=1.0, "RoboLevels: 0.0 < Strength <= 1.0")
GScript("""
if(QBC) {
QBCS=QueryBorderCrop(samples=Samples,thresh=QBC_Thresh,debug=DEBUG,ignore=Ignore,matrix=Matrix)
(DEBUG)? RT_Debug(QBCS): NOP
Eval(QBCS)
}
tclip=c.Crop(QBCropXP,QBCropYP,QBCropWP,QBCropHP)
QLMMS=QueryLumaMinMax(tclip,samples=Samples,ignore=Ignore,debug=DEBUG,matrix=Matrix)
(DEBUG)?RT_Debug(QLMMS):NOP
Eval(QLMMS)
if(IsRGB() || PC) {
CSMin = 0
CSMax = 255
} else {
CSMin = 16
CSMax = 235
}
ALMin = Int(CSMin - ((CSMin - QLMMMin) * Strength) + 0.5) # Round Up
ALMax = Int(CSMax - ((CSMax - QLMMMax) * Strength)) # Round down
(QBC) ? Crop(QBCropXM,QBCropYM,QBCropWM,QBCropHM): NOP
Levels(ALMin,1.0,ALMax,CSMin,CSMax,Coring=False) # DO NOT use Coring
""")
return Last
}


You also need the two scripts from 3 consecutive posts here (and RT_Stats plugin from 1st post in that thread,
both the scripts pointed to are contained in RT_Stats zip):
http://forum.doom9.org/showthread.php?p=1586857#post1586857

EDIT: Also need GScript.
QBC controls optional cropping, others can default. If no borders present then just simple "Robolevels()" will work fine.
Above script is work in progess but works fine.

[travolter]

I was testing RoboLevels in AVSP and its working pretty well ;) .. but using it realtime the video become freezed and does not start.
I need to use extra line in my script to do it realtime into ffdshow?

[StainlessS]

@travolter,
You could try it with QBC=false to switch off cropping, and/or Debug=true, and see what comes out of DebugView.
As far as I'm concerned, if it works ok in AVSP then it should work ok in whatever player you are using.
(as far as I know, ffdshow is just a codec) What is the actual player you are using.
Perhaps the player is failing because of the long timeout until first frame delivered, then its a player fault.
You might reduce timeout by setting samples lower but this may affect how well it auto levels.
You could also skip robolevels (just a temp name, a testing script to develop a plugin) and test with
QueryBorderCrop() and QueryLumaMinMax() separately just to see what the player dont like.
Have you tried with Mplayer or any other player?
I (just a few days ago) did 60 test clips which all worked perfectly in GOM, Mplayer HC, and MS MPlayer2 (the old one, dont like new one).
If posting further on this, please do it in the RT_Stats thread so as not to clutter this one. Thankyou.
EDIT: Make sure you are using latest version of RT_Stats and scripts.

[B3-bomber]

see new post

prefer that place since i can delete it later. can also auto expire.

[StainlessS]

Sorry if I was unclear, those were only d2v files.

Load VOB into DGIndex,
move bottom slider to START of area of interest and then press '[' to set start.
move slider to END of area of interest and then press ']' to set end.

Then on file menu click on "Save Project & Demux Video", it will save with name ending ".demuxed.m2v".
Check file size is suitable and then repost.

[B3-bomber]

http://www.sendspace.com/filegroup/iUGvV1drrxxBJQ26sf1l0HGq9HkKm4Ov

included d2v with the demuxed vid

side note: for sure source related. watching others on current settings dont need to change brightness to see dark areas.

[StainlessS]

Have looked at your source anime, and it looks like it is the intent of the makers that it be that dark
in places, looks like a coal cellar or something. It is I think unlikely that you can do anything other
than bring up lighting levels where everything will be almost solid grey rather than black,
and expose little detail, only noise.

Perhaps look at hdragc as Mournir suggested, any global leveling will fail (as for the supplied Robolevels).

Here are stats for your source clips and auto leveled (the crop fails on dark clip as it is nearly all black and so short,
ie crops out much of the video).


darkarea.demuxed.d2v
AVE_Y=27.58 MIN_Y= 11 MAX_Y=236 LMIN_Y= 15 LMAX_Y=227
AVE_U=128.09 MIN_U=110 MAX_U=152 LMIN_U=120 LMAX_U=145
AVE_V=128.44 MIN_V=117 MAX_V=203 LMIN_V=121 LMAX_V=151

darkarea.demuxed.d2v Auto Leveled
AVE_Y=34.29 MIN_Y= 16 MAX_Y=235 LMIN_Y= 16 LMAX_Y=235
AVE_U=128.14 MIN_U=109 MAX_U=150 LMIN_U=119 LMAX_U=145
AVE_V=128.66 MIN_V=116 MAX_V=207 LMIN_V=121 LMAX_V=156

lightarea.demuxed.d2v
AVE_Y=66.54 MIN_Y= 13 MAX_Y=237 LMIN_Y= 16 LMAX_Y=234
AVE_U=132.75 MIN_U=112 MAX_U=170 LMIN_U=118 LMAX_U=168
AVE_V=123.93 MIN_V= 91 MAX_V=139 LMIN_V= 99 LMAX_V=136

lightarea.demuxed.d2v Auto Leveled
AVE_Y=66.93 MIN_Y= 16 MAX_Y=235 LMIN_Y= 16 LMAX_Y=235
AVE_U=132.75 MIN_U=112 MAX_U=170 LMIN_U=118 LMAX_U=168
AVE_V=123.93 MIN_V= 91 MAX_V=139 LMIN_V= 99 LMAX_V=136


As you can see, MIN_Y and MAX_Y reasonably close to optimum anyway (although robolevels scan on full movie
would be be better than only on 10 to 20 seconds of clip).

EDIT: Global leveling (eg RoboLevels, on FULL movie) would not do any harm and may help just a little, but nothing spectacular would result
in the deliberately dark sections.

[B3-bomber]

i know i wont get it fully lightened without breaking stuff. but the brightness setting that i have on my monitor that shows it as it was aired on TV is much too high for overall viewing. this also doesnt seem to affect the actual other scenes oddly enough (my guess is they just plain did the entire thing a tad too dark). there have been a lot of complaints about the non TV sources for this. so yeah it may be deliberate but there is still some censorship built in and i just want it to play as a mostly normal scene and not a big wall of black (some eps are basically 80% of the super darkened crap because of this).


p.s. why the hell wont quick reply work even when i fully refresh the page? it shows me as logged in still too. unless i go out of the thread and reload it through clicking the thread it blocks me. at least using the post button works.

also may i get a little more detailed info on what to do for the robot thing or is there something simpler? i could actually give an idea of the light levels needed to see the show as it was on TV if i had a way to figure out if out driver and monitors did the same level of lighting...

[StainlessS]

You need RT_Stats and GScript,
RT_Stats: http://forum.doom9.org/showthread.php?t=165479
GScript: http://forum.doom9.org/showthread.php?t=147846&highlight=gscript

Also MS DebugView: http://technet.microsoft.com/en-us/sysinternals/bb896647.aspx

Rough script:

VIDOEOFILE="D:\AVS\AVI\lightarea.demuxed.d2v"
MPEG2Source(VIDOEOFILE)

# Your source above,
# RT_Stats & GScript plugins in avisynth plug dir and import below or put in plugs dir

Import ("D:\avs\BANK\QueryBorderCrop.avsi")
Import ("D:\avs\BANK\QueryLumaMinMAx.Avsi")

Robolevels(QBC=false,samples=24) # higher samples better, especially for your clip I think.
Return Last

Function RoboLevels(clip c,float "Strength",int "Samples",float "Ignore",Bool "QBC",Float "QBC_Thresh",int "X",int "Y",int "W",int "H", \
int "Matrix",bool "PC",bool "DEBUG") {
c
Frames=FrameCount
Strength=Float(Default(Strength,1.0))
Samples=(!Defined(Samples)) ? 24 : (Samples > Frames) ? Frames : (Samples < 1) ? 1 : Samples
Ignore=Float(Default(Ignore,0.2))
QBC=Default(QBC,True)
QBC_Thresh=Float(Default(QBC_Thresh,-32.0))
QBCropXP = Default(X,0)
QBCropYP = Default(Y,0)
QBCropWP = Default(W,0)
QBCropHP = Default(H,0)
Matrix = Default(Matrix,(width <= 720) ? 2 : 3)
PC=Default(PC,False)
DEBUG=Default(DEBUG,True)
Assert(Strength>0.0 && Strength <=1.0, "RoboLevels: 0.0 < Strength <= 1.0")
GScript("""
if(QBC) {
QBCS=QueryBorderCrop(samples=Samples,thresh=QBC_Thresh,debug=DEBUG,ignore=Ignore,matrix=Matrix)
(DEBUG)? RT_Debug(QBCS): NOP
Eval(QBCS)
}
tclip=c.Crop(QBCropXP,QBCropYP,QBCropWP,QBCropHP)
QLMMS=QueryLumaMinMax(tclip,samples=Samples,ignore=Ignore,debug=DEBUG,matrix=Matrix)
(DEBUG)?RT_Debug(QLMMS):NOP
Eval(QLMMS)
if(IsRGB() || PC) {
CSMin = 0
CSMax = 255
} else {
CSMin = 16
CSMax = 235
}
ALMin = Int(CSMin - ((CSMin - QLMMMin) * Strength) + 0.5) # Round Up
ALMax = Int(CSMax - ((CSMax - QLMMMax) * Strength)) # Round down
(QBC) ? Crop(QBCropXM,QBCropYM,QBCropWM,QBCropHM): NOP
(DEBUG)? RT_Debug("Levels("+String(ALMin)+",1.0,"+String(AlMax)+","+String(CSMin)+","+String(CSMax)+",Coring=False)"):NOP
Levels(ALMin,1.0,ALMax,CSMin,CSMax,Coring=False) # DO NOT use Coring
""")
return Last
}


I've added the RT_Debug line near the end to show something like


Levels(16,1.0,233,16,235,Coring=False)


You need to run DebugView and then run the script and Debugview will capture output.
The imported scripts are included in RT_Stats zip.
EDIT: The samples=24, line is the default, might want to increase to as big as you can stand,
(only samples 24 frames by default, spread evenly throughout clip).

EDIT: Also see here: http://forum.doom9.org/showthread.php?p=1598112#post1598112

[B3-bomber]

ok i still need help putting this stuff together. managed to get the 2 .zip, script had an issue at robotlevels line. if there is stuff that is supposed to be in a .avs in the plugins folder i need to know what the names are supposed to be and the contents.

warning: i have problems with complex avs scripts. at one point i tried animeIVTC script. i NEVER got all of that to function even with all the supposed required files.

also, what am i supposed to use to run this script? is it supposed to be in a .avs script and if yes what do i use to run it (got the errors in megui trying to preview the vid output) it didnt like this line either: MPEG2Source(VIDOEOFILE) says no function (yay for fresh installed avisynth and no plugins).

[manono]

i could actually give an idea of the light levels needed to see the show as it was on TV ...
Maybe your TV set needs calibrating. It looks to me too like your sample is supposed to be very dark.
also may i get a little more detailed info on what to do for the robot thing or is there something simpler?
There are lots of simpler filters to brighten things up. I often use YLevelsS:

http://avisynth.org/mediawiki/Ylevels

Or HDRAGC, as suggested earlier.

[StainlessS]

VIDOEOFILE="D:\AVS\AVI\lightarea.demuxed.d2v" where the d2v file is the one output by DGIndex.

RT_Stats, Grunt and GScript dll's need be in plugins dir.
Also put the two script from RT_Stats zip either in plugs dir or Import them as in example, obviously edit the directory names.

what am i supposed to use to run this script?

It looks like you do not use avisynth outside of eg MeGUI, use anything eg Mplayer to open script.

EDIT: Or see Manono's suggestions above

[LemMotlow]

d l t d

[manono]

Histograms of frames in the smples m2v's show the levels and colors are out of spec (darks compressed and crushed, brights off the wall).
There's nothing wrong with blacks being below 16 and whites above 235. Happens all the time. With :

LanczosResize(624,352)
ColorYUV(Analyze=True)
Limiter(Show="Luma")

you get the first picture below left from frame 199. The red dots show values below 16 (and barely below at that). Although it's quite dark, it's not really out of the ordinary for a really dark scene. The second picture below right is with:

YLevelsS(0,10.0,255,0,255)
Tweak(Bright=-14,Cont=0.93,Coring=False)
LanczosResize(624,352)
ColorYUV(Analyze=True)
Limiter(Show="Luma")

In neither of your two samples did I see any blown whites, although maybe they do blow out elsewhere.

[B3-bomber]

i watched this on TV using CRT. it wasnt overly dark then. i can watch it that way on this screen but it requires setting brightness to default on drivers (value of 0). doing that washes out EVERYTHING (OS menus, background, any other program used) but the video when it comes to colors.

they did this to the released source as a double censor thing. they already have blurred details on entrails and other body parts.

manono that screenshot on 2nd part looks a lot more how it was supposed to be. i dont expect it to ever be perfect but those really dark scenes should at least resemble the "night" scenes in most anime where u can see some stuff, not be pitch freaking black like it essentially is now. i never had any problems with the maximum, just didnt wanna wash out color detail for normal scenes.

i'll mess with this a bit later tonight maybe if not tomorrow night. at least having something work properly would be awesome (errors are annoying).

side note: i can pm what to search for to prove the complaints about the video being too dark. it isnt just my screen. just ask if u need info.

[LemMotlow]

d l t d

[manono]

That tells me that detail isn't just "dark", it's compressed and crushed.
It doesn't tell me that at all. The parts of the picture in my earlier post that are below 16 are just pure black with nothing else there.

The histogram for the original frame #39 in the "lightareas" sample showed brights smashing against the right wall of color histograms, with the brightest blues seriously clipped.
I see none of that. Have a look at the three pictures below, two with different kinds of histograms, the third using my earlier script. The only parts above 235 (the green dots) are the shiny metal of the bridge, there's very little of that, and it's completely normal. All this makes me wonder just how you're making your conversion to RGB in the first place. And why convert at all?

[B3-bomber]

Any tips on what these do so I can tweek them and check for actual results would be awesome:)

YLevelsS(0,10.0,255,0,255)
Tweak(Bright=-14,Cont=0.93,Coring=False)

I spent more time on the avisynth warpsharp site finding files I was missing (and a few on the forum that aren't there oddly, masktools).


LemMotlow, those screenshots of the redman holding a head are an improvement. Forgot to mention that earlier. On the default encode when it gets that far into the scene the head is no longer visible.

[manono]

Any tips on what these do so I can tweek them and check for actual results would be awesome:)
A=Last
B=A.YLevelsS(0,10.0,255,0,255)
B=B.Tweak(Bright=-14,Cont=0.93,Coring=False)
StackHorizontal(A,B)#perhaps followed by a resize so you can see them more easily

Or teach yourself to use AvsPMod where you can compare by switching tabs back and forth.

[B3-bomber]

Which one of these actually increases/decreases brightness and whats the value range for them. Adding more steps to something I have no clue how it works isn't the best idea.

Man that reminds me of debugging some old mirc code. One guy I asked basically told me get lost until I learned the entire new coding structure. I knew where the error was generally too, just not specifically what was messed up. Was all of maybe 10 minutes with someone who actually bothered to help me debug it. So yeah just what is present first. I'll ask questions if that doesn't do anything like I need it to:P

[manono]

If you're not even willing to read the docs, then why should anyone care enough to explain it to you?

Tweak (http://avisynth.org/mediawiki/Tweak)

Ylevels (http://avisynth.org/mediawiki/Ylevels)

StackHorizontal (http://avisynth.org/mediawiki/StackHorizontal)
Which one of these actually increases/decreases brightness...

If you understand that brightness determines the black levels, then Tweak's 'Bright=' does that.
...and whats the value range for them.

From the Tweak page:
bright (-255.0 to 255.0, default 0.0)

[B3-bomber]

sorry, didn't know those sites existed. it's been a few years since i've worked with this stuff.

[StainlessS]

@B3-bomber
The docs are also included via "Avisynth 2.5/Docs/English/Index.htm" although I doubt if YLevels is in there.
There is also a searchable compressed (CHM) help file version of docs via link in my sig, nice to have on a hot-key
eg "CTRL/ALT H". (EDIT: Although most up to date versions are in the Wiki)

[B3-bomber]

cool. i read the ylevels stuff. the scary thing is i had used levels at one point. that's how outdated my memories are.

got some progress working on the overly dark stuff. seems i have to use a sharpener on it though since it fuzzes the outlines. thanks for the information:)

[LemMotlow]

d l t d

[LemMotlow]

d l t d

[IanB]

Yes crushing blacks (and smashing whites) looses data. It's what you get with 8 bit integer arithmetic.

As you see in your histograms, when you stretch out the crushed blacks you get holes.

The maths is simple O=k*(I-16)+16, k being the 'stretch' factor.

For example with k=2 we get the following :-

Inpt Outpt
16 -> 16
17 -> 18
18 -> 20
19 -> 22
20 -> 24
...
32 -> 48
...
etc

You can mask and compensate for the effect with methods like dithering and non-linear stretching methods, but the data is gone.

[LemMotlow]

d l t d

[Mounir]

Something like intelligent dithering (like the msu super precision filter (http://www.compression.ru/video/super_resolution/super_precision_en.html)) would be better, i'm not sure we have that for avisynth yet, i recall we talked about it some time ago here (http://forum.doom9.org/showthread.php?t=159151&highlight=msu+super+precision)

[manono]

All this makes me wonder just how you're making your conversion to RGB in the first
place. And why convert at all?
I guess I'm misinformed.
Maybe. We'll see. You already know the reason for doing the conversion to RGB before using any of the
VDub filters that require RGB - because VDub stretches everything to 0-255 which often results in clipping.
So to prevent that from happening we like to do the conversion properly ourselves when the use of VDub
filters is necessary, whether we use the filters in our scripts or in VDub itself. Let's see what happens if we don't
do a proper conversion. I'm opening the lightarea.demuxed.m2v directly in VDubMod, scrolling to frame 39 and
using the Histogram in Color Tools. The result is the leftmost picture 1 of the set of four pictures
below. Does it look familiar? Opening an AviSynth script with no conversion in the script results in
picture 2, pretty much the same thing. Adding your conversion line to the script
(ConvertToRGB32(Matrix="Rec601")) gives you pic 3, something very similar to the first two. Now, convert it the
way you should have with ConvertToRGB(Matrix="PC.601") and you get the fourth picture. No clipping.

As you know, since you lectured me on RGB and YUV and computer monitors and television monitors, a video
looks different on a computer monitor than it does on a TV screen. The levels are different and they're
calibrated differently. What looks too dark to make out anything on a computer monitor just looks dark
but viewable on a television. Which is the way that dark scene is meant to appear - murky, difficult to make
out, which adds to the feelings of paranoia and horror it's meant to convey. I'm convinced the production
company that made this anime knew exactly what they were doing and that it looks exactly as it should
when viewed on a properly calibrated TV set. However...

...and that you won't get clean images using YLevels and Tweak alone...

You're saying that the AviSynth filters aren't good enough to adjust the brightness, contrast, levels,
gamma, and the like? Sheer and utter nonsense. Not only can you do it well using AviSynth, but you
should use AviSynth for such simple things (if they need it) to avoid the unnecessary colorspace
conversions and the problems (and mistakes) that result from colorspace conversions. The only time
I ever go to RGB is for some color work. Others use VDub only filters (ones such as Neat Video) for other purposes.

By the way, I'm sure you know your 'new and improved' darkarea_open_try8g.m2v is encoded as 4:3 while the
source is 16:9. And I'm sure you know you hard-telecined it and encoded it as interlaced, while the
source is encoded as progressive 23.976fps with pulldown. At least the samples are. So we won't use it as
an example of your restoration and encoding prowess.

[LemMotlow]

d l t d

[B3-bomber]

ok, is there anything that default drops crap frames they added in a lame attempt as a transition frame? they really would have done better just using the next one after this crap. the frame after the 2 ghosted spots he is shifted entirely to fit that position. i noticed these running an encoded ver at normal speed. i HATE when things like this are done.

http://img827.imageshack.us/img827/8125/halpk.png