Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th March 2019, 20:08   #201  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by der_groschi View Post
First of all, thanks LaTo for this awesome plugin, it's really become one of those tools i couldn't live without anymore.

I'm using it in a way that might be somewhat different from it's intended use.
Particularly, i use to combat the issue of highly inconsistent black levels i find on most DVD and bluray releases of older asian movies.

So i usually run it after dialing in the overall levels via SmoothAdjust as good as possible without clipping. Then i run AutoAdjust with auto_gain=true, gain_mode=1, with no gamma correction at all and a rather conservative dark_limit value; just enough to get the levels right for most scenes.

Now i always run into the problem of AutoAdjust being rather reluctant of changing very dark scenes, where muddy black levels are bothering me the most.
For brightly lit scenes a rather conservative dark_limit setting of 1.25 usually is sufficent, but for overall very dark scenes, i often have to crank it way up to 2.0 or even higher to get the black levels close to zero.

Is there any way to change this behavior, to allow AutoAdjust to change levels of very dark scenes by the same amount as it does brighter ones?

Or does anybody have an idea for a workaround?
I know this might not be exactly what this filter was originally made for, but finding a way to make it work this way would make my life a lot easier.
I realize I'm replying to you years late, but for anyone else who might come across this I found the same issue when more or less doing the same thing you're doing. Trying to fix poorly mastered DVDs with inconsistent and too high black levels. I found a few things helpful.

1) Filter the input: Use the external_clip function to feed in a slightly cropped and blurred version of the source to eliminate compression artifact ringing with unexpected/unintended dark and bright pixels. Crop off a few rows of pixels from each of the four sides before the blur to prevent any issues with cameras that leave a row of black or lower luminance pixels on one of the edges.

2) Exclude very few pixels. I use 0.01 to 0.001 for excludes.

3) Pre-process. As you noted autoadjust will not make dramatic changes to dark video with limited dynamic range. I stick a histogram of the source and output on the side of the video to see how the YUV levels look and watch through it. When I note a murky scene where AutoAdjust did not pull it down sufficiently or excessively stretched it out I add that frame range to my pre-processing in RGBAdjust.

So ahead of using AutoAdjust I do this:
Code:
ScriptClip(source_16b, "
RGBAdjust(rb=((xyx_l)*256), gb=((xyx_l)*256), bb=((xyx_l)*256), r=(1-(xyx_g)/256), g=(1-(xyx_g)/256), b=(1-(xyx_g)/256))
")
ConditionalReader("gain.txt", "xyx_g", false)
ConditionalReader("levels.txt", "xyx_l", false)
(working in 16-bits with AVIsynth+)

So in the levels.txt file (and probably gain.txt too) I will have something like:
Code:
Type float
Default 0.0

R 2379 2515 -10.0

R 3261 3301 -10.0

R 7476 7564 -10.0
4) Don't try to expand the whites with AutoAdjust. If you want to stretch them across whole large regions do it RGBAdjust (using gain) in front of AutoAdjust. AutoAdjust will squish excessive whites down to 235 even with bright_limit=1.0 (input_tv=false, output_tv=true). So as long as you don't boost them past 255 into clipping AutoAdjust will fix them.

Conversely it will do the inverse on the black end. If the blacks are too dark it will raise them up (with those options). So you can apply a broad general manual correction across an entire clip getting things in the general ballpark and let AutoAdjust take care of the scene by scene tweaking rather than relying on it to do all the correction.
Stereodude is offline   Reply With Quote
Old 25th April 2019, 02:59   #202  |  Link
HappyLee
Registered User
 
Join Date: Mar 2013
Posts: 27
First of all, I love this filter (or actually almost all the filters by LaTo). Recently I'm switching to AviSynth+ MT-r2772 (latest version), but when I use AutoAdjust, the first frame is always loaded incorrectly (heavy contrast and saturation), even if the script is as simple as this:
AutoAdjust(auto_gain=true)
Trim(1000,2000)

If it only happens on the very first frame, then it wouldn't be a big problem, but there was one time I was making a test clip encode using x265, and the encoded file has heavy flicker, like lots of frames having been loaded incorrectly, and that made me panic. I'm wondering if anyone has met the same problem, and how it could be fixed without switching back to AviSynth previous versions. Thanks.

Edit: Yes I've noticed there are other problems as well, not just the first frame. So could someone please help me confirm it or think of ideas to fix it. Thanks in advance!

Last edited by HappyLee; 25th April 2019 at 10:55.
HappyLee is offline   Reply With Quote
Old 25th April 2019, 13:21   #203  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
I think that I have noticed lately same type thing on first 1 or 2 frames using AutoAdjust (by frustrum [EDIT: Actually by LaTo, got mixed up with AutoLevels] , I think), also in some white balance scripts.
Dont think it used to happen about 6 months ago [possibly longer].

EDIT: Also, I think if you try trim off offending frames, it still affects the first few frames after trimming.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 25th April 2019 at 14:17.
StainlessS is offline   Reply With Quote
Old 25th April 2019, 13:37   #204  |  Link
HappyLee
Registered User
 
Join Date: Mar 2013
Posts: 27
Quote:
Originally Posted by StainlessS View Post
I think that I have noticed lately same type thing on first 1 or 2 frames using AutoAdjust (by frustrum, I think), also in some white balance scripts.
Dont think it used to happen about 6 months ago [possibly longer].
Thank you for confirming it. So it might be a glitch of the latest builds of AviSynth+? I guess switching back to AviSynth 2.6 could be the only solution for now?
HappyLee is offline   Reply With Quote
Old 25th April 2019, 13:44   #205  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Does the problem remain as in above edit, ie after trim off original offending frames ?
(might be a clue for somebody. I dont have Avs installed at the moment.)

EDIT: My mistake[got names mixed up], I thought I had tried AUTOLEVELS by Theodor Anschütz and Frustrum, but it was actually AutoAdjust by LaTo. (same plugin as you, so is confirmed for AutoAdjust).

EDIT: I have posted before about trim() not working as it used to, perhaps this is related.

EDIT: If you try [at and of script] trim off 1st couple of frames and problem remains, then try following the trim with DelayAudio(0.0), if this fixes problem,
then is the same problem as in above edit non functioning trim problem. [this prob really need be fixed, else we shall have no end of similar problems with scripts that used to work]
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 25th April 2019 at 14:20.
StainlessS is offline   Reply With Quote
Old 25th April 2019, 14:02   #206  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
One reason can be that the plugin assumes linear frame access, which is not necessarily true. Do the other glitches appear around scene changes?
pinterf is offline   Reply With Quote
Old 25th April 2019, 14:43   #207  |  Link
HappyLee
Registered User
 
Join Date: Mar 2013
Posts: 27
Trimming at the end of the script won't solve the problem.

And yes, pinterf. I detected similar glitches (flicker) around scene chances during the preview, and there was one time my encoded file by x265 also has heavy flicker caused by AutoAdjust.
HappyLee is offline   Reply With Quote
Old 25th April 2019, 18:36   #208  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Quote:
Trimming at the end of the script won't solve the problem.
That would imply that you did not try trim (or you would have said it does not solve the problem).
[EDIT: Trim() at end of script should not affect either frames before or after trimming, except those frames trimmed out will not be returned in the output clip (that is, when trim works correctly which is currently in question).]

But does it remove the bad frames when and if you do try it ? (or are they still there just at the new first frames)
and also what happens if adding DelayAudio(0.0) to end of script [DelayAudio is known to force trim to work as expected].
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 25th April 2019 at 21:26.
StainlessS is offline   Reply With Quote
Old 26th April 2019, 03:29   #209  |  Link
HappyLee
Registered User
 
Join Date: Mar 2013
Posts: 27
Of course I've tried Trim. I even added Trim in my previous post (test script):
*Original video here*
AutoAdjust(auto_gain=true)
Trim(1000,2000)

For the encoded file, frame 1000 of the original video would be broken. If I use Trim(2000,3000), frame 2000 would be broken. Broken frames happen not just the very first frame of the previous script, but the first frame the whole script is loaded.

This can probably be fixed by ConditionalFilter, but as I said, there might be bigger problems (more glitches) than the broken first frames.
HappyLee is offline   Reply With Quote
Old 6th May 2019, 12:47   #210  |  Link
Treaties Of Warp
Registered User
 
Join Date: Apr 2019
Posts: 14
For me this filter becomes more unstable in Avisynth when more filters are run with AutoAdjust, even if AutoAdjust is the first filter in the script. I was getting really weird and constant flashing in my output. When I disabled AutoAdjust, it went away. When I ran only AutoAdjust, the problem also went away. When I added LimitedSharpenFasterMod, then the flashing appeared, but only in the first few seconds of output. I didn't test further yet because I don't have time but I had some other filter or filters that were making the flashing constant. Maybe internal caching issue? Any way to disable it?
Treaties Of Warp is offline   Reply With Quote
Old 15th June 2019, 13:46   #211  |  Link
Lirk
Registered User
 
Join Date: Jan 2018
Posts: 33
Is it possible to use Autoadjust as Autolevels filter?
Lirk is offline   Reply With Quote
Old 26th July 2019, 13:11   #212  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
How does AutoAdjust know what color space to operate in when doing chroma processing? There's no flagging for rec601/rec709/etc. in the function call.

Also I presume development of this is dead? Otherwise I have a feature request. RGB based limiting. Instead of strictly looking at Y, looking at the RGB channels to figure out how far to expand the Y levels so that none of the color channels get clipped.
Stereodude is offline   Reply With Quote
Old 26th July 2019, 14:10   #213  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Quote:
How does AutoAdjust know what color space to operate in when doing chroma processing? There's no flagging for rec601/rec709/etc. in the function call.
There is no levels range indication in Levels() filter either, and levels limits range [Y=0->255, U:V=0->255, if no coring] and [Y=16->235, U:V=16->240 if coring].

LaTo aint been around for a while, but did pop up some time ago when someone posted a request, so who knows, maybe he's watching.
[There is no source available]

this may not be exactly how Levels is implemented, but at least very nearly the same.

Code:
    if (vi.IsYUV()) {
        for (int i=0; i<256; ++i) {
            float p;
            if (coring)
                p = ((i-16)*(255.0f/219.0f) - in_min) / divisor;
            else
                p = float(i - in_min) / divisor;                                        // range 0.0 -> 1.0 of input range

            p = pow(min(max(p, 0.0f), 1.0f), gamma);                                    // gamma
            p = p * (out_max - out_min) + out_min;                                      // output range with out_min offset
            int pp;

            if (coring)
                pp = int(p*(219.0f/255.0f)+16.5f);
            else
                pp = int(p+0.5f);                                                       // round to nearest luma level

            map[i] = min(max(pp, (coring) ? 16 : 0), (coring) ? 235 : 255);

            int q = ((i-128) * (out_max-out_min) + (divisor>>1)) / divisor + 128;
            mapchroma[i] = min(max(q, (coring) ? 16 : 0), (coring) ? 240 : 255);
        }
    } else if (vi.IsRGB()) {
        for (int i=0; i<256; ++i) {
            float p = float(i - in_min) / divisor;
            p = pow(min(max(p, 0.0f), 1.0f), gamma);
            p = p * (out_max - out_min) + out_min;
            int z=int(p+0.5f);
            map[i] = (z < 0) ? 0 : (z>255) ? 255 : z;
        }
    }
EDIT:
If coring set, then Levels assumes is TV range [but then after removal of 16 offset and scaling, weirdly assumes that arg input/output ranges are full range 0->255, probably to
ensure that gamma is affected properley when offset added back, but this makes it bloody awkard/hazardous to use anything other than 0,255 as either input or output range args when coring=true.
Eg if you supply coring=true, out range =16,235, then output range would be compressed TWICE ... Me dont like coring!
With Coring=true, should only use it to apply Gamma, anything else is dangerous unless you sort of backwards pre-calculate what the result will be, real awkward].

EDIT:
Backward pre-calc from AutoLevels source [for pre-calc input_min/input_max, output_min/output_max where coring=true].
Code:
double inv_coring_func(double y) { return y * (219.0/255.0) + 16.0; }				        // ssS: Changed (255.0/219.0) to (219.0/255.0)
EDIT: Actually, above was wrong in earlier AutoLevels (v0.6 and earlier), as noted in the comment.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 26th July 2019 at 14:49.
StainlessS is offline   Reply With Quote
Old 26th July 2019, 14:51   #214  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by StainlessS View Post
There is no levels range indication in Levels() filter either, and levels limits range [Y=0->255, U:V=0->255, if no coring] and [Y=16->235, U:V=16->240 if coring].

LaTo aint been around for a while, but did pop up some time ago when someone posted a request, so who knows, maybe he's watching.
[There is no source available]
Well, I can't even be certain that AutoAdjust does any conversion internally to RGB. I presume it does for it's chroma processing, but that's a guess. I suppose as long as it uses the same matrix for both conversions it probably doesn't make a big difference at least for 601/709.

Ideally I would like a mode to prevent clipping of any of the RGB color channels (even if chroma processing is not enabled). Currently it looks like it only considers Y which can cause an individual R,G, or B color channel to fall above or below the limits resulting in color channel clipping.

Last edited by Stereodude; 26th July 2019 at 15:03.
Stereodude is offline   Reply With Quote
Old 26th July 2019, 15:09   #215  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
I dont know what it does, but as it is quite a lot slower than autoLevels [with simple AutoAdjust(autogain=true)], I suppose it must be doing a lot of something, perhaps there is YUV->RGB->YUV conversion.
Would be nice if LaTo provided source.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 27th July 2019 at 17:35.
StainlessS is offline   Reply With Quote
Old 26th July 2019, 17:21   #216  |  Link
Stereodude
Registered User
 
Join Date: Dec 2002
Location: Region 0
Posts: 1,436
Quote:
Originally Posted by StainlessS View Post
This is my best guess [can produce weirdish numbers, eg -18.63]
Best guess of what?
Stereodude is offline   Reply With Quote
Old 27th July 2019, 17:58   #217  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
StereoDude,
Have deleted previous rubbish altogether, I'm quite sure that coring massager thing in AutoLevels is totally borked, and needs an alternative massage
so that when calling equivalent to Levels, we pre-compensate input_low and input_high so that we can get required result.
[problem due to coring messing things up so that required input_low/high dont do as required].
I can do it, but not today, time for the pub in a minute.
[I guess I could just add an arg to AutoLevels Levels() equivalent to skip coring on the input, and only core the output [EDIT, nah, that would not really work ideally]
[as we will want to give explicit/measured input_low & high, however I would still like to be able to do that with the Levels() filter so will probably
persist until we got it working without the coring skip].


If you would like to have a play, here a graf drawing whotsit (EDIT: Levels Simulator) [needs V.C Mohan Grid Filter, but can comment out that line if you wish]

Code:
Function LevelsSim(Int In_min,Float Gamma,Int In_max,Int Out_min,Int Out_max,Bool Coring,Int i,Int ret) {
    Gamma=Min(Max(Gamma,0.1),10.0)
    divisor = (in_max==in_min)  ? 1.0 : Float(in_max - in_min)
    R0=(coring)
        \ ? ((i-16)*(255.0/219.0)-in_min) / divisor
        \ : float(i - in_min) / divisor
    G = pow(Min(Max(R0,0.0),1.0),1.0/Gamma)
    R1 = G * float(out_max - out_min) + out_min
    R2 = (coring)
        \ ? min(max(Round(R1*(219.0/255.0)+16),16),235)
        \ : min(max(Round(R1),0),255)
    r =Select(ret,Round(R0*255.0),R1.Int,R2.Int)                   # Function Return R0, R1, or R2, depends upon Ret arg : *** NOTE *** R0=0.0->1.0, WILL BE SCALED TO 0->255 for graf
#RT_DebugF("%d] = %.2f",i,r)
    return r
}

K=BlankClip(width=256,height=256,Length=1,Color=$808080)

Ret=2           # return type (type=0 is multiplied by 255, originally 0.0->1.0)
Gamma=1.5
##############
In_min=0
In_max=255
Out_min=16
Out_max=235
Coring=false
Color=$00FF00
Point=K.BlankClip(width=3,height=3,Color=Color)
for(x=0,255) {
    y = LevelsSim(in_min,Gamma,in_max,Out_min,Out_max,Coring,x,Ret)
    K=K.Overlay(Point,x=x-1,y=y-1,Opacity=0.3)
}


In_min=0
In_max=255
Out_min=0
Out_max=255
Coring=true
Color=$FF0000
Point=K.BlankClip(width=3,height=3,Color=Color)
for(x=0,255) {
    y = LevelsSim(in_min,Gamma,in_max,Out_min,Out_max,Coring,x,Ret)
    K=K.Overlay(Point,x=x-1,y=y-1,Opacity=0.3)
}

K.grid          # V.C Mohan Grid
FlipVertical
Is arranged so that you can get results of the Levels() code at 3 different stages in the Levels function.
Graf looks like this


__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 28th July 2019 at 04:26.
StainlessS is offline   Reply With Quote
Old 25th March 2020, 21:53   #218  |  Link
Micheal813
Registered User
 
Join Date: Nov 2015
Posts: 59
I'm using AutoAdjust to tweak levels for VHS Captures. I've noticed that it doesn't necessarily give the same output even when using the same video and settings. The videos are compressed with Lagarith and I'm using AviSynth Plus.

Example:

Source=AviSource("Test.avi").AssumeTFF()
crp = Source.crop(8,8,-8,-8)

AutoAdjust(Source, external_clip = crp, high_quality=true, auto_gain=true, dark_limit=2.0, bright_limit=2.0, dark_exclude=.05, bright_exclude=.05, gain_mode=1, input_tv=True, output_tv=true, auto_balance=true)

I load the script into VirtualDub, select Fast Recompress, Lagarith, and then save the video. I reload the script without making any changes to the Test.avi or script and then save the video again. The second video may or may not be the exact same as the 1st. I base this on the file size and using a file comparison tool.

Is this normal?

Also, is it ok to use multi-threading (Prefetch()) with AutoAdjust? What would the correct MT mode be?

Last edited by Micheal813; 25th March 2020 at 21:55.
Micheal813 is offline   Reply With Quote
Old 25th March 2020, 22:21   #219  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Did you use Prefetch() when doing the AVI file saves ? [it ever happen without Prefetch ?]

(I'm guessin difference down to Prefetch, in some way, I've no idea hjow the multi-threading stufff works)
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 25th March 2020, 22:37   #220  |  Link
Micheal813
Registered User
 
Join Date: Nov 2015
Posts: 59
Quote:
Originally Posted by StainlessS View Post
Did you use Prefetch() when doing the AVI file saves ? [it ever happen without Prefetch ?]

(I'm guessin difference down to Prefetch, in some way, I've no idea hjow the multi-threading stufff works)

I did not use Prefetch.
Micheal813 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:21.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.