New filter: Autolevels

[T. Anschütz]

Hi,
some time ago I needed an autogain/autolevels filter and was unsatisfied with what ColorYUV had to offer, so I wrote my own. The main improvement over ColorYUV(autogain=true) is that the amount of gain is averaged over a number of frames which helps with scenes that have flashing lights etc.
This filter was useful for a VHS transfer I had to do recently. Maybe somebody else will find it useful, too.
The parameters are explained on the web page and in the readme file.

URL: http://ta41.zendurl.com/autolevels_en.php

Comments are always welcome, of course.

[IanB]

:( Any source code ?

Yes snapping the gain on a frame by frame basis, as ColorYUV does, sucks!

[TheRyuu]

How does it compare to HDRAGC?
(not at home otherwise I would try it)

I'll have to test this one out since I got a few bad sources that could use some gain.

[T. Anschütz]

:( Any source code ?

I re-uploaded the zip file with the source code. This is my first c++ program, so apologies if the code is awkward to read. Also, I haven't translated the comments, so they are all in German.

As for HDRAGC, I tried it out but it didn't seem suited well for my purposes. I think it uses a more complex algorithm, where my plugin basically just stretches the luma histogram and applies some averaging, but it was exactly what I needed.

[IanB]

Thanks for the source code.

As your filter make very heavy use of the cache you may wish to use the NewVideoFrame model for this filter instead of the MakeWriteable model for your output frame. This will avoid possibly needing to rerender the middle frame on subsequent calls.

Leak has brought to my attention some weaknesses in my algorithm for MakeWriteable VFB protection. It may be better not to depend on avisynth internals to defend against extreme cases of cache usage.

[T. Anschütz]

Thanks for the source code.

As your filter make very heavy use of the cache you may wish to use the NewVideoFrame model for this filter instead of the MakeWriteable model for your output frame. This will avoid possibly needing to rerender the middle frame on subsequent calls.

Leak has brought to my attention some weaknesses in my algorithm for MakeWriteable VFB protection. It may be better not to depend on avisynth internals to defend against extreme cases of cache usage.
Sorry for the late reply... I finally got around to looking at my code again. I noticed I was calling child->GetFrame() even for frames whose min / max / isSceneChange values were already known, which is of course silly, so I fixed that. It should call child->GetFrame() only once per frame now, at least for consecutive frames.

As for creating a new frame rather than modifying the original frame object, I see your point, but I'm assuming this isn't an issue any more, now that I'm doing my own caching (of the frame parameters), so I'm not calling GetFrame n times for each frame any more.
But maybe I'm wrong and using NewVideoFrame would still make things more efficient. If so, let me know.

Version 0.3 of the filter is up on my website.

[IanB]

boolean Autolevels::isSceneStart(int frameno, IScriptEnvironment* env) {
...
if (frameno > 1) {
PVideoFrame frame = child->GetFrame(frameno, env);
PVideoFrame prevframe = child->GetFrame(frameno-1, env);frame and prevframe are still fetched but are no longer used? :confused:
IntPair Autolevels::getMinMax(int frameno, IScriptEnvironment* env) {
PVideoFrame frame = child->GetFrame(frameno, env);
const unsigned char* pY = frame->GetReadPtr(PLANAR_Y);
...
hash_map<int, IntPair>::const_iterator mmIter = MinMaxCache.find(frameno);You still fetch the frames before checking you MinMaxCache, you probably only need to fetch it if there is no cache entry.

If you adjust your code to only fetch the "new" frame for the MinMaxCache and the "current" frame for outputing then the MakeWriteable model will be appropriate.

Also check on the libraries implementation of hash_map you may need to manually clear them in your destructor code.

[T. Anschütz]

You are absolutely right, I don't know what I was thinking. Thanks for spotting this!
I uploaded the fixed version, it is still v0.3, same filename.

With the unnecessary GetFrame calls are gone, the filter is a lot faster :)

I tried calling the hash_map destructor in the Autolevels destructor, but that crashed VDub, so I left it out for now. I need to actually read the documentation when I have some time.

[IanB]

Yes, it should be very much faster ;)

The short coming with some hash_map implementations is that it forgets the memory allocated to elements when it destructs.

As a workaround you may need to do a MinMaxCache.clear(), etc, somewhere in your code, Autolevels::~Autolevels might be a good place.

[rfmmars]

This is by far the best "autolevels" filter. It does get fooled on rare occasions in changing low light conditions, snapping quickly up and down a couples of levels. What you might do is to have it fade up or down slowly in low light so not to be noticed.

Excellent filter,

Richard
photorecall.net

[RickA]

Very nice. Will have to give this a try later on. Also, always good to see another Anschutz about. ;-)

Cheers,
Rick

[T. Anschütz]

Very nice. Will have to give this a try later on. Also, always good to see another Anschutz about. ;-)

Cheers,
Rick

http://forum.stuttgarter-zeitung.de/board/images/smiles/icon_beer.gif

[T. Anschütz]

This is by far the best "autolevels" filter. It does get fooled on rare occasions in changing low light conditions, snapping quickly up and down a couples of levels. What you might do is to have it fade up or down slowly in low light so not to be noticed.

Excellent filter,

Richard
photorecall.net
Thank you, I'm glad you like it.
What you are seeing is the filter "detecting" a scene change when there is none. To get rid of this, you can set sceneChgThresh to a higher value. At sceneChgThresh >= 256, scene change detection is effectively disabled and the "snapping" effect should be completely gone.

Of course, with scene change detection turned off, you may get undesirable effects at actual scene changes where the luma values change abruptly from one frame to the next. In these cases you don't want any fading to occur. So just try different values for sceneChgThresh to find out what works best in your situation.

Possibly, maybe, you will find that fades around scene changes aren't a problem and that sceneChgThresh = 256 is the way to go for you.

If you are a perfectionist, you can also deal with the false positives / false negatives by using the frameOverrides parameter, manually overriding the frames it falsely detects as scene changes or when it fails to detect one.

Even better would be a more intelligent scene change algorithm. Right now it looks at the maximum and minimum luma values of consecutive frames.
If someone knows a better algorithm that is not too complicated to implement, let me know.

[plane]

The link seems dead?

[T. Anschütz]

You are right, ZendURL is having problems. This link works for me:

http://www.zendurl.com/t/ta41/autolevels_en.php

[thymej]

Sorry thanks

[superuser]

thanks for the filter, will try it out.

[superuser]

thanks T. Anschütz for the filter. works gr8. If possible can you spin out a version say simple levels and provide more control to the user by input paramaters for the filters such as:

- gain to be achieved
- control for color control, if any, in luma and chroma plains (sorry this too much but thought may be will ask :) )

even in its native form filter is cool. thnxs again.

[AlanHK]

You are right, ZendURL is having problems. This link works for me:

http://www.zendurl.com/t/ta41/autolevels_en.php

I can't get this filter to work.

If I try to load the dll
LoadPlugin("p:\autolevels.dll")
I get an error message with some gibberish, as below.

I'm using Avisynth 2.57 on Win2k.

Also I note that the web page mentioned above has some problems, as the angle brackets used to indicate frame ranges are interpreted as HTML and thus invisible.

[unskinnyboy]

Does anyone have a working version of the latest version of Autolevels? If so, can they upload it somewhere, please? The previous URL is dead, and this don't seem to be hosted anywhere else.

[Chainmax]

I second that request, I'd like to try this filter out on some digicam footage and compare it to HDRAGC.

[NormanBates]

I found Autolevels v0.3 on my hdd and uploaded it for you.
Here is the link:
http://www.sendspace.com/file/7i6yx0

[unskinnyboy]

I found Autolevels v0.3 on my hdd and uploaded it for you.
Here is the link:
http://www.sendspace.com/file/7i6yx0
:thanks: This does seem the latest version!

[AlanHK]

Does anyone have a working version of the latest version of Autolevels? If so, can they upload it somewhere, please? The previous URL is dead, and this don't seem to be hosted anywhere else.

Attached.

But as you see above, I didn't have much luck with it.

[unskinnyboy]

AlanHK, Try the version NormanBates linked above. It works fine for me. Maybe yours is a corrupted version?

[AlanHK]

AlanHK, Try the version NormanBates linked above. It works fine for me. Maybe yours is a corrupted version?
They're identical.

[unskinnyboy]

They're identical.OK. Your attachment was unapproved (still is) when I posted my earlier reply, so I wasn't sure if it was. Not sure what your issue is then, sorry. Sure is specific to just you though. Works for me on my current Vista machine, and it used to work on my previous XP machine too (before it crashed a few months ago, that is).

[chriszxl]

links did not work............

[NormanBates]

Again:-)
I found Autolevels v0.3 on my hdd and uploaded it for you.
Here two links:
http://www.sendspace.com/file/z16kmr
http://rapidshare.com/files/135978459/autolevels0.3.zip.html

[Sharktooth]

direct download: http://www.webalice.it/f.corriga/misc/autolevels0.3.zip

[dreamy83]

URL is dead can some1 upload autolevels again
thanks

[Nightshiver]

No it's not. Use the direct download link from Sharky, it still works.

[AlanHK]

The file attached to my post above (http://forum.doom9.org/showthread.php?p=1136676#post1136676) is okay.

[IYG]

What is this filter's advantage?

[carlmart]

I would like to know what are the parameters to load on autolevels. The link is dead.

Any suggestions?

[sidewinder711]

Here you go:

http://www.mediafire.com/?ywjkgzmclgy

Cheers!

[EugenSU]

Something wrong occurs to colours after the filter. The picture is constantly painted in any colour. Has submitted on an input a black-and-white picture. On an exit has received floating displacement of colours. U and V components constantly float. I work with normal YUY2.

[frustum]

The wiki

http://avisynth.org/mediawiki/External_filters

says that autolevels works on YV12, YUY2, RGB32, RGB24 format pixels.

Certainly it works on YV12. If I convertToRGB24() before autolevels, it still works, but the effect is small. If I convertToRGB32() first I don't see any difference from the unprocessed video.

But my biggest problem is that ConvertToYUY2().autolevels() produces a very saturated purple video (it might be different on different clips, but this is what I'm seeing).

Is the wiki simply wrong?

I've worked around it with this:

(orig is in YUY2 format)
tmp = orig.convertToYV12().autolevels().convertToYUY2()
orig = MergeLuma(orig,tmp)

but if there is a more direct way to operate on YUY2() without the two conversions, I'd like to know.

[IanB]

Ouch! Yes this filter really only has a code path for YV12, although it should give correct results for RGB24 and near correct results for RGB32. It will be using the min(a, r, g, b) and the max(a, r, g, b) for its determinations, so the contents of the A channel can upset things. Unfortunately it will just make a mess of YUY2 chroma.

Given the filter just ploughs through RowSize bytes of data per line, you might get away with this :-(orig is in YUY2 format)
tmp = orig.GreyScale().autolevels() # make all chroma bytes 128
orig = MergeLuma(orig, tmp) # Overlay new luma bytes

[frustum]

Ian, your suggestion works well enough for the one scene I've tried it on. Thanks for the tip. Also thanks for the work you are doing on avisynth!

I've just had a look at the source for the autolevels plugin. I know nothing about avisynth data structures and internals, but it assumes the pixel data is planar. Won't it just treat y,u,v as three luma values? So if the image is very dark, it will think there are a bunch of y=128 pixels after performing the GrayScale() conversion, won't it?

Overall, it seems wrong to say autolevels() supports anything but YV12 (except in special cases) right now.

[Wilbert]

Did someone drop a mail/pm to the author about this issue?

[frustum]

Wilbert, the source code gives his name (Theodor Anschütz), but no contact info. When I tried to go to his website listed on the wiki for autolevels, it was a dead link.

[sinz718]

Awesome filter worked great for my current project, thanks!

[frustum]

I sent a pm to the author, but his profile says he joined doom9 in order to post his filter, then last posted a few months later, in the fall of 2007, so I suspect he won't notice it.

I've had a look at the code and it is something I could pick up and generalize, and perhaps improve it in other ways. It would be a good way for me to get my feet wet. However, the whole reason I started using avisynth was in order to transfer and restore about 35 8mm films to DVD in time for Christmas, as a present to my siblings. I may not have time to work on the plugin until that task is done.

However, I will mention a few weaknesses (other than blindly assuming YV12 format pixels) I see in the code, in order of difficulty.

(1) The average luma is computed by looking +/- N frames from the current frame (settable as a parameter). It starts its average from either frame -N or the most recent scene change before the current frame. It ends its average either at frame +N or at the frame before the last scene change before frame +N. This is all perfectly good unless there is more than one scene change detected after the current frame. This is cured by adding one "break;" after noticing a scene change after the current position, and is a very minor point.

(2) Once the rolling time averaged min/max luma for the current frame is computed, the luma distribution is shifted to map to 16 .. 235. This is desirable for some purposes, other times mapping to 0..255 is desirable. Perhaps this should be a parameter.

(3) The scene detection logic is very simple minded, although it probably works most of the time. The "min" and "max" luma for the frame are computed by ignoring roughly the outer 1% of the brightest/darkest pixels, so outliers don't have undue influence. If either the min or max for frame N has changed by more than a threshold value as compared to the previous frame, it is considered to be a new frame. This is probably pretty easy to produce false positives as well as false negatives.

Just from thought experiments, no actual data gathering, I think this can be improved. Using changes in the min/max is OK, but the threshold needs to be higher to avoid false positives, at the expense of false negatives. We can add other tests to help fix this.

Each frame in the averaging window has already exhaustively (and expensively) measured the luma histogram, but only the min/max are cached for the frame. It would be better save the whole histogram as well (only 256 ints). In addition to detecting whether the min/max changed significantly, with the histogram we could add tests for whether the mean (or median) has changed, and whether the autocorrelation between the histograms of successive frames has changed much. Further, rather than just considering whether frames n-1 and n correlate, look at the correlation for each successive pair. If there is a sudden change in the correlation it would be less prone to false positives. That is, even if the correlation changes a lot between frames n-1 and n, it is meaningful only if the correlation has been somewhat stable either before or after frame n. This will avoid a situation I've seen where in the middle of an ordinary scene the luma flashes rapidly up and down a few times.

[frustum]

Well, I couldn't resist, and I downloaded msvc++ 2010 express and set up virtualdub and avisynth on a virtual xp machine.

I now have a 0.4alpha version of autolevels. It fixes a couple small bugs, is probably measurably faster, supports yuy2, planar yuv, and interleaved rgb24/32 surfaces.

Great. The problem is that I take this dll which works fine in the virtual 32b xp machine and bring it to my real windows 7 64b box. It is running a 32b version of avisynth and 32b version of virtualdub too. When I try to load my newly minted autolevels, I get this bland error:
Avisynth open failure:
LoadPlugin: unable to load
"<path to my dll>\autolevels_0.4a.dll"
(<path to my invoking script>, line 5)

Am I overlooking something obvious, or is this where things start to get difficult?

[kemuri-_9]

you probably linked the dll in a fashion that requires the visual studio 2010 runtime to be installed on any computer you want to use the dll on.

[frustum]

kemuri-_9 -- you seem to be right. In the project properties for C/C+ code generation, I used to have "Runtime Library: Multi-threaded DLL (/MD)". Changing that to "Multi-threaded (/MT)" now allows my dll to run on my native machine.

However, the dll size went from 32 KB (similar to the size of the existing autolevels.dll) to a whopping 138 KB. Debugging info is disabled and optimization is enabled, as this is a release build.

I tried comparing my .vcxproj file to a 2006-era .vcproj file (from AddGrain) to find the difference, but despite being related, I can't correlate many of the options.

Are there any visual studio wizards that can spot the option which isn't set right? Meanwhile, I'll keep fiddling...

code generation command line:

/Zi /nologo /W3 /WX- /O2 /Oy- /D "WIN32" /D "NDEBUG" /D "_WINDOWS" /D "_USRDLL" /D "AUTOLEVELS_EXPORTS" /D "_WINDLL" /Gm- /EHsc /MT /GS /fp:precise /Zc:wchar_t /Zc:forScope /Fp"Release\autolevels.pch" /Fa"Release\" /Fo"Release\" /Fd"Release\vc100.pdb" /Gd /analyze- /errorReport:queue

linker command line:

/OUT:"C:\jim\video\autolevels\Release\autolevels.dll" /INCREMENTAL:NO /NOLOGO /DLL "kernel32.lib" /MANIFEST /ManifestFile:"Release\autolevels.dll.intermediate.manifest" /ALLOWISOLATION /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /PDB:"C:\jim\video\autolevels\Release\autolevels.pdb" /SUBSYSTEM:WINDOWS /OPT:REF /OPT:ICF /PGD:"C:\jim\video\autolevels\Release\autolevels.pgd" /TLBID:1 /DYNAMICBASE /NXCOMPAT /MACHINE:X86 /ERRORREPORT:QUEUE

[kemuri-_9]

what are you asking a question about?

the size increase with statically linking in the MSVC runtime into the DLL is normal.
the original size is small because it requires the use of an environment located MSVC runtime.

personally though, a ~100KB increase from statically linking in the MSVC runtime is extremely small.

[frustum]

kemuri-_9 -- that is reassuring, but I have looked at a bunch of the other plugins on warpenterprises. some come with source code, fewer still contain their msvc project files. It seems that people do usually specify /MT (multithreaded) instead of /MD (multithreaded DLL), yet all of the dlls weigh in under 40KB. In the grand scheme of things an extra 100KB isn't a whole lot these days, but it still seems to indicate I have some setting wrong.

[cretindesalpes]

It depends on which parts of the runtime library you are using. As kemuri said, no need to worry about these few extra kilobytes. Older MSVC versions may also produce lighter binaries given the same settings.

[kemuri-_9]

you still have some debugging on with the /Zi compilation option, other than that i don't see anything.

after fixing that i would say that you're being overly paranoid about the filesize.
most other plugins are not using msvc 2010 either, you could probably get a different (possibly smaller) filesize using an older verison.

[frustum]

It took a couple days, but I set up a web page for my version of autolevels, which is now revision 0.4

http://www.thebattles.net/video/autolevels.html

There are a number of ways this filter could be improved, but for now I need to set it aside so I can get back to my 8mm film conversion project. If you have any ideas, criticisms, or or questions, I'll be monitoring this thread, or you can email me at the address located on the web page I've just set up.

Edit 11/11/2010

I recommend not using this version, as the RGB path is too hacky, and the YUV path has only minor improvements over the 0.3 version. I'll release a better version soon.

[Hagbard23]

Thanks.. i will check this out on some of my old black/white films....i will do some testing on "regular" sources as well...

thanks for your effort...keep it up!

[lych_necross]

Sounds cool frustum. I'll check it out later tomorrow and let you know if I run into any problems. :)

[videoFred]

I have done some testing with the modified filter.
In RGB color space, autolevels works like it should, now.
Stretching the histogram to 0-255 is perfect.
I have tested it with VDubs levels filter, just to be sure.

The whites are a bit blown out both in YUY2 and in YV12 color space. The colors are much better in RGB color space, even if I convert back to YV12 afterwards.

I have disabled the scene change detection because this sometimes fails and then we have unwanted level changes in the middle of a scene. The filter works fine without the scene change detection. It needs a few frames (set with FilterRadius) to re-adjust at scene change, but that's not a problem for me. I have even tried FilterRadius=20 and this gives a very smooth levels adjustment.

Here are the test results, it is all transfered 8mm film.
Please notice the big difference in the red channel!
There might be some color shift, but I can live with that.

http://www.super-8.be/Doom/autolevels_001.jpg

http://www.super-8.be/Doom/autolevels_002.jpg

http://www.super-8.be/Doom/autolevels_003.jpg

http://www.super-8.be/Doom/autolevels_004.jpg


Fred.

[boondoggle]

I can confirm that stretching not working correctly in YV12. And also the red color problem described above. I used common VHS source not 8mm, not that it should matter..

[frustum]

Fred --

In your chicken picture, I'm not surprised that the whites are blown out, as they are a small percentage of the pixels. This is a feature of the algorithm that autolevels 0.3 uses, and which I didn't change in 0.4. The algorithm is to ignore the darkest and brightest 1/256th of all pixels (~ 0.4%) when determining the min/max luma for a frame. The intent is to avoid outliers in the distribution such that a stray noise pixels doesn't throw things off. If this feature didn't exist, if you had even one pixel with luma <= 16 and one with luma >= 235, autolevels would be a no-op.

Perhaps the best approach for this would be to add a parameter which specifies what fraction of outliers to ignore, and have it default to 1.0/256.0 for backwards compatibility. Another approach would be to still ignore outliers, but to clamp them to 16/235. Any opinion on which is better?

The color shift issue you see is no doubt due to the very hacky implementation of luma mapping for RGB I added. While it is infinitely better than what was there in 0.3 (which was to silently accept RGB as if it was YUV), it still leaves a lot to be desired. I spent only one evening on it and it seemed OK for the videos I tried, but I regret rushing it out now that I've seen some of the problems that occur. A great torture test is to try "somevideo.tweak(cont=0.05).converttoXXX.autolevels()" and see what happens beyond the expected heavy quantization.

Tonight I tried some less hacky but still hacky schemes to do the luma mapping in RGB space while still keeping the RGB proportions (in the just release 0.4, each of r,g,b are clipped independently). They all suffered from different failing corner cases.

So, I stole the yuv2rgb and rgb2yuv conversion routines from avisynth's guts (convert.h) and did the most direct code I could: one pixel at a time, convert rgb to yuv, remap the y using the same table I use in the yv12 code path, then convert back to rgb. That gives a nice image and isn't nearly as slow as I had feared it would be. If I go this route, I'd like to drop the "matrix" parameter, which I copied from the greyscale() filter. It is probably overkill as converttoyv12() from rgb is hardcoded for just one mapping function, not a choice of three.

Overall, I recommend not using 0.4 and getting tied into rgb (mis)behavior that I will fix soon.

BTW, I picked up this tip from a thread from a couple years ago. If you suspect that your components exceeding legal bounds, which is likely with autolevels' output, you can confirm it with limiter(), like this:

autolevels.limiter(show="luma")

Boondoggle --

Can you try your case with the 0.3 autolevels dll? I haven't changed yv12 processing, other than a very small fix to the scene detection logic.

[Emulgator]

Many thanks for the polishing of autolevels, frustrum !
Yesterday i had a short play with your autolevels 0.4
Perhaps the best approach for this would be to add a parameter which specifies what fraction of outliers to ignore
I second that. A percentage would be good, as in Photoshop, NikonScan etc.

For high resolution stills I tend to let some 0.05% of pixels go clamped.

For SD video it may be different and other people may have different needs,
so it would be nice to have a luma range and a clamp percentage to specify.

If we only could have autolevels controlling LaTo's Smooth Curve, Smooth Levels, Smooth Tweak, it would be awesome.
Since steady histograms come out there I find myself wanting no more histogram "holes".

[Gavino]

If I go this route, I'd like to drop the "matrix" parameter, which I copied from the greyscale() filter. It is probably overkill as converttoyv12() from rgb is hardcoded for just one mapping function, not a choice of three.
Not sure what you mean here - ConvertToYV12() from RGB supports the 'matrix' parameter (it converts first to YUY2 using the specified matrix, then to YV12).

[IanB]

The original code when used with RGB24 was almost doing the right thing (just wrong limits). In RGB each colour channel can range from 0 to 255. The colour depicted is related to the ratio of the R, G and B channels. This ratio must be maintained. You should just be looking for the Maximum and Minimum values of any channel and then scaling and biasing all RGB channels equally, the A channel should be ignored.

Like most filters that fiddle with brightness and contrast in YUV colour space we cheat and only process the Luma channel. The 2 chroma channels are not adjusted. The hue of the colour is related to the ratio of the U and V chroma channels. The saturation of the colour is related to the ratio of the U and V channels to the Luma channel. The luma only cheat may lead to noticable changes in saturation when the adjustment is not small.

Expect different results when processing RGB compared to YUV. The RGB results can be strictly correct, the YUV results will be a compromise. You can see the effect easily in VirtualDub with the inbuilt Levels filter by toggling the "Operate in luma instead of RGB" option.

And all YUV results should be the same. YV12 and YUY2 being different is a bug.

[frustum]

Gavino --

I looked at convert.h just enough to find the yuy2 to rgb conversion (and back) and didn't see any option for matrix. Now that you've clued me in, I see the matrix choices in convert_yuy2.cpp.

IanB --

Doing everything consistently in yuv space has the advantage that a given video source converted into rgb or yuv space running through the filter would give very similar results, whereas running it through using max(r,g,b) vs luma(r,g,b) would give different results. In RGB space, a pure blue pixel has already pegged the range (a 0x00 component and a 0xFF component) and no levels adjustment is possible, while in YUV space, such a pixel has a luma value of only ~11%.

I've looked a bit at some imagemagick filters for ideas and an array of options are offered, including adjusting each rgb channel independently, or by luma, or intensity, or brightness, or ...

http://www.fmwconcepts.com/imagemagick/autolevel/index.php

From the thumbnail example given it is hard to discern the difference between some of them. At any rate, I guess there isn't a one right way to do it, that different approaches make sense in different situations. Because I'm not doing much in the way of optimization (unrolling or assembly), each flavor of conversion routine probably costs only a few hundred bytes incrementally, so it wouldn't be unthinkable to offer a mode option to specify joint rgb vs separate rgb vs 709 luma vs 601 luma vs avg intensity. I just had a look at levels.cpp and it isn't trying any harder than I am to be fast, so I guess it is acceptable.

It would also be a relatively small extension to offer an auto gamma option based on the same statistics being gathered for autolevels and the same machinery for adjusting pixel values. From the same source as before:

http://www.fmwconcepts.com/imagemagick/autogamma/index.php

A more fully featured autolevels might want to support more of the features of levels. Ignoring all issues of feature creep and just dreaming out loud, I could imagine these features fitting together into one filter:

Version 0.3 options:

[filterRadius] -- number of frames before/after to average over

[sceneChgThresh] -- sensitivity to detecting scene changes

[frameOverrides] -- specify frames or ranges to ignore/enforce scene changes

new options:

[debug] -- info/debug option (in version 0.4 already)

[clip_low] -- fraction of dark pixels to ignore (default 1/256)
[clip_high] -- fraction of bright pixels to ignore (default 1/256)
[clip] -- specifies same value for clip_low, clip_high

[input_low] -- as in levels()
[input_high] -- as in levels()
[output_low] -- as in levels()
[output_high] -- as in levels()
[coring] -- as in levels()
[gamma] -- as in levels()

[autogamma] -- exclusive of use with [gamma], it is 0 by default, meaning don't autoadjust gamma, while 1.0 means estimate and apply a gamma; weights between the two change it in proportion, above 1.0 gooses it higher than what the algorithm calculates.

[mode] -- used only for rgb formats, "rgb" means adjust each r, g, b plane separately, "joint" means use the min/max of rgb component of each pixel and adjust all planes the same, while rec709/rec601/pc.709/pc.601/average are like the "matrix" option for converting rgb/yuv.

One thing is I would want the filter to produce the same results as the 0.3 version given the same parameters. This means that by default outliers in the distribution could map to luma < 16 and luma > 236. There should be some mechanism to specify that these values should be clamped to [16 .. 235], perhaps just another flag, but maybe there is some less verbose way to infer this without another parameter. For instance, mode could default to pc.601 and "pc.*" signifies whether luma is limited to [16 .. 235] when operating in yuv space.

[boondoggle]

I'm not sure if you still want the error testing with the 0.3 version, but i did it and the stretching error was there in YV12 as in the new version. I too agree on that the percentage option would be nice.

[frustum]

I had some time to work on autolevels and I've made some fixes and improvements to the filter. Perhaps I went overboard. Nevertheless, I'm offering it as a beta 0.6 of the filter; all feedback is welcomed.

Rather than doing all the pixel adjustment with new code and handling RGB adjustment by doing on the fly RGB to YUV to RGB remapping, I took a different approach. Instead, the filter measures the luma histogram and takes stats, but it recycles the code from the built-in levels() function to do the pixel value remapping. In essence, autolevels() is kind of a front end now to levels().

There are three benefits to this approach. First, the old autolevels() code only remapped the Y plane and didn't touch the U and V planes, which was an error. U and V are not orthogonal to Y; when the brightness of a pixel is increased, not only is Y is made larger, but U and V must move proportionally further away from their midpoint value, otherwise color saturation is lost. Second, the old autolevels allowed luma to be remapped outside the range of [16 ... 235]. Third, the levels() code handles RGB pixels in RGB space without all the YUV remapping. For an example of how processing U and V improves color fidelity, please see my webpage (http://www.thebattles.net/video/autolevels_comparison.html) about it.

Because I'm not doing RGB->YUV->RGB remapping, I've removed the "matrix" parameter which I had introduced in the previous version (0.4).

The other big new feature is I've added an autogamma option, along with the midpoint parameter. If autogamma=true, then a gamma is chosen such that the mean of the distribution is remapped to the midpoint of the range (0.5 by default, but usually I find a midpoint of 0.35 or 0.4 to be more pleasing). Like the levels, the gamma value is based on the mean luma value for a window of frames around the current frame. I've also added a filter called autogamma() which is identical in its functionality to autolevels, except that by default autogamma() has autolevels=false, autogamma=true by default. The simplest invocation would simply be out = in.autogamma().

While I was at it, I incorporated most of the features from levels(): gamma, input_high, input_low, output_high, output_low. "gamma" (default=1.0) can be used to manually specify a gamma remapping function that can be folded in to autolevels at zero per-pixel cost; it is ignored if autogamma=true. input_high and input_low are normally computed by the autolevels() function as the brightest and darkest pixels of the frame, but one or both can be overridden manually. output_high and output_low can be used to specify the output range of values (normally 16..235 for Y, 16..240 for U,V, and 0..255 for RGB).

There was a version 0.5 of the filter which I distributed to one person (videoFred) that added a few features. One of the complaints about autolevels() is that it ignores the darkest and brightest 1/256th of all pixels when determining the range of the luma distribution, which can be too much. The parameter "ignore" can specify the fraction of outliers to ignore, and ignore_high and ignore_low can be used to specify them separately if you want asymmetric tail clipping.

There is one last feature, also added by the 0.5 version. The autolevels and autogamma adjustments are based on the distribution of luma values in the frame. Sometimes the edges of the video are not representative of the image and we might want to ignore the edges for the purposes of gathering statistics (although the full frame is always processed). "border=20" would indicate to ignore 20 pixels along all four edges; border_t, border_b, border_l, border_r can be used to individually specify borders for top, bottom, left, right respectively.

Yes, there are a lot of parameters now, about 20 of them, but most of the time you can leave most of them at their default value.

There is one feature of levels() that I didn't implement: coring. The image luma statistics are taken before coring remapping, and if coring was enabled, the input/output high/low values are taken while Y has been mapped from [16..235]. Thus if the before-coring minimum Y value is 30, to map 30 to zero with coring enabled would require requesting an input_low of 14 (== 30-16), and an input_high of 191 (== (180-16)*255/219). This is just weird, so internally I keep coring off. I could be talked into adding it just to have a strict superset of levels() functionality.

The full webpage for the filter is here (http://www.thebattles.net/video/autolevels.html).

[AlanHK]

The full webpage for the filter is here (http://www.thebattles.net/video/autolevels.html).

Thanks.
The zip doesn't include any documentation, not even a readme. I can scrape the webpage, of course.

[julius666]

Hi, and thank you for your hard work frustrum.

Could you explain how sceneChgThresh works? How could I turn it off? Since I have a source that is one big scene (no real scenechanges) and sometimes autolevels makes a small, barely visible flicker. I'm pretty sure that autolevels reads those 'flickering points' as a new scene.

[frustum]

AlanHK -- I've updated the zip file; it was an oversight to not include the "Autolevels_ReadMe.txt" file. The contents of the readme match what is found on the webpage.

julius666 -- sceneChgThresh is unchanged from its operation since before I touched the filter. How it works is this. The filter computes the luma distribution for all the pixels in the frame, ignoring the smallest and largest N outliers in the distribution (normally 1/256), yielding low and high values. If the low value differs by more than "sceneChgThresh" from the previous frame's low value, it is decided that the current frame must be a new scene. Likewise, if the high value differs by more than "sceneChgThresh" from the previous frame's high value, it is decided that the current frame must be a new scene.

As the documentation explains, you can disable this feature by setting scheChgThresh to a high value which is impossible to trip, like 255.

[frustum]

I have revised my 0.6 beta release and now have a 0.6 "final", although I don't know if anybody other than me has used it at all.

It was probably unnecessary, but as I went 90% of the way towards covering all levels() functionality, I've gone and added the "coring" flag to autolevels as well. One oddity is that coring is true by default for levels() but false by default for autolevels() in order to backwards compatible with older versions of autolevels().

So now levels(a,b,c,d,e,coring=f) can be performed as autolevels(input_low=a,gamma=b,input_high=c,output_low=d,output_high=e,coring=f), not that you'd ever want to do this exactly. Other than that, all the new features of 0.6 beta remain the same.

Updated document and zip file containing the new dll, source, and docs here: http://www.thebattles.net/video/autolevels.html

All feedback is welcomed, even opinions like: hey, that is nice you are making all those changes, but you've gone off the rails.

[smok3]

a quick test, using script like this;

avisource("pathtoavi.avi")
info()
autolevels(sceneChgThresh=5, filterRadius=50)
converttoyuy2()
yadif()

interstingly every clip looks wrong one way or another, (either to bright or too dark or it fluctuates like somebody would forget to turn-off automatic settings on camcorder). i think it finds scene changes correctly, but really can't be sure.

[videoFred]

Autolevels 0.6 works very fine here. I'm testing it right now. Some of my 8mm transfers are captured in 14-140. (because I was to lazy to set my camera :p ) Those have to be stretched a lot by autolevels. Here it goes wrong sometimes. Blown out whites. Perhaps I do not understand a few parameter settings for files like these?

PS: to be more precise: those pixels are blown out before they reach the 255. I'm using Trevlacs histogram for VDub to check this. It can be seen as a small one line peak at the end of the histogram.

Fred.

[frustum]

smok3 -- "sceneChgThresh=5" means the scene change detection has a hair trigger. The whole point of autolevels is to smoothly track the changing scene lighting, but it attempts to find scene changes so as to allow sudden changes when warranted. With the threshold set so low (by default it is 20), you will keep getting false positives and thus no averaging. set "debug=false" and it will indicate which frames it thinks begin a new scene. Try increasing it until the scene change detections are plausible. Remember too that it isn't necessary to detect all scene changes. If scene N and scene N+1 have similar enough lighting that the scene change detection doesn't trigger, it probably means it is OK to average the luma statistics across the scene boundary.

Fred -- I'm not sure what to do about those pixels that are blown out either. Do you have any ideas for algorithms to help in such situations? The autolevels algorithms is simply to map the left end of the histogram to luma 16 and the right end of the distribution to 235. It should only blow out pixels which were to the right of the cutoff point. You can set the ignore parameter to set the fraction of pixels to sacrifice on the tails of the distribution. Setting it to zero will prevent any of them from getting saturated, but then even a single pixel in a frame which is white can prevent autolevels() from adjusting anything.

When operating in rgb mode, a single mapping table is set up to convert old_intensity to new_intensity. This table is applied independently to each of the RGB channels. If the gain is high, a value like (r,g,b)=(240,220,210) might map to (255,255,255). Whatever hue it had will get crushed into white. This is no different than the way levels() does it, and in fact, I stole the levels() code to do the pixel adjustment. A smarter, and slower, approach would be to find the max of the three components and if it exceeds 255, the lesser components would be scaled in proportion to the amount the largest component experienced to get to 255.

[videoFred]

Fred -- I'm not sure what to do about those pixels that are blown out either. Do you have any ideas for algorithms to help in such situations?

Creating algorithms is way beyond my knowledge, sorry.

But I have very good results with the border parameter. Border= 120 helps a lot in a situation like this.

Tweaking output_low and output_high also helps. Anyhow, of cource these narrow 14-140 captured files are extreme examples.

Fred.

[Mini-Me]

frustum, this is a very promising filter for people like me with a LOT of scenes to correct. I'm really glad you decided to adopt it. :)

I do see a potential area for improvement though: Because it relies on the built-in levels filter, Autolevels inherits the weaknesses of the former. Specifically, the contrast stretch and gamma shift are done at low precision, so they result in banding in the image and histogram. If you'd be willing to change the back end again or include the option for an alternate backend, you'd be able to bypass these issues and obtain higher-quality output. I'm not sure how many options are out there for high-precision levels adjustment with dithering, but I just noticed this one called SmoothAdjust (http://forum.doom9.org/showthread.php?t=154971) at the top of the forum. It seems to be YV12 only, so it's not ideal, but there might be other alternatives as well. Anyway, what are your thoughts on the basic idea? If the implementation would be too difficult or clumsy, what about debug text output listing the levels adjustments for each frame or section? (The desperate could then parse and format that output into trim and filter calls.)

[frustum]

Mini-Me --

I haven't studied smoothadjust, but it may make more sense to put autolevels logic into smoothadjust than it the other way around. Smoothadjust doesn't come with source code, so it would have to be called from within the filter. I know there is a way to do it, although I haven't looked into it. I'll add it to my list of things to think about when I next get time.

[boondoggle]

It seems we still have to do the ConvertToRGB before filtering?
I tried to fiddle around with the new settings and when I tried Autolevels(filterRadius=10,sceneChgThresh=255,autolevel=false, autogamma=false,border=5) which I thought would do nothing, it still did. And it seems like Fred says that the whites are blown out still. I know you say "detect scenes that should be kept dark (may not be easy to do)" but if you do implement it (I almost convinced my programming friend to look at it) I also, if possible, want to add to that feature a kind of logarithmic stretching(?) i.e. the more it needs to stretch luma the less it does. I dreamt of a script that degrained harder the more the luma was stretched :D .. keep versions comming though! :thanks:

[Jeremy Duncan]

Is autolevels() the same as ColorYUV(autogain=true) using frustum's autolevels_0.6_20110109.dll?

[frustum]

Jeremy, ColorYUV(autogain=true) looks only at the histogram of the current frame to adjust levels for that frame. This can lead to sudden jumps in the levels.

autolevels() (the original one or my recent versions) improve on the situation by using the histogram stats from N frames before and after the current frame to adjust the levels of the current frame. By using a rolling average it smooths the adjustment so there are no sudden jumps.

So, in short, no, autolevels() is not the same as ColorYUV(autogain=true)

[Jeremy Duncan]

Hi Frustum,

I use a frame doubler and I tried to use autolevels and when I framestepped the video would go dark tone and next frame light tone.

I traced the problem to the input_low, input_high, output_low, _output_high being different values.
So I fixed this dark and light tone error by setting it like this:
input_low=16, input_high=235, output_low=16, output_high=235, but this ruined the effect I was looking for which was improved frame doubling efficiency.

This was the code I used:

setmtmode(5,0)
SetMemoryMax(512)
video=ffdshow_source().changefps(29.976, linear=true)
A = video
setmtmode(2)
SetMemoryMax(512)
super = A.MSuper(pel=2, hpad=12, vpad=12, rfilter=1, levels=1, isse=true).autolevels()
backward_vec = MAnalyse(super, isb=true, blksize=16, levels=1, search=3, searchparam=2, isse=true, sadx264=7, pnew=180, lambda=2100, lsad=1300,global=true)
forward_vec = MAnalyse(super, isb=false, blksize=16, levels=1, search=3, searchparam=2, isse=true, sadx264=7, pnew=180, lambda=2100, lsad=1300,global=true)
backward_vec_1 = MRecalculate(super, backward_vec, blksize=8, search=3, searchparam=1)
forward_vec_1 = MRecalculate(super, forward_vec, blksize=8, search=3, searchparam=1)
A.MBlockFps(super, backward_vec_1, forward_vec_1, num=FramerateNumerator(A)*2, den=FramerateDenominator(A)*1,thscd1=400, mode=3, thres=100)
GetMTMode(false) > 0 ? distributor() : last

Then I remembered what my buddy subjunk did in a different thread and tried that and it fixed my tone problem, see the code below:

setmtmode(5,0)
SetMemoryMax(512)
video=ffdshow_source().changefps(29.976, linear=true)
A = video
setmtmode(2)
SetMemoryMax(512)
super_1 = A.MSuper(pel=2, hpad=12, vpad=12, rfilter=1, levels=1, isse=true).autolevels()
super_2 = A.MSuper(pel=2, hpad=12, vpad=12, rfilter=1, levels=1, isse=true)
backward_vec = MAnalyse(super_1, isb=true, blksize=16, levels=1, search=3, searchparam=2, isse=true, sadx264=7, pnew=180, lambda=2100, lsad=1300,global=true)
forward_vec = MAnalyse(super_1, isb=false, blksize=16, levels=1, search=3, searchparam=2, isse=true, sadx264=7, pnew=180, lambda=2100, lsad=1300,global=true)
backward_vec_1 = MRecalculate(super_1, backward_vec, blksize=8, search=3, searchparam=1)
forward_vec_1 = MRecalculate(super_1, forward_vec, blksize=8, search=3, searchparam=1)
A.MBlockFps(super_2, backward_vec_1, forward_vec_1, num=FramerateNumerator(A)*2, den=FramerateDenominator(A)*1,thscd1=400, mode=3, thres=100)
GetMTMode(false) > 0 ? distributor() : last

link to video clip I used, it's dancing, and helicopters, and a man riding his motorcycle. (http://www.mediafire.com/?3jvt2k2mu25fydo)

Thanks man. :)

[frustum]

Jeremy,

You are doing

A = video
...
super = A.MSuper(blah, blah, blah).autolevels()

I don't know the details of how mvtools works, but my vague understanding is that the superclip processes the original video and then stores this analyzed data (the motion vectors and who knows what else) as video within the superclip. Performing autolevels on the superclip will screw with this generated data.

Shouldn't you be doing

A = video.autolevels()
...
super = A.MSuper(pel=2, hpad=12, vpad=12, rfilter=1, levels=1, isse=true)

[Didée]

Shouldn't you be doing

A = video.autolevels()
...
super = A.MSuper(pel=2, hpad=12, vpad=12, rfilter=1, levels=1, isse=true)
Yes, definetly. MVTools2 does not store the vectors in the super clip, so that's not an issue. But doing the autolevels first is a) faster because of less image area to process (the super clip is *much* bigger), and b) avoids the dark/bright flickering issue. This issue was directly caused by the unlucky autolevels usage: in case of (exact) framerate doubling, the even frames are just the untouched original input frames, hence they aren't affected by the autolevels.

* * * *

BTW, why are all those "realtime" framerate doubling scripts using search=3/searchparam=2? This IMHO is plain silly! search=3 is very slow, and then the speed problem is fixed by using a very small search range and few superclip-'levels'.

Jeremy, that script has a search range of only 1 or 2 pixel! (levels=1, searchparam=1/2). "Motion search" is something else. :p

Really, there is no point in using the best-but-slowest search method, and then cripple it so badly that it becomes worse than other, more sane search methods would be to start with. :rolleyes:

[Jeremy Duncan]

Here is the code I updated:


setmtmode(5,0)
SetMemoryMax(512)
video=ffdshow_source().changefps(29.976, linear=true)
A = video.autolevels()
B = video
setmtmode(2)
SetMemoryMax(512)
super_1 = A.MSuper(pel=2, hpad=12, vpad=12, rfilter=2, isse=true)
super_2 = B.MSuper(pel=2, hpad=12, vpad=12, rfilter=1, isse=true)
backward_vec = MAnalyse(super_1, isb=true, blksize=16, levels=2, search=2, searchparam=1, isse=true, sadx264=7, lambda=200, dct=10)
forward_vec = MAnalyse(super_1, isb=false, blksize=16, levels=2, search=1, searchparam=1, isse=true, sadx264=7, lambda=200, dct=10)
backward_vec_1 = MRecalculate(super_1, backward_vec, blksize=8, search=2, searchparam=1, dct=10)
forward_vec_1 = MRecalculate(super_1, forward_vec, blksize=8, search=1, searchparam=1, dct=10)
B.MBlockFps(super_2, backward_vec_1, forward_vec_1, num=FramerateNumerator(A)*2, den=FramerateDenominator(A)*1,thscd1=350, thscd2=100, mode=3, thres=30)
GetMTMode(false) > 0 ? distributor() : last

I don't want to discuss my code in this thread though since it's about levels. Please go to the mvtools thread in the avisynth usage section where I posted my codes in response to Gavino if you want to talk about this code. Thank you both for pointing that out to me. The source is in that other thread, so please try the codes you suggest so you know for sure that the strobing light dark tones are fixed by your suggestions, as it is, it works.

[SubJunk]

Just wanted to say I found inconsistent results when using autolevels with mvtools. Using our test video (the PetitDragon one) it seems to improve as many frames as it worsens.
It's a great plugin for other purposes though, great job Jim :)

[frustum]

Just wanted to say I found inconsistent results when using autolevels with mvtools. Using our test video (the PetitDragon one) it seems to improve as many frames as it worsens.
It's a great plugin for other purposes though, great job Jim :)

Although I have no doubt that autolevels isn't guaranteed to only ever improve a clip (now wouldn't that be a great plugin), it would help to know what were the specific problems.

One common one is when a part of a scene should remain dark, autolevels will attempt to brighten it, which both affects the mood of the scene and results in a crappy, noisy picture.

Another common problem is too much clipping. The defaults for autolevel are set to match what it was with version 0.3, when I inherited it, which ignores the bottom 0.4% and top 0.4% of all pixels, pushing all of them into the clipping range. This is typically too much, and can often be fixed by setting it to something smaller like 0.1% or 0.05%.

For both of the above situations I have some ideas for heuristics which would improve the situation, but it doesn't seem possible to avoid all possible pitfalls.

Are you seeing some other problems? Are you using it on a yuv or rgb clip?

[SubJunk]

Sorry frustum, I was just replying to Jeremy.
Nothing was wrong with autolevels, it was more that it was inappropriate to use a plugin with this purpose in that particular script :)

[johnmeyer]

Another common problem is too much clipping. The defaults for autolevel are set to match what it was with version 0.3, when I inherited it, which ignores the bottom 0.4% and top 0.4% of all pixels, pushing all of them into the clipping range. This is typically too much, and can often be fixed by setting it to something smaller like 0.1% or 0.05%.

For both of the above situations I have some ideas for heuristics which would improve the situation, but it doesn't seem possible to avoid all possible pitfalls.

You've mentioned this several times in this thread, and to deal with it you added the "ignore" parameters. This greatly improved the usefulness of this filter. However, you've asked several times prior to this post for ideas for another heuristic to help cope with this fundamental problem, so here are a couple of ideas. Everything that follows, except for the last paragraph, focuses on calculations made within a frame (as opposed to the averaging that you do across "n" frames).

It seems to me that you need to introduce the additional concept of pixel grouping. Starting with a pathological case, under almost all circumstances I can think of, pure white single dots should be ignored when doing the levels calculation. To take the simple case, if a dot is 255 and all immediately adjacent pixels are a value "x" less than this, then the dot should be thrown out of all calculations.

This extreme case seems fairly straightforward (as are most extreme cases). As the number of adjacent pixels that are as bright or brighter (or as dark or darker) increases, more "thought" should be given to including this group of pixels in the calculations.

With this concept, the larger the grouping of spots that exceeds the upper or lower limits (ignore_high, ignore_low), the more likely your algorithm should include them in the calculations, even if the threshold for total number of pixels beyond the boundary hasn't been exceeded.

One other heuristic would be to add the concept of an outlier. You obviously know the statistical definition. The visual problem I'm trying to define is where you have something like a specular reflection from a window pane, something like you'd get if you were videotaping out the front of your car, and the sun hits the rear window or chrome bumper of the car in front of you. This would give you a big area of pure white, and the auto-exposure in the camera would reduce the overall exposure, but you would sill have the huge number of pure white pixels. The histogram of the result would have this huge number of pixels at or around 255, but then a huge gap between those and the rest of the histogram. Situations like this might be detected by looking for large gaps in the upper and lower boundaries of the histogram, with virtually no pixels at all between this big spike of pixels at the very end of the histogram, and the point at which the normal tail of the histogram resumes. If this additional heuristic existed, such video would still be automatically corrected, even though a very large percentage of pixels were nearly pure white.

The following really bad ASCII art is attempt to show this. My suggestion is that the "hump" on the right side should be ignored (the real-life example would probably have the grouping on the right much closer together so it looks more like a single spike). The length of the "dead zone" between the two humps would be the "tip off" that the upper pixels are bogus.

So here's the heuristic: the narrower the right-hand hump; the closer that hump is to 255; and the bigger the gap of almost no pixels between that hump and the tail of the main distribution, the more aggressively the algorithm should ignore all the upper pixels.

There is also the issue of a one-frame departures from average, such as you get when someone uses a camera flash. Such frames should not be part of the moving average. However, I assume you already do this.

|
| * *
| * * **
| * * *
| * *
| * * *
| * * *
| * *
| * * * *
| * ************* *
_______________________________________________[

[frustum]

John, thanks for the input. I had some time this weekend to think a little about autolevels, and one of the ideas is along the lines of your first comment -- a clustered group of N luma=255 pixels should matter more than N single pixels with luma=255 which are scattered around. My thought was to pre-process each frame with something like a median filter with a small radius (maybe just 1) to get rid of those specs. Perhaps one of the more sophisticated speckle removal filters would be enough. Stats would be taken on this munged frame and applied to the original frame pixels.

Warning: rambling thoughts ahead.

Another fundamental problem is this: the statistics take the mean histogram high/low values in a window +/-N frames of the current frame (ignoring frames judged to be from a different scene). The problem is that if the histogram changes suddenly, say due to the scene panning and something glaring in the sun pops into frame, this bright area will be visible in N+1 frames or fewer but it is averaged with 2N+1 frames by the time that bright spot becomes the center of the averaging window. To be more specific, say maxluma=200 for a good long time, then this bright object suddenly appears and stays in frame, and shows up as a spike near luma=240. maxluma averaged over the 2N+1 frame window will be (N*200 + (N+1)*240), or roughly 220 for larger values of N. This will cause all the pixels with luma > 220 to get saturated, wiping out all detail in the bright regions.

I think I should take this into account and do a smooth limiting function when calculating the average maxluma. Doing something simple like avg_maxluma = min( sum(maxluma(frame=current-N to current+N)), max(maxluma(frame=current-N to current+N)) ) would fail -- the first time the bright frame appeared in the averaging window would cause a sudden drop in brightness. Instead, it should have an influence based on its distance from the center of the window.

I've downloaded and perused a number of PDFs of people thinking about this same problem. One takes a very sensible approach of figuring out minluma and maxluma for each scene, then the correction for each frame is based on a combination of the minluma & maxluma of the current frame and the current scene. This algorithm isn't directly suitable for an avisynth filter, as a literal implementation would require searching forward and backward potentially for an unbounded number of frames until a scene boundary is detected. In real life, scenes lengths are typically less than a minute long, but that can still be more than 1000 frames. It might make sense to approximate the algorithm by having wide "scene window" and a smaller averaging window like currently exists.

Another failure mode of autolevels() which I had never thought about is the problem of fades and blended transitions. I was unaware of this because all of my work has been on 8mm home movies, and every cut is a jump cut. I think it would be relatively easy to look for a smooth transition to/from near black and to preserve it rather than attempting to boost those dark frames. Having that larger "scene" window would make it easy to see such transitions coming.

I had also long thought of some ways to improve autolevels' scene boundary detection logic using just the already-gathered histogram data, but before charging off to try it, I did a google search. It turns out that detecting scene boundaries is a ripe research area. Google on "shot boundary detection" and you'll find dozens of papers. There is even an annual shoot-out where researchers pit their latest algorithms against a battery of videos to see which works best. Many of these algorithms are not suitable for avisynth for the same reason stated before: they assume that the entire video will be processed in order, perhaps with two passes. avisynth filters can do this, but even with caching, the first frame out of the filter may take many seconds/minutes to produce.

One final observation -- none of the photo and video equalization algorithms operate in YUV space -- they either use RGB or better yet, HSV. It wouldn't be hard to change autolevels to convert from YUV to HSV, make adjustments, then convert back, but it would be performance hit and I know for a great number of avisynth users, speed is more important than quality (well, quality matters, but not if it drops processing to sub-realtime).

[johnmeyer]

Warning: rambling thoughts ahead.

Acthung: Even more rambling thoughts ahead ...

The problem is that if the histogram changes suddenly, say due to the scene panning and something glaring in the sun pops into frame, this bright area will be visible in N+1 frames or fewer but it is averaged with 2N+1 frames by the time that bright spot becomes the center of the averaging window.
To me, the heart of a better autolevels algorithm is dealing with these anomalies. Obviously, you first need to be able to correct a single frame and, to do this, you need to decide what constitutes correct levels. The original autolevels code did this and I think you have taken that part of the code forward. As I understand it, you basically apply a gamma function so the midtones are corrected more than the darkest and lightest part of the picture. To me, this is the toughest part of the whole thing and would be the place I would rely on other people's research. There is more art than science in figuring this out and doing it well.

However, I don't think I would look to averaging to provide a better fix to an individual frame's exposure. Averaging should instead, I think, be viewed ONLY as a way to avoid "hunting" and "pumping" problems, and not as a way to get a better exposure. After all, the sun may not just "pop" into the frame, but might be part of the scene for many minutes. If you are relying on averaging to make a proper correction, then in this example you'll never get anything different with which to average.

One takes a very sensible approach of figuring out minluma and maxluma for each scene, then the correction for each frame is based on a combination of the minluma & maxluma of the current frame and the current scene.
It is definitely true that some footage does exhibit the "shifted histogram" where the darkest pixels are way too bright or the brightest pixels too dark. Even when this occurs, you still need to truncate some outliers because there always seems to be a 0 or 255 out there somewhere, even when the exposure is clearly wrong. So simply using maxluma and minluma I think will lead to bad results, but using a slightly modified version, where you first throw out some outliers, might work. Concentrating on these values rather than using some sort of average would certainly stop the algorithm from trying to correct a sunset where the average luma is low, and there is almost nothing in the center of the histogram.

Once you correct the frame, I think I would use some sort of averageluma function to determine how to average the exposure. I actually think that you want to keep the number of frames averaged quite small, however. I can see no good coming from averaging more than about one or two seconds of video. Imagine standing in a forest at the edge of a lake. You start pointing the camera to the left, into the forest. The video is very dark, but the camera auto exposure turns up the gain so it isn't too dark. It still needs to be made lighter. The camera then pans across the lake, where the sun is hitting the water. The camera's autoexposure tracks the change, although probably with a lag. When the camera pan leaves the lake and points to the forest on your right, everything gets dark again. You don't want to dampen the corrections too much (by averaging LOTS of frames), or you'll end up not correcting anything.

I think you need to actually write down and clearly answer the question: "why am I averaging frames?" Make sure the objectives are clear, and that averaging is the best answer to the problem you are trying to solve.

So, I think if I were doing this, I'd spend most of my time trying to come up with a fantastic single frame exposure (levels) correction, and then at the end apply some "safety valves" that keep the correction from failing when subjected to specular reflections, scene changes, bright camera strobe flashes, and other short duration transient phenomenon.

Another failure mode of autolevels() which I had never thought about is the problem of fades and blended transitions. I was unaware of this because all of my work has been on 8mm home movies, and every cut is a jump cut.
I don't see why you have to consider this. I do film transfers all the time, but also edit lots of video. Fades and blended transitions don't happen in unedited video (I guess you can do them "in camera," but no one ever uses those features). Once footage has been edited, it then contains these transitions, but that is true of edited 8mm, Super 8, 16mm, 35mm, & other film formats as well. I would not bother to try to deal with this.

I had also long thought of some ways to improve autolevels' scene boundary detection logic using just the already-gathered histogram data, but before charging off to try it, ... I've written several scene detection scripts and, while they all fail sometimes, they generally work well enough to be useful. A simple

(YDifferenceToNext(j)>blankthreshold)&&YDifferenceToNext(i)<blankthreshold

works pretty darned well, where i=last and j=trim(i,1,0).

(This code is actually a fragment taken from a WriteIf statement so its syntax may be slightly screwed up.)

You can also use the scene change logic in MVTools2. This is actually faster, if you have the MT version of AVISynth, than is the script code above. I can't remember, but I think this code is lifted more or less intact from the MVTools2 doc:

SceneChange = MSCDetection (source_fields, backward_vec,thSCD1=BlockChangeThresh,thSCD2=Num_blocks_changed)

WriteFileIf(source_fields,filename, "(AverageLuma(SceneChange)>30)" , "current_frame+1", flush=false)


Finally, FWIW, here are the failures I have seen most often in both video and still photo "autolevel" algorithms.

1. Scenes which are naturally supposed to be dark, like sunsets, get gained up too much.

2. Bright objects, like someone walking directly in front of the video light on a camera-mounted light, cause the main video to go pitch black.

3. The autolevels algorithm attempts to always stretch the histogram to the limits, so at least one pixel is 255 and one pixel is 0. This often results in unnatural, high contrast.

4. The midtones are not stretched enough.

The last one is the toughest. Under- or over-exposed photos and video almost always compress the midtones (a necessary by-product of not using the whole exposure range). Simply moving those pixels up or down doesn't really solve the problem. Instead, the normal "hump" that represents the center range of exposure not only needs to be moved up or down, but also stretched out. I do this with custom gamma curves when I do my own corrections, but it is tricky stuff because you end up solarizing the image if you get the slope of the curves too steep.

[Ponder]

hi frustum, I like to try your autolevels, but it failed to load. I can load the 2007
version, perhap because I am using win2k. both versions placed at different folder, so
it is not conflicted.

[Caroliano]

Feature sugestion: possibility to make the estimatives from a separate clip, and apply in the original clip. Maybe even from a separate image. The original use I though for this feature was made obsolete by the "border" parameter. But still, it would be interesting to have.

EDIT: The output of version 0.6 is too saturated for my taste (see atachment). Can I control the degree of saturation correction that this new version aplies? I Haven't found any way, and as this saturation correction is dinamic, I can't just fix it aftewards...

EDIT2: Puttting an tweak(sat=0.65) before the autolevels() seems to do the trick.

[boondoggle]

@johnmeyer A while ago I tried to contact frustrum through email, but he was very busy. I really wanted him to put a feature to be able to limit the amount of stretching, which I found to be the biggest problem in most videos. I still hope he will find some time again on this very nice plugin. The settings I use now is Autolevels(border=12,filterRadius=4,sceneChgThresh=255,ignore_low=0.0002,ignore_high=0.0001). I dont know why but it seems filterradius eats lots of memory? more frames=more buffer?

[johnmeyer]

I am not doing any development on this filter but instead was just providing some "free advice." Therefore, I can't answer your question.

[boondoggle]

Ah yeah sorry, I just meant to bump(?) that I also had been on Frustrum about the very same feature requests as you. A max stretching value to stop stretching in darker videos.

[bizz & buzz]

How can I prevent Autolevels from "overexposing" bright areas? this plugin is very good but it causing me to lose bright area details.

[videoFred]

How can I prevent Autolevels from "overexposing" bright areas? this plugin is very good but it causing me to lose bright area details.

output_high :)

Fred.

[bizz & buzz]

*** I'm editing this post because the way I thought Aoutolevels() works was completely wrong. What follows is the corrected reply ***

Thanks Fred!

But Autolevels() just sets the max value according to Output_high and then expands the image accordingly. This does save some details, but also reduces overall brightness throughout the video ... So is there a way to affect the bright parts only ??

[Didée]

Well ... add border(s), do autolevels, crop border(s) again? If you add a border with luma=200, then the maximum bright-expansion will be 200-->[output_high].

Having parameters for black-stop and bright-stop would be more convenient, though. Until it's implemented, you could make a wrapper function to add those. ;)

[bizz & buzz]

Thanks a lot Didée!
Adding AddBorders did help in reducing the clipping, But unfortunately it also affected dark scenes, leaving them almost unchanged in comparison to not using AutoLevels at all. Problem solved when setting AutoLevels' Ignore_High parameter to 0.0001.
...Then I realized that Ignore_High alone was enough for the result I was aiming for :o .