[johnmeyer]

Originally Posted by frustum

Warning: rambling thoughts ahead.

Acthung: Even more rambling thoughts ahead ...

Originally Posted by frustum

The problem is that if the histogram changes suddenly, say due to the scene panning and something glaring in the sun pops into frame, this bright area will be visible in N+1 frames or fewer but it is averaged with 2N+1 frames by the time that bright spot becomes the center of the averaging window.

To me, the heart of a better autolevels algorithm is dealing with these anomalies. Obviously, you first need to be able to correct a single frame and, to do this, you need to decide what constitutes correct levels. The original autolevels code did this and I think you have taken that part of the code forward. As I understand it, you basically apply a gamma function so the midtones are corrected more than the darkest and lightest part of the picture. To me, this is the toughest part of the whole thing and would be the place I would rely on other people's research. There is more art than science in figuring this out and doing it well.

However, I don't think I would look to averaging to provide a better fix to an individual frame's exposure. Averaging should instead, I think, be viewed ONLY as a way to avoid "hunting" and "pumping" problems, and not as a way to get a better exposure. After all, the sun may not just "pop" into the frame, but might be part of the scene for many minutes. If you are relying on averaging to make a proper correction, then in this example you'll never get anything different with which to average.

Originally Posted by frustum

One takes a very sensible approach of figuring out minluma and maxluma for each scene, then the correction for each frame is based on a combination of the minluma & maxluma of the current frame and the current scene.

It is definitely true that some footage does exhibit the "shifted histogram" where the darkest pixels are way too bright or the brightest pixels too dark. Even when this occurs, you still need to truncate some outliers because there always seems to be a 0 or 255 out there somewhere, even when the exposure is clearly wrong. So simply using maxluma and minluma I think will lead to bad results, but using a slightly modified version, where you first throw out some outliers, might work. Concentrating on these values rather than using some sort of average would certainly stop the algorithm from trying to correct a sunset where the average luma is low, and there is almost nothing in the center of the histogram.

Once you correct the frame, I think I would use some sort of averageluma function to determine how to average the exposure. I actually think that you want to keep the number of frames averaged quite small, however. I can see no good coming from averaging more than about one or two seconds of video. Imagine standing in a forest at the edge of a lake. You start pointing the camera to the left, into the forest. The video is very dark, but the camera auto exposure turns up the gain so it isn't too dark. It still needs to be made lighter. The camera then pans across the lake, where the sun is hitting the water. The camera's autoexposure tracks the change, although probably with a lag. When the camera pan leaves the lake and points to the forest on your right, everything gets dark again. You don't want to dampen the corrections too much (by averaging LOTS of frames), or you'll end up not correcting anything.

I think you need to actually write down and clearly answer the question: "why am I averaging frames?" Make sure the objectives are clear, and that averaging is the best answer to the problem you are trying to solve.

So, I think if I were doing this, I'd spend most of my time trying to come up with a fantastic single frame exposure (levels) correction, and then at the end apply some "safety valves" that keep the correction from failing when subjected to specular reflections, scene changes, bright camera strobe flashes, and other short duration transient phenomenon.

Originally Posted by frustum

Another failure mode of autolevels() which I had never thought about is the problem of fades and blended transitions. I was unaware of this because all of my work has been on 8mm home movies, and every cut is a jump cut.

I don't see why you have to consider this. I do film transfers all the time, but also edit lots of video. Fades and blended transitions don't happen in unedited video (I guess you can do them "in camera," but no one ever uses those features). Once footage has been edited, it then contains these transitions, but that is true of edited 8mm, Super 8, 16mm, 35mm, & other film formats as well. I would not bother to try to deal with this.

Originally Posted by frustum

I had also long thought of some ways to improve autolevels' scene boundary detection logic using just the already-gathered histogram data, but before charging off to try it, ...

I've written several scene detection scripts and, while they all fail sometimes, they generally work well enough to be useful. A simple

(YDifferenceToNext(j)>blankthreshold)&&YDifferenceToNext(i)<blankthreshold

works pretty darned well, where i=last and j=trim(i,1,0).

(This code is actually a fragment taken from a WriteIf statement so its syntax may be slightly screwed up.)

You can also use the scene change logic in MVTools2. This is actually faster, if you have the MT version of AVISynth, than is the script code above. I can't remember, but I think this code is lifted more or less intact from the MVTools2 doc:

SceneChange = MSCDetection (source_fields, backward_vec,thSCD1=BlockChangeThresh,thSCD2=Num_blocks_changed) 

WriteFileIf(source_fields,filename, "(AverageLuma(SceneChange)>30)" , "current_frame+1", flush=false)

Finally, FWIW, here are the failures I have seen most often in both video and still photo "autolevel" algorithms.

1. Scenes which are naturally supposed to be dark, like sunsets, get gained up too much.

2. Bright objects, like someone walking directly in front of the video light on a camera-mounted light, cause the main video to go pitch black.

3. The autolevels algorithm attempts to always stretch the histogram to the limits, so at least one pixel is 255 and one pixel is 0. This often results in unnatural, high contrast.

4. The midtones are not stretched enough.

The last one is the toughest. Under- or over-exposed photos and video almost always compress the midtones (a necessary by-product of not using the whole exposure range). Simply moving those pixels up or down doesn't really solve the problem. Instead, the normal "hump" that represents the center range of exposure not only needs to be moved up or down, but also stretched out. I do this with custom gamma curves when I do my own corrections, but it is tricky stuff because you end up solarizing the image if you get the slope of the curves too steep.