Using per frame conditional filtering

[sh0dan]

A new filter has been put into the latest AviSynth core, and it features "conditional filtering" for the first time. Note that this is a completely new feature and most likely still contain bugs!

Basic usage is:

ConditionalFilter(Clip testclip, Clip source1, Clip source2, String condition1, String operator, String condition2,[bool "show"])

What is does is like this: It chooses between source1 and source2 based on how the testclip looks. The best way to illustrate is through an example:

vid = avisource("file")
vid_blur = vid.blur(1.5)
Conditionalfilter(vid, vid_blur, vid, "AverageLuma()", "lessthan","20")

This will choose frames from vid_blur when the average luma value of a frame is less than 20. Otherwise frames from vid will be returned. Adding "true" at the end will display the actual values on the screen.

The operator can be "equals", "morethan" or "lessthan". Or "=", ">" or "<" respectively.

Both conditions can be general expressions. That means you can use all internal functions, like sin(), cos(), variables, etc. Furthermore a few very basic functions has been put in (like AverageLuma). It is possible for filters to extend these functionalities, and adding more conditional functions.

For each frame, the variable "frame_number" is updated. You can even use this in your script. For instance:

ConditionalFilter(last,last.subtitle("S1"),last.subtitle("S2"),"current_frame%4","=","0")


Be aware, that when you reference a video clip or any other variable, it will be represented by the value it has at the end of the script.

The internal functions (so far) are:
AverageLuma(clip),
AverageChromaU(clip),
AverageChromaV(clip)
This will return the average pixel value of a plane. (Requires YV12, ISSE).

LumaDifference(clip,clip),
ChromaUDifference(clip,clip),
ChromaUDifference(clip,clip)
These return a float value between 0 and 255 of the absolute difference between two planes.(Requires YV12, ISSE)

When using these functions there are "implicit last" (first parameter doesn't have to be specified), so the first parameter is replaced by the testclip.

A further example:

vo=last
vi2=vo.trim(1,0)
vi=greyscale()
conditionalfilter(vo,vo,vi,"lumadifference(vi2)","<","5",true)

This will make frames on scene changes (or heavy movement) to become grey.

Suggestions for conditional filters are very welcome - even though I might not have time to implement them all.

You need the very latest binary to test this!

[IanB]

@Sh0dan : Very cool!

How about the following set:-

MinLuma() blackest pixel
MaxLuma() whitest pixel

Probably not quite so useful (signed values)
. MinChromaU()
. MaxChromaU()
. MinChromaV()
. MaxChromaV()

Maybe more useful as absolutes
. MinAbsChromaU()
. MaxAbsChromaU()
. MinAbsChromaV()
. MaxAbsChromaV()

More likely just to want the saturation (sqrt(U'^2+V'^2)) values
. MinChromaUV() or MinSaturation()
. MaxChromaUV() or MaxSaturation()
. AverageChromaUV() or AverageSaturation()

Possibly other statistical measures like Median value or upper and lower quartiles could
be good as well.

Also nothing in AviSynth says a plugin/filter has to return a Clip (they just all seem to). There are already functions like FrameRate() that return non-clip values so you probably don't need to build the world into this Filter.

IanB

[mustardman]

This is truly good news!

I did a similar post to the VirtualDub forum, as yet no replies (suspect all are asleep). The gist of my post is to correct random frame jumps ... from my VD post ...

"Say for example, I have really bad sync, or a film converted to video with random bad frame alignment... I look for the black frame band in the picture (x1,y1,x2,y2), and then shift the picture back to where it should be. Hence correcting random occurances,..."

I have very little idea on programming, so unfortunately I can't help out on that front.

But now one exists, how to actually use the filter to base the decision on a small area of the frame, x1y1x2y2, do I specify the 1st parameter as clipped video, or in the "AverageLuma" function??

vid = # original source video
shift_vid = vid.crop(0,y,width,height).addborders(0,0,0,y) # video moved up by y, bit more complex for a wrapping around function!

Conditionalfilter(vid.crop(x1,y1,x2,y2), vid, shift_vid, "AverageLuma()", "lessthan","20")

Conditionalfilter(vid, vid, shift_vid, "AverageLuma(x1,y1,x2,y2)", "lessthan","20")

Thanks for the work people!

[mustardman]

Hmmm. Went to get the latest binary as suggested by sh0dan, but the sourceforge page only contains the 2.5.1beta dated 12th March. As you posted today/yesterday, I assume this is not the binary you are referring to!

Where can I get the very latest?

Thanks.

PS: Sorry, my last post should read for the second script line...

Conditionalfilter(vid, vid, shift_vid, "AverageLuma(crop(x1,y1,x2,y2))", "lessthan","20")

[sh0dan]

@MustardMan: Follow the second link in my signature.
Actually both your scripts are correct.
If you don't specify a clip, the "testclip" is used as "implicit last".

@IanB: Yes - further single image statistical functions are in the works, and more compare functions should also be done.

[hakko504]

How bout something that can choose the frame with the least interlacing? I'm thinking of clips that are hard to IVTC then you could feed it both the IVTC2 and the Decombed version and get the best in return.

[WarpEnterprises]

1) You are right, "n" as frame number may be a little short.
But is it now

frame_number
or
current_frame

(I would prefer frame_number or maybe only frame)

2) I still don't understand why you use three parameters instead of one string evaluating to a bool. It would be much clearer to script writers.

4) what does that mean:
when you reference a video clip ... it will be represented by the value it has at the end of the script?
Putting a clip in the condition function feeds not the current value but the last in the script? That means it is strongly suggested to only use implicit last as this is how you feed the testclip?

5) I suppose the two choice clips must have the same video properties

6-9) very great - now using AviSynth gets really funny

[sh0dan]

Thread split - post usage here, and development issues to the new thread.

Usage related reply to WE:

Originally posted by WarpEnterprises
2) I still don't understand why you use three parameters instead of one string evaluating to a bool. It would be much clearer to script writers.

Adding a second version that only takes one string later is not a big deal. Right now this was the easiest way to control it.

4) what does that mean:
when you reference a video clip ... it will be represented by the value it has at the end of the script?
Putting a clip in the condition function feeds not the current value but the last in the script? That means it is strongly suggested to only use implicit last as this is how you feed the testclip?

That means, if you use variables in the condition tests, you may not redefine any of them later in the script.
The condition checks are run after the entire script has been parsed, and the graph has been created. Therefore any variables that has changed since the condition filter will contain their new values. An example:

vid=last.blur(1.5)
Conditionalfilter(last,last,last,"LumaDifference(vid)","=","0")
vid = avisource("hest.avi")

The lumaDifference will be calculated to the AVISource vid clip.
Unfortunately I cannot see any way to avoid this.

ad. 5). Yes - it is checked, just as colorspace, etc. (I may have missed a few in the exitement though).
ad. 6-9) Yes - User defined Filtering based on image properties opens up many possibilities.

[sh0dan]

Added the following conditional functions:

YDifferenceFromPrevious()
UDifferenceFromPrevious()
VDifferenceFromPrevious()

and

YDifferenceToNext()
UDifferenceToNext()
VDifferenceToNext()

These should be quite handy for detecting scene change transitions:

Example:

conditionalfilter(last,last,last.trim(1,0),"YDifferenceToNext()",">","10", show=true)

This will replace the last frame before a scenechange with the first frame after the scenechange. (Inspired by another thread).

This made me think of: Audio is always passed untouched through, from the testclip for now.

The CVS Binary is dated April 3rd.

[bilu]

@sh0dan

I've just realized this is the first frame-by-frame evaluation and conditional filtering ever done in Avisynth! Way to go, sh0dan!

Haven't tried it yet, it seems we can use any variables, right?
Now if we could have an "exists filename" condition and variable import...

I've got to show this to Ddogg

Bilu

[sh0dan]

Yes - you can use any variables - and in principle execute as many functions as you like, as long as it returns either bool, int or float.

[bilu]

And could I use Import() in one of the sources?

Bilu

[sh0dan]

Sources are static - only the two conditions are evaluated, so no.

[mustardman]

(@sh0dan).

Trying to use the new ConditionalFilter, I cannot get it to work. The conditional statement always returns an access violation error, except if I deliberately put in bad parameters (for example: try to use AverageLuma for RGB space).

Here is my script, 'TestArea' and 'Video' are returned as expected, 'FixVid' has the error reported as I have shown below.

Video = AVISource("e:\VD capture\CAM01 Comp VCRstab.avi")
Video = FlipVertical(Video) # Flip the video upside down to correct for MJPEG decoder
Video = AddBorders(Video,0,1,0,0) # Add one black line at the top of the frame
TestArea = ConvertToYV12(Crop(Video,0,0,260,200)) # AverageLuma function can only deal with YV12

FixVid = ConditionalFilter(TestArea,Subtitle(Video,"1ST CLIP"),Subtitle(Video,"2ND CLIP"),"AverageLuma()","LessThan","10")

#Return(TestArea)
#Return(Video)
Return(FixVid)

# Avisynth: caught an access violation at 0x025ea268 attempting to write 0x03471210

I know this is very new stuff, but I thought you should know.

The return size of the error video is the same size as the 'TestArea' clip, not the size of the 'Video' clip, which I found a bit strange.

Thanks

OK... some more info has come to light (ie: I am playing with it).

I tried your sample script for selecting a frame number, and it worked fine. So I dug.
It seems as though the test clip and the two source clips have to be the same size. Obviously for the two sources, but the test???? Also, it appears all 3 clips have to be in the same colour space or it will not work.

Anyway, my kludge to fix my current problem was to use StackVertical & StackHorizontal to return my small test area to full size, convert all clips to YV12, and back to RGB at the end, and it works great!!

This is going to save me weeks!!! (actually, without it, I probably would not even have tried!)

Much appreciated

[sh0dan]

Yes - the two source clips must be exactly alike - however previously the testclip also had to be the same as the two sources. This is now fixed in a new binary.
There is also proper error handling now. Sorry for the inconvinience.

[mustardman]

Thanks for the update to fix the size of the test clip. Still a bug though, the output clip size is that of the test clip, not the size of the two source clips!

Also, there appears to be a bug in the code (both this last build and the previous one I got containing the ConditionalFilter, maybe others as well) that brings up an "Unrecognised Exception" whenever I try to do any type of resize. An earlier version I have (v2.07) does resize OK.

Thanks for your efforts I am finding ConditionalFilter exceptional.

[mustardman]

This subject has been quiet for a couple of weeks, and I have got no feedback on my posts to the other threads on conditional filtering.

One was a bug that was introduced somewhere, probably not to do with conditional, but was in the CVS that had the conditional filter in it. "resize" does not work with any settings.

The other is a comment on ConditionalFilter, and perhaps someone in the know would like to address it while it is still experimental and can be changed...

I can do effectively the same thing with
1) ConditionalFilter(testclip,sourceA,sourceB,"AverageLuma()","<","128")
2) ConditionalFilter(anyclip,sourceA,sourceB,"AverageLuma(testclip)","<","128")

The only difference is that *if* "testclip" (or "anyclip") is a different size to "sourceA" ("sourceA" same size as "sourceB") the output of the filter is the size of "testclip".

Since syntax "2" is possible, why not have the general syntax for conditional filter to be...
ConditionalFilter(sourceA,sourceB,<test_expression1>,<condition>,<test_expression2>[,true|false])
...as I have noticed that AverageLuma (and other functions) are quite valid in place of <test_expression2>, as well as the first expression.

This would remove the requirement that the first clip ("testclip" above) has to be the same size as the desired output. This would mean the "last" image would always be used as "sourceA" unless specified.

Comments please!

PS: The ability to view the results of the two expressions is excellent! (using true/false)

[sh0dan]

(Merged with previous thread for reference)

"testclip" should not have anything to do with the output size. I may have an older CVS build up than I remember, but that was fixed some time ago.

When you are referencing to "size" - is it actually the length you are refering to??

For now it determined by the length of the testclip, correctly.

Anyway, how should it best be defined:

a) The longest of source1 and source2?
b) The shortest of source1 and 2?
c) The length of testclip? (as it is now)
d) source1 and source2 should be the same length?

Similar for sound - should it come from:

z) Testclip always (as it is now)
y) Source1 always.
x) Source2 always.
w) The one selected as input source (NOT a good idea!)

My personal opinion is b) z).

Testclip is delivered as "conditional last", which gives much easier syntax in the conditional filters.

[sh0dan]

A new function "ScriptClip" is introduced:

Syntax: ScriptClip([clip], "Script"[, show=true/false]).

This function further extends the possibilities of conditional filtering. A simple example:

AviSource("file.avi")
ScriptClip("Subtitle(String(YDifferenceFromPrevious()))")

This will print the difference from the previous frame onto the current one. A more advanced example:

AviSource("file.avi")
ScriptClip("Blur(YDifferenceFromPrevious()/20.0)")

This will apply blur on each frame based on the difference from the previous. This will also show how errors are reported on some frames

An advanced example:

function fmin(float f1, float f2) {
  return (f1<f2) ? f1 : f2
}

AviSource("file.avi")
ScriptClip("diff = YDifferenceToNext()"+chr(13)+"diff > 2.5 ? Blur(fmin(diff/20,1.5)) : TemporalSoften(2,7,7,3,2)")

This will apply temporalsoften to very static scenes, and apply a _variable_ blur on moving scenes. Blur is now capped properly. We also assign a variable - and this is why a line break is inserted.
Get the new binary at the second link in my sig.

"Restrictions" - the output of the script MUST be exactly like the clip deliviered to ScriptClip (Same Colorspace, Width and Height). Your returned clip is allowed to have different length - but the length from "clip" is always used. Audio from "clip" is passed through untouched.
For two very different sources (MPEG2DEC3 and AviSource) - you might run into colorspace mismatches. This is known quirk.

[WarpEnterprises]

1) ad "selection of properties": Why should the sound come from the testclip? Most of the current multi-clip-filters use Source1 as "default".

2) Is there a reference to the framenumber in ScriptClip?

3) So ScriptClip is some kind of generalisation of ConditionalFilter or what else are the pros/cons?