Dynamic (indexed) filtering

[Hotte]

Hi,
is there something in avisynth to alter filtering due to an index or indexed frame ?

Say I tied to clips together into one single AVI with an NLE. Clip A has little noise, clip B is very noisy.

I would like to tell avisynth to apply weak noise reduction to clip A but strong parameters to clip B. I could add an index frame or something at the beginning of each clip to tell avisynth which filtering it should apply. After the filtering avysynth would remove the index frames.

Any ideas how to do this ?
Thanks.

[TheFluff]

write a program that generates an avisynth script full of trim commands, or use one already written (YATTA would be one example but it's really dated now)

or use vapoursynth which has this sort of thing kinda natively

[StainlessS]

Perhaps this might help,

Avisource("D:\avs\test.avi")
ORG=Last

V1 = FFT3DFilter(Plane=0,Sigma=1.6)     # Light luma
V2 = FFT3DFilter(Plane=0,Sigma=2.0)     # Med   luma
V3 = FFT3DFilter(Plane=0,Sigma=4.0)     # High  luma
V4 = FFT3DFilter(Plane=3,Sigma=1.6)     # Light Chroma
V5 = FFT3DFilter(Plane=3,Sigma=2.0)     # Med   Chroma
V6 = FFT3DFilter(Plane=3,Sigma=4.0)     # High  Chroma
V7 = FFT3DFilter(Plane=4,Sigma=1.6)     # Light Luma+Chroma
V8 = FFT3DFilter(Plane=4,Sigma=2.0)     # Med   Luma+Chroma
V9 = FFT3DFilter(Plane=4,Sigma=4.0)     # High  Luma+Chroma
V10= FlipHorizontal()                   # Flip-H
V11= FlipVertical()                     # Flip-V
V12= Invert()                           # Invert


NickNames ="""  # Psuedonyms for clips
    L0   = 1    # Light Luma
    L1   = 2    # Med   Luma
    L2   = 3    # High  Luma
    C0   = 4    # Light Chroma
    C1   = 5    # Med   Chroma
    C2   = 6    # High  Chroma
    LC0  = 7    # Light Luma + Chroma
    LC1  = 8    # Med   Luma + Chroma
    LC2  = 9    # High  Luma + Chroma
    FH   = 10   # Flip-H
    FV   = 11   # Flip-V
    INV  = 12   # Invert
"""

SCMD="""          # Clip editing commands in string, can also use commands in file
    C0  0,99      # Light Chroma frames @ 0 -> 99
    L0  100,-200  # Light Luma frames @ 100, 200 frames ie frames 100->299
    INV 300,399   # Invert 300->399
    L0  400,499   # Light Luma frames 400->499
    FH  500,599   # Flip-H 500->599
    LC2 600,699   # High Luma + Chroma
    C1  800       # Med Chroma, Single frame
    1   900,999   # Light Luma, We used the clip number instead of a NickName
    FV  1000,1099 # Flip-V
    LC1 2000,0    # Med   Luma + Chroma, 2000 -> lastframe
"""

SHOW=True

ClipClop(ORG,V1,V2,V3,V4,V5,V6,V7,V8,V9,V10,V11,V12,scmd=SCMD,nickname=NickNames,show=SHOW)

ClipClop() here: http://forum.doom9.org/showthread.ph...light=clipclop

Replaces ranges in source clip with same range from alternate clips.
You can if you wish use clip number eg 1 to mean clip1 ie L0 (which will repesent V1).
Any ranges not replaced will remain as original clip.
Audio not affected.
Can replace as many ranges or single frames as you like (maximim that I've tried was about 40,000, parsed before first frame
served in about 1 second).
Max 255 replacement clips.

[Hotte]

Thanks stainless. I would not have been able to understand ClipClop() from the thread without this example, but your code makes it instantly clear.
The only issue is, that I would prefer not to determine the filter changes by given frame numbers but by key frames that are being inserted right before the clip area, where a new filter should be applied. I could easily do that in the NLE. I wished avisynth would then "react" to a certain keyframe or any other frame based information dynamically.
I do not know if the processing model of avisynth is capable of doing this - Just say no, if this approach is not possible.

[StainlessS]

I dont know of anything that can do as you wish in Avisynth, sorry.

[poisondeathray]

Was that just an oversimplified example ? Just two clips ? Appended together ?

Because there are obviously easier ways to that

Can you be more descriptive in what a typical use would be ? How many clips, variations, types of edits ?

Another way would be to use AVFS (avisynth virtual file system) and replace the clips with filtered avfs versions in the NLE once you've done your edits.

Or were you more interested in the mechanics and the approach as a theoretical experiment ?

[StainlessS]

And what do you mean by "keyframe or any other frame based information" or indexed frame ? and how would they denote filtering required ?

I dont use NLE's (well I did have a play a few times many years ago).

[poisondeathray]

@Hotte - If you're not familiar with AVFS, it's just an emulation method of making a "fake" AVI that allows you to use .avs scripts into NLE's and other programs. To the other program, it just "looks" like an uncompressed AVI . Nothing is rendered, you just mount the .avs. (You would have a different .avs for each version of filters, e.g heavy vs. light, etc...) . In most NLE's it's very easy to replace all instances of all videos with 2 clicks if you've organize the clips into a single folder. The problem you might have is memory issues if you deal with a lot of clips, or heavy filters

So another method you might use is use a "control layer" in the NLE to indicate to avisynth what filter to use with conditionalfilter() with one of the runtime functions like AverageChromaU . Since avisynth does this on a per frame basis, you use a solid color the same as your clip length as an overlay. The colors indicate what filters to apply over what sections. All you need to do is render out the overlay. So "red" might indicate filter this section with filter stack "A", "blue" might indicate filter this section with filter stack "B" , or something like AverageLuma() for white and dark. This isn't the same as what you asked for (inserting a single frame per section), but it works, as I've done this before

[Hotte]

poisondeathray: I would like to apply avisynth filters to a completed NLE-Project. The final video consists of numerous different clips and was exported as avi. Final step is to apply some subtle sharpening and degraining with tools only available with avisynth. In the NLE process I would like to predetermine which clip will have more or less sharpening or degraining when being post-processed by avisynth.
I think I could do that with StainlessS' approach. But then I would have to write a script or at least a textfile with all the frame-positions and filters to be applied. It seems much more practical to me if - in the NLE - I could insert a certain keyframe that would tell avisynth: Hey, the follwing frames need sharpen=3.0, and one clip later another keyframe (or information of whatever kind) which tells avisynth to apply sharpen=1.0 now, because this clip is already pretty sharp and so on.

I read that "conditional filtering" allows adaptive motion filtering so in principle it seems possible, however the available frame runtime functions seem difficult to be "misused" as keyframes (and - honestly - I do not understand all the mechanics).

Yeah it´s not a theoretical but a practical question.

[Hotte]

Just to add: I am not looking for something to add to the NLE or to use with AVFS. Forget about it. I export an AVI from NLE and load it into avisynth as a final process. And because there are some "special" frames coded into it, avisynth´s filters a being steered by these special frames. They are just there to set parameters. Could be a subtitle saying "sharpen=3.0". The problem is to evaluate such a frame.

[poisondeathray]

Originally Posted by Hotte

I read that "conditional filtering" allows adaptive motion filtering so in principle it seems possible, however the available frame runtime functions seem difficult to be "misused" as keyframes (and - honestly - I do not understand all the mechanics).

The control layer approach just chooses frames from a filtered version based on the color of the overlay. So if your filters are adaptive or temporal, then the final version will also be . It's 100% accurate , and has nothing to do with keyframes. I'm probably not describing it very well. You render out the AVI as normal. But you also render out the colored overlay. The colors tell avisynth which frames to choose from which filtered version .

[StainlessS]

You perhaps could do something like bottom quarter left frame Fine Red ($FF0000), bottom Right Fine Blue ($0000FF),
top lhs, maybe eg 6 or 8 32x32 blocks of either BLACK ($000000) or WHITE ($FFFFFF) binary encoded to mean a filter clip number.

Avisynth script could scan your clip looking for RED/BLUE "I'm here" type markers and write a ClipClop command file.
You would need to write the script for the V1, V2 etc clips as given in clipclop example but would not need NickName,
can directly use the binary coded clip number (clip 0 is reserved for original clip and so binary 0 would mean do not touch).

After that, run it though clipclop, no problem.

OR a control layer as PDR suggests, YOU have to have some way to encode command data, what can you suggest.

EDIT:

Could be a subtitle saying "sharpen=3.0"

,
Nah, OCR and then some kind AI to try to figure out what you want and then to write a good script, no chance.
YOU would have to decide what each numbered filter would do, and write the script as in the V1,V2 examples given in clipclop.

If you had only a simple filter in mind with only a couple of variables eg FFT3DFilter(Plane=4,Sigma=VAR_1,Sharpen=VAR_2)
then if you can find a way to encode to two variables into a control clip then those could be used to vary the VAR_1 VAR_2 variables,
either continuously (every frame) or if you can also encode some kind of "I'm HERE" marker, then effective from that frame onwards.

[Hotte]

Stainless:
OK, I understand the principle. Could you tell me how avisynth scans a clip e.g. for $FF0000 color blocks ? Just a few codelines perhaps ? Sorry, I am pretty new to avisynth and you guys are all so enormously experienced (Its my fault to grabbing for the stars with so limited knowledge....).

poisondeathray:
I understand! In NLE I would overlay my cut video with different solid colors over time (the control layer) where I would like a certain filter stack to be applied. Each color represents a certain filter stack. I would render out the original (avi1) plus the control layer as avi2 and feed both avi's into a conditionalfilter(). Sounds good!
Could you please, please help me with some code statements (say with simple blur(1.0) / blur(1.58) due to a different colorcode from avi2)
Thanks!

[poisondeathray]

Originally Posted by Hotte

poisondeathray:
I understand! In NLE I would overlay my cut video with different solid colors over time (the control layer) where I would like a certain filter stack to be applied. Each color represents a certain filter stack. I would render out the original (avi1) plus the control layer as avi2 and feed both avi's into a conditionalfilter(). Sounds good!
Could you please, please help me with some code statements (say with simple blur(1.0) / blur(1.58) due to a different colorcode from avi2)
Thanks!

Yes! that's it. When you edit it in the NLE, it helps to change the opacity of the control layer to 50% or something to "see" what is going on when you do the actual editing. Change it back when you export

The problem with what you initially wanted, is a single control frame doesn't indicate to avisynth anything about how many frames to affect (mark in, but no mark out point) . It also introduces other problems such as having to deleting it.



Sure, the 2 case example is straight forward. "Blur" , might not be easy to distinguish, so in this example, I choose levels to make a A) dark FilterA B) blown out FilterB. I chose the control layer to be black for FilterA, and red for FilterB. You can easily change the filters or control layer variables, or use different expressions like greater than, less than etc.....

For this example, the original clip is colorbars ("orig" in the script), frames 0-299 . Pretend this is the AVI that you exported

My control layer goes from 0-150 black, 151-299 red in this example . Pretend this is the overlay layer that you exported . Control layer doesn't have to be the same dimensions as main video. It can be 16x16 for example. That will export very fast from NLE

It's easy to debug with "show=true" to see what values are being returned . Comment it out of course when you use it for final encode. You can choose different runtime functions to measure (I used AverageChromaU in this example, for Red)
http://avisynth.nl/index.php/Conditi...time_Functions

You can use conditionalselect() for more cases as well, or combine conditional filter calls
http://avisynth.nl/index.php/ConditionalFilter

colorbars(width=640,height=480, pixel_type="yv12").showframenumber().trim(0,299)
orig=last

orig
levels(0,0.1,255,0,255,false)
subtitle("FilterA")
FilterA=last

orig
levels(0,3,255,0,255,false)
subtitle("FilterB")
FilterB=last

red=blankclip(orig, color=color_red)
black=blankclip(orig)

controllayer=black.trim(0,150)+red.trim(151,299)

ConditionalFilter(controllayer, filtera, filterb, AverageChromaU()", ">", "120", show=true)

[StainlessS]

SEE edit above

It would probably be easiest to have a separate control clip rather than a Fine Red/Blue I'm Here type clip (easier for you mostly, not really a problem this end).
You could have continuously varying VARS (as mentioned above) or FILTER_FROM_HERE type frames. For FILTER_FROM_HERE mode, non switch on
frames could be BLACK, and ignored (use previously set filter vars). (Or maybe just use the bottom half of frame to flag FILTER_FROM_HERE leaving top
half of frame to encode data).

If not ignored, then would need to extract some kind of data from eg Binary BLOCK coding in particular positions, OR
eg area 0,0,8,8 (x,y,w,h) might hold info as Average Luma data range 16->235 representing eg 0.0 to 1.0 for a particular variable for eg Sharpen.
eg area 8,0,8,8 (x,y,w,h) might hold info as Average Luma data range 16->235 representing eg 5.0 to 8.0, for some other function.
eg area 16,0,8,8, etc.
OR maybe if only eg 3 variables in total, simply encode 1 in each of the RGB channels, 0,0,0 being ignored and 1->255 being scaled to whatever
was required for particular output variable.

RT_Stats has a number of runtime functions to scan a frames, down to individual pixels for luma or RGB and can easily extract binary block coded
data or encoded as eg luma or RGB channel level. It really only requires that you can successfully encode something, we can extract it (but not subtitles, hehe).
I guess you could even encode ASCII text into Luma or RGB channels and RT_Stats could pick it out of that channel and convert it from a number eg
65 to ASCII 'A'.

It really all depends on what you can persuade your NLE to let you encode on your control clip key frames.
You could even if you like have eg BLACK frame = IGNORED, ORANGE (giving us exactly what you consider to be orange) meaning eg Degrain = 1.1, Sharpen 1.2.
RED = Degrain = 1.2, Sharpen 1.1
etc.

You need to figure out what you can encode, then we'll set about extracting that.

[poisondeathray]

Here is an example of a 3 state situation, using ConditionalSelect() . It's the same above, but the control layer is black , grey, and white , and goes from 0-100,101-200,201-300 in this simple example

I chose black = filterA , grey= original, white= filterB

colorbars(width=640,height=480, pixel_type="yv12").showframenumber().trim(0,299)
orig=last

orig
levels(0,0.1,255,0,255,false)
subtitle("FilterA")
FilterA=last

orig
levels(0,3,255,0,255,false)
subtitle("FilterB")
FilterB=last

black=blankclip(orig) #16
grey=blankclip(orig, color=$A9A9A9) #161
white=blankclip(orig, color=$FFFFFF) #235

controllayer=black.trim(0,100)+grey.trim(101,199)+white.trim(200,299)

ConditionalSelect(controllayer, "luma_av = AverageLuma()"+chr(13)+"luma_av < 170 ? (luma_av < 17 ? 2 : 1) : 0", FilterB, orig, FilterA, show=true)

How did I know to use those specific numbers <17 and <170? I used them in ConditionalFilter and used show=true. The values are 16 for black, 161 for the shade of grey I used (color=$A9A9A9), and 235 for white

[StainlessS]

PDR, can I suggest this as easier, especially when a few more Filters are added

colorbars(width=640,height=480, pixel_type="yv12").showframenumber().trim(0,299)
orig=last

orig
levels(0,0.1,255,0,255,false)
subtitle("FilterA")
FilterA=last

orig
levels(0,3,255,0,255,false)
subtitle("FilterB")
FilterB=last

black=blankclip(orig) #16
grey=blankclip(orig, color=$A9A9A9) #161
white=blankclip(orig, color=$FFFFFF) #235

controllayer=black.trim(0,100)+grey.trim(101,199)+white.trim(200,299)

Cond_S = """
    luma_av = AverageLuma()
    Result = luma_av < 170 ? (luma_av < 17 ? 2 : 1) : 0
    Return Result
"""

ConditionalSelect(controllayer,Cond_S, FilterB, orig, FilterA, show=true)

Only extracted the string expression to Cond_S and added the Result var

EDIT: From RT_Stats

RT_RgbChanAve(clip c,int "n"=current_frame,int "delta"=0,int "x"=0,int "y"=0,int "w"=0,int "h"=0,bool "interlaced"=false,
    int "chan"=0,int "mask"=NOT_USED, int "MaskMin"=128,"MaskMax"=255)
  Return int -1, if no valid pixels in Mask clip.
  Returns FLOAT average channel value (0.0 -> 255.0) in frame(n+delta) for area x,y,w,h, and selected Chan channel (0=R,1=G,2=B, RGB32 3=ALPHA).

Can just use eg

  Red = RT_RgbChanAve(0)   #   Red = RT_RgbChanAve(0,w=1,h=1) for single pixel sample at 0,0
  Grn = RT_RgbChanAve(1)
  Blu = RT_RgbChanAve(2)

if you want to use RGB color, can also perhaps use the RGB colors from avisynth Colors_Rgb_avsi, here a small sample

global color_aliceblue              = $F0F8FF
global color_antiquewhite           = $FAEBD7
global color_aqua                   = $00FFFF
global color_aquamarine             = $7FFFD4
global color_azure                  = $F0FFFF
global color_beige                  = $F5F5DC
global color_bisque                 = $FFE4C4

EDIT: If using the avsi, then this function might be easier

RT_RGB32AsInt(clip,int "n"=current_frame,int "delta"=0,int "x"=0,int "y"=0)
 Compile/Runtime clip function.
 Given an RGB32 clip that had a pixel value created from an Int, gets that pixel and returns as the
 original Int. By creating single pixel clips and stacking them horizontally/vertically,
 you can use a frame as a two dimensional array of Int, perhaps use the frame number for a third dimension.

Returns colors as RGB Int, will get from pixel(0,0) by default but be much quicker than scanning entire frame.

[poisondeathray]

Nice StainlessS, I'll keep it in mind. I've only used the 2 filter case before with my NLE, because I prefer to do it these sorts of things with remapframes() method in avisynth.

[Hotte]

Thank you guys - so much input! It will take me some time to go through your suggestions and come back with questions.

Short first question concerning performance: Are Filter A and B both calculated for the whole video or only for those parts of the video where they are finally applied to ? I use quite performance hungry filters. Selecting out of 3 or 4 variants might slow down heavily if everything is being calcualted for the whole clip ?

[StainlessS]

Only for those parts of the video where they are finally applied to.
Any filters not used at all (as in the ClipClop example) will only have the overhead of calling any filter constructors, ie before the first frame is served.