Apply filters with local gradient

[nji]

I recently ran into the task of restricting filters
(brightness, contrast, saturation, hue)
by a local mask.
With a kind support this was solved by an AviSynth script.
https://forum.doom9.org/showthread.php?p=1893743#post1893743
And as I normally do in VirtualDub2 I found a workaround there too.

No I would like to do the same filters in a "local gradient way"
(Portions of the area need some more contrast, others much more.
Saturation etc. dito).

It looks like no chance to fix that with VD2.

Being an absolute newbie to AviSynth I searched for infos.
I found something like:
http://avisynth.nl/index.php/Filter_SDK/GradientMask
http://avisynth.nl/index.php/GraMaMa
But have no idea if this is the right spot...

Any help appreciated.

[poisondeathray]

It depends on the complexity of the gradient.

Is it linear and simple, or complex pattern ? Are the edges sharp or feathered (gradual transition) in 1 or more directions ? Does gradient area change over time ?

If it's simple, it might be possible to do in avisynth directly

If it's complex you will probably want to do at least the gradient mask part in other programs , possibly other operations - such GUI's that can keyframe (e.g. resolve, blender, natron, hitfilm). The filter parameters can also be keyframed directly also, so >2 levels of control

To apply the filters through the gradient in avisynth, you can use overlay() or mt_merge()

[nji]

It's the most simple case:

No change over the time.
The gradient is given by a b/w mask, which determines amount of application.
Application should be one of: brightness, contrast, hue, saturation, ... perhaps sharpen.

[manono]

The gradient is given by a b/w mask, which determines amount of application.

A=Last
B=A.Filter Here
B=B.Another Filter If Needed
Mask=ImageSource("Mask.bmp")
Overlay(A,B,0,0,Mask)

[StainlessS]

Posts in a VDub2 thread.

First thought: this is implementation nonsence: filters are very expensive to manipulate parameters per-pixel in generic way.
You won't be happy to run this.

(... well, I won't argue with the boss. :)

But still...

Although I'm in SWE I at first place I don't look at implementation
but on what is needed, helpful, useful.
And it's just the movie why I started this thread
where a gradient of parameters is needed.
As - with the support of the kind members above -
the local restriction of filters is solved...
in my example it turned out, that the original film material
is "bleeched" by contrast, brightness and saturation
in a local gradient too!
I fiddled around with the filter stream/ pipeline "trick" (see above)
with all kind of masks and blending modes ... to no success.
But ... doing an older movie restoration ... this is will
be a very common needed task.
So how to do that?
(Probably will do a separate task, in VD, and if unsuccessful
in AviSynth subforum).

Second:
It may be "very expensive" ...
But your fine master blend does exactly that, ... in time gradient?
I personal don't care if something takes long to process.
As long it is done at all.

Please don't get me wrong.
I'm not argueing, but just put my thinking, to learn...

???

Greetings!

Normally filters are optimized to process massive amount of pixels at once.
Varying parameters each frame means the filter is processing w*h amount of pixels.
Varying each pixel ... means the filter is processing 1 pixel and then a shitload of work is required to change setup.

Something intermediate that can be done: run the filter N times to produce N resulting frames and then interpolate them.
However, few examples:

contrast. Filtering with average contrast is quite similar to average blending with full contrast. Would you even notice?
rotate. In this case increasing N is hopeless, it really ought to be per-pixel. Btw the result could be funny but I hardly imagine practical use :)

+++++ more.

[in say RGB] To alter eg Saturation of a single source pixel using say lumaY value in a single plane gradient GradY to control output,
would involve for each R,G,B channel [something like],

OutR[x,y] = R_Magic(InR[x,y],InG[x,y],InB[x,y],GradY[x,y])
OutG[x,y] = G_Magic(InR[x,y],InG[x,y],InB[x,y],GradY[x,y])
OutB[x,y] = B_Magic(InR[x,y],InG[x,y],InB[x,y],GradY[x,y])

Something like above for each and every pixel, and where X_Magic involves multiple terms in the calculations,
it will not be simple[and very very slow], as already suggested by Shekh.
[EDIT: For eg hue, would also need to alter behaviour depending upon which segment of the color wheel the input colour lives in]

Dont know if avs+ Expr() filter would be able to do as above [maybe using input channels extracted to eg Y8], and then results re-combined to RGB.
Also, maybe one of Wonkey_Monkey's weird rpn type filters could do it [I aint got a clue how to use either Expr or them there wonkey things].

Good luck to anyone that attempts something like above.

EDIT: And to do same for sharpening [as suggested somewhere], would also have to sample pixels surrounding [x,y] of each channel.

Doubt if anyone is likely to volunteer for any such simple project.

EDIT: Here the quote that mentions Sharpen:
It's the most simple case:

No change over the time.
The gradient is given by a b/w mask, which determines amount of application.
Application should be one of: brightness, contrast, hue, saturation, ... perhaps sharpen.

[nji]

A=Last
B=A.Filter Here
B=B.Another Filter If Needed
Mask=ImageSource("Mask.bmp")
Overlay(A,B,0,0,Mask)

Intuitive I "felt" this is not correct.

Thinking about it ... I would like to declare,
that overlaying as above is only correct
if the filter is a linear function.
For all other filters the overlaying does something "similar".

I really hope there is someone to confirm
or to contradict that.

And if it should be right:

Which of the mentioned filters are linear?

And - most important:
is there really no way to do it even with AviSynth?
Gradient application of filters should be very often needed.
Or is the approximation by overlaying good enough?

Help!

[poisondeathray]

Intuitive I "felt" this is not correct.

Thinking about it ... I would like to declare,
that overlaying as above is only correct
if the filter is a linear function.
For all other filters the overlaying does something "similar".

I really hope there is someone to confirm
or to contradict that.

And if it should be right:

Which of the mentioned filters are linear?

And - most important:
is there really no way to do it even with AviSynth?
Gradient application of filters should be very often needed.
Or is the approximation by overlaying good enough?

Help!




The mask can be a non linear gradient , so the application of filter(s) can be distributed along non linear gradient

Or are you asking something else ?

[nji]

Yes, actually I am "asking something else" :)

Are my thoughts above correct, or wrong?
If right:
Which of the filters actually are linear?
(R, G, B seems yes.
H, S, L seems not - if in RGB-format, but if you turn to HSL-format then yes, ???.
Contrast, Sharpen are not linear, ...)

Non-linear gradient of the mask does only equal the point-wise-application of a filter,
if the gradient is "like" the filter ...
This means: Not correct to approximate a non-linear filter like sharpen.

So: How is that common task done ... everywhere?

(Nothing but questions ...)

[StainlessS]

All overlay does is BLEND TWO frames together with [EDIT: per pixe] control via mask and [EDIT: Frame global control] via Opacity arg [there are a few other ops in addition to blend, eg lighten/darken, but still a sort of blend],
taking both mask pixel color and opacity into account, will result in eg 25% of pixel from Frame_1 and 100-25% of same pixel from Frame_2 [ie blended].

[manono]

I, for one, have no idea what he's asking. He's the one that said:


The gradient is given by a b/w mask, which determines amount of application.

So, if the mask is properly feathered (faded out), then it'll do the job of blending the filtered portion of the frame back into the unfiltered portion. And, as StainlessS says, the Opacity setting of Overlay can also help. Has he even tried yet to see if the script does what he wants?

[StainlessS]

He wants to eg change Hue of a pixel, with control on change via mask pixel value. [EDIT: I think]

[To tell the truth, I aint sure if its a crazy idea or genius, I suspect crazy]

EDIT: Same for other mentioned ops, like Sharpen, with control via mask pixel value.
I guess that simple Overlay blend with Pre-Sharpened frame should work OK [to some degree].

Presumably would need offset for Hue adjust, for +ve/-ve adjustment.

EDIT: He basically wants a Per Pixel filter, where the filter operation for the individual pixel is controlled via a mask pixel.

EDIT: Above based on this post from VDub forum:
... in addition to that the idea for that meta filter:

Take the b/w mask not only for switching on/off locally the manipulated filter
but ... well ... take the mask's values as degree of the set parameters.
I.e. degrees of the set parameters, not degree of the blending.
By this gradients could nicely be done.
Thinking about it ... yes, it's like master blend filter of the parameters,
but not in time but in space.
And - like there - would work only for some filters.
Cool?
Or complete nonsense?

[poisondeathray]

Yes, actually I am "asking something else" :)

Are my thoughts above correct, or wrong?
If right:
Which of the filters actually are linear?
(R, G, B seems yes.
H, S, L seems not - if in RGB-format, but if you turn to HSL-format then yes, ???.
Contrast, Sharpen are not linear, ...)

Non-linear gradient of the mask does only equal the point-wise-application of a filter,
if the gradient is "like" the filter ...
This means: Not correct to approximate a non-linear filter like sharpen.

So: How is that common task done ... everywhere?

(Nothing but questions ...)




Probably not what you're asking, but even if they appear to operate linearly, technically, NONE of the filters are actually linear when you just use them - because there is a gamma function "baked" in to everything with most common video formats . 1+1 does not equal 2.

In order for most video formats to be linear, you have to "degamma" or linearize it. Basically reverse the transfer function. Then all operations can be linear when operating in a linear space. 1+1 =2

(There are some exceptions, such as resampleHQ which resizes in linear light)



Or are you asking something else ?

[poisondeathray]

"Point-wise application of the filter" - If you meant "spatially" linear...

"sharpen" is slightly different because more than 1 pixel is changed. The "perception" of "sharpness" boils down to local contrast enhancement.

If I have 1 black pixel. You can't "sharpen" that. You need adjacent pixels for contrast.

But things like H,S,V can change per pixel if in a full color format (or "interpolated" to a full color format -because most consumer video formats will be 4:2:0 - the color information will actually be 1/2 width, 1/2 height). So you don't have a 1:1 pixel mapping for the color information and the luminance. If you convert it to (a new, interpolated) 4:4:4 . Then you do have (a new) 1:1 pixel mapping between Y,Cb,Cr planes . Then hue of individual pixels can be changed, and individual pixels controlled by the mask . Same with R,G,B. RGB is full color. Each R,G,B plane is the same dimension (no subsampling)

Or were you asking something else ?

[StainlessS]

PDR, :goodpost:
Some points that did not at all occur to me.

Then hue of individual pixels can be changed, and individual pixels controlled by the mask
Confirm, that is a theoretical observation, and not something that already exists for single pixel fixes ?

[poisondeathray]

Confirm, that is a theoretical observation, and not something that already exists for single pixel fixes ?

Bad wording

To clarify - if you change hue, say with tweak() or similar, yes you can control application of that layer per pixel by using a mask

But the effect itself applied to the filter layer does not control individual pixels directly because of subsampling .





But I have no idea if this is what he is asking...

[nji]

Thank you, all of you, I really appreciate the support given to me.
Not a hint of the rudeness that unfortunately is common in most
other forums.

And please, don't begin to insinuate bad things from my side
like "crazy idea" or "probably hasn't even tried"
as reaction when it becomes difficult to understand.

I can well asure to you that I only post a question after having
tried and thought by myself, and an unsuccessful search in the net.
(And, of course, I tried the "blending solution").

As I have the strong impression that you're really interested
I like to explain the thinking again:

As told, I recently stumbled upon the problem of
having to apply a filter not as usual uniforum but in a "patchy" way.

As I'm coming from the "math side" my thinking was that I
need a mechanism to ... well ... do a point-wise changing
of the parameters of the specific filter (say sharpen).

It didn't came to me to gradient blend a full sharpened layer,
as my "implicite intuition" hindered my on thinking that way.
(When I have to close a hole in a white wall, I don't look
around to see what "does the job". "Oh, the toothpaste!" :) )

As I was pointed often to the blending thing by you,
I tried it, and the result "looks" satisfying.
(Like the toothpaste BTW :))
So the problem is solved?

Question only academic?
No, I don't think so.
If, for example, you by some means know that the frame needs
pointwise-parameter-sharpen-parameter-changing,
as the disturbance process have been in that way...
wouldn't you want to revert it correctly?
And not only in way "it looks OK"?

2nd:
I've to admit that I'm interested in which cases the
blending thing does the job exactly.
As I said, it depends on the filter.
I suspect the less linear the filter is,
the larger the difference from correctness will be.
But I do not know.
And I'm a little lost about that.
And of course I cannot test it, as I don't know a method
to generate the pixel-wise parameter change
and compare it with the blending.

If you take away the time consuming aspect it should be
quite easy to implement:
Call the specific filter for every single pixel with
the parameter value that is given by the mask value.

And of course the problem is not in time.
So I had look at image manipulation progs and libs
without success.

[poisondeathray]

As I'm coming from the "math side" my thinking was that I
need a mechanism to ... well ... do a point-wise changing
of the parameters of the specific filter (say sharpen).


"Point wise changing ?" Or did you mean procedurally adjusting the filter values ? . e.g. say saturation 1.02 vs. 1.03 . Or sharpen 1.2 vs. 1.3. Or maybe 1.25 would be better - and preview it to "see" it ? ie. How do you know what values to use or what is the better "answer" or settings to use?

If this was the question, you can use sliders to control settings in avspmod like a GUI. You can also preview in avspmod with tabs and flip with number keys quickly so each swapped and superimposed (e.g. compare different versions, different settings how they look)

You can view different output "nodes" . It's similar to the "chains" or branches you were talking about in vdub. You're viewing different chains or different points in the chain. Say you wanted to view layer A separately with just the filter only, maybe put that in tab2 . Maybe tab3 has the filter settings stronger. Maybe tab4 has the overlay at opacity 100%. Maybe tab5 has the overlay with a feathered mask instead of a sharp binary mask to make it blend better. Tab1 has the "original" so you can see it quickly to compare



I tried it, and the result "looks" satisfying.


"satisfying", but not "great" ?

Why wasn't it better than satisfying ? Possibly it means you have to adjust the mask, and or filter, or add other filters. Wild guesses - There can be many other reasons the results were not ideal, and possibly many ways to improve on the current results.

You're looking a the 1 filter, 1 mask scenario - You can also combine multiple masks with blend modes and multiple filters



Question only academic?
No, I don't think so.
If, for example, you by some means know that the frame needs
pointwise-parameter-sharpen-parameter-changing,
as the disturbance process have been in that way...
wouldn't you want to revert it correctly?
And not only in way "it looks OK"?


Not at all. Make it better than OK.

Compositing is used every day , frequently, many different programs. eg . photographers use similar techniques all the time in photoshop, the underlying concepts are the same here, maybe slightly different for video and colorspace

Say you want to sharpen the clouds in the background, but nothing else. Or denoise shadows below a car. Or change the lipstick color of some beautiful woman. It's all done through masks



I've to admit that I'm interested in which cases the
blending thing does the job exactly.
As I said, it depends on the filter.
I suspect the less linear the filter is,
the larger the difference from correctness will be.
But I do not know.



It depends on what the source problem is, exactly. And how do you know what is "correct" exactly ? Your old video is probably lossy compressed and "not exact" in the first place

But if you "know" exactly what the problem is - you can be as complete and tedious as setting individual pixel values. e.g. pixel in x=34, y=45 has a YCbCr value of 134,65,74. That's 100% exact.

The "commonly" encountered scenarios that you won't be able to "fix" very easily in these scenarios is if there is channel clipping somewhere. e.g lets say part of the sky is blown out. Or some shadow details are crushed. Those are not as easily "fixable" by these methods such as a simple filter application +/- mask. You have to use other advanced methods




I have no idea if this is on the right track...it's just the wording here is very confusing... at least to me

[nji]

@PDR:
I'm sorry but I can't better explain than in my last post.
Sorry.
Maybe it's because I'm no native speaker.
(Well - actually I am. But not for English :))

A short addendum to my last long post:

As said in a post above using a non-linear mask
would surely make it mathematical correct.
But how do you know the exact mask-values?
The more non-linear the filter operates on a specific pixel
the more "non-gradient" will the mask point be.

2nd HSL:
I'm pretty sure R,G and B changing is linear.
So the blending workaround will do perfect.
But what about H,S and L?
If you have look of the color of a pixel as its "state",
then the filter changes it from stateA to stateB.
Having the state in RGB representation, changing the
say lightness does not look to be linear. Or H or S.
But if you do another representation (HSL)
it will be the same states of course.
But applying the change of H, S or L seems to be linear now.
BOING!!
"The situation in general is worrying"
(The inspector in "Die neun Leben des Tomas Katz")

[poisondeathray]

As said in a post above using a non-linear mask
would surely make it mathematical correct.
But how do you know the exact mask-values?
The more non-linear the filter operates on a specific pixel
the more "non-gradient" will the mask point be.


I think I understand what you're saying roughly, in the mathematical context

But in practice, with an old movie or video source, you're not going to have a exact gradient anyways

And if clipping ever occurs , anywhere, all bets are off



2nd HSL:
I'm pretty sure R,G and B changing is linear.
So the blending workaround will do perfect.
But what about H,S and L?
If you have look of the color of a pixel as its "state",
then the filter changes it from stateA to stateB.
Having the state in RGB representation, changing the
say lightness does look to be linear. Or H or S.
But if you do another representation (HSL)
it will be the same states of course.
But applying the change of H, S or L seems to be linear now.



Changing can be linear, but the actual end results will technically not be linear because of gamma error

[nji]

...
If you take away the time consuming aspect it should be
quite easy to implement:
Call the specific filter for every single pixel with
the parameter value that is given by the mask value.

Subject to my thinking about "linear filter" is correct
(sadly nobody confirmed or convincingly contradicts)
the mathematical correct implementation could be "sped up"
by not calling the "inner filter" for every pixel of the manipulated frame
but for ever value of the gradient mask:

Call the "inner filter" for each value v that arises from the gradient mask.
In the resulting frame set only the pixels (maybe by masking)
that have the value v in the gradient mask.
Cool, don't you agree?
OK, still 256 times for every frame.
But a solution at last (that doesn't exist anywhere).
And allows comparing to the "blend workaround".

---
"We don't need to worry about the crazy ones,
the normal ones are the problem!"

[nji]

... not to be misunderstood:

I'm not searching for a stupid one
who does the work for me.

I simply don't have the abilities.
(Short time ago I just learned how to even call AviSynth!)

Implementing the sketched algorithm would be helpful
as we would get a hint (by small test clips)
which filters will be approximated good/ bad with the "blend trick".
(As long as the mathematical prove is open).

OK, as far from me.

[poisondeathray]

Filter might be appear to be linear, but end result will visually be subject to gamma error and technically not be "linear"

https://en.wikipedia.org/wiki/Gamma_correction
https://www.giassa.net/?page_id=475
http://xahlee.info/img/what_is_gamma_correction.html
http://www.ericbrasseur.org/gamma.html

e.g. Some filter or manipulation applied and operating at the shadow levels will not result in the same magnitude of change when applied to pixels in the midtones because of that "baked" in gamma curve.

All higher end compositing is done in linear light. It's more accurate, and the math is correct because the slope is 1, not some power curve. But you usually don't have to resort to high end linear workflows to fix home video...





Implementing the sketched algorithm would be helpful
as we would get a hint (by small test clips)
which filters will be approximated good/ bad with the "blend trick".
(As long as the mathematical prove is open).


I really don't think it would be helpful. Are you more interested in the math or getting good results?

It does not model what is done in real life. How do you think people do it for real, professional photo or film repairs and compositing?

You're thinking too simplistically with a a single binary mask and single filter. It doesn't work like that for real repairs. Unless you have something very simple, then you would have had a perfect repair already.

"All" of them "work" with the "blend trick", because you add other filters in different combinations and blend modes with other layered masks.

Often you have to make use of other compositing tips/tricks to blend in the repair to make it seamless or close to perfect

Other tools I mentioned in your other thread are used as well. Look at some of the dedicated tools , far more control for color work, better filters, better monitoring and scopes. You can do the masking , feathering right in the program. You can view results in realtime. Make adjustment and see what it does. This is how it's done in real life





Call the "inner filter" for each value v that arises from the gradient mask.
In the resulting frame set only the pixels (maybe by masking)
that have the value v in the gradient mask.
Cool, don't you agree?
OK, still 256 times for every frame.
But a solution at last (that doesn't exist anywhere).
And allows comparing to the "blend workaround".




Another common "gotcha" here is YUV vs. RGB masks. If the mask and operations are in YUV it has to be Y 0-255. If you did the mask in some image program that RGB=>YUV "normal" will "map" it to Y 16-235. So you will not get expected results

But what if the filter parameter does not have 256 values ? Some filters have different scales, like 0-1.

Maybe a way you can visualize this is a 256 frame mask animation, varying the mask value. Frame 0, Y=0 (or if in RGB , [0,0,0]). Frame 1 Y=1, etc..

Then you can preview it in vdub or avspmod by advancing the frame . So if you apply filter "f" with setting "s" , on frame 0, 0% mask will be applied. At frame 127, it will be 50% blend. At frame 255, it will be 100% filter applied . Would something like that help you on your detour ?

[nji]

The proof of the pudding is in the eating.

[poisondeathray]

I think I understand what you're asking:

You want the pixel values from one layer (e.g. the gradient mask layer in this example , but it could be anything as a "control" layer) to directly "drive" the input filter parameter value on a per pixel basis; and "remap" the 0-255 range to the filter parameter value if required

e.g. if the control layer pixel at position x=10, y=10 has a RGB value of 60,60,60 (or in YUV a Y value of 60, or you can choose whatever channel to read), then the filter parameter value is 60 for that pixel at x=10,y=10 ( or normalized remapped equivalent to fit the scale of the setting; e.g. if filter goes from -10 to +10 , you'd remap 0-255 to -10 to +10) . And you can add modifiers if need be like +2 or whatever

And that' s done for each pixel position, and then you can apply those results through a mask

[nji]

Exactly!

That's what I'm trying to explain since the very beginning of the thread.
(My English must be much worse than I realized myself.
"Jack...! Zack...! I cannot to swim!!!" :))

However that sketched procedure is what I intuitive want to do.
And the "blending trick" proposed by manono is just an approximation in the general case.
For some filters (the "linear ones" probably) it will be the same.
Don't know for sure.

And the implementation would probably be as sketched in
https://forum.doom9.org/showthread.php?p=1894246#post1894246

But it's unrealistic to have a gradient filter with more than 256 values.
3 channels for example would mean 256^3 times calling the "inner filter"
- for every frame.
No, B/W should be enough to try ... for example gradient sharpening.

[poisondeathray]

for example gradient sharpening.

But for the per pixel application test, as mentioned earlier, some filters like "sharpen" - or any of the sharpen-like derivative filters - do not work on single pixels, single colors alone. If you isolated , cropped down to each 1x1 pixel, then applied sharpen per pixel, the reassembled - it wouldn't work. It requires the sampling and relationship of neighboring pixels and their relationships to each other. You can test this yourself by creating a single color "superpixel" e.g 80x80, and applying sharpen - nothing. Then place another 80x80 "superpixel" of different color beside it, and the edge between them will have a change. "Sharpen" relies on edges, and local contrast, so you can't calculate that on a single pixel by itself



For things working "linearly" - RGB is an additive color model , that might partially explain some of your RGB filter observations. But the "UV" in YUV are color difference channels; not as "simple" in that respect.

Note that each "concept" can have slightly different implementations or interpretations. You can have a RGB "saturation", or a YUV implementation of "saturation". You don't necessarily get the same or even similar results. Even within RGB "saturation" there are slightly different takes on it. e.g. Adobe has the classic saturation, but also vibrance saturation, slightly different formulas. Another example, "contrast" is implemented slightly differently in different programs - some have a different contrast center (what range is the center anchored on), and/or shadow/midtone/highlight contrast controls. In most programs, "contrast" should "push" both ways (brights get brighter, darks get darker). In avisynth tweak's "contrast" operates like a Y' gain, the whole waveform is shifted up only (everything brighter), instead of "pulled " or "pushed" apart at the contrast center

[StainlessS]

IIRC, the old TmpGenc had 3 different versions of contrast,
Bottom relative (same as Gain I think),
Center relative (center being eg 127.5 or 128),
and Top relative.
All three would be possible with single pixel, as not local contrast.
EDIT: RGB.

[nji]

... "Sharpen" relies on edges, and local contrast, so you can't calculate that on a single pixel by itself...


You're still misunderstanding.

What I mean is the most simple you can image.

In your sentence above remove "ON a single pixel" by "FOR a single pixel".
Even if the calculation of a single pixel would depend on ALL other pixels
of the frame ... no problem for what I mean.

As I mentioned already, I never wrote an AviSynth script,
but saw some of them.
So to my thinking the solution I mean would take not more than
say 30 lines.

[nji]

I have to admit ... I'm really getting exhausted, frustrated.
So much effort ... but still not a single step further, only misunderstanding.

I could try to sketch the alg in pseudo code (as I don't do AviSynth).

applyFilterByGradient (sourceClip, gradientMask, destClip) {
// probably possible to do the "innerFilter" as parameter too?

checks for compability of sourceClip, gradientMask, destClip etc.

for all pixels p in the sourceClip {

Definiton of innerFilter and corresponding mapGradientMaskToFilterParameter

filterParameter = mapGradientMaskToFilterParameter (gradientMask [p])
filteredFrame = innerFilter (sourceClip, filterParameter)
destClip [p] = filteredFrame [p];
}
}

Straight thought, simple implemtation.

As I was told the instantiation and call of innerFilter would be expensive
(and the alg above is in order of the number of pixels per frame)
there is a speed up (also sketched in the thread):

Don't iterate on the pixels, but on the 256 values of the b/w gradientmask.
For every value of that:
If the value is same than in the gradient mask at a pixel:
take the applied innerFilter for the dest.
(These things will probably been done by quick AviSynth masking etc.)

That alg has an order ("slow down") of 256.

In VD forum I learned that it was exactly that kind of in-between-
-manipulations of filter that is the great strenght of AviSynth,
and could never been done elsewhere.

BTW
There could be enhancements to let the gradientMask be a clip too.
And to take more than 1 varying paramter for the specific filter.

[nji]

BTW:

I was thinking on the requirements the "inner filter"
has to fulfull to be used in the sketched pattern.

In my example I was thinking about sharpening.
(As for color and HSL changement the "blending workaround"
should do the job too. Probably. Maybe. Dunno).

So the requirement:
Does not change the location of the pixel it operates at?

Not nescessarily.
(Hey, let's turn crazy NOW :))
As it actually still delivers a valid result,
if the pixel is locally moved.
So - for example - rotate.
(Don't ask me about the arty result :))
You won't be able to do that with the bl... blend job.

OK then.
The only restriction for the inner filter is
that it must not change frame dimensions or timeline.
(Although ... although ... I can well image
expanding the algorithm to timeline changes too.
("Let's do the timewarp ... agahain ... " :))
And - if ONE thing is for sure - I won't be able
to explain what I am thinking about :))

And all with ONE single pattern!

Sadly "no arms - no cookies".

[poisondeathray]

Got it; 256 filter states already calculated, already applied to whole image (or not necessarily, e.g. it might be top half only if that's where the mask or area of interest was was etc...then you could "speed" it up), or maybe it can be done in a way at runtime

Each pixel 's channel value that you're reading on the control layer "pulls" that version for that pixel

Is that accurate enough? Maybe for some cases. Some filters might require more "notches" or precision if you want an exact match . You might tweak sat=1.32 to match it "perfectly". 10bit values would give you 1024 but be even more horrendous to calculate and for speed

Not sure how to do it.

[StainlessS]

How does this sound,

8 bit only (others may expand plugin for 10 bit)

PixelSelectByMask(Clip SrcFrm,clip MaskFrm,Clip Data)

where,
SrcFrm is your single frame source clip
MaskFrm is single frame Mask clip [Planar Luma_Y]
Data, is precalc'ed 256 (for 8 bit) frame data clip [if somebody else updates then eg 1024 frame for 10 bit etc]

Y8, YV24, RGB24, RGB32 Only.

we need some name, unless PixelSelectbyMask() is acceptable (also suggestions for clip names if different from above).

You could do the other stuff in script, the real slow stuff could be achieved with this function.

could maybe have something like this done within 24 hours [EDIT: Make that 48, I could be busy].

Any good ???

EDIT: If not exactly as required, then suggest mod.
I note mention of Channel, should there also be Chan arg, where eg SrcFrm[x,y,Chan] modified via MaskFrm[x,y] to Data[MaskFrm[x,y]][x,y,Chan],
[EDIT: if Chan== -1, then affect all 3 channels(1 channel if Y8), -2 affect all 4 channels if RGB32]

What is you wantin' ?

State exactly what you want if not as above.

EDIT:
Or is it, Data = single frame, 1 dimension, 256x1 pixel array clip [which is really just a lut] ???

[nji]

Hey StainlessS.

Thank you VERY much for your offer!!!

Just to make sure we do not misunderstand each other:

The methode you propose is to use in the sketched alg
in https://forum.doom9.org/showthread.php?p=1894349#post1894349
The modification from paragraph
"... Don't iterate on the pixels, but ..."
Am I right?
If not, we are missing each other, and we have to make clear.

All this doesn't press in time.
But I think providing that script to the public
will increase the freedom to try manipulations in general
- not only for crazy nji :)

[StainlessS]

What I mean by above description is,

YOU create 256 modified input frame versions of SrcFrm(with varying control, as used/represented by Mask value) ; result in Data, where each frame is modified by eg increasing saturation (or whatever).
Result[x,y] pixel gotten from Data[MaskFrm[x,y]][x,y]

So, MaskFrm[x,y] selects which Data frame to get pixel from {and from the x,y coord}.

Is that as you want ?

EDIT:

Having looked at your outline, looks to me like you want this

For eg RGB 8 bit 3 channel, with possible 2^24 = 16777216 input colors, with 256 steps of control = lut size of 256 * 16777216 = 4294967296 colors LUT array,
with 3 bytes for each color being 12,884,901,888 or 12+ Billion bytes, not sure that is practical [and that's for only 8 bit].

I suggest that my solution of creating 256 modified frames using eg 'Tweak(Sat=G)' where G= is M*Mult variable with 256 values of M,
then get for each pixel M=MaskFrm[x,y] the pixel value of the modified frame[M] at x,y coords.

Using this method, you can calc result pixels per input frame in real time[well at every frame], rather than ahead of time(with massive lut), and due to eg frame caching, then speed will not be so bad,
although would need to precalc modified 256 frames at each input frame (1024 frames if 10 bit).

PDR, does above fit with your anticipated usage.

EDIT:
Don't iterate on the pixels, but on the 256 values of the b/w gradientmask.
For every value of that:
If the value is same than in the gradient mask at a pixel:
take the applied innerFilter for the dest.

That is the prep stage to create 256 frames in advance of each input frame.

[nji]

Back in 2020 ... my very best wishes for the ones reading this.
:) :) :) :) :) :) :) :) :) :) :) :) :) :) :) :) :) :) :)

StainlessS, this time it was ME to make the effort to understand.
I'm not sure that I get everything you proposed to 100%.
But ... yes ...

You several times asked, if that would be the thing I am "wanting".
I do not WANT a special implementation.
What I am talking about is an actually very simple idea:
Apply an (just call it: ) "inner filter" (say sharpen) on a SrcClip.
But the parameter of the inner filter should not be constant
(say sharpness = 42.42) but should be calculated for ever
source pixel of SrcClip with a different amount of sharpness.
This amount (for every pixel) is given by a "gradient mask".
To make it simple (in a first step) the "gradient mask"
shall be 8 bit only, and just 1 frame for the whole movie.
(Both (8 bit, 1 frame) should only be expanded in
an advanced version later.
As it makes it even more complicated to use/ understand
for a first step. And more than 8 bit would take even more time/ lut.
And a clip as gradientMask would have to be generated first.
So keep it simple for the first shot.)
That's the plain idea.

The implemenation should be the way that every user
just modifies its code by inserting
- the mapping function (from gradient value to value of inner filter parameter)
- the inner filter

How much sophisticated the implemenation will be
doesn't care at all, as long as it does the job.

If I get you right, then you have in mind to do an even more sophisticated
improvement concerning runtime, using the lut etc.?
This sounds good to me.
As the runtime will be the bottle neck of all.
(Maybe in a later version it could also do multithreaded,
as if this is apropriate anywhere ... then here?)
But, in first case, this is about to have a solution for the problem at all.
(Would you like me to tell you how many hours my IBM clone did on
Mandelbrot sets in the 80ies? :) )

I hope that this script will be basis for many others.

And it will show how much the (and if) there are difference
from the simple "blending workaround", depending on "inner filter"
and the used parameters.

Very exciting that, don't you agree?

[nji]

... addendum concerning runtime.

Hope this won't scare off:

The runtime (if I get it right) of that will be about

(Total number of possible (!) gradientFilterValues) times
that 1 call of the inner filter takes.
(As the mask operations would not take long I assume).

==> could (maybe later) be improved by
calculating not for all possible gradientFilterValues (say 256),
but only for the ones that are used in the gradientMask.
Although in practise users will use the full range (256) of gradient,
so it's not worthwhile making the code complicated for that.
But - as mentioned - multithreading would do very efficient,
as calculation the luts is independent.
And probably just using different AVS keywords for the loop?
(Don't know however)

OK, now it's up to you to propose the next step of improvement. :) :)
Although I don't have any idea how to do even better.
However - maybe should "taste the pudding" first.
Maybe at last it turns out that the result of point-wise filter application
and "frame blend job" is the same.
Then I would have to ask a mod to delete the thread
and for the rest of my life deny it ever exists. :) :)

[StainlessS]

No good news here, I tried two versions of the plugin to do as you want, with clip of only 256x256 pixels, consumes 11.5 GB of ram
out of my 12GB (only runs on x64), and takes bout 2 mins for single frame, crash after about 4 frames.
No idea why takes 11.5GB for single frame, 256 frames of 256x256 * 4 (RGB32) is only 64MB.
I'll try post a plug (with source) sometime today, maybe somebody has some idea why it takes so much ram.
(I did it back about 31 Dec, was a bit pissed off that it failed so badly).

[nji]

Hi StainlessS.
Thankyou for the message anyway.
To tell the truth I was kind of frustrated too,
because I thought that anybody who is familar with AviSynth
could do that (quite simple?) pattern.
Sadly I do not belong to them.
(I could do it with C++/ OpenGL right way).

Concerning your first try:
There must be bugs somewhere.
Either memory usage as well as run time is far too much.
To my estimation a 720p should do at about 1fps.
And it also should use not much mem.

Please, don't give up.

[StainlessS]

OK, played a bit more, faster and a bit more stable.
http://www.mediafire.com/file/ufswjavd5gsch2z/SelectPixelByMask_avs%252B_x86x64_dll_v0.00_20200111.7z/file


/*
Req:
Avs+
SelectPixelByMask
Grunt

SelectPixelByMask(clip Data, clip Mask)
Data 256 frames, single frame, Y8 or YV24, or RGB24, or RGB32. # EDIT: single frame, should not be there
Mask single frame, Y8 [EDIT: Changed, can be multiframe]
*/

Function GreyRamp() { # https://forum.videohelp.com/threads/385341-avisynth-how-to-add-borders#post2498735
black = BlankClip(color=$000000, width=1, height=256)
white = BlankClip(color=$ffffff, width=1, height=256)
StackHorizontal(black,white)
BilinearResize(512, 256)
Crop(128,0,-128,-0)
}

Function MyFunc(clip c, Float S, Float E) { # User written function to do whatever should be done via Mask value, FrameNo=Mask Value data
c.Trim(0,-1) # ENSURE Single frame
Loop(256) # 256 identical frames
Last.Animate (0, 255, "Tweak",
\ [* START Args *]
\ 0.0, [* Hue default 0.0 *]
\ 1.0, [* Sat default 1.0 *]
\ 0.0, [* Bright default 0.0 *]
\ S, [* Cont default 1.0 *]
\ False, [* Coring Default True *]
\ [* END Args *]
\ 0.0, [* Hue default 0.0 *]
\ 1.0, [* Sat default 1.0 *]
\ 0.0, [* Bright default 0.0 *]
\ E, [* Cont default 1.0 *]
\ False [* Coring Default True *]
\ )
Return Last # Return 256 frame Data clip
}


SSS="""
Data = Last.Trim(current_frame,-1).MyFunc(Start,End) # 256 frame Data clip
return SelectPixelByMask(Data,Mask)
"""

# CONTRAST
Start = 0.5 # Mask 0
End = 2.0 # Mask 255

###################
AviSource("D:\SELECTPIXELBYMASK\Parade.avi")
trim(500,0) # trim off static start
BilinearResize(256,256) # Try not to swamp memory too much
ConvertToYV24() # Sat will not work on Y8, And not RGB either (Tweak YUV only)
#ConvertToY8() # Sat will not work on RGB (Tweak YUV only)

#Return trim(0,-1).MyFunc(Start,End) # Show single frame 0, changed over 256 frames with MyFunc()

###################
Mask = GreyRamp().Trim(0,-1).ConvertToY8.Loop(Framecount) # Same Framecount as clip
#Mask=Mask.BilinearResize(Width,Height) # Rough ramp, single frame, same size as clip, just for testing
#return Mask.info

ARGS="Mask,Start,End"
Out = ScriptClip(SSS,Local=True,args=ARGS)
#return Out.info
Return StackHorizontal(Last.ShowFrameNumber,Out)


I've tried up to about 800 frames via Vdub2(x64, x86 dll will bomb sooner), on attempt close of VDub2, hangs
Not responding, force close seems not to release all mem. [EDIT: We do not use malloc/new in dll to allocate mem]

So, working to some degree. (only tested with YV24, also supports Y8, RGB24, RGB32)

Supplied with VS2008 Project files and source.

EDIT: Some of the source (implemented as source filter)

class SelectPixelByMask : public IClip {
private:
const VideoInfo vi;
bool parity;
int last; // remember last frame generated
PVideoFrame destframe;
//
PClip Data;
PClip Mask;
//
public:
SelectPixelByMask(const VideoInfo& _vi,PVideoFrame _destframe, bool _parity,PClip _Mask,PClip _Data,IScriptEnvironment* env);
~SelectPixelByMask();
void __stdcall GetAudio(void* buf, __int64 start, __int64 count, IScriptEnvironment* env) {}
const VideoInfo& __stdcall GetVideoInfo() { return vi; }
bool __stdcall GetParity(int n) { return (vi.IsFieldBased() ? (n&1) : false) ^ parity; }
int __stdcall SetCacheHints(int cachehints,int frame_range) {return 0;}
PVideoFrame __stdcall GetFrame(int n, IScriptEnvironment* env);
};

SelectPixelByMask::SelectPixelByMask(const VideoInfo& _vi,PVideoFrame _destframe, bool _parity,PClip _Mask,PClip _Data,IScriptEnvironment* env)
: vi(_vi), destframe(_destframe),parity(_parity), last(-1), Mask(_Mask), Data(_Data) {
}

SelectPixelByMask::~SelectPixelByMask() {
}

PVideoFrame __stdcall SelectPixelByMask::GetFrame(int n, IScriptEnvironment* env) {
n= (n<0) ? 0 : (n>=vi.num_frames) ? vi.num_frames-1 : n; // Range limit n
if(n != last) {
PVideoFrame pv[256];
const BYTE *datparr[256];
const BYTE *datparrU[256];
const BYTE *datparrV[256];
int pitcharr[256];
int pitcharrUV[256];
//
int w = vi.width;
int h = vi.height;
int i;
for(i=0; i<256 ;++i ) {
pv[i] = Data->GetFrame(i, env);
const int pitch = pv[i]->GetPitch(PLANAR_Y); // PLANAR_Y no prob for RGB (ie, first plane)
pitcharr[i]=pitch;
const BYTE *datp = pv[i]->GetReadPtr();
datparr[i] = (vi.IsRGB()) ? datp + ((h - 1) * pitch) : datp; // Top line
if(vi.IsYV24()) {
const int pitchUV = pv[i]->GetPitch(PLANAR_U);
pitcharrUV[i]=pitchUV;
datparrU[i] = pv[i]->GetReadPtr(PLANAR_U);
datparrV[i] = pv[i]->GetReadPtr(PLANAR_V);
}
}
BYTE* p = destframe->GetWritePtr(); // Try to reuse old frame
if (!p) {
destframe = env->NewVideoFrame(vi); // Ditch old frame and create a new frame
p = destframe->GetWritePtr();
}
const int destpitch = destframe->GetPitch(PLANAR_Y); // PLANAR_Y no prob for RGB (ie, first plane)
//
PVideoFrame maskfrm = Mask->GetFrame(n, env);
const BYTE * maskp = maskfrm->GetReadPtr(PLANAR_Y);
const int maskpitch = maskfrm->GetPitch(PLANAR_Y);
//
int x,y;
//
if(vi.IsRGB24()) {
p += ((h - 1) * destpitch); // RGB Upside down, start at top, ie match mask direction
for(y=0 ; y < h ; ++y) {
for(x=0 ; x < w ; ++x) {
const int m = maskp[x]; // BYTE Mask value
const BYTE * datap = datparr[m] - (y * pitcharr[m]); // RGB Upside down
const int x3=x*3;
p[x3 + 0] = datap[x3 + 0]; // B
p[x3 + 1] = datap[x3 + 1]; // G
p[x3 + 2] = datap[x3 + 2]; // R
}
maskp += maskpitch;
p -= destpitch; // RGB upside down
}
} else if(vi.IsRGB32()) {
p += ((h - 1) * destpitch); // RGB Upside down, start at top, ie match mask direction
for(y=0 ; y < h ; ++y) {
for(x=0 ; x < w ; ++x) {
const int m = maskp[x]; // BYTE Mask value
const BYTE * datap = datparr[m] - (y * pitcharr[m]); // RGB Upside down
const int x4=x*4;
p[x4 + 0] = datap[x4 + 0]; // B
p[x4 + 1] = datap[x4 + 1]; // G
p[x4 + 2] = datap[x4 + 2]; // R
p[x4 + 3] = datap[x4 + 3]; // A
}
maskp += maskpitch;
p -= destpitch; // RGB upside down
}
} else if(vi.IsY8()) { // Src, Y8
// p already OK
for(y=0 ; y < h ; ++y) {
for(x=0 ; x < w ; ++x) {
const int m = maskp[x]; // BYTE Mask value
const BYTE * datap = datparr[m] + (y * pitcharr[m]);
p[x] = datap[x];
}
maskp += maskpitch; // Incr mask pointer
p += destpitch; // Incr dest pointer
}
} else { // Src YV24
BYTE * pU = destframe->GetWritePtr(PLANAR_U);
BYTE * pV = destframe->GetWritePtr(PLANAR_V);
const int pitchUV = destframe->GetPitch(PLANAR_U);
for(y=0 ; y < h ; ++y) {
for(x=0 ; x < w ; ++x) {
const int m = maskp[x]; // BYTE Mask value
const BYTE * datap = datparr[m] + (y * pitcharr[m]);
const int datapitchUV = pitcharrUV[m];
const BYTE * datapU = datparrU[m] + (y * datapitchUV);
const BYTE * datapV = datparrV[m] + (y * datapitchUV);
p[x] = datap[x];
pU[x] = datapU[x];
pV[x] = datapV[x];
}
maskp += maskpitch; // Incr mask pointer
p += destpitch; // Incr dest pointer
pU += pitchUV; // Incr destpU pointer
pV += pitchUV; // Incr destpV pointer
}
}
last = n; // Remember last generated frame number
//
for(i=256; --i>=0 ; ) {
pv[i] = 0; // release data frames
}
}
return destframe;
}

AVSValue __cdecl Create_SelectPixelByMask(AVSValue args, void* user_data, IScriptEnvironment* env) {
const char * myName="SelectPixelByMask: ";
//
PClip Data = args[0].AsClip();
PClip Mask = args[1].AsClip();
const VideoInfo &Data_vi = Data->GetVideoInfo();
// Check supported ColorSpaces
if(!
(Data_vi.IsYV24() || Data_vi.IsY8() || Data_vi.IsRGB24() || Data_vi.IsRGB32())
) env->ThrowError("%sData clip 2, YV24, Y8, RGB24, and RGB32 Only",myName);
if(Data_vi.num_frames != 256) env->ThrowError("%sData Clip requires 256 frames",myName);

int w = Data_vi.width;
int h = Data_vi.height;

if(w==0) env->ThrowError("%sData Clip has no video",myName);

const VideoInfo &Mask_vi = Mask->GetVideoInfo();
if(!Mask_vi.IsY8()) env->ThrowError("%sMask clip Must be Y8",myName);
if(Mask_vi.width != w) env->ThrowError("%sMask Clip must have same width as Src clip",myName);
if(Mask_vi.height != h) env->ThrowError("%sMask Clip must have same height as Src clip",myName);
//
VideoInfo newvi = Data_vi; // Copy video info from Src
newvi.num_frames = Mask_vi.num_frames; // Result will be same frame count as maskframes
PVideoFrame destframe = env->NewVideoFrame(newvi); // Create initial/cache frame (will auto delete on close)
bool parity=false; // TODO
AVSValue ret = new SelectPixelByMask(newvi,destframe,parity,Mask,Data,env); // Dump child src, and use newvi (dont keep any lock on src)
return ret;
}

[StainlessS]

Another script.
Attempt at reducing tendency to run out of RAM, not really much better but perhaps of use.
Writes each output frame to image file.

I gave up at about frame 1400 and forced closure of VDub2 x64.
RAM climbs to about 11.8GB used, then drops to 11.3GB, at about 800 frames, RAM usage dropped to about 10.xGB, then started to rise agian.
We use for/next loop to access frames in sequence, instead of playing clip, intent to avoid dependance on previous frames (as would be case in playing),
however something is still hanging on to used RAM, no idea as to what or why.
Outputs current frame number to DebugView (Google)

/*
Req:
Avs+
SelectPixelByMask
Grunt
RT_Stats.
requires OUT\ directory in current dir

SelectPixelByMask(clip Data, clip Mask)
Data 256 frames, single frame, Y8 or YV24, or RGB24, or RGB32.
Mask Y8
*/

Function GreyRamp() { # https://forum.videohelp.com/threads/385341-avisynth-how-to-add-borders#post2498735
black = BlankClip(color=$000000, width=1, height=256)
white = BlankClip(color=$ffffff, width=1, height=256)
StackHorizontal(black,white)
BilinearResize(512, 256)
Crop(128,0,-128,-0)
}

Function MyFunc(clip c, Float S, Float E) { # User written function to do whatever should be done via Mask value, FrameNo=Mask Value data
c.Trim(0,-1) # ENSURE Single frame
Loop(256) # 256 identical frames
Last.Animate (0, 255, "Tweak",
\ [* START Args *]
\ 0.0, [* Hue default 0.0 *]
\ 1.0, [* Sat default 1.0 *]
\ 0.0, [* Bright default 0.0 *]
\ S, [* Cont default 1.0 *]
\ False, [* Coring Default True *]
\ [* END Args *]
\ 0.0, [* Hue default 0.0 *]
\ 1.0, [* Sat default 1.0 *]
\ 0.0, [* Bright default 0.0 *]
\ E, [* Cont default 1.0 *]
\ False [* Coring Default True *]
\ )
Return Last # Return 256 frame Data clip
}

# CONTRAST
Start = 0.5 # Mask 0
End = 2.0 # Mask 255
BaseName = ".\OUT\MyPic_"
Matrix = "Rec601"
myName = "SelPix2: "
DEBUG = True

###################
AviSource("D:\SELECTPIXELBYMASK\Parade.avi")
trim(60,0) # trim off static start
BilinearResize(256,256) # Try not to swamp memory too much
ConvertToYV24() # Sat will not work on Y8, And not RGB either (Tweak YUV only)
ORG=Last
FC=ORG.Framecount
#Return trim(0,-1).MyFunc(Start,End) # Show single frame 0, changed over 256 frames with MyFunc()

Msg=MessageClip("Done") # Create now when mem avilable
###################
Mask = GreyRamp().Trim(0,-1).ConvertToY8.Loop(FC) # Same Framecount as clip
#Mask=Mask.BilinearResize(Width,Height) # Rough ramp, single frame, same size as clip, just for testing
#return Mask.info

BaseName=RT_GetFullPathName(BaseName)

StartTime=RT_TimerHP
For(n=0,Framecount-1) {
current_frame=n
Data = ORG.Trim(n,-1).MyFunc(Start,End) # 256 frame Data clip
Out=SelectPixelByMask(Data,Mask).Loop(FC)
Out=Out.ConvertToRGB32(matrix=Matrix).ImageWriter(file=BaseName, start=n,end=n, type="png")
Out.RT_AverageLuma(w=1,h=1) # Force write
if(DEBUG) {
Tim = RT_TimerHP
RT_DebugF("%d] %f%% %.2fFPS",n,(n+1)*100.0/FC,1/(Tim-StartTime),name=myName)
StartTime=Tim
}
Out=0
Data=0
}

Return Msg

EDITED:

EDIT: System seems a bit unstable after running either script, reboot a good idea.
EDIT: Updated, Crash at about 1445 frames, about 3.17FPS at start, dropping to maybe 2.5FPS at frame 500.

[nji]

Oh man - this looks far more complex than I had thought.

The less I'm able to contribute.

A thing I know about is that the collaboration between VD and AviSynth
is problematic sometimes when the script is very complex.
For example using MSU script for interpolating ... when doing
fastward/-backward in VD it crashes regulary. Have to jump to timeline.

About the code in general.
If I'm about to encircle strange effects I usually strip the code more and more
to see what's the cause.
So maybe disable multithread first.
If it still has unexpected effects (mem, time) maybe strip (comment)
lines of code one by one?
Especially mem usage should be really low for the alg ... I think.
The 256 masks can be handled not all at once but one after the other,
and for each iteration there are only 3 to 4 additional masks.
(==> There must be some strange thing happening).

Sorry I cannot tribute more than silly thoughts. :)

[StainlessS]

So maybe disable multithread first.
Is not multithreaded.
maybe strip (comment)
Not relevant.
The 256 masks can be handled not all at once but one after the other,
and for each iteration there are only 3 to 4 additional masks.
Total non-starter, and back to ~2 mins (or more) per frame.

I think problem in Avisynth, is hanging on to mem somewhere.

[nji]

Total non-starter, and back to ~2 mins (or more) per frame.

I think problem in Avisynth, is hanging on to mem somewhere.

Another stupid thouht :) from my side:

So maybe the mem problem of AviSynth is the cause for the "run"time?
(Getting/ Letting mem usually takes long)

And then ... maybe worthwile to dig further to get a potential bug
Avisynth?

[StainlessS]

Removed script functions [ GreyRamp(), MyFunc() ], embed code directly into main level script.
Not much difference. (craps out about 1440 frames)


/*
Req:
Avs+
SelectPixelByMask
RT_Stats.
requires OUT\ directory in current dir

SelectPixelByMask(clip Data, clip Mask)
Data 256 frames, Y8 or YV24, or RGB24, or RGB32.
Mask Y8
*/

# CONTRAST
StartCont = 0.5 # Mask 0
EndCont = 2.0 # Mask 255
BaseName = ".\OUT\MyPic_"
Matrix = "Rec601"
myName = "SelPix3: "
DEBUG = True

###################
AviSource("D:\SELECTPIXELBYMASK\Parade.avi")
trim(60,0) # trim off static start
BilinearResize(256,256) # Try not to swamp memory too much
ConvertToYV24() # Sat will not work on Y8, And not RGB either (Tweak YUV only)
ORG=Last
FC=ORG.Framecount

Msg=MessageClip("Done") # Create now when mem avilable
###################
black = BlankClip(Length=1,color=$000000, width=1, height=256,Pixel_type="Y8")
white = BlankClip(Length=1,color=$ffffff, width=1, height=256,Pixel_type="Y8")
Mask=StackHorizontal(black,white).BilinearResize(512, 256).Crop(128,0,-128,-0).Loop(FC)

#Mask=Mask.BilinearResize(Width,Height) # Rough ramp, single frame, same size as clip, just for testing
#return Mask.info

BaseName=RT_GetFullPathName(BaseName)

StartTime=RT_TimerHP
For(n=0,FC-1) {
current_frame=n
Data=ORG.Trim(0,-1).Loop(256) # 256 identical frames
Data=Data.Animate (0, 255, "Tweak",
\ [* START Args *]
\ 0.0, [* Hue default 0.0 *]
\ 1.0, [* Sat default 1.0 *]
\ 0.0, [* Bright default 0.0 *]
\ StartCont, [* Cont default 1.0 *]
\ False, [* Coring Default True *]
\ [* END Args *]
\ 0.0, [* Hue default 0.0 *]
\ 1.0, [* Sat default 1.0 *]
\ 0.0, [* Bright default 0.0 *]
\ EndCont, [* Cont default 1.0 *]
\ False [* Coring Default True *]
\ )
Out=Data.SelectPixelByMask(Mask).Loop(FC)
Out=Out.ConvertToRGB32(matrix=Matrix).ImageWriter(file=BaseName, start=n,end=n, type="png")
Out.RT_AverageLuma(w=1,h=1) # Force write image file
if(DEBUG) {
Tim = RT_TimerHP
RT_DebugF("%d] %f%% %.2fFPS",n,(n+1)*100.0/FC,1/(Tim-StartTime),name=myName)
StartTime=Tim
}
Out=0
Data=0
}

Return Msg

[nji]

Please excuse: I made a severe typo in my post before.
What I meant was:

If traced down to the line of AviSynth code which causes the mem problem,
and it should not grap/ keep as much,
then maybe it's an error of AviSynth?
(Which would make this investigation even more valuable)

[StainlessS]

Which would make this investigation even more valuable
Maybe so, but nothing that I can do about it.

[nji]

Maybe so, but nothing that I can do about it.

No, of course.
That would be an issue for the AviSynth development team then.

But, if I get you right, it is not as far, that it is known if it's an error in your script or in a specific AviSynth statement?

EDIT:
A third option (that this alg can't be done without huge ressources of space and time)
cannot be true. That's what I do know for sure.

[zorr]

This piqued my interest. Unfortunately my code-fu is not very strong when it comes to Avisynth, but I was able to implement this in a VapourSynth script.

I haven't done any extensive testing but it seems to run ok, memory consumption is about 2,4GB when running with a 1080x576 source video. The 8-bit mask in this case had all 256 values in it, the script will skip calculating the effect for the missing mask values and will be more efficient when there are less mask values. Speed was about 1fps with 256 mask values and about 5-6 fps with 16 mask values, using a horizontal blur effect as the test function (I had another heavy task running in the background so these measurements are not very accurate).

If you're interested I can post the script (to VapourSynth section of this forum).

[nji]

... I was able to implement this in a VapourSynth script.
...If you're interested I can post the script (to VapourSynth section of this forum).

Yes, please do.

@StainlessS:
Why does the Avisynth version crash/ take so much mem?

[StainlessS]

If you're interested I can post the script (to VapourSynth section of this forum).
I guess you could do, I'll take a peek when I have time.

Why does the Avisynth version crash/ take so much mem?
No idea, nothing that I'm doing [EDIT: I think], I dont allocate any RAM, is connected with the way avisynth is implemented.
EDIT: I do create a new frame on filter construction, but should be destroyed when destructing [when setting the relevant clips to 0 in script].

EDIT: Zorr,
Would your code work with all filters, eg could you for instance using a gray ramp, blur max 1.58 at frame left, null in middle and sharpen max 1.0 at frame right.
(or eg temporal smooth or sharpen, or MC filtering, or frame replacement filtering, or is it just a LUT).

EDIT: nji
I think that you mentioned OpenGL somewhere. IIRC, that stuff has some kind of pixel filtering operations, so it is likely exactly what you want
(dont know much at all about OpenGL, and not relally feeling inclined to duplicate OpenGL as some kind of Avisynth filter).