V2.60 - variables?

[martin53]

The following is a fragment of a 800+ lines script. This morning, after bugfixes somewhere else far away in the script, it produces weird results. :mad:

RT_Debug("R before AverageLuma", string(R), isFloat(R)?"float":"not float")
Y = cYUV.AverageLuma - 16 * 255.0/219
RT_Debug("R after AverageLuma", string(R), isFloat(R)?"float":"not float")
The code runs in a GScript in a GScriptclip:
cRGB.Echo(cPre, cGamutMask, cR, cG, cB, cRMask, cGMask, cBMask, cYUV)
GScriptClip("GScript("+chr(34)+chr(34)+chr(34)+""" # "
...
"""+chr(34)+chr(34)+chr(34)+")", args="cYUV, cR, cG, cB, cRMask, cGMask, cBMask, ymax, ydif, amount, border, RBplaus, Gplaus, pt1, rules, reduce, haze, ExtColRange, power, P, THRESH, KR, KG, KB, UVRANGE_CHECK_YMIN, info")


DebugView output:

[1836] RT_Debug: R before AverageLuma 0.943805 float
[1836] RT_Debug: R after AverageLuma 202 float
Clip size is 540x360, it is a ImageSource() sequence with 1230 jpg pictures.
Effect is only present at frame#0. I have several similar sequences, example results are:

[6508] RT_Debug: R before AverageLuma 1.002005 float
[6508] RT_Debug: R after AverageLuma 162 float

[4368] RT_Debug: R before AverageLuma 0.927987 float
[4368] RT_Debug: R after AverageLuma 135 float

[6700] RT_Debug: R before AverageLuma 0.963102 float
[6700] RT_Debug: R after AverageLuma 160 float


System:
Win7 x86
AviSynth 2.60 build Jan14 2013
AvsPmod 2.4.2
dither 1.23.0
GRunT 1.0.1
GScript 1.1
MaskTools v2.0a48
RT_Stats V1.22 26
MYStats v2.6

btw yesterday's version of full script is here (http://forum.doom9.org/showthread.php?t=168062)

[martin53]

P.S. Renaming R to red throughout the GScriptClip salves the symptom - is not a favourable solution however
(I intend to allow the user to extend the script code with a string, and R, G, B are intended variable names for the code string)

Also, changed AverageLuma() to RT_AverageLuma() as a test and the effect was still present.

[martin53]

Other variables inside the GScriptClip were affected too :eek:, and not only at first frame, but only sporadically and I could not identify a pattern. Script start/reload or jumps in the timeline seem to trigger it. If it happens at a frame, it will always happen the same way when the frame is activated the same way, i.e. if it is again reached with 'forward 1 second', but not when it is reached by advancing frame by frame.
But because of the type of my function, it ends with 'Preroll \n return last'. Therefore, I expected that a jump to a frame should do the same as advancing to it frame be frame :confused:

The affected variables had one letter names. I found a maybe assicoated problem the other day, where 'c' inside one ScriptClip suddenly was no more the c it had been two lines above, but the c from another Scriptclip. In that case it became apparent because these two 'c' had different pixel types.

Maybe I should place a bet that one letter variable names in ScriptClips are vulnerable:sly:. I can not rewrite the ScriptClips to work without GRunt, so I can not recheck without having that plugin installed, but I doubt it is the reason.

[Gavino]

Sporadic failures and weird effects like getting different results simply from changing variable names are classic symptoms of memory corruption. These problems can be difficult to resolve as the symptoms often appear in a part of the script completely unrelated to where the corruption actually occurs.

Since GScript and GRunT are being used, I feel duty bound to investigate (although the problem could easily be something else).
Unfortunately (due to a hard disk failure), I am temporarily without a computer of my own, so can't do very much at the moment. However, I will take a look at your script when I find some time (probably not until at least tomorrow).

This morning, after bugfixes somewhere else far away in the script, it produces weird results.
Can you say exactly what changes you made?
These may be related to the problem, although they could simply be unrelated changes which (like changing variable names) alter memory contents in such a way as to trigger the symptoms you saw.

[martin53]

Gavino,
thanks for noticing my post.

The changes were about making the script suitable for RGB as well as YUV input, mainly optimizing the chain(s) of internal clips and their pixel formats - more towards script start and outside the RTE.

As I wrote, I also noticed a puzzling effect that the clip 'c' of one ScriptClip sporadically appears inside another ScriptClip in the place of its own 'c'. I gave that observation as a PM to StainlessS, because that appeared directly after a RT_LumaSceneChange() line.

Considering the things I observed, I'd say: as I understand, general memory corruption might cause program crashes, script error messages etc - I did never see that.
The problem is more narrow to the variables storage, it sees. I observe
- the problem happens in RTE,
- it corrupts variable content inside the RTE (also clip variables),
- and it is triggered by RTE functions with non clip return value (i.e. AverageLuma etc, execution at RTE 'compile time')
- when a new variable is declared with the function result.
I am unsure if the variable content is overwritten. I feel it's more likely that variables point to other variables' content (or contain a copy of another variable's content) instead of their own.

E.g. the RT_LumaSceneChange() writes to a variable, and very few lines below, this variable and another global variable with the previously active current_frame trigger scene change mechanisms of the script. Sporadically, the DebugView window indicates that scene change is triggered although the variable value was below the threshold, and the previous frame was current-1 i.e. the trigger conditions are not met.

[StainlessS]

This is the snip of script referred to by Martin53, with a couple of additions noted by ssS:


Import("_AWB_130829.txt")

function foobar(clip c) {
info = true
RT_Debug("c", c.PixelType)
c
ConvertToYV24
GScriptClip("GScript("+chr(34)+chr(34)+chr(34)+""" # "
RT_Debug("Last in ScriptClip BEFORE TRY", Last.PixelType) ### ssS ADDED
c = last ### ssS ADDED
RT_Debug("c in ScriptClip BEFORE TRY", c.PixelType) ### ssS ADDED
Try {
RT_Debug("Last in ScriptClip TRY", Last.PixelType) ### ssS ADDED
c = last
RT_Debug("c in ScriptClip TRY", c.PixelType) ### ssS ADDED
# fc = RT_LumaSceneChange(c, c, delta=-1) ### comment this line and it changes
# fc = RT_LumaMovement(c, c, delta=-1) ### comment this line and it changes
# fc = RT_AverageLuma(c, delta=-1) ### comment this line and it changes
fc = YDifferenceFromPrevious(c) ### comment this line and it changes
RT_Debug("Last in ScriptClip TRY AFTER YDifferenceFromPrevious", Last.PixelType) ### ssS ADDED
RT_Debug("c in ScriptClip TRY AFTER YDifferenceFromPrevious", c.PixelType) ### ssS ADDED
c
RT_Debug("c in ScriptClip", c.PixelType)
cu = c.UToY

c
cAdjusted = last

return cAdjusted
} catch(err) {
RT_Debug("ScriptClip 1 ########", err)
}
"""+chr(34)+chr(34)+chr(34)+")", args="info") # "
}
#=====================================================

ColorBars(pixel_type="RGB32")
c = Last.ConvertToYV24
ConvertToRGB()
GScriptClip(""" #"
c = last.blur(1.58)
x=round(1.5*256)
y=round(1.5*152)
w=round(1.5*54)
R = c.ShowRed(pixel_type="Y8").RT_YPlaneMax(x=x, y=y, w=w, h=w, threshold=1.0)
G = c.ShowGreen(pixel_type="Y8").RT_YPlaneMax(x=x, y=y, w=w, h=w, threshold=1.0)
B = c.ShowBlue(pixel_type="Y8").RT_YPlaneMax(x=x, y=y, w=w, h=w, threshold=1.0)
RGBBalancedShift(1.0/R, 1.0/G, 1.0/B)
return last
""") #"
ConvertToYV24()
Overlay(BlankClip(last, width=54, height=54, color_yuv=$ff8080), x=256, y=152)
c2 = last
RT_Debug("c2", c2.PixelType)
c2.foobar()

return convertToRGB32()


Debug

00000005 1.92376041 RT_Debug: c2 YV24
00000006 1.94367695 RT_Debug: c YV24
00000007 2.64812660 RT_Debug: Last in ScriptClip BEFORE TRY YV24
00000008 2.65295386 RT_Debug: c in ScriptClip BEFORE TRY YV24
00000009 2.65365505 RT_Debug: Last in ScriptClip TRY YV24
00000010 2.65422726 RT_Debug: c in ScriptClip TRY YV24
00000011 2.70037937 RT_Debug: Last in ScriptClip TRY AFTER YDifferenceFromPrevious YV24
00000012 2.70092607 RT_Debug: c in ScriptClip TRY AFTER YDifferenceFromPrevious RGB32
00000013 2.70149994 RT_Debug: c in ScriptClip RGB32
00000014 2.70209956 RT_Debug: ScriptClip 1 ######## UVtoY: YUV data only!
00000015 2.72632909 RT_Debug: ([GScript], line 17)
00000016 2.74119353 RT_Debug: Last in ScriptClip BEFORE TRY YV24
00000017 2.74202228 RT_Debug: c in ScriptClip BEFORE TRY YV24
00000018 2.74276805 RT_Debug: Last in ScriptClip TRY YV24
00000019 2.74346662 RT_Debug: c in ScriptClip TRY YV24
00000020 2.79378557 RT_Debug: Last in ScriptClip TRY AFTER YDifferenceFromPrevious YV24
00000021 2.79432392 RT_Debug: c in ScriptClip TRY AFTER YDifferenceFromPrevious RGB32
00000022 2.79486251 RT_Debug: c in ScriptClip RGB32
00000023 2.79543447 RT_Debug: ScriptClip 1 ######## UVtoY: YUV data only!
00000024 2.79546285 RT_Debug: ([GScript], line 17)
00000025 2.81296062 RT_Debug: Last in ScriptClip BEFORE TRY YV24
00000026 2.81405020 RT_Debug: c in ScriptClip BEFORE TRY YV24
00000027 2.81406498 RT_Debug: Last in ScriptClip TRY YV24
00000028 2.81476545 RT_Debug: c in ScriptClip TRY YV24
00000029 2.86504054 RT_Debug: Last in ScriptClip TRY AFTER YDifferenceFromPrevious YV24
00000030 2.86558700 RT_Debug: c in ScriptClip TRY AFTER YDifferenceFromPrevious RGB32
00000031 2.86611915 RT_Debug: c in ScriptClip RGB32
00000032 2.86686778 RT_Debug: ScriptClip 1 ######## UVtoY: YUV data only!


I changed RT_LumaSceneChange to YDifferenceToPrevious to see if RT_LumaSceneChange were guilty party, problem persists.

EDIT: Simplified last part of script maintaining problem


ColorBars(pixel_type="RGB32")
c = Last.ConvertToYV24
ConvertToRGB()
GScriptClip(""" #"
c = last # Comment out removes problem, c is RGB32 ssS
return last
""") #"
ConvertToYV24()
RT_Debug("Last", Last.PixelType)
foobar()

return convertToRGB32()

[StainlessS]

Added Width, height test


Import("_AWB_130829.txt")

function foobar(clip c) {
info = true
RT_Debug("c", c.PixelType)
c
ConvertToYV24
GScriptClip("GScript("+chr(34)+chr(34)+chr(34)+""" # "
RT_Debug("Last in ScriptClip BEFORE TRY", Last.PixelType) ### ssS ADDED
c = last ### ssS ADDED
RT_Debug("c in ScriptClip BEFORE TRY", c.PixelType) ### ssS ADDED
RT_Debug("C.Width="+String(c.width),"C.Height="+String(c.height)) ### ssS ADDED, Width Height TEST
Try {
RT_Debug("Last in ScriptClip TRY", Last.PixelType) ### ssS ADDED
c = last
RT_Debug("c in ScriptClip TRY", c.PixelType) ### ssS ADDED
# fc = RT_LumaSceneChange(c, c, delta=-1) ### comment this line and it changes
# fc = RT_LumaMovement(c, c, delta=-1) ### comment this line and it changes
# fc = RT_AverageLuma(c, delta=-1) ### comment this line and it changes
fc = YDifferenceFromPrevious(c) ### comment this line and it changes
RT_Debug("Last in ScriptClip TRY AFTER YDifferenceFromPrevious", Last.PixelType) ### ssS ADDED
RT_Debug("c in ScriptClip TRY AFTER YDifferenceFromPrevious", c.PixelType) ### ssS ADDED
RT_Debug("C.Width="+String(c.width),"C.Height="+String(c.height))
c
RT_Debug("c in ScriptClip", c.PixelType)
cu = c.UToY

c
cAdjusted = last

return cAdjusted
} catch(err) {
RT_Debug("ScriptClip 1 ########", err)
}
"""+chr(34)+chr(34)+chr(34)+")", args="info") # "
}
#=====================================================

ColorBars(pixel_type="RGB32")
c = Last.ConvertToYV24
ConvertToRGB()
GScriptClip(""" #"
c = last.BilinearResize(width*2,height*2) # Comment out removes problem, c is RGB32, TEST Width Height ssS
return last
""") #"
RT_Debug("Last", Last.PixelType)
RT_Debug("c", c.PixelType)
RT_Debug("Calling FooBar()")
foobar()

return convertToRGB32()



00000005 1.48841429 RT_Debug: Last RGB32
00000006 1.48900259 RT_Debug: c YV24
00000007 1.48907018 RT_Debug: Calling FooBar()

00000008 1.49098611 RT_Debug: c RGB32
00000009 1.89604425 RT_Debug: Last in ScriptClip BEFORE TRY YV24
00000010 1.89666355 RT_Debug: c in ScriptClip BEFORE TRY YV24
00000011 1.89855063 RT_Debug: C.Width=640 C.Height=480
00000012 1.89933240 RT_Debug: Last in ScriptClip TRY YV24
00000013 1.89990711 RT_Debug: c in ScriptClip TRY YV24
00000014 1.90891588 RT_Debug: Last in ScriptClip TRY AFTER YDifferenceFromPrevious YV24
00000015 1.90944564 RT_Debug: c in ScriptClip TRY AFTER YDifferenceFromPrevious RGB32
00000016 1.91120744 RT_Debug: C.Width=1280 C.Height=960
00000017 1.91174400 RT_Debug: c in ScriptClip RGB32
00000018 1.91232562 RT_Debug: ScriptClip 1 ######## UVtoY: YUV data only!
00000019 1.91235268 RT_Debug: ([GScript], line 19)
00000020 1.94156682 RT_Debug: Last in ScriptClip BEFORE TRY YV24
00000021 1.94212198 RT_Debug: c in ScriptClip BEFORE TRY YV24
00000022 1.94385016 RT_Debug: C.Width=640 C.Height=480
00000023 1.94437897 RT_Debug: Last in ScriptClip TRY YV24
00000024 1.94490564 RT_Debug: c in ScriptClip TRY YV24
00000025 1.96107233 RT_Debug: Last in ScriptClip TRY AFTER YDifferenceFromPrevious YV24
00000026 1.96169102 RT_Debug: c in ScriptClip TRY AFTER YDifferenceFromPrevious RGB32
00000027 1.96342146 RT_Debug: C.Width=1280 C.Height=960
00000028 1.96394753 RT_Debug: c in ScriptClip RGB32
00000029 1.96451962 RT_Debug: ScriptClip 1 ######## UVtoY: YUV data only!


Seems to be using main level GScriptClip instance of clip c, with doubled width/height.

[Gavino]

Thanks for the info, Mr S.
I think I see what might be the problem.


EDIT: Simplified last part of script maintaining problem
...
GScriptClip(""" #"
c = last # Comment out removes problem, c is RGB32 ssS
return last
""") #"
...
Try this instead (also with 1st example):
GScriptClip(""" #"
c = last # Comment out removes problem, c is RGB32 ssS
return last
""", local=true) #"
Short of time right now. If it works, I'll explain why later (or maybe you can work it out yourself).

[StainlessS]

That works a treat big G. EDIT: Also with original snippit.

From Grunt docs

bool local - if true, the filter will evaluate its run-time script in a new variable scope, avoiding unintended sharing of
variables between run-time scripts.
Default is true if args is also specified, otherwise false (to preserve backwards compatibility).

A short example (based on the original) shows how this greatly simplifies passing function parameters into a run-time script.

function bracket_luma(clip c, float th1, float th2) {
Assert(0 <= th1 && th1 < th2 && th2 <= 255, "Invalid thresholds!")
ScriptClip(c, """
avl = AverageLuma()
avl <= th1 ? last.BlankClip() : avl >= th2 ? last.BlankClip(color=color_white) : last
""", args="th1,th2", local=true)
}

This is much easier than the standard approach of dynamically building the runtime script using string concatenation, or passing
the values via global variables.

"I really do not like this global variable business with ScriptClip ..." - stickboy

And because the run-time script is evaluated in its own scope, there is now no problem in calling bracket_luma
more than once in the same script (previously the variables th1 and th2 of different instances could interfere with each
other).


Was not aware of the local arg to Grunt, keep forgetting that Grunt is there. :)

[Gavino]

Does this now explain all martin53's problems, including the wierd changes to variable R?
I was initially puzzled by String(R) giving "202" (and similar values without decimal places) when isFloat(R) was true, until I remembered that ints also satisfy IsFloat(), so the output is consistent with R being changed to an int.
This could be explained if, further up the filter chain, there was another run-time script that used a variable called R and set it to an integer, where the corresponding (G)Scriptclip() call did not use args nor set local=true. (local defaults to true if args is used).

The reason it shows up after a run-time function call (eg AverageLuma) is that these functions will immediately fetch one or more frames from their input clip, which will cause frame requests to be passed up the filter chain. As a result, any run-time scripts further up the chain will be evaluated while still in the middle of evaluating the original run-time script (the one calling AverageLuma), potentially changing variables that it uses. Using local=true prevents this interference (when you use GRunT, of course :)).

[StainlessS]

This also seems to cure problem (last line of GScriptClip in foobar)


"""+chr(34)+chr(34)+chr(34)+")", args="info",after_frame=true) # "


EDIT: Or perhaps moves potential problem into main level GScriptClip()

[martin53]

, local=true
haven't got much time at the moment - this means I just used the plugin the wrong way ? If it's like that, so sorry to irritate you! Going to read the doc later this evening.

But wait - local=true is the default, if args is used...? With the script that initiated this thread, I use the args argument with both GScriptClip calls (unlike in the example given here), thus never gave a thought to local.

[Gavino]

This also seems to cure problem (last line of GScriptClip in foobar)

"""+chr(34)+chr(34)+chr(34)+")", args="info",after_frame=true) # "

Yes, that makes sense.
With after_frame=true, ScriptClip fetches a frame from its input before evaluating its own run-time script - any run-time scripts further up the chain will thus be evaluated at that point. Then, when its own script is evaluated, calls to AverageLuma (etc) will find their input in the cache and not call up the chain again, so you don't get the variables being changed in the middle.
(However, this wouldn't help with a more complex example where AverageLuma was called on a different clip, one not in the input chain of the ScriptClip.)
(EDIT - even on the simple example, it won't work if the function is YDifferenceToNext(), since that fetches a frame that won't be in the cache.)

haven't got much time at the moment - this means I just used the plugin the wrong way ? If it's like that, so sorry to irritate you!
No need to apologise, it's not a problem.
The run-time environment is very complex, especially when you have several run-time filters in the same script.

But wait - local=true is the default, if args is used...? With the script that initiated this thread, I use the args argument with both GScriptClip calls (unlike in the example given here), thus never gave a thought to local.
Yes, I saw that args was used in all the calls in your file _AWB_130829.txt, but that just defines the functions - what about the script that actually calls AWB()? Did it by any chance use ScriptClip on the input chain leading to the call (as in StainlessS's example)?
Can you post a complete script that reproduces the problem?

[martin53]

Can you post a complete script that reproduces the problem?
Yes, fortunately.
With both attachments, pls remove extension .txt

_AWB_130830_bug.avsi contains the called functions (complete, I hope)

_AWB_130830_bug.avs is the calling script.
It uses ColorBars as source. The bug appears directly when (re-)opening the AvsPmod F5 preview, and three times more when 'up' ore 'down' are pressed to move by 30 frames. It disappears as soon as one advances by 1 frame, or at the 4th time 'up'/down', reappears with F5.

See RT_Debug output
RT_Debug: R before AverageLuma 0.978490 isFloat= true isInt= false
RT_Debug: R after AverageLuma 180 isFloat= true isInt= true

AvsPmod and all other versions see 1st post, thank you for investigating.

EDIT: Adding local=true to the ScriptClip that
- uses R, G, B and
- has R=180 (I added a debut output after uploading)
- does not use args,
indeed cures the symptom.
I haven't got, however, why a missing 'local' in this place can affect the AWB script which uses local=true?

[Gavino]

I haven't got, however, why a missing 'local' in this place can affect the AWB script which uses local=true?
Still waiting for your attachments to be approved, so I haven't seen the details yet, but I imagine it is something similar to the example StainlessS posted. There, a run-time script with local=false gets evaluated in the middle of another one (for reasons explained in my earlier post). Because local=false, it does not start a new scope and so continues at the same scope as the other run-time script, allowing it to change its variables.

[Gavino]

Yes, indeed it is what I expected.
The (second) ScriptClip() inside AWB sets variable R, then calls cYUV.AverageLuma, requiring a frame to be fetched from cYUV.
Because cYUV is (indirectly) derived from the ScriptClip() in the top-level script, that ScriptClip's text is evaluated at this point. The outer ScriptClip runs at the same scope (and sees the same R) because it has local=false, so R gets set to 180 and the original value is lost.

I can't exactly see why it doesn't happen on every frame, but I expect it has to do with the order in which frames are fetched from the various clips in your function, and the operation of the Avisynth cache.

Setting local=true on the outer ScriptClip resolves the problem, since it will then run in its own scope with its own variable R.

For the benefit of anyone else who may come across this thread, please note that the problem of unintended sharing of variables between run-time scripts is not the fault of GRunT - it is actually a 'feature' of the standard Avisynth run-time filters, to which GRunT provides the solution (via local=true).

[StainlessS]

Are there any well known scripts that make good use of this 'feature' in Standard Avisynth Run-time to pass info between SciptClip instances ?

[martin53]

Are there any well known scripts that make good use of this 'feature' in Standard Avisynth Run-time to pass info between SciptClip instances ?
To my knowledge no.

But please remember that without GRunT, this 'feature' is also the only way to pass variables from outside RTE to a ScriptClip - besides nasty global vars. So Gavino's decision to make the 'local' default dependent on the use of 'args=' for compatibility deserves accordance.

[martin53]

I noticed that I made another mistake in calling RGBbalancedShift from a ScriptClip and not setting its 'RTE' parameter.

Do you know a better way to automatically determine if a script is running outside or inside Runtime Environment than...?
Try { current_frame RTE=true } catch (err_msg) { RTE=false }

[StainlessS]

Do you know a better way to automatically determine if a script is running outside or inside Runtime Environment than...?
Try { current_frame RTE=true } catch (err_msg) { RTE=false }

Not that I'm aware of. Note that script setting of current_frame will produce false +ve (some scripts do that).
Also NOTE, RT_VarExist("current_frame") should also work.

[Gavino]

Are there any well known scripts that make good use of this 'feature' in Standard Avisynth Run-time to pass info between SciptClip instances ?
I can't think of any 'well known' ones, but lots of published examples (in the forum and in the wiki) seem to do it, especially older ones - perhaps because people didn't realise at first that you could have multi-line ScriptClips.

It was certainly one of my prime objectives that GRunT should be 100% backwards compatible so that all existing scripts would continue to work.

Do you know a better way to automatically determine if a script is running outside or inside Runtime Environment than...?
Try { current_frame RTE=true } catch (err_msg) { RTE=false }
I don't think there is any better way - that is essentially the way the RTE itself does the check (except it does it in C++ rather than in script language, and also checks for it being an int).

Of course, it can be 'fooled' by setting current_frame manually in the script.

[Gavino]

Of course, it can be 'fooled' by setting current_frame manually in the script.
... or if WriteFileStart() or WriteFileEnd() have been used at a previous point in the script (discussion here (http://forum.doom9.org/showthread.php?p=1329748#post1329748)).

[martin53]

... or if WriteFileStart() or WriteFileEnd() have been used at a previous point in the script (discussion here (http://forum.doom9.org/showthread.php?p=1329748#post1329748)).

So,
Try { RTE=current_frame<0?false:true } catch (err_msg) { RTE=false }

[Gavino]

Yes, good idea.

BTW You can also write it simply as:
Try { RTE = (current_frame >= 0) } catch (err_msg) { RTE=false }
(Some people seem to prefer the longer form, but really it's redundant.)