script to detect the left edge

[mathmax]

Hi

I would like to get the position of the left edge of my video. For that i wrote this script:

global thresh = 10 # luma threshold to determine edge

function FindEdge(clip c, int x) {
   return (x >= 30 || AverageLuma(c.Crop(x-2, 0, 2, -0)) > thresh) ? x : FindEdge(c, x+2)
}

source = mpeg2source("VTS_01_1.d2v")

bc = ScriptClip(source, """
BlankClip(length=3000, width=FindEdge(last, 10), height=200,fps=25, color=$ff0000)
""")

overlay(source, bc)

but I get an error of argument in AverageLuma()... I guess because the function is not directly in ScriptClip() but in a recursive sub-function.

How can I solve this problem?

[Gavino]

GRunT allows calls to AverageLuma, etc, from user functions called inside ScriptClip, etc.

But there is another problem with your script - the clip returned by ScriptClip must have the same properties (dimensions, etc) as the source clip. So you will need to move the Overlay inside the ScriptClip call.

ScriptClip(source, """
bc = BlankClip(length=3000, width=FindEdge(last, 10), height=200,fps=25, color=$ff0000)
overlay(bc)
""")

[mathmax]

works fine, thank you

now I have to make the value stable in time... ideally, I would like the average position of the edge by scene, else I'll create shaking if the detection is slightly different from a frame to another. How can I do that?

[Ghitulescu]

Maybe you'd also like to have a look on software-based TBC solutions, already discussed here.

[Gavino]

Originally Posted by mathmax

ideally, I would like the average position of the edge by scene, else I'll create shaking if the detection is slightly different from a frame to another. How can I do that?

Tricky - perhaps you could use a GScript loop to loop over the relevant frames and work out the average?
Do you know the frame ranges for each scene or are you expecting the script to figure that out for itself?

[mathmax]

Originally Posted by Ghitulescu

Maybe you'd also like to have a look on software-based TBC solutions, already discussed here.

no.. it's not TBC.. but if you have any TBC solutions I'm also interested for another video Where was it discussed?

Originally Posted by Gavino;

Tricky - perhaps you could use a GScript loop to loop over the relevant frames and work out the average?
Do you know the frame ranges for each scene or are you expecting the script to figure that out for itself?

I would rather do an average position of the edge, scene by scene. But I think this might be tricky cause the avisynth works frame by frame so it must be difficult to go back on preceding frames after the end of a scene to process the preceding frames... or maybe it's possible to make 2 passes?
And yes, I expect the script to figure out the frame range for each scene by itself, if possible

[jmac698]

Hmm, what if you average all frames in the scene? You will find the minimal left edge that way, and it will be stable.

[mathmax]

Originally Posted by jmac698

Hmm, what if you average all frames in the scene? You will find the minimal left edge that way, and it will be stable.

how would you do that?

[jmac698]

I found a way to directly find and add up every left edge position. You can use the average or minimum for each scene. This is in beta but I'll leave it out there for comments.
The demo creates two scenes with different average shifts. The script then adds up the edge positions in each frame, finally a crop routine crops off the average edge. It's slow to start.

#Find avg left edge of scene by jmac v0.5
#Requires MaskTools, works only with YV12
#Works correctly as intended, changes from Gavino
blankclip(pixel_type="YV12",color_yuv=$408080,length=100)
scene1=randomshift.shift(4,0)
scene2=randomshift.shift(8,0)
thresh=18
range=30
scene1+scene2
autocropleft(scene1,thresh,range)+autocropleft(scene2,thresh,range)

function autocropleft(clip v, int thresh, int range){
  x=minx(v,thresh,range,0,range)#the minimum
  #x=minx(v,thresh,range,0,0)/v.framecount#the average
  v.shift(-x,0)
  subtitle(string(x))
}

function findleftedge(clip v, int thresh, int range){
  #This finds the the first pixel>thresh
  #If the actual picture contains less than thresh, it won't find the edge
  src=v.crop(0,0,range,0)
  ramp=mt_lutspa(src,mode="absolute", expr="x")#increasing linear ramp
  mt_lutxy (src,ramp,yexpr="x "+string(thresh)+" > y "+string(range)+" ?")
  current_frame=0
  YPlaneMin
}

function avgx(clip v, int thresh, int range, int i,int x){
  x=x+findleftedge(v.trim(i,-1),thresh,range)
  i<v.framecount-1?minx(v,thresh,range,i+1,x):x
}

function minx(clip v, int thresh, int range, int i,int x){
  x=min(x,findleftedge(v.trim(i,-1),thresh,range))
  i<v.framecount-1?minx(v,thresh,range,i+1,x):x
}

function randomshift(clip v){
  ScriptClip(v,"""
    shift(rand(2)+rand(2)+rand(2),0)
  """)
}

function shift(clip v, int sx, int sy) {
  v#shift a video sx, sy pixels, sx<0 is left, sy<0 is down
  pointresize(width*2,height*2)#to allow 1 pixel shifts in YV12
  sx=sx*2
  sy=sy*2
  l=(sx<0)?-sx:0
  r=(sx<0)?0:sx
  t=(sy<0)?0:sy
  b=(sy<0)?-sy:0
  crop(l,t,-r,-b)
  addborders(r,b,l,t)
  bilinearresize(v.width,v.height)#puts some blur in it
}

[Gavino]

Your findleftedge function looks wrong to me. I think it should be:

function findleftedge(clip v, int thresh, int range){
    src=v.crop(0,0,range,0)
    ramp=mt_lutspa(src,mode="absolute", expr="x")#increasing linear ramp
    mt_lutxy (src,ramp,yexpr="x "+string(thresh)+" > y "+string(range)+" ?")
    current_frame=0
    YPlaneMin ()
}

In the minx function, the last line should be:

  i<v.framecount-1?minx(v,thresh,range,i+1,x):x

[jmac698]

G,
Caught the 2nd one. My leftedge worked but found the rightmost pixel<thresh, where yours finds the leftmost>thresh. Updated.

[Gavino]

Originally Posted by jmac698

function avgx(clip v, int thresh, int range, int i,int x){
  current_frame=i
  x=x+findleftedge(v.trim(i,i),thresh,range)
  i<v.framecount-1?minx(v,thresh,range,i+1,x):x
}

Why set current_frame here?
It has no effect, since it is always set to 0 in findleftedge (and in any case, it is a local variable).

Also, note for future reference that trim(i,i) will select the entire clip rather than a single frame when i=0. You get away with it here (and in minx) because findleftedge only uses the first frame, but it's best to use trim(i,-1) when selecting a single frame to avoid potential problems.

[jmac698]

Oh, I'm sure that's just leftover stuff. It seems using current_frame=i could save me from trimming.

[Gavino]

Originally Posted by jmac698

It seems using current_frame=i could save me from trimming.

Yes, instead of passing v.trim(i,i), you could pass v and i as separate parameters and set current_frame=i in findleftedge.
But setting it in the calling function (avgx or minx) won't work unless you make it global.

[mathmax]

thank you so much for working on that script..
I just don't understand well how the function findleftedge() works? Especially the functions mt_lutspa(), mt_lutxy() and YPlaneMin() are not well documented..
Could you describe what it does?

Thak you

[jmac698]

function findleftedge(clip v, int thresh, int range){
  #This finds the the first pixel>thresh
  #If the actual picture contains less than thresh, it won't find the edge
  src=v.crop(0,0,range,0)
  ramp=mt_lutspa(src,mode="absolute", expr="x")#increasing linear ramp
  mt_lutxy (src,ramp,yexpr="x "+string(thresh)+" > y "+string(range)+" ?")
  current_frame=0
  YPlaneMin
}

src is set to crop the video to the first Range pixels. Hopefully this is faster but also, the edge finding technique is limited to about 256 pixels anyhow.
Ramp is a video with increasing brightness pixels. The picture looks like this: 0,1,2,3... 255
mt_lutxy: This does a calculation on two videos, we use x to refer to the first one (src) so x=src, y=ramp. So we calculate "x thresh > y range ?" where thresh and range are substituted with the actual numbers, using string(thresh) etc.
This weird notation is called Polish, it's like it's written backwards. Here's normal (infix) 2+3 and here's Polish 2 3 +. So we are using the operation ? which takes 5 parameters, two values to compare, the type of comparison, what to return if it's true, what to return if it's false.
So we are saying, if x>thresh then return y, otherwise return range.
In the big picture what this does is test each pixel in src against thresh. Since the left edge is going to be black, this is not true at first so we have the left edge all set to Range (30). Finally we find the picture and set the picture to the same pixel in y, the Ramp. Let's say we're at pixel 10. Then ramp is 10 at this point, so actually Ramp happens to contain the pixel x-coordinate. Now our picture is like this:
30 30 30 30 30 30 30 30 30 10 11 12 13 14....
So now we return the minimum pixel value in the whole picture, which is 10, so we found the edge. Note that this works for every line at once so we find the left most pixel > thresh for the whole height.
There is a problem, what if the picture contains an 8 somewhere? Then we get 8 as the answer which is meaningless, because that's not the edge at all. Really it's only going to work for edges<16 because a picture usually has pixels starting at brightness 16.
The use of YPlaneMin is not really documented, this is a runtime function that usually only works in ScriptClip, and it's updated on every frame, but due to implementation, you can read this value exactly once in your script at the position current_frame. I use the trick, it's actually reading once each frame in the scene and compiling all the information before the video ever starts playing, which is why it's so slow to start up. The answers are already known before you play it.
Lutspa is meant just to construct an image through calculations of the x/y coordinates. "x+y" would make a diagonal ramp.

[Gavino]

Originally Posted by jmac698

So we are using the operation ? which takes 5 parameters, two values to compare, the type of comparison, what to return if it's true, what to return if it's false.

Strictly speaking, it takes 3 parameters: a boolean value and the true and false return values. The comparison (here two values to compare and the type of comparison) is a subexpression in its own right and could be anything (of arbitrary complexity) that delivers a boolean result.

The use of YPlaneMin is not really documented, this is a runtime function that usually only works in ScriptClip, and it's updated on every frame, but due to implementation, you can read this value exactly once in your script at the position current_frame.

See here for the documentation of YPlaneMin.
You're not limited to reading it just once in the script when using the 'current_frame' trick- I think what you mean is that each use gets the value (at compile-time) for a single frame (identified by the value of current_frame at the time).

[mathmax]

Originally Posted by jmac698

function findleftedge(clip v, int thresh, int range){
  #This finds the the first pixel>thresh
  #If the actual picture contains less than thresh, it won't find the edge
  src=v.crop(0,0,range,0)
  ramp=mt_lutspa(src,mode="absolute", expr="x")#increasing linear ramp
  mt_lutxy (src,ramp,yexpr="x "+string(thresh)+" > y "+string(range)+" ?")
  current_frame=0
  YPlaneMin
}

src is set to crop the video to the first Range pixels. Hopefully this is faster but also, the edge finding technique is limited to about 256 pixels anyhow.
Ramp is a video with increasing brightness pixels. The picture looks like this: 0,1,2,3... 255
mt_lutxy: This does a calculation on two videos, we use x to refer to the first one (src) so x=src, y=ramp. So we calculate "x thresh > y range ?" where thresh and range are substituted with the actual numbers, using string(thresh) etc.
This weird notation is called Polish, it's like it's written backwards. Here's normal (infix) 2+3 and here's Polish 2 3 +. So we are using the operation ? which takes 5 parameters, two values to compare, the type of comparison, what to return if it's true, what to return if it's false.
So we are saying, if x>thresh then return y, otherwise return range.
In the big picture what this does is test each pixel in src against thresh. Since the left edge is going to be black, this is not true at first so we have the left edge all set to Range (30). Finally we find the picture and set the picture to the same pixel in y, the Ramp. Let's say we're at pixel 10. Then ramp is 10 at this point, so actually Ramp happens to contain the pixel x-coordinate. Now our picture is like this:
30 30 30 30 30 30 30 30 30 10 11 12 13 14....
So now we return the minimum pixel value in the whole picture, which is 10, so we found the edge. Note that this works for every line at once so we find the left most pixel > thresh for the whole height.
There is a problem, what if the picture contains an 8 somewhere? Then we get 8 as the answer which is meaningless, because that's not the edge at all. Really it's only going to work for edges<16 because a picture usually has pixels starting at brightness 16.
The use of YPlaneMin is not really documented, this is a runtime function that usually only works in ScriptClip, and it's updated on every frame, but due to implementation, you can read this value exactly once in your script at the position current_frame. I use the trick, it's actually reading once each frame in the scene and compiling all the information before the video ever starts playing, which is why it's so slow to start up. The answers are already known before you play it.
Lutspa is meant just to construct an image through calculations of the x/y coordinates. "x+y" would make a diagonal ramp.

thank you for this explanation I understand everything except the "current_frame" trick with the YPlaneMin() function... :-/

[Gavino]

Originally Posted by mathmax

I understand everything except the "current_frame" trick with the YPlaneMin() function... :-/

This is basically a 'cheat' I worked out based on the internal workings of the run-time environment. Filters like ScriptClip set current_frame for each frame rendered at run-time and the value is used by functions like AverageLuma and YPlaneMin to know which frame to operate on. By setting current_frame manually, these functions (intended to be used only at run-time) can be made to return the value for a specific frame at compile-time instead of giving an error.

Strictly, this is cheating as it relies on the internal workings of Avisynth which could in principle be changed in the future.