Simple 30->24 fps Srestore decimate not working

[johnmeyer]

Here's a sample clip from the football game I'm restoring:

test clip.mpg (http://www.mediafire.com/?2v3co32uylsqhq2)

It is NTSC video (29.97) from film. The pulldown is a simple frame blend (not field) every fifth frame. All that needs to be done is to detect this blended frame and decimate it.

I did this with several other films and everything worked great. However, I've tried every Srestore parameter and cannot get it to work. I've tried first correcting the contrast and then using that corrected clip, but I still get incorrect decimation. The problem is that the script is decimating more than every fifth frame.

Here's the script I'm using (using SRestore v2.7e):

loadPlugin("C:\Program Files\AviSynth 2.5\plugins\Srestore\mt_masktools-25.dll")
loadPlugin("c:\Program Files\AviSynth 2.5\plugins\MPEGDecoder.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\Srestore\srestore.avs")

source=mpeg2source("E:\Richards\1970 Stanford at Washington State\test clip.d2v")

output=srestore(source,frate=23.976, omode=1)

#stackhorizontal(source.SeparateFields(),output.SeparateFields())

return outputI've also tried using TFM/TDecimate, but that doesn't work.

BTW, this script is obviously NOT doing any decimation (omode=1), but instead lets me observe (using stackhorizontal) which frames will be decimated. If I can get it working, I'll change omode to "omode=6".

I'm using AVISynth 2.60 build Mar 9, 2013. Masktools 2.0.36.0. I've tried first converting the video to AVI in order to avoid any issues with caching when using DGIndex/MPEG2Source to read the video file (I've sometimes had caching issues).

Thanks in advance for any hints or help.

[johnmeyer]

Well, I spent another two hours on this, but can't get it to work. I am really puzzled because I've used SRestore many times before. I had one problem when I stupidly used SetMTMode(), but I'm not doing that here. I've tried dozens of different settings, but the blend detection seems to be random.

I'll keep trying ... hopefully someone will be able to offer some advice.

[martin53]

I guess you're not just asking for SelectEvery(5,1,2,3)? I cannot help you with Srestore, but offer a different solution and maybe initiate a discussion about timeline manipulation from AviSynth's runtime environment.

First, I looked into Srestore.avsi, but I'm too lazy and stupid to understand that at quick. Framerate reduction seems to be done in the 2nd last line 'oclp.trim(opos,0)', where opos includes a difference between curent_frame and the sum of removed frames at current_frame position. It is embedded in a ScriptClip() call.

Second, it is not difficult to determine a blended frame inside a ScriptClip(). You need RT_Stats (http://forum.doom9.org/showthread.php?p=1584313#post1584313) for the example script.
DirectShowSource("test clip.mpg")
ScriptClip(""" # "
c = last
var1 = RT_LumaCorrelation(c, c, delta=-3, delta2=-2)+RT_LumaCorrelation(c, c, delta=-2, delta2=-1)
var2 = RT_LumaCorrelation(c, c, delta=-2, delta2=-1)+RT_LumaCorrelation(c, c, delta=-1, delta2=+0)
var3 = RT_LumaCorrelation(c, c, delta=-1, delta2=+0)+RT_LumaCorrelation(c, c, delta=+0, delta2=+1)
var4 = RT_LumaCorrelation(c, c, delta=+0, delta2=+1)+RT_LumaCorrelation(c, c, delta=+1, delta2=+2)
var5 = RT_LumaCorrelation(c, c, delta=+1, delta2=+2)+RT_LumaCorrelation(c, c, delta=+2, delta2=+3)
RT_Debug(RT_String("%d %f %f %f %f %f", current_frame, var1, var2, var3, var4, var5))
var3 > max(var1,var2,var4,var5) ? c.subtitle("____to be removed") : c
""") #"

It evaluates 5 frames and assumes that the blended one is the one that has best correlation with its neighbors. Theoretically, more than one frame could share the same floating point correlation, then the script would show not enough frames "to be removed".

Third, I know of no way to permanently change the clip's timeline from a ScriptClip() (!)
Manipulation of the current_frame - no problem. But timeline manipulation needs to persist outside the runtime environment. Clip properties like framecount are determined at script start, and not modifiable during script run. The question is: How can the clip, say at position 123, know how many frames were dropped in the range 0...122?

I can imagine two solutions: recursion and restriction to linear editing. Recursion means, at position 9999 the script needs 9998 calls to the RTE, i.e. for each frame, the whole chain down to frame #0 is re-evaluated. Because that might be an issue with PC resources, restriction to linear editing might be favorable: The clip *must* be navigated from frame #0 towards the end. A global "frame offset" variable is used to shift the clip and this variable is incremented each time the ScriptClip() encounters a frame to drop.
Based on that, the following script should work,
global DroppedFrames = 0
DirectShowSource("test clip.mpg")
ShowFrameNumber()
c = last
ScriptClip(""" # "
var1 = RT_LumaCorrelation(c, c, delta=-3, delta2=-2, w=-62)+RT_LumaCorrelation(c, c, delta=-2, delta2=-1, w=-62)
var2 = RT_LumaCorrelation(c, c, delta=-2, delta2=-1, w=-62)+RT_LumaCorrelation(c, c, delta=-1, delta2=+0, w=-62)
var3 = RT_LumaCorrelation(c, c, delta=-1, delta2=+0, w=-62)+RT_LumaCorrelation(c, c, delta=+0, delta2=+1, w=-62)
var4 = RT_LumaCorrelation(c, c, delta=+0, delta2=+1, w=-62)+RT_LumaCorrelation(c, c, delta=+1, delta2=+2, w=-62)
var5 = RT_LumaCorrelation(c, c, delta=+1, delta2=+2, w=-62)+RT_LumaCorrelation(c, c, delta=+2, delta2=+3, w=-62)
RT_Debug(RT_String("%d %f %f %f %f %f", current_frame, var1, var2, var3, var4, var5))
#~ (var3 > max(var1,var2,var4,var5)) ? c.Subtitle("____to be removed") : c
global DroppedFrames = (var3 > max(var1,var2,var4,var5)) ? DroppedFrames+1 : DroppedFrames
Trim(DroppedFrames,0)
Subtitle(string(DroppedFrames))
""", after_frame=true) #"
... but it doesn't! At least on my PC, the result is catastrophic, the clip jumps back and forth!
EDIT: Is not an AviSynth issue, but my fault using DirectShowSource and right keyboard autorepeat. DirectShow is unreliable regarding timeline.

I tested positive, however, a ScriptClip variant that writes to a file while you move once through the clip. The file then contains the neccessary Trim() commands to do the actual frame dropping.
global DroppedFrames = 0
RT_TxtWriteFile("""DirectShowSource("test clip.mpg")""", "TrimList.avs")
RT_TxtWriteFile("ShowFrameNumber()", "TrimList.avs", append=true)
DirectShowSource("test clip.mpg")
ShowFrameNumber()
ScriptClip(""" # "
c = last
var1 = RT_LumaCorrelation(c, c, delta=-3, delta2=-2, w=-62)+RT_LumaCorrelation(c, c, delta=-2, delta2=-1, w=-62)
var2 = RT_LumaCorrelation(c, c, delta=-2, delta2=-1, w=-62)+RT_LumaCorrelation(c, c, delta=-1, delta2=+0, w=-62)
var3 = RT_LumaCorrelation(c, c, delta=-1, delta2=+0, w=-62)+RT_LumaCorrelation(c, c, delta=+0, delta2=+1, w=-62)
var4 = RT_LumaCorrelation(c, c, delta=+0, delta2=+1, w=-62)+RT_LumaCorrelation(c, c, delta=+1, delta2=+2, w=-62)
var5 = RT_LumaCorrelation(c, c, delta=+1, delta2=+2, w=-62)+RT_LumaCorrelation(c, c, delta=+2, delta2=+3, w=-62)
RT_Debug(RT_String("%d %f %f %f %f %f", current_frame, var1, var2, var3, var4, var5))
action = var3 > max(var1,var2,var4,var5) ? true : false
action ? RT_TxtWriteFile("Trim(0,"+string(current_frame-1-DroppedFrames)+")++Trim("+string(current_frame+1-DroppedFrames)+",0)", "TrimList.avs", append=true) : nop()
global DroppedFrames = action ? DroppedFrames+1 : DroppedFrames
last
""", after_frame=true) #"

Warning - no special handling for situations where frame #0 or the last frame are to be dropped is included. The frame counting arithmetic is incorrect in these cases.

[Gavino]

The clip *must* be navigated from frame #0 towards the end. A global "frame offset" variable is used to shift the clip and this variable is incremented each time the ScriptClip() encounters a frame to drop.
Based on that, the following script should work,
global DroppedFrames = 0
DirectShowSource("test clip.mpg")
ShowFrameNumber()
c = last
ScriptClip(""" # "
var1 = RT_LumaCorrelation(c, c, delta=-3, delta2=-2, w=-62)+RT_LumaCorrelation(c, c, delta=-2, delta2=-1, w=-62)
var2 = RT_LumaCorrelation(c, c, delta=-2, delta2=-1, w=-62)+RT_LumaCorrelation(c, c, delta=-1, delta2=+0, w=-62)
var3 = RT_LumaCorrelation(c, c, delta=-1, delta2=+0, w=-62)+RT_LumaCorrelation(c, c, delta=+0, delta2=+1, w=-62)
var4 = RT_LumaCorrelation(c, c, delta=+0, delta2=+1, w=-62)+RT_LumaCorrelation(c, c, delta=+1, delta2=+2, w=-62)
var5 = RT_LumaCorrelation(c, c, delta=+1, delta2=+2, w=-62)+RT_LumaCorrelation(c, c, delta=+2, delta2=+3, w=-62)
RT_Debug(RT_String("%d %f %f %f %f %f", current_frame, var1, var2, var3, var4, var5))
#~ (var3 > max(var1,var2,var4,var5)) ? c.Subtitle("____to be removed") : c
global DroppedFrames = (var3 > max(var1,var2,var4,var5)) ? DroppedFrames+1 : DroppedFrames
Trim(DroppedFrames,0)
Subtitle(string(DroppedFrames))
""", after_frame=true) #"
... but it doesn't! At least on my PC, the result is catastrophic, the clip jumps back and forth! I have no clue why.
You need to keep the frames being examined in step with the deletions. Insert, as the first line in the ScriptClip():
c = Trim(DroppedFrames, 0)

(BTW, DroppedFrames does not need to be global.)

For a similar problem, see here.

For another approach, doing deletion at compile-time, see this thread.

[StainlessS]

Might be of interest: http://forum.doom9.org/showthread.php?t=167971


FrameSel() is a simple plugin to select individual frames from a clip.

Can select frames numbers by direct arguments to filter, or in a string, or in a command file.

Video:- Planar, YUY2, RGB32, RGB24.

Audio:- Returns NO AUDIO (Does not really make sense for individual frames).

FrameSel(Clip, int F1, ... , int Fn, string 'scmd',string 'cmd', bool 'show', bool 'ver',bool "reject",bool "ordered",bool "debug")

Reject bool, Default False. If true then selects frames NOT selected rather than selected frames.
eg if you have a 5 frame clip (0->4) and have commands to select 4,2 and 0, then reject=true would actually
select frames 1 and 3 instead.
If reject=true, then orderering occurs no matter whether Ordered is set true or false and the frames are returned
in sequential order.
Reject might be of use to view all frames NOT in a frames command set.



Mpeg2Source("test clip.d2v")

Crop(100,24,-120,-24)

DROPLIST="Droplist.txt"
RT_FileDelete(DROPLIST)

PrevFrm = -2 # Init

ScriptClip(""" # "
c = last
corr1 = (PrevFrm==current_frame-1) ? corr2 : RT_LumaCorrelation(c, c, delta=-3, delta2=-2)
corr2 = (PrevFrm==current_frame-1) ? corr3 : RT_LumaCorrelation(c, c, delta=-2, delta2=-1)
corr3 = (PrevFrm==current_frame-1) ? corr4 : RT_LumaCorrelation(c, c, delta=-1, delta2= 0)
corr4 = (PrevFrm==current_frame-1) ? corr5 : RT_LumaCorrelation(c, c, delta= 0, delta2= 1)
corr5 = (PrevFrm==current_frame-1) ? corr6 : RT_LumaCorrelation(c, c, delta= 1, delta2= 2)
corr6 = RT_LumaCorrelation(c, c, delta= 2, delta2= 3)
v1 = (corr1 + corr2) / 2.0
v2 = (corr2 + corr3) / 2.0
v3 = (corr3 + corr4) / 2.0
v4 = (corr4 + corr5) / 2.0
v5 = (corr5 + corr6) / 2.0
# RT_Debug(RT_String("%d %6.4f %6.4f %6.4f %6.4f %6.4f", current_frame, v1, v2, v3, v4, v5))
action = v3 > max(v1,v2,v4,v5) ? true : false
action ? RT_TxtWriteFile(string(current_frame),DROPLIST, append=true) : nop()
action ? SubTitle("DROP",align=5,size=32) : NOP
RT_SubTitle("%6.4f %6.4f %6.4f %6.4f %6.4f %6.4f",corr1,corr2,corr3,corr4,corr5,corr6)
RT_SubTitle(" %6.4f %6.4f %6.4f %6.4f %6.4f",v1,v2,v3,v4,v5,y=20)
PrevFrm = current_frame
last
""", after_frame=true) #"
return last

EDITED:


Mpeg2Source("test clip.d2v")
DROPLIST="Droplist.txt"
FrameSel(cmd=DROPLIST,show=true,reject=true)
return last


Not perfect but may be better than source, perhaps some kind of Deshake beforehand might improve things.
EDIT: There are a number of consecutive blends in source, perhaps that is throwing srestore off.

[johnmeyer]

Thanks for all the input, but as I said in my initial post, my problem is not the decimation, but the detection. Srestore is sometimes replacing two or even three frames in a group of five, and is sometimes missing really obvious (visually) blended frames. Also, note that in the script I posted in #1 above, "omode=1". This does not perform any frame deletion. Decimation can always be done in a second pass, using a list created in pass one.

So my problem is understanding why this clip causes problems for SRestore, while similar clips (progressive, with one blended frame every fifth frame) from other sources work just fine.

I suppose I can try to find the several dozen places where the pulldown cadence changes, and then use a bunch of SelectEvery() statements, but that is going to take a long time.

I won't be at the main computer until Monday, but at that time I'll try some of the ideas posted and see if I can finally solve this problem. During the course of trying to make this work I did look at srestore.avsi, but decided not to modify it. Based on the posts here, I'm now going to look at doing just that, but the focus will not be on the decimation logic, but on the detection code. That is where the problem lies. For those who didn't follow the evolution of srestore, it initially didn't do any decimation at all, but instead was designed to simply replace the blended field/frame with a neighboring frame, with the understanding that you would use TDecimate (or your favorite decimator) to eliminate the resulting dups.

BTW, I have experimented using different versions of Masktools, and have found that I get different results with different versions. Therefore this variable is in the mix.

[edit]One thing I forgot to mention is that I have tried creating a DClip that has been stabilized with Depan and then cropped. I did this because I realized that there is a lot of jumpiness from film gate weave and that this motion may be causing the detection problems. So far, this hasn't improved the detection, but I'm still working on it.

[martin53]

You need to keep the frames being examined in step with the deletions. Insert, as the first line in the ScriptClip():
c = Trim(DroppedFrames, 0)

Thanks for your attention! (and johnmeyer, please excuse that I use your thread to reply here)

I pondered on that and tried it, but it worked no better. Will thoroughly read the links.
Found out that DirectShowSource when serving MPEG2 to an autorepeat forward in AvsPmod causes the jumps. Single frames good, so not AviSynth guilty, but DirectShowSource. Above example gives good output with good source filter.

BTW, DroppedFrames does not need to be global.
Indeed! I always programmed with the wrong concept on my mind that RTE variables lived only during one call, but the example proves you are right!
ColorBars()
ScriptClip(""" #"
try {
subtitle(s)
} catch(err) {}
s = "Here I am"
last
""") #"
ScriptClip(""" #"
try {
subtitle(s + " too", align=9)
} catch(err) {}
last
""") #"

[Gavino]

I always programmed with the wrong concept on my mind that RTE variables lived only during one call
The only time that happens is when you use local=true with GRunT.

[johnmeyer]

Even though some of the discussion has moved away from detecting blends, it has actually been quite useful because it has forced me to consider developing my own logic that might be better than SRestore, at least for this video.

Here is my idea.

The blended frame that happens every fifth frame is obviously nothing more than a blend of the two adjacent frames. I should therefore be able to construct this blend almost perfectly by simply doing a blend of the adjacent frames. Because the video underwent compression when it was encoded, my blend will not be a pixel-for-pixel exact duplicate, but it should be close.

Next, building on some ideas I learned from Didée, in this thread: Inverse of Decimate() (http://forum.doom9.org/showthread.php?p=1409324#post1409324) I should be able to create a second clip where each frame is a blend of its two neighbors. I will then interleave this with the original clip. If I then compare adjacent frames in this new stream, most frames should be quite different because in each comparison I'll be comparing a blend with an original frame. However, when the blend I create is compared to the blend created by the telecine, I should get an almost exact duplicate match which should be (relatively) easy to detect. I'll then use the tricks I learned from Didée in that thread linked to above to decimate all the frame I added, plus the frames marked as duplicates.

I'll report back.

[StainlessS]

What are you planning to do at eg frames 59/60 and 64/65 where you have double blends ?

EDIT: Both above mentioned blend pair frames are blends partially made from a frame for which there is no clean frame.

[martin53]

Thanks for all the input, but as I said in my initial post, my problem is not the decimation, but the detection.
Didn't intend to respond off topic, please accept my excuse if you feel I did.

Reviewing Srestore, I am sorry to say that although you value it, which I respect, I see it uses risky methods and might be more unreliable than you guess.
(1) The UPN expression in line 59 is buggy, evaluates only to "x 128 - 2 ^ x 128 - 2 ^ y 128 - 2 ^ + / 255 *", while "x y - == 128 ..." has no effect on the result. This might make bmask, and hence pmask, pp2 and pp3 invalid, if the original intention of the expression is no more fulfilled.
(2) Lines 155...173 and 207...210 use Select() statements inside the RTE. I found this to be unreliable in a non-reproduceable manner (one day it worked, the next day it wouldn't) and do not use it any more. Wrong selections might go unnoticed.
(3) Line 232 makes a timeline manipulation from RTE. - You did not use decimation with the example, I understand. But you still used replacement, i.e. either a Select() or the Trim() is definitely effective, and the above examples show that the results of timeline manipulation from RTE gan get seemingly unpredictable.
EDIT: Since I identified DirectShowSource (keep forgetting how careful it must be used), i'm wrong with (3).

Maybe an AviSynth version update you made has a side effect on (2) or (3), You might re-check with already successfully processed clips. When I had your clip open with AvsPmod today with the above timeline example, the result was completely unreasonable. And I experienced the same thing with (2) last autumn. Suddenly there was a moment when these Select() lines started to make problems, and I never identified the cause.
Finally, you use MaskTools 2.0.36.0 . You might consider to update to 2.0.alpha 48 although the changelog does not explicitly state a fix related to this thread.

[martin53]

Here is my idea.
The blended frame ... is a blend of the two adjacent frames. ... I should therefore be able to construct this blend almost perfectly by simply doing a blend of the adjacent frames.
Your idea is a good start. But sometimes the blend is 10%+90%, sometimes it's 90%+10%. Usually you will be unable to reproduce the blend in a way that allows a near 100% comparison. I experience this with the Super 8 clips from my family, where the beamer and the camera were synched, but not tight enough, so depending on the mechanical film transport resistance, the sync moves back and forth a bit.

Therefore, I used in the above example the next deduction from your idea: A blended frame should have less difference from its adjacent frames than a non blended one.
The RT_LumaCorrelation() function directly gives you a high result if two frames are similar. That's why I added RT_LumaCorrelation() between current and previous, and between current and next frame, to create the "blended" measures var1...var5. Typically, I assume there's no need for subtracting frames etc. since RT_LumaCorrelation() is here.
Before, a typical solution would have been 'AverageLuma(Overlay(mode="difference"))'

Probably, the above example is almost all you need (except for your mpeg2source ...)

[johnmeyer]

And what are you planning to do at eg frames 59/60 and 64/65 where you have double blends ?Since it's a football game, punt.

Seriously, part of the plan I'm evolving, which is based on that thread I linked to, is to decimate the worst (or best) "n out m" frames, similar to Cycle/CycleR in TDecimate. That would remove double blends. However, I'd then be left with a jump that I'd have to fill in with FillDrops() or something similar.

Another possibility is to just leave one of the blends. The main part of my restoration is to adjust the bad levels, but I was hoping to also use my version of VideoFred's scripts to further improve the film. That of course requires having a clean progressive source, but the occasional blended frame may not screw things up too badly.

[johnmeyer]

Your idea is a good start. But sometimes the blend is 10%+90%, sometimes it's 90%+10%. Oh, I didn't notice that. If true, my idea is DOA.

Lines 155...173 and 207...210 use Select() statements inside the RTE. I found this to be unreliable in a non-reproduceable manner (one day it worked, the next day it wouldn't) and do not use it any more. Wrong selections might go unnoticed."Unreliable in a non-reproduceable manner" exactly describes what is happening and why I'm so puzzled with this. You have done a good job of deconstructing SRestore and pointing out flaws. Despite these flaws, it has worked for me and for other people on many different clips. That is why it is so puzzling as to why it completely fails on this clip no matter what settings are used.

I do have one other way to proceed. Years ago I developed a process for decimating the pulldown frame that results from capturing film from a shutterless movie projector using 1/1000 of a second shutter speed on a video camera. I did this by capturing all the MIC parameters from TFM in pass 1; putting these into a spreadsheet where I developed logic that analyzes these parameters, looking both backwards and forwards, for several fields in each direction; and then I wrote Excel macros that exports parameters that are used by MultiDecimate and fed into TDecimate during pass 2. I've used this for years and it works flawlessly. I may be able to adapt that process to this situation.

[Gavino]

Lines 155...173 and 207...210 use Select() statements inside the RTE. I found this to be unreliable in a non-reproduceable manner (one day it worked, the next day it wouldn't) and do not use it any more. Wrong selections might go unnoticed.
I'd be interested to see if you can reproduce such problems.
Select() is such a simple function it's hard to imagine it going wrong, either inside or outside the RTE.

Regarding strange results with SRestore(), note that it cannot be called more than once in a script, and I figured it can go unpredictably wrong if you use certain variable names in your script. See this post.

[martin53]

@Gavino,
will be pleased to report, but it's improbable. I worked on that frame substitution script (http://forum.doom9.org/showthread.php?p=1600151#post1600151) when it happened. You can find essential infos in the changelog and the fSelectFromStr() function.
But it's usually not reasonable to post such a script the day you observe the problem. The 'bug environment' is just too confusing & complex.

@johnmeyer,
this time it seems it's not Select(). I can confirm that Srestore() replaces many frames with your settings, making the movie jumpy instead of deblending properly. I made an output file, then changed Srestore to avoid the Select() function (see attachment) and made another output file. Both are 100% identical. Changing the thresholds also hadn't helped btw.

[johnmeyer]

@johnmeyer,
this time it seems it's not Select(). I can confirm that Srestore() replaces many frames with your settings, making the movie jumpy instead of deblending properly. I made an output file, then changed Srestore to avoid the Select() function (see attachment) and made another output file. Both are 100% identical. Changing the thresholds also hadn't helped btw.Thank you very much for that. It is very useful to have my findings confirmed. What you report is exactly what I was finding.

I'll try a few more things tomorrow, but it looks like this particular decimation problem just can't be solved.

[martin53]

Investigation results:
When output frame is replaced, it is always replaced with previous frame. That comes from line #232, 'oclp.loop(1-opos,0,0)'. opos is -1 because of line #231, 'opos=opos+dup...', dup is -1 because of line #224 'dup= ... bbool==false ? 0 : ... om==1?-1:1', bbool is calculated in line #177 'bbool = .8*bc*cb>bb*cc && .8*bc*cn>bn*cc && bc*bc>cc'.
E.g. bc means blended clip current frame. So a comparison is made whether 0.8 times the product of current blended with previous clear clip frames is bigger than product of previous blended with current clear clip frames, etc.
Frame #6 is the first replaced frame. It has a 'blended' value of 5, while the surrounding frames have 0.125, and a 'clear' value of 12, the previous frames having 62 and the following also 12. The next replaced frame #8 again has a maximum of its surrounding 'blended' values which is 4 between 0.125 and 3, and the 'clear' value is a minimum of 4 following a 12 and next being 25.
The 'blended' values are 128 minus minimum luma of a metrics clip 'bclp', the 'clear' values are maximum luma minus 128 (lines #135ff). So, for the replaced frames, his clip has a high minimum luma and a low maximum luma, i.e. little contrast.
'bclp' is defined in line #47 'bclp = ... mt_lutxy(diff,diff.trim(1,0),expr=code0 ...)' - means, an expression 'code0' is calculated for current and next frame of clip 'diff'.
'diff' is a mt_makediff() of the input clip's current and next frame. It is reasonable that current and next frames are evaluated twice, because the results are not directly fed to the abovementioned 'bc' etc. variables, but to the variables for later frames, and then shifted through while current_frame advances.
So, 'code0' deserves some attention. It is a UPN string (line #44). "RT_Debug(mt_infix(code0))" helps to translate it: "((x-128)*(y-128)>0 ? (abs(x-128)<abs(y-128) ? (x-128)*(128-x) : (y-128)*(128-y)) : (x+y-256)*(x+y-256))*0.25 +128"
Interpretation: if diff and diff.Trim(1,0) are both brighter or darker than the average luma value of 128, then the negative square of the less contrastful pixel is used (the distance from 128 is squared). Otherwise, i.e. if diff and diff.Trim(1,0) have opposite distances from 128, then the square of the sum of their distances is calculated.
Because later, the maximum over the frame is used to determine the 'clarity', and the minimum to determine the 'blendedness', the blendedness comes from the pixel where both frames are bright or dark, and one is close to 128, while the clarity comes from the pixel where they are different in sign and amplitide.
Assumed diff were the clip, it would look similar to frame difference metrics I created over the months. But the input clip for the expression is not the clip itself, but already mt_makediff(). This makes for a second derivative over time which I can't explain.

To have a look at bclp, the return line from post #1 can be replaced with 'StackHorizontal(source,bclp.BicubicResize(source.width,source.height))'. I don't feel qualified to interpret it.

As a conclusion, I see no general defect in the code of the function, but some assumptions are made which may or may not apply to a certain footage.

P.S. for the investigations , I inserted 'ScriptClip("""last.subtitle(String(current_frame), size=9)""")' before the Srestore call to see the original frame number but minimize confusing the script with frame number text.

[johnmeyer]

Your description is very useful to me because there are a few code constructs in the SRestore.avs script with which I am not familiar and therefore I can not quite follow what the script is doing.

If I can understand it, perhaps I can patch it to work with this source.

[StainlessS]

M53, great respect that you decided to dissect srestore, I had a look and found it less than obvious to understand, and there is an awful lot of it.
Would have been nice if original authors had tried less to keep its operation a secret, did not however totally understand your description either,
but that is understandable, is a somewhat complex script. When you sort it out, can you please give a rendition with more full names (variables)
and any kind of additional comments on source would be greatly appreciated. If not, then what you have done already is a already a great help
to I'm guessing, a lot of people. Thanx very much.

[StainlessS]

John, still have not given up (totally) on this, but no real success at all as yet.
Have got to get rid of double blends first and and then singles, no way to detect doubles afterwards.
EDIT(or what is left of a double blend, afterwards).

[martin53]

John,
hope you don't mind if we use this thread further to design a solution for the original problem 'identify and remove blended IVTC-like frames', even if the solution does not mean enhancing or skillfully using Srestore.

If you concentrate your efforts on Srestore, please tell so, so we possibly start a new thread.

Some months ago, I figured out a concept for a measure of 'blendedness'.
The idea is: For three frames, the center frame will have a similarity with the preceding frame. Whatever this similarity is: How will it change when the comparison is not the preceding frame alone, but a blend with much of the preceding frame, and a bit of the following frame? With some real footage without blended frames, I measured the ratio of (Correlation with -1)+Correlation with +2)/(2*Correlation with (50% of -1 and 50% of +1))
c = last
average=0.0
c50 = merge(c.DuplicateFrame(0), c.DuplicateFrame(c.FrameCount()-1).DeleteFrame(0), weight=0.5)
c.ScriptClip(""" #"
c=last
average = average+ (RT_LumaCorrelation(c, c, delta2=-1)+RT_LumaCorrelation(c, c, delta2=1))/(RT_LumaCorrelation(c, c50, delta2=0)*2)
c.Subtitle(RT_string("%f", average/(current_frame+1)))
""", args="c50", local=false) #"
(needs GRunt plugin - and RT_Stats, of course)
as 0.96 . And btw, this is a measure for motion. Without motion, it is 1.0 . If the subtle advantage of the blended clip is discarded for the moment, then it can be said that for an unblended frame, the similarities will be about the same. But it is obvious, that a blended frame is more similar to the blend. As an improvement, the similarities with the preceding frame, and a blend containing only a little bit of the following frame, and the same to the opposite side, can be measured and both results used.
#=====================================================================================================================
function BlendMeasure(clip c) {
BLEND = 0.2
#clip 'previous frame' has 1st frame duplicated
cp = c.DuplicateFrame(0)
#clip 'next frame' has 1st frame dropped. To compensate for clip length, last frame is duplicated
cn = c.DuplicateFrame(c.FrameCount()-1).DeleteFrame(0)
cp98 = Overlay(cp, cn, opacity=BLEND, mode="lighten") #Merge(cp,cn,weight=BLEND)
cn98 = Overlay(cn, cp, opacity=BLEND, mode="lighten") #Merge(cn,cp,weight=BLEND)
cContainer = blankclip(c, pixel_type="RGB32", width=1, height=1)

cContainer.GScriptClip(""" #"
#1st check: Is central frame more similar to predecessor with a bit successor, or predecessor alone? Then it is assumed to be blended with successor
#(possible mistake: successor is very similar to central frame and predecessor very dissimilar)
#similarity is measured with RT_LumaCorrelation, although chroma is not taken into account. To take chroma into account, e.g. add 'Overlay(lumaClip, ChromaUClip)' and same with chromaVClip
sim1 = RT_LumaCorrelation(c, cp98) - RT_LumaCorrelation(c, c, delta2=-1)

#2nd check: Is central frame more similar to successor with a bit predecessor, or successor alone? Then it is assumed to be blended with predecessor
sim2 = RT_LumaCorrelation(c, cn98) - RT_LumaCorrelation(c, c, delta2=1)

#If both sim1 and sim2 are positive, then the central frame was more similar to both blended comparison frames: It is almost sure it is blended.
#If one of the sim clips is negative, then the central frame is presumably not blended; so a negative opposite sim is clipped to zero
sim1 = sim2 < 0 ? min(0.0, sim1) : sim1
sim2 = sim1 < 0 ? min(0.0, sim2) : sim2

#sim1+sim2 is now a measure for blend of the central frame. Positive number means it blended, negative means it is self-contained.
#The number is wrapped into a container clip, to take advantage of the AviSynth frame cache when frames need to be evaluated repeatedly.
BlankClip(last, color=RT_FloatAsRGBA(sim1+sim2))
""", args="c, cp98, cn98") #"
}
#=====================================================================================================================

... insert your source filter here or at top of script

c = last
cBlend = c.BlendMeasure()
c.ScriptClip(""" #"
c=last
RT_RGB32AsFloat(cBlend, delta=0) >= max(0, RT_RGB32AsFloat(cBlend, delta=-1), RT_RGB32AsFloat(cBlend, delta=1)) ? c.Subtitle("blended", align=5) : c
subtitle(string(RT_RGB32AsFloat(cBlend, delta=0)), align=5, y=240)
""", args="cBlend") #"

- Alas, the result is at best mediocre. I did already replace the merge() filter with Overlay(mode="lighten") because the blending of this clip emphasizes bright colors. But I notice that motion has an non-negligible effect on the correlation. Maybe the correlation between previous and following frame can be useful as an effective compensation value...
:)
Still, I hope I can stimulate your phantasy with this post.
I also have a suggestion for a frame reduction method based on the blend detection, but let's first make the blend detection reliable.

Because also Srestore is based on difference metrics, maybe the abrupt changes in motion (players, zoom, frame jumpiness) confuse it, too.

[johnmeyer]

hope you don't mind if we use this thread further to design a solution for the original problem 'identify and remove blended IVTC-like frames', even if the solution does not mean enhancing or skillfully using Srestore. If you concentrate your efforts on Srestore, please tell so, so we possibly start a new thread.I don't mind at all. My goal is to fix the video.

However, since SRestore does work extremely well on some clips, understanding how it works is probably an important step in constructing something better. You took a good first step in that direction in your earlier posts.

I have been sidetracked today transferring several dozen reels of Polavision film (Polaroid's instant film from the 1970s & 1980s). While this is completely OT, if you want to see how amazing AVISynth's filters can be, check out this before/after showing what can be done to remove the horrendous chemical pollution that almost completely ruins this film (the film is developed inside the cartridge, and has been sitting in this chemical filth for thirty years).

Polavision Before/After (http://www.youtube.com/watch?v=ud1XNIoh1P8)

[Didée]

Just chiming in to say: it's indeed a mystery that Srestore struggles s ohard with this clip, while it works so well almost everywhere else. Though, I never got the nerves to really analyse how-and-why Srestore is doing its analyses ... the programming logic simpy is too cryptic.

For your ideas of scripting new tools, I'm a bit alarmed when seeing things like "look for the most-something / the least-something in a group of X". Keep in mind - this may be okay for a e.g. 1-in-5 detection, but can easily fail for a general detection method. Eg. the usual PAL blended sources are basically clean-blend-c-b-c-b-etc., so you can't rely on "the majority".

Reflating the detection idea from Restore24 (if anyone remembers that one): R24 did perform edge detection on the frames and analysed the edgemasks. The principle is quite simple: a clean frame will have a lesser amount of more strong edges, and a blended fram will have a bigger amount of less strong edges. This detection worked pretty well. While R24 is so outdated you'll hardly get it to run on today's Avisynth, the detection logic is still prime. As long as there is anyting in motion, it never fails.

[martin53]

R24 analysed the edgemasks: a clean frame will have a lesser amount of more strong edges, and a blended fram will have a bigger amount of less strong edges.
The edges are basically made with mt_edge(), the weight for strong edges is made with gamma in Levels(). Please point out where the lesser amount of edges is checked, I find only AverageLuma() which in contrary favors a bigger amount of edges, instead a lower.

(1)
logic(tmpp.DEdgemask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5",setdivisor=true,divisor=6),
\ tmpp.DEdgemask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5",setdivisor=true,divisor=6),"max")
==> similar to mt_logic(mt_edge("1 0 -1 2 0 -2 1 0 -1", thY1=0, thY2=255), mt_edge("1 2 1 0 0 0 -1 -2 -1", thY1=0, thY2=255), "max", chroma="128") i guess?
(2)
levels(0,0.5,160,0,255,false)
(3)
YV12layer(edge_p2, edge_p1.invert(),"mul",chroma=false)
==> Overlay(edge_p2, edge_p1.invert(),mode="multiply")
(4)
AverageLuma(edge_n1_dark_n)
Maybe I missed something. Apart from that it looks familiar, I also like the gamma knack on the edges, and tests did not verify the luma correlation idea.
Today, in fact there is a plugin that could help with the amount of edges: RT_YInRange(). EDIT: or better RT_AverageLuma(clip, mask=clip, MaskMin=1, MaskMax=255)

As long as there is anyting in motion, it never fails.
With my own footage, I made a different experience. Scene change detection seems mandatory to me. Otherwise, the scene with many objects, though blended, is preferred against the frame with good quality but lack of contrastful objects. Same issue appears when the major detail leaves the scenery.

[martin53]

This script does a good job with blend detection of the test clip. :)
Autolevels() is compulsive. Without it, it will typically identify adjacent frames instead the really blended ones - don't know why yet.
And please note: not a single global variable, full nonlinear access.

#=====================================================================================================================
# The function returns 1 x1x RGB32 clip as container of a float number representing the luma edge contrast of the frame.
# The number is independent from number of edges, just contrast.
# Chroma is not used.
# Needs GRunt and RT_Stats v1.27 or higher plugins.
function EdgeMagnitude(clip c) {
#prepare a clip that yields the strong edges of the primary clip
c
mt_logic(mt_edge("1 0 -1 2 0 -2 1 0 -1", thY1=0, thY2=255), mt_edge("1 2 1 0 0 0 -1 -2 -1", thY1=0, thY2=255), "max", chroma="128")
cmask = last

#prepare the container 1x1 pixel clip
Blankclip(c, pixel_type="RGB32", width=600, height=100)

GScriptClip(""" # "
Try {
BlankClip(last, color=RT_FloatAsRGBA(RT_YPNorm(cmask, mask=cmask, p=2, MaskMin=2, MaskMax=255)))
} catch(err_msg) {
RT_Debug("EdgeMeasure:", err_msg)
}
""", args="cmask") #"
last
}
#=====================================================================================================================

mpeg2source("E:\Richards\1970 Stanford at Washington State\test clip.d2v")

AutoLevels(border_l=100, border_t=16, border_r=116, border_b=20)
c = last
cEdgeMagnitude = c.EdgeMagnitude()
c.GScriptClip(""" #"
c=last
RT_RGB32AsFloat(cEdgeMagnitude, delta=0) < min(
\RT_RGB32AsFloat(cEdgeMagnitude, delta=-2),
\RT_RGB32AsFloat(cEdgeMagnitude, delta=-1),
\RT_RGB32AsFloat(cEdgeMagnitude, delta=1),
\RT_RGB32AsFloat(cEdgeMagnitude, delta=2)
\) ? c.Subtitle("blended", align=8, y=16) : c
subtitle(string(RT_RGB32AsFloat(cEdgeMagnitude, delta=0)), align=8)
""", args="cEdgeMagnitude") #"

[johnmeyer]

All I get with this script, when applied to the test clip, is a subtitle overlay of "0.000000". I never see "blended."

I've checked my work several times. I'll keep trying ...

[edit] see my posts below ... I got it working.

[johnmeyer]

It looks like: GScriptClip(""" # "
Try {
BlankClip(last, color=RT_FloatAsRGBA(RT_YPNorm(cmask, mask=cmask, p=2, MaskMin=2, MaskMax=255)))
} catch(err_msg) {
RT_Debug("EdgeMeasure:", err_msg)
}
""", args="cmask") #"always returns zero. However, when I look at the mask itself, the blended frames are visually quite obvious.

[edit] see my posts below, I got it working.

[johnmeyer]

Needed later version of RT_Stats. Script now working.

The safest thing for me to do next is to export the raw numbers and then use them in the spreadsheet logic I referred to earlier to in my earlier post. I can use that to create a true 1 in 5 decimation pattern. The extra logic in the spreadsheet will make sure that I decimate one in every five (the script above sometimes goes 10+ frames without a blend being reported), and will also help pick the worse of the two blends when there is a double blend. The conditional logic required for this is too much work to write and debug within AVISynth.

Thank you very much for this. I can now finish my work.

[edit]Sorry for the edits, but I'm trying not to make too many posts in a row. Here are the frame numbers of the blends missed by the script:

4
185
392
395
465
485

In addition, due to not having scene detection, 187 and 394 are flagged as blended. I have to decide whether I want to take the extra step of adding scene detection.

[martin53]

Would a script that outputs the numbers of scene change frames support you, or maybe you already have this?

[johnmeyer]

Would a script that outputs the numbers of scene change frames support you, or maybe you already have this?Yes, I do have several scene change scripts. One uses depan; another uses MVTools2; another simply uses YDifference. They all work to some degree, although all are easily fooled by an object entering the scene close to the camera, and also by rapid camera panning (like you get when the camera pans during an American football kickoff).

So, since I already have some code that sort of works, I did think of adding scene change detection to this script. However, I then realized that the process which created the blends almost certainly did not look at scene changes. Therefore there is just as much likelihood that a blend can happen on the frames at the scene change boundary as on any other frame.

Put another way: scene changes and frame blends are mutually exclusive.

So, I don't think I can use a scene change flag. Instead, I think the key thing to making this work well relies on having some sort of "tendency" override. The idea behind this is that the blends should usually happen every five frames. If this was exactly true, and never changed, then we could simply use SelectEvery(5,...) and go home. However, because of edits, phase of the moon, etc., the pattern is sometimes interrupted and therefore we need SRestore or your script.

So my idea is this: once the detection has been done (i.e., your script as it is written above), then if there is a gap of ten frames or more without a blend, there needs to be a reexamination of the 2-3 frames in the area where a blend was expected. In a normal programming language this would be simple: set up two arrays, one filled with metrics from your script and one filled with an alternate detection process (e.g., SRestore). Then, when a potential failure was detected, switch to the other array of blend metrics and look at the frame where a blend was expected, plus or minus one frame. If one is found, add that to the list of blended frame numbers.

However, that is not easy in this environment, and that is why I am still planning to export your metrics (and possible metrics I create from SRestore) to an Excel spreadsheet and use that to create a decimation file that I then feed to TDecimate during a second pass. It is a little clunky, but I've done similar things before and not only does it work, but it is "obvious" in the sense that the decimation is easy to see and control in the spreadsheet.

[martin53]

Since you wrote you are going to export the frame numbers or the edge magnitudes to a spreadsheet, I didn't (don't) see reason in suggesting a loop script that produces exact 1-out-of-five blend hints, although it is not complicated with the GScript-enhanced AviSynth language.

But it suffers from the thing you observed, too: close to scene changes, a blend in the 1st scene might go undetected, a comparingly unblended frame of the 2nd scene spotted as the blended frame just because the 2nd scene has less edges ... etc.

My favorite solution would be to split the clip into scenes first, and then do an exact 1-out-of-five decimation to keep the original overall speed.

The combination to feed the hinting loop with scene changes, and only loop inside scene boundaries, is still a feasible task. Again, when the edge magnitudes are exported, the scene changes can also be exported and all calculation can be done with the spreadsheet.

Btw: the few frames you mention that in fact are blended but not signalled, are neighbors of even more blended frames or scene changes I think. This is what I saw at least. To make the clip free of blends they should be processed. But with dropping a gap in the timeline would become noticeable, I'm afraid? I thought it's better to drop 1/5 in a regular pattern and then maybe _replace_ instead of drop the few remaining bad frames.

[martin53]

John,
to compensate for being unable to help with Srestore() (http://forum.doom9.org/showthread.php?p=1650423#post1650423) on the test clip, I extended it to be multi-callable. Now you can use StackHorizontal(Srestore(parameters1), Srestore(parameters2)).

Hope you can save future time with this.

[johnmeyer]

to compensate for being unable to help with Srestore() (http://forum.doom9.org/showthread.php?p=1650423#post1650423) on the test clip, I extended it to be multi-callable. Now you can use StackHorizontal(Srestore(parameters1), Srestore(parameters2)).Thanks for that. I'm sure that will be useful in other, future projects.

Just a quick note on how this project is progressing. I had to manually edit the original video because there is a strong white flash each time the film camera was stopped and then restarted. This is a common problem for all amateur movie cameras. I therefore had to edit the video to remove these flashes. After I did this, I exported all the cut points which gave me a perfect scene change list.

Since this editing interrupted the blend cadence, I added some conditional logic in my spreadsheet which eliminates looking for minimums beyond the scene change, in either direction. I'm still scrubbing the timeline and making sure everything works before I write out the new file, but I'm pretty sure it is going to work just fine. I'll then have to decide if I want to make a second pass and attempt to replace the remaining blends (which were part of a double-blend) with motion estimated frames.

[johnmeyer]

I would appreciate a few hints on how to tune Martin53's excellent script because, unfortunately, it fails on other parts of the video, even though those parts look almost the same as the clip I submitted.

Here's an example of a second clip from the same source, showing where the detection fails:

Clip 2 (http://www.mediafire.com/?p5i49pb7s85dtqx)

Here is a list of frame numbers where the script makes the wrong identification. A frame number by itself means that there is a blend, but the script didn't identify it; if there is an "*" after the number, then that means the script made an identification of a blend that actually wasn't a blended frame.

11, 46, 48, 51, 56, 61, 63*, 69*, 71, 75*, 76, 78*, 91, 106, 108*, 181.

I only entered the frame numbers for the first scene. To make it easy to see what is happening, go directly to frame number 61 and then go frame-by-frame as the receiver goes in motion, starting at the twenty-yard number.

I tried adjusting the RT_YPNorm settings, because I can more easily understand what that is doing. However, I think the problem may be with the two mt_edge masks, and I am not exactly sure how to adjust those mask parameters in a way that would alter -- and hopefully improve -- the detection.

So, if anyone has any hints or ideas, I'd love to hear them. I actually did produce an entire decimated version of the video, and in some sections it is perfect, but in other sections, the detection breaks down and I end up with big jumps where good frames have been decimated.

Thanks!

[martin53]

1) The EdgeMagnitude() script I posted contains an exponent to weigh strong edges over those with small contrast. You can imagine it as contrast^exponent, and obviously this must be >1 to weigh 1 strong edge more than two weak ones from a blended frame.

You can freely play with this constant.
BlankClip(last, color=RT_FloatAsRGBA(RT_YPNorm(cmask, mask=cmask, p=2, MaskMin=2, MaskMax=255)))

I don't get the absolute same results as you at the moment, even with 2. For lack of time from my side, please check if this preprocessing filter for the edge gauge enhances the detection.
(use Median128.EdgeMagnitude)

What it does is: evaluate on frame by frame basis the luma median and then apply the levels filter with a gamma to bend the luma curve in order to get about a luma median of 128 over the frame.
AutoGain etc to similar things. The avantage of the Median128() filter is that you don't need any crop because it automatically ignores the black border and only takes reasonable lumas into account.

function Median128(clip c) {
# to bend luma to transform AverageLuma()->128, Levels()# gamma is: gamma = ln( AverageLuma() /255 ) / ln(128/255)
c.ScriptClip(""" #"
last
last
Levels(0, log(float(RT_YPlaneMedian(last, mask=last, MaskMin=17,MaskMax=234))/255)/log(128.0/255), 255 ,0 ,255 ,true)
""", local=true) #"
}

[johnmeyer]

You can freely play with this constant.BlankClip(last, color=RT_FloatAsRGBA(RT_YPNorm(cmask, mask=cmask, p=2, MaskMin=2, MaskMax=255)))For lack of time from my side, please check if this preprocessing filter for the edge gauge enhances the detection ... (use Median128.EdgeMagnitude)OK, I played around with your suggestions, and further deconstructed your script. I think the problem may be with RT_YPNorm.

Before I came to that conclusion, I did try using your Median128 function as a pre-process, but that did not seem to make any difference at all in the detection quality, and when I looked at the masks created, it seemed to degrade the non-blended mask so that the edges were less distinct. This might actually make the detection worse.

I then looked directly at the edge mask clip (edge= ...) and visually it appears to be doing exactly what it should, and the blends are quite apparent, as you can see from these pictures of the edge mask from two adjacent frames, one normal and the other blended (these were created using the original script posted a few days ago):

Normal
http://i177.photobucket.com/albums/w208/johnmeyer/Normal_zps0cd972be.jpg

Blended
http://i177.photobucket.com/albums/w208/johnmeyer/Blended_zps4f82daa9.jpg

As you can see, the edges in the second, blended frame are not as bright and are less distinct because the edge lines are slightly wider and "softer."

This made me focus on the line in your code which actually creates the metric that is used to determine which frame is blended:RT_YPNorm(cmask, mask=cmask, p=2, MaskMin=2, MaskMax=255)Changing the power (p) or any of the other scaling metrics really doesn't do much because all the adjacent vectors also get scaled in the same way, and therefore the local minimum (which is the blended frame) isn't going to change to a different frame.

I tried to understand what RT_YPNorm does and how it works, but when I tried to read about the "Minkowski distance," I intantly went beyond my math skills pay grade (I can handle most branches of calculus, but I never understood a single word of statistics). I did try, blindly, altering some of the YPNorm arguments, and this did actually change which frames got selected as blended, using your simple two-adjacent frame local minimum logic. This is what makes me think that I am on the right track in focusing on YPNorm.

So, in summary, I think your blend detection mask is working correctly, but the attempt to statistically evaluate the frame and return a single "blendedness" vector is where the approach fails on some frames. My sense of this is that this approach to detecting blended frames is very close to working and just needs one more tweak or idea to make it work quite well.

The parameters for YPNorm are given here: RT_Stats v1.24 (http://forum.doom9.org/showthread.php?p=1645670#post1645670)

[martin53]

Hi John,
makes me feel happy that you are working on the basics. Hope you don't mind I try to explain in words what is happening, and add some thoughts.

The first algorithm to process the script is a double mt_edge(). You saw the output, so we don't need to talk much about this. The one important thing is: edges with more contrast give brighter lines.

The RT_YPNorm() can be explained in simple words. Grace to StainlessS, the statistics functions like this one work on the frame, but also accept a mask clip, where the mask pixel's luma turns base pixel relevance on or off.
In this case, we give the clip itself as the mask and say 'MaskMin=2'. This means: only those pixels of the edges frame are statistically evaluated, which have a luma of at least 2.
This approach solves Didée's note, that a non blended frame has less edges. Because RT_MYPNorm() disregards all pixels that are not part of an edge, the amount of edges is eliminated from the result. :)
At the next step, what I wrote today was in fact wrong. If we just calculated the average luma of the relevant pixels, we'd already get a higher value for a non blended frame. It is unneccessary to use an exponent >1.
So the math behind this is just: (average luma of pixels with luma >=2)/(count of those pixels).
The Minkowski norm does no more than e.g. allow instead of adding up the plain values to calculate the average, squaring them in advance and later taking the square root of the sum - but with no restriction to the exponent 2, instead allowing 'any integer' exponent. That favors bright pixels and further increases the result when a frame has high-contrast edges.

The little script above also compares these edge magnitudes and decides a local minimum to be blended. I assume you removed this too simple decision making and transferred it to the spread sheet. :cool:

I think the remaining problems might come from several influences, besides of scene changes.
Remember Didée's note that this algorithm never fails. This is true if the scenery's edges are constant. But it can still fail if a) good edges enter or leave the frame, b) exposure varies, so the same objects have different-contrast edges, c) camera motion blur.
Solution approach for a): I know of no perfect solution. A mask could be overlayed that progressively dims the pixels close to the border. This would smoothen the effect.
Solution approach for b): AutoGain, Median128 ... are attempts. Any other flicker filter might be better here.
c: No remedy I can think of.

Additional problem sources for the given clip
d) If there is much small detail, maybe the mask approach to use the 2nd next row/column may be not as clever as it looks at first sight (though I marvel it).
Solution approach: The mt_edge commands could be changed for a test into
'mt_edge("1 -1 0 2 -2 0 1 -1 0", thY1=0, thY2=255), mt_edge("1 2 1 -1 -2 -1 0 0 0", thY1=0, thY2=255)', I do not expect much improvement.
e) The projection frame is part of the video frame. The edge between the content and the brownish frame area might confuse the algorithm.
Solution approach: The RT_YPNorm() filter supports x,y,w,h parameters:
'RT_YPNorm(cmask, mask=cmask, x=100, y=20, w=-120, h=-40, p=2, MaskMin=2, MaskMax=255)'
f) rolling shutter, i.e. no constant blend over whole frame, see frames 182, 183.

EDIT: I noticed RT_YPNorm() should better have a MaskMin=1.
Might try this script. Please change path strings if you don't have C:\TEMP. If you render the clip from start to end, you'll find the data for the spreadsheet in the file EdgeMagnitude.csv.

... your source filter
c = last
cmask = mt_logic(mt_edge("1 0 -1 2 0 -2 1 0 -1", thY1=0, thY2=255), mt_edge("1 2 1 0 0 0 -1 -2 -1", thY1=0, thY2=255), "max", chroma="128")
PATH = "C:\Temp\"
RT_TxtWriteFile("Frame; EdgeMagnitude", PATH+"EdgeMagnitude.csv", append=false)
c.GScriptClip(""" #"
c = last
edgeMag = RT_YPNorm(cmask, mask=cmask, x=100, y=40, w=-120, h=-40, p=2, MaskMin=1, MaskMax=255)
RT_TxtWriteFile(string(current_frame)+";"+string(edgeMag), PATH+"EdgeMagnitude.csv", append=true)
c
subtitle(string(edgeMag), align=8)
""", args="cmask, PATH") #"
StackHorizontal(last, cmask)

[johnmeyer]

Martin,

Thanks for that lengthy and very helpful post.

The Minkowski norm does no more than e.g. allow instead of adding up the plain values to calculate the average, squaring them in advance and later taking the square root of the sum - but with no restriction to the exponent 2, instead allowing 'any integer' exponent. That favors bright pixels and further increases the result when a frame has high-contrast edges.In looking at the edge-detected frames (see the two still photos in my previous post), it is clear that an effective statistical analysis must give a LOT of weight to pixel values very near 255. I tried to achieve this by increasing MaskMin so as not to include really dark pixels, and then I set mu to the same value as MaskMin. This seems like it should shift the knee of the exponential curve so the inflection point won't happen until the pixels get really bright. However, I didn't get better detection with this idea.

I assume you removed this too simple decision making and transferred it to the spread sheet.Yes, I did that. Using the two-pass method, with a spreadsheet in the middle, lets me create arbitrarily complex decision logic using the vectors created by the script. This lets me get around some strange problems, such as the case where the edge stats monotonically decrease over a range of a dozen frames or more. When this happens, there is no local minimum, and blended frames are not flagged. With my spreadsheet logic, I can look for a trend line, then remove that trend, and then look for local minimums.

However, the problem is not being able to intelligently parse the edge detection numbers, but instead it is the numbers themselves. Which brings me to ...
But it can still fail if a) good edges enter or leave the frame, b) exposure varies, so the same objects have different-contrast edges, c) camera motion blur.I don't think the failures I'm seeing in the second clip I uploaded yesterday (see my previous post) are caused by either of those problems. If you start with frame 61 (which should be flagged as a blend) and look at the next 50 frames, you'll see a lot of failures and I don't think the detection failure is caused by any of the three things you describe. Instead, I think the problem is not setting the parameters of RT_YPNorm correctly. That is what I'm continuing to work on.
I noticed RT_YPNorm() should better have a MaskMin=1I actually got better performance when I increased this parameter. If I set it higher, doesn't it exclude pixels that are darker than this value? It seems like this would be a good thing, since the valuable information for detecting blends is stored in the brighter pixels.
d) If there is much small detail, maybe the mask approach to use the 2nd next row/column may be not as clever as it looks at first sight (though I marvel it).
Solution approach: The mt_edge commands could be changed for a test intoI did try your alternative convolution kernels, but did not get results that were better. Your original two kernels, if I understand how they work, seem to produce good results. I think the first kernel creates edges looking in the horizontal direction and the second creates edges looking in the vertical direction, and then you create a result by taking the brighter pixel from each x,y location from each of the two mt_edge results.
The projection frame is part of the video frame. One thing I have done in all my work is to simply add a crop statement [crop(100,16,-110,-20)] in the first line of the script to get rid of the wasted space that resulted from encoding 4:3 material using a 16:9 format.

So, I'm learning a lot, but I still haven't improved the detection quality from what I got with your initial script.

[martin53]

I opended the result of the last script in a spreadsheet and did a local minimum decision there. The figures looked fine to me in general: There were more than one out of five frames marked to be a minimum, but a pattern with about every fifth line was noticeable. Please point out where you identify bad decisions, so I can concentrate on these frames.

I had thought that MaskMin=2 would remove noise from the edge clip, but I then learned that (probably due to the overexposure of the clip) the edges have small magnitude and results were much closer to truth with MaskMin=1.
It is possible to manually bias the kernel divisor, that might be a means against a potential rounding issue. Add ' 3' to the kernel strings. Lower values create saturation. E.g. at frame #7 of Clip2, MaskMin=1 and =2 both work. Mathematically, =1 is correct.

I don't encourage to set mu>MaskMin. For even exponents, it will spoil the result. To explain: (luma-mu)^2 for luma=1 and mu=2 is (1-2)^2 which is positive and will pretend a luma of 3 to the algorithm. All lumas below mu will then be resemble lumas above mu :(
mu=MaskMin is a good point! :cool:

As a summary so far, with kernel divisor 3, p=3, mu=1, MaskMin=1 e.g. frame #27 has a significant dip of 16.81 vs. 18.71 and 18.68 of the neighbor frames.

[johnmeyer]

It sounds like you are saying that, with the latest clip, you were able to correctly detect the frames in the range of 61 through 120, which is where most of the problems occur for me. If so, then I must be using different versions of the various dependent plugins. This is where I wish there were a tool in AVISynth that would report all DLLs loaded, and their version number. I try to explicitly load all my plugins, but sometimes there are plugins that get pulled into the script (from my default plugin directory) without my knowledge.

I'll do some more tests, and see if I can determine the exact version of the DLLs I'm using. Masktools and RTStats seem like the only two that would matter, but maybe there is something else that is causing the difference between the better results you are getting and the detection failures that I am seeing.

[johnmeyer]

It is possible to manually bias the kernel divisor, that might be a means against a potential rounding issue. Add ' 3' to the kernel strings.

As a summary so far, with kernel divisor 3, p=3, mu=1, MaskMin=1 e.g. frame #27 has a significant dip of 16.81 vs. 18.71 and 18.68 of the neighbor frames.I am puzzled and frustrated because I am not getting the same results as you. I've copied at the end of this post the script I am using. Virtually all of it is copy/pasted from what you've posted. You can see two lines that are commented out. These are the alternative settings that you've suggested. None of the permutations of these settings gives me numbers even remotely close to the numbers you are reporting.

With the script as shown, frame 26 shows a metric of 27.712540 and frame 27 shows a metric of 30.616247.

One more puzzling thing is that frame 26 is the one with the blend, and not 27 (which is what you are reporting). I am reading the frames from VirtualDub, which starts numbering at frame 0, not frame 1 so perhaps that explains the difference in numbering.

As far as scaling the convolution kernel, I must be misunderstanding what you are telling me to do. If I add "3" to each number in the convolution kernel, I get 96.262840 for frame 26 and 94.856087 for frame 27. As you can tell from these numbers, this gives me the wrong result because frame 26 (the one with the blend) is no longer smaller than its neighbor. I then tried adding three to all the kernel numbers, but left the central number at zero, since I think the idea is to replace that pixel with a weight of its neighbors. That is what I've shown below. That still didn't give me anything that matched your results.

I must be doing something really dumb here ...

I've shown the date/time of each of the plugins that I'm loading explicitly, but perhaps there is some other plugin that is being called that is different that what you are using.

[edit]I realized just after I posted that mt_edge lets you specify a power just by adding one more parameter to the string. So, this is what I should be using in order to multiply by three: mt_logic(mt_edge("1 0 -1 2 0 -2 1 0 -1 3", thY1=0, thY2=255), mt_edge("1 2 1 0 0 0 -1 -2 -1 3", thY1=0, thY2=255), "max", chroma="128")
Using this line in the script below still doesn't give me results that match your numbers, and it still fails on the frames I listed in my earlier post.

#This script looks for blended frames in a progressive source. It outputs numbers for each
#frame that show the "blendedness" of the frame compared to the four closest neighbors.
#It then exports the frame number along with the blendedness number. This will then
#be used by Excel spreadsheet logic to create a decimation file to be used by TDecimate.

#Original script by "martin53"

Loadplugin("C:\Program Files\AviSynth 2.5\plugins\GRunT.dll") #Sept. 27, 2008
Loadplugin("C:\Program Files\AviSynth 2.5\plugins\RT_Stats.dll") #October 17, 2013
Loadplugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools-25.dll") #December 31, 2010 version 2

filename = "e:\output.txt"
sep=", "

source = mpeg2source("E:\Richards\1970 Stanford at Washington State\Clip2.d2v")

HighContrast = AutoLevels(source,border_l=100, border_t=16, border_r=116, border_b=20)

cEdgeMagnitude = HighContrast.EdgeMagnitude()

# This puts the EdgeMagnitude number on the screen
HighContrast.GScriptClip(""" #"
c=last
RT_RGB32AsFloat(cEdgeMagnitude, delta=0) < min(
\RT_RGB32AsFloat(cEdgeMagnitude, delta=-2),
\RT_RGB32AsFloat(cEdgeMagnitude, delta=-1),
\RT_RGB32AsFloat(cEdgeMagnitude, delta=1),
\RT_RGB32AsFloat(cEdgeMagnitude, delta=2)
\) ? HighContrast.Subtitle("blended", align=8, y=16) : HighContrast
subtitle(string(RT_RGB32AsFloat(cEdgeMagnitude, delta=0)), align=8)
""", args="cEdgeMagnitude,HighContrast") #"

#WriteFile(HighContrast,filename,"current_frame", "sep", "String(RT_RGB32AsFloat(cEdgeMagnitude, delta=0))")




#=====================================================================================================================
# The function returns 1 x1x RGB32 clip as container of a float number representing the luma edge contrast of the frame.
# The number is independent from number of edges, just contrast.
# Chroma is not used.
# Needs GRunt and RT_Stats v1.27 or higher plugins.
function EdgeMagnitude(clip c) {
#prepare a clip that yields the strong edges of the primary clip
c
mt_logic(mt_edge("1 0 -1 2 0 -2 1 0 -1", thY1=0, thY2=255), mt_edge("1 2 1 0 0 0 -1 -2 -1", thY1=0, thY2=255), "max", chroma="128")

#The following alternative adds 3 to each kernel parameter (except the central number)
# mt_logic(mt_edge("4 3 2 5 0 1 4 3 2", thY1=0, thY2=255), mt_edge("4 5 4 3 0 3 2 1 2", thY1=0, thY2=255), "max", chroma="128")
cmask = last

#prepare the container 1x1 pixel clip
Blankclip(c, pixel_type="RGB32", width=600, height=100)

GScriptClip(""" # "
Try {
# BlankClip(last, color=RT_FloatAsRGBA(RT_YPNorm(cmask, mask=cmask, p=2, MaskMin=2, MaskMax=255)))
BlankClip(last, color=RT_FloatAsRGBA(RT_YPNorm(cmask, mask=cmask, x=100, y=40, w=-120, h=-40, p=3,mu=1, MaskMin=1, MaskMax=255)))

} catch(err_msg) {
RT_Debug("EdgeMeasure:", err_msg)
}
""", args="cmask") #"
last
}

[martin53]

John,
my output and EdgeMagnitude table here (http://www.mediafire.com/download/l525bpxgkhaq0te/Clip2.zip).

The frame offset was introduced when I converted the mpeg2 input to huffyuv, because I had troble with my mpeg2source. Sorry that confused you.
As for the different figures, I left out everything that is no more needed. This is the complete script I use now (cEdgeMask accelerated the availability of 5 values per frame with respect to recalculating them, but now only one value is used per frame).
PATH = "E:\Richards\1970 Stanford at Washington State\"
#~ PATH = ".\"
SOURCE = "Clip2.d2v"
CSV = "EdgeMagnitude.csv"
mpeg2source(PATH+SOURCE)
c = last
cmask = mt_logic(mt_edge("1 0 -1 2 0 -2 1 0 -1 3", thY1=0, thY2=255), mt_edge("1 2 1 0 0 0 -1 -2 -1 3", thY1=0, thY2=255), "max", chroma="128")
RT_TxtWriteFile("Frame; EdgeMagnitude", PATH+CSV, append=false)
c.GScriptClip(""" #"
c = last
edgeMag = RT_YPNorm(cmask, mask=cmask, x=100, y=40, w=-120, h=-40, p=3, mu=1, MaskMin=1, MaskMax=255)
RT_TxtWriteFile(string(current_frame)+";"+string(edgeMag), PATH+CSV, append=true)
c
subtitle(string(edgeMag, "%.2f"), align=8)
""", args="cmask, PATH, CSV") #"
StackHorizontal(last, cmask)

EDIT: I use AviSynth v2.60alpha5. mt_masktools26.dll is dated 12/31/2010, 966,656 bytes, alpha48 afaik, this might be essential. RT_Stats26.dll v1.27 is dated 20/17/2013, 307,200 bytes Never knew of a version with broken RT_YPNorm(). If it's there, it's OK. GRunT.dll 09/27/2008, 77,824 bytes. If GScriptClip() is there, it's probably OK.

EDIT2: From the former 16 frames, all frames in the pattern 'last letter is 1 or 6' have at least lower figures than their neighbors (leave decision to spreadsheet), all other frames don't, except for these 3:
#51 not detected but almost no motion to detect, unimportant for final quality
#71 not detected, reason is good global sharpness compared to previous frame, insufficient local motion to overcompensate that
#181 not detected, reason is even more blended neighbor 182

[johnmeyer]

OK, that puts us on exactly the same page. I compared my CSV file to yours, using your script, and it matches exactly. However, it also confirms the places in which the edge detection fails. Here is a partial list of "important" failures. When there is little or no motion, it is almost impossible to detect a blended frame (other than the faded dust spots), and failure to detect a blend correctly won't matter much, as long as I decimate approximately the correct number of frames. However, when a blend is missed during motion, then any failure becomes quite apparent because it introduces a jump in the action.

Here is the list of the "important" misses during motion in the first part of the clip:

71 - missed
86 - missed
88 - false local min
91 - missed
95 - false local min
96 - missed
109 - false local min
111 - missed

However, I think I can proceed with the script as written because I can create, in the spreadsheet, the equivalent of selectevery(5,0,1,2,3), which will then be overridden whenever the local minimum is unusually "obvious."

Thanks for all the help. Hopefully this will be useful to other people if SRestore doesn't work on their particular source.

BTW, I noticed that you didn't use Autolevels in your latest version of the script. It seems to work without it. Removing it does eliminate one possible problem, if the Autolevels algorithms actually introduce exposure variations.

[johnmeyer]

I thought it would be worthwhile to post one more time in order to provide a tidy ending to this very useful (to me, at least) thread.

First of all, I cannot thank Martin53 (and Stainless and Didee) enough for all the help you have given me on this project. I am happy to report that as of two minutes ago, everything is working perfectly. I used the script Martin53 provided earlier today and created the CSV file. I then imported that into a spreadsheet, along with my scene detect frame numbers. I then used some rather convoluted logic in the spreadsheet to locally look backwards and forwards at both the blend metrics and the scene change markers. Without going into all the detail, the spreadsheet looks for a really large negative value for the blend metric. This is then considered to be a blend, no matter what. The spreadsheet then performs the equivalent of a selectevery(5,0,1,2,3) so that every fifth frame after this "big blend" metric is decimated. This continues until the pattern is interrupted by either a scene change or another really large blend metric. The logic gets a little convoluted because of the various conditions under which the blend cadence can change.

The beauty of doing this in a spreadsheet is that I can instantly see the results of various logic changes, and I can simply go to the frame number in VirtualDub and make sure the decimation is going to work correctly. I then export the frame number and decimation (1/0) information in mfile.txt format, use MultiDecimate.exe to create the other files needed by Multidecimate.dll, and then run multidecimate in pass=2 mode to actually do the decimation.

It all works like a charm!

If I were doing this for a commercial release of this film, I'd next use a variation of the same script on the decimated file, looking for the blends that remain (which are from the double-blends). I'd then simply replace (no decimation) those blends with motion estimated frames, using MugFunky's old "filldrops()" function. Actually I may still do this, because it is a pretty simple step, and since I've been working with this, on and off, for almost three weeks, I might as well do it right.

FWIW, here is a link to a portion of the Excel spreadsheet I created. It is a real clunker, but it does the job:

Blended Frame Decimate.xls (https://dl.dropboxusercontent.com/u/1561578/Blended%20Frame%20Decimate.xls)