Repairing bad ivtc foreground with progressive background

[kriNon]

I have a progressive DVD where every so often I will find a section where the foreground has been IVTC'd incorrectly and then put over a progressive background.

Is there any way I could repair the IVTC on the foreground while leaving the background still mostly unaffected?

I have found some success with using QTGMC(InputType=2), however this resulted in some issues including splodgy noise, haloing, detail loss, in other parts of the video.


EDIT: New sample: https://mega.nz/#!LQEWmIrS!dqF4mN8YBJh7eK-73y9V0icL-Yi_7cGgM1F0H3VjaQ8

[poisondeathray]

Did you try something simpler / better first ? Like custom IVTC settings ? Likely your PP settings are causing those issues, it's deinterlacing those frames when it shouldn't be

[kriNon]

Yeah, so I have tried using multiple different deinterlacing and IVTC tools and so far the only thing to have any real success is QTGMC. I have spent a lot of time tweaking QTGMC, but honestly I don't get much better of a result than just using QTGMC(InputType=2). Are you saying I should try a different SrchClipPP?

[poisondeathray]

Post a sample of original source

[kriNon]

the sample I posted was mostly unprocessed, it was just resized. Here is a completely unprocessed sample though: https://mega.nz/#!fB9ggQzK!K3wEkK--dmgePZPUgY2GWMwz02fRYBvItO-L5DtBJ2Q

[kriNon]

The sample I posted earlier was just resized with no further processing, here is a completely unprocessed sample though:
https://mega.nz/#!fB9ggQzK!K3wEkK--dmgePZPUgY2GWMwz02fRYBvItO-L5DtBJ2Q

[kriNon]

What I linked was pretty much the original source, but it had been resized. Here is a sample that hasn't been resized though in case that helps: https://mega.nz/#!fB9ggQzK!K3wEkK--dmgePZPUgY2GWMwz02fRYBvItO-L5DtBJ2Q

[kriNon]

Here's a copy of the original source: https://mega.nz/#!LQEWmIrS!dqF4mN8YBJh7eK-73y9V0icL-Yi_7cGgM1F0H3VjaQ8

I was thinking that it would be good to be able to detect which frames have issues, and then I could either edit them manually or use QTGMC on only these frames.

I was thinking one way of detecting these frames could be to QTGMC the entire source, and then look at the difference between the QTGMC'd video track and the source. If they are different enough I could potentially also check for aliasing to detect these frames more accurately.

Although I would like to hear if anyone has any better ideas about how to implement this?

[LemMotlow]

Here's a copy of the original source: https://mega.nz/#!LQEWmIrS!dqF4mN8YBJh7eK-73y9V0icL-Yi_7cGgM1F0H3VjaQ8Looks to me like your source is plain progressive frames encoded as interlaced, having been badly deinterlaced or inverse telecined earlier. The only thing you'll get by deinterlacing is duplicate frames.

[kriNon]

Looks to me like your source is plain progressive frames encoded as interlaced, having been badly deinterlaced or inverse telecined earlier. The only thing you'll get by deinterlacing is duplicate frames.

Yeah, that's exactly what has happened to the source. My main goal right now is to try to find a way of detecting these frames where it was badly deinterlaced/ivtc'd. From there I can either try to find a way of fixing it, possibly by interpolating the bad frames, manually editing the frames or finding a filter that works (possibly QTGMC) and only running it on frames where the issue is detected.

Also, although you wouldn't expect a deinterlacing to do anything to help with this issue, QTGMC actually works fairly well. The issue is that it causes some issues in other parts of the video, such as enhancing noise and creating haloing.

[kriNon]

Looks to me like your source is plain progressive frames encoded as interlaced, having been badly deinterlaced or inverse telecined earlier. The only thing you'll get by deinterlacing is duplicate frames.

Yeah, that's exactly what's happening with the source.

My current aim is to find a method of detecting which frames have this issue, after which I can either manually edit these frames, or filter just these frames.

Although unexpected, unlike most deinterlacers QTGMC actually works fairly well for fixing these frames. The issue is that QTGMC creates noise and introduces haloing. As such if I am to continue using QTGMC, I would like to limit it to frames that actually need it, hence the need for detecting which frames need to be processed.

Alternatively I could edit these frames manually once I have found all the frames that require editing.

[StainlessS]

ClipClop() can replace frames/ranges of a clip with same frames from one of up to 255 replacement clips.:- http://forum.doom9.org/showthread.php?t=162266&highlight=clipclop

An editing utility SawBones/FrameSurgeon, allows use of VirtualDub to view eg multiple clips in a 2x2 stacked window, and mark for replacement frames/ranges in src with same range from another clip, can also eg delete frames/ranges. Framesurgeon uses ClipClop and Prune plugins for replacement and deletion.

Sawbones/FrameSurgeon:- https://forum.doom9.org/showthread.php?t=173158&highlight=sawbones

Prune: https://forum.doom9.org/showthread.php?t=162446&highlight=prune

So, could do main processing now, and later replace dodgy frames when you figure out how.

If an auto detection script is written, can just create a frames file for use with ClipClop.

EDIT: EG,

1 100,-10 # replace 10 frames, 100 -> 109 with frames from replacement clip 1
1 100,109 # same
1 100 109 # same
1 100 # just frame 100, with rep clip 1

[kriNon]

I will definitely look into using ClipClop with QTGMC to replace any bad frames. Sawbones/FrameSurgeon look useful in a worst-case scenario, however I would much prefer not to have to look through 24 episodes manually looking for issues.

I would prefer to do my main processing later as I am still finalising some of my filtering and I'd like to get the encode all done at once.

Not sure how I should go about doing an auto-detection script though, if anyone has any ideas please let me know. I'm probably gonna give it a go tomorrow or the next day.

Ideally I'd like to find a faster/more efficient method to running the entire video through QTGMC and then looking for frames with a large enough difference to the original.

[StainlessS]

However you do it, your gonna wanna do the QTGMC thing first, as you will need access to that result to use any kind of auto
frames file generator (otherwise you would need to use something other than ClipClop to replace frames, it needs the frames file
before it starts.) You likely would also want to view frames to be replaced, unless QTGMC output already existed, its gonna take
an awful long time (probably multiple view times) for you to do anything.

EDIT: I thought you were in NTSC land (just looked at your sample).
That problem is quite horrible, and difficult to detect when its happening I would think.
QTGMC has not got much chance to improve that, was horrible to begin with (when playing in DGIndex).

d2v file

DGIndexProjectFile16
1
D:\New Folder\VTS_02_1.demuxed.m2v

Stream_Type=0
MPEG_Type=2
iDCT_Algorithm=6
YUVRGB_Scale=1
Luminance_Filter=0,0
Clipping=0,0,0,0
Aspect_Ratio=16:9
Picture_Size=720x576
Field_Operation=0
Frame_Rate=25000 (25/1)
Location=0,0,0,1368

900 5 0 0 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 538035 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 1077074 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 1615337 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 2154563 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 2693421 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 3232202 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 3772038 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 4310295 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 4848535 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 5386604 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 5924738 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 6462888 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 7000902 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 7538770 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 8075756 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 8612069 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 9150842 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 9688120 0 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 a2 ff

FINISHED 100.00% VIDEO


UpConv=1 made no difference.


VideoFileName ="D:\New Folder\out.d2v"
MPEG2Source(VideoFileName,UpConv=1) # Upconv to convert to YUY2, unmangle chroma if encoded as interlaced.
#TFM(d2v=VideoFileName) TDecimate(mode=(TFM_ANIME)?1:0)
Return Last


EDIT: And the log file (Shows Interlaced all the way through playing in DGIndex, up until last frame where displays Progressive).


Stream Type: Elementary
Profile: main@main
Frame Size: 720x576
Display Size: [not specified]
Aspect Ratio: 16:9 [3]
Frame Rate: 25.000000 fps
Video Type: PAL
Frame Type: Progressive # EDIT: I think this just shows whatever it showed in DGIndex on last frame
Coding Type: B
Colorimetry: BT.470-2 B,G*
Frame Structure: Frame
Field Order: Top
Coded Number: 227
Playback Number: 227
Frame Repeats: 0
Field Repeats: 0
VOB ID:
Cell ID:
Bitrate: 8.875 Mbps
Bitrate (Avg): 8.975 Mbps
Bitrate (Max): 9.079 Mbps
Timestamp:
Elapsed: 0:00:09
Remain: FINISH
FPS: 24.92
Info:

[kuchikirukia]

If QTGMC is giving you haloing try turning down the sharpness.

[kriNon]

So I believe I have found a way to detect the bad frames.

Using this script:

SetMemoryMax(2048)
video1=MPEG2Source("I:\[NHK]\Welcome to the NHK [DVDISO-PAL]\Disk1\episode1.d2v")
video2=video1.QTGMC(InputType=2)
video3=subtract(video1,video2)
video4=subtract(BlankClip(video3),BlankClip(video3))
compare(video4, video3)


compare will give an output at the top of the screen, and from what I have seen, if Mean Abs Dev is greater than about 0.97 then it will be a bad frame.

Does anyone know how I could get the value of Mean Abs Dev from the compare function, and then if it is greater than 0.97 output the frame number, ideally into a text file?

I guess I'll probably need to use a combination of ConditionalFilter and WriteFile? How would I go about getting an output from compare though? Any ideas?

[poisondeathray]

So I believe I have found a way to detect the bad frames.

Using this script:

SetMemoryMax(2048)
video1=MPEG2Source("I:\[NHK]\Welcome to the NHK [DVDISO-PAL]\Disk1\episode1.d2v")
video2=video1.QTGMC(InputType=2)
video3=subtract(video1,video2)
video4=subtract(BlankClip(video3),BlankClip(video3))
compare(video4, video3)


compare will give an output at the top of the screen, and from what I have seen, if Mean Abs Dev is greater than about 0.97 then it will be a bad frame.

Does anyone know how I could get the value of Mean Abs Dev from the compare function, and then if it is greater than 0.97 output the frame number, ideally into a text file?

I guess I'll probably need to use a combination of ConditionalFilter and WriteFile? How would I go about getting an output from compare though? Any ideas?


I don't think compare() (PSNR) has an equivalent internal runtime function . But you might be able to use something similar like lumadifference

This seems to work ok on your test clip with a threshold of 1.31 (use show=false to disable the debug overlay) . You might have to tweak the value for other sections


video1=MPEG2Source("VTS_02_1.demuxed.d2v", cpu=0)
video2=video1.QTGMC(InputType=2)
video3=subtract(video1,video2)
video4=subtract(BlankClip(video3),BlankClip(video3))

ConditionalFilter(video4, video2, video1, "LumaDifference(video4,video3)", ">", "1.31", show=true)

[kriNon]

Yeah, that works perfectly poisondeathray, thanks!

I'll have a look at figuring out how to output the frames to a text file using WriteFile tomorrow. Looks like detection works and is fairly consistent.

[poisondeathray]

Yeah, that works perfectly poisondeathray, thanks!

I'll have a look at figuring out how to output the frames to a text file using WriteFile tomorrow. Looks like detection works and is fairly consistent.

WriteFileIF() takes the same basic syntax


video1=MPEG2Source("VTS_02_1.demuxed.d2v", cpu=0)
video2=video1.QTGMC(InputType=2)
video3=subtract(video1,video2)
video4=subtract(BlankClip(video3),BlankClip(video3))



WriteFileIf(video4, "log.txt", "LumaDifference(video4,video3) > 1.31", "current_frame")


But the log will write the current frame number when value >1.31 (or whatever you specify as the condition) . When you have strings of consecutive frames detected e.g. 206,207,208,209, they will be on separate lines . So if you were to use the RX() interpolation function as discussed in PM, you would need to group them somehow

[StainlessS]

and difficult to detect when its happening I would think
Am delighted to be proved wrong :)

you would need to group them somehow

From FrameSel:- https://forum.doom9.org/showthread.php?t=167971&highlight=framesel


FrameSel_CmdReWrite(Clip c,String ofn, int F1, ... , int FN, String "SCmd"="", String "Cmd"="", \
Bool "Reject"=False, Bool "Ordered"=True, Bool "Debug"=False, Bool "Range"=True, Bool "Space"=False, Bool "Prune"=False)

Args identical to FrameSel (up to Debug), with exception of the additional compulsory 2nd String arg which is
an output filename, and missing Ver and Show and Extract args.
See FrameSel() description for other arguments.

ofn, String, Compulsory.
Output file name. It is advised to use a different name to the Cmd file, although there is nothing to stop you
overwriting an original Cmd file if you like a little gamble.(Output file only written after fully read in and
all Ordering, Rejecting etc already processed. Only likely error would be during file writing eg eject floppy disk).

Range, bool, Default=True
False = Write individual frame numbers only, to output file.
True = (Default) Write comma separated Frame Ranges to output file (only where adjacent frames).

Space, bool, Default=False
False = (Default) Write comma separators in output file for ranges.
True = Write SPACE separators in output file instead of comma for ranges.

Prune, bool, Default=False
False = (Default) Write FrameSel style frames or ranges to output file.
True = Write Prune style frames or ranges to output file (with a clip index first eg '0,10,20' instead of '10,20').
Prune supports up to 256 input clips that is the reason it needs a clip index.
Prune Supports Audio (whereas FrameSel does not) and also allows you to audio fadein/out at splices.
This function can only output Prune command files, it cannot read Prune command files.

FrameSel_CmdReWrite, takes either Frame numbers as direct arguments, or in the SCmd string, or in the Cmd file (at least one must
be specified). If more than one command method is used, will process Cmd file first, then SCmd string second and lastly
directly supplied frame numbers. The command frames will be acted upon via the Reject, and Ordered, args (as in FrameSel)
and then re-written to the output command file. Any comments in SCmd string or Cmd file will not be written to the output
command file.

The FrameSel_CmdReWrite() function returns the total number of frames written to the output file (including ranges).

This function could have several uses:-
To combine multiple frame specifying command methods into a single command file, and if Ordered, then remove duplicates.
Convert single frame command files to frame range command files, (smaller and perhaps easier to peruse).
Convert frame range command files to single frame command files, for use in another plugin that expects only frame numbers.
Convert a Reject=True command set to Reject=False command file, and vice versa.
Convert a command file of single frames/ranges to reject from a clip, into a Prune command file (Prune does not itself have a
Reject arg but it does fadein/out audio at splices and can also coalesce single frames into ranges before splicing so that
it does not fadein/out at every single individually specified frame).
Perhaps more uses.

*** NOTE ***, if you would normally use Reject=True in FrameSel, and you do a conversion with Reject=True, then the resultant
command file should be used with Reject=False in FrameSel.



Something like:- (In separate script after creating frames file).

c = Avisource(...) # Needs source clip, only for Reject mode really (so it knows how long the clip is and what to reject),
SPACE =True # Space separator, else Comma
FrameSel_CmdReWrite(c,"OutputRanges.txt",Cmd="InputFrames.txt",Range=True,Space=SPACE)
Return MessageClip("Done")


EDIT: I trust that below format will work with whatever the RX script is

10 14
210 215


or SPACE=False

10,14
210,215

[poisondeathray]

EDIT: I trust that below format will work with whatever the RX script is

10 14
210 215



Sorry I should have posted the details. The OP couldn't post for some reason early on and we had a discussion in PM

It's just an mvtools2 interpolation function, to interpolate over bad frames. It allows for runs of more than single frames, but the problem is it requires manual identification of frame ranges. If you could ID them "automatically" with some degree of accuracy it might be a good option. It's certainly "cleaner" than QTGMC in this case, and fixes the aliasing . But we all know that's only because this example scene was "simple" - mvtools2 interpolation can make a big mess of things on more complex scenes/motion

Here is the PM


In this zip file there are 2 versions, one done with avisynth "rx.mp4", and another with motion tracking guided replace "tracking.mp4" . No other filtering, just replacing
https://www.mediafire.com/?c53a9afmpticcl5


"rx" is a function I wrote a long time ago (with big help from gavino) for replacing frames using adjacent "good" frames . You write the frame number, and the number of frames to replace. For example (10,2) would replace frames 10,11 using frames 9 and 12 as guidance

If you look critically at the 2 examples, you will notice the avisynth version has a bit of slight "wobbling" or "warping" in the repaired frames . You might be able to fix by playing with blocksize or other mvtools2 settings.

The motion tracked repair is more "stable" , but motion tracked repairs are only suitable for some types of scenes. If there was other types of motion, more foreground object movement etc... you might have to resort to other types of approaches.

Global filtering (e.g. QTMGC) on all frames is a last resort IMO, because you damage all the frames. But on the other hand , it's "easier" and faster. Or another approach is you could replace select frames with QTMGC (or other filtered) . e.g. using clipclop or replaceframes functions


MPEG2Source("VTS_02_1.demuxed.d2v", cpu=0)
rx(175,3)
rx(187,1)
rx(196,1)
rx(198,3)
rx(203,1)
rx(206,4)
rx(211,2)
rx(215,1)
rx(218,1)
rx(221,1)
rx(223,1)


function RX(clip Source, int N, int X)
{
# N is number of the 1st frame in Source that needs replacing.
# X is total number of frames to replace
#e.g. RX(101, 5) would replace 101,102,103,104,105 , by using 100 and 106 as reference points for mflowfps interpolation

start=Source.trim(N-1,-1) #one good frame before, used for interpolation reference point
end=Source.trim(N+X,-1) #one good frame after, used for interpolation reference point

start+end
AssumeFPS(1) #temporarily FPS=1 to use mflowfps

super = MSuper()
backward_vec = MAnalyse(super, isb = true)
forward_vec = MAnalyse(super, isb = false)
MFlowFps(super, backward_vec, forward_vec, blend=false, num=X+1, den=1) #num=X+1
AssumeFPS(FrameRate(Source)) #return back to normal source framerate for joining
Trim(1, framecount-1) #trim ends, leaving replacement frames

Source.trim(0,-N) ++ last ++ Source.trim(N+X+1,0)
}

[StainlessS]

Thanx PDR, unfortunately, dont think that I'm in any fit state to do anything, at the mo.
Will come back again in the morn.

EDIT: Ive had a post from user about not being able to post, perhaps a site problem. (originally thought that I was blocking him).

[kriNon]

Hey, thought that I should post an update.

So far I've managed to run the script on the first four episodes, and it has been quite successful in finding any errors. From what I have seen the script does tend to find a lot of false positives, but it is also quite sensitive to actual errors, finding some errors so small that I couldn't even notice straight away. The script especially finds a lot of false positives when the video is either panning, shaking or fading between two clips as a transition. What I have been doing is running the script and then checking each frame output manually for errors, which has sped up this process significantly.

What I have noticed is that there are three main different types of error that are detected.
The first type being where in some frames, most visible near lines, it seems like two frames have been overlaid but with a very slight offset, resulting in double lines running parallel. This is most often not very noticeable by eye.
The second type typically occurs during motion, and is where to some limited extent, the frame has elements of the frame before/after it overlaid upon it with.
The third type is the type that is shown in the sample I posted, which results in aliasing.

It seems that each type of error would require some slightly different filtering, some individual frames might even need to be modified in an image editing program.

For now I am going to prioritise checking each episode for errors using the script, and then rechecking the list of errors manually.

I will probably end up using a mixture of QTGMC, RX, Image editing software, and possibly After Effects if I can figure out how to achieve results like what poisondeathray did.

[poisondeathray]

What I have noticed is that there are three main different types of error that are detected.
The first type being where in some frames, most visible near lines, it seems like two frames have been overlaid but with a very slight offset, resulting in double lines running parallel. This is most often not very noticeable by eye.
The second type typically occurs during motion, and is where to some limited extent, the frame has elements of the frame before/after it overlaid upon it with.
The third type is the type that is shown in the sample I posted, which results in aliasing.


Do you want to post examples of the other types ? It's not entirely clear to me but it almost sounds like you're referring to field blending ? There might be other approaches you could try

(I guess I should have mentioned this too)
In our PM's , I mentioned the NTSC version probably being better, but the OP apparently has examined all versions including broadcasted versions and apparently they all have even worse issues...

But I want to emphasize when you are combining techniques (including some manual work), you want to have access to all material that you can. It can save lots of time when you are compositing repairs. A botched part of frame on the PAL version might be able to be masked out with a NTSC version and so forth.

[kriNon]

So it turns out that I had made a mistake, and when I was looking through the list of errors I had been checking the frame numbers against the QTGMC'd track, instead of the source. Without having checked extensively, it looks like the first and second types of artifacting were caused by QTGMC and don't exist in the source. I will be rechecking errors using the proper source, and I'll post again either if I run into a problem, or when I have finished checking for errors. This might take a while.

Also, all of the issues that exist in the PAL version also exist in the NTSC version, however the NTSC version has further issues. Using the NTSC version for compositing repairs isn't an option.

[kriNon]

Alright, so I ran the script correctly this time, and after getting a list of frame numbers I then checked them manually, which resulted in me finding many damaged frames. However in the time that I was checking for errors, I found an example of an entire scene which was not detected by the script, nor was even really that close to being detected. From that I've realized that I'm going to need to find a better method of checking for errors.
Here's the scene where it wasn't detected: https://mega.nz/#!fQ8jRTKL!PR9dEmjECkpooRC8etDcbir6UY0UyPJg5hIasyQv2UI

Nonetheless I do suspect that we found the majority of the damaged frames. I have been looking into better detection methods, however I haven't been able to find anything that is able to detect the above scene. From what I have seen I believe that the damage occurs only in the luma component of the video.

[kriNon]

So I have spent some time trying to find another better method of detection and I believe I have found one, although I do still need to figure out some of the logic. What I found is that the previous method used relied heavily on Motion Analysis to work, which meant that in some scenarios where there is little to no motion, errors will not be detected.

After doing some testing using a variety of different detection methods, I have found one that I believe will work well. What I found is that it is possible to detect the bad frames using this script:

video1=MPEG2Source("I:\[NHK]\Welcome to the NHK [DVDISO-PAL]\Disk1\episode1.d2v", cpu=0)
track2=binarize(CombMask(video1)).greyscale().invert()

ConditionalFilter(video1, video1, video1, "AverageLuma(track2)", ">", "0", show=true)
return(last)

Whenever there is an error, the AverageLuma of track2 will fall significantly, relative to the surrounding frames.

As shown in this graph here:http://i.imgur.com/5K71kII.png

It is not possible to check if the AverageLuma of track2 is below a threshold because it changes significantly throughout the episode.
I am in the process of trying to figure out a way of separating these values which do not fit the trend of the graph.
I will also need to use some kind of scene detection to run this test on a per scene basis, because the graph I posted is discontinuous at each scene change.

EDIT:
I have found a way of detecting the outliers in the data using matlab which I will post here:
%mydata is the array of AverageLuma values
x=1:length(myData);
TF=smooth(x,myData,0.1,'rloess');

Test=(TF-myData)./TF;
for i=1:length(myData)
if Test(i)<0
Test(i)=-1;
elseif Test(i)
end
end

alpha=0.15;
for i=1:length(myData)
if Test(i)>alpha
disp(i);
disp(Test(i));
end
end

From here all I would need to do is implement scene detection.

I am thinking maybe it would be best to detect scene changes, output each frame number to a text document with its corresponding scene number and AverageLuma value, and then do all of the logic in matlab.

I'll post my results tomorrow, hopefully this works!