Noob question: What massage would allow shift vs brightness comparison?

[MrPete]

I'm not exactly sure how to even describe this. Below are three frame-extracts. They are not going to line up perfectly (partly because they are actually manual screen grabs as should be obvious).

The actual difference:
Cap 1: Baseline
Cap 2: Cap 1 but with pulldown; also *slightly* distorted
Cap 3: Next frame, clearly shifted a little everywhere

What I want: a way to transmogrify the data so I can compare 1-2 and see that they are "more similar" than 1-3, because 1-2 has less shift.

I *think* that means I want to totally discount the fact that #2 happens to be: blurrier, lighter, etc.

So... what kind of transformation on the data could help accomplish this? AND, how do you think about such a question?

I'm very much in the early stages of learning how to think about these things properly, so probably as much as if not more than "the answer", I am very interested in how you would approach such a question!

Thanks sooo much, and Happy New Year!
pete

Cap1: http://www.ds.org/files/avisynth/cap1.jpg
Cap2: http://www.ds.org/files/avisynth/cap2.jpg
Cap3: http://www.ds.org/files/avisynth/cap3-11372.jpg

[johnmeyer]

If these are frames in a video, and if your main objective is to make the frames align, using the sprocket holes as alignment guides, you could use Deshaker inside of Virtualdub, and restrict deshaker so it only looks at pixels from the left side of the frame. It will then adjust all frames so the sprocket holes are in exactly the same X-Y location, from frame-to-frame.

[MrPete]

I do use Deshaker, but that comes after I have rejected the frames that I don't want.

My goal for this phase is to discover which sets of frames correspond to a single film frame.

In my example, the first two captures are for one film frame, while the third capture is for the next frame.

I've tried pixel-difference counting (RT_LumaPixelsDifferent) and luma-difference (RT_YDifference). Neither one provides useful results... because the luma difference for the first two frames is so large.

Thinking "out loud"... I've seen edge-detection algorithms that reduce a photo to a sharpened outline of edges. Is it possible that this can be done in AVIsynth? If so, perhaps that would help me deted edge-motion...

[johnmeyer]

I designed a film to transfer system ten years ago. I removed the shutter from the projector, run it at full speed, and point the camera directly into the projector lens at point-blank range. I then use software to detect and remove fields that either contain pulldown (where the film is moving). These fields do not match either adjacent field. Once that is done, I then decimate any fields that are duplicates.

I would suggest that you look at TIVTC which includes the field matching software (TFM) and the field decimation software (TDecimate). You may also find a use for SRestore, even though it was built for another purpose.

[MrPete]

Thanks, John. I'll take a look! You're correct: it's the recognition of pulldown fields that is challenging me right now.

[MrPete]

One tiny problem: TFM appears to strongly assume interlaced video. Mine is not. It is NEVER combed.

Do you suspect it might be able to field-match even with no combs at all?

[johnmeyer]

Non-interlaced (progressive) video is the same as interlaced video, but with no temporal difference between "fields" (although the field concept obviously doesn't apply). Most plugins/filters designed for interlaced will work for progressive. The reverse, of course, is not true at all. "Combs" are actually a residual artifact in some telecine, but not all. They are not part of the actual matching and decimation process, but instead are usually handled by post processing (see the "pp" parameter in TDecimate).

[MrPete]

That's all true, John.

However (based on code examination and now completing a few experiments), the TFM() routine depends entirely on discovering combing to detect which fields/frames to be removed :(

It's in the code, it's in the intermediate file (if two-pass is used)...

My result: same input and output every time :(

This is essentially what I saw with the RestoreSuper8Frames() routine. It too is searching for combing to discover frames to be removed.

That's why I have been going down this alternate path...

[johnmeyer]

Why don't you post some sample video so we can actually see what's going on?

[MrPete]

Sure! Here's some fresh samples online:

Here is the shortest subclip -- just 26 frames. (http://www.ds.org/files/avisynth/capsample26.avi)
Same clip, ~100 frames (80MB) (http://www.ds.org/files/avisynth/capsample100.avi)
More. Over 200 frames (223MB) (http://www.ds.org/files/avisynth/capsample1.avi)

These are YV12, encoded using Lagarith lossless to avoid losing anything... http://lags.leetcode.net/codec.html

[johnmeyer]

Sorry, I can't play the video. Someone else will have to help.

[feisty2]

Sorry, I can't play the video. Someone else will have to help.

I loaded the video by lwlibavvideosource in avisynth and it returns a 1080p yv12 clip with no error

[MrPete]

I use lagarith lossless compression to retain all bits. Is that bad?

If so... what's the usual thing for online sharing here?

Sorry to be such a noob ;)

[feisty2]

you might wanna try x264 -- qp 0 for lossless encoding, it's a little bit slower and the file size should be also a little bit smaller, and pick ".mp4" as the container
the point is such combo (h264 + mp4) could be played everywhere, like USD in currency

[MrPete]

@feisty -- I **like** that idea.

Can you point me to a tutorial on how to use it effectively?

I loaded it up... used those options... and got a 250% increase in size and an mp4 that doesn't play for me :(

Obviously I'm doing something wrong...

[feisty2]

x264.exe --output video.mp4 --preset ultrafast --qp 0 input.avs

[MrPete]

Wow.

I feel like I just ran into a political buzz saw. There's apparently no official x264 build with mp4 support? I'm honestly not sure why; it doesn't seem worth reading through all the back-and-forth on this.

If you know of a place for downloading x264.exe WITH mp4 support, I'd love to hear about it. Otherwise, I can't imagine why any normal person would try to use x264 for creating mp4 files (one of its primary purposes) if it can't actually create mp4 files.

I don't want to get into the details of x264 vs mp4 and why/why-not here. I just want a solution. :)

(If there's something valuable I can learn or contribute by getting into it, then please aim me in the right direction... but this feels like a huge distraction!)

[feisty2]

uh... I didn't know the official x264 builds have no "mp4" support, sorry for that
I use x264 tmod for everyday encoding, and it supports "mp4" output,
you can get it here (https://github.com/astrataro/x264_tMod)
binary (http://tmod.nmm-hd.org/x264/x264_rev2431_tMod.7z)

[creaothceann]

MKV > MP4 anyway...

[feisty2]

MKV > MP4 anyway...

however it works inside really doesn't matter (at least to me), it gives you the wanted mp4 file and without an extra manual mkv to mp4 remux, that's good enough to me

[johnmeyer]

I got back to my main editing computer today and used ffmpeg to convert the file to uncompressed and then looked at it.

The video is indeed progressive. You can easily remove all the pulldown frames using the technique and technology provided by StainlessS that I used in this thread:

Bad 1950s Kinescope - Hopeless? (http://forum.doom9.org/showthread.php?t=167591)

All you need to do is mask the frame so you are only looking at the portion of the frame between the sprocket holes. Then, trigger a decimate so that it only happens when the luma threshhold between those sprocket holes goes above some number. StainlessS's technology is the key to this step.

There will then be some dupes left, and you can use any of several duplicate removal filters to get rid of those.

That's the general outline of how to proceed.

[johnmeyer]

BTW, if you want further help with this, you really need to post something with a little movement. A static shot of flowers in a hot house is probably the worst clip you could have picked.

[MrPete]

Thanks, feisty2...

[MrPete]

Thanks John, yes the "between sprockets" will work in many situations... probably about 2/3 of what I've been capturing. I originally started down that path but hesitated due to the other 1/3 of my situations:
* Sadly, a lot of my older film (mostly B&W) is clear around the sprocket holes.
* Another situation is when the edge has a lot of flare due to improper film handling. (More of that than I wish :) )

However, one way or another, I suspect this *will* be doable. (Worst case, maybe I can write code to find the left border and just monitor near the border. Even clear edges have a dark border next to the image!)

I've just finished connecting the *proper* sync signal, using a Component->Composite converter. Oh Joy! VERY stable captures. The only thing that knocks it into an alternate frame now is some kind of "tug" either due to the takeup reel not having enough power to move the mass of 400' near the end of a capture... or some kind of hold-back from the feed reel (edges a little out of whack.

I'd still love to find a way to eventually do it based on the image itself... but practically speaking... I'll live w/ whatever works.

Time to go try some things, after I get some sleep...

[MrPete]

Woah.

@johnmeyer, the Zebra thread -- and StainlessS's plugin -- look PERFECT. I don't see why the same technique, used very simply, can't work here. "All we need" is to detect the difference between neighboring frames +/-...

(Yeah, I know it is wayyy more complicated than that. You guys invested more than a few Round Tuit's on that football video... in this case I suspect I'll end up with something that can auto-decimate captured film even with defects in the capture sequence. Worth spending some time)

But Zebra gives me hope... Time to go play!

I'll put up a chunk of my latest re-capture, including a transition between even and odd-frame pulldown. (Yikes... and a section that stays right in between... upper/lower half... about as awful as it gets :( )

BTW, I've been putting up hard things on purpose :) -- IMHO, easy clips tend to distract toward simplistic solutions that don't actually work in the harder cases. I'll work on supplying both easy and hard...

[MrPete]

however it works inside really doesn't matter (at least to me), it gives you the wanted mp4 file and without an extra manual mkv to mp4 remux, that's good enough to me

I agree.

So, feisty2... would you believe that the current x264 tmod build is compiled such that it cannot read an AVS, nor an AVI?

This is quite silly, to put it nicely. If you have a windows binary that actually works, maybe you can share it somewhere. I do have x64...

[feisty2]

I agree.

So, feisty2... would you believe that the current x264 tmod build is compiled such that it cannot read an AVS, nor an AVI?

This is quite silly, to put it nicely. If you have a windows binary that actually works, maybe you can share it somewhere. I do have x64...

it does read avisynth scripts, and, u know 32bit program and 64bit program cannot communicate directly, right? :p
perhaps ur using a 32bit avisynth? then you have to use 32bit x264

[StainlessS]

Here something I was playing with some time ago, no idea if it can be persuaded to do anything useful.


DVD=AVISource("D:\dvd_sample.avi") #.GreyScale()
VHS=AVISource("D:\vhs2_sample.avi") #.GreyScale()

Function xLuma(clip c1,clip c2,int n,int rad) {
X = rad*2
Y = rad*2
W = - rad*2 # As in crop(), clip width relative, same as c.width - X - RADIUS
H = - rad*2 # clip height relative, same as c.height - Y - RADIUS
BestX = 255 # Nothing in particular
BestY = 255
diff = -1.0
GScript("""
for (i= - rad, rad) {
for (j= - rad, rad) {
th = RT_LumaDifference(c1, c2,n=n, x=X,y=Y,w=W,h=H,n2=n, x2=X+i, y2=Y+j)
th=((255.0-th)/127.5)-1.0 # LumaDiff as for LumaCorrelation, (-1.0 -> 1.0)
if(th>diff) {
diff=th
BestX = i
BestY = j
}
}
}
S="Lx="+String(BestX)+" Ly="+String(BestY)+" Lth="+String(th)
""")
Return S
}
Function xCorr(clip c1,clip c2,int n,int rad) {
X = rad*2
Y = rad*2
W = - rad*2 # As in crop(), clip width relative, same as c.width - X - RADIUS
H = - rad*2 # clip height relative, same as c.height - Y - RADIUS
BestX = 255 # Nothing in particular
BestY = 255
diff = -1.0
GScript("""
for (i= - rad, rad) {
for (j= - rad, rad) {
th = RT_LumaCorrelation(c1, c2,n=n, x=X,y=Y,w=W,h=H,n2=n, x2=X+i, y2=Y+j)
if(th>diff) {
diff=th
BestX = i
BestY = j
}
}
}
""")
S="Cx="+String(BestX)+" Cy="+String(BestY)+" Cth="+String(th)
Return S
}

frms=min(DVD.FrameCount,VHS.FrameCount)

GScript("""
for(n=1,frms-1) {
SL=xLuma(DVD,VHS,n,5)
SC=xCorr(DVD,VHS,n,5)
Eval(SL) Eval(SC)
BADS=(Lx==Cx && Ly==Cy) ? "" : " MISMATCH"
if(BADS!="") {
RT_Debug(SL,"::::",SC,BADS)
}
}
""")



Return DVD
#Return VHS
Return StackHorizontal(DVD,VHS)


Might be able to get sub pixel accuracy by upscale resize.
Also, think probably be able to get useful info if examining strips rather than full image, eg vertical strip near both
left and right edges to ascertain horizontal zoom, etc.

Although that would need mod to script, just something to play with if you are bored.

[MrPete]

feisty2, thanks for hanging in there with this noob... ok, with bits matched, I do get a result... and VLC and WMP can both view it. But...
- QuickTime just gets all-black (most variants) or all-white if I convert to RGB32 or YV24 in the AVS before it gets to x264.
- Vdub/AVIsynth can't read the mp4 at all... I assume I'm missing a codec of some kind... or else my system is completely hosed.

Maybe I'll open a separate thread on this... getting OT from my original question!

[MrPete]

StainlessS... looks interesting!

OK... Real Life has intervened... gotta go pack boxes for a while...

If anybody is interested... uploading new clips... in Lagarith format (until I am happy with mp4!)
An easy case. Black borders, alternating good/pulldown. (http://www.ds.org/files/avisynth/dmate-ez-lags.avi)
"Hard" case - begins with frames that have upper/lower pulldown instead of good/bad frames. Last part is more normal. (http://www.ds.org/files/avisynth/dmate-hardsplit-lags.avi)
"Hard" case - left border is clear; B/W movie (http://www.ds.org/files/avisynth/dmate-hardclear-lags.avi)
"Hard" case - border "burn" due to exposure (http://www.ds.org/files/avisynth/dmate-hardburn-lags.avi)

More laterz...

[foxyshadis]

feisty2, thanks for hanging in there with this noob... ok, with bits matched, I do get a result... and VLC and WMP can both view it. But...
- QuickTime just gets all-black (most variants) or all-white if I convert to RGB32 or YV24 in the AVS before it gets to x264.
- Vdub/AVIsynth can't read the mp4 at all... I assume I'm missing a codec of some kind... or else my system is completely hosed.

Maybe I'll open a separate thread on this... getting OT from my original question!

VDub will never support MP4, most likely. Avisynth can read it with LSMASHVideoSource (lsmashsource.dll) or FFVideoSource (ffms2.dll).

The QT problem is weird, but QT is very, very weird too, so I wouldn't rely too much on it. x264 will normally convert everything to YV12, did you force it to encode to YV24 with --output-csp i444? In that case it's no surprise QT can't handle it, very few players can handle anything other than 8-bit YV12 H.264.

[MrPete]

(I didn't tell x264 to do anything other than make an mp4. The encoding difference was what I fed into it from the AVS :) )

[feisty2]

Quicktime only supports what it wants, that's the Apple style :)

[MrPete]

Thanks all... I'm making good progress. With proper parameter tweaks, I think StainlessS' Zebra() will enable me to do it! My detection will have at least two levels available:
a) Watching for bright/dark frames in the sprocket-hole area, based on "black" set by sampling the RIGHT edge of the frame
b) If (a) fails, then watch for overall bright/dark frames, looking in particular at column-based Zebra() (ie rows=false)

I need to learn a few things to put it all together... (see other posts...) but it is getting close!

[StainlessS]

Do not underestimate the YInRange band in Zebra, it is very 'tuneable'.

[MrPete]

Do not underestimate the YInRange band in Zebra, it is very 'tuneable'.

You've got that right! That's exactly what I'm doing.

For analyzing the "dark" area between sprocket holes, I dynamically pull the current frame's black-point (which varies frame-to-frame due to auto-exposure adjustments)... and use that value + a constant as the LO value for YInRange :)

[Asmodian]

creaothceann only meant that MKV was better than MP4 anyway not that the tmod build does a MKV -> MP4 conversion internally.

x264.exe --output video.mkv --preset slow --qp 0 input.avs

Is also universally supported by anyone who might look at a sample.

[MrPete]

@StainlessS -- I'm making great progress. Very very close...

I have run into something with RT_YDifference() et al that surprises me a bit. In many situations, what it provides is exactly what I want. However, for identifying whether I want odd or even frames for decimation, I need to grab a bunch of signed, not unsigned, Y differences. In such a case, I care whether Y is lower on the first or second frame of the pair!

Am I missing something? I'm hoping I don't need to write a bunch of code to replicate what you've already got running sooo well.

[StainlessS]

RT_YDifference returns the average pixel difference, signed would be of dubious value, imagine a checker board pattern where
black checker boad pixels were +1 and white -1 (in comparison with the alternate frame), the result would be 0 and seemingly identical.
RT_YDifference is a measure of 'sameness'. (EDIT: or rather NOT sameness, 0 = identical)
Sounds like you would be wanting AverageLuma or YPlaneMin, or YPlaneMax, or YPlaneMedian, as well as or instead of.

EDIT: You could always use eg Subtract() and measure result of that.

EDIT: The RT_LumaCorrelation thing (as previously posted) is reasonably tolerant of differing brightness between test frames,
whereas RT_LumaDifference and RT_YDifference are not.

[MrPete]

I guess there's no reason why Subtract() shouldn't work! Hmmm...

I wonder if you might have an idea for a conceptual puzzle I've got on this. I have the basic engine working ok... but sometimes it gets fooled:

Basic concept: look for, and follow, patterns of alternating bright/dark frames in some portion of the image.

a) If the pattern's there on the left / sprocket hole area, that's most reliable; use it.
b) If not (a) but there's a bright/dark pattern in the overall image, use that.
c) If no pattern, just continue using whatever was last seen, until a pattern is seen again.

What's fooling my system is surprising to me: it's actually sequences that match pattern (a)! I think this is because:

1) It's not always obvious whether one sample is brighter. Consider a fade-in or fade-out (or ISO-change causing the same effect):


HigherY=25 HigherY=30
LowY=20 LowY=24

The middle two samples are pretty close to one another... and at some point the sequence actually CAN shift from odd-good to even-good frames.

2) I'm not actually allowing a change between odd/even captures all the time... for consistency and speed, as long as there's a significant difference between Y values of successive frame pairs overall, I don't analyze each frame in detail. I only look in detail at each frame if the overall pattern breaks.

What I've seen a few times is that #2 gets fooled sometimes, into running with the exact-wrong frames for a while.

Now... possibly using Subtract() to get a signed-difference between adjacent frames will solve it completely... but I'm wondering if anyone has ever developed an overall Better Solution for this kind of problem, where I'm trying to lock into a pattern... but also be smart about the occasional pattern-shift.

I'll keep at it (tonight ;) )... just curious about other ideas...

[StainlessS]

Would it be of use to get differences of Y eg

D1=F2Y-F1Y D2=F3Y-F2Y D3=D4Y-F3Y

They would of course be signed.

[MrPete]

Would be of use to get differences of Y eg

D1=F2Y-F1Y D2=F3Y-F2Y D3=D4Y-F3Y

They would of course be signed.

Yeah. I got sidetracked by RT_YDifference...

I suspect just getting average luma for each and subtracting as needed will be quicker than a bunch of Subtract() operations.

I will soon turn to performance (good performance, and good performance evaluation tools, have always been a hot button for me...)

I see that RT_Debug* has some kind of microtimer in its display. That warms the cockles of my heart :)

(Is there a function that retrieves fractional seconds/ms/us/?? If so, maybe I can write up a little developer perf-tuning library someday...)

[StainlessS]

Would RT_TimerHP() do ?

EDIT:

RT_TimerHP()
Returns a higher precision time than RT_Timer (If available on your system, otherwise uses same routine as RT_Timer).
Note, where RT_Timer returns time elapsed in seconds since start of process, RT_TimerHP is not defined to return anything
in particular, the only thing that is meaningful is the difference between returns from 2 calls to RT_TimerHP.
DO NOT mix times from RT_Timer with RT_TimerHP, results may be meaningless.
Usage:
s=RT_TimerHP() Test() e=RT_TimerHP()
Str="Test Start="+String(s,"%.3f")+" End="+String(e,"%.3f")+" Time="+String(e-s,"%.6")+"secs"
SubTitle(str)

[MrPete]

Nice. Yes. :)

[StainlessS]

microtimer in its display
The readout prior to any RT_DebugF() output is generated by DebugView, you can turn to 'clock time' rather than time since capture started.
Also can set filter to only show debug stuff containing/not containing certain strings. I usually include a colon in my output.
The filter is a real good idea if using Avisynth v2.58, IIRC it outputs load of debug stuff and could make what you are looking for tricky.