Software TBC - I finally made one

[jmac698]

I've had a few dedicated souls searching and tracking me down because of a script I posted for a TBC. I hate to disapoint you, but there is no general TBC in software. I have done some significant work which is useful in rare cases, and I have 3 hopeful ideas left. Here's what I have so far (internally):

-say you have a colorized VHS and want to match it to a b/w dvd. You can stabilize against the DVD and colorize it
-You have a noisy VHS and you are using multiple pass averaging to denoise it. You can stabilize the copies relative to each other (thus they all have the same, but wrong jitter)
-There's some small chance of reducing jitter statistically, but only if you have a primitive capture card
-There's some small chance of reducing jitter by tweaking the horizontal shift of a bt8x8 card (so you can actually see the edge of the hsync)
-There's a mathematical algorithm to line up pictures, but I haven't implemented it
-I can record a test signal in part of the picture and perfectly line it up later. This is useless except for measuring jitter statistics.

It's been also because there's various subproblems; matching colors of two sources, and fixing dropped frames. I've made progress in these too:
-I know how to derive the tweak statement to match two videos, but it may need an FFT. I have one idea to possibly script it.
-I can create and analyze a test signal to prove if there is any frame dropping and find any patterns
-I know how to script a frame drop/dub fixer but it will have to make guesses in low motion areas
-I've found capture settings which reduce frame drops

I discovered some interestings things;
-Some capture cards do have a TBC, but it produces the wrong results
-Getting good sync with no frame drops really varies by card, driver and OS, and each may need different capture settings

And I have some ideas that might work with no special equipment:
-You can digitize video with a sound card, however you only get 12 samples per line. You can see the sync and should be able to measure the jitter, however the picture influences the signal so I think you'd have to use DSP to combine both recordings (capture card and sound card at the same time) to extract accurate timing information, and this is going to be hard.

Thanks for your inquiries, it will keep me interested.

Update:
I've finally managed to tweak the chip of a capture card to give me the raw hsync signal, which should in theory let me create a software TBC (similar to a hardware TBC). This works only on specially captured videos. Strangely enough, lining up the hsync doesn't fix the picture. I'm still investigating what's going on.
The good news is, if this works no one will ever have to buy a TBC again - just download my program and use your capture card for an instant hardware TBC!
The hardware is fully capable of doing this, if I can find the right algorithm.

Update2:
It works.
http://forum.doom9.org/showthread.php?t=162726

[Mounir]

To match the colours of a given video there is the excellent filter Coulourlike (http://forum.doom9.org/showthread.php?t=96308) i've tried it, it does an excellent job but you have to adjust the levels of the video first

[Mini-Me]

jmac, I've been working on related problems to deal with some of my own captures. We actually bumped into each other on the Virtualdub forum in December, where I had the same findings as you about the proper settings for frame-accurate captures. :)

In case anyone else is wondering, here are those settings (assuming jmac hasn't found even better ones). In VirtualDub, go to Capture->Timing:
The ONLY checkbox that should be ticked is "Correct video timing for fewer frame drops/inserts."
Resync mode should be "Sync audio to video by resampling the audio to a faster or slower rate."
Audio latency determination should be automatic.
Frame drops are extremely rare (most two hour captures have none at all), whereas they happened several times per capture with more commonly recommended settings. Virtualdub doesn't even indicate the remaining frame drops in the statistics, so they're probably unavoidable glitches that occurred earlier in the chain. Frame insertions do not happen at all, although occasionally a field will be a duplicate of the last field (possibly shifted). This is also unreflected by the statistics, so it also probably happens before the signal reaches Virtualdub.

Anyway, I've been working on similar problems. jmac, you've been working on horizontal stabilization, correct? I remember you linked to a research paper by a mathematician, who approached the problem with some kind of polynomial fitting (or something) and had really impressive picture samples. Did that not end up working well, or did you not get the chance to implement it?

As Mounir mentioned, ColourLike is useful for matching two captures. When I have time, I plan to create a smooth, dithered version of it (like Smoothlevels), which correctly matches both luminance and colors in the LAB colorspace. (I need to do that to fix a certain problem I have with my captures, so it's going to happen someday...I just don't know when. :p)

For jmac and everyone: I've made significant progress on vertical stabilization, but it pretty much assumes reasonable horizontal stabilization. (My AG-1980's built-in TBC keeps the individual lines rock solid, but the fields can be jumpy, and my AVT-8710 is somehow worthless.) My stabilization function currently works much, MUCH better than g-force's Stab function, but I'm still modifying it every few days, so it's not ready for posting. It can match local field height to minimize a measure of combing (or simply eliminate spikes), it has a nonlocal method that tries to target "anchors" like timestamps, and the two can be used together and weighted. I'm still testing whether a much stronger version of Stab works better than my current nonlocal matching, so I might replace that...but anyway, it's coming sooner or later. :) I've already made a near-unwatchably jumpy clip pretty smooth, and it could probably be made perfect with slightly more aggressive parameters. I just haven't had the time to test that.

Like jmac, I tend to average multiple VHS captures (median), after matching their field heights and running them through a [very immature] function that replaces blown out, duplicate, torn, flickering, and corrupt fields. I hope to eventually post that function too, but it's slow and complex ScriptClip stuff, and the parameters are still in "the try some numbers for the various thresholds and cross your fingers" stage. It's nowhere near ready.

I'm curious how jmac approached horizontal jitter matching for each line (I remember the thread about splitting a clip into lines), but here are the functions I've been using to vertically align one capture with another and paint in the missing pixels:

# Pass in clip with separated fields
function MatchFieldHeight(clip c, clip target, string "fill", bool "mvtools", int "maxdisp", int "metric")
{
# Defaults:
# Fill black lines? Valid values are "target" and
# "mirror", and other values leave the lines black.
fill = Default(fill, "target")
# Use MVTools or DepanEstimate? DepanEstimate can make unpredictable
# undercompensation errors when the movement is large.
mvtools = Default(mvtools, True)
# Limit maximum displacement with DepanEstimate?
# I haven't figured out how to translate this to MVTools yet.
maxdisp = Default(maxdisp, 50)
# Metric controls DCT parameter of MAnalyse. Default is 0 for SAD, but
# other modes may work better for difficult scenes and captures with
# different luma levels or flicker.
metric = Default(metric, 0)

# Note on other MAnalyse Params:
# blksize 32 gives wrong results; using default of 8, but 16 is probably viable.
# search = 7 is an exhaustive vertical search.
# searchparam is search radius; search 2 * radius + 1 at each level;
# using the default of 2, since maxdisp doesn't directly translate.
# Truemotion is for high correlation/global motion (which is what we want).

# Get motion data
interleaved = Interleave(target, c, target)
vectors = mvtools ? MAnalyse(MSuper(interleaved, pel = 1), blksize = 8, search = 7, \
searchparam = 2, truemotion = True, dct = metric) : NOP()
motion = mvtools \
? MDepan(interleaved, vectors, zoom = False, rot = False, range = 0) \
: DePanEstimate(interleaved, range = 1, trust = 0.0, dxmax = 0, dymax = maxdisp)

# Match the field heights
mirrorval = fill == "mirror" ? 3 : 0
result = Depan(interleaved, data = motion, offset = 1, subpixel = 0, mirror = mirrorval)
result = result.SelectEvery(3, 2)

# Return, and paint in black lines if necessary
return fill == "target" ? result.FillBlackLines(target) : result
}

function FillBlackLines(clip dest, clip source, bool "protectpixels", bool "debug")
{
# Defaults
protectpixels = Default(protectpixels, False)
debug = Default(debug, False)

# Convert to planar if necessary
planar = dest.IsPlanar()
dest = planar ? dest : dest.ConvertToYV16()
source = source.ConvertToYV16()

# Create a mask for pure black pixels
destblackmask = mt_lut(dest, "x 0 == 255 0 ?")
sourceblackmask = mt_lut(source, "x 0 == 255 0 ?")
mask = mt_lutxy(destblackmask, sourceblackmask, "x y > 255 0 ?")

# Eliminate spurious matches at user request; could horizontally downsize,
# binarize, and upsize for a better result, but that's slow and unnecessary
# given visual black = 16 luma.
mask = protectpixels ? mask.mt_inpand(mode = "horizontal") : mask

# Merge the fields (all channels) using the mask's Y channel
mask = MaskYUVFromY(mask)
dest = mt_merge(dest, source, mask, u = 3, v = 3, chroma = "process")

# Return result or mask in original format (unless RGB)
return debug ? \
planar ? mask : mask.ConvertToYUY2() \
: planar ? dest : dest.ConvertToYUY2()
}

function MaskYUVFromY(clip mask)
{
u = mask.UToY()
maskuv = mask.BicubicResize(u.width, u.height)
return YToUV(maskuv, maskuv, mask)
}

[jmac698]

Great to see this work going on! I'm hoping to do some more work soon, I've just been doing some housecleaning and getting organized, I have my equipment sitting on a table ready for some more tests...
colourlike - I'm aware of it, but it's not what I need. To use it, you need every frame of two videos (reference and destination). I want to use, say one example frame and adjust colours for the whole destination. To do that, I can use the histograms of both and find the best fit in terms of brightness, contrast, tint, and saturation. I just realized how to do this.
It also doesn't always work well, it seems obvious to me that you should be using it on L*a*b* colorspace (as mini-me realized), so I'm also looking at yet another sub-problem of converting colorspaces. There's some sophisticated filters for something similar that HTPC people are using to project colour-corrected movies in realtime, I'll have to look into them. And as mini-me pointed it, it will be smooth as well.
I also want to look into the non-linearity of the digitizer, I know a way to calibrate each level of the ADC very accurately.

About the mathematician's paper, yes I talked to her but she was sick at the time and haven't bugged her for the matlab codes. I haven't implemented that yet.

The problem with dropouts is something about the driver and directshow api, that the capture rate has to be set before you start capturing, so there's no way to just allow the frames as they come in.

What was your color problem? Sounds interesting :)

[Mini-Me]

Great to see this work going on! I'm hoping to do some more work soon, I've just been doing some housecleaning and getting organized, I have my equipment sitting on a table ready for some more tests...
colourlike - I'm aware of it, but it's not what I need. To use it, you need every frame of two videos (reference and destination). I want to use, say one example frame and adjust colours for the whole destination. To do that, I can use the histograms of both and find the best fit in terms of brightness, contrast, tint, and saturation. I just realized how to do this.
It also doesn't always work well, it seems obvious to me that you should be using it on L*a*b* colorspace (as mini-me realized), so I'm also looking at yet another sub-problem of converting colorspaces. There's some sophisticated filters for something similar that HTPC people are using to project colour-corrected movies in realtime, I'll have to look into them. And as mini-me pointed it, it will be smooth as well.
I also want to look into the non-linearity of the digitizer, I know a way to calibrate each level of the ADC very accurately.

About the mathematician's paper, yes I talked to her but she was sick at the time and haven't bugged her for the matlab codes. I haven't implemented that yet.

The problem with dropouts is something about the driver and directshow api, that the capture rate has to be set before you start capturing, so there's no way to just allow the frames as they come in.

What was your color problem? Sounds interesting :)

Going from image stabilization to histogram matching is a bit off topic, but it's your thread, soooo... :)

The image on my VHS tapes has a texture to it that looks like a coarse-grained and bumpy "scanline" noise pattern...and I thought it was one. I my damnedest to use NeatVideo to clean it, and I attacked it with several different painstakingly crafted 100% noise profiles. In the end, even overkill settings had trouble totally eliminating the pattern, unless I used spatial denoising only. Stronger temporal settings actually made NeatVideo deliberately retain it in certain areas, since it's so stable within any given scene! Basically, NeatVideo needed to turn my video into plastic to get rid of the pattern. Didee suggested Vinverse, and it helped, but it blurred unnecessarily. I say "unnecessarily," because I had figured out what the real problem was, and I realized the solution required no blurring or denoising whatsoever.

On a hunch, I had decided to do clip.AssumeTFF.SeparateFields.AssumeFrameBased.AssumeTFF.SeparateFields() and check the levels of the odd and even lines. I noticed that the levels were obviously different, so I figured out which lines seemed brighter/darker for even fields and odd fields and adjusted them accordingly with levels. After viewing the result, the pattern was TOTALLY gone in the scenes I targeted, partially gone in others, and...much, much worse in the scenes where the parity was different.

As it turns out, the even lines of each individual field always have different luma levels from the odd lines (maybe chroma too, but I forget). The magnitude of the difference changes from field to field, and you can't even predict from scene to scene which lines are going to be the dark lines and which will be the light lines. For all I know, it may not even be a simple contrast or gamma difference but something that would require arbitrary curves to correct in some cases. The only thing that is consistent about it is that when the image is woven, the pattern is dark, dark, light, light (or vice versa, and the top of the frame can start anywhere in the middle of that pattern).

Funny enough, I need exactly what you need: I need to individually match the levels or curves on a frame-by-frame basis! (Or rather, I need to match each half-field to the levels of the whole field). You actually can use ColourLike and HistogramAdjust for this purpose, but it's painfully slow, because you have to do everything within ScriptClip (including writing a histogram to a file and reading back from that file, every single frame). I actually tried this (I have the scripts somewhere around here in some forgotten test directory), and it eliminated the pattern entirely throughout the whole video, leaving it clean...but the loss in precision resulted in unacceptable banding and posterization.

I have the source code for ColourLike, and I plan to heavily modify it when I get the time. My first modification will be including a mode that processes each frame individually (without writing to a file). Second, I want to overhaul the core algorithm: Instead of just scaling the histograms, ColourLike constructs a 256-point curve to map the CDF (cumulative histogram) of each channel of the first clip to each channel of the second. This is a more robust approach that handles any monotonic transformation from one histogram to another, but it's still not ideal:
First, the 256-point curve is a quantized approximation of the transformation of one CDF to another. I want to create a much higher-resolution curve (e.g. 65536 high-precision samples) or even interpolate each pixel lookup directly from the two CDF's.
Performing a higher-resolution lookup would still be useless if we point-sampled it for every pixel. Instead, we should use something like serpentine Floyd-Steinberg dithering (or something better, which I believe the guy from the HQResample thread discovered). That way, if we have a stretched curve, we won't lose tons of precision. For instance, if 128 maps to quantized 129, and 129 maps to quantized 140, we won't leave ten entire luma values from 130-139 unused in the final image. ;) Instead, our diffused error allows us to smoothly sample anywhere in the given range. Ten unused values is an extreme case that wouldn't apply to my use case anyway, but it can and does happen to people who use ColourLike to match two unrelated clips.
Then, we get into the colorspace problem, where YUV has weird color-mixing issues and is totally inappropriate for matching colors...so before we even calculate the CDF's, we need to convert internally to high-precision L*a*b.
Once I implement all of this functionality, it wouldn't take much more work to let users supply custom curves/levels for application in L*a*b space...so taken together, I'll inadvertantly be reimplementing LaTo's SmoothLevels too. Sigh. :p Mine will probably be comparatively slow as shit though, so I imagine they'll have slightly different use cases at least.

[jmac698]

I read the wikipedia entry on CDF, looked at the text file of colourlike, and knew immediately that's what it was doing :) Then I thought of using image registration techniques to match two histograms, this would give a brightness and contrast. I'll have to look at CDF again to see if it also can give me a brightness and contrast.
Once I just have these two numbers, I can use other tools like smoothadjust to implement them.
t3dlut might work for converting to L*a*b.
Your problem with brightness patterns sounds easy to solve; just take the averageluma of each field to guess the brightness difference.
I agree we need a single pass color matcher.
Another way around the quantization errors is something else I've thought of; some kind of HDR from multiple pass captures. I was thinking this would b ea good way to bypass the non-linearity limitations of the capture as well.
Let me see what might be a good approach and I'll get back to you; sounds like we could share some work on this.
8 bits is fine for presentation; not good for manipulation! We so need 16bit video.. that's why I wrote Deepcolor tools.

[Mini-Me]

I read the wikipedia entry on CDF, looked at the text file of colourlike, and knew immediately that's what it was doing :) Then I thought of using image registration techniques to match two histograms, this would give a brightness and contrast. I'll have to look at CDF again to see if it also can give me a brightness and contrast.
Once I just have these two numbers, I can use other tools like smoothadjust to implement them.

Matching histograms exactly requires curves (and they still only work if the curve from one CDF to another is monotonically increasing), but if you only want them to cover the same range of values, you can do that easily enough with ScriptClip.

The functions YPlaneMax and YPlaneMin work like an inverse CDF: Pass in the percentage of outliers you want to ignore, and viola, you have the highest and lowest luma values in your scene! If you pass them into Levels as input, your histogram will fit perfectly to whatever output levels you set. For instance, you could stretch the histogram of clip A from luma 16 to luma 235 like this within(G)ScriptClip:
A = A.Levels(A.YPlaneMin(), 1.0, A.YPlaneMax(), 16, 235)

If you want to match the levels of clip A to clip B and ignore 0.01% of outliers:
A = A.Levels(A.YPlaneMin(0.01), 1.0, A.YPlaneMax(0.01), B.YPlaneMin(0.01), B.YPlaneMax(0.01))

Instead of using a hardcoded gamma of 1.0, you can also match the midpoints better by comparing the two AverageLuma values and calculating a gamma shift from them. I forget the exact math, but I'm pretty sure you can calculate it precisely given the known information...or if it's the median you need, then you'd want to use YPlaneMax(50.0) in your calculations instead of the average.

There are two big problems though:
Using Levels (or anything else) to change contrast or gamma in 8 bits will lose you precision and create some amount of banding/posterization. You can use SmoothLevels, but it's slow as hell inside of ScriptClip.
Exactly matching two histograms requires curves; it's not enough just to match the the endpoints and midpoints.

t3dlut might work for converting to L*a*b.
I remember seeing a thread about colorspace conversions where that filter was mentioned. It's not ideal to convert to L*a*b in a separate filter though, because it has to be quantized to 8 bits before you can even work with it, and converting it back will result in another loss of precision. Dithering would help, but the perfect solution would convert to high-precision (16-bit or even 32-bit) L*a*b, calculate and apply your levels/curves there, convert to high-precision YUV, and then dither back to 8-bit YUV.

Your problem with brightness patterns sounds easy to solve; just take the averageluma of each field to guess the brightness difference.
I'm actually really tempted to try this, using the method I outlined above for you. It might even work. There's no guarantee though, because the shift may not be that mathematically simple, and it might actually require an arbitrary curve to match the histograms perfectly (so that the pattern doesn't appear at any brightness level).

I ultimately wouldn't want to do it without dithering though, and SmoothLevels and ScriptClip don't play nicely together. That's why I've just been daydreaming away about overhauling ColourLike. :p

I agree we need a single pass color matcher.
Another way around the quantization errors is something else I've thought of; some kind of HDR from multiple pass captures. I was thinking this would b ea good way to bypass the non-linearity limitations of the capture as well.
That's a really clever idea: You could capture the lower half of the range separately from the upper half, fitting both into the full range. It's kind of the opposite approach from what CretinDesalpes has taken: Whereas he separates the most significant bits from the least significant bits, you would separate the low range from the high range, HDR-style.

That said, the real problem is storing high precision data it in such a way that Avisynth can hold it and filters can perform meaningful operations on it...between your method and Cretin's, I wonder which would work better with existing, "unaware" filters? It probably depends on the filter though.

Let me see what might be a good approach and I'll get back to you; sounds like we could share some work on this.
8 bits is fine for presentation; not good for manipulation! We so need 16bit video.. that's why I wrote Deepcolor tools.

Most definitely...
It will sadly still be a long time though. Even if Avisynth adopted high-precision formats and the core filters were modified to work with them, there are still a ton of common filters that would take years to catch up. Someday, though...

[jmac698]

I confirmed that it can be done, you can convert to XYZ in td3lut and I suggested to the author to support the 16bit video hack of smoothlevels. Then I looked at the formula to go to L*a*b* and it can probably be done in masktools. However, this really encourages me to write a plugin. That would be a fairly big project though, and it certainly wouldn't be optimized.

Meanwhile we can do some tests in a good research level image program like ImageJ (or the Fiji distribution).

My next problem is estimating scale in one direction. mvtools only has a zoom factor, you have to supply aspect. Perhaps that with show or one of the fftfilters can get me the raw fft so I can find it myself.

There's a trick to find the pixel with peak value in script, it's to make a linear ramp video and then use it as a mask somehow and the luma of the result becomes your x coordinate.

[FlimsyFeet]

Since this discussion seems to be now about colour matching, I just wondering if the page here (http://www.mee.tcd.ie/~sigmedia/Research/ColourGrading) is of any use, or if you are already aware of this method. Or maybe I should have bumped the ColourLike thread.

[jmac698]

I found a working link to one of the papers:
http://www.mee.tcd.ie/~sigmedia/pmwiki/uploads/Main.Publications/fpitie07b.pdf
That's a good reference.
I was following another approach; create a model of how the colors are affected by a VHS and estimate the parameters to that model. Things like a linear color shift by saturation or by frequency. I've already modeled my capture card; it's pretty much a strict addition and multiplication by the brightness and contrast capture controls (with little details like which happens first and what clipping there is before you even get to it). The other simplification is that it only has to be done once; I don't expect the color to change for the entire material except in bad frames.

Matching laserdisc to dvd might not work, one reason being there was a new transfer which was color graded differently, so I'm back to the general problem of just making two images look similar.
I also have to think about the effects of sharpness, noise, rainbowing, dotcrawl, and geometric matching. For example I just did a color/geometric matching from laser to dvd and you can still see a border between sharp/blurry and noisy/smooth.
Rainbowing would mess up the statistics. Dot crawl is easy to remove if you have two frames of something that's still.
I'm also interested in merging SVHS and VHS copies of the same material; how they differ and finding a model which closely matches it. I've found tint changes from the same VCR through either composite or s-video connector; why is this so? I can also see differences per head; slight luma changes over the frame repeating every 4 frames.

I guess what I am saying is that, I don't see the need to match curves, because why would gamma change between video tapes or passes of video tapes, even under different capture settings?

Btw I also tried a neat experiment; what if you bought many copies of the same VHS tape and merged them? You should eventually eliminate all variability. Unfortunately when I tried that, one of the tapes had a manufacturing defect and I think I only bought two copies.

[pandy]

http://www.cl.cam.ac.uk/~mgk25/tv-crypt/image-processing/antisky.html

[jmac698]

I finally made a software TBC, however it's not working as expected. The line at the left is the HSYNC, I believe, but if it isn't, then that would explain why it's not working :)
So it's hsync, color burst, and picture right? Actually I didn't expect it to be a thin line but more like a wide black line down the screen, if so, then what's that bright line?
http://screenshotcomparison.com/comparison/86066

[2Bdecided]

That's the colour burst and part of the back porch. No sync pulse there at all. No idea what the vertical white line in the colour burst (green vertical bar) is - some artefact of that VCR or capturing, and not generally useful I think. I've never seen one like that.

http://en.wikipedia.org/wiki/Analog_television#Structure_of_a_video_signal

and

http://www.davidstringer.info/training/pics/pmcross.jpg
which shows what you would see, from
http://www.davidstringer.info/training/how_television_works.htm

Cheers,
David.

[jmac698]

:thanks:
That's helpful! His is even green too :)
Don't worry, I'll redo the capture and move it more to the left. I'll also remove the NTSC pedastal to see black level.

[Mounir]

Vertical green bar hmm this remind me of my previous standard vcr (pal vcr with ntsc playback(pal 60) capability..suposedly) which displayed a vertical green bar on the left part of the image top to bottom whenever i played ntsc tapes.
When i bought a higher end vcr it was better indeed.

It's either this or a macrovision related issue imo