Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 24th July 2017, 23:09   #161  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,167
I agree. It was a cool idea that seemed like it could have worked.

I wonder if some of StainlessS' RT_Stats code could be used? I've been able to use it for all sorts of line-to-line comparisons. The shifting part of the code (shifting an entire line left/right) should be straightforward, as long as you have a clear left edge (which is what he was using). The bigger issue is that time base errors are often not consistent across the entire frame, left to right. However, even a first-order correction would be welcome.
johnmeyer is online now   Reply With Quote
Old 24th July 2017, 23:25   #162  |  Link
lordsmurf
Registered User
 
lordsmurf's Avatar
 
Join Date: Jan 2003
Posts: 120
What's missing is temporal intelligence.

When you take a video with horizontal jitter, and hit it with a harsh temporal NR, you'll notice that the wiggling largely dissipates. What you're left with, of course, is nasty temporal artifacts (aka "mouse trails", ghosting). But motion-stable artifacts!

The solution for a software TBC could be to run temporal NR, use the results as a baseline for where the lines should be, and then shift accordingly.

More chaotic timing noise would need some degree of both the line shifting, and temporal NR. Perhaps even something semi-intelligent, maybe based on Donald Graft's dynamic NR, to reduce ghosting artifacts.

I think this filter was going down the wrong path.
(1) You never have the sync data. Proof-of-concept was somewhat stupid, as it required a VHS tape be captured with special extra data. That's just not real-world whatsoever. I think that was a sidetrack tangent to where it could have gone.
(2) Timing errors are almost never static, but continuously move. It's analog. Analog is controlled/restrained chaos. The quicker you realize this, the better you are long-term.

I wish somebody would take up where jmac698 stopped. He was on to something.
__________________
Back in town.
If you want my advice, then find me at the DigitalFAQ forum. Glad to assist.

Last edited by lordsmurf; 24th July 2017 at 23:31.
lordsmurf is offline   Reply With Quote
Old 25th July 2017, 01:07   #163  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 6,952
You need to be logged into your DigitalFAQ.com account to download above linked files.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 28th July 2017, 12:07   #164  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
@johnmeyer
"The bigger issue is that time base errors are often not consistent across the entire frame, left to right."
Correct. That's why it started as fast line shifter then turned into software tbc. Both the left and right edges have jitter, which means there's some horizontal resizing per line. The filter does work dynamically per line.

@lordsmurf
"You never have the sync data" Semi-true; for a professional, it is assumed they are going to buy the equipment they need to get the sync data. Not only that, there were a huge number of cards based on that chipset which could be made to work, most famously the WinTV series.

Second, I've successfully applied it even without sync data; if the capture window is large to enough to leave black borders on both sides (video supplied by cherbette).

You misunderstand that I never came into this thinking it would be a polished, end-user solution; it was experimental, meaning I was trying to understand the nature of the problem to start. And I did learn something; I learned specifically that left-aligning the sync is not the solution; I have to fit between two edges or two syncs. I just put it out there expecting people to play with the functions, but no one ever did

This is how engineering is done, start by a) capturing data on the problem b) creating a model for the problem c) solving the model d) apply the solution to the real-world e) refine and iterate.

I also showed to a lot of people that it can be done in software and helped dispel a lot of the misinformation that you need some kind of hardware box. The internet was adamant that it was impossible without sync, that's not true though.

Anyhow, there is another way. There's a paper which works by using mathematical properties (an L1 norm*) which have a good chance of saying if lines are aligned. Someone else tried to implement it too, but we were both stuck over one sentence which wasn't explained. I was in contact with the author but she was very sick at the time. I can still finish this someday.

Another experiment someone did is detecting alignment with a Fourier Transform, but that didn't work out.
*an L1 norm is based on Least Absolute Deviations

"What's missing is temporal intelligence" what you're getting is a frame blending with random horizontal offset. What you call stable jitter motion is due to this, mathematically speaking:
the jitter is guassian random, and as you average frames, the jitter is reduced by the square root of the number of frames, making it look more stable. However, the ghosting is increasing in the same proportion. Features which don't move will literally be guassian blurred. Features that move will also have motion blur. I don't think using this as a reference would really work.

Using my process of engineering I could attempt to prove this; I would simulate jitter, apply temporal denoising, then try to fix it. That's going through the steps of modelling and simulation.

I still can't explain why there is jitter to begin with; the capture card clearly adjusts sample rate based on the sync timing, it is in fact a built-in TBC.

Last edited by jmac698; 28th July 2017 at 12:26.
jmac698 is offline   Reply With Quote
Old 28th July 2017, 12:26   #165  |  Link
lordsmurf
Registered User
 
lordsmurf's Avatar
 
Join Date: Jan 2003
Posts: 120
I have at least one sample clip that could be right-aligned, as the black image border is visible. If you can align that, the video should be steady.

I'm still not sure you understand what I mean by "temporal intelligence". You'd not average anything, aside from analysis. If you sample the same line across (X) [3-7] frames, and at least (Y) [25-75] % is still the same (noting "the same" occurs AFTER the pixel shift), then decide if the line shall be moved in the direction of the majority position. Temporal analysis, not temporal application.

This would process slow as hell, but in the modern computing era (Skylake+), it can probably be done. I'd imagine its probably not worse than Mercalli on an old CPU.

I can think it up, but programmers (even mathematicians) more gift than I must create it. That part is beyond me (although I could probably learn, given many years).
__________________
Back in town.
If you want my advice, then find me at the DigitalFAQ forum. Glad to assist.

Last edited by lordsmurf; 28th July 2017 at 12:30.
lordsmurf is offline   Reply With Quote
Old 28th July 2017, 12:41   #166  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
As to the person who couldn't get it to work 3 years ago; you were using it wrong. I believe I can script around the problems with the varying thresh levels.
I agree with your conclusions that the border detection wasn't good enough. I was expecting that other scripters around here would easily be able to address that problem. If anyone can post a sample I could work on that problem and it will still work.
The basics of this are sound and proven; all you need is some black border, and a better border detection. No need for sync.
jmac698 is offline   Reply With Quote
Old 28th July 2017, 12:45   #167  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
Post your sample.

Yes I understand you meant it as a detect clip. Pans would obviously make it fail, because is it a pan or jitter? What about diagonal lines? I think it could work in some scenes but not others. It's worth an experiment though; I believe to always keep an open mind and each failure only helps you learn more

Last edited by jmac698; 28th July 2017 at 12:48.
jmac698 is offline   Reply With Quote
Old 28th July 2017, 12:54   #168  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
ohh... I have to post this before I forget. I had an idea for static scenes (stationary scenery with just characters moving, end credits, titles etc.). All you have to do is align the frame relative to the last one; more than that, to the average offset of all the static frames, this then has to average close to the true position.

Code:
    ****  +4
 **** +1
  **** +2
2nd line, being the left-most aligned, is taken as a reference line. From that, we get +1 and +3 offsets, average that to +2 offset and move that line (in all frames) to that position. It is now much closer to the true offset, and it's guaranteed to converge to the true offset by the properties of statistics. I'd say a 1 second scene is easily enough to get nearly perfect sync (30 frames can account for 5 pixels of offset). Each line will independently line up with those above/below like magic

*1 proof:
https://math.stackexchange.com/quest...-the-true-mean

*2 assuming jitter is i.i.d.

Last edited by jmac698; 28th July 2017 at 13:06.
jmac698 is offline   Reply With Quote
Old 28th July 2017, 12:58   #169  |  Link
lordsmurf
Registered User
 
lordsmurf's Avatar
 
Join Date: Jan 2003
Posts: 120
Quote:
Originally Posted by jmac698 View Post
Pans would obviously make it fail
Not necessarily.

It would make it again hit the CPU and/or RAM, but this one is (probably?) easy. Analyze by frame temporally, then by the line. If the entire frame shows signs of panning, then said motion should be accounted for. Yet another %/slider/option should be present, to provide threshold for "what is a pan". Because, as you say, all clips are different.

To not waste CPU cycles, put the entire frame range into RAM.

I have 16gb RAM, i7-6700k CPU. I routinely get annoyed by Avisynth filters not taking advantage of the resources, usually using less than 25% (even with MT options).
__________________
Back in town.
If you want my advice, then find me at the DigitalFAQ forum. Glad to assist.
lordsmurf is offline   Reply With Quote
Old 28th July 2017, 13:10   #170  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
@lordsmurf
Great idea! Though I thought of something like that after I posted.
jmac698 is offline   Reply With Quote
Old 28th July 2017, 14:53   #171  |  Link
lordsmurf
Registered User
 
lordsmurf's Avatar
 
Join Date: Jan 2003
Posts: 120
I'm going to fully recapture the footage I want you to test a correction with. I pulled the tape tonight, and will get back to you sometime soon. I'll see what my capturing options are.
__________________
Back in town.
If you want my advice, then find me at the DigitalFAQ forum. Glad to assist.
lordsmurf is offline   Reply With Quote
Old 28th July 2017, 15:06   #172  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
My suggestion is, try to get black on both sides, and turn up the brightness (with procamp) so we can distinguish the blacker than black from the dark. In cases I've seen, there was a clear difference.

Ohh.. now I had another idea. By the same process, if you capture a scene many times (too many to be practical, tbh), it should naturally average out as well.

How would you feel about capturing a few seconds 9 times? haha
jmac698 is offline   Reply With Quote
Old 28th July 2017, 15:11   #173  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
I think what I need now is an edge filter followed by RANSAC (this is an algorithm to get rid of outliers; what should remain is true corresponding edges) and can use that for alignment.
The other way is correlation/fourier, that tries to find the best "sum" that matches position and scale.
jmac698 is offline   Reply With Quote
Old 28th July 2017, 20:45   #174  |  Link
lordsmurf
Registered User
 
lordsmurf's Avatar
 
Join Date: Jan 2003
Posts: 120
Quote:
Originally Posted by jmac698 View Post
RANSAC (this is an algorithm to get rid of outliers;
Ah, that gets into statistics, which I do have some skill at.

You need to be careful about the stats method across all frames. You don't want 100% averaging, as outliers can drag the norm in its direction. What you want is a truncated mean of some sort, and I find a winsorized mean to often be more accurate.

Quote:
The other way is correlation/fourier, that tries to find the best "sum" that matches position and scale.
I'm not sure a fourier transform is the way to go. For one thing, even with a still image, it just kills a CPU. Does the sum exclude outliers?

Quote:
Ohh.. now I had another idea. By the same process, if you capture a scene many times (too many to be practical, tbh), it should naturally average out as well.
I think that would be more academic than anything else.

Generally damaged footage will already be digital, with access to the tape unavailable. Aside from really bad nth gen errors embedded/compounded on the tape, a quality S-VHS VCR with line TBC (or Es10 on passthrough) would easily correct the timing problem, thus negating the filter's need.

I mean, yeah, sure, I can do it. But I don't think it will help much. In my experience, multi capturing doesn't result in a big difference, if at all. Again, embedded errors, meaning the positioning would remain mostly or entirely unchanged.
__________________
Back in town.
If you want my advice, then find me at the DigitalFAQ forum. Glad to assist.
lordsmurf is offline   Reply With Quote
Old 28th July 2017, 22:36   #175  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
If your sample has embedded errors, don't bother then.

I've been looking at robust statistics lately, and ran across windsoring, but it has a bias. That would still leave the line with an outlier of jitter sticking out (just not as much).

Robust methods are rated by their breakdown point, which means what % of the data has to be corrupted in order to give you a bad estimate. A method called least median of squares has 50% breakdown, which is really good.

Imagine a graph where the x positions are non-linear, based on the cummulative sum of a guassian. It will be .5 halfway, increase quickly then taper off..
Code:
*   *  ****  *   *
The Y positions, you plot your data, but sorted and cummulative. If the data comes from a guassian, this should now form a straight diagonal line. It's similar to the concept of log/log graph paper. This straight line has an intercept (where the line touches the x-axis, ideally 0), which is the mean, and a slope (normally 1) which is the standard deviation. At this point you would call it the linear regression method. If the mean of the data is higher, the line should shift to the right. If the sample is 'fatter' (more variation), the area under the curve is always higher, so the line slope should increase more, and slant up.

Modify this concept again, but choose only the points that cause the least deviation from a straight line. Yes that means trying every set of points and see how good of a line they make. However this is now called the least median of squares. But you can see how it's getting rid of outliers, it chooses only the most 'clean' data. It works like a majority vote; outliers are completely eliminated from consideration. Using the chosen points to form a slope, is like taking an average of the best data. You can see it's adaptive; it uses the most clean data available and then reduces to a simple average (the sample mean is the population mean, it's consistent, efficient, unbiased, but not robust).

You might think that a guassian is infinite, however our data isn't, and it's not an actual graph but a virtual calculation, so you can include all the data arbitrarily.

This method of choosing only certain points is what puts 'median' in the name. It's also what gives it it's robustness, as you know a median isn't affected by outliers. In the worst case think of 3 points in a triangle; the longest side would be the best fit line.

There is an even more powerful method called maximum likelihood, but if you solve it for the case of Gaussian/normal, you get the method above.

This method is the best you can do; it's consistent, which means it always gets closer to the population mean (the true mean) as you use more data, it's efficient, which means it reaches the true conclusion the quickest as you use more data, it's unbiased.

Last edited by jmac698; 28th July 2017 at 23:19.
jmac698 is offline   Reply With Quote
Old 28th July 2017, 23:29   #176  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
Ok I finally realized your point, this whole process is moot because if you have the tape then you can use a TBC to begin with. So baked in jitter is the only problem that needs solving here? Does TBC also fail sometimes?
I mean, it was still useful to me and some other people who couldn't get a TBC for whatever reason.

Last edited by jmac698; 28th July 2017 at 23:31.
jmac698 is offline   Reply With Quote
Old 29th July 2017, 04:54   #177  |  Link
lordsmurf
Registered User
 
lordsmurf's Avatar
 
Join Date: Jan 2003
Posts: 120
Quote:
Originally Posted by jmac698 View Post
I've been looking at robust statistics lately, and ran across windsoring, but it has a bias.
That's the point. You need bias. If the frame exceeds x pixels, then it should be ignored. Otherwise erratic skew can throw off the mean used for the line readjustment.

This should also be a variable switch, letting you adjust it higher or lower. Note the minimum must be 3 pixels, while something like 20 disables it.

Quote:
Originally Posted by jmac698 View Post
this whole process is moot because if you have the tape then you can use a TBC to begin with.
Exactly.

Quote:
So baked in jitter is the only problem that needs solving here?
Yes.

Quote:
Does TBC also fail sometimes?
TBC is useless on embedded errors, and fails 99% of the time. It's rare to have an embedded error respond to TBC corrections.

Quote:
I mean, it was still useful to me and some other people who couldn't get a TBC for whatever reason.
You'd be wasting time reinventing the wheel solely for the cheapness of others. You can buy a good ES10 for under $150, or JVC for under $300. If you're cheap, then buy it, use it, resell it.

Let's focus on currently-uncorrectable problems with video, creating something that does not exist to address it. And software TBC (lack thereof) is still a major issue.

I think we're on to something here.

I know MUCH more than I did 6-7 years ago when we met and this project was started, and you're probably the same. Let's finish it.
__________________
Back in town.
If you want my advice, then find me at the DigitalFAQ forum. Glad to assist.
lordsmurf is offline   Reply With Quote
Reply

Tags
tbc

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 04:10.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.