Karyudo
16th July 2004, 21:55
(this has been mostly reposted, separating it from the existing TooT thread.)
Thanks to E-male's recent work in this area, there are now some cool new tools that assume you have access to more than one copy of a source!
His TooT (Two out of Three) filter works great where you have three copies of the same source: two will be nearly identical, and it's clear the oddball to be ignored is the third. However, it's more common (not to mention cheaper) to have just two copies of a given source. For example, I have two copies of the "faces" THX LD of The Empire Strikes Back, which is already more than most people want to pay good money for!
Clearly, there is almost certainly enough *picture* data to create a clean copy from two flawed sources, but without a way to "inform" the filter, there's not enough *decision-making* data (with three sources, it's the redundancy that provides the decision-making data). Without any decision-making data, a filter can't choose which of the two copies is correct when it finds differences between them.
I've thought of some possible solutions:
1) a "control" clip that is only used to decide on differences between two clips; and/or
2) some temporal analysis to see which of two clips is more likely to be flawed; and/or
3) some spatial analysis to see which of two clips is more likely to be flawed.
In case (1), the control clip could be a lesser-quality capture (for example, a lower-quality capture from VHS used to compare two higher-quality LD captures), or a capture from a different release of the same material (for example, I have a copy of The Empire Strikes Back Special Edition that could help a filter decide on which of two "faces" captures is right at any given point).
In case (2), deriving some sort of numerical value for frames 1 and 3 (that is, nearly-identical frame 1's and frame 3's one frame before and after dissimilar frame 2's) might indicate whether frame 2a is more likely correct than frame 2b. For example, in a low-motion scene, if the values for frame 1 and 3 are 115 and 117 respectively, and the values calculated for frames 2a and 2b are 12 and 118, then it's more likely that picture data from frame 2b is a better choice than data from frame 2a. Some user-selectable threshold would be desirable, to reduce artifacts (like motion blur from temporal smoothers).
In case (3), comparing a pixel to the surrounding pixels in both sources may indicate which source is more likely correct. For example, if a pixel differs from its 8 nearest (spatial) neighbours by no more than 5 levels in one source, but differs by 100 levels in the other source, then the first source is likely the one to choose. Again, some user-selectable threshold would be desirable, to reduce artifacts.
In fact, just using the temporal and spatial techniques developed for other filters should probably be enough, because a lot of those filters are both trying to identify problems *and create data to fix them*. In the case of two flawed sources, the problem is merely to decide which of the two is more likely correct; no picture information needs to be created. As long as even a quick-and-dirty analysis is reasonably sound, the resulting picture should be perfect.
I'm trying to simulate (theoretically -- I'm no coder) a decision process that a person would use to decide between two frame candidates.
If you were to step through a pair of sources, knowing that it was equally likely that each was flawed at any given frame, then there would be some clues available to tell you which was the better choice:
1. If you had another copy -- even though it didn't match exactly -- you'd be able to see which of the two candidates looked more like the third copy. The more the flawed copy deviated, the easier it would be. And the less it deviated, the less it would matter! Of course, making the filter "see" the clip in a way that would let it approximate the decision-making process a person would use is the tough part (as underlined by E-male in the TooT thread).
2. If the frames before and after the candidate frames didn't have a flaw, you'd probably be able to see the flaw pop on for the one frame in question, and you'd know to keep the other frame candidate.
3. If there's junk flying around the frame anyway (like an explosion), then the size, shape, hardness and intensity of the flaw would give you information about whether it's a video problem, a film problem, or supposed to be there! For example, a black or white line of a dozen pixels or so, confined to a single horizontal line, is probably a dropout, and should be fixed.
I don't know how good just a Type 1 comparison of something like a whole-frame histogram would be in determining which source is right. A user-selected threshold value would probably be good. Because you're only ever replacing from a frame that *is* the same (except for the flaw), the control source shouldn't need to be colour-corrected or super-close on the alignment, etc.
At least, I hope not!
In fact, since most (all??) video flaws and most film flaws are primarily black or white, how about comparing monochrome histograms, with a heavy bias to the 0-54 and 200-255 range? I suspect that it would be fairly informative to look at these highly-specific histograms for decision-making. I mean, that's about what you're doing when you look at the frame yourself and note, "There's a black spot here," isn't it?
Two further details/abstractions might be possible:
- using two different sources (for example, from two different video transfers), eliminate dust, etc., included as part of the video on pre-recorded sources. Note that none of the analysis above can correct dust, etc., that's actually been recorded (as opposed to dropouts or rot) on a tape or LD!
That might be a bit of a magic bullet, and would definitely involve *a lot* of pre-processing, in order to get the sources to match closely enough for this sort of comparison. So perhaps a less-automated solution might be better?
- The Decomb package by Donald Graft (aka neuron2) allows dumping the field-matching results to a text file, and allows a user-created text file to override the field-matching process. In a similar way, a list of data from frames that differ between two versions (including the filter's selection) could be exported to the proposed comparison filter, and then this (hopefully short) list of frames could be reviewed by the user, to semi-automatically clean the video. An
imported text file could include rows with [frame_number, source] included (for example, "2354, a") that would tell the filter which of the input sources should be used. This would get a lot more complicated in the case of both frames being flawed, but in different parts of the picture.
Anyway, that's a lot of rambling on my part. It's what I do best; I have no coding ability...
Thanks to E-male's recent work in this area, there are now some cool new tools that assume you have access to more than one copy of a source!
His TooT (Two out of Three) filter works great where you have three copies of the same source: two will be nearly identical, and it's clear the oddball to be ignored is the third. However, it's more common (not to mention cheaper) to have just two copies of a given source. For example, I have two copies of the "faces" THX LD of The Empire Strikes Back, which is already more than most people want to pay good money for!
Clearly, there is almost certainly enough *picture* data to create a clean copy from two flawed sources, but without a way to "inform" the filter, there's not enough *decision-making* data (with three sources, it's the redundancy that provides the decision-making data). Without any decision-making data, a filter can't choose which of the two copies is correct when it finds differences between them.
I've thought of some possible solutions:
1) a "control" clip that is only used to decide on differences between two clips; and/or
2) some temporal analysis to see which of two clips is more likely to be flawed; and/or
3) some spatial analysis to see which of two clips is more likely to be flawed.
In case (1), the control clip could be a lesser-quality capture (for example, a lower-quality capture from VHS used to compare two higher-quality LD captures), or a capture from a different release of the same material (for example, I have a copy of The Empire Strikes Back Special Edition that could help a filter decide on which of two "faces" captures is right at any given point).
In case (2), deriving some sort of numerical value for frames 1 and 3 (that is, nearly-identical frame 1's and frame 3's one frame before and after dissimilar frame 2's) might indicate whether frame 2a is more likely correct than frame 2b. For example, in a low-motion scene, if the values for frame 1 and 3 are 115 and 117 respectively, and the values calculated for frames 2a and 2b are 12 and 118, then it's more likely that picture data from frame 2b is a better choice than data from frame 2a. Some user-selectable threshold would be desirable, to reduce artifacts (like motion blur from temporal smoothers).
In case (3), comparing a pixel to the surrounding pixels in both sources may indicate which source is more likely correct. For example, if a pixel differs from its 8 nearest (spatial) neighbours by no more than 5 levels in one source, but differs by 100 levels in the other source, then the first source is likely the one to choose. Again, some user-selectable threshold would be desirable, to reduce artifacts.
In fact, just using the temporal and spatial techniques developed for other filters should probably be enough, because a lot of those filters are both trying to identify problems *and create data to fix them*. In the case of two flawed sources, the problem is merely to decide which of the two is more likely correct; no picture information needs to be created. As long as even a quick-and-dirty analysis is reasonably sound, the resulting picture should be perfect.
I'm trying to simulate (theoretically -- I'm no coder) a decision process that a person would use to decide between two frame candidates.
If you were to step through a pair of sources, knowing that it was equally likely that each was flawed at any given frame, then there would be some clues available to tell you which was the better choice:
1. If you had another copy -- even though it didn't match exactly -- you'd be able to see which of the two candidates looked more like the third copy. The more the flawed copy deviated, the easier it would be. And the less it deviated, the less it would matter! Of course, making the filter "see" the clip in a way that would let it approximate the decision-making process a person would use is the tough part (as underlined by E-male in the TooT thread).
2. If the frames before and after the candidate frames didn't have a flaw, you'd probably be able to see the flaw pop on for the one frame in question, and you'd know to keep the other frame candidate.
3. If there's junk flying around the frame anyway (like an explosion), then the size, shape, hardness and intensity of the flaw would give you information about whether it's a video problem, a film problem, or supposed to be there! For example, a black or white line of a dozen pixels or so, confined to a single horizontal line, is probably a dropout, and should be fixed.
I don't know how good just a Type 1 comparison of something like a whole-frame histogram would be in determining which source is right. A user-selected threshold value would probably be good. Because you're only ever replacing from a frame that *is* the same (except for the flaw), the control source shouldn't need to be colour-corrected or super-close on the alignment, etc.
At least, I hope not!
In fact, since most (all??) video flaws and most film flaws are primarily black or white, how about comparing monochrome histograms, with a heavy bias to the 0-54 and 200-255 range? I suspect that it would be fairly informative to look at these highly-specific histograms for decision-making. I mean, that's about what you're doing when you look at the frame yourself and note, "There's a black spot here," isn't it?
Two further details/abstractions might be possible:
- using two different sources (for example, from two different video transfers), eliminate dust, etc., included as part of the video on pre-recorded sources. Note that none of the analysis above can correct dust, etc., that's actually been recorded (as opposed to dropouts or rot) on a tape or LD!
That might be a bit of a magic bullet, and would definitely involve *a lot* of pre-processing, in order to get the sources to match closely enough for this sort of comparison. So perhaps a less-automated solution might be better?
- The Decomb package by Donald Graft (aka neuron2) allows dumping the field-matching results to a text file, and allows a user-created text file to override the field-matching process. In a similar way, a list of data from frames that differ between two versions (including the filter's selection) could be exported to the proposed comparison filter, and then this (hopefully short) list of frames could be reviewed by the user, to semi-automatically clean the video. An
imported text file could include rows with [frame_number, source] included (for example, "2354, a") that would tell the filter which of the input sources should be used. This would get a lot more complicated in the case of both frames being flawed, but in different parts of the picture.
Anyway, that's a lot of rambling on my part. It's what I do best; I have no coding ability...