Choose the best frames from identical sources

[thecoreyburton]

Assuming I have two sources that are identical in timing and frames but were compressed differently, is there a plugin or script that can make a single video from the clearest bits of both sources?

A good example of this is Youtube in regards to videos that have both VP9 and H264 streams - each codec has strengths and weaknesses with the large amount of compression they're using, so would it be possible to pick the best frames from each to make an output video that's slightly better in quality than either of the originals are individually?

Also note that this is only for cases when a better source can't be found. There's no substitute for quality input.

[johnmeyer]

I think someone else asked for this a few months back, and StainlessS came up with a script. I didn't have time to do an extensive search, but I think this thread was inspired by the original request:

SpatialAlign v1.0 : Spatial Alignment of clips (http://forum.doom9.org/showthread.php?t=172136)

It doesn't actually analyze the two frames to figure out which is "better" (that would be really tough to do), but it does let you get them lined up.

[feisty2]

The point is that the mathematical formula of "better quality" is yet undefined

[Sharc]

One would need to have a clean reference to which the 2 clips can be compared, in order to select the "more similar" picture (assuming that this is the better one), e.g. based on PSNR or similar.
Obviously we normally don't have access to the clean reference clip. Also, the switching between the selected pictures originating from different encoders might create unexpected annoying effects.

[colours]

Ditto the two posts above.

I actually do this regularly—such is the life of working with broadcast television and disgusting interlacing-related artifacts—but there are a lot of caveats and you have to carefully consider the tradeoffs you're making.

If you have multiple sources that have more or less the same amount of detail, but lots of blocking and mosquito noise, you can do this:

a = ffvideosource("foo.mkv")
b = dgsource("bar.m2ts")
c = mpeg2source("baz.d2v")
ab = repair(a, b, 1)
bc = repair(b, c, 1)
ca = repair(c, a, 1)
raveragew(ab, 22.0/64.0, bc, 21.0/64.0, ca, 21.0/64.0, lsb_out=true)
ditherpost(mode=6)

You can adjust to however many input clips you have in the obvious way, but there's usually rapidly diminishing returns beyond three input clips. This unfortunately tends to wash out a bit of detail, but the reduced noise is usually worth it.

If you have two sources where one of them has more fine detail than the other, and they're both reasonably free of blocking and mosquito noise, then you can try this:

function combine(clip a, clip b)
{
function blurl(clip c, bool lsb_inout)
{
lsb_inout ? c : c.dither_convert_8_to_16()
dither_box_filter16(radius=3)
dither_box_filter16(radius=3)
lsb_inout ? last : ditherpost(mode=6)
}

ah = mt_makediff(a,a.blurl(false),u=3,v=3)
al = mt_makediff(a,ah,u=3,v=3)
bh = mt_makediff(b,b.blurl(false),u=3,v=3)
bl = mt_makediff(b,bh,u=3,v=3)
mt_lutxy(ah, bh, "x 128 - 3 ^ y 128 - 3 ^ + x 128 - 2 ^ y 128 - 2 ^ 0.0001 + + / 128 +", u=3, v=3)
mt_adddiff(raveragew(al,33.0/64.0,bl,31.0/64.0), last, u=3, v=3)
}

Variations of this idea can be used to automatically colour-correct one source to match another source, etc.

For stuff like merging the VP9 and H.264 streams on YouTube, however, I don't think there's a good way of doing that (yet). Last time I tried, FFMS2 barfed on the VP9 input files, duplicating and discarding random frames everywhere, but I suppose that even if there were a source filter that worked with VP9 files, the artifacts between the two streams are too similar for either of the above approaches to be of any use.

[StainlessS]

A couple of threads on similar

http://forum.doom9.org/showthread.php?t=170261&highlight=findcuts

http://forum.doom9.org/showthread.php?t=172088&highlight=findcuts

[jmac698]

There's another case where this is useful, digital TV streams.

One way, a 24fps source on a 60fps channel. I found some patterns, where the picture was always clearest (in mpeg2) on the P frame after an I frame. This goes in groups of 15, which is not always in sync with the 24fps of the source, but you can improve a lot of the frames. In fact you get the same source frame repeated 2 or 3 times and can select the best one. So if the new source begins on an I frame, the frame after that is clearest.

Another case is where you merge two recordings of the same material. The source is exactly the same, the encodings (even with the same parameters) are different. Same idea as above, take the P after the I frame from each source. Or you can even average them.

[MysteryX]

For stuff like merging the VP9 and H.264 streams on YouTube, however, I don't think there's a good way of doing that (yet). Last time I tried, FFMS2 barfed on the VP9 input files, duplicating and discarding random frames everywhere, but I suppose that even if there were a source filter that worked with VP9 files, the artifacts between the two streams are too similar for either of the above approaches to be of any use.
LSMASHSOURCE works well to open VP9

If someone comes up with a script that generates a better video by taking both YouTube videos, that would be interesting.

However, there will be quality loss by the re-encoding that may nullify whatever small quality gain you may get.

[thecoreyburton]

MysteryX, you're right. I use LWLibAVVideoSource and it doesn't have the frame duplication problem that's present in FFMS2. The videos are out of sync on first load up of the script (in Virtualdub for instance), but if you close it and reopen it they're perfectly in sync for that load and all subsequent loads. If you're getting ghosting with the script then this is the problem and a reload fixes it.

I'm testing a bit of a variation of the script Colours wrote that works nicely. I managed to find two videos produced at the same time by the same channel that exhibit compression artifacts in both formats (one has a portion that looks okay in VP9 but not H264 and the other has a portion that looks okay in H264 but not VP9). The issue here is that neither of the clips themselves are guaranteed to be consistently superior to the other.

I've managed to remove a lot of the compression artifacts, but there is some detail loss - it's not too drastic and it certainly outweights the initial problem (at least for me), but I'm experimenting with sharpening the source slightly at various points and changing the ditherpost mode in order to preserve any detail I can.

Edit: Here's the script and the video files (https://mega.nz/#!YwwHjDpR!YkuygqQYAA6iv1KRHqeoyxej5I71zGQcIceaN9QRvnc) as promised, along with some more information.

I acquired these videos using 4K Video Downloader (I compared the videos to the ones aquired through youtube-dl which grabs the streams directly and confirmed that there's no conversion or transcoding of the videos after the downloading is done). The MKV contains the VP9 video with a transcoded vorbis audio track. The MP4 contains the H264 video with the original AAC audio track, making it the better choice for audio source if it's required.

In S05E02 at frame 722 the VP9/MKV looks significantly better than the H264/MP4. In S05E04 at frame 1568 the H264/MP4 looks significantly better than the VP9/MKV. Detail loss is very noticable in this clip, but I think that's related to how bad one of the sources are. Colors are also more vibrant at frame 700 of S05E02 in the VP9 clip, the red bar getting a more blue-ish hue in the H264 clip and in the output clip.

Here's the WIP script, this seems to produce the cleanest result with relatively good detail retention, but it's still not perfect. A lot of it was trial an error there's a good chance I'm not using one of the plugins in the most efficient way. Anything after DitherPost is just a bit of personal post processing on my behalf and isn't necessary to repair the videos.

MKVVideo=LWLibAVVideoSource("erb_s05e02.mkv")
MP4Video=LWLibAVVideoSource("erb_s05e02.mp4")
RepairMKV=Repair(MKVVideo, MP4Video, 4)
RepairMP4=Repair(MP4Video, MKVVideo, 4)
Audio=FFAudioSource("erb_s05e02.mp4")
Video=RAverageW(RepairMKV, 0.5, RepairMP4, 0.5, LSB_Out=True)
AudioDub(Video,Audio)
DitherPost(Mode=6)
Santiag()
LimitedSharpenFaster(ss_x=2.0, ss_y=2.0, strength=16, overshoot=0, undershoot=0, soft=0, edgemode=0)

S05E02, frame 722, MP4/H264:
http://i63.tinypic.com/21msocm.png
S05E02, frame 722, MKV/VP9:
http://i66.tinypic.com/2h7kpe1.png
S05E02, frame 722, script output:
http://i67.tinypic.com/2ldvm7a.png

S05E04, frame 1568, MP4/H264:
http://i65.tinypic.com/2e5odaq.png
S05E04, frame 1568, MKV/VP9:
http://i68.tinypic.com/a26cs4.png
S05E04, frame 1568, script output:
http://i65.tinypic.com/34sihph.png

The result is definitely much more pleasant to watch without the problem sections, but there's definitely quite a bit of detail lost. I'm not sure how to go about increasing the detail much more without reintroducing the artifacts that were removed, but I'm happy with the result so far - so thank you all!

[thecoreyburton]

ConvertToRGB(matrix="Rec709")

Adding this to the end of the script corrects the colors to the more accurate ones found in the VP9/MKV.

[GMJCZP]

Thecoreyburton:
For curiosity, why you use Santiag?

[thecoreyburton]

In the particular sample videos, the titles exhibit some aliasing and Santiag helps with that. Anything after DitherPost in my script isn't necessary in the restoration process but is something I would use on these videos in particular.

[GMJCZP]

Ok. Thanks for your reply.