Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 16th June 2016, 04:52   #1  |  Link
thecoreyburton
Guest
 
Posts: n/a
Choose the best frames from identical sources

Assuming I have two sources that are identical in timing and frames but were compressed differently, is there a plugin or script that can make a single video from the clearest bits of both sources?

A good example of this is Youtube in regards to videos that have both VP9 and H264 streams - each codec has strengths and weaknesses with the large amount of compression they're using, so would it be possible to pick the best frames from each to make an output video that's slightly better in quality than either of the originals are individually?

Also note that this is only for cases when a better source can't be found. There's no substitute for quality input.

Last edited by thecoreyburton; 16th June 2016 at 05:03.
  Reply With Quote
Old 16th June 2016, 06:50   #2  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
I think someone else asked for this a few months back, and StainlessS came up with a script. I didn't have time to do an extensive search, but I think this thread was inspired by the original request:

SpatialAlign v1.0 : Spatial Alignment of clips

It doesn't actually analyze the two frames to figure out which is "better" (that would be really tough to do), but it does let you get them lined up.
johnmeyer is offline   Reply With Quote
Old 16th June 2016, 06:54   #3  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
The point is that the mathematical formula of "better quality" is yet undefined
feisty2 is offline   Reply With Quote
Old 16th June 2016, 07:25   #4  |  Link
Sharc
Registered User
 
Join Date: May 2006
Posts: 3,997
One would need to have a clean reference to which the 2 clips can be compared, in order to select the "more similar" picture (assuming that this is the better one), e.g. based on PSNR or similar.
Obviously we normally don't have access to the clean reference clip. Also, the switching between the selected pictures originating from different encoders might create unexpected annoying effects.

Last edited by Sharc; 16th June 2016 at 07:29.
Sharc is offline   Reply With Quote
Old 16th June 2016, 09:02   #5  |  Link
colours
Registered User
 
colours's Avatar
 
Join Date: Mar 2014
Posts: 308
Ditto the two posts above.

I actually do this regularly—such is the life of working with broadcast television and disgusting interlacing-related artifacts—but there are a lot of caveats and you have to carefully consider the tradeoffs you're making.

If you have multiple sources that have more or less the same amount of detail, but lots of blocking and mosquito noise, you can do this:

Code:
a = ffvideosource("foo.mkv")
b = dgsource("bar.m2ts")
c = mpeg2source("baz.d2v")
ab = repair(a, b, 1)
bc = repair(b, c, 1)
ca = repair(c, a, 1)
raveragew(ab, 22.0/64.0, bc, 21.0/64.0, ca, 21.0/64.0, lsb_out=true)
ditherpost(mode=6)
You can adjust to however many input clips you have in the obvious way, but there's usually rapidly diminishing returns beyond three input clips. This unfortunately tends to wash out a bit of detail, but the reduced noise is usually worth it.

If you have two sources where one of them has more fine detail than the other, and they're both reasonably free of blocking and mosquito noise, then you can try this:

Code:
function combine(clip a, clip b)
{
function blurl(clip c, bool lsb_inout)
{
lsb_inout ? c : c.dither_convert_8_to_16()
dither_box_filter16(radius=3)
dither_box_filter16(radius=3)
lsb_inout ? last : ditherpost(mode=6)
}

ah = mt_makediff(a,a.blurl(false),u=3,v=3)
al = mt_makediff(a,ah,u=3,v=3)
bh = mt_makediff(b,b.blurl(false),u=3,v=3)
bl = mt_makediff(b,bh,u=3,v=3)
mt_lutxy(ah, bh, "x 128 - 3 ^ y 128 - 3 ^ + x 128 - 2 ^ y 128 - 2 ^ 0.0001 + + / 128 +", u=3, v=3)
mt_adddiff(raveragew(al,33.0/64.0,bl,31.0/64.0), last, u=3, v=3)
}
Variations of this idea can be used to automatically colour-correct one source to match another source, etc.

For stuff like merging the VP9 and H.264 streams on YouTube, however, I don't think there's a good way of doing that (yet). Last time I tried, FFMS2 barfed on the VP9 input files, duplicating and discarding random frames everywhere, but I suppose that even if there were a source filter that worked with VP9 files, the artifacts between the two streams are too similar for either of the above approaches to be of any use.
__________________
Say no to AviSynth 2.5.8 and DirectShowSource!

Last edited by colours; 16th June 2016 at 09:18.
colours is offline   Reply With Quote
Old 16th June 2016, 13:45   #6  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
A couple of threads on similar

http://forum.doom9.org/showthread.ph...light=findcuts

http://forum.doom9.org/showthread.ph...light=findcuts
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 17th June 2016, 16:08   #7  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,867
There's another case where this is useful, digital TV streams.

One way, a 24fps source on a 60fps channel. I found some patterns, where the picture was always clearest (in mpeg2) on the P frame after an I frame. This goes in groups of 15, which is not always in sync with the 24fps of the source, but you can improve a lot of the frames. In fact you get the same source frame repeated 2 or 3 times and can select the best one. So if the new source begins on an I frame, the frame after that is clearest.

Another case is where you merge two recordings of the same material. The source is exactly the same, the encodings (even with the same parameters) are different. Same idea as above, take the P after the I frame from each source. Or you can even average them.
jmac698 is offline   Reply With Quote
Old 18th June 2016, 16:42   #8  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,559
Quote:
Originally Posted by colours View Post
For stuff like merging the VP9 and H.264 streams on YouTube, however, I don't think there's a good way of doing that (yet). Last time I tried, FFMS2 barfed on the VP9 input files, duplicating and discarding random frames everywhere, but I suppose that even if there were a source filter that worked with VP9 files, the artifacts between the two streams are too similar for either of the above approaches to be of any use.
LSMASHSOURCE works well to open VP9

If someone comes up with a script that generates a better video by taking both YouTube videos, that would be interesting.

However, there will be quality loss by the re-encoding that may nullify whatever small quality gain you may get.
MysteryX is offline   Reply With Quote
Old 19th June 2016, 02:32   #9  |  Link
thecoreyburton
Guest
 
Posts: n/a
MysteryX, you're right. I use LWLibAVVideoSource and it doesn't have the frame duplication problem that's present in FFMS2. The videos are out of sync on first load up of the script (in Virtualdub for instance), but if you close it and reopen it they're perfectly in sync for that load and all subsequent loads. If you're getting ghosting with the script then this is the problem and a reload fixes it.

I'm testing a bit of a variation of the script Colours wrote that works nicely. I managed to find two videos produced at the same time by the same channel that exhibit compression artifacts in both formats (one has a portion that looks okay in VP9 but not H264 and the other has a portion that looks okay in H264 but not VP9). The issue here is that neither of the clips themselves are guaranteed to be consistently superior to the other.

I've managed to remove a lot of the compression artifacts, but there is some detail loss - it's not too drastic and it certainly outweights the initial problem (at least for me), but I'm experimenting with sharpening the source slightly at various points and changing the ditherpost mode in order to preserve any detail I can.

Edit: Here's the script and the video files as promised, along with some more information.

I acquired these videos using 4K Video Downloader (I compared the videos to the ones aquired through youtube-dl which grabs the streams directly and confirmed that there's no conversion or transcoding of the videos after the downloading is done). The MKV contains the VP9 video with a transcoded vorbis audio track. The MP4 contains the H264 video with the original AAC audio track, making it the better choice for audio source if it's required.

In S05E02 at frame 722 the VP9/MKV looks significantly better than the H264/MP4. In S05E04 at frame 1568 the H264/MP4 looks significantly better than the VP9/MKV. Detail loss is very noticable in this clip, but I think that's related to how bad one of the sources are. Colors are also more vibrant at frame 700 of S05E02 in the VP9 clip, the red bar getting a more blue-ish hue in the H264 clip and in the output clip.

Here's the WIP script, this seems to produce the cleanest result with relatively good detail retention, but it's still not perfect. A lot of it was trial an error there's a good chance I'm not using one of the plugins in the most efficient way. Anything after DitherPost is just a bit of personal post processing on my behalf and isn't necessary to repair the videos.

Quote:
MKVVideo=LWLibAVVideoSource("erb_s05e02.mkv")
MP4Video=LWLibAVVideoSource("erb_s05e02.mp4")
RepairMKV=Repair(MKVVideo, MP4Video, 4)
RepairMP4=Repair(MP4Video, MKVVideo, 4)
Audio=FFAudioSource("erb_s05e02.mp4")
Video=RAverageW(RepairMKV, 0.5, RepairMP4, 0.5, LSB_Out=True)
AudioDub(Video,Audio)
DitherPost(Mode=6)
Santiag()
LimitedSharpenFaster(ss_x=2.0, ss_y=2.0, strength=16, overshoot=0, undershoot=0, soft=0, edgemode=0)
S05E02, frame 722, MP4/H264:

S05E02, frame 722, MKV/VP9:

S05E02, frame 722, script output:


S05E04, frame 1568, MP4/H264:

S05E04, frame 1568, MKV/VP9:

S05E04, frame 1568, script output:


The result is definitely much more pleasant to watch without the problem sections, but there's definitely quite a bit of detail lost. I'm not sure how to go about increasing the detail much more without reintroducing the artifacts that were removed, but I'm happy with the result so far - so thank you all!

Last edited by thecoreyburton; 22nd June 2016 at 04:07.
  Reply With Quote
Old 21st June 2016, 10:19   #10  |  Link
thecoreyburton
Guest
 
Posts: n/a
Quote:
ConvertToRGB(matrix="Rec709")
Adding this to the end of the script corrects the colors to the more accurate ones found in the VP9/MKV.
  Reply With Quote
Old 22nd June 2016, 00:25   #11  |  Link
GMJCZP
Registered User
 
GMJCZP's Avatar
 
Join Date: Apr 2010
Location: I have a statue in Hakodate, Japan
Posts: 744
Thecoreyburton:
For curiosity, why you use Santiag?
__________________
By law and justice!

GMJCZP's Arsenal
GMJCZP is offline   Reply With Quote
Old 22nd June 2016, 04:09   #12  |  Link
thecoreyburton
Guest
 
Posts: n/a
In the particular sample videos, the titles exhibit some aliasing and Santiag helps with that. Anything after DitherPost in my script isn't necessary in the restoration process but is something I would use on these videos in particular.
  Reply With Quote
Old 23rd June 2016, 00:49   #13  |  Link
GMJCZP
Registered User
 
GMJCZP's Avatar
 
Join Date: Apr 2010
Location: I have a statue in Hakodate, Japan
Posts: 744
Ok. Thanks for your reply.
__________________
By law and justice!

GMJCZP's Arsenal
GMJCZP is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 00:06.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.