Restoring quality original from multiple bad copies?

[redfordxx]

Hi, I 'd like to ask if anyone has an experience or idea how to get best result from multiple clip averaging.

Suppose you have four clips which are compressed from same source but in low quality and the source is not available.

Each clip has slightly different resolution, detail (and original noise) retention. Also there might be various denoising and (over)sharpening in the original encoding preprocessing...we don't know. And finally, different quantization artifacts, ringing, blocking, smearing... But, at least, as each clip has different resolution, the blocking and ringing is happening always on different places.

So, after upsizing the clips to same resolution and correcting the color levels if necessary, what would be next steps...

There could be blind averaging of all clips which definitely removes lot of artifacts but it also removes some detail if it is present somewhere and also brings some artifacts to the final results.

Other method would be to find the best clip, interleave with the other ones, temporal denoise (dfttest is my current favorite) and use the denoised best clip as result. But there would be unwanted detail loss due to denoising and/or less efficient artifact removing and definitely no use of detail in other clips.

I was thinking about some smart way like
"hey, this pixel is same in all clips except one so probably it is correct and let's ignore this one when averaging"
or
"this pixel is local max in two clips and local min in other two, so probably it is ringing or blocking"
or
"there is one edge or line or dot in one clip where in other one is nothing so maybe it is detail which is smeared in other clips

BTW: I can even virtually double or triple the number of sources by using the untouched, deblocked and deringed clips and compare them for differences.

So, anyone has some ideas to throw?
Thanx
R.

[Mounir]

Send me a sample (20sec) of the best copy you have ill see what i can do

[redfordxx]

The point is not to do whatever you can with one clip, but to use all available clips together.

I was thinking more about putting some ideas together first...But, if you are willing to try yourself, it should be preferably on video you like.

The source can be youtube, for example. For each clip there are multiple resolutions and for each resolution there are up to three different codecs (flv,mp4,vp8).
So when you find some clip you like, you might try to enhance it yourself.

[redfordxx]

The example of the script I was mentioning in first post can look like follows:srcfile1="trailer 360p.flv"
srcfile2="trailer 360p.mp4"
srcfile3="trailer 480p.webm"
srcfile4="trailer 480p.mp4"
srcfile5="trailer 480p.flv"

o1=FFVideoSource(srcfile1, pp="hb/vb:10:60", threads=1)
o2=FFVideoSource(srcfile2, pp="hb/vb:10:60", threads=1)
o3=FFVideoSource(srcfile3, pp="hb/vb:10:60", threads=1)
o4=FFVideoSource(srcfile4, pp="hb/vb:10:60", threads=1)
o5=FFVideoSource(srcfile5, pp="hb/vb:10:60", threads=1)

r1=o1.AddBorders(0,0,0,0).BicubicResize(640,480,0,0.75,0,0,0,0)#.Subtitle(srcfile1)#.AssumeFPS(25,1)
r2=o2.AddBorders(0,0,0,0).BicubicResize(640,480,0,0.75,0,0,0,0)#.Subtitle(srcfile2)#.AssumeFPS(25,1)
r3=o3.AddBorders(0,0,0,0).BicubicResize(640,480,0,0.75,0,0,0,0)#.Subtitle(srcfile3)#.AssumeFPS(25,1)
r4=o4.AddBorders(0,0,0,0).BicubicResize(640,480,0,0.75,0,0,0,0)#.Subtitle(srcfile4)#.AssumeFPS(25,1)
r5=o5.AddBorders(0,0,0,0).BicubicResize(640,480,0,0.75,0,0,0,0)#.Subtitle(srcfile5)#.AssumeFPS(25,1)

Interleave(r1,r3,r4,r5,r2)
dfttest(tbsize=5,sigma=3)
SelectEvery(5,3)
But this is definitely not, what can be achieved at best.

[librarian]

You could see this (http://forum.doom9.org/showthread.php?t=159780), particularly Didée's posts. I was forgetting to mention this thread ("http://forum.doom9.org/showthread.php?t=140282) too.

[redfordxx]

Thanx, I'll look into that...

[StainlessS]

@Librarian,
Your 2nd link is a bit broken

http://"http//forum. etc

should be

http://forum. etc

EDIT: or this

http://forum.doom9.org/showthread.php?t=140282

[redfordxx]

Hi, here my approach, to make it short only as a symbolic script (this is not working script, only the structure and my own functions are missing):# I have original clips o1,o2,o3,o4

#for each of them colormatrix, levels and other fixes

#cropping, padding, upsampling so they match in dimension like this
r2=o2.AddMirrors(0,4,0,6).BicubicResize(w,h,0,0.75,6.5,1.3,-4.9,-1.2)#AddMirrors is my function instead of AddBorders

#create average
ax=0.25
avg0=RAverageW(r1,ax,r2,ax,r3,ax,r4,ax)

#motioncompensate all clips frame per frame to the average clip
avg01234=Interleave(avg0,r1,r2,r3,r4)
avg01234mc=RefineAverage4(avg01234, -1 ,pnw=5,blk=8,lambda0=2000) #results interleaved mocomped average and mocomped clips

#find detail in all clips (local extremes via convolution), adjust/enhance it
str="1 1 1 1 1"# 1 1 1 1 1 1"
cavg01234mc=avg01234mc.mt_convolution(str,str,chroma="process")
dcavg01234mc=RAverageW(cavg01234mc,-1,avg01234mc,1,bias=128)
dcavg00000mc=dcavg01234mc.SelectEvery(5, 0,0,0,0,0) #this is repeating average to have it available to process along interleaved clips
adj01234mc=mt_lutxy(dcavg00000mc,dcavg01234mc,"x 128 < y 128 < y x min y ? x 128 > y 128 > y x max y ? y ? ?", chroma="process")# this is finding where the detail in average is sharper than in original
adjavg01234mc=RAverageW(cavg01234mc,1,adj01234mc,1,bias=-128)#merge back the found detail

# this is 3D denoising of the adjusted mocomped sources all together (not temporal), going to 16bit, average, going back to 8bit
dft1234mc=adjavg01234mc.SelectEvery(5, 4,0,1,3,2,4,0,1).dfttest(tbsize=5,sigma=1,lsb=true).SelectEvery(8, 2,4,3,5)
avg0dft=RAverageW(dft1234mc.SelectEvery(4,0),ax,dft1234mc.SelectEvery(4,1),ax,dft1234mc.SelectEvery(4,2),ax,dft1234mc.SelectEvery(4,3),ax,mode=8,lsb_in=true,lsb_out=false)#true).DitherPost()

#result dubbing audio from source of choice
AudioDub(avg0dft,o3)

Personally I was very surprised how much average helped in terms of denoising and deblocking.
The crutial thing, restoring the detail (red), is working but I am not sure whether good enough.
MC, denoise and 16bit might be in many cases slow overkill.

If anyone sees my point and happens to have better idea especially about the detail enhancement, please comment.

[redfordxx]

As I got a request on PM, I put here some of my functions used:
function RefineAverage4(clip itl, int outc, int "ppel", int "blk", int "ovrlp", int "lambda0", float "thSC", bool "Flow", int "Fmode", int "srch", bool "chrom", int "lvls", int "sp", int "pnw") {
ppel = default( ppel , 4 )
pnw = default( pnw , 50 )
lvls = default( lvls , 2 )
srch = default( srch , 4 )
sp = default( sp , 1 )
blk = default( blk , 8 )
olp = default( ovrlp , int(blk/2) )
lambda0= default( lambda0, 1000 )
thSC = default( thSC, 400 )
Fmode = default( Fmode, 0 )
chrom = default( chrom, true )
Flow = default( Flow , false )

super=itl.MSuper(pel=ppel,levels=lvls)
vb1=MAnalyse(super,isb=true ,delta=1,levels=lvls,chroma=chrom,blksize=blk,overlap=olp,search=srch,searchparam=sp,lambda=lambda0*blk*blk/64,pnew=pnw)
vb2=MAnalyse(super,isb=true ,delta=2,levels=lvls,chroma=chrom,blksize=blk,overlap=olp,search=srch,searchparam=sp,lambda=lambda0*blk*blk/64,pnew=pnw)
vb3=MAnalyse(super,isb=true ,delta=3,levels=lvls,chroma=chrom,blksize=blk,overlap=olp,search=srch,searchparam=sp,lambda=lambda0*blk*blk/64,pnew=pnw)
vb4=MAnalyse(super,isb=true ,delta=4,levels=lvls,chroma=chrom,blksize=blk,overlap=olp,search=srch,searchparam=sp,lambda=lambda0*blk*blk/64,pnew=pnw)
mb1= (Flow==true) ? MFlow(itl,super,vb1,mode=Fmode,thSCD1=thSC) : MCompensate(itl,super,vb1,thSCD1=thSC)
mb2= (Flow==true) ? MFlow(itl,super,vb2,mode=Fmode,thSCD1=thSC) : MCompensate(itl,super,vb2,thSCD1=thSC)
mb3= (Flow==true) ? MFlow(itl,super,vb3,mode=Fmode,thSCD1=thSC) : MCompensate(itl,super,vb3,thSCD1=thSC)
mb4= (Flow==true) ? MFlow(itl,super,vb4,mode=Fmode,thSCD1=thSC) : MCompensate(itl,super,vb4,thSCD1=thSC)
mb1=mb1.SelectEvery(5,0)
mb2=mb2.SelectEvery(5,0)
mb3=mb3.SelectEvery(5,0)
mb4=mb4.SelectEvery(5,0)
ax=0.25
avg=RAverageW(mb1,ax,mb2,ax,mb3,ax,mb4,ax)
Interleave(avg,mb1,mb2,mb3,mb4)
(outc==-1) ? last : SelectEvery(5, outc)
}



function ReplaceClip (clip c1, clip c2, int start, int end){
(start==0) ? c2.trim(0,end) : c1.trim(0,-start)+c2.trim(start,end)
(end==0) ? last : last+c1.trim(max(end+1,start-end),0)
}

function AddMirrors(clip c, int l, int t, int r, int b) {
c
(l==0) ? last : StackHorizontal(last.Crop(0,0,l,0).FlipHorizontal,last)
(r==0) ? last : StackHorizontal(last,last.Crop(last.width-r,0,0,0).FlipHorizontal)
(t==0) ? last : StackVertical(last.Crop(0,0,0,t).FlipVertical,last)
(b==0) ? last : StackVertical(last,last.Crop(0,last.height-b,0,0).FlipVertical)
}

[Gavino]

function ReplaceClip (clip c1, clip c2, int start, int end){
(start==0) ? c2.trim(0,end) : c1.trim(0,start-1)+c2.trim(start,end)
(end==0) ? last : last+c1.trim(max(end+1,start-end),0)
}
This doesn't work properly when start=1.
Use trim(0, -start) instead.

[jmac698]

I've done this to a lot of similar mpeg2 recordings. They were of separate analog broadcasts. The theory is that the source was always the same but the noise, interference, and blocking artefacts were always different. It worked quite well. I tried comparing median to average and the effect was subtle. Anyhow try median, it should completely eliminating ringing by definition with 3 copies in 3 different resolutions (and ringing in different spots).

[redfordxx]

OK, I will try...you were the one who asked me to implement median to RedAverage, right? Well, I am bad with time now:(

Would you recommend any plugin which does median of any number of clips? I saw thread where it was done via interlacing and masktools, so it was probably slow...

[librarian]

For any number of clips Medianblur (slow). For 3 to 5 clips better(IMHO) use the scripts I referred to in my previous post.