An idea for restoring blurry video: mocomp'd noise as detail

[Dark Shikari]

One trick I've found for making crappy-quality video more palatable (in particular the awful quality proleague Starcraft VODs) is to add a constant layer of grain to the background to simulate detail.

This works shockingly well with the right parameters (AddGrain(60,0.5,0.5,constant=true) is my current) ... but the instant something moves, whether it be the whole screen or an object (say, someone's face), the illusion breaks, since the detail doesn't move with the object.

Would it be possible to use MVTools to motion-compensate this constant fake grain layer to truly create the false perception of detail? This could significantly improve the visual quality of badly encoded videos.

[Fizick]

short answer is NO.
(new tracking function(s) must be implemented).
it was discussed some similar tasks like colorize of black-white movies, etc.

it is quite trivial for constant scene (talking head) with fixed base frame, but not in general.

[Dark Shikari]

short answer is NO.
(new tracking function(s) must be implemented).
it was discussed some similar tasks like colorize of black-white movies, etc.

it is quite trivial for constant scene (talking head) with fixed base frame, but not in general.Would it be possible to solely compensate for global motion (pans only) and nothing else? That would be sufficient for 99% of my source video here, as the objects in Starcraft games are small enough that the illusion holds even though they are moving with a constant grain layer superimposed on them.

[Sagekilla]

Tracking would require some sort of segmentation of the image then of sorts, or am I wrong here?

[*.mp4 guy]

The thread title is not as informative as it could be, you might get more suggestions if it reflected what you are trying to do. "motion compensated noise as detail" "stabilizing added noise to simulate detail" etc.

I'm only mentioning this because I thought this thread was about something completly different based on the title, and other people who could help might make the opposite mistake.

It might be worth trying something like this pseudo code.


source=last
grainy = source.addgrain(128, 0, 0)
fv1 = source.mvtools
fv2 = source.mvtools
bv1 = source.mvtools
bv2 = source.mvtools

grainy 2 = grainy.mvdegrain2(fv1...)

grainy 3 = grainy2.mvdegrain2(fv1...).addgrain(64, 0, 0)

grainy 4 = grainy3.mvdegrain2(fv1...)

etc.

[Dark Shikari]

The thread title is not as informative as it could be, you might get more suggestions if it reflected what you are trying to do. "motion compensated noise as detail" "stabilizing added noise to simulate detail" etc.fixed

[MfA]

Even if you have a good motion field you will still have to come up with a way to "patch up" the noise each time you motion compensate it, motion compensation errors will build up (blocking or blurring depending on the MC). If it's simple gaussian noise a simple leaky bucket approach might work (ie. every time scale the motion compensated noise and add some new noise) but if you want larger scale structure you will have to come up with something better.

[Fizick]

Dark Shikari,
here are links I recall and find:
http://forum.doom9.org/showthread.php?t=105495
http://forum.doom9.org/showthread.php?t=138467
http://forum.doom9.org/showthread.php?t=141041
http://forum.doom9.org/showthread.php?t=137235
http://forum.doom9.org/showthread.php?t=138467

MfA,
you are too pessimistic in almost any recent post!
Where are your creative suggestions? :)


i do not know how exactly imlement it at moving scenes. for Panning colorize we need rewrite DePan.
All I can easy to do right now as a toy, is to add some function or option to MVTools for static scene. for example fixed base frame (and variable deltaframe)

[Dark Shikari]

I think I figured out one way to do it:

Do the following:

1. Have a frame representing the current field of "detail" added to the clip.

2. Perform motion estimation between frame n and n+1.

3. Compensate 1) with the vectors from 2).

4. Modify MVCompensate such that if a section of the frame is declared "intra", the compensated frame section is transparent. Make this work with overlap, so the transparency for any given pixel is determined according to "how intra" it is.

5. Insert new detail from a new call to AddGrain() into the field of detail according to the transparency. Remove the transparency, now that its done. This should be a trivial masktools call.

6. Use that field of detail for frame n+1.

7. Repeat for the next frame.

One note: if there is no YUV colorspace with transparency available, you could simply use the U or V plane, as the "detail" is entirely luma-based.

[Fizick]

Post your script! :)

[Dark Shikari]

Post your script! :)Are you sure that's even possible in an Avisynth script?

[Fizick]

I do no see how to do 5) (EDIT: not 5 but 7)
Some recursion?

[Fizick]

source=avisource("film.avi")
noise=blankclip(clip=source,color=grey).addgrain(10)
supersource=mvsuper(source)
supernoise=mvsuper(noise)
v=mvanalyse(supesource)
# csource=mvcompensate(source,supersource,v)
cnoise=mvTransparentCompensateStage4to6(noise,supernoise,v)
add(csource,cnoise)

But the is no any "intra" in MVTools. Probably you say about empty borders in MVDepan?

EDIT: scipt may be longer if somebody try to use AvsRecursion extention by Kassandro
http://home.arcor.de/kassandro/AvsRecursion/AvsRecursion.htm

[Dark Shikari]

But the is no any "intra" in MVTools. Probably you say about empty borders in MVDepan?What happens when a block cannot find a valid MV, though? I know in MVFlow it blurs the two together instead of calculating an intermediate block.

(Example: the entire screen moves 50 pixels to the left. There are now 50 pixels on the right which are completely new. Where will they get their detail from? We have to introduce new detail, of course.)

[Fizick]

mvanalyse can find some similar block at other corner of frame :)

of cource, some of them wiil be have large SAD, but not in general case for real video.

[Dark Shikari]

mvanalyse can find some similar block at other corner of frame :)Hmm, that actually wouldn't be a bad thing: since the noise is random anyways, its not a problem if you randomly grab noise from some other area of the frame, as that will work just as well.

[Didée]

Ah, an idea's resurrection. I remember scharfis_brain and me cracking our brains about this problem, yeeaars ago.


The most simple thing I would try first:


1. generate static grain.

2. compute vectors N-1 <--> N on the source

3. compensate grainlayer with vectors of (2).

3a. Update: [grainlayer] := [compensated grainlayer]

4. next frame: goto 2


For (2) I would first try to use pel=1 and overlap=0, to hopefully avoid degradation of the grainlayer. Pel>0 and overlap>0 surely will cause dagradation due to the respampling operations. Pel=1 will reproduce exact pixels. In motion areas there will be some blocking introduced into the grainlayer, sure; but hehe, blocking is not very visible in a grainlayer. ;)


(3.) simply means that for each frame we proceed, MVCompensate would use the already-compensated grainlayer as base for the current Compensation.

Somewhere inbetween, include step (X) to reset the grainlayer on scenechanges.


In the past, (3) was not possible to do in a script. (Perhaps it was for plugin writers, but not for us scripters.)

Perhaps with MVTools v2, it is possible because the 'super'frame is public? One could try to us ConditionalFilter or AVSReccursion to update the 'super'frame on a per-frame-basis, via crop+stack?

[Dark Shikari]

For (2) I would first try to use pel=1 and overlap=0, to hopefully avoid degradation of the grainlayer. Pel>0 and overlap>0 surely will cause dagradation due to the respampling operations. Pel=1 will reproduce exact pixels. In motion areas there will be some blocking introduced into the grainlayer, sure; but hehe, blocking is not very visible in a grainlayer. ;)Exactly what I was thinking: this isn't ideal, but its an easy hack to avoid having to regenerate grain.

(3.) simply means that for each frame we proceed, MVCompensate would use the already-compensated grainlayer as base for the current Compensation.

Somewhere inbetween, include step (X) to reset the grainlayer on scenechanges.


In the past, (3) was not possible to do in a script. (Perhaps it was for plugin writers, but not for us scripters.)

Perhaps with MVTools v2, it is possible because the 'super'frame is public? One could try to us ConditionalFilter or AVSReccursion to update the 'super'frame on a per-frame-basis, via crop+stack?If we only need 3) in realtime rather than for arbitrary seeking, we could do it like I did in my old crappy GrainOptimizer plugin: simply completely ignore being consistent and start the calculations from wherever playback begins.

[AntiJw]

One trick I've found for making crappy-quality video more palatable (in particular the awful quality proleague Starcraft VODs) is to add a constant layer of grain to the background to simulate detail.
...Nice, I didn't know you watched them too :)
I like them with English commentating:
http://www.youtube.com/user/VioleTAK
although the video-quality is even worse then (because of reencoding) :p

EDIT: Cool. I just found GOMTV-live (after your recommendation): http://www.gomtv.net/classics2/live/

[Fizick]

it too late to read (=understand) the comments,
here is the toy to play with (it is not what do you want finally, but first approximation) :

LINK REMOVED

AVISource("some.avi")
nref=0 # static refrence frame
noise=blankclip(clip=last,color_yuv=$808080).addgrain(1000)
super=mvsuper()
supernoise=mvsuper(noise)
v=super.mvanalyse(delta=-nref) # negative as a mark
cnoise=mvcompensate(noise,supernoise,v)
mt_lutxy(last,cnoise,"x y + 128 -")

[Dark Shikari]

Nice, I didn't know you watched them too :)
I like them with English commentating:
http://www.youtube.com/user/VioleTAK
although the video-quality is even worse then (because of reencoding) :pI watch Tasteless's GOMTV narration live every weekend :p

[Dark Shikari]

I had to modify this to the following to get it to work:

nref=0
noise=last.blankclip(pixel_type="YV12",color_yuv=$808080).addgrain(60,0.5,0.5)
super=mvsuper()
supernoise=mvsuper(noise)
v=super.mvanalyse(delta=-nref,overlap=0) # negative as a mark
cnoise=mvcompensate(noise,supernoise,v)
mt_lutxy(last,cnoise,"x y + 128 -")

But it doesn't really work at all: the grain isn't constant at all and it looks exactly as if I ran addgrain() normally without constant=true.

Also, what happened to the "pel" option for MVAnalyse?

[Sagekilla]

Perhaps because you called it as AddGrain(60,0.5,0.5) which implies constant=false?

[Dark Shikari]

Perhaps because you called it as AddGrain(60,0.5,0.5) which implies constant=false?The original script had constant=false as well, and calling it with "constant=true" certainly doesn't work: it looks exactly as if I had done it with normal addgrain(constant=true), with no motion tracking at all.

[Fizick]

it should works at static scene only. May be with small oscillatory (returning to same scane) panning like shaking.
Reference frame is FIXED to constant (=0).

[Fizick]

Seems, you need in "inloop" mode=2 of old MVTools.

converttoYV12()
noise=blankclip(clip=last,color_yuv=$808080).addgrain(1000)
v=mvanalyse()
cnoise=mvcompensate(noise,v,mode=2)
mt_lutxy(last,cnoise,"x y + 128 -")

http://manao4.free.fr/MVTools-v0.9.9.1.zip

Please confirm. I can re-implement it to v2.X

BTW, I use AddgrainC dated 11 August 2006 г., 13:25:12

[Dark Shikari]

It... seems to work.

Motion tracking really could use some work, and it looks a little weird because it appears to be duplicating a lot of blocks of grain in flat areas, but overall the effect seems to be somewhat near what is wanted.

It seems it would be better if it had more uniform grain instead of repeating the same grain pattern every 8 pixels.

[Fizick]

I release v2.2.1 with mode=2 option.
Really I removed the "mode" parameter :) and added more general parameter "recursion". Here is crappy text from doc:
recursion is percent weight of previously compensated frames used for new compensation in special recursive mode. Rest weight is taken (uniformly) from given reference frame (used in ordinary mode). Recursion=100 is full recursion similar to in-loop mode=2 of old MVTools v0.9.x. Do not use recursive mode unless you know what do you do. Default recursion=0.

[Dark Shikari]

I release v2.2.1 with mode=2 option.
Really I removed the "mode" parameter :) and added more general parameter "recursion". Here is crappy text from doc:
recursion is percent weight of previously compensated frames used for new compensation in special recursive mode. Rest weight is taken (uniformly) from given reference frame (used in ordinary mode). Recursion=100 is full recursion similar to in-loop mode=2 of old MVTools v0.9.x. Do not use recursive mode unless you know what do you do. Default recursion=0.Great, I'll try it. It should now be fast enough to work in realtime, unlike MVTools 0.9.x :p

[Fizick]

I said nothing about speed ;)

[zee944]

Very intriguing topic. I hope you can polish this method out.

If I get it right, the current script works on the whole image, not only on the sides (panning).

Just an idea: what about adding more grain around the edges, less to the flat surfaces? Would this improve the illusion?