How to add artificial details to a video?

[zee944]

First off, before anyone starts to think I'm being stupid, I'd like to pin down that I'm aware I can never call forth details never existed on the source. But I do think there has to a be a way to make the image to LOOK like it had enough details to cheat our eyes.

I imagine something like Shortcut PhotoZoom Pro does. This software is a photo tweaker for printing, has a very good resizing method (S-Spline), and during resizing you can add artificial details to the picture. Here's a sample of the effect:

http://i179.photobucket.com/albums/w286/zee944/samohung-luckystars_1.jpg
http://i179.photobucket.com/albums/w286/zee944/samohung-luckystars_2_200zoom_0deta.jpg
http://i179.photobucket.com/albums/w286/zee944/samohung-luckystars_3_200zoom_100de.jpg

No, I don't say this type of added artificial details are perfect, nor I say it brings the photo quality to another level, but you get the idea. It works more or less, still makes the photo to appear a little bit more detailed. Is there such a filter to achieve this in AviSynth? Or even in VirtualDub or some other software.

I've read through several threads before I posted this. Please redirect me if I overlooked something very obvious filter, method, software or anything which suits to this job. What I actually want to do now is to resize a non-anamorphic clip from DVD to anamorphic, and somehow (at least seemingly) compensate the lack of details.

PS.: I've read a lot about 'blind deconvolution' but I was never able to actually download a working script or plugin for that.

[martino]

AddGrain.

[zee944]

AddGrain.

Seriously. I need something more effective and tricky solution than just adding equal grain to the whole image.

[buzzqw]

grainfactory ?

BHH

[Didée]

Seriously. I need something more effective and tricky solution than just adding equal grain to the whole image.
Seriously #1: You have Avisynth. Arm yourself with MaskTools, AddGrain/C, whatever suits. Then go ahead and make the effective and tricky solution.

Seriously #2: Detail is static, or quasi-static. Grain/Noise is random.
Your referenced app "PhotoZoom" has a rather easy job to do: it works on only ONE Frame. On one frame, you can't see if the fake detail is static or random.
But you want to apply something to a video sequence, and that a lot more complicated. When just adding grain, obviously you get just random grain. To make it look more like real detail, you would need to add motion-stabilized grain. Time ago, scharfis_brain and me racked our brains about how to implement such a procedure in a script, but we didn't succeed. With the currently available tools, it seems close to impossible to motion-stabilize grain over a longer time period (i.e. more than just 2/3/4 frames or so).


Seriously #3: Just to be clear - from the look of it, PhotoZoom just adds medium-frequency grain to areas where there is medium pixel variance. That's not very complicated to do. But again, working on one single frame is a completely different story than working on frame sequences that run with 25~30 frames per second.
You might try the following: extract a sequence of, say, 100 frames from your source to single bitmaps. Then use PhotoZoom to add "detail" to all those 100 frames. Then play those processed 100 frames sequentially, and see how that looks like. Most probably you will find that the effect achieved by PhotoZoom is just ... noise. :p


OTOH ... if you non-anamorphic DVD source is in good shape (better than the posted example), SeeSaw or even LSF(Lmode=3) can achieve a quite noticeable effect.

[2Bdecided]

Funny - here I am with mc_spuds spending hours trying to do the opposite! ;)

Of course, if you do add noise, it's going to make it harder to encode. The MPEG-2 encoding will often soften the noise and turn it into blocks.

(sorry to state the obvious!)

Cheers,
David.

[Socio]

Like Didee said use either SeeSaw or LimitedSharpen faster to sharpen and bring out some "artificial detail" without screwing up the PQ.

Here is something I did real quick using SeeSaw, DeHalo_Alpha and denoiser script I had:

Your original:
http://i179.photobucket.com/albums/w286/zee944/samohung-luckystars_1.jpg

with Seesaw:
http://img292.imageshack.us/img292/3603/samoseesawkp5.jpg

Since you want detail the only trick is tweaking SeeSaws sharpening as high as you can go without getting oversharpening halos.

You can also use Dehalo_Alpha and a good denoiser after sharpening like I did to take care of some of the haloing that might occur with out harming the detail to much. You just have to find the right balance between them and SeeSaw.

[zee944]

Seriously #1: You have Avisynth. Arm yourself with MaskTools, AddGrain/C, whatever suits. Then go ahead and make the effective and tricky solution.


Oh, I think you misunderstood me, I didn't try to be the smart guy here. What I've tried to say there must be already working, smarter methods done by experts - something like buzzqw suggested, grainfactory which, at least as I found out, adds more grain to darker parts etc.

Thanks Didée for taking the time for a detailed answer. I've been planning to use SeeSaw and/or LimitedSharpen anyway (I like them both, use them regularly) just felt the lack of details will still be a problem.

Funny - here I am with mc_spuds spending hours trying to do the opposite! ;)

Like Didee said use either SeeSaw or LimitedSharpen faster to sharpen and bring out some "artificial detail" without screwing up the PQ.

Thank you both of you for the tips! I'm planning to use SeeSaw/LimitedSharpen. DeHalo_alpha is great but I hope I won't need it.

Probably my first post is misleading - I didn't think only about grain! I've been thinking of any kind of artificial, generated, calculated details.

I put another question here, please, please comment, I'm hungry for your opinions:

Should I forget all those algorythms which use the neighbour frames to extract more details? Blind deconvolution, "super resolution", ReFocus etc. Aren't there a single one which gives noticeable results?

[2Bdecided]

Should I forget all those algorythms which use the neighbour frames to extract more details? Blind deconvolution, "super resolution", ReFocus etc. Aren't there a single one which gives noticeable results?Some work remarkably well. Some seem broken (i.e. worse than a normal resize). The commercial versions all have demos - many do a single frame, not the whole video (batch mode though?). There's also fixmyvideo.

mc_spuds wasn't a suggestion - it won't do what you want it to do at all!

Cheers,
David.

[J_Darnley]

I had a thought about how to do this. I was looking at the TemporalDegrain script and saw that it adds the difference from a denoised clip.
# Contra-sharpening: sharpen the denoised clip, but don't add more than what was removed previously.
# Here: A simple area-based version with relaxed restriction. The full version is more complicated.
s = NR2.minblur(1,1) # Damp down remaining spots of the denoised clip.
allD = mt_makediff(o,NR2) # The difference achieved by the denoising.
ssD = mt_makediff(s,s.removegrain(11,-1)) # The difference of a simple kernel blur.
ssDD = ssD.repair(allD,1) # Limit the difference to the max of what the denoising removed locally.
ssDD = SSDD.mt_lutxy(ssD,"x 128 - abs y 128 - abs < x y ?") # abs(diff) after limiting may not be bigger than before.

NR2.mt_adddiff(ssDD,U=2,V=2) # Apply the limited difference. (Sharpening is just inverse blurring.)

So what I used is:
source = ImageSource("D:\Downloads\samohung-luckystars_1.jpg").bicubicresize(448,256).converttoYV12()
original = source
denoise = original.removegrain(11,-1)
diff = mt_makediff(original, denoise)
enhanced = original.mt_adddiff(diff, U=2, V=2).mt_adddiff(diff, U=2, V=2).mt_adddiff(diff, U=2, V=2)

Interleave(original.subtitle("ORIG"), enhanced.subtitle("ENH"))
http://users.telenet.be/darnley/images/screenshots/detail-enhance-1.pnghttp://users.telenet.be/darnley/images/screenshots/detail-enhance-2.png
Unfortunately it doesn't look very good as it does halo (I shouldn't have repeated the mt_adddiff three times) and bring out the blocking from your image which could be there in the source or due to the JPEG compression. Someone could develop this idea a little further to enhance what limited details might be in the video. This could prevent edge enhancement problems whilst possibly bringing out some more detail. MVDegrain or MVDenoise might be useful on a video.

[Didée]

I had a thought about how to do this. I was looking at the TemporalDegrain script and saw that it adds the difference from a denoised clip.
...
Someone could develop this idea a little further to enhance what limited details might be in the video.
It's already done (http://forum.doom9.org/showthread.php?t=104701). ;)

See, adding the difference of a "RemoveGrain(11)" basically is the very same as simply doing "sharpen(1)". Adding it three times would be the same as "sharpen(3.0)" (which isn't allowed, max is 1.2 or something). Avoiding halo'ing by excluding the edges from sharpening is old stuff, too ... it was discussed in the past, and btw that's what LimitedSharpen is doing with Lmode=3 (sort of) or edgemode=2 (literally).

So, sorry to spoil the fun, but I don't see any breaking news in this ...

[zee944]

Some work remarkably well. Some seem broken (i.e. worse than a normal resize).

Can you give me some suggestions? Which are the ones that doesn't work for sure, and which of them worth a try?

And anyone else who has opinion about these video enhancers, please drop a comment. :)

mc_spuds wasn't a suggestion - it won't do what you want it to do at all!

Sorry, my fault. I was talking about about LSF and SeeSaw.

[2Bdecided]

QE super resolution is OK, but seems to assume all content is interlaced. Single frame output only.

PhotoAcute Studio is OK, but is made for photos so painful to get video in and out.

VideoEnhancer and YUVsoft Super Resolution worked OK though I thought they looked a little over sharp. I can't remember if the output was a single frame or a video (demos have expired here!) - may have been video.

Topaz moment cannot deinterlace (it should do using the underlying media player, but the deinterlace filter fails on my system), and in any case the output just looked weird (like crystals!). Single frame output only.

fixmyvideo.com looked OK, though some movement became very strange. Video output (three formats). Designed for upsizing very small videos, so maybe no good with your source.

If anyone know of any others, please post!

My conclusion with this lot was, basically, don't bother. Use tricks in AVIsynth instead.

Cheers,
David.

[*.mp4 guy]

Are you looking to achieve somethig like this (http://img136.imageshack.us/img136/7185/fakedetailje2.jpg) (all done in avisynth)? if you are, your pretty much out of luck when dealing with video, motion makes noise much easier to spot. You can work around this by adding lots of fancy noise, then using high temporal radius motion compensated denoising, followed by ttempsmooth to calm down static areas, but its slow, and not entirely effective.

To get interesting fake detail, you should try adding noise, then removing it with a denoiser that has a bit of pattern recognition, something like fft3d, dfttest, frfun3/7, or tnlmeans. Set the denoisers strength so it just falls short of being able to remove the noise. Semi pattern aware denoiser, like the ones listed will remove less noise in complex areas, and will incorporate the noise into the structure of the complex areas to a certain extent, this is usually more convincing then just adding noise.

[45tripp]

Are you looking to achieve somethig like this (http://img136.imageshack.us/img136/7185/fakedetailje2.jpg) (all done in avisynth)?

create an oil painting?

[zee944]

QE super resolution is OK, but seems to assume all content is interlaced. Single frame output only.

Thanks David for the dtails, will do some research and tryout.

Are you looking to achieve somethig like this (http://img136.imageshack.us/img136/7185/fakedetailje2.jpg) (all done in avisynth)?

Ugh, no - I don't like the results, it looks more a cartoon now!


To get interesting fake detail, you should try adding noise, then removing it with a denoiser that has a bit of pattern recognition, something like fft3d, dfttest, frfun3/7, or tnlmeans. Set the denoisers strength so it just falls short of being able to remove the noise. Semi pattern aware denoiser, like the ones listed will remove less noise in complex areas, and will incorporate the noise into the structure of the complex areas to a certain extent, this is usually more convincing then just adding noise.

Was the sample you linked above made this way, or it's a different idea? Thanks anyway.

[*.mp4 guy]

Was the sample you linked above made this way, or it's a different idea? Thanks anyway.[/QUOTE]

Yes, thats how the fake detail was added to that sample.

Just out of curiosity, does this (http://img136.imageshack.us/img136/5919/fakedetail2xy4.jpg) look better to you then the other sample i posted.

[zee944]

Just out of curiosity, does this (http://img136.imageshack.us/img136/5919/fakedetail2xy4.jpg) look better to you then the other sample i posted.

Yeah, a little better than the previous - but these fake details seem only noise. Much more than the image processed with Photozoom Pro in the first post.

[TSchniede]

I made a script which should in theory create pseudo-details in video.

Here is my first attempt of a motion compensated grain generator. It is very slow, so I haven't tested it extensively.


######################################################################################################################
#
# MCGrain motion compensated grain
#
# Version History
# version 1.1 minor bugfix (thSAD of mvdegrain wasn't properly set)
# version 1.0 first release
#
# by TSchniede
# requires mvtools, mt_masktools and addgrain
#
# mf is maskfactor - sort of strenth setting for the grain-use, it works most on smooth areas (low detail)
# noiseclip supply your own noise - luma==128 is supposed to be untouched video
# gstddev standard deviation of the grain
#
# maskflat apply noise only to flat areas?
# smoothclip smooth the clip with the applied noise and reduce the choppyness
#
# other options are simply the exposed mvtool-options
#
# Idea: motion compensate 1 frame of noise up to 3 frames in both directions, then take the next.
# it IS choppy but it correlates better than other noise with the motion,
# so to even it out somewhat smooth it up with mvdegrain3
#
# yes this IS really,really, slow!
#
######################################################################################################################

function MCGrain(clip source, int "block", int "ol", bool "tm", int "sh", int "ts", int "fbl", int "pelf", int "mf", clip "noiseclip", float "gstddev", bool "maskflat", bool "smoothclip"){
block = default(block,8)
ol = default(ol, block/2)
tm = default(tm,true)
sh = default(sh, 2)
ts = default(ts,320)
fbl= default(fbl,0)
mf= default(mf,64)
pelf = default(pelf,2)
gstddev = default(gstddev, 60.0)
maskflat = default(maskflat, true)
smoothclip = default(smoothclip, true)

# generate the motion vectors of the clip
backward_vec3 = source.MVAnalyse(isb = true, blksize=block, truemotion=tm, delta = 3, pel = pelf, overlap=ol, idx=1, dct=fbl)
backward_vec2 = source.MVAnalyse(isb = true, blksize=block, truemotion=tm, delta = 2, pel = pelf, overlap=ol, idx=1, dct=fbl)
backward_vec1 = source.MVAnalyse(isb = true, blksize=block, truemotion=tm, delta = 1, pel = pelf, overlap=ol, idx=1, dct=fbl)
forward_vec1 = source.MVAnalyse(isb = false, blksize=block, truemotion=tm, delta = 1, pel = pelf, overlap=ol, idx=1, dct=fbl)
forward_vec2 = source.MVAnalyse(isb = false, blksize=block, truemotion=tm, delta = 2, pel = pelf, overlap=ol, idx=1, dct=fbl)
forward_vec3 = source.MVAnalyse(isb = false, blksize=block, truemotion=tm, delta = 3, pel = pelf, overlap=ol, idx=1, dct=fbl)

# prepare the noise
noiseclip=default(noiseclip, source.converttoyv12().mt_lut(expr="128",u=1,v=1).addgrain(gstddev,0,0))

# mask the smooth/flat areas
mask=mt_lutxy(source.converttoyv12(),source.converttoyv12().mt_convolution(horizontal="1 1 1 1 1", vertical="1 1 1 1 1", u=1 , v=1),
\ expr="x y - abs "+string(mf)+" *",u=1,v=1)

# generate the motion compensated noise (makes troubles on scene changes!)
bclip3 = noiseclip.MVCompensate(vectors=backward_vec3, idx=2)
bclip2 = noiseclip.MVCompensate(vectors=backward_vec2, idx=2)
bclip1 = noiseclip.MVCompensate(vectors=backward_vec1, idx=2)
fclip1 = noiseclip.MVCompensate(vectors=forward_vec1, idx=2)
fclip2 = noiseclip.MVCompensate(vectors=forward_vec2, idx=2)
fclip3 = noiseclip.MVCompensate(vectors=forward_vec3, idx=2)

# arrange the frames to a continous clip
noise =interleave(bclip3,bclip2,bclip1,noiseclip,fclip1,fclip2,fclip3).SelectRangeEvery(47,7,21,false) #

# add the noise to the input
noisemerged = mt_lutxy(source.converttoyv12(),noise, expr="x y 128 - +",u=1,v=1)

# add noise only to flat areas
maskflat ? mt_merge(source.converttoyv12(),noisemerged,mask.mt_invert(),y=3,u=1,v=1) : noisemerged

# ensure correct color format
isyuy2(source) ? last.converttoyuy2() : last
# smooth clip & noise
smoothclip ? mvdegrain3(last,backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,idx=3,thSAD=ts) : last
# return the original chroma data
return mergechroma(last,source,1.0)
}

#filterpath="avs game\filter\"
#LoadPlugin(filterpath+"addgrain.dll")
#LoadPlugin(filterpath+"mvtools.dll")
#LoadPlugin(filterpath+"mt_masktools.dll")

[TSchniede]

By the way have you tried NNEDI for upsizing? http://forum.doom9.org/showthread.php?p=1045226#post1045226

As far as I know, the new preset makes an error with the posted script, field=1 worked for upsizing.

[Gladiator:)]

Also u can use awarpsharp with a high threshold power
But a small cartoonising will be there.....