New approach to fine detail sharpening without edge halos. Very fast, HD also.

[Hotte]

Hi,

this message has been edited to point your attention to ready to go solution of how you can

1. fine-sharpen your Full-HD material
2. without edge halos or pushed up contrast of mid-detail
3. with optional and highly effective temporal degraining and denoising

Look at post #62 of page 4 in this thread.

---- Old discussion
I have had so much benefit from this forum in search of "my" sharpening solution. It seems I´ve got it now and would like to share it with you.

The Full-HD videos of my Nikon D5300 Camera are best with defensive internal sharpening and sharpening in post. Due to agressive noise-reduction out-of camera fine detail is somewhat underdeveloped. So a detail-sharpener is needed which emphasizes fine contrast without pushing overall contrast of mid-detail and without ugly edge haloing. After WEEKS of research I came back to this 10-year old avisynth code:

http://forum.doom9.org/archive/index.php/t-76257.htm.

This is a fast and simple yet very effective unsharp mask with edge limiting. Thanks to who ever wrote this!

If you set strength (s) to 40 and small edge mask width (w) to 1-5, you will get a fine detail emphasizing effect without new edge halos. There are only two remaining issues: Deliberatly soft structures like clouds tend to form colour blocks and may get sharp edges where there are none. Second, already EXISTING edge halos (due to in-camera presharping) will get emphasized too. So I added/changed the following:

I replaced the old simple blur(1.58) with the more developed binominalBlur() from the variableblur package and the cloud-issue was almost gone (GaussianBlur did not show better results but only slowed down performance heavily). Second I applied DeHalo_alpha() to reduce remaining edge halos.

This is now my favourite sharpening solution among those I tried out. Consider that you have to balance detail sharpening and noise enhancement using the "s" and "w" parameters.

Feel free to try out this code with your high-definition footage. Your comments and suggestions are most welcome.

#################
# Sharpening (unsharp masking) of fine details avoiding edge halos or harsh contrast (by Hotte).
#
# Make sure you installed external filters "variableblur" and "DeHalo_alpha" with dependant filters
# Load source and convert. Binominal blur needs YV12.

x=avisource("blabla.avi").convertToYV12

# Set unsharp-mask parameters.
# s=strength of sharpening, 1 to 40=subtle, 70=significant, 100=strong (few artefacts), 127=maximum (strong artefacts)
# w=width of mask, 1-5=very fine detail only (almost no halos), 12=overall (some stronger halos)

s=40
w=5

# perform unsharp masking (see link above): fine blur, subtract negative, subtract whole mask from original

i=x.binomialBlur(varY=0.5, varC=0, Y=3, U=2, V=2, useMMX=true)
j=subtract(i,x).levels(0+s,1,255-s,0,255)
w=w*(128/s)
k=j.levels(127-w,1,128+w,127-w,128+w)
subtract(x,k)

# A subtle dehalo of large sharpening edges. highsens limits edgemask to larger edges rather than fine ones.
# DeHaloAlpha reduces sharpening a tiny bit. Increase width of sharpening mask by 1 or 2 for compensation.
# If you like geometrical structures being less aliased turn ss to 1.5, but it will reduce speed to 40%.

DeHalo_alpha(highsens=100,ss=1.0)
#################

In my environment this code runs multithreaded without issues if you put the following statements on top. This way I am getting crazy 35fps and 'round 80% CPU usage on an elderly i7 Quadcore (supersampling switched off (ss=1.0)):

SetMemoryMax(1024)
SetMTMode (4,4) # announce multithreading Mode 4, use 4 cpu-cores

[feisty2]

Function SharpLimit16 (clip input, clip "sharp", float "str", bool "tv_range")
{
sharp = Default (sharp, UnDefined ())
str = Default (str, 1.00)
tv_range = Default (tv_range, True)

Sharpdif = Dither_sub16 (sharp, input, dif=True, y=3, u=3, v=3)
Dif = Dither_lut16 (Sharpdif, "x 256 / 128 - abs 4 / 1 4 / ^ 4 * "+String (str)+" * x 256 / 128 - x 256 / 128 - abs 1.001 + / * 128 + 256 *", y=3, u=3, v=3)

output = input.Dither_add16 (Dif, dif=True, y=3, u=3, v=3)

return output
}

pass ur source clip and regularly sharpened clip to the function above, the output clip would be free of artifacts

[Hotte]

Hi feisty2,

thanks. Is this suggestion intended to be a replacement for dehalo_alpha or another suggestion for sharpening ?

I was trying to replace dehalo_alpha but in my code like this:

....y=subtract(x,k)

#DeHalo_alpha(highsens=100,ss=1.0)
z=SharpLimit16(x,y)
return(z)

where x represents the original and y the sharpended clip. However it returns Undefined() to be an undefined function. I do not know how to manage this. Could you help ?

Hotte

[feisty2]

it's a simple function, it does a nonlinear limit to ur sharpened clip so the artifacts will be gone
very easy to use
a=last
b=a.#any sharp filter#
a.sharplimit16 (b)
* both source clip and sharpened clip gotta be stacked 16 bpc format

[creaothceann]

[code] http://i.imgur.com/PsLxH88.gif

[Hotte]

Hi feisty2,

I was trying to use your function as a replacment for dehalo_alpha like this:

....
b=subtract(x,k) # final step of my sharpen code applied to input clip x
x.sharplimit16(b) # call your function to remove sharpening artefacts

This throws: there is no function named "UnDefined""

If ignore the "sharp = ...."-line in your code it throws: "there is no function named "Dither_sub16""

I could not find a plugin that contains "Dither_sub16".

What's wrong ? Thanks.

[foxyshadis]

If ignore the "sharp = ...."-line in your code it throws: "there is no function named "Dither_sub16""

I could not find a plugin that contains "Dither_sub16".

Dither plugin (http://forum.doom9.org/showthread.php?p=1386559). It needs its own wiki page at some point.

[Hotte]

Thanks foxyshadis for this hint. I managed to install dither, but it was difficult since there where two syntax errors in the linked version of dither.avsi which where very difficult to find. Very strange.

Anyway after all that I did not get any satisfying result from feisty2's receipe. Instead some heavy white snow-like artifacts where added to the video.

I would like to come back to my original script. Any feedback on sharpness out there ?

Still there are things to improve, e.g.

1. Any ideas how to control a bit the fine grainy flickering that could show up, when fine detail sharpening is pushed up to heavily (I do not mean classical sensor noise here)

2. Is there a basic approach e.g. with dither to reduce color banding. I find dither very, very promising, but is difficult to understand, maybe there is some out of the box recommendation to try out for 8-bit fHD input resolution ?

3. Any suggestions to improve edge halo suppression even more than with (my parameters of) dehalo_alpha ?

[feisty2]

dats because ur doin it wrong
I'll show u a more detailed example of how to use it

a=last.Dither_convert_8_to_16 ()
b=last.binomialBlur(varY=0.5, varC=0, Y=3, U=2, V=2, useMMX=true).Dither_convert_8_to_16 ()
sharp=dither_add16 (a,dither_sub16 (a,b,dif=true),dif=true)
a.sharplimit16 (sharp)
ditherpost (mode=6)

edit: u need avs 2.6 or avs+ to use this function

[Hotte]

Hi feisty2,

thank you. This time your code worked immediately. Indeed it does fine sharpening pretty much the way my code does, so I had a very close comparative look.

Sharpening results are almost exactly the same if I apply s=40 and w=1 to my code (my code finesharpens a tiny, tiny bit more). There is no visible difference in artefacts of whatever kind, even if you zoom in 300%. Your code also benefits a lot if I add dehalo_alpha (like in my code) to get rid of existing edge halos. Your code runs some 30% slower both solo or with setmtmode(4,4).

What do I learn ? Two methods - same result. It seems that we´ve reached the feasible limits in detail sharpening.

Being an obvious expert in handling the complex dither-tool, I would love if you could answer the following questions:

1. How can I influence the strength of sharpening with your code ?
2. There is a very, very tiny flair of color shifting e.g. with clouds to pink (looks a bit like chromatic abberation) which is typical for unsharp mask processes - in both approaches. Do you know how to limit this ?
2. I've got some temporal color-banding in blue skies. Is there something in dither I could do against ?
3. Is there something in dither to calm the grainy flickering (I do not mean sensor noise) that my arise with strong detail sharpening ?

Thanks!

[feisty2]

1 "str" controls the strength, or sometimes u can just repeat the code several times if you want extra strong sharpening
2 use an internal resizer or dither_resize16(nr) on chroma planes, adjust src_left and src_top parameters to shift the chroma a little bit left(or right), a little bit up (or down), till theres no visible chroma shift
2 as the dither document says, "gradfun3" can remove bandings
3 search "motion compensated sharp" on forum, temporal processing is mvtools job, not dithers, there re 2 ways to limit sharp result temporally, the motion block average method (mdegrain) and the domain limit method (mcompensate+mt_clamp)

[Hotte]

Now it is getting really interesting. Modfying "str" scales detail sharpening indeed very effectively. Compared to my approach, dither performs somewhat better with very high values like 3 or 4 since it creates less artefacts. This is useful when temporal denoise gets in. It took me quite a while to understand the principles of motion-compensated MDegrain but thanks to example scripts in this forum I finally got there.

Also I experimented with GradFun3 to smoothen a bit the blockyness of colour which may arise whilst sharpening.

Bottom line is that improvements in both calming down the grain and colour blocks/bands are only subtle but maybe worth the price for the significant performance decrease.

I will soon be coming up with a full script

Thanks feisty2 for your hints! There is only one thing I was not able to manage: Chroma shifting. I simply was not able to shift the chroma plains. Could you give me an example ? (You really have to have to intensively study these matters to put things into something that works).
But I think chroma shifting is maybe not the right approach. The problem is, that after sharpening if you have a very close look a clear blue sky will have many blocks in it with a magenta tone and light clouds will have blocks with a light yellowish tone. You do not see it at frist glance, but you notice a very slight color change. When you zoom in heavily you see these blocks and bands that were not there before. It's not a big problem and the sharpness benefit counts much more, but I ask you, if there is something we could do against this effect ?

[feisty2]

EDIT:
just checked the code, it doesn't process chroma at all, so, little bit confused about the "color block" u were talkin about, cuz it sounds a lot like a hangover of shitty chroma got sharpened kinda stuff
post a picture maybe?

[Hotte]

Thanks for looking into this. Yes, it's time for pictures now.
I prepared 4 samples, all a bit overdone with sharpening to demonstrate the effect.

No1: Camera .MOV dragged into EDIUS and exported as AVI with Canopus HQ 8-bit intermediate Codec with very high quality settings. AVI back to Edius and shoot.

No2: Like No1 but reimported AVI sharpened with EDIUS's internal sharpener (setting 33).

No3: AVI No1 brought into avisynth and processed with nothing but your dithercode, str=2.0. Detail sharpening appears somehow like EDIUS, but less halos - just nice (dehalo_alpha would make it even nicer).

No4: Same as No3 but with remarks showing the increase of artefacts compared to No2 or No1.

I know we already have a fine solution here, but maybe we can try to get rid of some residual artefacts.

Thanks!

[feisty2]

its better to use an online image host site than upload attaches, it would waste the already very limited resources of doom9 forum server and would have to wait for admins approval which is gonna take a long time
anyways, if ur curious if theres somethin even better
I got a "deconvolution" like "magic" sharpen function suite
everything is done under avisynth except the wiener filter part
but its extremely slow, so damn slow you cant even imagine, Id say, 0.001 fps at 720x480 would be the speed of a super nice computer running the script
so, not useful at all, but it produces nice result
a quick preview and comparison, click the thumb to view the full size image
http://thumbnails111.imagebam.com/36196/3df43f361956265.jpg (http://www.imagebam.com/image/3df43f361956265)
http://thumbnails109.imagebam.com/36196/d2fa74361956251.jpg (http://www.imagebam.com/image/d2fa74361956251)
http://thumbnails110.imagebam.com/36196/656393361956218.jpg (http://www.imagebam.com/image/656393361956218)
http://thumbnails110.imagebam.com/36196/39e160361956234.jpg (http://www.imagebam.com/image/39e160361956234)

[feisty2]

the "dither" function I wrote above (actually modified from didees script, I didn't invent it) is a part of that veeeeeeeery slow sharpening, it's like a lite version of lsfmod

[Hotte]

Can you see my pictures (the ruined castle wall) now ? I can.
Yeah the clip with the lady looks nice at first glance - although you would not see artifacts like I found at that size. And I like that face but not on my screen for 1 year to become sharper...

I found out, that these magenta artefacts are exactly the same with my code, because it seems that we have the same unsharp mask approach. Moreover, they where much stronger, before I replaced blur(1.58) with binominal blur.

Somewhere I read, that unsharp masking produces color shifting. But if I use unsharp masks in my photo-raw-editor there is never any chroma shifting...

Anyway some tweaking with dehalo_alpha, mdegrain, gradfun and well balanced sharpening parameters gives excellent results in pretty short time.

Maybe someone out there`s got an idea ?

[poisondeathray]

@Hotte - did you realize you uploaded lossy jpg's ?

Blocking & banding artifacts are in the source image fom the crappy compression from your camera (Those are mostly h264 compression artifacts, not JPEG artifacts). Any sharpening approach will tend to enhance them

Take your original MOV source and use histogram("luma") to visualize them. It's an enhanced luma view

[feisty2]

im on cell now, will check ur pics l8r
are u sure u were seeing my pics at full size which is 720*352
and dont look at her face, take a glance at hair

[Hotte]

@poisondeathray, yes I know its not free from artefacts - which cam isn´t ? (btw jpg-picture compression did not add visible artefacts - it looks at home just like you see it on the web here)

Say you are right then I still do not understand what the trick is with the EDIUS sharpener: It boosts detail the way the unsharp mask does, but there are (almost) no magenta shifts. I cannot explain that from histogram-luma.

Im pretty sure it happens somewhere on the way we do the masking process with avisynth. Maybe it is not trivial.

If you have good quality footage and would like to give it a try, I would be very happy if you could give some feedback.

[Hotte]

feisty, had a close comparative look now at this lady.

"new" indeed sets a benchmark here: Pulling out all the detail without halos or grain increase - just keeping the natural look. Wonderful! It is so sad that it runs so slow.

I am keen to work with that sample. But with all that conversion here and there, I will probably not have a comparable starting position. Or could you post a 10 picture avi or so to a hosting website ?

[poisondeathray]

@poisondeathray, yes I know its not free from artefacts - which cam isn´t ? (btw jpg-picture compression did not add visible artefacts - it looks at home just like you see it on the web here)


Are you kidding? There are many camera / recording setups that don't have these type of h264 block artifacts. You record to uncompressed or external recorder. You don't need uncompressed, even lightly compressed like ProRes , DNxHD. Even cheapo consumer camcorders with HDMI can do this

There are 2 isssues you are discussing. 1) block artifacts (already present) 2) magenta shift

I suspect Edius doesn't have the magenta shift because it's working in RGB

[feisty2]

u want my sample?
sure, when I get home I'll upload the samples (original untouched clip and the sharpened clip via my approach)
lsfmod indeed produces halo free and no grain amplified result also
but the whole image just became oil painted after sharpening
I checked lsfmod code, all nice code but the results just not quiet satisfying
I guess it might be just the crappy (too rough, not very precise) blur kernels it uses
edit: removegrain does not look like a nice kernel for sharpening to me
but somehow lsfmod picks it as the kernel

[feisty2]

all "magic" comes at price
this case, the price is ur time
u can enjoy the nice "magical" result if ur willing to waste such a long time to sharpen some videos

[feisty2]

o=last
a=last.Dither_convert_8_to_16 ()
b=last.binomialBlur(varY=0.5, varC=0, Y=3, U=2, V=2, useMMX=true).Dither_convert_8_to_16 ()
sharp=dither_add16 (a,dither_sub16 (a,b,dif=true),dif=true)
a.sharplimit16 (sharp)
ditherpost (mode=6)
s=last
mask=mt_hysteresis (o.mt_edge ("cartoon"),o.mt_edge ("hprewitt")).mt_expand_multi(sw=6, sh=6).mt_inpand_multi(sw=6, sh=6).tmaskcleaner ()
mt_merge (o,s,mask,y=3,u=3,v=3,luma=true)

solution to ur problem

[Hotte]

@poisondeathray: Of course having artefact free raw material is better. However most ordinary videographers will have some in-camera compression, nowadays usually h.264 and .mov if it is a mainstream dslr. Nevertheless some of those are looking for better sharpening than in-camera. I already achieved much better sharpening using avisynth, which is our discussion here.

What you said about EDIUS is interesting: I do not know if the algorithm they use inside is RGB. But I tried out my initial unsharpmask code and replaced bionomialblur() with gaussianblur(). Gaussian also delivers good results, but contrary to binomialblur() it allows the code to run with RGB (canopus codec exports the MOV as an AVI in YUY2) like this:

x=avisource("blabla.avi").convertToRGB24
i=x.gaussianBlur()
...

Result: Again magenta artefacts. Mmmmhhhhhh.....

[feisty2]

u can use any yv12 filter on rgb colorspace
there's something called "rgb48y", check the "dither" document

[Hotte]

wow, feisty2, can´t belive it...
I will do some testing tonight...can´t wait (like my boss).

[feisty2]

u can simply convey every rgb channel on a grayscale yv12 frame
example: showred ("yv12"), it will separate red channel from ur image and store it on yv12 format
detailed example:

xxxsource
a=last
a.showred ("yv12")
#processes here#
r=last
a.showgreen ("yv12")
#processes here#
g=last
a.showblue ("yv12")
#processes here#
b=last
mergergb (r,g,b,"rgb24")

there, rgb image filtered by yv12 only filters and without colorspace conversion

[feisty2]

samples!
http://www.sharebeast.com/72oa3rblzoax
http://www.sharebeast.com/woo4wbqk4h9g
soft.mkv is the source clip
sharp.mkv is the detail sharpened clip
both are in an "encodable" rgb48y format, encode and process friendly, but not for the final result
soft.mkv was converted to rgb48y from a dvd I bought
both clips were encoded by x264 (qp=0, lossless encoding)
to convert them to regular rgb clips
2 methods
1 rgb24 output
ditherpost (mode=6)
mergergb (selectevery (3,0),selectevery (3,1),selectevery (3,2),"rgb24")
2 rgb48 output
converttoy8 ()
Dither_convey_rgb48_on_yv12(selectevery (3,0),selectevery (3,1),selectevery (3,2))
*raw output, requires special encode commands

[poisondeathray]

@poisondeathray: Of course having artefact free raw material is better. However most ordinary videographers will have some in-camera compression, nowadays usually h.264 and .mov if it is a mainstream dslr. Nevertheless some of those are looking for better sharpening than in-camera. I already achieved much better sharpening using avisynth, which is our discussion here.


Of course, but you were discussing the "block" artifacts after sharpening. I was merely pointing out they were already there, and that any approach to sharpening or local contrast enhancement will tend to increase their visibility

There are many "hacks" for increasing internal bitrate recordings of various DSLRs. This alone will significantly reduce those artifacts - Magic Lantern for Canon, GH2 hacks etc... Look into it for your model

[Hotte]

feisty, I cannot get tmaskcleaner to work (unknow function). I found a zip with tmaskcleaner.dll (well hidden somewhere) and put it into the avisynth/plugins directory, but it does not work - even, when I Load the plugin (can´t load) at the beginning of the script.

btw do I use 32bit or 64bit versions of dll`s ? I have git Win7 64bit and avisynth 2.6

Thanx

[Hotte]

poisondeathray, thank you for your input. Yes, yes, yes I am very keen to get a higher bitrate for my NikonD5300! But I could not find a website. Any idea ?
Forget about that. After some more investigation I understood that the D5300 ist just simply to new to have been conquered. Buty anyway thanks for the idea. Never thought these options do exist.

[poisondeathray]

poisondeathray, thank you for your input. Yes, yes, yes I am very keen to get a higher bitrate for my NikonD5300! But I could not find a website. Any idea ?
Forget about that. After some more investigation I understood that the D5300 ist just simply to new to have been conquered. Buty anyway thanks for the idea. Never thought these options do exist.

That's too bad. You can clearly see the improvement in all camera models and makes. It really makes a big difference

eg. for D5100
http://www.personal-view.com/talks/discussion/comment/163187#Comment_163187

Sometimes Vitaliy needs some motivation or convincing to work on new models (donations help). There are also other variants of hacks you can check , like NikonHacker. Maybe you just have to wait a few months...



Anyways we are getting off topic here, lets stick to the avisynth discussion...

[Hotte]

Sorry feisty2, maybe these are a stupid questions but I cannot get your code to work:


avisource("blabla.avi").convertToYV12
o=last
a=last.Dither_convert_8_to_16 ()
b=last.binomialBlur(varY=0.5, varC=0, Y=3, U=2, V=2, useMMX=true).Dither_convert_8_to_16 ()
sharp=dither_add16 (a,dither_sub16 (a,b,dif=true),dif=true)
a.sharplimit16 (sharp)
ditherpost (mode=6)
s=last
mask=mt_hysteresis (o.mt_edge ("cartoon"),o.mt_edge ("hprewitt")).mt_expand_multi(sw=6, sh=6).mt_inpand_multi(sw=6, sh=6).tmaskcleaner ()
mt_merge (o,s,mask,y=3,u=3,v=3,luma=true)


Answer "The scripts's return value was not a video clip".
Ok, I changed:


...
p=mt_merge (o,s,mask,y=3,u=3,v=3,luma=true)
return(p)

Answer: "There is no function named "tmaskcleaner""
Although tmaskcleaner.dll x86 Version ‎24.‎12.‎2013 ‏‎21:03 is in the avisynth/plugins directory.

Ok, another approach. Put a load at the beginning:

LoadPlugin("C:\Program Files (x86)\AviSynth\plugins\tmaskcleaner")

or

LoadPlugin("C:\Program Files (x86)\AviSynth\plugins\tmaskcleaner.dll")


...answer is "Unable to load "C:\..."

If I just outcomment tmaskcleaner

...
s=last
mask=mt_hysteresis (o.mt_edge ("cartoon"),o.mt_edge ("hprewitt")).mt_expand_multi(sw=6, sh=6).mt_inpand_multi(sw=6, sh=6)#.tmaskcleaner ()
p=mt_merge (o,s,mask,y=3,u=3,v=3,luma=true)
return(p)


I am getting a sharpend result which shows absolutely no change to previous results.

What am I doing wrong?

[StainlessS]

I dont know dither at all (got it, never tried it), but, are you using v2.6, Avisynth v2.58 cannot 'see' v2.6 plugins. (tmaskcleaner)

[feisty2]

uh, tmaskcleaner is just to remove some random dots, remove it if u cant load it
not working? but it works perfectly on my computer
ur problems are all about flat areas got sharped
so, I masked flat parts out
no, the problem areas are not even processed at all, 100% from the source clip
so.... I donno whats happening on ur computer

[Hotte]

@StainlessS: A version mess... I`ve got 2.60build2012May16 and 2.60build2013Sep28. Both don't find tmaskcleaner... and the latter throws caching errors with dither.avsi!

@feisty2:
btw. which version are you using ?
It`s not a problem, I just removed tmaskcleaner, and yes, I did a stupid mistake with "... return value was not a video clip"
It's just that "mask=mt_hysteresis...mt_merge..." make no visible change to the result.

So I will experiment a bit with your excellent RGB tip and try to flatten out magenta somehow...
...and not drive the thread into further detail of magenta block artefacts that might not show up elsewhere and are maybe only a result of bad source encoding (as poisondeathray suspected).

I will soon be coming back with a final summary of this sharpen approach.

Thank you for your generous and highly qualified support so far. I´ve learned so much about the power of avisynth.

[feisty2]

u overrated me, after all, how much can a 17 years old kid know :p
I just happen to like to dig around "Google"
look, the whole thing just seems weird to me
u said the source clip is free (at least visually free) from those block and color messes
so here's what the code I wrote above does
it replaces problem block, discolored parts of the sharpened clip with the original unprocessed clip by using a mask function
so how could it be no change?

[Hotte]

Just a short message tonight, I'll be out for two days.

Yes, feisty2, I was wrong, you were right: There definitely IS a change. Is does away with fine sharpening artifacts so delivers even better results, though in some rare flat areas it does not sharpen (obviously because you restore the original). Unfortunately in chroma there is no significant improvement. For some strange reason flat blue here and there becomes flat magenta. But all in all it is not a big problem. You have to have a very, very close look to see it. What counts more is, that the sharpening is really good. I will be trying with some separated RGB settings and gradfun3....

I will be coming back soon. Don´t forget to post that lady-video (just a short clip). I would like to see how our sharpening looks in comparison. If possible put it somewhere where you do not have to download some software or register. Thanks so far I´ll be back soon.

[poisondeathray]

Unfortunately in chroma there is no significant improvement. For some strange reason flat blue here and there becomes flat magenta. But all in all it is not a big problem. You have to have a very, very close look to see it.

You must be looking at the full video or testing on another example, or you didn't quite use the same script - because he replaced it with a mask. If you are getting areas that still have it, then the mask isn't perfect (this can easily happen because the mask was for a still image, on other frames you're bound to get holes in the mask.) You can use amplified subtract or amplied differences (or other variations) to verify this, or view the mask directly to see the relationship. There is zero pixel difference in that region on the still, but maybe a few stray pixels near the borders of the edges

eg.
subtract(filtered, original).Levels(127, 1, 129, 0, 255)

But how useful a mask like that would be on actual footage instead of a still is debatable... or at minimum you would have to tweak different masks for different shots



If you still feel something in masktools is contributing to it, and it's not from the pre-exisiting artifacts (which exist in all 3 channels, you can visualize the chroma channels by UtoY() and VtoY() , then test on a clean clip with similar blue gradient. Even a CG gradient will demonstrate an issue if there is a math calculation bug. I just think it's not merely a coincidence that you get "blocks" of magenta here and there that just happen to coincide with the h264 compression artifacts

The other sharpening algorithm in Edius probably isn't using this approach of blurring and subtracting. That probably has something to do with it. And even if it is, "subtract" is defined differently in avisynth than other programs like photoshop, after effects , probably Edius etc... .Not only is the formula different, those other programs are working in RGB for their channel operations

[feisty2]

thats not a "still" mask
mt_edge creates mask based on the edge of "every" frame
the mask changes by every frame unless the clip itself is still
so, the mask is a "video" too, not a still picture

[poisondeathray]

thats not a "still" mask
mt_edge creates mask based on the edge of "every" frame
the mask changes by every frame unless the clip itself is still
so, the mask is a "video" too, not a still picture

Obviously :)

What I meant is you refined the mask for a still, not the video. Unless he uploaded the video somewhere? He mentioned something that suggests holes in the mask. Well there were hardly any holes when I looked, there were no magenta shifts like the unmasked version. It was a very good mask for that still in terms of using masktools, almost as good as if you rotoed it. So I am assuming he is checking and referring to the actual video.

What I am saying is there are other scenes, other content, other characteristics to the video, other types of edges. And that perhaps you need to adjust the settings in other shots, threshold values - Or did you think this would work perfectly for everything with those settings without looking at them ?

[feisty2]

holes?
I used expand.inpand to remove holes
maybe do it some more times?

[poisondeathray]

holes?
I used expand.inpand to remove holes
maybe do it some more times?

We don't know what is going on for sure, because he didn't post a video sample, only an image

But I don't see any big holes with your script on the still image. But what he says suggests either he's messed it up, or actually checking the video (not the stilll), or another video

"For some strange reason flat blue here and there becomes flat magenta" - I don't see that with the mask

Since we don't know what is going on for sure it's hard to suggest something, so I would wait for clarification




I think the artifacts have everything to do with this. I would treat the underlying problem first

[foxyshadis]

I wouldn't be surprised if it was a case where moving the U and V by 1 value each changed the color visibly, if it's that flat and you flipped back and forth enough.

[Hotte]

Hi everybody - thank you for your precious contribution.

I am most happy do upload a short video sample. Would 1 or max. 2 seconds be ok ?, because I have very very low upload speed and canopus avi are huge.

Would it be best to just upload the out of camera original ?

Could you recommend a storage site where I could upload without any registersoftwaredownloadwhatever hassel ?

I can do it only on the weekend cause I got that stuff at home.

Thanks.

[feisty2]

yeah, paid online hosting sites

[creaothceann]

Could you recommend a storage site where I could upload

MEGA, MediaFire

[feisty2]

found FQSharp (deconvolution filter)
now everything is done under avisynth
cool