Plum, blind deconvolution enhanced by pixel/block matching

[feisty2]

Some initial observations:


1) It's &^&%$*#* slow.

2) It crashes on large image sets (probably GPU memory issue, I tested on a 4GB card, but I wasn't about to switch to "CPU" mode yet for the KNLMeans part, because of possible different results. GPU-z does spike to 100% GPU/memory at times, but even on "smaller" images which complete ) .

3) It does mess up some test patterns. Partially because I didn't play with the settings enough - "you dummy you're supposed to use the 'right' settings"... But because it's so slow I can't try out different settings quickly on multiple sources at one go. Hard to give proper feedback or get a "feeling" for how it reacts. Then when I come back a few days later and if I didn't make notes I have to start over again.

You might ask who the &*^* watches test patterns?... Those low level tests have high predictive value where you're going to get problems manifesting in real life situations . If you can tune an algorithm to "pass" or at least look good on various patterns, without making a mess of others, then you've essentially got a "perfect" filter. (But you'll never get a perfect storm of "ideal" inputs, there's almost always some problems with source.) Or at the very least you can help to identify what areas you might improve on.

If I had time I'd try to give proper , more specific feedback, but this thing is so slow. The warp drive is down, and so are the impulse engines...

can you post the test patterns that it messed up?
I could use them as references when I'm trying to improve this thing, if you have read the script by now, you might have noticed there're a few "magic constants" which were calculated by mathematica as the result of curve fitting on the test set I got, but they could be kind of overfitting cuz I don't have a set that covers everything

[poisondeathray]

Actually, it seems to do worse then some simpler sharpeners on any static test pattern so far. I realize there is a temporal component to this filter, but I never even got to motion tests yet. Might not be using the correct settings or cutoff. Or maybe you actually can't have your cake and eat it too.

Just download a random free one, like a zone plate / wedges/ trumpets - and if you crash, maybe crop to a region of interest to test

eg.
http://www.bealecorner.org/red/test-patterns/

[feisty2]

Actually, it seems to do worse then some simpler sharpeners on any static test pattern so far. I realize there is a temporal component to this filter, but I never even got to motion tests yet. Might not be using the correct settings or cutoff. Or maybe you actually can't have your cake and eat it too.

Just download a random free one, like a zone plate / wedges/ trumpets - and if you crash, maybe crop to a region of interest to test

eg.
http://www.bealecorner.org/red/test-patterns/

yeah, the final estimation would be rendered probably 70% useless if you're testing single frame samples

[poisondeathray]

yeah, the final estimation would be rendered probably 70% useless if you're testing single frame samples

It's easy to animate charts to induce motion. You can even impart real camera motion , but motion tracking a source and applyin that to the test chart. You can add motion blur to simulate shutter blur if you wanted to (or not.) Do you think it will do any better ? My guess is probably not. My preliminary impression on very limited testing is high contrast lines tend to be a problem here.

That's just a small part of the testing. Sure it might look ok on some more organic sources, but there will be parts that will look bad if it fails on a motion zone plate

[feisty2]

It's easy to animate charts to induce motion. You can even impart real camera motion , but motion tracking a source and applyin that to the test chart. You can add motion blur to simulate shutter blur if you wanted to (or not.) Do you think it will do any better ? My guess is probably not. My preliminary impression on very limited testing is high contrast lines tend to be a problem here.

That's just a small part of the testing. Sure it might look ok on some more organic sources, but there will be parts that will look bad if it fails on a motion zone plate

wow, these test patterns are real LARGE!!
it crashes the source filter (lsmash) even before I get started!

[poisondeathray]

You can crop to region of interest... and animate that



This 4 frame test chart sample is from real camera (ie. non synthetic , with real noise, real camera issues like aliasing)

UT Video ~2.3MB
https://www.mediafire.com/?d3xbclg2xb5tad5

[feisty2]

It's easy to animate charts to induce motion. You can even impart real camera motion , but motion tracking a source and applyin that to the test chart. You can add motion blur to simulate shutter blur if you wanted to (or not.) Do you think it will do any better ? My guess is probably not. My preliminary impression on very limited testing is high contrast lines tend to be a problem here.

That's just a small part of the testing. Sure it might look ok on some more organic sources, but there will be parts that will look bad if it fails on a motion zone plate

I tested it on "Zone720-hardedge-B.png" and I'm not seeing anything messed up,,,

BEFORE:
http://i.imgur.com/qknUq4X.png
http://i.imgur.com/BK1J3Jc.png
http://i.imgur.com/TDDcJbu.png
http://i.imgur.com/YeAmKIh.png

AFTER:

clp = core.lsmas.LWLibavSource('blah.png')
clp = core.fmtc.bitdepth(clp,bits=32,fulls=True,fulld=True)
clp = core.std.ShufflePlanes(clp,0,vs.GRAY)

ref = Plum.Basic(clp)
clp = Plum.Final([clp, ref], freq_margin=0) #freq_margin=0 means the anti-bloat filter is off, kind of cheating here cuz normally it should be something greater than 0, but it doesn't make sense on test patterns

http://i.imgur.com/ZVp61NG.png
http://i.imgur.com/aueH9FM.png
http://i.imgur.com/mhp9UVK.png
http://i.imgur.com/ifPXVtO.png


the results are exactly what I expected, "just don't do nothing at all if the source is already good enough"

can you post your test script?

[poisondeathray]

Go look at the fine patterns .

I was intrigued earlier about some claims about "sharpening" without haloing artifacts. But I don't think it's possible unless it's very weak strength, or unless you do some sort of masking or differential application. You're going to mess up the fine patterns eventually when you turn up the strength. There's got to be some give and take. If you enhance the coarse details, the fine details are going to be oversharpened eventually. That will show up on the test pattern

I used only default settings so far, should I have been using something different ?

[feisty2]

Go look at the fine patterns .

I was intrigued earlier about some claims about "sharpening" without haloing artifacts. But I don't think it's possible unless it's very weak. You're going to mess up the fine patterns eventually when you turn up the strength.

I used only default settings so far, should I have been using something different ?

the first image I posted above is the "finest" part I could find in that test image.

make "freq_margin" 0 if you're doing tests on test patterns

[feisty2]

another one, with very "fine" features (from "star-chart-bars-full-600dpi.png")
same script as in #57

BEFORE:
http://i.imgur.com/NMjAbUC.png

AFTER:
http://i.imgur.com/YDKmeGc.png

still nothing messed up

[poisondeathray]

so freq_margin=0 is the tweaked setting for high con / test patterns ?

so it takes an a$$ long time to do nothing :devil: ? Well, why not just do nothing ? :D . Hell of a lot faster

Try it on the "video" test pattern in 56 , both with default, and freq_margin=0 "cheat" settings . Both are worse than original or some weak simple sharpening

[feisty2]

Go look at the fine patterns .

I was intrigued earlier about some claims about "sharpening" without haloing artifacts. But I don't think it's possible unless it's very weak strength, or unless you do some sort of masking or differential application.

Non-Local Error is a self-adaptive algorithm, it automatically ignores the very sharp edges cuz it weights on "value" instead of "euclidean distance", which means it's a non-linear filter and it does not have a "nice" frequency domain representation like normal USM filters

so it does not produce ringings/halos


so it takes an a$$ long time to do nothing ? Well, why not just do nothing ? . Hell of a lot faster.
it does certainly "something" if the algorithm thinks the image should get sharpened..:)

[feisty2]

You can crop to region of interest... and animate that



This 4 frame test chart sample is from real camera (ie. non synthetic , with real noise, real camera issues like aliasing)

UT Video ~2.3MB
https://www.mediafire.com/?d3xbclg2xb5tad5

can you try with another codec and another container...
maybe something less,, alternative?
I tried like 5 versions of lsmash and all crashed trying to read this thing..

say, maybe, h264 lossless + mkv?

[poisondeathray]

Image is "already sharp" so do nothing - ok that makes sense. But you're using freq_margin=0 - is that what would be used for "normal" situations ? Because it messes up pattern when you don't use that setting .

Crap I thought UT was pretty standard. I think it might be because this is the VFW version

You can convert with ffmpeg

ffmpeg -i input.ext -c:v libx264rgb -crf 0 -an output.ext

[poisondeathray]

Why can ffms2 open it (the original ULRG sample), but lsmash can't ? AVISource works too, but relies on your VFW system

[feisty2]

Why can ffms2 open it (the original ULRG sample), but lsmash can't ?

who knows, I've downloaded ffms and now doing my test with it.

[feisty2]

Why can ffms2 open it (the original ULRG sample), but lsmash can't ? AVISource works too, but relies on your VFW system

I've spotted the problem for your test video
open Plum.py and go to line 125, change

clip = core.Shrink(limited)

to

clip = limited #core.Shrink(limited)


"Shrink" is a helper function and it tries to shrink the image down to make it look "finer" and more "delicate"
it's a nice feature and generally works for most videos but fails with your test clip cuz your test clip has some severe aliasing and it messed the "aliasing" up, so, it made the bad even worse and that's it...


But you're using freq_margin=0 - is that what would be used for "normal" situations ? Because it messes up pattern when you don't use that setting .

your test clip is screwed not because of "freq_margin", it's just too aliased to get shrunk, comment out the "Shrink" function and it's gonna be fine (but you might want the "Shrink" function back when you're doing tests on real life videos)

[poisondeathray]

ok thanks , I'll eventually keep on doing more tests, but I'll keep those 2 things in mind with different sources

The aliasing might look "severe" but that's actually pretty typical pattern for consumer point and shoot. For modern HD and UHD cameras, you 'll commonly see aliasing above a certain resolution (higher than the one shown there, but the pattern will be similar). For common DSLR, it's usually much worse, with ugly color moire patterns on top



your test clip is screwed not because of "freq_margin", it's just too aliased to get shrunk, comment out the "Shrink" function and it's gonna be fine (but you might want the "Shrink" function back when you're doing tests on real life videos)

But "clean" test patterns get screwed up (or look less than ideal) because of not adjusting it to zero - so there is a direct relationship independent of shrink

So what values of freq_margin are you typically using? Do you make adjustments on the fly ? Or was that only for synthetic clean test chart and you typically leave it ? I see in the py as 20 , and must be : freq_margin < 0 or freq_margin > 100-cutoff

[feisty2]

ok thanks , I'll eventually keep on doing more tests, but I'll keep those 2 things in mind with different sources

The aliasing might look "severe" but that's actually pretty typical pattern for consumer point and shoot. For modern HD and UHD cameras, you 'll commonly see aliasing above a certain resolution (higher than the one shown there, but the pattern will be similar). For common DSLR, it's usually much worse, with ugly color moire patterns on top



But "clean" test patterns get screwed up (or look less than ideal) because of not adjusting it to zero - so there is a direct relationship independent of shrink

So what values of freq_margin are you typically using? Do you make adjustments on the fly ? Or was that only for synthetic clean test chart and you typically leave it ? I see in the py as 20 , and must be : freq_margin < 0 or freq_margin > 100-cutoff

I did my tests on a lot of music videos which were shot on 35mm film and then scanned to digital format (some were shot in digital format, but shot with fancy cameras like RED), and I compared the DVD release and the master tape of the same video and tried to match them, I didn't do any test on videos shot by cheap cameras

[poisondeathray]

Actually "expensive" cameras like RED , Alexa, high end Sony's show the SAME pattern of aliasing and moire, the difference being at the higher resolutions and finer patterns. Just download some sample test footage, there's plenty around. The reason I uploaded that "low resolution" example was because of the crashing. You can't process footage from the other cameras at native dimensions with this, at least not currently . It might not be GPU memory/usage, because even "small" dimension tests max it out this 4GB card without crashing, so not sure about why

[feisty2]

Actually "expensive" cameras like RED , Alexa, high end Sony's show the SAME pattern of aliasing and moire, the difference being at the higher resolutions and finer patterns. Just download some sample test footage, there's plenty around. The reason I uploaded that "low resolution" example was because of the crashing. You can't process footage from the other cameras at native resolution with your filter

That aliasing in ur test clip is not temporally stable, yes you can kill the spatial aliasing by downscaling but that's not gonna work in temporal dimension, you still have that "shimmering" thing and I've never seen that kind of temporal aliasing in my test videos

If the aliasing was temporally stable, the "Shrink" function might have just worked cuz the motion compensation in the final estimation could have swept that crap away,,,

[poisondeathray]

That aliasing in ur test clip is not temporally stable, yes you can kill the spatial aliasing by downscaling but that's not gonna work in temporal dimension, you still have that "shimmering" thing and I've never seen that kind of temporal aliasing in my test videos

If the aliasing was temporally stable, the "Shrink" function might have just worked cuz the motion compensation in the final estimation could have swept that crap away,,,


The same thing happens with >$50K cameras (at native dimensions) . The fine details are aliased . This is how "resolution" is defined - the limit at which you can distinguish "clean" patterns. Yes, when you downsample to HD (or in your case SD), you lose the high freq. details, thus you lose the moire/aliasing/shimmering . That's one reason why oversampling is fantastic. Your SD test cases are downsampled and "clean" in that respect

ALL cameras have aliasing with finer details, higher "resolutions" . For example, the Alexa is clean until about 800-900 lines then you get the same shimmering if you move the camera when shooting something similar . No, you don't typically shoot a test pattern, but things like that occur in real life, e.g. a brick wall, tiles on a building in the distance etc... That's why in some respects, a "cheap" consumer UHD camera can outperform a $50K "HD" camera

Synthetic test pattern can be completely clean ,even in motion if you wanted it to be, but that doesn't necessarily reflect real world acquisition formats - but it's important include those types of low level tests




Traditional linear /simple sharpener will create problems on the finer patterns (when there was none to begin with) because of the artifacting and haloing, but still sharpen the coarse patterns. If you set low level it might be acceptable (because your footage might not have those fine details to begin with). But when set freq_margin to zero, it does nothing here, even on coarse patterns - I guess I'm trying to get a feeling for how to tweak/adjust this filter on different sources, but it's so hard because it's so slow

(another thing I miss is tabs in avspmod, it's easy to compare different settings in different tabs, and you can "learn" a filter a lot faster. I made request for vapoursynth editor before, but got shot down)

[madshi]

FWIW, I have some photos (converted from RAW to TIF) shot with RED Helium here, but I don't see aliasing in those. I see color fringing, but overall it's rather soft (when 1:1 pixel peeping).

[feisty2]

The same thing happens with >$50K cameras (at native dimensions) . The fine details are aliased . This is how "resolution" is defined - the limit at which you can distinguish "clean" patterns. Yes, when you downsample to HD (or in your case SD), you lose the high freq. details, thus you lose the moire/aliasing/shimmering . That's one reason why oversampling is fantastic. Your SD test cases are downsampled and "clean" in that respect

ALL cameras have aliasing with finer details, higher "resolutions" . For example, the Alexa is clean until about 800-900 lines then you get the same shimmering if you move the camera when shooting something similar . No, you don't typically shoot a test pattern, but things like that occur in real life, e.g. a brick wall, tiles on a building in the distance etc... That's why in some respects, a "cheap" consumer UHD camera can outperform a $50K "HD" camera

Synthetic test pattern can be completely clean ,even in motion if you wanted it to be, but that doesn't necessarily reflect real world acquisition formats - but it's important include those types of low level tests




Traditional linear /simple sharpener will create problems on the finer patterns (when there was none to begin with) because of the artifacting and haloing, but still sharpen the coarse patterns. If you set low level it might be acceptable (because your footage might not have those fine details to begin with). But when set freq_margin to zero, it does nothing here, even on coarse patterns - I guess I'm trying to get a feeling for how to tweak/adjust this filter on different sources, but it's so hard because it's so slow

(another thing I miss is tabs in avspmod, it's easy to compare different settings in different tabs, and you can "learn" a filter a lot faster. I made request for vapoursynth editor before, but got shot down)

ok, new information noted
issue: because aliasing is rarely seen in SD and HD videos shot by professional cameras, I never considered if that would be a problem, my main focus was on ringing cuz that's what happens with regular USM filters, the default settings might fail if the source suffers from obvious aliasing..
solution: turn off the "Shrink" filter


you seem to be very interested in the "freq_margin" parameter, I should probably explain how it works..

freq_margin was designed to act as an anti-bloat filter, it saves the image from "xylographing" as already explained here (https://forum.doom9.org/showthread.php?p=1776542#post1776542), it works as an extension to the "cutoff" parameter

first we decompose the image into MANY frequency-wise components, and rank them.
say,
0 -> lowest frequency component,
100-> highest frequency component,
Plum being a high pass filter does not affect frequencies labeled from 0 to "cutoff", frequencies labeled from "cutoff" to 100 will be processed.

now because high frequencies in the sharpened clip have more "energy", if we merge source[0, cutoff] with sharpened[cutoff, 100], there will be more energy than the source clip and the result will look "xylographed" (energy(sharpened[cutoff, 100]) > energy(source[cutoff, 100]))

to keep the overall energy amount approximately unaffected, we have to "sacrifice" certain frequencies in the source clip, it can't be the low frequencies, cuz that's the base of the entire image, it can't be the high frequencies, cuz that's what we want, fine details and stuff..

so certain medium frequencies have to die for the greater good (or more of, they are already shifted to high frequencies in the sharpened clip anyways), and "freq_margin" gets to decide, how many medium frequency components should die, basically, source[cutoff, cutoff+freq_margin] will be removed

so the final result is actually:
source[0, cutoff] + sharpened[cutoff+freq_margin, 100]

[poisondeathray]

FWIW, I have some photos (converted from RAW to TIF) shot with RED Helium here, but I don't see aliasing in those. I see color fringing, but overall it's rather soft (when 1:1 pixel peeping).

Anytime you have footage that is "soft" without aliasing, very likely it's been shot with a strong OLPF - i.e the fine details and high freq details have already been cut off



you seem to be very interested in the "freq_margin" parameter, I should probably explain how it works..

freq_margin was designed to act as an anti-bloat filter, it saves the image from "xylographing" as already explained here, it works as an extension to the "cutoff" parameter

first we decompose the image into MANY frequency-wise components, and rank them.
say,
0 -> lowest frequency component,
100-> highest frequency component,
Plum being a high pass filter does not affect frequencies labeled from 0 to "cutoff", frequencies labeled from "cutoff" to 100 will be processed.

now because high frequencies in the sharpened clip have more "energy", if we merge source[0, cutoff] with sharpened[cutoff, 100], there will be more energy than the source clip and the result will look "xylographed" (energy(sharpened[cutoff, 100]) > energy(source[cutoff, 100]))

to keep the overall energy amount approximately unaffected, we have to "sacrifice" certain frequencies in the source clip, it can't be the low frequencies, cuz that's the base of the entire image, it can't be the high frequencies, cuz that's what we want, fine details and stuff..

so certain medium frequencies have to die for the greater good (or more of, they are already shifted to high frequencies in the sharpened clip anyways), and "freq_margin" gets to decide, how many medium frequency components should die, basically, source[cutoff, cutoff+freq_margin] will be removed

so the final result is actually:
source[0, cutoff] + sharpened[cutoff+freq_margin, 100]


Ok thanks, it will take some time to digest...

I am interested, because I think this is the "key" to being useful in actual usage - being able to accurate categorize and filter certain ranges . I need to play with it more

[poisondeathray]

Is there a way to code a helper function or visualization aid (preferably quick processing preview, minus all the processing) for what "it" thinks is a frequency range ? For example , maybe shade or overlay the values (0-30) (or whatever you enter) red or something ?

[feisty2]

Is there a way to code a helper function or visualization aid (preferably quick processing preview, minus all the processing) for what "it" thinks is a frequency range ? For example , maybe shade or overlay the values (0-30) (or whatever you enter) red or something ?

What frequency range? Are you saying like calculate "cutoff" and "freq_merge" automatically?

[poisondeathray]

What frequency range? Are you saying like calculate "cutoff" and "freq_merge" automatically?



I mean as a debugging option. You have normalized to the range 0-100, so let's say your cutoff is 20. This means 0-20 will not be affected, right? I want to "see" what it has categorized as 0-20 . Is this the correct cutoff to use for that particular source, or your particular goal ? Maybe the value should have be 36.5 for this specific goal. Or maybe to visualize the modulation effect of a different freq_margin value. I supposed you could just use crazy strenght settings and see where the effect is applied , but remember this is a slow filter . The visualization of exclusion/inclusion areas without actually applying the effect won't go through all the steps/filters and should be faster shouldn't it ?

[feisty2]

I mean as a debugging option. You have normalized to the range 0-100, so let's say your cutoff is 20. This means 0-20 will not be affected, right? I want to "see" what it has categorized as 0-20 . Is this the correct cutoff to use for that particular source, or your particular goal ? Maybe the value should have be 36.5 for this specific goal. Or maybe to visualize the modulation effect of a different freq_margin value. I supposed you could just use crazy strenght settings and see where the effect is applied , but remember this is a slow filter . The visualization of exclusion/inclusion areas without actually applying the effect won't go through all the steps/filters and should be faster shouldn't it ?

"cutoff" is a frequency-wise threshold, you simply can't map it to a mask kind of thing, cuz every pixel is processed in the spatial domain, but you can indeed check the low frequency components and it's fast

call the "lowpass" function here,

def lowpass(src, p):
upsmp = core.fmtc.resample(src, src.width*2, src.height*2, kernel="gauss", a1=100, fulls=True, fulld=True)
clip = core.fmtc.resample(upsmp, src.width, src.height, kernel="gauss", a1=p, fulls=True, fulld=True)
return clip

e.g.
lowpass(clp, 20) -> clp[0, 20]
core.std.MakeDiff(clp, lowpass(clp, 45)) -> clp[55, 100]

[feisty2]

I mean as a debugging option. You have normalized to the range 0-100, so let's say your cutoff is 20. This means 0-20 will not be affected, right? I want to "see" what it has categorized as 0-20 . Is this the correct cutoff to use for that particular source, or your particular goal ? Maybe the value should have be 36.5 for this specific goal. Or maybe to visualize the modulation effect of a different freq_margin value. I supposed you could just use crazy strenght settings and see where the effect is applied , but remember this is a slow filter . The visualization of exclusion/inclusion areas without actually applying the effect won't go through all the steps/filters and should be faster shouldn't it ?

a few more things you should probably be careful with:

the final estimation does not work on videos with sudden scene changes, you might wanna cut each individual scene out and process them separately, either manually or with some batch processing program

if your test videos are very large and you don't have a supercomputer, either downscale them or crop them, I've never tested this thing on any video larger than 720p on my i7-4790k/GTX 970/16G RAM

pel = 2 is faster and does not affect the quality much

use vspipe + rawsource and save the intermediate results (like Super or Basic) as uncompressed raw binaries on your hard drive, it's gonna be A LOT faster if you plan to actually apply this thing to your videos

[feisty2]

I uploaded the alpha version (prototype) of the next Plum release: https://github.com/IFeelBloated/Plum/blob/master/PlumAlpha.py
it features 2 major changes:
the motion estimation part was mostly redesigned
the basic estimation now produces much less aliasing with NLMeans rectifying

you could replace the old version with this alpha test version now and then continue with your tests

[poisondeathray]

thanks feisty ,

I'll have to get back to it later this week, but this deserves some more testing on varied sources



Have you tried on "typical" DVD source yet, where there is a bit of grain / noise ? I noticed many of your tests are from music DVD's kylie & such , which might be "cleaner" than say some action movie DVD . What are your general impressions in that scenario

[feisty2]

thanks feisty ,

I'll have to get back to it later this week, but this deserves some more testing on varied sources



Have you tried on "typical" DVD source yet, where there is a bit of grain / noise ? I noticed many of your tests are from music DVD's kylie & such , which might be "cleaner" than say some action movie DVD . What are your general impressions in that scenario

I don't really have any DVD movies, all movie disks I got are bluray release, cuz unlike early music videos which were shot on 35mm film but mastered at an NTSC/PAL resolution, movies were at least mastered at 2k or even higher resolution, so I will have a native HD copy with bluray release, and DVD releases are just garbage. (relatively speaking)

[poisondeathray]

Ok or cropped BD (or full if you don't crash :) ) ; I'm interested in the context of underlying grain/noise . I'll eventually test these but I'm wondering if you have to prefilter or use special treatment . For example, large grain film stock vs. fine grain pattern

[feisty2]

thanks feisty ,

I'll have to get back to it later this week, but this deserves some more testing on varied sources



Have you tried on "typical" DVD source yet, where there is a bit of grain / noise ? I noticed many of your tests are from music DVD's kylie & such , which might be "cleaner" than say some action movie DVD . What are your general impressions in that scenario

if you're doing the new alpha version,
there's a value that controls the aliasing-fine structure tradeoff and ain't getting parameterized yet..

line 143

dif = core.NLMeans(dif, a, 4, 3.2, src)

"3.2" is the tradeoff value here, larger = smoother result(less aliasing) but weaker enhancement on fine structures

[feisty2]

I removed some useless legacy stuff in the alpha version, please update your PlumAlpha.py
and I did some tests on a high quality (mastertape, ProRes 422HQ, zero compression artifacts) 1080p clip (Ariana Grande - Love Me Harder)
HD videos (even the master tapes) are generally much softer than (master tape quality) SD videos, so it makes sense to sharpen them if you prefer a sharper image..

source (cropped)
http://i.imgur.com/eKsp84M.png

USM (naïve)

clp = core.std.MergeDiff(clp, core.std.MakeDiff(clp, core.std.Convolution(clp,[1,2,1,2,4,2,1,2,1])))

http://i.imgur.com/NhwG52e.png

smarter USM (LSFmod)

clp = havsfunc.LSFmod(clp,defaults="slow",strength=164)

http://i.imgur.com/KDYABhC.png

even smarter USM (finesharp)

clp = finesharp.sharpen(clp, sstr=1.2)

http://i.imgur.com/3ZG6TTq.png

plum (alpha ver)

ref = PlumAlpha.Basic(clp)
clp = PlumAlpha.Final([clp, ref], [PlumAlpha.Super(clp), PlumAlpha.Super(ref)], cutoff=20, freq_margin=12)

http://i.imgur.com/k8zzrUd.png

[feisty2]

conclusion:
the Plum result looks more like a native 1080p picture than the source image, some people might think the Plum result should be the ground truth and the source image should be a 720p upscale in a blind test...
other algorithms enhanced the overall sharpness (while Plum didn't), but failed to make the image more "delicate"

[madshi]

Personally, I'm a big fan of the "delicate" look, but I can identify some problems in the Plum screenshot:

1) There are some "distortions". E.g. the 3rd candle from the right gets fatter/distorted.
2) Some parts of the image get sharpened *a lot* (e.g. the arm chair lattice), other parts not at all (e.g. the face or the candle lamp).
3) There are some ringing artifacts (e.g. above the "bow" at the top center of the image).
4) There are some weird artifacts (e.g. left of the arm chair some of the bright window pixels "streak" into the dark drapes (or whatever that is)).
5) Sometimes some things/lines get broader instead of thinner, e.g. the sharpening of the wood plate at the very left side of the arm chair looks kinda weird to me.

I've been working on an "ideal deconvolution" filter, which almost perfectly reverts a gaussian blur. At relatively low sharpening strengths like in this screenshot comparison my wip filter looks very similar to FineSharp. Which makes me think that maybe Plum is going a bit overboard with just sharpening high frequencies to be even higher/sharper, but completely ignoring the rest of the image? I think I'd prefer Plum to sharpen e.g. the face or the candle lamp or the bra or [...], too, similar to FineSharp. Although, granted, where Plum does get active, it looks nice, probably nicer than FineSharp. The artifacts do worry me, though.

Just my 2 cents, of course. What do you think?

[poisondeathray]

conclusion:
the Plum result looks more like a native 1080p picture than the source image, some people might think the Plum result should be the ground truth and the source image should be a 720p upscale in a blind test...
other algorithms enhanced the overall sharpness (while Plum didn't), but failed to make the image more "delicate"


Keep it coming . I knew you would pop out another version, so I didn't even test the last update :)

One problem is what is "native 1080p" picture supposed to be ? Often it is supposed to be "soft" , as it's shot with diffusion filters and soft dof focus on purpose. Other times it's supposed to be razor sharp, eg. maybe reality TV or documentary.



Some pros/cons to each of them . Depends what you're going for

If you nearest neighbor 2x to look more closely, the plum version has problems with haloing around the light, edge of skirt by her knee

Some look semi-unnatural like the USM, the singer is supposed to be plane in focus, some sharpen the BG too much for this type of shot I would argue

[feisty2]

after several tests on different kinds of videos, I decided to remove NLMeans rectifying, it generally has more cons than pros..
and I changed the default value for a few parameters, you may update PlumAlpha.py if you want to.

new test result of the same video at #86, it now sharpens more low frequency details and hopefully solved some of the mentioned problems..
there still might be some problems, but the tradeoff looks good to me

source (1080p full frame)
https://s3.postimg.org/ifbci86cx/src.png
plum

ref = PlumAlpha.Basic(clp)
clp = PlumAlpha.Final([clp, ref], [PlumAlpha.Super(clp), PlumAlpha.Super(ref)], cutoff=20, freq_margin=12)

https://s18.postimg.org/5dfsdtajb/plum.png

[feisty2]

Personally, I'm a big fan of the "delicate" look, but I can identify some problems in the Plum screenshot:

1) There are some "distortions". E.g. the 3rd candle from the right gets fatter/distorted.
2) Some parts of the image get sharpened *a lot* (e.g. the arm chair lattice), other parts not at all (e.g. the face or the candle lamp).
3) There are some ringing artifacts (e.g. above the "bow" at the top center of the image).
4) There are some weird artifacts (e.g. left of the arm chair some of the bright window pixels "streak" into the dark drapes (or whatever that is)).
5) Sometimes some things/lines get broader instead of thinner, e.g. the sharpening of the wood plate at the very left side of the arm chair looks kinda weird to me.

I've been working on an "ideal deconvolution" filter, which almost perfectly reverts a gaussian blur. At relatively low sharpening strengths like in this screenshot comparison my wip filter looks very similar to FineSharp. Which makes me think that maybe Plum is going a bit overboard with just sharpening high frequencies to be even higher/sharper, but completely ignoring the rest of the image? I think I'd prefer Plum to sharpen e.g. the face or the candle lamp or the bra or [...], too, similar to FineSharp. Although, granted, where Plum does get active, it looks nice, probably nicer than FineSharp. The artifacts do worry me, though.

Just my 2 cents, of course. What do you think?

I did a new test, what do you think about the new result?

[feisty2]

One problem is what is "native 1080p" picture supposed to be ? Often it is supposed to be "soft" , as it's shot with diffusion filters and soft dof focus on purpose. Other times it's supposed to be razor sharp, eg. maybe reality TV or documentary.



Some pros/cons to each of them . Depends what you're going for

but even the sharpest (non-CG) 1080p video I've ever seen is much softer than the master tape image I posted at #1...
this particular video might have some diffusion filter applied to generate the dreamy haze, but nothing really, nothing 1080p has the sharpness even close to the master tape image I posted at #1




If you nearest neighbor 2x to look more closely, the plum version has problems with haloing around the light, edge of skirt by her knee

Some look semi-unnatural like the USM, the singer is supposed to be plane in focus, some sharpen the BG too much for this type of shot I would argue


some of the problems might have been solved in the new test result, I guess...

the arm chair is getting a lot more sharpening than Ariana because there's simply "nothing" to start with for the Ariana part in the image, it's detail-less, while the arm chair has plenty of complex but soft patterns that are prefect for sharpening, I mean Ariana's bra is getting some sharpening in the new result..

[feisty2]

any more feedbacks?
I'm gonna move on and test another video if y'all happy with the result of this Ariana video..?

[madshi]

I did a new test, what do you think about the new result?
Looks like a solid improvement to me. Still not "perfect", though (but that might impossible to achieve, I don't know). Here's the plum image with a few image locations marked where I think plum shows some artifacts:

http://madshi.net/plum.png

And generally, I still think lower frequencies could use (even) more sharpening. But then, I've only looked at this one image. It's never good to judge an algo by just looking at one single image. So please take my comments with a big pinch of salt.

[feisty2]

Looks like a solid improvement to me. Still not "perfect", though (but that might impossible to achieve, I don't know). Here's the plum image with a few image locations marked where I think plum shows some artifacts:

http://madshi.net/plum.png


I think those tiny pixel-wise artifacts might be gone with a different set of "wn" and "scale" values,,, I haven't tested thoroughly, I just thought, these pixel-wise artifacts should be "invisible" in motion anyways so guess I'm gonna move on to another test sample now :devil:


And generally, I still think lower frequencies could use (even) more sharpening. But then, I've only looked at this one image. It's never good to judge an algo by just looking at one single image. So please take my comments with a big pinch of salt.

that's more of a personal taste kinda thing, I just love the high micro contrast sort of "delicate" image (the high frequencies should be, real HIGH, but it shouldn't change the "nature" of the original image in general), you could easily get more low frequency sharpening with a lower "cutoff"

[MonoS]

Probably the default Plum parameters aren't the best when applied to anime material.
The source is just detelecined and cropped.

In order: Source, Plum, PlumAlpha
https://abload.de/img/clipboard01ydsee.png

[feisty2]

Probably the default Plum parameters aren't the best when applied to anime material.
The source is just detelecined and cropped.

In order: Source, Plum, PlumAlpha
https://abload.de/img/clipboard01ydsee.png

no, I don't make filters for CG stuff,,, it's just way too different from photo-like stuff...

you want things to be extra "delicate" with photographic materials (high micro contrast), but not with CG graphics, vectorization is the ultimate answer to all CG graphics out there

photographic materials are just, more "fragile" in every single way than CG graphics, one mistake and all that precious photographic texture is gone permanently, it degrades to some semi-CG picture, which is probably the reason why stuff like waifu2x fails miserably on photographic images

[poisondeathray]

no, I don't make filters for CG stuff,,, it's just way too different from photo-like stuff...

you want things to be extra "delicate" with photographic materials (high micro contrast), but not with CG graphics, vectorization is the ultimate answer to all CG graphics out there

photographic materials are just, more "fragile" in every single way than CG graphics, one mistake and all that precious photographic texture is gone permanently, it degrades to some semi-CG picture, which is probably the reason why stuff like waifu2x fails miserably on photographic images


Be careful how you use the term "CG" so loosely. There is realistic CG that is indistinguishable from real life photos. You're referring to a specific subset of CG

[feisty2]

Be careful how you use the term "CG" so loosely. There is realistic CG that is indistinguishable from real life photos. You're referring to a specific subset of CG

you know what I was trying to say, the cheesy basic CG, not the fancy Hollywood stuff, and that's good enough.

[poisondeathray]

you know what I was trying to say, the cheesy basic CG, not the fancy Hollywood stuff, and that's good enough.

yes, I knew what you were saying, but it's better to be clear about intended usage or applicable scenarios (maybe in the documentation or readme). You're going to save yourself some headaches when someone doing a quick search , or doing some quick tests , asks questions or how come so and so doesn't work . And you will save users the headache of having to test the slow filter on stuff it wasn't meant to be used on