Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 30th November 2021, 17:50   #661  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,351
Is there a special scene or frame you have issues with? I tested with the Carrie call (I use tr 1 and 2 and thSAD=400 for the 2nd call) and found it good enough.
Also keep in mind this is a BW film so plane=0 and chroma=false.

The film shows "fizz" grain. It's in between the Carrie example and something stronger like 300. I found that a blksize of 32 is good to avoid catching these kind of big grains, along refinemotion this is a good approach. Filters like SPresso or STTWM are not suited here because they only filter minimal change noise, these are more for DVDs or faint grain.
My general approach with strong grain is a rather strong prefilter (with LFR), and a rather not so strong filter without LFR.

Personally for my encodes I don't like the clean look because they either look psychovisually blurry or sometimes it shows the "screen" effect, like grain stuck over a glass, so I apply some artificial grain after denoising.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread

Last edited by Dogway; 30th November 2021 at 17:59.
Dogway is offline   Reply With Quote
Old 30th November 2021, 21:13   #662  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,510
Quote:
Originally Posted by Dogway View Post
Is there a special scene or frame you have issues with? I tested with the Carrie call (I use tr 1 and 2 and thSAD=400 for the 2nd call) and found it good enough.
Also keep in mind this is a BW film so plane=0 and chroma=false.
The issue is the resulting bitrate, too high for a BW movie.

The best results are from SPresso+STPresso+SMDegrain but the speed is in the order of 0.5 fpsL too slow. I am eager to see your porting of STPresso if it could achieve better speeds and the same quality.

Plus, I have found that, strangely, if you apply plane=0 and chroma=false, you obtain a lower noise cleanup.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 1st December 2021, 09:29   #663  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,708
Quote:
Originally Posted by tormento View Post
The issue is the resulting bitrate, too high for a BW movie.
BW movies are often very grainy. Also, chroma information is a much smaller portion of data than luma.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 1st December 2021, 09:33   #664  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,510
Quote:
Originally Posted by Boulder View Post
BW movies are often very grainy.
Results vary by the "type" of grain, i.e. silver particle size, distribution, etc. Some BW movies are really easy to compress and you can achieve really high compression ratio.
Quote:
Originally Posted by Boulder View Post
Also, chroma information is a much smaller portion of data than luma.
The strange thing is that using plane=0 and chroma=false, lowers the noise reduction, even using Greyscale() to cleanup chroma completely. That's a thing I will never understand.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 1st December 2021, 11:25   #665  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,351
The chroma plane which is plain grey will be considered for the SAD calculation so it lowers the overall SAD value, moreover when I use a scaleCSAD of 1 for HD sources. Chroma planes are always greyish so the difference is not much but it might be noticeable. You have to counteract this with a higher thSAD.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread
Dogway is offline   Reply With Quote
Old 2nd December 2021, 17:46   #666  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,510
@dogway

Do you see anything wrong with

SMDegrain (tr=6,PreFilter=4,thSAD=900,contrasharp=false,refinemotion=true, plane=0, chroma=false, Globals=2)
SMDegrain (tr=6, thSAD=900,contrasharp=false,refinemotion=true, plane=0, chroma=false, Globals=1)


instead of using higher tr and thSAD?

Which choice is, quality wise, better?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 2nd December 2021, 18:56   #667  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,351
Ah, I noticed some issues, well Globals is something I haven't touched in a long time.
I recommend you for the time being to not use Globals. Other than that I see the output is very clean, I'm not accustomed to using such high values, maybe you can add some LFR recovery. Maybe the first pass can use blksize of 32 to speed up processing, but this depends on what details are being wiped or not, it depend on grain amount.

Using two passes is fine, it indirectly multiplies the effect for a fraction of the performance. (-6) -6 0 6 (6) = 24 frames for the current frame output.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread
Dogway is offline   Reply With Quote
Old 2nd December 2021, 20:32   #668  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,510
Quote:
Originally Posted by Dogway View Post
Ah, I noticed some issues, well Globals is something I haven't touched in a long time.
Please I think that the possibility to calculate vectors only once is useful.
Quote:
Originally Posted by Dogway View Post
I recommend you for the time being to not use Globals.
So should I use 2 lines of SMDegrain without passing vectors?
Quote:
Originally Posted by Dogway View Post
Other than that I see the output is very clean
I just can see some issues right before scene change, where details get somehow lost.
Quote:
Originally Posted by Dogway View Post
I'm not accustomed to using such high values, maybe you can add some LFR recovery.
Perhaps you are not fond of old movies, where grain is not details but simply useless noise. I keep on battling with that.

Do you suggest me to put LFR in the first line, in the second or both?

12 angry men, which I sent you a clip of, is really a bloody mess. With x264 crf 20, I am sweating to keep bitrate unter 4000kb/s and for a BW movie it is really a lot.
__________________
@turment on Telegram

Last edited by tormento; 2nd December 2021 at 20:47.
tormento is offline   Reply With Quote
Old 3rd December 2021, 08:53   #669  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,351
I already fixed the issue with Globals. There are already many changes for new SMDegrain, I was waiting to add RGB support and cumulative YPlaneMin/YPlaneMax to ex_retinex but I lost part of that work plus I was having issues with runtime variables readback.

I only need to finish an issue with UHDhalf prefiltering (switched position so prefiltering happens on dowscaled version) and I do the release. Next v3.4.0 is a good candidate for long stable release.

Running 2 passes is a good idea for several reasons, as you might know 2 mean averages approximates a binomial weight:

ex_boxblur(3,mode="weighted") <-> ex_boxblur(1,mode="mean").ex_boxblur(1,mode="mean")

This is called tent blur and in the context of motion vectors this is more performant. Besides you calculate new vectors from a cleaner source so its a finer refinement of MV.

For MDegrain this is like running a median twice, so it approximates a weighted median.
Together you perform a stronger denoising but with stronger center weighting, in other words better protection and speed than a single call with higher 'tr'.

With Globals this is not possible because you are reusing the "dirty" Globals of the first pass (the output Globals of the first prefiltering) and the only benefit is the weighted median. (In contrast using the first pass as a prefiltering only you do refine the MV but don't perform the weighted median)
Globals was designed when combining filters that use MV in rather clean sources, for example for temporal limiting, frame interpolation, etc.

I do like old films, actually I hate new digital recorded movies but lately I have been only developing rather than encoding. If you use such high thSAD or tr, LFR and DCTFlicker by extensions are a must. By Didée words grain is a high frequency only property but DCT encoding propagates this to low frequency. 'DCTFlicker' is not perfect so I recommend running both at first pass and lower thSAD and tr for the second pass to don't destroy what was recovered.

You can optimize x264 bitrate by using preset 'slower' or 'very slow'. Using 'zones' for the opening and credits. Also using a range limiter in avisynth. And a light blur where motion vectors fail (the 'mfilter' arg)
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread

Last edited by Dogway; 3rd December 2021 at 09:05.
Dogway is offline   Reply With Quote
Old 3rd December 2021, 10:16   #670  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,510
Quote:
Originally Posted by Dogway View Post
You can optimize x264 bitrate by using preset 'slower' or 'very slow'. Using 'zones' for the opening and credits. Also using a range limiter in avisynth. And a light blur where motion vectors fail (the 'mfilter' arg)
Thanks for your precious advices.

I am using slow preset as my CPU is the real bottleneck of my computer.

What about introducing a "pass" or "recursion" parameter, to internally apply the consecutive denoising? Perhaps it could lead to more speed and less roundings.
__________________
@turment on Telegram

Last edited by tormento; 3rd December 2021 at 10:18.
tormento is offline   Reply With Quote
Old 3rd December 2021, 16:52   #671  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,351
Ok, I just uploaded a new version. It needs latest avisynth, test32 I think. By the way I included a new prefilter, ex_FluxSmoothST(), this is MinBlur + FluxSmoothT, which share the same concept but one is spatial and the other temporal. So now DFTTest is 4, KNLMeanCL 5 and BM3D 6.

I'm reworking the whole packs to be numerically stable and fast by using the latest updates by pinterf (we should build him a monument or at least a xmas present :D )

You might notice some speed improvements when I update ExTools, currently I'm finishing QC but it might take more days to be sure I don't screw it. I mainly have to assure that the new f32 scale_inputs works fine across the board along the new range_max constant (as full scale).

ex_retinex() needs more work to include cumulative stats, they will be in next version.

Do you mean recursion for MDegrain only (not MV)? That's possible. I will run tests to see how it improves things. It fits in the scope of current recursion solution by prepending an "r", so it would work as mode="rMDegrain".


EDIT: sneakingly added recursion to MDegrain, test it out to find if it fills your needs. I might give it a revision for the next version, as Didée was doing something similar back then.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread

Last edited by Dogway; 3rd December 2021 at 17:18.
Dogway is offline   Reply With Quote
Old 3rd December 2021, 18:39   #672  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,510
Quote:
Originally Posted by Dogway View Post
So now DFTTest is 4, KNLMeanCL 5 and BM3D 6.
I'd prefer to have an added number than to change years of habits.
Quote:
Originally Posted by Dogway View Post
Do you mean recursion for MDegrain only (not MV)?
I meant a way to replicate 2 calls of SMDegrain consecutively but internally and optimized, as we were talking about the opportunity to have 2 calls with minor THSAD instead of 1 only with bigger one.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 6th December 2021, 12:02   #673  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,510
Quote:
Originally Posted by Dogway View Post
I will run tests to see how it improves things. It fits in the scope of current recursion solution by prepending an "r", so it would work as mode="rMDegrain".
I have looked at SMDegrain AVSI but I can't get how to make recursion work.

I can't see any parameter to add but only a check about the "r" letter on MDegrain and I can't understand how to make it work.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 6th December 2021, 13:32   #674  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,351
Yes, declare mode="rMDegrain" in the call and it will use recursion, but only for the MDegrain part not the vectors. As for to replicate 2 consecutive calls, I think it's best to do it explicitly as currently is done, I prefer simple things that can be seen and there wouldn't be speed gain at all (you only save a single TV->PC range).

Bu the way I'm reworking ex_gaussianblur() (used in LFR and ex_retinex()) as it was designed with a single resolution size, but it changes depending on resolution so I had to make a multi-regression fit. The good part is that it will fit even better than before. I will release ExTools in 2 days.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread
Dogway is offline   Reply With Quote
Old 6th December 2021, 14:46   #675  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,510
Quote:
Originally Posted by Dogway View Post
As for to replicate 2 consecutive calls, I think it's best to do it explicitly as currently is done
I hoped to save some video memory, as the dual line recursion eats it like peanuts and I have to increase the number of threads: AVS doesn't like 2xSMDegrain so much and sits idle for the most of machine time.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 6th December 2021, 15:21   #676  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,351
Well yes, that's an issue on mvtools, running manalyze several times has a great impact on performance, that's why among other things I don't like creating more MV in scripts like STPresso. DCFlicker uses it too, so it's a bit on the heavy size.

Back in the time I used to split the first pass of SMDegrain into a lossless file and then run the second call on this, this usually was faster than in a single script. MPPipeline also helped on this regard. Now for HD I don't see lossless being an option, maybe pseudo-lossless can cut it, a x264 profile at 12-bit with pseudo-lossless settings.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread
Dogway is offline   Reply With Quote
Old 6th December 2021, 17:20   #677  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,510
Quote:
Originally Posted by Dogway View Post
Now for HD I don't see lossless being an option, maybe pseudo-lossless can cut it, a x264 profile at 12-bit with pseudo-lossless settings.
x264 CRF 0 is of great use but unfortunately Nvidia can read 8 bit AVC 4:2:0 only, leaving me with the option to index it with some software only. HEVC is out of question, as it is too much CPU intensive.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 11th December 2021, 14:56   #678  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 2,351
I have been making a SAD sampler for the last 2 days without success, so I will leave this here in case someone sees an issue.

Code:
# ex_Luma_Rebuild(s0=6)
# SAD_sampler(show=2)


## Dependencies:
##      ExTools
##      FredAverage
##      ResizersPack
##      Grade          (for show)
##      MasksPack      (for show)
##      TransformsPack (for show)

# show: 0: no debug, 1: show sampled area 2: show sampled area in diff mode
function SAD_sampler(clip src, int "show") {

    show = Default(show, 0)

    mmod(src,128,mode="crop")
    b=ConvertBits(8, dither=-1).ExtractY()

    w = width()
    h = height()
    davg=0 grey=0
    id = 0 fr = 0
#    x  = 1 y  = 1
    OK = false

    while (!OK) {

        for (x = 1, w/128, 1) {

            stridex = 128*x

            for (y = 1, h/128, 1) {

                stridey = 128*y

                c       = b.Crop(stridex-128,stridey-128,-w+stridex,-h+stridey)
                a       = c.FredAverage()
                d       = c.TemporalSoften(10,255,0,30,2)
              # e       = c.ex_edge("frei-chen",0,255)
                diff    = ex_makediff(a, d, aug=5, metric="none", UV=128)
              # diff    = ex_logic(diff, e, "max", UV=128)

                ScriptClip(diff, function[a,d,x,y,c,show] () {

                    davg = (AverageLuma(last,-5)+AverageLuma(last)+AverageLuma(last,+5))/3.
                    grey = (AverageLuma(a,   -5)+AverageLuma(a)   +AverageLuma(a,   +5))/3.
                    OK   = (davg < 40.) && (90. < grey < 126.)
                    d    = show > 1 ? last : show > 0 ? c : c
                    OK ? propSet(d,"_x_id",-1,0).propSet(  "_y_id",-1,0).propSet("_diff",davg,0).propSet("_grey",grey,0) : \
                         propSet(d,"_x_id", x,0).propSet(  "_y_id", y,0).propSet("_diff",davg,0).propSet("_grey",grey,0)
                    } )

                yy = propGetInt("_y_id")
                xx = propGetInt("_x_id")
                EOL = (w+h)/128.
                t1  = id/EOL == 1.0
                t2  = id/EOL == 2.0
                t3  = id/EOL == 3.0
                t4  = id/EOL == 4.0
                fr  = t4 ? fr+24*3 : fr
                b   = t1 ? b.PadResize(w,h,biasw=64,biash=0)   : \
                      t2 ? b.PadResize(w,h,biasw=0, biash=64)  : \
                      t3 ? b.PadResize(w,h,biasw=-64,biash=0)  : \
                      t4 ? b.PadResize(w,h,biasw=0,biash=-64).selectevery(1,fr) : b

                if (show > 0 && yy<0) {
                    grey = round(propGetFloat("_grey"))
                    davg = round(propGetFloat("_diff"))
                    src  = b.subtitle(Format("id:{id} fr:{fr}\nx:{x} y:{y}\ng:{grey} d:{davg}"),lsp=1)
                    bg   = BoxMask(src,x*128-128,x*128,y*128-128,y*128,invert=false,matchformat=true)
                    ex_blend(src,bg,"multiply",opacity=0.3)
                }
                x  = yy<0 ? w/128 : t1 || t2 || t3 || t4 ? 1 : xx
                y  = yy<0 ? h/128 : t1 || t2 || t3 || t4 ? 1 : yy
                id = yy == h/128 ? id : id + 1
                id = t4 ? 0 : id
                OK = yy<0

            }
        }
    OK ? last : Undefined()
}
I also tested adding the variables into an array to save on propSet/Get calls, but array items can't be dynamic it seems.
I will continue with last fixes for ExTools, SharpenersPack and SMDegrain.
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread

Last edited by Dogway; 11th December 2021 at 15:04.
Dogway is offline   Reply With Quote
Old 14th December 2021, 00:05   #679  |  Link
anton_foy
Registered User
 
Join Date: Dec 2005
Location: Sweden
Posts: 702
Quote:
Originally Posted by Dogway View Post
Strange, what version of avs+ are you on? propNumElements() works for me outside runtime environments.
Same error here with LSFmod Im on Avs+ version r3577 (testbulid 34), error at line 541 in LSFmod-script.
EDIT: also smdegrain at line 61.

Last edited by anton_foy; 14th December 2021 at 00:17.
anton_foy is offline   Reply With Quote
Old 14th December 2021, 00:44   #680  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,510
Trying ResizersPack-7.1~Dogway.avsi

Script error: Invalid arguments to function 'width'.
(D:/Programmi/Media/AviSynth+/plugins64/ResizersPack-7.1~Dogway.avsi, line 171)
(D:/Programmi/Media/AviSynth+/plugins64/ResizersPack-7.1~Dogway.avsi, line 150)
(D:/Programmi/Media/AviSynth+/plugins64/TransformsPack-1.0rc29~Dogway.avsi, line 136)
(D:/Programmi/Media/AviSynth+/plugins64/TransformsPack-1.0rc29~Dogway.avsi, line 145)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 914)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 917)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 228)
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Reply

Tags
avisynth, dogway, filters, hbd, packs

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:21.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.