Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 11th June 2021, 08:08   #41  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,914
Quote:
Originally Posted by Dogway View Post
With the new mvtools2 version DCT is now much faster, so I defaulted DCT to 5, if everyone agrees.
"dct = 5: SATD instead of SAD for luma."
So for this mode you noticed a speed increase? There was a fix indeed, but I couldn't tell the magnitude of this change, did not perform a before-after test.
pinterf is offline   Reply With Quote
Old 11th June 2021, 16:41   #42  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 1,745
Quote:
Originally Posted by Dogway View Post
only 8-bit is used for motion vectors, MDegrain uses a MSuper of original bitdepth, and everything else is processed in original bitdepth.
Even if I feed a HBD source? So some dithering/truncating occurs?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 11th June 2021, 18:36   #43  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 1,164
SMDegrain 3.2.0d final is released. I had changed DCT to 5 by reading the changelog only, now I run some benchs and indeed, no speed gain so reverted. Also optimized ex_luma_rebuild() and some overall cleanup. In the future I want to add some frequency restoration system, this is something that I also looked into 6 years ago.
Code:
OLD DCT = 0   7.600fps
OLD DCT = 5   6.300fps
NEW DCT = 5   6.300fps
NEW DCT = 0   7.700fps
@Tormento, yes, the performance/gain ratio is very bad for HBD motion search. ex_luma_rebuild() makes up for an extra 16% precision by converting to PC levels which is very fast with the latest update. You can run tests to see the difference, check below example.

@pinterf, I was confused about bitdepth handling with motion vectors, is the next valid code or do I need MScaleVect()?

Code:
super_search = MSuper(rfilter=2)

bv3 = super_search.MAnalyse(isb = true,  delta = 3, overlap= 4)
bv2 = super_search.MAnalyse(isb = true,  delta = 2, overlap= 4)
bv1 = super_search.MAnalyse(isb = true,  delta = 1, overlap= 4)
fv1 = super_search.MAnalyse(isb = false, delta = 1, overlap= 4)
fv2 = super_search.MAnalyse(isb = false, delta = 2, overlap= 4)
fv3 = super_search.MAnalyse(isb = false, delta = 3, overlap= 4)

convertbits(16)
MDegrain3(MSuper(levels=1), bv1, fv1, bv2, fv2 ,bv3, fv3, thSAD=300, thSADC=150)
Dogway is offline   Reply With Quote
Old 11th June 2021, 19:43   #44  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,914
Quote:
Originally Posted by Dogway View Post
SMDegrain 3.2.0d final is released. I had changed DCT to 5 by reading the changelog only, now I run some benchs and indeed, no speed gain so reverted.
dct>0 can be quicker indeed, except for 8x8 block sizes at 8 bit videos. Other than that special format (which is computed by integer dct code) the others are using fftw3, which remained unchanged but the epilog code got a fix in the SSE2 code.
pinterf is offline   Reply With Quote
Old 11th June 2021, 19:55   #45  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,914
Quote:
Originally Posted by Dogway View Post
SMDegrain 3.2.0d final is released. I had changed DCT to 5 by reading the changelog only, now I run some benchs and indeed, no speed gain so reverted. Also optimized ex_luma_rebuild() and some overall cleanup. In the future I want to add some frequency restoration system, this is something that I also looked into 6 years ago.
Code:
OLD DCT = 0   7.600fps
OLD DCT = 5   6.300fps
NEW DCT = 5   6.300fps
NEW DCT = 0   7.700fps
@Tormento, yes, the performance/gain ratio is very bad for HBD motion search. ex_luma_rebuild() makes up for an extra 16% precision by converting to PC levels which is very fast with the latest update. You can run tests to see the difference, check below example.

@pinterf, I was confused about bitdepth handling with motion vectors, is the next valid code or do I need MScaleVect()?

Code:
super_search = MSuper(rfilter=2)

bv3 = super_search.MAnalyse(isb = true,  delta = 3, overlap= 4)
bv2 = super_search.MAnalyse(isb = true,  delta = 2, overlap= 4)
bv1 = super_search.MAnalyse(isb = true,  delta = 1, overlap= 4)
fv1 = super_search.MAnalyse(isb = false, delta = 1, overlap= 4)
fv2 = super_search.MAnalyse(isb = false, delta = 2, overlap= 4)
fv3 = super_search.MAnalyse(isb = false, delta = 3, overlap= 4)

convertbits(16)
MDegrain3(MSuper(levels=1), bv1, fv1, bv2, fv2 ,bv3, fv3, thSAD=300, thSADC=150)
The code is valid.
The super clip provided to MDegrain must be originated from the same format as the video to denoise.
It can even be a 4:4:4 32 bit float as well, while the vectors can be computed from a plain 8 bit YV12.

I still do not understand what is luma_rebuild for.
"makes up for an extra 16% precision". If this extra range would help with anything then using a 16 bit clip for motion vector search would yield an extra 25600% precision increase. In real life: when you compare the vectors from a 8 bit clip or its ConvertBits(16) counterpart, there will be basically no difference.
What I can imagine that you stretch the Y range and thus it will have a relatively larger weight when chroma is involved.
Btw. you can adjust luma/chroma weights in SAD search with MAnalyze "scaleCSAD" parameter which fine tunes chroma SAD weight relative to luma SAD.
pinterf is offline   Reply With Quote
Old 11th June 2021, 20:53   #46  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 1,164
Bitdepth has more precision, but is not feature bound. Take for example the "opposite" of a TV->PC conversion, a source in log space (or a very low contrast processed source), the SAD will be misleading even in float precision because it doesn't represent human vision.

luma_rebuild() also has a brightening option for the dark areas for the same reason. Human vision is very adaptive to dark (more so in dim/dark surround) so it makes sense to brighten this up to give it more SAD weight and avoid smearing. If this wasn't true we would just try to find motion vectors in linear light and not gamma encoded sources.

I'm using scaleCSAD (MAnalyze), but only for chroma subsampling as written in the docs. Does it have an effect on thSADC of MDegrain? Meaning are they mutually exclusive?
Dogway is offline   Reply With Quote
Old 11th June 2021, 20:59   #47  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,914
I see. So it is for lifting the shadows. thsadc can be used independendly from scaleCSAD
pinterf is offline   Reply With Quote
Old 11th June 2021, 21:52   #48  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 1,745
Quote:
Originally Posted by Dogway View Post
yes, the performance/gain ratio is very bad for HBD motion search
I am a bit confused now about the differences between the parameters in your SMDegrain and real.finder's one.

I am looking at your html file, that perhaps you have not updated yet looking at the changelog, and I can see the lsb_in and lsb_out parameters. Weren't they the parameters used when AVS did not support HBD?

Let's have some example:
  1. I have a 8 bit input, I want to internally work in 8 bit and output 8 bit
  2. I have a 8 bit input, I want to internally work in 16/32 bit and output 8/10/12/16 bit
  3. I have a 10/12/16 bit input, I want to internally work in 10/16/32 bit and output 8/10/12/16 bit
Could you please write the correct parameters to be used for each case?

Thanks in advance.
__________________
@turment on Telegram

Last edited by tormento; 11th June 2021 at 21:55.
tormento is offline   Reply With Quote
Old 11th June 2021, 22:13   #49  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 1,164
The documentation hasn't been updated, not even real.finder updated it so it is very old by current terms. I will update when I finish Transforms Pack which will be very soon.

My updated SMDegrain tries to be as simple as possible so I removed redundant arguments like HBD in/out and things like that.
For example 2 precede it with ConvertBits(bits), others need nothing.
Dogway is offline   Reply With Quote
Old 12th June 2021, 08:27   #50  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 5,085
Quote:
Originally Posted by Dogway View Post
luma_rebuild() also has a brightening option for the dark areas for the same reason. Human vision is very adaptive to dark (more so in dim/dark surround) so it makes sense to brighten this up to give it more SAD weight and avoid smearing. If this wasn't true we would just try to find motion vectors in linear light and not gamma encoded sources.
I've always wondered a bit about luma_rebuild. Using s0=1.0 makes the image generally much darker compared to the original clip than cretindesalpes's original version which has s0=2.0. I thought it would make the clip have less dark areas.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 12th June 2021, 10:04   #51  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 1,164
I don't remember what was the default before my recent edits but I usually try to leave defaults in "safe" settings otherwise people may complain too much, but if there's a consensus I can set it to 2.0, it's still safe (I've gone to 6.0 on a recent project). s0=1.0 by default just converts to PC levels so it appears darker when seen on TV levels preview (ie. AvsPmod)
Dogway is offline   Reply With Quote
Old 16th June 2021, 23:51   #52  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 1,745
Quote:
Originally Posted by Dogway View Post
ExTools
I just updated to 1.7 version and I get:

Script error: Invalid arguments to function 'ex_Yexpr'.
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.2.0d~Dogway.avsi, line 618)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.2.0d~Dogway.avsi, line 161)
(F:\In\1_30 Febbre da cavallo\febbre_4.avs, line 20)
(F:\In\1_30 Febbre da cavallo\febbre_4_temp\febbre_4_source.avs, line 1)


My script is simple:

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_30 Febbre da cavallo\febbre.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)
Prefetch(6)


1.6 is working fine, SMDegrain is the last one.
__________________
@turment on Telegram

Last edited by tormento; 17th June 2021 at 00:04.
tormento is offline   Reply With Quote
Old 17th June 2021, 00:51   #53  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 1,164
Yes, ExTools is a bit developed ahead as I'm making time to solve this issue and rebase the other filters.

For the moment use v1.6. I think I will doing other things until the issue which is critical is solved.
Dogway is offline   Reply With Quote
Old 17th June 2021, 09:15   #54  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 1,745
Quote:
Originally Posted by Dogway View Post
For the moment use v1.6. I think I will doing other things until the issue which is critical is solved.
Do you plan to abandon dependencies on Zs_RF_Shared?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 17th June 2021, 09:36   #55  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 456
Quote:
Originally Posted by tormento View Post
Do you plan to abandon dependencies on Zs_RF_Shared?
SMDegrain of Dogway don't dependencies on Zs_RF_Shared
kedautinh12 is offline   Reply With Quote
Old 17th June 2021, 09:37   #56  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 1,745
Quote:
Originally Posted by kedautinh12 View Post
SMDegrain of Dogway don't dependencies on Zs_RF_Shared
Strange. If I recall well, I tried on one of the first beta and it needed.

I will try again with latest one.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 17th June 2021, 17:42   #57  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,225
it's still needed for https://github.com/Dogway/Avisynth-S....2.0d.avsi#L39

edit: and also https://github.com/Dogway/Avisynth-S...2.0d.avsi#L145

also there are bug that case error in this line https://github.com/Dogway/Avisynth-S...2.0d.avsi#L170 with subpixel=3 that fixed in https://github.com/Dogway/Avisynth-Scripts/pull/11
__________________
See My Avisynth Stuff

Last edited by real.finder; 18th June 2021 at 01:50.
real.finder is offline   Reply With Quote
Old 17th June 2021, 22:38   #58  |  Link
Dogway
Registered User
 
Join Date: Nov 2009
Posts: 1,164
Quote:
Originally Posted by real.finder View Post
also there are bug that case error in this line https://github.com/Dogway/Avisynth-S...2.0d.avsi#L170 with subpixel=3 that fixed in https://github.com/Dogway/Avisynth-Scripts/pull/11
You mean ex_luma_rebuild()? Expr() has issues dealing with clip luma range as explained in this post. So development is pretty much halted until solved.
Dogway is offline   Reply With Quote
Old 18th June 2021, 01:50   #59  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,225
Quote:
Originally Posted by Dogway View Post
You mean ex_luma_rebuild()?
it's another one with subpixel=3
__________________
See My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 18th June 2021, 16:59   #60  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 1,745
Tried pure speed comparison between SMDegrain and NotSMDegrain on an old movie. Latest versions of AviSynth+, AVSI and dll filters.

Script for SMDegrain

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_30 Febbre da cavallo\febbre.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)
Prefetch(6)


Script for NotSMDegrain

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_30 Febbre da cavallo\febbre.dgi",ct=24,cb=24,cl=0,cr=0)
NotSMDegrain (tr=6, thSAD=600, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)
Prefetch(6)


SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

AVSMeter 3.0.8.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.1 (r3413, master, x86_64) (3.7.1.0)

Number of frames: 2712
Length (hh:mm:ss.ms): 00:01:48.480
Frame width: 1920
Frame height: 1032
Framerate: 25.000 (25/1)
Colorspace: YUV420P16

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.918 | 959.8 | 5.764
Process memory usage (max): 2162 MiB
Thread count: 75
CPU usage (average): 72.4%

Time (elapsed): 00:07:50.498


NotSMDegrain (tr=6, thSAD=600, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.956 | 125000 | 6.124
Process memory usage (max): 1961 MiB
Thread count: 76
CPU usage (average): 73.0%

Time (elapsed): 00:07:22.854


SMDegrain (tr=7, thSAD=700, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.788 | 1055 | 4.498
Process memory usage (max): 2020 MiB
Thread count: 75
CPU usage (average): 73.2%


NotSMDegrain (tr=7, thSAD=700, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.477 | 243903 | 5.029
Process memory usage (max): 1783 MiB
Thread count: 74
CPU usage (average): 72.5%


SMDegrain (tr=9, thSAD=900, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.683 | 714286 | 3.412
Process memory usage (max): 2278 MiB
Thread count: 75
CPU usage (average): 73.1%


NotSMDegrain (tr=9, thSAD=900, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.835 | 625000 | 3.909
Process memory usage (max): 1964 MiB
Thread count: 75
CPU usage (average): 72.7%
__________________
@turment on Telegram

Last edited by tormento; 18th June 2021 at 17:03.
tormento is offline   Reply With Quote
Reply

Tags
avs+, dogway, filters, hbd, packs

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 08:50.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.