mClean spatio/temporal denoiser v3.2 (01 March 2018) - Page 17

Taurus · 6th February 2018, 13:13

Quote:

Originally Posted by burfadel

I'll make a list of required depenedencies and versions with the next release.

Thank You

!

ryrynz · 8th February 2018, 01:14

Anyone done any comparisons vs FluxSmooth?

Khun_Doug · 16th February 2018, 18:34

I noticed something peculiar with v2.3. I have not done regression testing so the problem may exist in older versions. I am attempting to clean a B&W HD rip. The source is cropped 220 on the left and 220 on the right. I believe the HD source is VC1. I am using an I7-6850K with 12 threads. I use a prefetch of 8, and later I tried 10. What I notice is the speed starts at 22 fps. Inevitably at around the 6 minute point into the film the speed slows to around 15 fps and CPU also drops significantly. The script to pre-render the lossless video is now running 5 hours and the speed is 5.6 fps. This is only reading the source, cropping, and using mClean. I also noticed that ffmpeg went from 2.5 GB up to 3.3 GB of memory.

Does this sound like some type of memory leak? Why would it only seem to occur on this particular video? I have used mClean on several other sources, some of those also B&W, and I have not seen this happen. Any thoughts? Perhaps something with AviSynth+?

burfadel · 18th February 2018, 08:38

Are all the filters used the latest? I'll release a major update to the script shortly. It's kind of ready now apart from one thing I want in the release.

Khun_Doug · 21st February 2018, 19:24

As far as I know, the filters are the latest versions. I began to think the problem was the source video. The ripping software identified the soundtrack incorrectly and nothing could detect any playable audio in the rip. My thought is the odd audio frames confused the readers, causing them to slow down as they got further into the source. I did a second rip of the BD and forced the audio to AC3, and the problem went away.

However, I did notice something else. I have seen this happen on two different rips so I know the problem is not a unique thing. On some of the older sources the borders are not completely clean. I've see this on Summer Lovers and Forbidden Planet, so far. The result is at the edge where the source isn't clean the border seems to get amplified, or widened. The rest of the image is clean, just not at the edge. And it can be very noticeable depending on the scenes. Is there a setting that will have the script to be less aggressive on the borders? If not, is there that such a thing could be added? For instance, a parameter to be less aggressive on the right, bottom, etc. I can get a screen shot if needed.

burfadel · 21st February 2018, 19:42

It might need a bit more padding. Still working on the next version, there are considerable changes.

Khun_Doug · 21st February 2018, 21:57

If you would like some testing to see how the new version handles this, I am willing to do some testing. These are examples of what I see.

burfadel · 24th February 2018, 10:43

Just posted a new version in the first post with considerable changes. There's still some minor tweaking and another feature I want to add to it, but that's to come later.

burfadel · 24th February 2018, 10:46

Quote:

Originally Posted by burfadel

I'll make a list of required depenedencies and versions with the next release. No plans for vapoursynth conversion, as not sure the actual syntax. Also aren't the plugins a little different now, such that the output wouldn't be the same?

Forgot about this post

. I don't have time currently to do this, I'll post it within the next few days.

If you have the very latest RGTools, Avisynth+ build, modplus, MVTools, Masktools, f3kdb, and chikuzen DCTFilter update, you should be fine

Please note I just updated the first post with the correct v3.0 version, not the earlier test version as previously.

zub35 · 24th February 2018, 22:18

On 3.0 again compatibility issues. Only works with "Intel SDE"
Maybe problem in masktools. First MT-mode, second ST-mode.
http://images2.imagebam.com/f2/65/15...b760586423.png

burfadel · 25th February 2018, 03:21

Are you using the latest MaskTools and MVTools by Pinterf?

zub35 · 25th February 2018, 03:30

Yeah. I tried not only the latter, alas, everywhere this problem.
Try to check yourself through the flag -p4p (sde.exe -p4p -- AVSMeter64.exe test.avs)
This will be emulation Pentium4-Prescott and Core2Quad-Kentsfield even on new CPUs

burfadel · 25th February 2018, 09:32

Quote:

Originally Posted by zub35

Yeah. I tried not only the latter, alas, everywhere this problem.
Try to check yourself through the flag -p4p (sde.exe -p4p -- AVSMeter64.exe test.avs)
This will be emulation Pentium4-Prescott and Core2Quad-Kentsfield even on new CPUs

If it's a problem with one of the tools, it would be best to post it in the appropriate thread. Most of the tools were updated by Pinterf, who did a great job

, but it may have caused problems with older processors with certain functions.

For the relevant versions:
Masktools: https://forum.doom9.org/showthread.php?t=174333
MVTools: https://forum.doom9.org/showthread.php?t=173356
RGTools: http://forum.doom9.net/showpost.php?...&postcount=239 (although it's not a specific thread for it)
Avisynth+: https://forum.doom9.org/showthread.p...68856&page=199

I've updated the first post with v3.1 that has some tweaking. The changes won't resolve the issue you mentioned.

burfadel · 25th February 2018, 09:45

Recommended versions (or later):

Avisynth+ r2636: https://github.com/pinterf/AviSynthPlus/releases
RGTools 0.96: https://github.com/pinterf/RgTools/releases
Masktools2 2.2.14: https://github.com/pinterf/masktools/releases/
MVTools2 2.7.25: https://github.com/pinterf/mvtools/releases/
Modplus: http://www.avisynth.nl/users/vcmohan...s/modPlus.html (link down the bottom)
f3kdb 2.0pre2: https://forum.doom9.org/showthread.php?t=161411 (link at top of code window for 2.0pre2, http://nmm.me/tr)
AutoAdjust v2.60 : https://forum.doom9.org/showthread.php?t=167573

pinterf · 25th February 2018, 10:39

Masktools, rgtools, mvtools2 and avs+ should work with sse2 and up. Earlier there were a couple of issues with sse2 only computers with specific filter parameters, but they were fixed (I was aware of).

zub35 · 25th February 2018, 13:28

Quote:

Originally Posted by burfadel

I've updated the first post with v3.1 that has some tweaking. The changes won't resolve the issue you mentioned.

This will sound very strange, but version 3.1 has worked without compatibility issues. While the 3.0 version gives a problem.

pinterf, Let's try to figure it out. I will send you in private messages script version 3.0 and we will try to find out why this happens, so that in the future there was no such.

burfadel · 25th February 2018, 14:01

That's weird, I can't think of a valid reason for that. The changes were with MAnalyse thSAD, and using reduceflicker function of clense only if it proved beneficial.

pinterf · 25th February 2018, 14:12

There was a non-guarded SSE4.1 instruction in the convolution code in masktools2. Interactive test is on the way.

zub35 · 25th February 2018, 16:06

burfadel, if deband=0 BitsPerComponent error

phazer11 · 26th February 2018, 03:41

Love the filter burfadel, really the only thing saving one of my encodes atm. Perhaps you could help me eliminate a filter?

Currently I'm having to use the following script (after a solid few weeks of working on it) to get good results for this aggravating source. I had hoped mClean by itself might be able to do everything I needed but the defaults aren't quite good enough and I was having trouble sorting through for the syntax so had to combine it with my previous best script to have it eliminate the final resistance of noise.

Code:

SMDegrain(tr = 12, thSAD = 800, thSADC = 400, contrasharp = false, refinemotion = false, interlaced = false, pel = 1, subpixel = 3, prefilter = 0, blksize = 16, search = 5, Truemotion = false, thSCD1 = 2400, lsb = false)
mClean()

Also would anyone of you fine people be able to help burfadel port his beautiful filter over to Vapoursynth?

21st February 2018, 21:57	#327 \| Link
Khun_Doug Registered User Join Date: Jul 2016 Posts: 100	If you would like some testing to see how the new version handles this, I am willing to do some testing. These are examples of what I see. Attached Images

24th February 2018, 22:18	#330 \| Link
zub35 Registered User Join Date: Oct 2016 Posts: 56	On 3.0 again compatibility issues. Only works with "Intel SDE" Maybe problem in masktools. First MT-mode, second ST-mode. http://images2.imagebam.com/f2/65/15...b760586423.png Last edited by zub35; 24th February 2018 at 22:43.

25th February 2018, 03:30	#332 \| Link
zub35 Registered User Join Date: Oct 2016 Posts: 56	Yeah. I tried not only the latter, alas, everywhere this problem. Try to check yourself through the flag -p4p (sde.exe -p4p -- AVSMeter64.exe test.avs) This will be emulation Pentium4-Prescott and Core2Quad-Kentsfield even on new CPUs Last edited by zub35; 25th February 2018 at 03:48.

25th February 2018, 09:45	#334 \| Link
burfadel Registered User Join Date: Aug 2006 Posts: 2,229	Recommended versions (or later): Avisynth+ r2636: https://github.com/pinterf/AviSynthPlus/releases RGTools 0.96: https://github.com/pinterf/RgTools/releases Masktools2 2.2.14: https://github.com/pinterf/masktools/releases/ MVTools2 2.7.25: https://github.com/pinterf/mvtools/releases/ Modplus: http://www.avisynth.nl/users/vcmohan...s/modPlus.html (link down the bottom) f3kdb 2.0pre2: https://forum.doom9.org/showthread.php?t=161411 (link at top of code window for 2.0pre2, http://nmm.me/tr) AutoAdjust v2.60 : https://forum.doom9.org/showthread.php?t=167573 Last edited by burfadel; 3rd March 2018 at 02:01. Reason: Updated Avisynth version

25th February 2018, 10:39	#335 \| Link
pinterf Registered User Join Date: Jan 2014 Posts: 2,314	Masktools, rgtools, mvtools2 and avs+ should work with sse2 and up. Earlier there were a couple of issues with sse2 only computers with specific filter parameters, but they were fixed (I was aware of). __________________ AviSynth+ on github, Other repos: RgTools, Masktools2, MvTools2, TIVTC, Average

8th February 2018, 01:14	#322 \| Link
ryrynz Registered User Join Date: Mar 2009 Posts: 3,650	Anyone done any comparisons vs FluxSmooth?

16th February 2018, 18:34	#323 \| Link
Khun_Doug Registered User Join Date: Jul 2016 Posts: 100	I noticed something peculiar with v2.3. I have not done regression testing so the problem may exist in older versions. I am attempting to clean a B&W HD rip. The source is cropped 220 on the left and 220 on the right. I believe the HD source is VC1. I am using an I7-6850K with 12 threads. I use a prefetch of 8, and later I tried 10. What I notice is the speed starts at 22 fps. Inevitably at around the 6 minute point into the film the speed slows to around 15 fps and CPU also drops significantly. The script to pre-render the lossless video is now running 5 hours and the speed is 5.6 fps. This is only reading the source, cropping, and using mClean. I also noticed that ffmpeg went from 2.5 GB up to 3.3 GB of memory. Does this sound like some type of memory leak? Why would it only seem to occur on this particular video? I have used mClean on several other sources, some of those also B&W, and I have not seen this happen. Any thoughts? Perhaps something with AviSynth+?

18th February 2018, 08:38	#324 \| Link
burfadel Registered User Join Date: Aug 2006 Posts: 2,229	Are all the filters used the latest? I'll release a major update to the script shortly. It's kind of ready now apart from one thing I want in the release.

21st February 2018, 19:24	#325 \| Link
Khun_Doug Registered User Join Date: Jul 2016 Posts: 100	As far as I know, the filters are the latest versions. I began to think the problem was the source video. The ripping software identified the soundtrack incorrectly and nothing could detect any playable audio in the rip. My thought is the odd audio frames confused the readers, causing them to slow down as they got further into the source. I did a second rip of the BD and forced the audio to AC3, and the problem went away. However, I did notice something else. I have seen this happen on two different rips so I know the problem is not a unique thing. On some of the older sources the borders are not completely clean. I've see this on Summer Lovers and Forbidden Planet, so far. The result is at the edge where the source isn't clean the border seems to get amplified, or widened. The rest of the image is clean, just not at the edge. And it can be very noticeable depending on the scenes. Is there a setting that will have the script to be less aggressive on the borders? If not, is there that such a thing could be added? For instance, a parameter to be less aggressive on the right, bottom, etc. I can get a screen shot if needed.

21st February 2018, 19:42	#326 \| Link
burfadel Registered User Join Date: Aug 2006 Posts: 2,229	It might need a bit more padding. Still working on the next version, there are considerable changes.

24th February 2018, 10:43	#328 \| Link
burfadel Registered User Join Date: Aug 2006 Posts: 2,229	Just posted a new version in the first post with considerable changes. There's still some minor tweaking and another feature I want to add to it, but that's to come later.

25th February 2018, 03:21	#331 \| Link
burfadel Registered User Join Date: Aug 2006 Posts: 2,229	Are you using the latest MaskTools and MVTools by Pinterf?

25th February 2018, 14:01	#337 \| Link
burfadel Registered User Join Date: Aug 2006 Posts: 2,229	That's weird, I can't think of a valid reason for that. The changes were with MAnalyse thSAD, and using reduceflicker function of clense only if it proved beneficial.

25th February 2018, 14:12	#338 \| Link
pinterf Registered User Join Date: Jan 2014 Posts: 2,314	There was a non-guarded SSE4.1 instruction in the convolution code in masktools2. Interactive test is on the way. __________________ AviSynth+ on github, Other repos: RgTools, Masktools2, MvTools2, TIVTC, Average

25th February 2018, 16:06	#339 \| Link
zub35 Registered User Join Date: Oct 2016 Posts: 56	burfadel, if deband=0 BitsPerComponent error

26th February 2018, 03:41	#340 \| Link
phazer11 Registered User Join Date: Feb 2018 Posts: 11	Love the filter burfadel, really the only thing saving one of my encodes atm. Perhaps you could help me eliminate a filter? Currently I'm having to use the following script (after a solid few weeks of working on it) to get good results for this aggravating source. I had hoped mClean by itself might be able to do everything I needed but the defaults aren't quite good enough and I was having trouble sorting through for the syntax so had to combine it with my previous best script to have it eliminate the final resistance of noise. Code: SMDegrain(tr = 12, thSAD = 800, thSADC = 400, contrasharp = false, refinemotion = false, interlaced = false, pel = 1, subpixel = 3, prefilter = 0, blksize = 16, search = 5, Truemotion = false, thSCD1 = 2400, lsb = false) mClean() Also would anyone of you fine people be able to help burfadel port his beautiful filter over to Vapoursynth? Last edited by phazer11; 26th February 2018 at 03:58.