BM3D (VapourSynth) versus MVTools

[guest]

I just wanted to show what problems I get, when using BM3D CUDA....(Avisynth)

BM3D CPU is good :)

https://www.mediafire.com/file/ai7y4ild8fe6opn/bm3d_cuda_test.mkv/file

[kedautinh12]

Latest ver
https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/issues/7#issuecomment-886173346

[tormento]

I read that since test8 AVX support has been removed.

Does it mean that now it requires AVX2? That would be so bad for my i7-2600k...

[kedautinh12]

I read that since test8 AVX support has been removed.

Does it mean that now it requires AVX2? That would be so bad for my i7-2600k...

No it's required below AVX can use this plugin

[DTL]

I read that since test8 AVX support has been removed.

Does it mean that now it requires AVX2? That would be so bad for my i7-2600k...

May you finally can move to about 10-years old AVX2 capable chip. The era of AVX2 still going widely. The AVX was very short transition from SSE2 to AVX2.

AVX is float only 256bit that is still slowly in compare with 16 and may be 32 bit integer AVX2 for 8 and 16bit AVS formats.

It is too slow to process of 8bit samples as 32bit floats in AVX - it makes processing samples half of possible per op. Also the available to store in register file chunk of data is half sized. Also conversions from-to float and 8/16bit unsigned int are slow (and require special limiting-saturation as separate ops that is built-in into bit-truncation packing ops of integers).

There is even an auto-saturated integer addition for 16bit arguments exist already in AVX2. It is only about 1/3 slower at some chips families in compare with standard addition. Looks saturated addition can be dispatched on 2 ports and standard addition on up to 3 ports at once so super-scalarity is only a bit limited for saturated addition.

[tormento]

May you finally can move to about 10-years old AVX2 capable chip.
Whenever I will get the money to do it.

Can I send you my bank account number? :p

Funny enough, now that I am saving to buy a new Intel, they disabled AVX512 and AMD enabled them.

It's a strange world.

[DTL]

" AMD enabled them."

AMD with AVX512 (F + BMI at least) and large cache and large core number should be a winner. Now it is unbalanced world - intel have AVX512 2048 bytes register file + nice instructions set but low core number and low cache size.

AMD have large core number and large cache but only AVX2 old 512 bytes sized register file and a decade+ old instructions set only. No nice data shift in variable byte granularity and inbetween 'registers' with single instruction to support typical moving pictures data processing operations like scaling/shifting/searching. Old hack with using palignr instruction for shift-rotate with variable bytes granularity is only work for 128bit lanes and require pairing/cascading with permutes/blends to support 256bit or data shift inbetween 256bit data chunks. It is not as nice as newer AVX512 byte moving instructions.

The era of 10+ years old AVX2 is ending and it is again new waiting time who make new winner CPU chip of both 2048bytes register file + many cores + large cache + nice new instructions for multimedia.

[kedautinh12]

" AMD enabled them."

AMD with AVX512 (F + BMI at least) and large cache and large core number should be a winner. Now it is unbalanced world - intel have AVX512 2048 bytes register file + nice instructions set but low core number and low cache size.

AMD have large core number and large cache but only AVX2 old 512 bytes sized register file and a decade+ old instructions set only. No nice data shift in variable byte granularity and inbetween 'registers' with single instruction to support typical moving pictures data processing operations like scaling/shifting/searching. Old hack with using palignr instruction for shift-rotate with variable bytes granularity is only work for 128bit lanes and require pairing/cascading with permutes/blends to support 256bit or data shift inbetween 256bit data chunks. It is not as nice as newer AVX512 byte moving instructions.

The era of 10+ years old AVX2 is ending and it is again new waiting time who make new winner CPU chip of both 2048bytes register file + many cores + large cache + nice new instructions for multimedia.

Intel 12th had more core

[DTL]

Intel 12 is strange hybrid to be partially good at new and at old applications so have only part of cores with AVX512 SIMD coprocessor and part of cores with AVX2 only. So its nightmare for threads planner - it MUST put AVX512 applications only at P-cores with AVX512 co-SIMD and never put to E-cores (or got invalid instruction crash when attempt to execute AVX512 instruction). Its magic require Win11 threads scheduler and may be special build of application. The only sad workaround for old windows is to disable degraded E-cores at all in BIOS.

Last full AVX2/AVX512 compatible desktop intel is 11.

Intel 12 may be either specially designed low-cost chip with partial AVX512 co-SIMD support on a part of cores or may be sorted out Xeon chips with defected and disabled at factory AVX512 coprocessors at one or more cores. Because full cores of AVX512 require too much transistor and power budget for general purpose desktop CPU. Xeons typically $$$$ to $$$$$ priced with all cores AVX512 support.

So intel 12 still some transient from AVX2 era to AVX512 in some possible future of desktop CPUs. AVX512 is still low supported by desktop apps and intel need to show good benchmarks at the old more or less multithreaded desktop applications with no any use of SIMD at all. Current intel production simply can not sell for $$$ the massive multicores CPUs with all cores AVX512 for a small number of AVS+ plugins and some cryptomining. The residual desktop market of CPUs require hard balance between cores number and cores complexity and cache size. AVX512 is very expensive part of core and too few used by desktop software. May be x264/265 will be redesigned to use it too someday. Typical advanced workstation for movie pictures data processing is Xeon-based.

[tormento]

Intel 12th had more core
Intel is laser cutting all new desktop processors not to have AVX512.

To have them back, you need to jump to Xeon and its HPC derivatives.

[guest]

Latest ver
https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/issues/7#issuecomment-886173346

I wonder what the difference is between "test#9", and this :-

https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/actions/runs/2675015268

[kedautinh12]

I wonder what the difference is between "test#9", and this :-

https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/actions/runs/2675015268

Same cause it's from aabebd9 commit

[DTL]

About 1 year after 4th July 2021, 18:10 thread starting:

It looks new tests for season of mid-2022 ready - https://forum.doom9.org/showthread.php?p=1972003#post1972003

BM3D (CUDA ?) vs mvtools2 (DX12-ME accelerated MAnalyse + interpolated overlap MDegrainN +MVLPF +motion-adaptive thSAD) - speed/quality.

[kedautinh12]

New ver:
https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/issues/7#issuecomment-886173346

[SaurusX]

Any idea what may be causing this error?
http://i.ibb.co/0JP3Hwy/AVS-error.jpg

I'm using MeGUI x64 build 2913.

[poisondeathray]

Any idea what may be causing this error?
http://i.ibb.co/0JP3Hwy/AVS-error.jpg

I'm using MeGUI x64 build 2913.


You probably need a newer version of avs+ ; bm3d requires features like arrays which are not supported in older avs+ versions

Megui 2913 was released (January 19, 2020) , the bundled avs+ version is probably too old

[kedautinh12]

You need update avs+ manually and use with MeGUI AvisynthWrapper from Pinterf:
https://forum.doom9.org/showthread.php?p=1913117#post1913117

Avs+ latest ver:
https://gitlab.com/uvz/AviSynthPlus-Builds

[Stereodude]

Sorry if this is a stupid question, but I just discovered BM3D and some of the parameters aren't so clear to me (like are there downsides other than memory / CPU usage).

Are these reasonable parameters for standard def 480p content (MPEG-2 DVD source 24p live action / not animation)?

Adjusting things by eye I have:
BM3D_CPU(sigma=5.0, block_step=3, radius=4)
BM3D_VAggregate(radius=4)

[Stereodude]

I'm running the last test release and it's a slideshow if chroma=true using CUDA. Chroma=true is like 1/10th the speed of Chroma = false. I get a few frames at time (seems to correlate to my prefetch setting) and long pauses in between. My GPU (3070 Ti) doesn't show that it's loaded heavily or out of memory. Has anyone seen something like this? Running on the CPU is very close in speed of chroma = true or false.

[Stereodude]

I'm running the last test release and it's a slideshow if chroma=true using CUDA. Chroma=true is like 1/10th the speed of Chroma = false. I get a few frames at time (seems to correlate to my prefetch setting) and long pauses in between. My GPU (3070 Ti) doesn't show that it's loaded heavily or out of memory. Has anyone seen something like this? Running on the CPU is very close in speed of chroma = true or false.
FWIW, I was trying to use this on 1080p content with a radius of 4 with chroma = true when it performed so slowly. It seems that with chroma = true the radius has a much larger effect on the speed when compared to chroma = false. A radius of 1 or 2 with chroma = true behaved much better.

[Emulgator]

I can confirm that if I stack algos that internally rely on BM3D (just 1 feisty2 algo which is full of BM3D anyway),
plus maybe SMDegrain(prefilter=8 (BM3D)), and sometimes a third instance of BM3D may join the party hidden somewhere),
I get a huge penalty depending on prefetch depth.
The individual depths of individual calls to BM3D overlaid by prefetch needs to shovel/calculate huge data pages back and forth, thus becoming a gridlock.
Trying to split some load into BM3D_CPU and BM3D_CUDA may help here, I tried that and it helped.
ANd if not: In harder cases I just have to split the load, tolerate an intermediate generation and run 2 passes.

[Stereodude]

BM3D feels like a cheat code for encoding. 1080p Blu-ray and 2160p HDR Blu-ray content that is already quite clean looking with very little to no visible apparent noise to remove will encode to about half the bitrate with the same exact same x265 settings after BM3D NR (vs. no BM3D NR). In some cases I can't even see a visible difference in terms of noise even while pixel peeping from running it through BM3D, but the resulting bitrate of the x265 encode is substantially lower. I've got CRF 18 1080p x265 encodes where the resulting video looks great to me with nothing objectionable in the encode that I can notice and the HEVC video track is smaller in size than the lossless audio for the movie. :eek:

[DTL]

We expect same methods to be implemented in the MPEG encoder engine so the pre-processing with 'external denoising' can be skipped. Some motion to this way may be in the AV1 coder.

[tormento]

We expect same methods to be implemented in the MPEG encoder engine so the pre-processing with 'external denoising' can be skipped. Some motion to this way may be in the AV1 coder.


When? :)

BTW BM3D is my favorite denoiser by far.

I hope it won’t become an abandonware as it has still some great potential such as running the temporal part in the GPU routine or having the capability to run multiple pass with lower settings, such as SMSegrain does, without leaving GPU memory space. I tried to ask Asd~G to maintain it but he is not interested. Perhaps Pinterf could maintain it as it does with KNLMeansCL.

[cubicibo]

What's a "MPEG encoder engine"? VTM? Unusable.

[Z2697]

Personally speaking, if a encoder has a denoising feature, I definitely don't want to use it (on anything I actually care about).

It means you have the denoising based on a process that's highly dependent on the encoding process, and there's virtually no way to verify / evaluate the effect without doing the encoding, and the added encoding distortion.
The result will vary based on the encoding settings, including quality / bitrate! (there's a bunch of settings that can cause the ME result to change, other than the ME settins)
Even the denoising itself will probably change the ME result.
The granularity of the ME also cannot be changed (otherwise where's the difference than denoise separately?), you only have the PU level MVs, or even worse, look-ahead rough search.
The goal of ME also not quite the same. Denoising VS Encoding, obviously. I know there must be a lot of overlap between these two, but still not the same.

It also means you equivalently have the denoising as the absolute final step of your pre-processing. (if you are planning to do denoising only, then great, it might actually not that big of a problem, but still, consider the previouslyt described problems)
Which may... just maybe fine sometimes, but not ideal.

If you just want a encoder feature that "beat 'em up and save bits", then that's probably it, but other than that, I'd say I'm very doubtful.
A good encoder should be good at one thing, and one thing only, that's encoding.

[DTL]

Typically most of MPEG encoders already working as denoising engines but with very poor quality. Because they can not fit all possible input noise-data into typically required output bitrate. Making input less noisy mean encoder can make better image without wasting bits to encode noise.

"Even the denoising itself will probably change the ME result."

Yes - you can see it in multi-generation method for mvtools. It also takes lots of computing resources because instead of single pass ME+block_averaging we need to make many sequential ME+block_averaging with lower settings in attempt to have less details missed.

The ME engine of BM3D may be also adapded to this method - only required to add the second input-clip to ME engine. Typically ME engine of denoisers (and MPEG encoders) work with single input clip and search ME of current frame vs other frame of the same input clip.

For multi-generation denoise the ME engine search current frame vs frame from different input clip.

1st generation denoise - ME engine work with single imput clip. Searches of current frame vs other frame on the input clip (max noised).
Nth generation denoise - ME engine work with N-1 generation denoised clip and original noisy input clip. This allows to search vs non-distorted input and miss less details.

"The goal of ME also not quite the same. Denoising VS Encoding,"

Denoising ME must be more perfect - error in ME for denoising cause details loss/degradation. Error in ME for MPEG encoding only cause bitrate increasing because of less motion-compensated data found. It also mean hardware ME engines that we have in GPUs are from MPEG encoders and not designed to be very great quality.

The ME engine for denoiser may be several orders of 10-magnitude more complex (in computing complexity) in comparison with 'already very slow' MPEG encoder ME. Because we have lots of methods to make MVs better by statistical analysis. But it require lots of ME engines running in parallel and also in many generations of MVs refining.

[Z2697]

Block matching is not motion estimation, I think.

[DTL]

BM marked as 3D works in both same frame (2D) and +1D temporal. It can run on single frame only in 2D mode (?). It come from static image denoise when single frame denoise also required. Same as KNLmeans method.

[Z2697]

BM3D is pure spatial denoiser. The third dimension is the "matched blocks", IIRC.
A "temporal" version is called V-BM3D. But I think it still isn't doing motion estimation.
The "temporal effectiveness" of V-BM3D is very limited, because there's no motion compensation, and the "temporal matching" part works differently than NLMeans.

[DTL]

BM3D is spatio-temporal with 'very big' possible temporal 'radius' up to 16 - http://avisynth.nl/index.php/BM3DCUDA

int radius = 0
The temporal radius for denoising, valid range [1, 16].
For each processed frame, (radius * 2 + 1) frames will be requested, and the filtering result will be returned to these frames by BM3D_VAggregate.
Increasing radius only increases tiny computational cost in block-matching and aggregation, and will not affect collaborative filtering, but the memory consumption can grow quadratically.
Thus, feel free to use large radius as long as your RAM is large enough :D

"because there's no motion compensation,"

I think block-matching directly mean finding and averaging only 'matching' blocks - it mean both search and 'compensation' in the same frame and in the several frames.

[Z2697]

BM3D is spatio-temporal with 'very big' possible temporal 'radius' up to 16 - http://avisynth.nl/index.php/BM3DCUDA

int radius = 0
The temporal radius for denoising, valid range [1, 16].
For each processed frame, (radius * 2 + 1) frames will be requested, and the filtering result will be returned to these frames by BM3D_VAggregate.
Increasing radius only increases tiny computational cost in block-matching and aggregation, and will not affect collaborative filtering, but the memory consumption can grow quadratically.
Thus, feel free to use large radius as long as your RAM is large enough :D

"because there's no motion compensation,"

I think block-matching directly mean finding and averaging only 'matching' blocks - it mean both search and 'compensation' in the same frame and in the several frames.

I know.
It's called VBM3D.
Not "3D meaning temporal".

[Stereodude]

BM3D is pure spatial denoiser. The third dimension is the "matched blocks", IIRC.
A "temporal" version is called V-BM3D. But I think it still isn't doing motion estimation.
The "temporal effectiveness" of V-BM3D is very limited, because there's no motion compensation, and the "temporal matching" part works differently than NLMeans.
What is the radius parameter doing in BM3D? The documentation says it looks across multiple frames of video (as long as the radius isn't set to 0).

I didn't test this aspect too much, I mostly picked radius settings that fit in the memory of my GPU and didn't slow encoding too much, but in my brief tests I didn't see much difference in the radius setting, though I have to be honest, I don't know what I should have looked for or in what sort of content it would make a difference.

[tormento]

What is the radius parameter doing in BM3D? The documentation says it looks across multiple frames of video (as long as the radius isn't set to 0).

I didn't test this aspect too much, I mostly picked radius settings that fit in the memory of my GPU and didn't slow encoding too much, but in my brief tests I didn't see much difference in the radius setting, though I have to be honest, I don't know what I should have looked for or in what sort of content it would make a difference.


The larger radius you give, the more precise the denoising is, mostly for high denoise values. There is a precise formula to calculate memory consumption on the GitHub repo.

[Stereodude]

The larger radius you give, the more precise the denoising is, mostly for high denoise values. There is a precise formula to calculate memory consumption on the GitHub repo.
What is considered a high denoise (sigma?) value?

[Z2697]

What is the radius parameter doing in BM3D? The documentation says it looks across multiple frames of video (as long as the radius isn't set to 0).

I didn't test this aspect too much, I mostly picked radius settings that fit in the memory of my GPU and didn't slow encoding too much, but in my brief tests I didn't see much difference in the radius setting, though I have to be honest, I don't know what I should have looked for or in what sort of content it would make a difference.

BM3D, for each "block to be denoised", finds similar block (matching) and group them like a "pillar" which is where the third dimension comes from, and then do the rest steps.
V-BM3D, as I understand it, extends the matching range to adjacent +- "radius" frames, the rest is pretty much the same.

[tormento]

What is considered a high denoise (sigma?) value?
I usually set sigma=4 for clean source up to 10-12 for bad ones but, in that case, more processing is often needed. I read that BM3D gives the best with sigma > 8 but I find it pleasant even with 4 for recent BDs.

For some strange reason, even values of sigma give better results than odd ones.

[Stereodude]

I usually set sigma=4 for clean source up to 10-12 for bad ones but, in that case, more processing is often needed. I read that BM3D gives the best with sigma > 8 but I find it pleasant even with 4 for recent BDs.

For some strange reason, even values of sigma give better results than odd ones.
Interesting... I've found that a sigma of 2.0 is plenty for 1080p BD sources that have no to very little frame to frame "grain"/noise on top of the image. For really clean BDs there are no readily apparent differences in the post processed output image, but after BM3D it compresses considerably better.

2.5 or 3.0 clean up light grain/noise nicely. By the time the grain/noise gets really heavy I've found large sigma values negatively affect sharpness and detail making the image look unnatural/overly processed. I prefer to use a sigma of like 5 to knock down the finer parts of the grain/noise which helps compressability significantly, but leaves visible coarseness which presumably the director wanted. Obviously the compressed HEVC video track has higher bitrates than a clean source would, but they're still smaller than the source and look more pleasing to my eyes than going crazy with the sigma.

I hadn't noticed anything about whole numbers or even numbers vs odd or fractional settings.

[Z2697]

I usually use 1 or even less sigma LOL
And because BM3D and even VBM3D is not great at temporal noise I use it together with MDegrain, or even as it's reference, or limit (like this (https://github.com/Mr-Z-2697/z-vsPyScripts/blob/b427bf5ffc54bb4abfaf72b373b8d39fe332e2a5/zvs.py#L242-L247), as I stole the idea from MCTD)
BTW the sigma of BM3D (mawen1250) and BM3DCUDA/CPU (WolframRhodium) does not have exact same effect, somehow. The latter is roughly 5/3 times "stronger" in my opinion. It's BM3DCUDA I was talking about in first line.

[Stereodude]

I usually use 1 or even less sigma LOL
And because BM3D and even VBM3D is not great at temporal noise I use it together with MDegrain.
BTW the sigma of BM3D (mawen1250) and BM3DCUDA/CPU (WolframRhodium) does not have exact same effect, somehow. The latter is roughly 5/3 times "stronger" in my opinion. It's BM3DCUDA I was talking about in first line.
I also gave BM3DCUDA/CPU numbers. I found very little downside to using a really higher sigma value like 8 on the first few discs I tried it on where 2.5 or 3 would have done the trick. On clean sources in static shots a high sigma doesn't really seem to negatively affect the image from what I could see. I subtracted the after BM3DCUDA image from the source to see what it was removing / changing. When content is really noisy was where I started re-evaluating the idea of using a higher than necessary sigma. Things that are moving quickly or camera movement were blurred inconsistently or perhaps erroneously with high sigma values (like 10+). I was stepping through it frame by frame, not watching it in motion, so maybe in motion it wouldn't have been noticeable, but it didn't sit well with me.

[Z2697]

The "visibility" of the effect, by the same sigma, whether it's downside or not, will vary depending on the noise strength. For most contents I normally filter, sigma=3 already looks bad.

[tormento]

The only negative side of BM3D is that, with high sigma values, it can expose the underlying blockiness of high crf values, previously hidden by grain/noise. Unfortunately, when that happens, there is very little you can do. Once the compression went too high, you have to deal with that blocks in some way. One good example is Evangelion BDs, where they applied too much AVC compression over an unprocessed very bad master. You have to choose between bad noise or blockiness. Any hint is welcome.

[Z2697]

High sigma blockiness comes from the block_step (being > 1).
Which I think, is always there, just lower sigma makes less significant.
Lower block_step makes it less significant as well, and like what mawen1250 does in his wrapper, having the 2 steps use different block_step also helps.

[DTL]

You have to choose between bad noise or blockiness. Any hint is welcome.

There are very many methods possible. First you can try to use (many ?) 'deblock' plugins before denoise.

Next is to make processing of several copies of the clip with somehow shifted block-tesselation grid and calculate average of the result. Same as overlapped processing in mvtools. The external plugin exist for overlapped averaging - BlockOverlap.

[Z2697]

No you shouldn't. The block is created by BM3D, you should tweak the settings instead of degrading it further.

[tormento]

Next is to make processing of several copies of the clip with somehow shifted block-tesselation grid and calculate average of the result. Same as overlapped processing in mvtools. The external plugin exist for overlapped averaging - BlockOverlap.
That goes beyond my average AVS script writing. :p

Will try VsDeblockPP7.

[DTL]

The block is created by BM3D,

If BM3D create blockiness - it is also possible to decrease it using 2 processing pipelines of original and shifted clips and aggregate at the output with BlockOverlap plugin. If the block-tesselation grid of the BM3D is rectangular and stable it may work good enough.

Script is simple to test:

orig=last
shifted=Crop+AddBorders to shift in H and V directions to 1/2 of block size

orig_fitered=BM3D(orig)
shifted_filtered=BM3D(shifted)

BlockOverlap(orig_filtered, shifted_filtered) (apply crop-shift back if required by BlockOverlap - not remember it right)

If cropping is not nice and edges quality must be highest - the orig and shifted clips may be created with AddBorders only with a bit bigger frame size and Crop added sides after BlockOverlap.

BM3D may be very old and abandoned so developers do not add internal overlapped processing to decrease output block artifacts like was added in mvtools.

[tormento]

BM3D may be very old and abandoned so developers do not add internal overlapped processing to decrease output block artifacts like was added in mvtools.
BM3D isn't old, it's just been "abandoned" as it happens whenever it's one person only to create builds. IMHO it's one of the best denoisers so far.

[DTL]

If developer still active and the filter create block artifacts - you can try to ask about implementing block overlapping to decrease it. It may be somehow better in performance in comparison with scripted form. Also BlockOverlap support only 2x-diagonal offset mode and with other software methods you can use 4x overlap like default in mvtools with overlap=blocksize/2. It makes better quality but takes 4x more blocks to process.

[tormento]

If developer still active and the filter create block artifacts - you can try to ask about implementing block overlapping to decrease it. It may be somehow better in performance in comparison with scripted form. Also BlockOverlap support only 2x-diagonal offset mode and with other software methods you can use 4x overlap like default in mvtools with overlap=blocksize/2. It makes better quality but takes 4x more blocks to process.


Good idea. I opened an issue and I hope he will read it.