LTSMC - where to download?

[Aquilonious]

@To all -
I've been through quite a handful of video forums and I still cannot find a host where I can download LTSMC which, supposedly, is even better than PixieDust at grain removal. I've seen the results in several threads here and elsewhere and they are truly excellent.

Can anyone point me to where I can download LTSMC, or is it a subset of a package of filters, like MVTools? Also, can you point me to some sample LTSMC scripts that would be effective at removing a heavy amount of grain?


@Didee -
Can you help me out here? You mentioned TimeDenoise. How does it compare with LTSMC, or PixieDust?

So far I've been using PixieDust and even though it's glacially slow, I really like the results. But I have some sources which are even too much for PD, and I need a more "heavy duty" filter like what I've seen with LTSMC to get the heavy grain out.

[unskinnyboy]

http://home.arcor.de/dhanselmann/_stuff/LTSMC.rar

But Didée says it isn't ready yet as per here (http://www.neuron2.net/board/viewtopic.php?t=787).

[Didée]

Aquilonious -

from unskinnyboy's links you get the basic framework of LTSMC, and brief information on how to get it run.

And indeed, I do not recommend it for general usage, yet.

The point is: while motion compensated denoising is a smart and powerful concept in general, actual implementations have their weaknesses and shortcomings. In particular blocking artefacts (on non-compensatable image parts), creation of artificial motion (e.g. on static objects that are lit by flaring lighting), and inefficiency on big value changes (e.g. line flicker from deinterlacing).
LTSMC was started with the goal to get past most or all of these problems. As of now, however, it's still far away from there. Very far. In fact, the framework was put online to not get accused of "not telling what I did" in some other posts/forums, and to perhaps give some ideas for others that like to fiddle around.
If there was a plugin for the deblocking routine I have in mind, a reasonable alpha of LTSMC could be made. Alas, there isn't - and I'm overbusy with other stuff, can't work on a dozen different things at a time.

Despite all of that, in case of rather extreme noise where anything else seems to fail, it may give remarkable results ... patience and much of RAM presumed.

With TimeDenoise I've not toyed too much. In one case it worked very good, in another case it worked rather poor.
Compairing them makes little sense - TimeDenoise is a combo of several spatio/temporal filters, with MVDenoise put in the middle. LTSMC tries to prove one single methode solely: motion compensated denoising.

PixieDust, having been a long-time favorite of mine, mostly suffers from the fact that it seemingly works only on +/- 1 frame on the temporal axis. For rather light noise, that's sufficient. For heavy noise or grain, it is not.

[Aquilonious]

Aquilonious -
PixieDust, having been a long-time favorite of mine, mostly suffers from the fact that it seemingly works only on +/- 1 frame on the temporal axis. For rather light noise, that's sufficient. For heavy noise or grain, it is not.
Is 768MB of RAM enough? CCE takes about 256MB when running. My memory timings are pretty tight and it's running in dual channel 400FSB. I'm still getting the $$$ together to build yet another system-- an Athlon 64 with 1GB low-latency RAM, two Maxtor MaxLine III's in RAID 0, and a Radeon 1800XL, among other components.

I tried running PixieDust(8) twice in series on a grainy source but ugh, CCE was only producing about .03 frames/sec! With one PD pass it CCE will range from .25 with occaisonal peaks up to 5 frames/sec.

Perhaps I'm not running PD correctly. Here's my script:

LoadPlugin("C:\Other\Plugins\LoadPluginEx.dll")
LoadPlugin("C:\Other\Plugins\DustV5.dll")
ConvertToYUY2()
PixieDust(8)
ConvertToYV12()

Do I need both lines 3 and 5 to run PD correctly? Doesn't Rebuilder convert to YUY by default?

A few other Q's:

1. Does PD(Dust) use MMX and/or SSEx? If so, is there any appreciable speed increase?

3. I use Deen at it's default values. Is there anything I can do with Deen's values to remove more grain?

3. Other than PD, what other grain removal filters do you regularly use?

There are so many grain filters. I've tried STMedian which I didn't like much because it blurs the output too much for my taste. Also tried RemoveGrain but it didn't remove much grain. I'd like to try FluxSmooth but don't know how to write a script for it.

Thanks for your help. :)

[mg262]

If there was a plugin for the deblocking routine I have in mindDo tell! (If it doesn't take too long to describe.) Having said that, I need to apologise for asking you for plug-in ideas and then having missed your "ideas bin" until yesterday... I've been pretty slow on video work recently, due to multiple real-life happenings.

[Didée]

No tragedy. Important things have to come first. We're here because it's fun, aren't we. ;)

Do tell! (If it doesn't take too long to describe.) OK, a short version of the basic procedure:

a) Apply a median-like filter, probably of radius 2 or 3, that for every center pixel takes into account only pixels of the same 8x8 block. Now we have removed detail, but the blocking is still 99% intact.

b) Now run a blocking detection like in kassandro's Deblock(), i.e. search for hard edge steepyness. Perhaps blur the found values along the borders.

bb) Safety check to not consider horizontal/vertical detail as blocking: detected blocking on only one of a block's 4 borders is invalid. There must be blocking detected on at least 2 block borders; if this isn't, do nothing. (Blocking on only one border can happen, but is seldom.)

c) For those borders detected as being blocky, create compensation by blurring across the borders.

d) Construct full block compensation by "blurring" the border compensation over the whole 8x8 block, i.e. fade out each border compensation to the opposite border.

e) Apply block compensation to input.


This will remove blocking without blurring detail.

Imagining a frame in form of a height field, both the problem and the needed operation get clearly obvious.

[mg262]

a) Apply a median-like filter, probably of radius 2 or 3, that for every center pixel takes into account only pixels of the same 8x8 block. Now we have removed detail, but the blocking is still 99% intact. BlockMedian, 16 October 2005 (http://people.pwf.cam.ac.uk/mg262/posts/BlockMedian_16Oct05.dll)
Source (http://people.pwf.cam.ac.uk/mg262/posts/BlockMedian_16Oct05.cpp)
BlockMedian(int radius)
BlockMedian(int horizontalradius, int verticalradius)

These use a square window, with radius taken as inclusive (so radius 3 means up to 7 x 7 pixels are averaged). Chroma is trashed; I can change that to copy or compute chroma block-subordinate median if you prefer.

I'm using a standard median-finding algorithm, but the method is still pretty slow... so perhaps something like BlockMedian(0,3).BlockMedian(3,0) might be worth trying (disclaimer: I haven't yet checked that that is faster)? Or alternatively the mean should be much faster to compute...

No hurry to test this, by the way.

b, c, d) ... the principles are described very clearly but there are details I'd like to check (e.g. exact preferred methods for deciding whether an edge is steep, blurring over an edge, fading out)-- will write more on this later.

Edit: Actually, those examples ^ ^ span most of my questions. Would you prefer these steps to be separate plug-ins, or a single plug-in, or somewhere in between (e.g. you could have b+bb generate a mask of the blocks and then mask the compensation).

[Mug Funky]

mg262, you are the man :)

i'll try this when i get a chance (right now i'm using blindpp with some high/low pass separation, but i'm not happy with it conceptually, as no matter how i think of it, it should work the same as blindpp on its own, but somehow it's slightly more effective at detail preservation and block removal. i put it down to a fluke :)).

one thing though - blockiness may not always be limited to 8x8 borders relative to the frame. wouldn't high quants + motion mess the deblocking up a bit? like if a q31 block from a previous or next frame were moved to an odd position on the current frame, then the residual was also coded at q31, wouldn't there be blocking right inside a macroblock? this would be true for non-q31 blocks as well depending on the performance of the encoder used and whether it was a dark area or not...

maybe a deblocker that had access to motion-vector data could take this stuff into account - kind of like forcing an in-loop filter into mpeg-2... if the reference frames are deblocked before the current frame is constructed from them, then further deblocking would only be needed on mod 8 edges.

dunno, just thinking out loud here. it's possible that i've missed something quite important.

[Didée]

Mug Funky -

that's a problem indeed. - But it's kind of a "2nd order" problem. I'd say, first concentrate on blocking that's aligned to a codec's block grid, 8x8 or 16x16. Seems complicated enough ...
Also, while this here might even lead to a general purpose deblocker, the actual context is "deblocking a motion compensated frame", where the case of "references to blocky blocks" hardly is an issue.


Clouded -

first, thanks for BlockMedian. Does the intended job, and I like it :)

However, there are some issues in above a)..e) list that should be noted:

step a) may even disturb, depending on the block's actual content. It was meant to possibly make the detection of blocking easier. But after actually trying BlockMedian, it seems that radii > 1 even create or emphasise blocking. Obviously, that's evil for calculating correct border compensations.
Proposal: we first leave this out, and later see if some sort of pre-processing might make the detection of blocking more reliable.

step bb), after looking closer at samples, also is not always beneficial - hard transitions through blockiness very well may occur on only one block border. (e.g. when posterization happens on originally very smooth gradients).
If current block shows blocking on only one border, perhaps it's better to look what's happening at it's neighbor blocks, or so ...
Proposal: we first leave this out, and later see if ... ;)

So, the most important thing is, of course, the decision at block borders if blocking is present, or not. I think that the current method of kassandro's MDeblock is a good one ... perhaps some tweaking could be done. (To me it seems that MDeblock does make a tad too much of false decisions, both in not-detecting and wrongly-detecting blocking.)
The 2nd really important point is d) - making a full block compensation based on the calculated border compensations. IMO the weakness of MDeblock is that it only processes block border pixels. Doing so is okay for rather light blocking. But in case of slightly stronger blocking, MDeblock will only soften the transitions between the blocks, but the blocking in itself stays intact.

Let me amuse you with some quick scribbling-with-the-mouse:

http://img426.imageshack.us/img426/2664/blocks8jm.png

With some musing over this, the aim should get clear: at the block borders, decide what to do with the borders. Then gradually spread those border compensations over the whole block, so that the complete block gets 'skewed' to fit with its neighbor blocks. Since this correction necessarily creates a "low frequency" change to the block as a whole, any detail a block still might be carrying would not be affected.

[mg262]

Grrr... wrote out a proper reply to this and then lost it. Briefly

step a) may even disturb, depending on the block's actual content. It was meant to possibly make the detection of blocking easier. But after actually trying BlockMedian, it seems that radii > 1 even create or emphasise blocking. Obviously, that's evil for calculating correct border compensations.
Proposal: we first leave this out, and later see if some sort of pre-processing might make the detection of blocking more reliable.
-- I hope this isn't due to a bug; I tested it in various ways before releasing, but anyway: source code tidied somewhat and linked to above in case anyone is kind enough to skim for correctness.
-- Mug Funky... best to wait for remaining steps to be implemented before testing... sorry if I didn't make this clear above, but it's just step a) that that filter dealt with -- and as noted it doesn't seem to be doing what is wanted.

I like the diagram -- makes me think of old platform games! Fade in/out is clear, at least if it is to be linear... if left-hand pixel increased by 4, righthand pixel increase by 12, get this increase pattern:

4 5 6 7 9 10 11 12

But I have some questions about how we obtain the 4 and 12.

First note that trying to do horizontal and vertical simultaneously may be a bad idea... think about corner pixels being increased twice. So I will assume we do horizontal first, take the result and apply vertical to it.

-- Are the 4 and 12 obtained on a per-scanline basis or a per-blockline (i.e. group of eight scanlines) basis?

Independently of that, there are many methods imaginable. For example:

i) kassandro's MDeblock method
http://home.arcor.de/kassandro/MDeblock/MDeblock.htm
(I have read this but have not yet thought about it hard.)

ii) Imagine this:
x0|x1 | is a block boundary.

we can just replace x0 and x1 by their average. This is a bad method since we will end up with lots of pairs of vertical lines of the same colour... so it might be better to move x0 and x1 some fraction of the way towards their average.

iii) Imagine this:
x0 x1|x2 x3 where | is a block boundary.

Two points, namely x0 and x3, define a line (in the graph of luma against position); we just adjust x1 and x2 to lie on that line. This is not great as it stands because x0 and x3 will also change, so we won't have a line any more... but the method can probably be adjusted to force a line in the output.

iv) Imagine this:
x0 x1 x2|x3 x4 x5 where | is a block boundary.

Now x0, x1, x4, x5 together define a cubic curve and we can proceed as in the line case.

I'm game to try anything, but I would rather you made the call :)

@Mug Funky,
For the moment I'm just doing as I'm told :D. Changing the block size would be easy... I already have an internal variable called blocksize in anticipation of that. Apart from that, I think you've just outlined a good reason why deblocking often belongs inside a decoder... in fact, I've been wanting to try the following for a long time -- and in some ways it's very similar to Didée's method:

The coefficients for each DCT block in frequency space define a continuous surface in spatial-space. What we normally do is sample from the surface on an 8 x 8 grid to get the pixel luma values. But there is no reason why we can't sample elsewhere -- in particular, we can sample halfway between this block and the next block to the right -- i.e. 0.5 to the right of the rightmost pixels. Similarly we can sample 0.5 to the left of the leftmost pixels in the next blocK. Now these numbers, calculated from the left and calculated from the right, should be very similar... and we see blocks precisely when they are dissimilar. Now: when they are dissimilar we can tweak the coefficients until they match... this can be done quite nicely with the least squares method I've used elsewhere. In other words, we adjust the two neighbouring surfaces until their edges touch. (And we could also try matching the slopes of the surfaces.)

But obviously that is something that should sensibly be done as part of a decoder...

Clouded

[Mug Funky]

that's a pretty good idea!

the DGdecode source is available... i'm not sure how readable it is (not looked at it), but IIRC DG's done a lot of work documenting it. perhaps the postprocessing part is written such that you wouldn't need to rewrite anything, but i doubt it.

in a completely unrelated matter, it'd also be cool for DGdecode to be able to return the motion vectors in a service clip, mvtools style :) would speed up mvBobbing of ex-DVD sources.

[mg262]

Mmmm... for me, working on other people's code is no longer fun. I normally only do when I'm paid to. Hence and for other reasons I much prefer people to share ideas than to share code. But I'm very happy for anyone to try implementing the above if they find it interesting... and I will even try and write out the maths if it is helpful.