Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
2nd March 2005, 10:23 | #81 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,389
|
kassandro -
as it is currently written, that LUTxy command performs limiting on the Y plane only. U & V planes are passed through from RemoveDust just as they are. To do the same limiting for chroma, one would specify uexpr & vexpr with the same string as yexpr, and call U=3 & V=3 in the LUTxy comand. However when I tried that, I didn't see any remarkable visual difference. So I took the faster option with just passing chroma untouched. Granted, that clip snipplet doesn't show all possible aspects. But then, that wasn't my aim. It is so that I really like RemoveDust in general. But there are motion characteristics where I don't like the plain RemoveDust at all, which held me back from using it. The most important one is just this: loss of detail in *slowly* moving areas. If motion is slow, then detail loss is easily noiticeable. In high motion, it's much less of a problem. And so I just took a random example that showed the problem I'm mostly concerned about. It wasn't meant at all to be a "general" comparison. Then, one could see such "hard limiting" as a poor technique. In fact it is, because it's just a dumb "cutting-off the effect". But looking at it from another angle, one could say as well like this: Generally, RemoveDust does the right thing. But in certain cases, it just does too much. Therefore, if limiting the effect to rather low values still is able to remove the noise/grain we want to remove, then there is no need to do any more. Or, in other words: The filter is working in the right direction, but is working too diligent. We better tell it where to stop. Keeping in mind that all of this is working without true motion compensation, I think that this is a really great filter combination for noise and grain removal. And I could imagine that it gets even more impressing when combined with motion compensation, even if the incredible speed is lost then. Alas I've currently not enough free time to look into that. perhaps someone might want to try that. Valky: Just try it and see for yourself. Not that difficult, it's just a few lines. Very roughly from memory: Together with XviD encoding, I got ~5fps for PixieDust and ~20fps for LRemoveDust. LRemoveDust encoding was only marginally slower than without any filtering - XviD was the most limiting factor, there. For plain rendering without encoding, (L)RemoveDust is in the range of 8~10 times faster than PixieDust. Regarding *that* difference in performance, I think the results are so close to PixieDust, it's almost impertinent
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) Last edited by Didée; 2nd March 2005 at 10:25. |
2nd March 2005, 15:15 | #82 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,718
|
Didée,
would it be possible to make a YUY2 version of LRemoveDust? It seems very interesting but would require converting to YV12 and then back to YUY2 to feed to CCE.
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
2nd March 2005, 16:25 | #83 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,389
|
Yes and no. Kassandro's RemoveDust function supports YUY2 natively (in fact, it supports all colorspaces, afaik). But the limiting for LRemoveDust is done through MaskTool's "yv12lutxy" ... the name says it all. Because of that, at least for doing luma limiting there's no way to avoid the yuy2->yv12->yuy2 journey (unless someone codes a plugin for that). However, passing through the untouched "original" yuy2 color planes (as produced by RemoveDust, without limiting) is perfectly possible, with a small speed penalty.
If I made no error (you know ... writing a script without possibility to test it), the following should do it: Code:
function LRemoveDust(clip input, int _mode, int "limit") { limit = default(limit,4) LL = string(limit) STR = "x "+LL+" + y < y "+LL+" - x "+LL+" - y > y "+LL+" + x ? ?" repmode = 2 clensed = Clense(input) rep = Repair(clensed, input, mode=repmode) rg = RemoveGrain(rep, mode=_mode) trep = TemporalRepair(rg, rep) trep2 = isyuy2(input) \ ? trep.converttoyv12() \ : trep input2 = isyuy2(input) \ ? input.converttoyv12() \ : input limit!=0 ? yv12lutxy(trep2,input2,yexpr=STR,U=2,V=2) : last out = isyuy2(input) \ ? last.converttoyuy2().mergechroma(trep) \ : last return out }
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
2nd March 2005, 16:27 | #84 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,718
|
Thanks a bunch, I'll have to try it on my next capture tomorrow
EDIT: A quick test shows it works, at least no errors pop up
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... Last edited by Boulder; 2nd March 2005 at 16:30. |
3rd March 2005, 22:12 | #85 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,718
|
I did some tests and found LRemoveDust wonderful! The YUY2 version is quite a bit slower (something like 0.29RT vs. 0.44RT), and I noticed that conversion to YV12 and back to YUY2 (and then back to YV12 in the encoder) is not noticable with TV captures so I ended up using the original function. LRemoveDust keeps details in motion better than RemoveGrain().RemoveDirt() so it looks like it'll become the next allround filter for my analogue captures -- at least till kassandro updates RemoveDirt
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
4th March 2005, 11:08 | #86 | Link | ||
Registered User
Join Date: May 2003
Location: Germany
Posts: 502
|
Quote:
When I did my first analog capture recently, I saw that when using RemoveGrain with AVIsource, VdubMod crashes upon exit. I had the same problem with RemoveDirt once. The probable reason is that I didn't deallocate some objects properly and AVISource doesn't like that. This is a bug of AVIsource as well and I recommended to fix it long ago, because under certain circumstances objects cannot be deallocated. This is certainly not a serious problem and it will be fixed with the next version of RemoveGrain in the summer. YV12LUTxy is a very useful, reasonably fast and also very smart (because of using look up tables) filter, which should be made internal, but in LRemoveDust it can be replaced by the much faster LimitChange. However, when I tested an optimised version two days ago it didn't work anymore, but I hope to post an inofficial version next weekend, which should speed up LRemoveDust a little. It will also come with a modified inofficial version of Clense, which then can be used together with mvtools to provide a slow, motion compensated version of RemoveDust. It should reduce motion blurring considerably but also worsen compression. Quote:
|
||
4th March 2005, 11:20 | #87 | Link | |||
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,718
|
Quote:
Quote:
Quote:
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
|||
4th March 2005, 12:07 | #88 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,718
|
I just had a walk with the dog and this thought came to my mind:
1)source is YUY2 2)convert to YV12 3)use LRemoveDust_YV12, enable greyscale mode for even faster processing 4)convert to YUY2 5)put the original chroma information back 6)denoise the chroma (CNR2) This way there would be no loss, right? Since luma noise is much more easily seen by the human eye, it wouldn't matter if chroma wasn't processed by LRemoveDust and you can use other plugins like CNR2, which is also fast, for that.
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
4th March 2005, 15:20 | #89 | Link | |
Registered User
Join Date: May 2003
Location: Germany
Posts: 502
|
Quote:
Actually, I can outwit Avisynth's color space limitations. For Avisynth a planar YUY2 frame will look just like an ordinary interleaved YUY2. However, the information is organized differently on the planar YUY2 frames. Say, you have a 720x576 interleaved YUY2 frame. Then the first byte is the luma of the first pixel, the second byte ist the U value of pixels 1 and 2, the third byte is luma of pixel 2 and the 4th byte is the V value for pixels 1 and 2 and then it goes on with pixels 3 and 4, etc. On a planar YUY2 frame, you have first the 720 luma values of the first line, then the 360 U values of the line and finally the 360 V values of the line. Thus I have only rearranged the values of the pixel lines, but any yv12 filter could handle such frames with only tiny changes just like yv12 frames. However, ordinary spatial YUY2 filters like a horzontal crop would be very destructive (a vertical crop wouldn't hurt) for such frames. Temporal filters, which treat luma and chroma equally could be applied to planar and interleaved YUY2 in the same way. |
|
6th March 2005, 06:01 | #90 | Link |
Registered User
Join Date: May 2003
Location: Germany
Posts: 502
|
motion compensated RemoveDust
Here is my motion compensated RemoveDust:
Code:
function MCRemoveDust(clip input, int _smooth, bool _grey) { repmode = 2 clmode = 4 quad = Quadruple(input) # each pixel is quadrupled, SSETools are needed qpel = RemoveGrain(quad, mode=12, modeU=-1) # blur the luma for searching motion vectors bvectors = MVAnalyse(qpel, blksize=16, lambda=200, isb=true) fvectors = MVAnalyse(qpel, blksize=16, lambda=200, isb=false) backward = MVCompensate(quad, bvectors) forward = MVCompensate(quad, fvectors) clensed = mcclense(quad, backward, forward, grey=_grey).shrinkby2(grey=_grey) #SSETools are needed for shrinking rep=Repair(clensed, input, mode=repmode, modeU=_grey ? -1 : repmode) rg = RemoveGrain(rep, mode=clmode, modeU=_grey ? -1 : clmode) return TemporalRepair(rg, rep, grey=_grey, smooth=_smooth) } Code:
function LRemoveDust(clip input, int _mode, int "limit") { limit=default(limit,4) repmode = 2 clensed = Clense(input) rep=Repair(clensed, input, mode=repmode) rg = RemoveGrain(rep, mode=_mode) rd=TemporalRepair(rg, rep) return LimitChange(rd, input, limit, limitU=255) } Code:
function LRemoveDust(clip input, int _mode, int "limit") { limit=default(limit,4) repmode = 2 clensed = Clense(input) rep=Repair(clensed, input, mode=repmode) rg = RemoveGrain(rep, mode=_mode) rd=TemporalRepair(rg, rep) return LimitChange(rd, input, limit) } Last edited by kassandro; 6th March 2005 at 06:03. |
6th March 2005, 09:28 | #91 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,718
|
That's a nice one, got another capture coming in later today so I guess it's time to play
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
6th March 2005, 16:32 | #92 | Link |
Registered User
Join Date: May 2003
Location: Germany
Posts: 502
|
Sorry, Boulder. Something must have gone wrong here. I just ran a test and it turned out, that MCRemoveDust gives exactly the same as RemoveDust. There must be misunderstanding of the MVTools on my side.
|
6th March 2005, 23:35 | #93 | Link |
Registered User
Join Date: May 2003
Location: Germany
Posts: 502
|
Unfortunately, the Filter mcclense, part of a slightly changed RemoveGrain plugin, was working exactly as clense, whence MCRemoveDust did work exactly as the RemoveDust. I have now uploaded a corrected and tested version. The script function remains unchanged. I did a test with a short 5500 frames part of a fairly clean black&white movie from 1952. Motion was definitely sharper as with the original RemoveDust. On the other hand in such old movies motion is always blurred a little. Thus this video was not very suitable for MCRemoveDust. Here are the compression results:
Code:
MCRemoveDust 18.896.476 Bytes RemoveDust 18.791.986 Bytes |
8th March 2005, 13:09 | #94 | Link |
Registered User
Join Date: May 2003
Location: Germany
Posts: 502
|
new comparison
I just made a comparison with a high quality source from 2002 (63958 frames, 720x576, with ogg vorbis audio):
Code:
unfiltered 347.059.657 Bytes RemoveDust 284.028.286 Bytes MCRemoveDust 280.028.971 Bytes LRemoveDust 297.983.847 Bytes Because the source contained relatively little grain and was virtually dirt free, the compression gain was only 18.2% with RemoveDust and 19.3% with MCRemoveDust, less than half what I experience in the average with my usually much older films. Encoding time with MCRemoveDust was a little bit more than two times the encoding time with RemoveDust. Using LRemoveDust (should be identical with Didee's version) the output is 6.4% larger than with MCRemoveDust. The comparison with LRemoveDust is a little bit unfair, because it uses TemporalRepair with smooth=0, the default value, but RemoveDust, MCRemoveDust use smooth=1, but the impact should not be very significant (1-2%), because TemporalRepair only corrects the spatial denoiser RemoveGrain, such that it cannot touch (temporally) static pixels. Last edited by kassandro; 8th March 2005 at 19:29. |
8th March 2005, 14:29 | #95 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,718
|
How does LRemoveDust (luma only) compare? Using MCRemoveDust is unfortunately out of my league since the encoding time is already quite long and I can't run the computer overnight. However, it could be used on shorter clips and the result first saved in lossless format to speed up the encoding to the final format.
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... |
8th March 2005, 19:30 | #96 | Link | |
Registered User
Join Date: May 2003
Location: Germany
Posts: 502
|
Quote:
|
|
9th March 2005, 02:30 | #97 | Link |
Registered User
Join Date: Oct 2003
Posts: 155
|
Sorry, but I can't get LRemoveDust to work. This is my script...
Code:
LoadPlugin("c:\filters\decomb521.DLL") LoadPlugin("c:\filters\RemoveGrainS.dll") LoadPlugin("c:\filters\RepairS.dll") Import("c:\filters\LRemoveDust.avs") AviSource("c:\capture (2005-03-08 at 00-05-53).avi") Trim(0,7322) Telecide(order=1,post=0) Decimate(cycle=5,mode=2) ConvertToYV12() LRemoveDust() Is there anything wrong ? |
9th March 2005, 03:24 | #98 | Link | |
Registered User
Join Date: Oct 2003
Posts: 155
|
OK. I got LRemoveDust to work with this...
Code:
LoadPlugin("c:\filters\decomb521.DLL") LoadPlugin("c:\filters\RemoveGrainS.dll") LoadPlugin("c:\filters\RepairS.dll") LoadPlugin("c:\filters\MaskTools156.dll") Import("c:\filters\LRemoveDust.avs") AviSource("c:\capture (2005-03-08 at 00-05-53).avi") Trim(0,7322) Telecide(order=1,post=0) Decimate(cycle=5,mode=2) ConvertToYV12() LRemoveDust(4,2) Quote:
I wonder what is wrong... Last edited by vigi_lante; 9th March 2005 at 03:49. |
|
9th March 2005, 09:00 | #99 | Link | ||
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,389
|
The question was:
Quote:
Quote:
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
||
9th March 2005, 09:01 | #100 | Link |
Pig on the wing
Join Date: Mar 2002
Location: Finland
Posts: 5,718
|
It should be faster, I do it like this:
org=last ConverttoYV12() LRemoveDust(4,2) GreyScale() ConverttoYUY2() MergeChroma(org) and it's still much faster than PixieDust. I have tweaked the function so that only luma is processed by adding the necessary grey=true and modeU=-1 parameters where needed, but the difference shouldn't be too big. Then again, I have a SSE2 CPU which gives a nice boost thanks to kassandro's optimizations EDIT: Didée's right..if you've tried using PixieDust in your IVTC script, it will show you what slow is
__________________
And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon... Last edited by Boulder; 9th March 2005 at 09:18. |
Thread Tools | Search this Thread |
Display Modes | |
|
|