Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
30th June 2004, 12:58 | #1 | Link |
Clouded
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
|
PixieDustPP by Didée (was: Dust and blocks, a.k.a making dust even slower)
Edit: Go down here to get Didée's new script, which alleviates a lot of the quirks of PixieDust.
________________________________________________________ Original (very old) post... For those of you that can bear to spend the time, I've found that the following is quite effective against the artefacts of dust on a very noisy source: Split the source into four copies, shifted like this: a=last b=crop(2, 0, -2, 0) c=crop(0, 2, 0, -2) d=crop(2, 2 ,-2, -2) Dust each one separately, and recombine. (For the source in question, which is extremely noisy, I use peachsmoother on the result.) The most annoying thing about the process is that you can't run multiple copies of dust in one script. I had thought that 2.55's dllname_functionname calling syntax might allow you to get round the problem by duplicating the dust DLL, but I can't get that the work. (See here http://www.avisynth.org/AviSynthPlugins for the calling syntax.) I've resorted to using a preprocessing script, four dust scripts and a combining script, but usually one of the dust scripts crashes. I doesn't anyone knows of a lightweight program that will duplicate the Vdub direct stream copy (or even ->huffy) but that retries a script if there's a crash? Last edited by mg262; 24th October 2005 at 13:39. Reason: Update thread title and link to PixieDustPP |
30th June 2004, 13:13 | #2 | Link |
Registered User
Join Date: Sep 2002
Location: Germany
Posts: 352
|
well, i guess that's because of some global variables in Dust.
if we only had the sources of Dust... can you provide some screenshots that shows which artefacts are eliminated by using dust this way? how do you combine the 4 results of Dust? A workaround to the inability to call Dust multiple times could be the following: - don't crop but shift the 4 clips. (crop + addBorders) -> same size - append them together to one clip - call Dust - split them to their original length (trim) - combine them like this you have called Dust only once. Greetings, Malcolm |
30th June 2004, 14:00 | #3 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Yep - it was scharfis_brain who came up with that nice little idea yesterday. "Scharfi for President!"
I was suspicious if it would work, because Dust still has to access the same frame multiple times. But now I can confirm: yes, this method indeed works! Just be reminded that you have to keep your frame at a resolution of MOD16 horizontal and MOD8 vertical - otherwise Dust will give you nice artefacts. Pink ones, mostly And more: by adding black borders in order to keep MODs, Dust's ME will most likely fail to use the border blocks as references. Padding the borders with mirrors of themselves before cropping is - still not optimal, but supposingly better than adding black borders. - Didée
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
30th June 2004, 14:09 | #4 | Link | |
Clouded
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
|
I tried the idea on the assumption that dust divided the screen into cells and found the best match for each cell in the previous frame by motion compensation... so that cells did not necessarily match up on boundaries. So removing a couple of pixels from the left (say) shifts the cell boundaries, and averaging removes the boundary affect
I haven't put that very well, but what I mean is that I was hoping that the artefacts (squares) in each version would be slightly offset so that averaging them basically cancelled them out. Whether or not that explanation is correct, the process certainly seemed to yield a benefit. Quote:
The noise is mostly a sense of squares appearing in disappearing over time, so it's hard to see the difference in a static shot, but I'll try and find one that shows it and post it up later... I will post screenshots in a bit, although it's |
|
30th June 2004, 14:12 | #5 | Link |
Clouded
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
|
Didee - just caught your reply. So that's where the pink junk comes from! Luckily I'm going to be cropping everything anyway, so it doesn't matter too much about the border effects.
Does anyone have any idea what size blocks dust works with? I found it hard to tell from stills... offsetting by 2 was really just a guess. |
30th June 2004, 14:42 | #6 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Accidentialy, I've made myself a little function some time ago
Code:
Function BorderMirror( clip clp, int "hh", int "vv") { hh = default( hh, 8 ) vv = default( vv, 8 ) top=clp.crop(0,0,-0,vv) bot=clp.crop(0,clp.height-vv,-0,-0) stackvertical( top.flipvertical(),clp,bot.flipvertical() ) left=last.crop(0,0,hh,-0) right=last.crop(last.width-hh,0,hh,-0) stackhorizontal(left.fliphorizontal(),last,right.fliphorizontal()) return last } BTW, I like to use this function for visualizing if my general cropping is tight enough, or if there are any serious artefacts around the borders in the source itself. If so, they become much more obvious this way. *** I think Dust uses 8-pixel cells internally. (But don't nail me down on that!). Then, I think you can get a pretty nice artefact reduction already with only 2 offsetted sources, instead of 4. Just in case you're in a hurry In that case, make the 2 copies offset by 2 pixel horizontally + 2 pixel vertically. Speak, offset them diagonal to each other. And my final note: Although I know about these artefacts, and usually find them myself in my encodings, I hardly can spot any of them when actually watching my stuff as a normal spectator - Didée
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
30th June 2004, 15:01 | #7 | Link | |
Clouded
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
|
Alternate idea for multidusting:
Quote:
Didée - thanks again. I'm going to try changing the offset to 4 and seeing how that looks in a bit... |
|
30th June 2004, 15:04 | #8 | Link | |
Moderator
Join Date: Nov 2001
Location: Netherlands
Posts: 6,364
|
Quote:
|
|
30th June 2004, 20:41 | #10 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Yes, that's exactly the kind of problem.
BUT: (Ah, one more chance to go into lenghts about my beloved Dust ) Soulhunter, I'm pretty sure you've called PixieDust(limit) with a relative high value for "limit". Greater than 2 or 3, that is. There's one *VERY* important thing about Dust's "limit" parameter - and a assume many people are not aware of it: The "limit" parameter of Dust is NOT a conventional threshold as used in most other filters! Exclamation! Most conventional filters use thresholds in the meaning of: IF ( |test_pixel-reference_pixel| <= threshold ) THEN do_processing ELSE do_nothing Or in words: only do any processing if the neighbor pixel is not more different from the center pixel than (threshold). Now ... Dust works different than the rest of the world. It uses no thresholds at all - at least not any that we know of. The "limit" parameter describes how much any pixel may be changed maximally. I.e. when calling PixieDust(limit=2), than no pixel will be changed more than +/- 2 maximally. There is no reason of upping the "limit" in order to catch more noise. Dust will always reckognize all noise it is able to reckognize, no matter what value for "limit" one sets. And in fact, a limit of "2" is fully sufficient for most sources. "1" is not a ridiculous low value - it is a very good value for "calming down" the noise, for people that prefer to let some noise in the source for encoding. "3" is already suited for strong noise, IMHO. And the more you allow it a pixel to change, the stronger the artefacts, and the higher the risk to get them. (Heavy noise as i.e. in analog capturing is another story - but that's the domain of GoldDust, not PixieDust. Think about the name: if the noise has a "per-pixel-characteristic", then you need ... which Dust flavour?) Now, if anyone thinks "limit" would be a "normal" parameter, and uses it in some "normal" range like 4~~7, then ... he gets blocking for no benefit. Personally, I use PixieDust with a limit of "2" in 99% of all cases. And I have very little problems with blocking. End of schoolmastering (sorry). But if at least one person out there in the wide world has had an "AHA!" effect now, then it was worth it. - Didée
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) Last edited by Didée; 30th June 2004 at 20:51. |
30th June 2004, 20:44 | #11 | Link |
Registered User
Join Date: Sep 2002
Posts: 88
|
An idea for multidusting.
1.Use two (or more) DustV5.dlls. Put original DustV5.dll in your plugins' directory, and copy it into another directoy. or Rename DustV5.dll (eg.Renamed_Dust.dll). In this case, you can place this renamed dll in your plugins' directory, even though there is also original DustV5.dll, too. 2.Make the function for multidusting and use it like this: Code:
#LoadPlugin("your_directory\DustV5.dll") AviSource("hoge.avi") #your source SeparateFields() even = SelectEven().FaeryDust(...) odd = SelectOdd().MultiDust(...) InterLeave(even, odd).Weave() function MultiDust(clip clip, int "limit", string "output") { LoadPlugin("another_path\DustV5.dll") #or #LoadPlugin("(maybe)your_directory\Renamed_Dust.dll") clip = FaeryDust(clip, limit, output) return clip } Last edited by niiyan; 30th June 2004 at 20:47. |
30th June 2004, 20:59 | #12 | Link | |
Bored...
Join Date: Apr 2003
Location: Unknown
Posts: 2,812
|
Quote:
But to tell the truth, I dont know what are PixieDust's defaults... Bye Last edited by Soulhunter; 16th September 2005 at 14:22. |
|
30th June 2004, 21:15 | #13 | Link | ||
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Quote:
Quote:
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
||
30th June 2004, 21:27 | #14 | Link |
Clouded
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
|
Didée - I always work with VHS captures, some of which are very noisy... I spent a couple of weeks last Christmas testing every smoother I could get my hands on and dust was definitely the best.
I typically use Pixiedust at 5 (& yup, I know what limit is) because beyond that the artefacts become very noticeable. That won't clean off all the junk, but using another smoother (spatial?) at light settings fixes most things. I haven't seen much difference between Golddust and high values of Pixie, but I haven't looked extensively -- if you think Gold is better, then I'll run another set of tests... Soulhunter - those clips are brilliant and show the effect much more clearly than anything I have. Is any chance you could try the 4xdust method on the first of those clips for comparison? (Or I'll do it if you like.) |
1st July 2004, 05:15 | #15 | Link | |
Registered User
Join Date: May 2003
Location: Germany
Posts: 502
|
Quote:
|
|
1st July 2004, 12:03 | #16 | Link | |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Quote:
You can easily verify this by taking the difference between the source and source.dust(x) - the maximal difference will be +/- x. Maybe with some outliers. Of course, such small differences are very hard to see - *generally*. But, if you run it with a limit 5 (default), then the maximum possible delta in two neighbor blocks is 10 - and on flat, gradient-like areas, this results in the mentioned problem. Check Soulhunters screenshots. edit: What Dust would need is an internal postprocessing step, similar to what you are doing in RemoveDirt, kassandro. A pity that Steady didn't do that. - Didée
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) Last edited by Didée; 1st July 2004 at 12:08. |
|
1st July 2004, 16:31 | #18 | Link |
Registered User
Join Date: May 2003
Location: Germany
Posts: 502
|
Didée,
you would be right, if for a whole line there would be difference of 10 between neighbouring pixels, then you would see blockiness like in Soulhunter's screen shots. However, if he really cleans everywhere, but with change limitation to avoid artifacts, then blockiness would hardly be possible. Blockiness is possible, if he would use dct/idct filtering like trbarry, or he would clean only some blocks, like RemoveDirt, which has no change limitation, or if he would use a block rather than a pixel dependent cleaning routine. For me personally change limitation is a very primitive kind of artifact protection, because such kind of cleaning has very little effect on sharp dirt. On the other hand, change limitation can be nicely combined with minmax clipping, which I employ in RemoveDirt. Consequently I could get a temporal change limited cleaner with 200-400 fps on my 1.3 GHZ Celeron, which doesn't require any deblocking. It should be very similar to Faerydust. Now spatial cleaning is a very different subject. Firstly, there are the convolution based denoisers, which I deeply hate. Secondly, there is trbarry's dct filter and vaguedenoiser, which seems to use a dwt (= discret wavelet transform) instead of dct. Unlike convolution based filters these filters are based on very sound mathematical ideas. Nevertheless they do not make sense to me, because any advanced codec does such kind of filtering all the time and in a more compression efficient way. Thus instead of using such filters one should better raise the quantiser of the codec. Then finally there is trbarry's Undot, which uses some kind of spatial minmax clipping. This is a fast, simple, virtually artifact free, spatial denoiser, which makes sense for any progressive clip but unfortunately takes out only very little noise. I pushed the idea of Undot a lot further and RemoveGrain is the result. An inofficial release can be found by searching for RemoveGrain in this forum. It is stable but slow. The official release will be SSE turbo charged. Because of the above variety Pixiedust's spatial cleaning method is hard to guess, but temporally with change limitation (change limitation does not make sense for Undot like denoisers) I can certainly do better. Dust was written before Avisynth went yv12. As we can read from the above documentation, it converts internally the interleaved yuy2 color space to a planar color space, processes the frame and then converts it back to yuy2. Thus yv12 would have been the ideal color space for Dust, because all these conversion would not have been necessary. |
1st July 2004, 17:31 | #19 | Link | |
Bored...
Join Date: Apr 2003
Location: Unknown
Posts: 2,812
|
Quote:
Code:
# Multidust AVS Function MultiDust(clip clip, int "limit", string "output") { LoadPlugin("C:\Programme\AviSynth 2.5\x\dustv6.dll") Clip = PixieDust() Return clip } Code:
LoadPlugin("C:\Programme\AviSynth 2.5\plugins\dustv5.dll") Import("C:\Programme\AviSynth 2.5\x\Multidust.avs") mpeg2source("C:\Blah\test4.d2v") SeparateFields() Even = SelectEven().PixieDust() Odd = SelectOdd().MultiDust() InterLeave(even, odd).Weave() |
|
1st July 2004, 18:45 | #20 | Link |
Clouded
Join Date: Jul 2003
Location: Cambridge, UK
Posts: 1,148
|
What's the error message? Should it be
Clip = clip.PixieDust() ? I wasn't really talking about fields, but this... a = Pixiedust(5) LoadPlugin("C:\Capture\AviSynth 2.5\plugins\old\Dust2.dll") b = crop(2, 0, -2, -0).BorderMirrorH(2).Pixiedust(5) LoadPlugin("C:\Capture\AviSynth 2.5\plugins\old\Dust3.dll") c = crop(0, 2, -0, -2).BorderMirrorV(2).Pixiedust(5) LoadPlugin("C:\Capture\AviSynth 2.5\plugins\old\Dust4.dll") d = crop(2, 2 ,-2, -2).BorderMirror(2,2).Pixiedust(5) ab = layer(a, b, "add", 127, 2, 0) cd = layer(c, d, "add", 127, 2, 0) layer(ab, cd, "add", 127, 0, 2) which does work, with bordermirror as above (Didée) and BorderMirrorH having the obvious meaning. Mohan |
|
|