Yellow wavy lines

[cjv]

Hi,
I'm not really sure which section this post belongs in, I apologize if this is off topic..

I have been trying to clean up an old Arnold Schwarzenegger clip that I have..for a few days now..without much luck. I have tried various combinations of SpatialSoftenMMX, TemporalSmoother, TemporalSoften, cnr2, sshiq, etc...

They do an excellent job of reducing the noise (which ironically I don't really mind since its from the 70's), but there are these yellow wavy lines running through the entire movie that I cannot seem to get rid of, or even diminish somewhat. I realize garbage in = garbage out..maybe this source is beyond hope.

This is the source:
http://members.shaw.ca/cjv99/shot.jpg

Any ideas would be greatly appreciated..thanks
cjv

[WarpEnterprises]

Those lines come from some signal interference, you won't get rid of them, except maybe:
Have you (captured) then full size (576 lines) movie, too?
Since the two interlace fields are shifted in time it could be possible to do a strong temporal chroma smooth or such.
If you have the fullsize post it.

[cjv]

Thanks for checking it out,

Unfortunately I didn't capture this. I've had it for a while time, but just recently got a standalone and I'm in the process of converting to VCD (eventually)..but wanted to clean it up somewhat first..the orig. res is 384x288 div4.

I tried something like this:

S=AVISource("mov-huff.avi").BicubicResize(640,480,0,0.65)
TS=S.TemporalSmoother(3).MergeLuma(S)
TS=TS.cnr2().TemporalSmoother(2,1).BilinearResize(384,288)
return TS

It helped somewhat..is my script way off, or is there anything else you might suggest?

cjv

[WarpEnterprises]

1. Why do you make it bigger and then smaller again?
2. Is it possible for you to post or email me some 20 frames (huffyuy) for further thoughts?

[neuron2]

Stuff like that can only be removed by applying notch filters in the fourier transform domain. There are no avisynth filters for that.

[WarpEnterprises]

...but I've found a VD-filter (that works with AVS) that seems to do that. The help is in some chinese/tai language, so I can't configure it correctly, but with AVS:

LoadVirtualDubPlugin("c:\programme\virtualdub\plugins\wNR.vdf","WNR",0)
Avisource("c:\_video\problem.avi").converttoRGB32()

wnr(0, 0, 0, 0, 0, 65280, 0, 0, 0)
wnr(0, 0, 0, 0, 0, 65280, 0, 0, 0)
wnr(0, 0, 0, 0, 0, 65280, 0, 0, 0)

this is what it looks like:
(attachement).

[WarpEnterprises]

Here is the filter mentioned above (it's called Wavelet Noise Reduction).
Maybe someone can translate the help-text.
(extra post as only 1 attachement possible)

[cjv]

@WarpEnterprises:

Thank you SO much! I am SO grateful you took the time to help me out. If you only knew how much time I have spent on this clip!
The results are simply amazing and I'm amazed how you figured out those parameters from the help file :)

I have posted 2 original clips (huffy..3megs each):
http://members.shaw.ca/cjv99/huffy-25frames-dark.avi http://members.shaw.ca/cjv99/huffy-25frames-light.avi

in case you have any more ideas on how to clean up this crappy, crappy source (I plan on cropping of course)..or even care to spend any more time. I usually use crystal clean DVD sources..and this is my first attempt at actually trying to fix something. Regardless, you have helped me greatly..and I really appreciate it.

cjv
PS. I upsize because I was under the impression that temporal filters work better on larger sources..and then bilinear downsize in order to hopefully conceal any artifacts and help compression..although I could be totally off?

[WarpEnterprises]

I found the filter some time ago and considered it useless since I didn't understand the settings.
But with neuron2's hint I remembered this.
Maybe the parameter can be more optimized, I will play a little and keep you informed (I hope tomorrow).
IMHO do not enlarge to filter. It's sometimes good to filter before downsizing but artificially upsizing is not good.
Also I think (at least for mpg compression and ONLY IF ITS NECESSARY) a good idea is to do:
downsizing > temporal filtering > spatial filtering > sharpening
or for a little more quality and much more cpu time
temporal filtering > spatial filtering > downsizing > sharpening
(sharpening for VCD res and xVCD-bitrates/VBR only)

Edit:
I think I've figured out what the VD-params are:
Show selected plane: (debug) shows only the selected plane
Mode: (debug) High-enhances the contrast Diff-shows the diff between original and filtered
Thresholds: can be set for each plane (Luminance Y, two color channels Cr, Cb), for X- and Y-direction and 3 different frequencies (0,1,2) seperately.
For your case the noise is low-freq in X-direction, therefore X2 must be set. And it seems to be only in one color channel.
So you can try yourself (in VirtualDub!) by setting mode to diff and then trying different values in the Cr/Cb X2 settings.
To get those settings to AVS: press ctrl-s in VD and look in the vcf-file at the line:
VirtualDub.video.filters.Add("wavelet noise reduction (0.1)");
VirtualDub.video.filters.instance[0].Config(0, 0, 0, 255, 255, 255, 255, 255, 255);
Those numbers are what you need for avisynth.

2nd Edit:
After looking to your videos try this VirtualDub-Settings (make a .vcf-file and load it with ctrl-l):
VirtualDub.video.filters.Clear();
VirtualDub.video.filters.Add("wavelet noise reduction (0.1)");
VirtualDub.video.filters.instance[0].Config(16448, 8224, 8224, 0, 4096, 65280, 0, 0, 0);
VirtualDub.video.filters.Add("wavelet noise reduction (0.1)");
VirtualDub.video.filters.instance[1].Config(16448, 8224, 8224, 0, 4096, 65280, 0, 0, 0);
VirtualDub.video.filters.Add("warp sharp (1.1)");
VirtualDub.video.filters.instance[2].Config(102);

This fiters the 1-pixel-noise together with the yellow stripes.
You need the WarpSharp filter from DonaldGraft/AveryLee, which seems quite good suited for your clip (normally it's for cartoons, but in your case it helps to sharpen although there is much of this 1-pixel-noise).

Enjoy VirtualDub & Avisynth.

[neuron2]

@WarpEnterprises

As my friend OUTPinged_ would say, "you be da man"!

Gotta grab this and your help info for the filters website.

[cjv]

Wow, the improvement on the walls/flat areas is unbelievable. I implemented your settings, a very light temporal smoother, resized, then applied a really cool sharpening filter..xsharpen.

I tried playing with warpsharp earlier, but that was with the yellow lines still there..you can imagine it wasn't pretty..but the way you have looks good to me.

Currently its encoding at 2-3 fps, using XviD MPEG quant..with a little bit of noise added postproc with ffdshow..it looks sweet! (VCD is next)
Garbage in does not always equal garbage out :)
Oh yea, the wavlet filter seems unstable on my Tbird, lots of crashes with illegal opcode, but works fine on my P4...hmmm...must be some SSE2 in there.
Thanks again,

cjv

[Belgabor]

The text file is japanese. I'll try to get babelfish to give something that resembles a translation ;)

Cheers
Belgabor

[WarpEnterprises]

Just tried the filter on macroblock removal. It does a really incredible job!
The parameters 0,1,2 correspond to pixel wavelengths of 8-16, 16-32, 32-64
That means it is sensitive on "waves" whith that pixel distance from top to top and will perform a notch filter on these.
The GUI has a little bug which doesn't let you enter the numbers directly.
The effects are quite mild, the thresholds can hardly be made to large.

Btw. I found the site I downloaded it some time ago:
http://members.tripod.co.jp/thejam79/index.html

[Defiler]

This filter is awesome. It really puts a hurting on color bleeding, in its most aggressive mode.
The preview window is great too.

[bill_baroud]

i haven't tested it, but i have a translation of the readme ....

Big Thanks to blueskyfloating from Ishin-Digital for his help ^^

enjoy


WAVELET NOISE REDUCTION (WNR 0.1) : thejam

this is a filter to reduce the wavelet noise


use condition.
it needs a CPU that supports MMX and SSE
it would only work on displays that the horizontal and verticle ratios are multiplies of 8


installation
directly copy snr.vdf into virtualdub's plugin folder

uninstall
please delete wnr.vdf


explanation
filter enable, turning the filter on, off

MODE
it's display modes
normal = normal mode
high = high wavelet (which includes lots of noise and surroundings ) increases them.
diff = increases the strenght of each level before and after the filter (make weak weaker, strong, stronger)

Treshold
select Plain in Y Cb C
Use the slider to adjust the shiki value (??? I have no idea what's that) X is horizontal, and Y is verticle. and both have 3 levels.

Show Preview
it shows the preview panel


adjust methods
while looking at the preview panel, put it in high wavelet mode, and set the shikii value (still no idea)



HOW TO USE IN AVISYNTH

adjust using virtualdub
use save procesing setting and save as .vcf
save the following settings into the avs
loadplugin("the path copied/wnr.vdf")
reading the source
wnr("--.vcf")



ABOUT INTERLACE PICTURES

ones that have unlocked the interlace feature or horizontals only (Y0-Y2 are all zeroes) can safely process without any problems
Other than that, since this filter process it by frames, it might have problems.
If you want to process by fields, add deinterlace's filter before and after it.. the process is described following.
deinterlace (mode: unfold)
wavelet noise reduction
deinterlace (mode:fold)


ABOUT PROCESSING IT TOO SLOW
anyway, it's very heavy.
if threshold is zero, at that time, the processing would be skipped,
so if you set other stuff to zero, it could be a little faster.
normally, you would only have to use the brightness signal (Y)'s X0, Y0, X1, and Y1. (no idea.. what this means..)

RAISONS AND EXPLANATIONS
I'm not an expert on graphics, math, and programming, so there might be some misunderstanding and big bugs or mistakes. I will apologize it here first.
the most important part is the param explaination



hmm sorry for the spelling mistake, that has been translated in hurry ^^;;

[Belgabor]

It didnt go as well as I hoped and I had to recriut a japanese speaking frind of mine to help (thanks again Klepto :D) and still couldn't figure out all and finally it doesnt help very much, but here you go:

wavelet noise reduction (wnr 0.1) : thejam

This is a wavelet noise reduction filter.


Requires:
- CPU with MMX and SSE
- Height and width must be multiples of 8.

To install:
- Copy wnr.vdf to the Virtualdub plugins directory

To uninstall
- Delete wnr.vdf


Explanation:

filter enable
Turn the filter on and off.

MODE
Operation mode of the filter.
Normal: Normal mode
High: Emphasis on the high frequency component (which includes mainly noise and
contours)
Diff: This will show the diffrence in pictures before and after you use the filter.

Threshold
Select the plane with Y Cb Cr
Threshold values are set with the sliders.
X is horizontal, Y is vertical, and there are 3 levels respectively

Show Preview
Shows the preview.


Method of Adjustment:
While looking at the preview picture, please set threshold value to the high
frequency indicatory mode.


How to use it with Avisynth
Adjust the settings with Virtualdub
Use "save processing settings" to create a .vcf file
Put the following into the avs script:
LoadPlugin("your path\wnr.vdf")
WNR("~.vcf")


Interlaced material
Interlaced material is no problem if you have no noise in Y direction
(Y0-Y2 thresholds set to zero).
(Couldn't make sense of this sentence, but I think it means something like: )
Otherwise you run into trouble.
If you want to process the noise on field level, put a deinterlace filter
before and after this one like shown below:
deinterlace (mode:unfold)
wavelet noise reduction
deinterlace (mode:fold)


If you have trouble with a too low processing speed
Setting Thresholds to 0 can gain a little speed. (There's more in this sentence,
but i think thats the essence)
Usually its enough to set X0,Y0,X1 and Y1 in the luminance plane (Y).



Disclaimer:
I'm a picture specialist, no mathematician or programmer, so there might be
some errors or fatal bugs. Thank you in advance.


Edit: DOH! Well, you can choose which translation suits you better ;)

Cheers
Belgabor

[bill_baroud]

well i'm not english so i can't judge ;)

but i'm first :D

[Belgabor]

:p

well, i think a combination of both would make a quite good one =D

Other thing, the filter wont work for me :(

Hitting 'Preview' kills vdub and seeking with it enabled gives me a "Internal error: MMX state left on T:\projects\virtualdub_old\main\VideoSource.cpp:1101"

Anyone else with a Athlon 1400 who has the same prob/whom it works for?

Belgabor

[WarpEnterprises]

MMX-error: is your video a DIVX? If yes see http://virtualdub.org/virtualdub_kb
And is the resolution a multiple of 8?

For docu: I'm going to condense your translations into one, add some thing I've found and post it. Thanks!

[Defiler]

Belgabor: If that's a Thunderbird CPU, it doesn't have SSE, and thus cannot run this filter. You'll need a Morgan/Palomino/Thoroughbred.

Thanks for the translations, guys. It's awesome that this filter can run saved .vcf settings from inside AviSynth.

[Belgabor]

@WarpEnterprises: No, its HuffYUV and both are divisible by 8

@Defiler: I thought as much, but on the other hand _2DCleanerYUY, which supposedly needs MMX2 and SSE works like charm ?.?
Perhaps someone else with a Athlon could check if he gets the same works/works not pattern.

Guess i gotta find a prog that checks what your cpu supports :)
(damn, i put it in myself and cant really remember)

EDIT: Found one. Defiler is right, but I wonder why _2DcleanerYUY works then...

EDIT2: Damn, i REALLY sould leran to hit 'update' before i post ;)

Cheers
Belgabor

[Defiler]

You can use WCPUID to determine that.
http://www.h-oda.com/
I only have Athlon XPs and newer now, or else I'd perform the tests you mentioned.

[WarpEnterprises]

Here is our combined help-file with a quite simple How-To! (THX to all)
(attachment)

[Defiler]

Has anyone managed to make this load a .vcf file containing settings? It gives me an error. If I copy and paste out of the .vcf file, all is well.
Example:
wnr("d:\path\blah.vcf") fails, even when "blah.vcf" only contains

VirtualDub.video.filters.Add("wavelet noise reduction (0.1)");
VirtualDub.video.filters.instance[0].Config(0, 0, 0, 0, 0, 0, 0, 65280, 65280);
However, this works fine:
wnr(0, 0, 0, 0, 0, 0, 0, 65280, 65280)

[WarpEnterprises]

You have to use
LoadPlugin instead of LoadVirtualDubPlugin

If thats ok for you then I dont know.
What error?

On my system it works both ways....

[Defiler]

It complains when I use LoadPlugin. I'll give it another try. Thanks for pointing me in the right direction.

[Si]

for info - I've an Athlon 1.33 which only supports MMX and I can't run this filter.
regards
Simon