Question about internal filters in VirtualDub

[ArdenDag]

Hi. :)

I have a few questions about the internal filters available for use in VirtualDub. First of all, I was wondering if these filters are available for use (and to be referenced) inside an .avs script file that VirtualDub can open (and thus make it easier for reference and to save). Is there a .dll file with calls to them? Are there features which VirtualDub can't set directly? Are they usable only internally from VirtualDub? Specifically the filters I'm looking for are the 'Sharpening' and 'Smoothing' filters. I am looking to sharpen and smooth some movies of mine (Anime), and the internal filters in VDub do the job, but I'd rather have some permanent way of scripting them for future jobs.

Are there better filters for the above tasks? Specifically (like I said) for anime, where the quality loss isn't going to be as noticable in certain instances, but MUCH MORE SO in others (you animephiles know what I mean).

Also, is there any better subtitling utility then TextSub for use with VirtualDub? I want to put permanent subtitles in my .avi files, and I intended to just add borders via .avs script and put the subtitles in them afterwards (so the picture isn't cluttered by the annoying subtitles, of course) so my 16:9 picture will be letterboxed with subtitles in the black.

I appreciate any and all help :)

[ArdenDag]

Also, I just encoded a video of mine with the following script:

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\decomb.dll")

mpeg2source("C:\Documents and Settings\Arden\Desktop\Kiki's Delivery Service.d2v")

Telecide(Guide=1,Chroma=True)
Decimate(Mode=2,Quality=3,Cycle=5)

crop(24,73,672,344)

BicubicResize(512,288,0,0.5)

and got a very nice quality using DivX 5.0 with 800 kbps (700 MB avi with 1:43 of video)

Now, I edited the script a bit, changed the bicubic resize to 0.75 (sharp bicubip), and added the following in Virtual Dub:

Saturation x 120% (I wanted more vivid colors, some of them were washed out)

and Sharpen (by 32) which is right in the middle.

And now was using the DivX 5.02 PRO codec (I removed the GAINware)

And I noticed a HORRIBLE amount of quality loss on the final product.

Can anyone point me to where the quality loss might have resulted from? (BTW, viewing it frame by frame for the output in VDub didn't show the quality loss I got writing the .avi file, plus I didn't get the sharpness and Saturation I wanted, too) Is this partially because of the script file loaded into VDub and the order in which things get done? There, again, lies my first question, are there script functions I can call do to this? :D

[neuron2]

If it looked fine previewing in VirtualDub but not in your DivX encoded video, then you need to look at your encoding parameters first. Did you use a sufficient bit rate for example? If the encoding is all OK, your DirectShow filter (or ffdshow, etc., if installed) may be ruining things on playback as well.

@ArdenDag

You cannot access VirtualDub internal filters through Avisynth scripts, but there are equivalent (and often better) native Avisynth filters.

[ArdenDag]

Thanks.

I went to the avisynth site and downloaded the 2 filters I needed most (sharpening and saturating).

About the video quality on my encode. I encoded both at the same bitrate, the only changes I made I listed (changing Smooth Bicubic resizing to Sharp Bicubic resizing, and adding the internal VDub filtering for Sharpness and Saturation)

Is there a better order for when to sharpen/saturate an encode? I heard that certain orders cause for a better encode quality. Any help is appreciated :)

[ArdenDag]

Well, I played around a bit, and did a bunch of testing on partial AVI files. Here are my results:

I tested putting the sharpening before and after resizing and also before IVTCing (bad idea, lol), the filter would sharpen the interlaced/telecined fields before they were IVTCed ;)...

Anyway, I played around with different settings, different qualities with the different resize filters, seems the best I could do was:

Using the sharp bicubic resizing
Using MSharpen at 100 strength with a threshold of 15 (instead of 10)
Using 1.4x saturation

Provided me with decent results on my encode (first with 100% quality, then with 800kbps bitrate encode)

If I had the patience, I'd upload a few to ask for comparisons, but I'm lazy, and I doubt anyone would really bother to compare short AVI files ;)

I'm still curious if there is a better sequence for sharpening,resizing,IVTCing, etc etc...


Oh, I almost forgot... I removed postprocessing on my IVTCed film, it didn't seem to affect the end result the way I thought it would, it actually improved it, is this because I am coming from DVD quality material into an encode? It should also speed up my encode time, correct? Is it better to change some of the parameters in Decimate() and Telecide() instead of changing the postprocessing to false?

lots of questions from a n00blet, sorry if I'm being troublesome :)

[manono]

Hi Arden Dag-

If you have one of the newer AviSynths installed, then you can use the Tweak Filter for your saturation. It's built into AviSynth.

Before going to an outside sharpener, I think I'd just try a sharper resizer first. My first choice would be LanczosResize. But remember that a sharper resizer hurts compressibility. And MSharpen and the Sharpen Filter hurt it even more (probably causing the quality loss you mentioned earlier). Not that it's not worth using in certain circumstances, but it may raise your average quant enough to be noticeable (when doing the encodes for the same file size). But you might compare the average quant of your various tests. You can get it from inside the analyze.log which gets created at the beginning of the second pass (DivX5).

Don't put any of the filters in before the IVTC process or it can get seriously screwed up.

If you remove the Decomb Post Processing (the FieldDeinterlace component of Telecide), then, yes, the encoding speeds up. It might be risky to remove the post processing though, unless you're pretty sure no interlaced frames have slipped by Telecide.

[ArdenDag]

If you have one of the newer AviSynths installed, then you can use the Tweak Filter for your saturation. It's built into AviSynth.

That's the filter I'm using, I downloaded it from AviSynth's website, but I'm using VirtualDub to process my scripts, so I just load it manually in my .avs file.

Before going to an outside sharpener, I think I'd just try a sharper resizer first. My first choice would be LanczosResize.

I was doing research on the bicubic resize filter, it took me a while, but I found the link to the parameters on AviSynth's website, and though it was informative, it also confused me :)

I read there that the LanczosResize filter works better for the 0.5+ bicubic sharpening, so I will attempt this in some tests without sharpening, and then again with very light sharpening effects, and see the differences.

But remember that a sharper resizer hurts compressibility. And MSharpen and the Sharpen Filter hurt it even more (probably causing the quality loss you mentioned earlier). Not that it's not worth using in certain circumstances, but it may raise your average quant enough to be noticeable (when doing the encodes for the same file size). But you might compare the average quant of your various tests. You can get it from inside the analyze.log which gets created at the beginning of the second pass (DivX5).

I think I need to start checking that, heh. I was running most of my first tests under 1 pass 100% quality (quant 2). This was just a comparison of whether the sharpening effects were causing noise on my .avi files, often there was too much noise created to make it worth the effect of the sharpening (this was using MSharpen). I'll see if running my resize under Lanczos will improve the sharpness enough to make it worthwhile removing MSharpen. It might, it might not. It looks like this movie I'm working on it just that poorly made *shrug*. The color balance looks horrible, and I know the initial film wasn't this washed out. I spit on the HK bootleggers! :D

If you remove the Decomb Post Processing (the FieldDeinterlace component of Telecide), then, yes, the encoding speeds up. It might be risky to remove the post processing though, unless you're pretty sure no interlaced frames have slipped by Telecide.

Ok, here's what gets me a little confused. I know that Telecide will adjust for shifts in the telecine pattern, and often it can slip up on the shift, but I've noticed that it has done this (from what I can see) only done it 3 or 4 times in the movie (god I hate these Hong Kong bootleg DVDs... ARGH!) and often it will be a short 'blip' of what looks like interlaced material while watching the movie. Most people would not be offended with this while watching, but to me it's just annoying for it to be there if I can somehow remove it manually. I know there is override functionality in telecide, decimate and fielddeinterlace, so if I find the frames that this happens on, I can tell them to fix it manually. This, although, required quite a bit of work, and a lot of time :) so I have two choices here.

Leave the post processing on, and let Decomb fix all possible problems (and in turn I notice that the quality DOES reduce a bit), or get off my lazy ass and watch the movie, note where the problems occur, and manually fix them before doing my final encode. :)

Sigh, all this could be easily avoided if Disney would just release the movies I want in the US... (Including this, and Laputa)They wouldn't butcher the quality so I wouldn't have to go through this!


manono, you don't know how much I appreciate all the help you've given me since I've started posting here :)

[ArdenDag]

Ok, I have encoded a copy with the following script:

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\decomb.dll")
LoadPlugin("C:\Docume~1\Arden\Desktop\Arden'~1\Videot~1\Virtua~1\Plugins\msharpen.dll")
LoadPlugin("C:\Docume~1\Arden\Desktop\Arden'~1\Videot~1\Virtua~1\Plugins\tweak.dll")

mpeg2source("C:\Documents and Settings\Arden\Desktop\Kiki's Delivery Service.d2v")

Telecide(Guide=1,Chroma=true,post=false)
Decimate(Mode=2,Quality=3,Cycle=5)

crop(24,73,672,344)
AddBorders(0,80,0,80)
LanczosResize(640,480)

MSharpen(strength=100,threshold=20)
Tweak(sat=1.4)

I also added subtitles via TextSub (which is why I added the borders) and the audio is muxed. I wonder if anyone wouldn't mind downloading a clip I've uploaded to tell me if it looks ok?

EDIT: Removed dead link

Thanks in advance :D

[Angrychair]

A couple of things I'd do that would greatly improve your quality are to first off, use a light temporal stabilizer (I like fluxsmooth a lot myself) and move over to the alpha xvid. Chroma motion estimation is a huge boon for those of us that encode anime, and using a color stabilizer to lower motion in areas of the screen that aren't supposed to have any is a good idea.

Good luck to you.

[Suikun]

You shouldn't put the black borders there. A bad thing about them is that some 16*16 macroblocks a halfway into them which doesn't exactly help compression. If you really need them you should put them after resizing and add a multiple of 16 for the upper border.
You should always filter before resizing, as resizing also resizes noise. You should also consider something like a very slight Convolution3d or DctFilter (note that DctFilter has to be the very last filter in your script) which can help compression to some extent.
And, of course, use XviD =).
Happy encoding.

[neuron2]

Originally posted by ArdenDag

Telecide(Guide=1,Chroma=true,post=false)
The chroma option has no bearing when post=false.

[Stereodude]

This is just my uneducated guess, but that bicubic resize takes place in a YUY2 colorspace. Virtualdub's take place in a RGB color space.

I've been doing some HDTV to DVD spec mpeg-2 work. If I do the resize in a YUY2 color space the resulting MPEG-2 looks very subpar when compared to doing the resize in VD. You can convert to RGB in AVIsynth with ConvertToRGB() but the slowdown is very, very bad. Doing a bicubic resize in VD is about 4x as fast as a converting to RGB and then resizing in AVI synth.

ConvertToRGB()
BicubicResize(720,480)
= slow

BicubicResize(720,480)
= looks bad

I think your best bet is keep resizing in VD unless there's some reason why you don't want to use VD at all.

Stereodude