Best filter for compression errors in cartoon

[Slogra]

I want to encode Beauty and the Beast SE R2. Unfortunately this DVD has a bit low bitrate because 3 different versions of the movies are put on one DVD (without seamless breaching). So there are some compression artifacs at the edges of the drawings in almost very scene.

Which filter should i use? And how do i use a filter in Gordian Knot?

I never use any 'external filters' in Gknot, so i don't know how to use them :( . I guess i have to edit the AVS file before encoding, load the plugin in it and set the parameter for the plugin. The last part is what i don't understand, how do i know which parameter i can use?

[bkam]

To the moderators, I know this question has been sort of covered before, but as a fellow newbie it took me quite a while to piece together enough information to start using filters by using search (I specifically wanted to try convolution3d) so I thought I would post him specific instructions.

I'm sort of a newbie too so I'm not sure how much I can help you, but for noisy sources I use convolution3d. This is a filter by vlad that does both spatial and temporal smoothing, which should removie noise and increase compressibility, and just makes the final avi look better in most cases to me. You can try it yourself and decide whether it helps in this case. You can download it from the avisynth site (http://www.avisynth.org/index.php?page=Section+3%3A+Filters+and+colorspaces#q28), and put the file convolution3d.dll in your gordianknot folder. Then you have to edit your avs file in GordianKnot in order to use c3d (or any other filter). I'll use c3d as an example of how to set up filters... First make sure you have your d2v file set up and your resolution, etc, the way you want it. Then go to the window where you can see the results of your cropping, etc, the one where you normally hit Save & Encode to start the process. Hit Save & Encode, and select the resize filter you want, and any of the other options in there, but instead of clicking Save & Encode click just Save. This will bring up the screen that lets you save your avs file, but it won't bring up the DivX codec options. If you want to do your credits separately the way GordianKnot does if you just click "Both" (at 20% quality with max quantizers or whatever--just read low quality), then you have to have set your credits frame, then save two different avs files by clicking "Movie Only" then saving (I save movie.avs), then opening the Save & Encode window and clicking "Credits Only" then saving (movie_Credits.avs).

I hope I explained that well enough, please ask if I didn't.

Next comes editing the avs files. Go to the directory where you saved them, and open the main movie avs file. At the bottom of the LoadPlugins section, add a line for your filter. I usually just copy the "SimpleResize.dll" line then type in whatever filter, in this case LoadPlugin("C:\PROGRA~1\GORDIA~1\Convolution3d.dll")

I then go down and find the resizing line, there is a heading that says # RESIZING and under that a line that says either BilinearResize(parameters) or BicubicResize(parameters) or SimpleResize(parameters). Now you have to decide whether you want to apply c3d before or after resizing. There is a whole thread (http://forum.doom9.org/showthread.php?s=&threadid=36344&highlight=convolution3d+resizing) on this topic but from what I gather, you should put your c3d line before resizing if you want quality, but it will take longer. You should put it after if you want compressibility, and it will take less time. I think in your case you're trying to improve the quality, so you would want to put it before.

So here is the actual line for the c3d. I always use one of the two following lines (I know, I am a total newbie and c3d is much more powerful than this):

Convolution3d(preset="movieHQ")
Convolution3d(preset="movieLQ")

because I only encode movies. There are also presets for anime though that might work better for you:

Convolution3d(preset="animeHQ")
Convolution3d(preset="animeLQ")
Convolution3d(preset="animeBQ")

By the way, HQ is High Quality, LQ is Low Quality, and BQ is Bad Quality from what I can tell. You can read all about the parameters of c3d here (http://hellninjacommando.com/con3d/readme.html), but I couldn't find anywhere that explained the seven parameters enough to screw with them.

Once you have added this line, save your avs, then go into GordianKnot onto the Encoder page. You have to set it up yourself, so click Add Job. It has three "select" buttons that let you choose your movie avs, your credits avs, and the output avi file you want (this should be fairly straightforward). After this just set up the rest of the options however you want. From what I can gather, B-Frames are good, GMC might be ok but might screw up panning shots, and QPel just sucks. Psychovisuals is up to you, they do increase compressibility if you are worried about that. Then set up your audio and add the job as you normally would.

One thing is that convolution3d makes your movies more compressible, so if you want to run checks, it's sort of tough in GordianKnot (although it can be done, I don't like the way it works out). I prefer using jonny's DivX 5 Enc (http://jonny.leffe.dnsalias.com/) to do my compressibility tests, since I can make multiple avs if I want to check different filters' effects on my compressibility, or I can easily change DivX options from there and test those too. Note that it seems like a really good idea to leave that program's setting on Slowest (it's really the codec's setting, I think, but just leave it). I tried the same avs and same divx settings on slowest and fastest and the difference was 71% compressibility for slowest and 38% for fastest (way off!).

Anyway I hope I helped you more than confused you. I took a long time to figure that out, being a newbie as well, so I hope it saves you some time.

bkam

[Slogra]

I've got good and bad news:

The good news is: I got the filter working!
Thx, you really helped me a lot. It's a great summary how to use filters and i've saved your message to a text file :).

The bad news is that i couldn't get rid of the compression errors. Maybe i should've said that the movie is clean of noise, there are only errors due to high compression. I've tried animeHQ, animeLQ and animeBQ, but really nothing helped against these specific errors. I think i might have a real problem here, with no good solution :(.

Anyway, i'm gonna try the filter now on another movie. I'm curious what results i get. And i will try Divx 5 Enc's compressability test.

[hakko504]

Started thinking a little about this and I think you should test MarcFD's version of Mpeg2dec (http://ziquash.chez.tiscali.fr/), especially with setting cpu=6. This is a setting that will try to reduce compression artifacts. While you're at Marc's site also get Chroma Noise Reducer. This is also a quite useful filter for cartoons. I'd say that something like this can give quite OK results:LoadPlugin("mpeg2dec3.dll") #Add path
Mpeg2source("BatB.d2v",cpu=6) #Add path
CNR2() #default values are quite OK
Convolution3D(preset="animeLQ") #also try BQ and HQ.
crop(8,0,-8,0) #Change to whatever you find suitable
BicubicResize(512,384) #Change to whatever you find suitableAnd make sure you use the right versions of the filters, YUY2 for AviSynth 2.0x and YV12 for AviSynth2.5!

[RadicalEd]

for removing compression artifacts SmartSmootherHiQ is still the best AFAIK
http://cultact-server.novi.dk/kpo/avisynth/smooth_hiq_as.html

[Slogra]

I tried all the filters you recommended but i didn't succeed. I guess the compression artifact are just too great to muffle away with a filter :(.

But i will not give up... i'll try some different setting soon

[jonny]

@bkam:

Yes, Fastest...Slowest are divx 5 codec settings

[bkam]

Am I right that slowest is the "recommended" setting? Because I got huge differences in compressibility between slowest and fastest with your program.

BTW thanks for the incredible program! I'm a big fan.

[jonny]

Yes, use always slowest!

I've added the perf/quality options only for those how want to test :)

Strange thing: two of the pref/quality values produce exactly the same result! if i remember correctly slower & medium, anyway you can verify this with a small clip... one of the values is only "cosmetic" :D...

[bkam]

Maybe this has been addressed in other threads but I was trying to get the low-down on pre-processing. When you just use GKnot you don't really run into it, but I see the settings on your program. From what I can gather it's some sort of smoother? But is it temporal, spatial, what are the settings like, etc.? I was wondering if you could clarify this a little for me. Thanks :).

[jonny]

Try this: http://www.divx.com/support/guides/guide.php?gid=1

[bkam]

Thanks for the link, but I'm still confused. I understand this is a different question than the post. This is an official DivX page which also states that QPel is good, and that GMC and Psychovisuals (even strong) get the "green flag" (as it says somewhere on the page) for use in all encoding, which I believe is not the consensus. If the information about Pre Processing Source is correct on this page, then it is basically a combination of spatial and temporal filters for averaging. If this is correct then it's basically c3d, I believe. My question is about the actual quality of this sort of pre-processing compared to avisynth filters (in your opinion or the consensus about it) until I run some tests.

[jonny]

Preprocessing increase compressibility for sure.

About the quality... i don't like to use preprocessing so i can't tell you what is the quality of the divx 5 preprocessing, probably doing some search you'll find better answers about this.

Anyway the best thing it's to do your experiments.

[hassegubben]

I know the problem very well. I´ve tried to encode the "new" beauty and the beast several times, using several filters. I started out with sharp bicubic(usually my favourite filter)but the filter enhanced the artifacts in the original material. I read a lot in the forums about the problems and finally understood that bilinear is the best way to go with "hard" movies with a lot of artifacts. The result was that slow scenes was in parity with the original regarding quality, but stills from the fast scenes looked shitty(colour artifacts introduced), but this is not a problem since you can´t spot the artifacts when watching the movie. Trying to get rid of the "original" artifacts would only blur the movie in one way or the other. Reproducing the original with as crisp settings as possible was good enough for me. Just my thoughts...

[manono]

bkam-

If this is correct then it's basically c3d

Only in the sense that they're both using a combination of spatial and temporal smoothers. My opinion is that the algorithms used for the Pre-processing in DivX5 are fairly primitive although relatively fast. I'd much rather use C3D or FluxSmooth (which is faster than C3D).

[bkam]

Originally posted by manono
I'd much rather use C3D or FluxSmooth (which is faster than C3D).

Thanks for clarifying, and for mentioning FluxSmooth. I hadn't heard of that so it's something new to research :).

[manono]

Hi-

SansGrip has come on strong lately as an AviSynth filter developer. If you want to try FluxSmooth or any of his other filters, you can get them Here (http://www.jungleweb.net/~sansgrip/avisynth/). There are threads in the AviSynth forum about most of them.