improving compressibility of clean dvd source prior to encoding with XviD

[Smiff]

first i hope this hasn' been beaten to death in the big threads but i think this is worth a topic of it's own (?). after reading through much of this forum i'm leaning towards two methods (goal is best quality final result after compressing good modern DVD source to 1 CD, time being not too important).

1) MPEG2Dec3.dll, CPU=4 or 6 (any recommendations?)

OR

2) Convolution3d (0, 4, 4, 4, 4, 2.8, 0) - or thereabouts - ?

Other than that it's just a crop & resize, i don't want an over-filtered look. I wonder what happens when you use Deblocking&Dering both on the initial MPEG2 (input, MPEG2Dec3.dll) and the final MPEG4 (playback, FFDshow) - is that bad?

And specifically, this is for Xvid encoding, in case that makes any difference (how the encoder copes with noise etc.)

[Smiff]

and before any gets upset and says "every dvd is different" - i know, i'm asking what are the likely pros and cons of each method above, hopefully from someone with a good technical understanding of the internals of Xvid, and what are people's personal experiences... because i said time to encode isn't important, but there's no way i'd have to time to try every method, with all their parameters, on every source :)

[-h]

Pre-processing will always be good for aiding MPEG-4 compression - I'd say apply as much as you can until just before it affects the image in a subjectively negative way.

A good post-processor (like ffdshow) should not over-smooth the output - if you've filtered your source enough to give mostly quants of 2 and 3 in the output, the post-processor will do very little resmoothing on those frames since there will be few artifacts to remove.

-h

[Smiff]

ok, so you want to be using as much filtering as you can on the input side, so you get a clean result without post-processing, then you can enable PP to clear up remaining artifacts? That's the rough idea I think. Sorry i'm still pretty new to this. So which method do people recommend to pre-process the MPEG2? Is MPEG2DEC3's PP (if you call it PP) more "intelligent" than something like Convolution3d?, i.e. could it have access to more information in the original stream? (again trying to avoid that generic blurred look as far as possible :)).

[yaz]

hi smiff !

Originally posted by Smiff
and before any gets upset and says "every dvd is different" - i know, i'm asking what are the likely pros and cons of each method above, hopefully from someone with a good technical understanding of the internals of Xvid, and what are people's personal experiences...

hmmm ... i don't wanna be upset but ... :-) like or not, every source is (more or less) different. don't expect any 'general/the_only_one/final/ultimate/...' solution. imho, that solutions can generate more problems than solve.
u can collect 'personal experience' quite easily from the threads all over here & u'll know pros&cons. i don't like to say it but use 'search' ! both mpeg2dec3pp & conv3d has its own thread, lotsa people piled lotsa things there. read it, it worth.

Originally posted by Smiff
... because i said time to encode isn't important, but there's no way i'd have to time to try every method, with all their parameters, on every source :)

that's what's the hardest to save; time. yes, u must play with filters, settings, mods, ... so as to find that unique solution best for that clip/film u work with. that's the way it goes.

about your question. personally, i'm hardly against any l'art pour l'art filtering. on one hand, before u use any tool u must know what's it for & what u loose/gain when using it. all filters are designed&tuned for/against some specific 'noise', so, if u haven't that why to use ? all filters available are thoroughly tested & discussed. here or in other threads/forums. use 'search' (sorry:-)
on the other, some people seem to forget that noise is(ok, can be) a relevant part of the source. getting rid of it means (can mean) loosing information i.e. worsening the quality. instead of some noise u have clear picture wout detailes/shades/... after all, it's up to your preference what u choose&use.
if u prefer 1cd shots (like me:-) wout any experience, just follow the guides. if u have problems afterwards (i bet, u'll have :-)) u can look for solutions for that spec problem.

y

ps once we all were beginners & some remain that forever :-)

[Smiff]

yaz, hi, fair enough :)

I have read a fair bit of discussion about parameters for each filter, what i haven't seen much of is comparison of different filters, and for real film not anime and all that weird stuff that geeks are into ;)

At the risk of going slightly warezy for a second, has anyone here seen the recent Austin Powers 3 release by DVL? (dvl-ap3.avi) It's made with Xvid (btw, anyone wish there was a tag to show which build of Xvid and/or settings was used?) and has an amazingly smooth appearance... very watchable on a TV with no PP (they've been following -h's advice above! ;)). I don't know how to contact these groups (and not sure i'd want to :p and certainly not sure they'd want to discuss it lol) so my only hope is that someone here can look at a clip and say "hey, that was done like this" heh :)

Anyway i just thought there must be loads of people who have read the threads for Convolution3D and MPEG2DEC3 and many others and settled on one in particular... i'd love to hear which you've chosen (neither? both? comeon don't be shy with your lee7 settingz ;)) :) Because the truth is, i'm sure, that people find a method they like and apply it to many sessions.. i just want a reasonably good compromise that works for many films.. hopefully by the time i get there a stable Xvid will be out ;)).

[duartix]

Well, first of all what I would like would be a point (could be a post or a sticky) where we could gather all these tools. Yesterday I spent over 2 hours between forums, Avisynth and VDub pages, Avisynth FAQs, etc. only to find a few of them.

I believe it would help so many people to know where we can find:
[list=1]
that special version of mpegdec with the luma option (MPEG2DEC3?)
Convolution3D filter
SpatialSoftenMMX filter
Cnr2 filter
MAM (Motion Adaptive Mixer) filter
...
[/list=1]

[Smiff]

MPEG2DEC3:
No homepage (?). Download from this thread:
http://forum.doom9.org/showthread.php?s=&threadid=34253&perpage=20&pagenumber=6
URL for latest version at time of writing (beta6):
http://forum.doom9.org/attachment.php?s=&postid=194418

Convolution3D has a homepage:
http://hellninjacommando.com/con3d/
Interesting thread about the background to Convolution3d: http://forum.doom9.org/showthread.php?s=&threadid=29829

MAM - no homepage (?), here's a link to v0.6 spe3, the latest (?):
http://forum.doom9.org/attachment.php?s=&postid=194670

SpatialSoftenMMX - AviSynth FAQ points to http://users.win.be/dividee/ but not there... SpatialSoften is built into AviSynth, is this same?

[iago]

In addition to above:

Cnr2 -> http://ziquash.chez.tiscali.fr/ (Marc FD's site)

(AFAIK, Marc doesn't host "MAM" on his website any more, and "MPEG2Dec3 beta6" by Marc currently doesn't have a homepage, but I just requested from Marc in the related avisynth thread to make it accessible on his web page. ;))

regards,
iago

[Smiff]

i can see there's a fair interest in this topic by the number of reads... question now would be: which of these filters are now obselete (can be replaced by filters that most people would agree do the same job better)?

[Smiff]

Oops seems we missed a good candidate - SmoothHiQ (attached, v1.01) At first glance seems very similar to Convolution3D - ?)

[Wilbert]

@Smiff,

These can all be found in the faq (yesterday I added a link to the newest MAM). Also a thread linking to MPEG2DEC3 can be found in the faq.

SpatialSoftenMMX - AviSynth FAQ points to http://users.win.be/dividee/ but not there... SpatialSoften is built into AviSynth, is this same?

No it is not the same. And yes it can be found on dividee's homepage. I quote (on 18/08):

"The radius parameter in SpatialSoftenMMX 0.9 by Mathias Born wasn't working. A radius of 2 was always used. He probably forgot to modify one line before releasing it , i don't think it was done on purpose. Download (*) a fixed version."

(*) --> link to download version

[Acaila]

Wouldn't it be better to have Avisynth contain the fixed version instead of the broken one??

[Smiff]

oh yeah missed that, thanks. was providing list 'cos someone asked. my question: what have people found to work best... 2nd Q: shouldn't some de-noising be built into Xvid? - discussion?. thanks. I can put Q another way - if one of these plugins' methods was to be taken as the "standard" for inclusion in Xvid, right now, which would it be and why? :)

[Wilbert]

Wouldn't it be better to have Avisynth contain the fixed version instead of the broken one??

SpatialSoftenMMX is not contained in AviSynth, it is an external plugin. It comes with GordianKnot.

[Prosper]

As a good starting point, I usually try both
mpeg2dec3 (,cpu=4)
then resize().crop()
then Convolution3d (0, 4, 4, 4, 4, 2, 0).

Usually ends up making the source 30-50% more compressable, with negligable effect on quality. But for each source, it is best to experiment.

[Emp3r0r]

resize then crop?

for clean dvd sources I've found that weak conv3D, lanczos, and h263 quant type works well.

Some obvious things (I use) to increase compression from most ---> least.

Resize Method (Bilinear ---> Lanczos)
Quant Type (h263 ---> mpeg)
Resolution (low ---> high)
B-frames (none ---> 5)
Convolution3D (1,32,32,32,32,28 ---> 0,1,1,1,1,1.5)

[Smiff]

i think everyone agrees you crop first - but:

Resize then Convolution3D? or Covolution3D then resize? I was first using the latter, because i thought the image would be less distorted, then i changed to putting the Convolution3D last because i thought it would increase compressibility. Anyone got any thoughts on this? Also which resize method - i'm currently using Lanczos, but it might not be best suited here. Emp3r0r you think bilinear is good in this situation?


My current script looks like this:

pre-process source (CPU=4, I think 6 (dering) is too much)
crop black borders
resize (Lanczos) (keeping h. res to 576 or 608)
Convolution3d (0, 4, 6, 4, 6, 2.8, 0) (experimenting there)

Also i'm staying right away from b-frames atm, they seem too "beta", but i am using q-pel - this wise?

[iago]

... i'm hardly against any l'art pour l'art filtering. on one hand, before u use any tool u must know what's it for & what u loose/gain when using it. all filters are designed&tuned for/against some specific 'noise', so, if u haven't that why to use ? ... (yaz)

@everyone (which means not especially aimed towards anybody ;))

I just wanted to say that I completely agree with yaz. Imho, it is really not so difficult to ruin an encode with excessive/unnecessary filtering, though it may come with a big compressibility increase and even reach an average quantizer of ~ 2 ;). The aim of any filtering process is to "increase" the perceptional quality of an encode, not to "decrease" it for the sake of "only" lowering quants. (Believe me you can produce really crappy encodes even with (average) quantizer of (~) 2 as a result of unnecessary filtering.)

best regards,
iago

edit: For instance, even a setting of "only" cpu=4 with Marc's MPEG2Dec3.dll (with no other filters) might sometimes be excessive/unnecessary for certain sources.

[Smiff]

I agree with the philisophy of that, however, with no filtering, even a short movie at a low resolution gives me a result that's terrible, full of artifacts if you watch w/o postprocessing.

This is why i'm thinking Xvid would really benefit from its own more intelligent routine for discarding detail. At least Convolution3D etc gives you some control over how the loss of detail happens (vs no filtering at all)... maybe i'm just setting up Xvid totally wrong (using mostly default settings). :p but i doubt it cos i'm really gettting nice results now, almost comparable with release groups, if i say so myself :) I mean this is partly that my standards have gone up a lot since i started fiddling with this stuff, and since Xvid got so good ;). But i still think a pro will come along, look at my encodes and go "hah. he's done [insert stupid n00b mistake here, like getting filter order wrong]."

[Acaila]

My current script looks like this:

pre-process source (CPU=4, I think 6 (dering) is too much)
crop black borders
resize (Lanczos) (keeping h. res to 576 or 608)
Convolution3d (0, 4, 6, 4, 6, 2.8, 0) (experimenting there)
If this is for what you call "a clean dvd source" then you're definately overfiltering.
Pre-processing is only useful (IMHO) for removing compression artifacts when you're re-encoding an already encoded movie. Otherwise it removes way too much detail.
Convolution3D is a great filter to use, but like any smoother using it before resizing keeps more detail. Personally I wouldn't use the spatial part but only the temporal smoothing, because like you said it's already clean. Using spatial smoothing on a clean source is the best way to remove detail, and I know you don't want that to happen...

[Smiff]

yeah i think i am too.

So you'd use something like

Convolution3d (0, 0, 0, 4, 4, 2.8, 0)

I could try

Convolution3d (0, 4, 2, 4, 4, 2.8, 0)

(low Spacial Chroma Threshold). I had the Chroma and Luma values backwards before (!)). You want to be using less chroma than luma smoothing to avoid obvious filtered look right? Thanks Acaila.

Anyone had any luck substituting Convolution3D for SmootherHiQ?

[Acaila]

You want to be using less chroma than luma smoothing to avoid obvious filtered look right?No you had it right the first time.

Filtering luma is much more visible than chroma, however filtering luma also has a much greater effect on compressibility (because chroma is already subsampled). So you'll want to have higher chroma thresholds than luma thresholds.

Convolution3D can filter both spatially and temporally, SmoothHiQ can only filter spatially. SmoothHiQ uses an edge detection algorithm so it's quite good at preserving detail while still smoothing spatially. I'm not sure if Convolution3D also has edge detection.

[iago]

SmoothHiQ uses an edge detection algorithm so it's quite good at preserving detail while still smoothing spatially. (Acaila)

SmoothHiQ really seems to be a successful filter in keeping detail, though I have used it only for limited times and only in my "re-encodes" of some DivX and XviD rips (mostly to fit 2CD rips onto 1CD).


@Acaila

What SmoothHiQ settings do/did you use for your normal movie (DVD source) rips?

regards,
iago

[Acaila]

I'm usually not much of a fan of spatial smoothing, so for the average clean DVD source I don't it at all. Only a very weak TemporalSoften, usually (1,3,0), to get rid of the DVD grain noise.

Although my favorite spatial filter is SmoothHiQ, I only use it on Anime. And then only when the compression is getting high enough that blocks start appearing. I usually start with temporal filtering and only use spatial filtering as a last resort (both because spatial filtering hurts detail sooner and because it hardly increases compressability compared to temporal filtering).

The settings I use are mostly either SmoothHiQ(3,5,15,192,3) for light filtering, and SmoothHiQ(5,15,25,192,3) for heavy filtering.

[Smiff]

hmm yeah i guess noise/"grain" is really what we want to get rid of... i'll try that TemporalSmoother(1,3,0) - that's (int radius, int luma_threshold, int chroma_threshold), so you're doing no chroma filtering, why please? And is this a better choice than Convolution3D with the spatial stuff set to zero (or doesn't it matter)?

[Acaila]

so you're doing no chroma filtering, why please?Don't filter unless you have to. To get rid of the grain luma needs to be filtered. Although filtering chroma will probably not be visible, it's not something that needs to be done to get a better picture. Filtering chroma up to temporalsoften(1,0,10) will give you only a 1-2% benefit to compressibility: not worth it for something that's not needed.
Btw, the (1,3,0) was in combination with the H.263 matrix, if you're using the MPEG matrix you'll want to use something a little bit stronger like (1,5,0) because MPEG adds noise compared to H.263.

And is this a better choice than Convolution3D with the spatial stuff set to zero (or doesn't it matter)?I've never compared them like that, besides I can't tell you what looks better to you, only what I prefer :). But why use a filtered designed for two things and you'll only be using one of those things, when you can use a filter that was specifically made for the one thing you want to do?

[Smiff]

hmmm interesting Convolution3D v1.00 is out (attached) and i see the "movieHQ" preset is Convolution3D (0, 3, 4, 3, 4, 2.8, 0), rather more than Acaila reckons but looks alright to me... different people have different tolerances.. i will try with just TemporalSoften first, see how that goes.

Acaila thanks a lot for the information above, that's exactly the kind of technical detail i was looking for :)

Edit: Get Conv3D from homepage: http://hellninjacommando.com/con3d/

[Smiff]

ooops lost a night's encoding due to using "insta-build" (or whatever its called) version... normal service will resume shortly ;)

so current script is something like:

pre-process source (MPEG2DEC3, CPU=4)
crop borders
some denoise process, TemporalSoften(1,5,0) for now, could also be Conv3D, this is the tricky bit
resize (lanczos)
then encode

in that order.

Any advantage to using H263 for both passes rather than Modulated?

[Smiff]

ok, i found TemporalSoften to be not very effective, and the result of that encode to be "messy" and not so pleasing to the eye. I have found Convolution3D to be far more effective than anything else I've tried so far.. i'm thinking my "clean dvd source" is really far from reference quality DVD mastering, like i would think most films except a rare few very expensive ones where great care was taken (a la Moulin Rouge).

Actually, my current view is that you a better off _slightly_ over-smoothing, so as to get a very nice XviD encode, and then if you like adding some noise with ffdshow on playback... this may well go against what a "purist" would like, but, the result really is IMHO, great. I am just going on a subjective "sit in front of the tv and enjoy the film", not some technical analysis or comparison. And after all, we're trying to compress a 5GB+ DVD (already hugely compressed!) down to 700MB (!), so we can't expect too much. Also in case it matters, i'm mostly interested in how things look played back on a TV, not a computer montitor.

So my next attempt is going to be a light Convolution3D, probably the "movieHQ" preset, which i expect to produce a very pleasing result :) (even if it does double the encode time). The only thing to decide now is whether to apply it before or after the resize, and what kind of resizing to use.

[iago]

@Smiff

LOL :D

Well, actually I'm kinda tired of writing about convolution3d to many different threads in both XviD and Avisynth forums for days, but...

* Use convolution3d "before" resize to preserve more details, as vlad also suggests.

* Use at least BicubicResize(width,height,0,0.5) or LanczosResize(width,height) imho to keep sharpness and compensate for the possible softening effect.

regards,
iago

[Smiff]

iago, yes thank you, i think i've nearly found the "best" result i'm going to get, and then will leave the good people of Doom9 in peace :)

At least hopefully this thread will be useful to other newbies who might have been using Xvid straight. Personally i think i've (with everyone's help) pretty much solved the mystery of why movies downloaded mostly looked better than my own encodes. Which is nice.