Here's a way to get the bitrate down

[Richard Iredale]

I've had some experience with CCE 262, and think it's a terrific encoder for our miniDV camera video. I have always wondered, however, why I have to struggle to get the Q down to a reasonable value even when running at a fairly high bitrate. I have gradually come to the conclusion that CCE is forced to work overtime trying to encode the video noise that is present in our DV footage. By contrast, the Hollywood 35mm film frames are virtually noise-free, and I've suspected this is the reason why a Hollywood DVD can get away with a 4-6Mb/sec bitrate.

Today I had some time to kill, and played around with VirtualDub and some of the filters that others have written for that remarkable program. In particular, I experimented with the "Smart Smoother" filter. The basic idea is to blend noise pixels together while somehow not softening actual image detail.

In one experiment, I took 10 seconds of an indoor choir shot (ambient lighting, fairly grainy) and ran it through the VirtualDub filter with the parameters set in the middle ranges. I then created another copy, this time with the parameters set aggressively. I joined the three segments together (original, moderate tweaking, aggressive tweaking) and first viewed them and then ran the string through CCE 2.62, using VBR encoding.

The results: From a visual perspective, the first and second clips looked virtually the same from a "detail" standpoint, and the background noise looked slightly different, but still evident. The third clip clearly had a sort of "smearing" effect over the background noise, and although the detail was still pretty much there, the video had a mild sort of "colorized" effect.

But what a difference in the bitrate and Q curves! Opening the "Advanced" window in CCE I was able to see that there was a large difference in bitrates, and the greatest difference was between the first and second clips! The first clip section ran about 8Mb/sec, the second was 6Mb/sec, and the third about 5.7Mb/sec. Q was about 8, 7, and 7.

In other words, although the "Smart Smoother" seemed to have very little visual effect on the noise level (and no effect on the sharpness), it was responsible for reducing the bitrate 25%.

Admittedly, this was only a brief test, but the results were a shock to me. For those of you working with re-encoding Hollywood DVDs, there is little advantage with this VirtualDub step, since your source images are already clean. But for those of you working with DV footage, this might be a way to greatly increase compression efficiency.

CCE also includes a filter function that can be adjusted. I do not know how sophisticated this filter is. One can certainly do a simple spatial filtering and reduce the bitrate, but of course sharpness is lost in the bargain.

There are many "noise reduction" filters written for VirtualDub. I simply chose one that looked promising. Others may create an even greater impact...

[auenf]

ive always found that DV footage can be rather grainy, which maybe the noise you're refering too, also interlaced content doesnt encode as well as Progressive (FILM based) content.

Enf...

[Arky]

With the limitations on lens size for compact miniDV cameras, there is a subsequent limitation on the amount of light available to the CCD (charge coupled device). A byproduct of this relative lack of illumination is that the CCD and it's associated electronics generate more electronic noise. This can be simply demonstrated by using your camera in dark conditions and seeing the dramatic increase in 'mosquito noise' as compared to footage shot in bright sunlight. therefore, it is no surprise that footage imported to one's MAC or PC, and viewed on its high resolution monitor, appears to be grainy, and presents MPEG encoders with a certain degree of difficulty, given that the very efficiency of MPEG encoding relies on the discardability of a certain portion of the image, provided that that portion remains constant from one frame to the next.

I think, therefore, that Richard's findings are not surprising, in principle, but in magnitude, are rather an eye-opener (no pun intended).

The relative noisiness of miniDV footage has also encouraged me, on some occasions, to use ProCoder instead, since it has an integrated Median Blur filter which works ok on very low settings.

Perhaps, in future, I should make more of an effort to filter my DV footage with AVIsynth prior to it reaching CCE!

Incidentally, the above represents a very good endorsement of the merits of Canon's XL-1 camcorder, given that it has a significantly larger (Flourite) lens than any consumer level models.


Arky ;o)

[dirk67]

Hi,

try the 2D-Cleaner optimized in VirtualDub. It is similar to the Smart Smoother while being MUCH faster, exspecially on Athlons. I use it for all my VHS captures.

Greetings,
Dirk

[bb]

As far as I can see there's nothing new in this thread. It's been well-known since I don't know how many years that smoothing filters can greatly improve compressibility.

But I don't want to my post to be destructive only. So here's something for you to try: Convolution3D is known to increase compressibility very well.

bb

[DDogg]

Guys, I swear to you. Use CCE for everything else but use ProCorder for interlaced DV. I spent two damn years doing tests like Richard Iredale so that I could do a quality job of encoding DV at SVCD rates. I beat my head on the desk in frustration many times and finally gave up.

I happened upon a chance to use ProCorder one day and said "what the hell" I'll try it again. Did nothing but load and select SVCD as output and pressed the encode button. Boom! Best DV on SVCD rate encode I ever saw.

I don't know if the magic is actually in the encoder or if they have designed some specialized filtering and such. All I know is nothing can touch it for doing DV, and only for proprieties sake do I add a IMO.

BTW, I am not trying to preach the gospel here. Rather, I am just hoping to save somebody the frustration and hundreds of hours of testing that I went through.

[RB]

Procoder supports "field based" encoding, that is, unlike CCE which encodes always whole (interlaced) frames, it can encode each field of the interlaced source separately, creating an MPEG2 stream of "field" picture type. This is a spec of MPEG2 but especially some low-cost DVD players cannot play it correctly.

[DDogg]

I would like to learn more about this. Any specific links or just Canopus Website?

Maybe this is why it is so much better when doing interlace but about the same or less quality when doing progressive. I always wondered how that could be.

As for as DVD players I have had these disk run on dozens of brands with never a problem.

[RB]

Look here (http://www.dvd-svcd-forum.de/phorum/ikonboard.cgi?;act=ST;f=28;t=9158). The thread starts out in English but turns into German later, sorry. I think that Forum is where the German "ProCoder experts" hang out, I'm sure they'll be happy to reply to your questions in English.

[DDogg]

Thanks for the link RB.

From what I can gather, the field based encoding method is an option that has to be turned on in the advanced menu. I don't use it so compatibility with various players it is not an issue for me.

[WarpEnterprises]

Even the quality you get WITHOUT field encoding is outstanding, but only in the very slow MasterQuality mode.
Procoder achieves it by dynamically changing the encoding parameters (GOP type, quant scale, scan type,...) so you can't even use BitRateViewer to compare against e.g. CCE.
AFAIK there is no explicit filtering, but of course by the adapting quant method the picture gets softer if bitrate is getting short, but there won't be any blocks soon.

At least in the 1.035 version the fadings are the weak point.
As is discussed thereover you should not use field encoding if you want to stay compatible.

[nsdn]

I found this thread really interesting, since the only way I have ever been able to get a good SVCD from DV is by deinterlacing and using the temporal smoother.

Now I want to convert DV to DVD and I have tryed TMPGE at the max VBR (2000min, 7000 Ave 9000 max) and I can still see some artifacts (PAL). Its really irritating because I thought once I bought a dvd burner all my problems would go away :(

Is it worth trying procoder for DV to DVD transcoding?

is there any recommendations on temporal smoother settings for avisynth to product a good looking DV to DVD?? I assume the MPEG 2 stream from DVD2SVCD will be DVD complient if you do the appropriate things...

[WarpEnterprises]

I don't know how experienced you are, but are you really sure you made the rest of the TMPG settings properly?
(GOP, interlace flag, NO resizing in TMPG)

[nsdn]

I must admit I just used the standard TMPGE template for DVD, just increased the bitrates to the max.

I found this site http://dvd-hq.info/Compression.html which has a guide for setting up GOP and I frames etc...I will have a fiddle with that tonight. Although I have a engineering background...I just don't have time to learn the necessary stuff (got 4 kids :) I was kinda hoping for a quick solution...but ...yeah I know now i was dreaming..hehe

I have since tried procoder and it seems to produce a excellant result (can only compare on monitor), except I can't find any authoring program that will burn the stream (s) for me (they either want to re encode or do not support my NEC drive) Spuceup does not seem to like my burner...nero does not like the stream (s). Procoder either produces elementary separate WAV audio and M2V stream or program MP2 stream. Nero does not like the M2P stream even after I rename it to MPG let alone separate audio/video files :(

[ronnylov]

Originally posted by nsdn
I must admit I just used the standard TMPGE template for DVD, just increased the bitrates to the max.

I found this site http://dvd-hq.info/Compression.html which has a guide for setting up GOP and I frames etc...I will have a fiddle with that tonight. Although I have a engineering background...I just don't have time to learn the necessary stuff (got 4 kids :) I was kinda hoping for a quick solution...but ...yeah I know now i was dreaming..hehe

I have since tried procoder and it seems to produce a excellant result (can only compare on monitor), except I can't find any authoring program that will burn the stream (s) for me (they either want to re encode or do not support my NEC drive) Spuceup does not seem to like my burner...nero does not like the stream (s). Procoder either produces elementary separate WAV audio and M2V stream or program MP2 stream. Nero does not like the M2P stream even after I rename it to MPG let alone separate audio/video files :(

Just rename *.m2v to *.mpv and then you can import it into spruceup. Create titleset in spruceup and burn the file structure to DVD with Nero. You don't need to do the actual burning with SpruceUp.

[nsdn]

Thanks for all your help....seems I had a few of errors, Spruce up did not like the disk because nero did not erase it properly...and would not erase the disk without locking up....got tmpge author and that works a treat. Authored the procoder stuff untouched.

Warpdrive, no I was not resizing, and I had bottom field selected, and interlaced video. I changed the GoP settings per that web page, I did everything it said...and although it looks slightly better than it did before, it still is not as good as procoder, although I would say procoder is slightly softer. Looking at the end result on the TV it is as near as the original cam tape as I am ever going to hope for. I have not given up on TMPGE, because I like using it. Be nice if someone put up a DV template for TMPGE that does not care about bitrate....just about quality.