The Blue Aurora Borealis That Would Not Die

[sadie]

@Mp4Guy

In fact, it would nearly seem that it is because your camcorder reproduces less detail that it avoids the tendency to produce banding. Secondly, if I read you right, the compression process that takes place 'invisibly' in the camcorder is hardly a 'standard' process but varies from brand to brand or even model to model. I naively had thought that quality lay mostly with the optics and chip factors. Old school. Moreover, I wonder if the noise issue I have is unique to the camera or is more likely common to all GS400s. Has anyone ever bothered to rank/compare camcorders on the basis of their internal encoding prowess? And more pertinent to the present discussion, is it your opinion then that attempting to do the a la carte treatment on a clip using dedicated fine-tuned programs for capture, avi rendering and mpeg2 compression is probably futile since the damage has already been done, and I'm just as well off settling for the ease of Pinnacle Studio and use a post-processing filter in a pinch at playback? Which is fine since, like theories about the universe, I prefer tech solutions which are simple yet elegant.

@Henryhk

Clearly, your proposal of doing zone treatments is the most concrete workaround to my problem. However, you forget that in a 1-hour film shot mostly outdoors at the height of summer, the blue sky is...er...not an infrequent visitor. I think I estimated at least 30 zones to fully do justice to all the problem areas. To be sure the passing nun is the most difficult of them all, but there are plenty more. So we cannot blame it all on religion. And after all, as you yourself pointed out, the great majority of these anomalies play out only on the computer monitor. Last night did a fully authored Procoder DVD encode and on the TV playback of that scene, the banding was nearly completely gone. Superior even to Chilledout's test. The other good news is that yesterday I found out that my new ISP is set up for delivering h264 content wirelessly from my computer to the TV. Maybe if won't be necessary after all to strangle Chillout until he coughs up his encoding secrets.

I'd like to evoke an idea from my opening blurb. Is there not a tool or technique available that will allow one to operate on a contiguous area define by a magic wand similar to that of Photoshop? For example, I heard of one utililty that can track and eliminate a particle of dust as it moves across the screen. It only works on Mac computers though.

[henryho_hk]

sadie, the blue sky has no problem, the graininess has. Did you push the gain over +6db? Also, the zoom and pan scenes should be much easier than the fading scenes.

[chilledoutuk]

OK sadie had a brainwave basically I believe that your rising gradient in the blue skys problem is caused by you cross-fading blue sky’s together which have a slightly different gradient which basically means you get a raising gradient effect.

I have encoded a segment between fades and it seems to not have any problems but to fully test this theory you need to either try a different transition effect such as a fade out fade in or perhaps a wipe.

If you try encoding I think you will find that there is nothing wrong with the source video.

Here are a couple of clips I encoded from your video but from a part without any fades it seems to support my theory.

4500kbps Xvid encode
http://www.sendspace.com/file/0e88z1

2500kbps H.264 encode Mainconcept
http://www.sendspace.com/file/jscy4n

[sadie]

Well, it looks like I finally decided to bite the bullet. Inspite of your generous time and effort dedicated to this issue, I've concluded that ultimately my problem is a unique blend of a noisy source with an aggravating dissolve in a DV format that developers probably didn't have in mind when they produced xvid. I mean via their DVNR processes and the 'pseudo-interlaced' format of the DVD, Warner Brothers and Dreamworks already made life a heck of a lot easier for most of you and your standard usage of the encoder.

Finally I opted for a two-pass xvid encode of my 1h20min film that gave me a 38% vs DVD file size with an average 2750kb/s bitrate. Larger than my target reduction, but the challenge demanded it. I didn't use zones, since it seems to me xvid ought not need it if it's doing it's job in 2-pass mode. Although slightly inferior to Mpg2 quality on the monitor, when viewed on TV the differences are close to nil. Oddly the award for the largest spike in bitrate, 5000kb/s went to a panned scene of rainfalling over windswept palm trees. As for the wavy curtain, it is not severe, although the deband filter is a must for viewing on the monitor. Henry will be happy to know that I found a new aurora over the city of Hong Kong. It was not blue (nighttime cityscape). I used the area deinterlacer + smartdelace combo instead of the tomsmocomp delacer which I feel smears things out just a bit much.

Which leads me to the following theoretical question: Smoothing aside, in xvid which should give the more optimal color results: my route with a full process in vdubmod, or a tomsmocomp avs fast process in vdubmod? I ask this since the former is a straight shot RGB in/out whereas the latter although taking advantage of not having to undergo vdub's filter conversion dynamic did necessitate an avisynth conversion to YUY2 for tomsmocomp filter to work.

Although this goes beyond the scope of this board, I would love any more input in dealing with source noise factor. Henry, what did you mean by 'pushing gain over 6db'? As far as I know the only gain I've ever pushed is on my 1977 Ibanez. Beyond what I've already surmised about, I wonder if noise could more adequately be dealt with at the pre rendering stage while still in DV. Pinnacle's tool doesn't seem all that sophisticated. By the way, can noise increase with time as the camcorder ages? What about DV tape quality?

Another theoretical question. If I understand x264, it's supposed to allow much greater compression to much lower biterates than xvid or divx. What about moderate to high bitrates. Under normal circumstances and with kbs in the 2500-3000 range would you expect a noticeable improvement of x264 over xvid? In other words, and to open a new can of worms, if my priority no longer is to get file size down as much as possible but to simply get the best quality possible out of a non-HD DV source up to and including DVD bitrate will h264 still have a decided advantage? What I'm driving at is, if x264/h264/avc is the standard of the future destined to supercede DVD, is it in my interest now to use it to produce my final full quality video and prefer it to (if not abandon) mpeg2? Hmm, same question asked three ways. Hopefully one of them is clear.

Last related issue. It's standard fodder about how image quality varies on the television vs the monitor due to those elusive issue re/ color space and more. It's also a given that the classic CRT television is fast becoming a relic and flat progressive lcd, plasma and soon to be laser will wholly dominate the future. Does this mean that the encodes that I am making now, compromised for TV, will became redundant and visual quality will suffer when viewed on these new technologies? I've not yet viewed any of my DV material on anything other than the venerable tube.

As I sense this thread will wind its way down, I want to thank one and all for the useful input and interesting discussion. Hopefully we'll continue to meet up on other threads.

[henryho_hk]

Hong Kong? I have not seen the blue sky here for a few weeks. ~lol~ I think "gain" is the setting that boost the brightness. :P The higher the gain boost, the more the noise. DV cams do produce more noises when the CCD ages, also when it overheats. Aged tapes? They would give block noises (block decode error) but not grainy noises.

Optimal xvid encodes should involve minimal color conversion. Hence, vdubmod fast recompress is the right way. When the bitrate is high (with respect to the frame size), xvid is better than x264 as more details can be retained (the trade-off is obviously more artifacts when the bitrate is insufficient).

[chilledoutuk]

OK for starters xvid does not retain more detail than decent h.264 codecs partly due to the better use of qpel.

Secondly i would like to know why my post has been completely ignored i have found out the reason why your getting the probelms on the blue skys and bassically its down to you using crossfadding.
If you use a different transistion then there will be no problem.

For example how often do you see cross fade transitions of videos with large areas of colour that has a gradient you dont.

you freakishly managed to find something that would confusse the hell out of macroblock based codecs yet your not willing to adapt to this and use a different transistion.

[sadie]

Chilledout, apologies. I probably should have explicitly acknowledged your extra little test and I did notice that the banding effect was somewhat reduced in between the dissolves. But not to rile you or anything, but I'd always thought this aspect of the problem was self-evident.

Back on Oct 10 I said:

This rising curtain happens elsewhere in my video and will at turns exhibit itself as rising or falling depending on if the scene is dissolving in or out. The 'curtain' isn't apparent between straight cuts, but only where there is a transition effect employed.

It was precisely why I posted the clip I did: only six seconds but with a dissolve at either end. This thread was only ever about what mechanism causes the anomaly (noise in flat chroma) and what to do about it. And you provided us with perhaps the best correction to date. I finally didn't go the x264 route because it's still all to new to me and rather inconvenient to use/view.

But I can assure you there is no question of my changing my manner of filming simply to please a possible technical defect of my camera or a limitation associated with an encoder. I said in my opening post that I probably overuse these dissolves. But any reduction of the use of this filter in the future will be based purely on aesthetic principles. Besides which whether I'm 'willing to adapt to this and use a different transition' begs the question since this video is already a done deal. While we're there, I could also adapt by choosing not to film the sky.

Finally, you're absolutely wrong that using another filter will fix the problem. It will only reduce it somewhat. As I mentioned to Henry not long ago, there are some 30 other problem points in the film and the vast majority have nothing to with dissolve contexts.

One loose end, Chilledout, that touches on a Q. that has bugged me for a while. You resized one test to 640x480. Why did you not do it to 640x512 which is the more exact proportional reduction from 720x576. I thought that due to pixel size issues the 4:3 aspect would automatically adjust itself to fill the screen in the player anyway. Or is it a toss-up situation?

So the jury is still out, to judge by the two of you, as to the relative viability of x264/xvid at high bitrates. Feel free to elaborate.

[henryho_hk]

Art vs Tech. ^_^

720x576 --crop--> 704x576 --resize--> 640x480 is the correct way to produce an AVI for viewing as square pixel (i.e., no resize by the playback software) on a computer screen.

[chilledoutuk]

I dont want to sound patronising but when one films and produces you have to take into consideration everything including the fact that your going to use a Lossy compression to deliver your content.

I understand that if you have finished production that you cant go changing the trasistion effect but its something your should keep in mind for the future.

ON another note the problem could be caused by pinacle studios effects have you tried any other editors such as vegas video.

What would be great is if you could supply 2 dv uneditied source clips of sceens with blue skys so that we can try out all our different NL editing applications to see if that has any effect.

henry explained why i used the resolution i did for 640x480 but also because it is a pefect 4:3 aspect and its also a standard VGA resolution.