How does a camcorder produce interlaced DV?

[wayback]

As will shortly be obvious, I'm new here.

I just got a Canon Elura 100 miniDV camcorder (NTSC), and am trying to learn how to produce the best progressive DivX files from the interlaced DV. I've done a lot of reading on deinterlacing, but still don't have a good understanding of exactly what the interlaced DV format really is in the first place.

I assume each line of each field is not captured in sequence as the CRT electron gun would have done back in the day, but what actually does happen? Are the two field captures actually separated in time by 1/60th second? Does the camera take a full capture of the whole sensor for each field, and then toss the lines that aren't part of the particular field? Or does it only capture the lines of the particular field each time? Either way, are all the lines of a field captured at the same instant?

And finally, is there anything peculiar about the nature of NTSC DV that suggests the use of a particular deinterlacer? If it matters, I think I would like to end up with 29.97 frames per second, 640x480, and am trying to figure out how to get there using AviSynth, VirtualDubMod, and either Divx or Xvid.

[Blue_MiSfit]

Well, I'm not an expert on interlacing, but from what I understand it works like this.

The camera sensor takes 60 pictures per second (59.94 actually), but only actually scans half the vertical resolution each time.

AFAIK, there's nothing specific about NTSC DV that suggests using one particular deinterlacing method... I usually get good results with TDeint, Yadif, or MVBob.selecteven.

After you achieve proper 29.97p or 59.94p, you can resize to square pixel 4x3, which would be 720x544 (mod16) for NTSC, assuming you want to interpolate instead of throwing data away.

~MiSfit

[communist]

Are the two field captures actually separated in time by 1/60th second?

Either way, are all the lines of a field captured at the same instant?
Yes Field1 and Field2 are captured at different times separated by 1/60th second in NTSC land. Capturing them at the same time would net you a progressive picture. Check out 100fps.com

Deinterlaced NTSC DV would end up being 720x480. From there you resize it to what you need and also make sure you get a 4:3 aspect ratio. I dont think there is a particular deinterlacer more suited for standard interlaced NTSC than PAL. You will have to evaluate them yourself and make a decision between speed quality and what you prefer. Currently these filters are quite popular:
LeakKernelDeint
TDeint
MCBob
MVBob
Yadif

[wayback]

Thanks for the responses. And wouldn't it be resized to 720x540 with square pixels after deinterlacing? That would be 4:3.

[um3k]

It would most certainly not be resized to 720x540. Here are some proper ways to do it:

1. The traditional method would be to crop eight pixels off each side (resulting in 704x480) and resize to 640x480.
2. If you do not want to scale down the image, you could crop as above and resize to 704x528.
3. If you prefer not to crop, resizing directly to 720x528 will work as well.

All of the options listed here will result in mod16 frame sizes with exact (or nearly so) square pixels. The first one may be performed before or after deinterlacing, as it does not resize the image vertically. Two and three can only be done after deinterlacing, because resizing an interlaced image vertically is a no-no.

My source: http://lipas.uwasa.fi/~f76998/video/conversion/

[stegre]

Does the camera take a full capture of the whole sensor for each field, and then toss the lines that aren't part of the particular field? Or does it only capture the lines of the particular field each time? Either way, are all the lines of a field captured at the same instant?

I spent like a week researching this, a couple of months ago, and the answers are not simple. I started making a whole web page with diagrams (http://www.headbands.com/gspot/misc/dvinterlace/) but never really did get to finish it with any explanatory portion. Anyway, I found this whole subject is much more complicated than I had ever imagined.

> Does the camera take a full capture of the whole sensor for each field, and then toss the lines that aren't part of the particular field?

Yes, this is a 95% accurate description of what they call field integration mode. And as far as I can see, it'd be better if they did exactly what you said (i.e. completely "tossing out" the odd lines during the 1st 60th of a sec and then completely tossing the even ones during the next 60th). But for some reason, in practice, the engineers decided it was better not to "toss" any info. When they are outputting the odd lines, they also average in some of each even line below the odd line in question. The reasons I've seen cited are that averaging two lines helps cancel out "noise", and the fact that they by actually adding two lines they get double the light sensitivity. Regardless of what or how much averaging they do that way, the resulting output goes only to the odd lines of the output device at this time. The averaged output represents a window in time completely contained within this first 60th of a second, and the two lines being averaged were both "exposed" (after being reset) at the same point in time and for the same duration of time. Like regular cameras, DV cams do have an "exposure time", but it cannot logically exceed 1/60th of a second in this mode (not in this mode, anyway - but see below for more about this). Electronically speaking, the data is shifted out during the next 60th of a sec, but that's not relevant to the discussion.

Meanwhile, a 60th of a second later, they "reset" the entire CCD & re-expose it to get the second field of the frame. This, of course, is sent to the even lines of the output device (live monitor or DV tape "file"). But before sending each of these even lines, they average in some of the corresponding odd line below it, just as described above for the odd lines.

Then there's another mode, frame integration mode, which doesn't sacrifice vertical resolution and also gets good light sensitivity, achieving both by using longer exposures. In a sense, it sacrifices "temporal resolution", though as a moving image will blur (the latter may be artistically desirable, but that's another discussion). Anyway, this latter mode surprised me, because I had previously stated in some post that it was "obvious" that the longest exposure time possible an interlaced DVD camera was 1/60th sec. But soon found out I was wrong. Apparently they can "reset" the odd lines and even lines separately. These "resets" are skewed by 1/60 sec. But after resetting, say the odd lines, they can continue letting those odd lines those accumulate light beyond the end of the field time - in theory for as long as a 30th sec total. To put it simply, the tail end of odd lines' exposure overlaps the "head end" of the even lines' exposure. In this mode they never average adjacent lines, or you'd have a mess. Despite the "overlap", the one set of lines is always further along in its exposure cycle than the other, and represents an average of a "window in time" that's 1/60 sec skewed from the other.

I got involved in all this "research" trying to understand a semi-progressive scan technique called "mechanical shutter" (which works amazingly well). To make it as brief as possible , it seems somewhere along the line, someone realized, "hey, if we're in this "frame mode" and the exposure is set to exceed 1/60 sec then there is some period of "overlap". What if we let everything run exactly as described above, but leave the entire CCD in the dark (by literally blocking it with a mechanical shutter) most of the time. But for a brief instant, during an overlap interval, we open and then close the shutter." Repeat this 30 times per second.

Then the odd lines and the even lines represent exactly the same window in time. No electronic or other logic changes to the camera are needed; even more importantly no special "output hardware" is needed. The viewer watching live would see an "interlaced" display - that is first the odd lines fill up screen and then the even lines. But the resulting image has zero interlace artifacts.

Anyway, as mentioned at the top, the diagrams / unfinished web page I started making is here ( http://www.headbands.com/gspot/misc/dvinterlace/). The last post I made concerning them is here ( http://forum.doom9.org/showthread.php?p=998622#post998622), and it all got started because of a question about this really nice progressive looking footage which I've analyzed in this other web page ( http://www.ftyps.com/unrelated/interlace/) I made here. As you'll see on the final page, for obtaining images without interlace artifacts, this mechanical shutter system seems to outperform any software de-interlacer - - because there's really are no artifacts to begin with. Apparently for certain reasons the shutter system does not technically qualify as true "progressive scan", but it sure comes close to it in certain regards.

Disclaimer: All of the above is my understanding of this entire matter after several days of research - but I am by no means an expert. In fact, I knew almost nothing about the entire DV interlace subject before I started that research. So please, if anyone's reading this and there are any misstatements, misconceptions, or outright falsehoods - be sure to let me know...

[wayback]

Thanks very much, stegre.

It seems the Frame Integration Mode would also greatly improve low-light performance because the maximum exposure time is doubled, perhaps at the price of some smearing of rapidly moving objects.

And I guess in theory the camera could switch into that mode as the light level drops.

Is there a way to know which method my Canon uses? I would assume it's field integration only. Does that sound likely?

Well, I guess the question then becomes whether this averaging of adjacent lines has any implications for how a deinterlacer should work for this material. Is there an existing deinterlacer that should work better for this than others? From what I see here, it appears that's not the case.

And by the way, could the mechanical shutter instead be a liquid crystal layer that's switched between opaque and transparent?

[zambelli]

Disclaimer: All of the above is my understanding of this entire matter after several days of research - but I am by no means an expert. In fact, I knew almost nothing about the entire DV interlace subject before I started that research. So please, if anyone's reading this and there are any misstatements, misconceptions, or outright falsehoods - be sure to let me know...
:goodpost:
Nice work! This is an excellent explanation, one that even I could wrap my head around on the first read. :)

[Mtz]

At that time I asked: "how to encode the progressive recording of HC96? As interlaced or progressive?" Because Gspot reported as interlaced and AutoGK analyze reported progressive.
Thanks Steve for your work. I hope someday to finish it with your conclusions and something in Gspot to report this type of "progressive".

enjoy,
Mtz

[Chainmax]

I'd say McBob+SelectEven for maximum quality (extremely slow though) or TDeint+TMM(+EEDI2) for very good quality at a reasonable speed.

[zambelli]

Stegre, what do you think about making a new thread out of post #6 and asking the mod to make it into a sticky?

[chipzoller]

I'd say McBob+SelectEven for maximum quality (extremely slow though) or TDeint+TMM(+EEDI2) for very good quality at a reasonable speed.
I'd say Yadif for very good quality and very fast speed. It's my new favorite all-purpose deinterlacer right now thanks to Fizick.

[zambelli]

I'd say Yadif for very good quality and very fast speed. It's my new favorite all-purpose deinterlacer right now thanks to Fizick.
How does it do in terms of reliability? How often are new bugs found or interlaced artifacts sneek through?

[chipzoller]

I've tested it extensively and have not come across any artifacts or anything else strange. It's very close to the quality that mcbob and tdeint+eedi2 offer but years faster, and this is in both mode 0 and 1.

[wayback]

I appreciate the responses to my newb question.

Could someone point me to an example, or collection of examples, of .avs files that I might use to do this conversion? I've never used avisynth before, and it's a bit confusing.

I'm starting with NTSC DV, which is 720x480 interlaced, unsquare pixels, 4:3 aspect ratio, 29.97 fps, brought in with DVIO. I'm using the Cedocida DV codec feeding avisynth 2.57. I believe Cedocida will output either YUY2 or YV12. Then to VirtualDubMod in fast recompress mode to produce the DivX output.

I need to end up with DivX 640x480 progressive, square pixels, 4:3 aspect ratio, 29.97 fps. I want as good quality as I can get, even if the processing time is long.

I've got all kinds of questions about what to do in avisynth, and in what order. I know I need to deinterlace, and resize, and possibly separate into fields, but I just need to look as some examples of how this could all be done for one or more of the high-quality deinterlacers.

[Blue_MiSfit]

Well, your basic workflow will be this (I think)

Have Cedocida decode DV to YUY2.


avisource("fantastic_movie.avi")

#Right now it's YUY2, 720x480, Interlaced, 29.97fps, 4:3

AssumeBFF() #Just in case, since DV is BFF
YADIF(mode=0) #Deinterlace to 29.97p

BicubicResize(640,480) #Whatever algorithm suits your needs

ConvertToYV12() #Convert progressive YUY2 to progressive YV12


That should output progressive 640x480 @ 29.97p, and do a damned nice job of it.

If you really want to be masochistic with the time required, switch out Yadif (which in this case is operating in same rate aka 29.97p mode) with MCBob().SelectEven(), which will bob-deinterlace to 59.94p VERY nicely, and then decimate away half the frames to get 29.97p.

If you can manage it, and have high motion stuff, you might think about just encoding to 59.94p. The difference can be pretty stunning.

So to sum things up with some important key points:

1)Deinterlace YUY2 first.

IIRC, decoding NTSC DV (4:1:1) to YUY2 (4:2:2) introduces less chroma artifacting than decoding straight to YV12(4:2:0). Of course, you're going to convert to YV12 for MPEG encoding, but you might as well have full precision through your filter chain.

Always remember the difference between bobbing and same rate deinterlacing. Try different algorithms, as YMMV ;)

2) Resize and do the YV12 down conversion as a last step.

I'm not an expert, but I have done a fair amount of DV -> MPEG-4 work, and this workflow is good for me.

Another note, DV is frequently noisy, and doesn't compress especially well. This depends largely on the motion content, and quality of camera / lens etc, but you will usually benefit with some noise reduction. I always de-noise the chroma somewhat, usually with fft3dfilter / fft3dgpu, and sometimes the luma a bit as well.

Again, the golden rule is YMMV, and try lots of stuff to get a good idea of what works and what doesn't. In AviSynth, I like to do frame by frame comparison like this (for example, comparing an unfiltered frame to one processed by fft3dgpu at moderate chroma settings)


a=last
b=a.fft3dgpu(sigma=2, plane=3).subtitle("fft3dgpu")

interleave(a,b)


I spend lots of time stepping through the video in VirtualDub frame by frame and doing comparisons. This method can be applied to almost any filter, provided you keep your wits about you (no comparing bobbers and same rate deinterlacers without decimating)

Good luck!

~MiSfit

[um3k]

One thing to add: change the resize in Blue_MiSfit's script to this:

BicubicResize(640,480,1./3,1./3,8,0,704,480)

That will give proper square pixels.
Explanation: http://www.mir.com/DMG/aspect.html
In depth: http://lipas.uwasa.fi/~f76998/video/conversion/

[zambelli]

Another note, DV is frequently noisy, and doesn't compress especially well. This depends largely on the motion content, and quality of camera / lens etc, but you will usually benefit with some noise reduction. I always de-noise the chroma somewhat, usually with fft3dfilter / fft3dgpu, and sometimes the luma a bit as well.
I'd like to second that recommendation. Denoising DV can do wonders for your output. Though it's probably a topic for itself, I've found that denoising with FFT3D before deinterlacing gives me slightly better results, though plenty of people would recommend exactly the opposite. If performing denoising before deinterlacing, remember to use the "interlaced=true" parameter in FFT3D, like this:
AssumeBFF()
FFT3DFilter(sigma=2.0, plane=4, interlaced=true)
YADIF(mode=0)
You can then compare that with this:
AssumeBFF()
YADIF(mode=0)
FFT3DFilter(sigma=2.0, plane=4, interlaced=false)

Other tips and tricks:
1) Check YUV levels in your video. It's amazing how often DV camcorders record useful information in the 0..15 and 236..255 range that ends up getting thrown away if the range isn't tweaked during preprocessing. I've been able to recover texture in seemingly entirely white and black areas on more than one occasion by just adjusting the range a little.

2) If dealing with PAL DV, set Cedocida to output YV12 MPEG-2 interlaced video. You'll get the true 4:2:0 video without any upsampling or downsampling.

Actually, if somebody wants to create a "best practices for processing DV" topic, this all might be better suited there.

[chipzoller]

For those of you using Yadif, you may want to set the order in the params where order=0 = BFF and order=1 = TFF. Yadif does, however, require the field parity be set at least with AssumexFF() before calling the filter itself.

[wayback]

Thanks very much for the replies. A few questions:

1. I assume the very best quality, particularly when viewing things that move, would be obtained at double the original frame rate, which as I understand it is what bobbing produces. But that would also produce twice the file size. Right?

2. I think I understand about cropping before resizing, in theory, but the final test would be whether square objects in the original video are still square after processing. Or, if they aren't, whether I like the distortion produced by not cropping, which I assume would make things skinnier than they were in the original.

Actually, I guess I really should see if objects which are square in real life are still square in the processed video, which would eliminate any distortions the camera might produce in its processing.

3. How do you tweak YUV levels during processing to make sure you don't lose the extremes? I assume Cedocida automatically preserves them. What do I need to do to preserve them through processing?

> Actually, if somebody wants to create a "best practices for processing DV" topic, this all might be better suited there.

That would be fine if someone prefers to start a new thread. There is already a DV to DivX sticky thread, but it uses a completely different method - processing in VDub, etc.

Thanks again for the suggestions.

[Didée]

I'd say Yadif for very good quality ... I've tested it extensively and have not come across any artifacts or anything else strange. It's very close to the quality that mcbob and tdeint+eedi2 offer


Yadif
http://img443.imageshack.us/img443/2374/toytrainyadifce1.th.jpg (http://img443.imageshack.us/my.php?image=toytrainyadifce1.jpg)

MCBob
http://img411.imageshack.us/img411/9697/toytrainmcbobfs6.th.jpg (http://img411.imageshack.us/my.php?image=toytrainmcbobfs6.jpg)

[Blue_MiSfit]

Bobbing won't produce exactly twice the filesize.. Okay, maybe for uncompressed video, but remember that the difference between frames becomes less, so you should be able to compress rather well. But yes, it will take more bitrate to look "as good" as the same video at 29.97. It all depends on whether or not there's enough motion in your source to justify it.

@ Didee:

Wow, I guess I haven't set up MCBob in awhile. Theres some serious differences there. All residual combing seems to be gone, and there is much more detail. The chroma seems much more accurate as well. One good (slow) bobber!

~MiSfit

[chipzoller]

Yes, it does a very good job, but drastically slower than Yadif. If you need the best deinterlacing possible no matter the speed, use mcbob. If you need good fast deinterlacing, use Yadif.
Didée, are you still working on improving mcbob either in speed or quality?

[wonkey_monkey]

Didée, are you still working on improving mcbob either in speed or quality?

I've only just started writing plugins, but if I can do anything to help improve the speed of mcbob (like writing customised versions of plugins to do only what mcbob needs) I'd be happy to try.

One thing that suggested itself to me was a custom version of EEDI2 which doesn't interpolate unmasked areas - that way only those areas that mcbob needs to fill in are processed. Unless I've misread the mcbob script and something like this is already being done...

David

[wayback]

I've tried a number of the suggestions provided here, and the results look pretty good. Thanks very much.

But, I can't help but notice that the best quality, by a good margin, is the original imported DV played in Windows MediaPlayer 10. If I play the raw file in VirtualDub using the Cedocida DV decoder, it looks a good bit more "jumpy".

Now MP is resizing to 4:3, and I assume doing on-the-fly deinterlacing, and Cedocida is doing neither of those. So of course there are lots of jaggies and things appear a bit stretched out. But I was not expecting the jumpiness.

And the problem is that the jumpiness is carried through AviSynth, VirtualDub, and DivX to the final compressed video. So it seems I have a problem at the head end of this process.

The sample footage is just hand-held panning around my back yard, so there is some jiggle. But while it looks relatively smooth in MP, the very same file played with Cedocida/VirtualDub looks as though image stabilization has been turned off. The difference is quite noticable.

Can anyone explain what's going on here, and suggest what I can do about it?

[wonkey_monkey]

If you've been using yadif(mode=0), that will remove the interlacing and keep only one field from a frame. Media Player is (presumably) doing the (quick and dirty) equivalent of yadif(mode=1), which keeps both fields from each frame, thus doubling the number of frames, and resulting in smoother motion.

David

[wayback]

Yes, that appears to be the explanation. Media Player is presumably set to double-rate deinterlace DV automatically, but of course VirtualDub doesn't do that.

But Media Player doesn't do that when playing back a DivX file which was encoded with the "Preserve Interlace" setting. And apparently the DivX playback codec doesn't either. I tried doing an preserved-interlace conversion, and it looks terrible.

So, I tried a Mode 1 conversion, leaving the DivX bitrate the same (2000 kpbs). It not only looked much smoother, but frankly I didn't notice any degradation in the non-moving scenes.

And since presumably any modern computer could display 60 fps ok, and if the bitrate doesn't have to be increased to get the same quality, but smoother, output, then why, exactly, don't we do double-rate deinterlacing on everything? Why are all the file-sharing DVD rips same-rate deinterlaced? I guess I must be wrong about the bitrate not needing to be significantly higher.

If I do everything in double-rate deinterlacing, is there anybody who wouldn't be able to view the resulting DivX properly? I wonder if stand-alone DivX players like my Philips 642 will play it ok - I'll burn a test DVD and see. And if I go that way, is Yadif still the right deinterlacer to use?

[wayback]

The Philips 642 DVD player handles 30fps fine, but gets nowhere with DivX 60fps. It produces about a second of audio, then freezes up. The screen stays black.

Rats.

[plane]

@ Didee:

Wow, I guess I haven't set up MCBob in awhile. Theres some serious differences there. All residual combing seems to be gone, and there is much more detail. The chroma seems much more accurate as well. One good (slow) bobber!

~MiSfit

Where can I get the lastest ver of MCBob? :confused:

[Didée]

Thread: MCBob (http://forum.doom9.org/showthread.php?t=124676).

[legoman666]

If you're looking to preserve the quality during motion, you might try a 2 pass xvid or divx encode. (or maybe your 2000kbps bitrate is high enough that you don't get any blockiness during motion, i dont know, just a suggestion)

[plane]

Other tips and tricks:
1) Check YUV levels in your video. It's amazing how often DV camcorders record useful information in the 0..15 and 236..255 range that ends up getting thrown away if the range isn't tweaked during preprocessing. I've been able to recover texture in seemingly entirely white and black areas on more than one occasion by just adjusting the range a little.

Can I know your tweaking method? :devil:

[Blue_MiSfit]

Indeed. I would like to know how to do this (should be simple), as I do a lot of work with 60i NTSC DV.

[zambelli]

Indeed. I would like to know how to do this (should be simple), as I do a lot of work with 60i NTSC DV.
I start by using Avisynth's ColorYUV(analyze=true) and Histogram("levels") function in YV12 colorspace (not both at the same time though). Scrub through the video and note how frequent and dense the information in the sub-16 and super-235 areas is. Try to find a few frames that are underexposed and a few frames that are overexposed.

Then load the MaskTools plugin into your script and paste this function into it:
function Levels_Smooth(clip clp, int min_before, float gamma, int max_before, int min_after, int max_after)
{
pc2tv = clp.Levels(min_before, gamma, max_before, min_after, max_after, coring=false)
pc2tvd=yv12lutxy(clp,pc2tv,"x y - 4 * 128 +","x y - 4 * 128 +","x y - 4 * 128 +",U=3,V=3)
yv12lutxy(clp,pc2tvd.removegrain(mode=19),"x y 128 - 4 / -","x y 128 - 4 / -","x y 128 - 4 / -",U=3,V=3)
}
This is a modification of a function Didee suggested a while back for doing "smooth" levels remapping.

Call ehe function right before your histogram/analysis call, like this:

Levels_Smooth(0,1.0,255,16,235)

Then go back to those underexposed and overexposed frames and see if the remapping brings out any additional detail. Using Interleave() to do a before-after comparison is very helpful.

Note that not all camcorder shot video needs to be remapped from 0..255 to 16..235. Sometimes the low and high range data really is junk. The best way to figure that out is to keep looking at those under/overexposed frames and figure out how far the range needs to be remapped before the lost details are completely brought out. For example, you might need to only remap 8..235 to 16..235 because everything below 8 and above 235 is junk.

[Mtz]

If performing denoising before deinterlacing, remember to use the "interlaced=true" parameter in FFT3D, like this:
AssumeBFF()
FFT3DFilter(sigma=2.0, plane=4, interlaced=true)
YADIF(mode=0)
...
Actually, if somebody wants to create a "best practices for processing DV" topic, this all might be better suited there.
Try to use nnedi instead of yadif. :rolleyes:

enjoy,
Mtz

[Tack]

Yadif
http://img443.imageshack.us/img443/2374/toytrainyadifce1.th.jpg (http://img443.imageshack.us/my.php?image=toytrainyadifce1.jpg)

MCBob
http://img411.imageshack.us/img411/9697/toytrainmcbobfs6.th.jpg (http://img411.imageshack.us/my.php?image=toytrainmcbobfs6.jpg)
This is a pretty compelling example. Didée, could you make a short clip of this source available? I'd like to experiment.

[Didée]

VQEG Test Sequences: http://media.xiph.org/vqeg/TestSeqences/Reference/

The toy train one is "src10_ref__625".

Thumbnails of all sequences: http://media.xiph.org/vqeg/TestSeqences/ThumbNails/

[wayback]

One thing to add: change the resize in Blue_MiSfit's script to this:

BicubicResize(640,480,1./3,1./3,8,0,704,480)

That will give proper square pixels.
Explanation: http://www.mir.com/DMG/aspect.html
In depth: http://lipas.uwasa.fi/~f76998/video/conversion/

Sorry to bring this back up again, but could someone explain the terms in the resize script line? Well, particularly the 1./3,1./3,8,0 part. It's supposed to crop 8 pixels off each side of a 720x480 NTSC DV frame, then resize down to 640x480.

[2Bdecided]

The last four numbers are the crop part (exactly the same as AVIsynth's crop command, but executed within the resizer).

The fractions (1./3 = 1/3 - the decimal point just forces AVIsynth to treat the result as floating point, rather than an integer - essential, since you can't express 1/3 as a whole number!) are the b and c coefficients of the resizer - see the mediawiki on bicubicresize for more details.

Cheers,
David.