Correct way to capture PAL progressive VHS?

[LeoH]

Hi all,

I've got a question about capturing an old progressive PAL VHS with my ADVC-100. As I understand it, the ADVC-100 will only produce interlaced DV files. The trouble is, that this produces jagged edge lines on objects from what looks like field shifting. However, if I render the captured file as "progressive" 25fps in Vegas - the jagged edges disappear but the resolution seems to get worse. Maybe just an illusion? Anyway, is this an acceptable way of capturing progressive PAL VHS tapes - or am I doing it all wrong? I'm concerned that rendering in Vegas as progressive is not the right way to do it, but I'm not sure what else to do. I've tried Deinterlace in AviSynth, but the results weren't very good. I'm planning on putting this to PAL DVD, after cleaning in AviSynth. Any help or suggestion? Thanks!!!

Source frame 1 untouched (ADVC-100 capture):
http://s90519770.onlinehome.us/cap1interlaced.png

Frame 1 rendered "progressive" in Vegas:
http://s90519770.onlinehome.us/cap1progressive.png

Source frame 2 untouched (ADVC-100 capture):
http://s90519770.onlinehome.us/cap2interlaced.png

Frame 2 rendered "progressive" in Vegas:
http://s90519770.onlinehome.us/cap2progressive.png

-LeoH:)

[jggimi]

Moved to DV forum.

[LeoH]

Yes I'm planning on putting it to PAL DVD. But since the original material is progressive, I feel like it should be kept progressive no? Also, all the noise reduction I do in AviSynth seems to work much better with progressive frames. I'm not arguing, just wondering...! Thanks!

LeoH :)

[SomeJoe]

Originally posted by LeoH
Yes I'm planning on putting it to PAL DVD. But since the original material is progressive, I feel like it should be kept progressive no?

What makes you believe that the source material is progressive?

It doesn't look progressive to me. In your source frame 1, the gentleman on the right has clear interlacing lines from motion visible at the border of his dark shirt.

This appears to be interlaced video, and should be processed and encoded as interlaced. AVIsynth filters (including denoisers like convolution3D) can be scripted to work properly with interlaced video. I don't have a convolution3D interlaced example script here with me, but when I get home tonight I can post one.

[LeoH]

Well in this previous thread I started before I figured out where the jagged lines were coming from, I was told by neuron2 that I have progressive material. http://forum.doom9.org/showthread.php?s=&threadid=77833

Those lines you see at the shirt border are not from motion, but from mismatched fields or something. Here's a short clip of the ADVC-100 capture:



Clip 1 (21MB) (http://s90519770.onlinehome.us/Test1Short.avi)


LeoH:)

[SomeJoe]

OK, I reviewed the other thread.

I have another idea of what might be wrong, but I'll have to get home where I can step through the video in VirtualDub to confirm. (About to leave work). Check back in this thread a little later.

[LeoH]

Thanks for looking into it. At first I thought the jagged edges might be caused by some field order problem, but so far as I can tell everything is set correctly. Unless my ADVC-100 is somehow screwed up... :( So normally, when capturing PAL progressive material as interlaced DV, these jagged edges shouldn't appear right?

Thanks for any help or advice,

LeoH :)

Here is a much longer clip if you have a fast connection:

Clip 2 (138MB) (http://s90519770.onlinehome.us/Test1.avi)

[SomeJoe]

OK, finally finished all my testing. Your video is indeed originally from a progressive source. Of course, each progressive frame from the film is split into two interlaced fields on tape, and when captured.

I looked at a lot of things with your video, and I think I finally came up with something that fully explains the artifacts you're seeing. First, the problems that this is NOT:

1. Field grouping. i.e. Suppose the ADVC-100 captured fields and grouped them into DV frames where each field in the DV frame came from a different film frame. This can be corrected by:

DoubleWeave.SelectOdd()

Apparently, this is not the problem, as DoubleWeave.SelectOdd() made the problem worse.

2. Chroma upsampling error (CUE). The chroma upsampling error occurs as a result of a code error in any video codec (MPEG-2 or DV) that has to take interlaced-encoded 4:2:0 content and upsample it to 4:2:2 or 4:4:4. (Note that not all codecs/implementations suffer from this bug, but many do). While this bug is normally seen with MPEG-2 codecs, it applies to you also because PAL DV is 4:2:0. This can be corrected by:

FixBrokenChromaUpsampling()

This isn't the problem either, because FixBrokenChromaUpsampling() didn't make any difference. Also, to have this bug present, your DV codec installed on the computer would have to be one that suffers from this bug, and I believe only the Microsoft DV codec does. If you have any other DV codec installed (like Main Concept or Panasonic), you should be fine. Note that I can't tell for sure, since the samples I downloaded from you are HuffYUV .avis, which have already been decoded by your DV codec.

3. VCR head misalignment. What can happen here is that the heads of your VCR can be misaligned (or the heads of the deck that recorded the signal on tape were), and every other scan line is played back with slightly different offset timing. This causes each bottom field to be shifted horizontally from the top field. This can be corrected by:


original=AVISource("Test1Short.avi")
original=original.ConvertToYUY2(interlaced=false)

field1=original.SeparateFields().SelectOdd()
field2=original.SeparateFields().SelectEven()

field2=BlankClip(field2).Layer(field2,"add",255,1,0,0,TRUE)

new=Interleave(field1,field2)
new=new.Weave()

return new


where the red "0" in the code above is replaced with a horizontal pixel offset from field 1. I tried values ranging from -2 to 2, and anything other than 0 misaligned the fields more. So this is apparently not the problem.


OK, so having investigated all that, that leaves only one other conclusion. I believe you're seeing what's called the Interlaced Chroma Problem (ICP), and it comes from the fact that interlaced 4:2:0 as used in PAL DV is an inherently bad design. This is not a bug, because the algorithms are doing exactly what they're supposed to, and this is a completely different problem than the Chroma Upsampling Error (CUE).

The problem arises because in each field, chroma samples only exist on every other field line, and in the opposite field, the chroma samples are horizontally offset from the ones in the first field. The resulting interpolation of chroma information results in "filled in" chroma areas that ignore the chroma information in the other field.

With true interlaced material where each field is different, this problem is less noticable, although it still exists. With your material (that is actually progressive material but was encoded interlaced by the ADVC-100), it's highly noticable.

Note that NTSC DV doesn't suffer from this issue because the chroma downsampling is 4:1:1 vice 4:2:0.

There is not any type of filter that can completely remove these artifacts, although I cleaned it up pretty good with:

Convolution3D(0, 32, 128, 16, 64, 10, 0)

If you want to be less aggressive with the cleanup to avoid losing too much detail, try:

Convolution3D(0, 6, 10, 6, 8, 2.8, 0)

The final AVISynth script I used was:


LoadPlugin("convolution3D.dll")

AVISource("Test1Short.avi")
ConvertToYUY2(interlaced=false)
Convolution3D(0, 32, 128, 16, 64, 10, 0)


The reference that contains a lot of information on the Chroma Upsampling Error and Interlaced Chroma Problem is http://www.hometheaterhifi.com/volume_8_2/dvd-benchmark-special-report-chroma-bug-4-2001.html.

Other than this AVISynth filtering, the only other way you will be able to improve quality is to capture the video with a non-4:2:0 codec, which would mean ditching the ADVC-100. This footage would look perfectly pristine if captured as D1 (8-bit uncompressed) on a professional capture board.

[trevlac]

Originally posted by SomeJoe

OK, so having investigated all that, that leaves only one other conclusion. I believe you're seeing what's called the Interlaced Chroma Problem (ICP), and it comes from the fact that interlaced 4:2:0 as used in PAL DV is an inherently bad design. This is not a bug, because the algorithms are doing exactly what they're supposed to, and this is a completely different problem than the Chroma Upsampling Error (CUE).

The problem arises because in each field, chroma samples only exist on every other field line, and in the opposite field, the chroma samples are horizontally offset from the ones in the first field. The resulting interpolation of chroma information results in "filled in" chroma areas that ignore the chroma information in the other field.



I don't think this is the problem. If you kill the chroma (like with Tweak(sat=0), I believe you would still see the artifacts. The artifact seems to be in high contrast areas. Can't say I know what it is, but I'd test thru my process if I were Leo.

Start by making sure there is no edge enhancement / contrast bosting type of thing happening on the VCR. Also try another VCR or the RCA outs to see if this makes a difference.

[LeoH]

Thanks for all the testing SomeJoe, at least I have a good idea of what the problem isn't now! ;) I'm reading up on that ICP problem. Anyway, I'm hestitant to use Convolution3D to clean up the problem, since I'd rather attempt to find the root of the trouble if possible. I was also having some luck with the "field bob" plug from VDubMod, but again it's just sort of a band-aid. Like Trevlac was suggesting, I think I'll try to do some more testing of my VCR and see if perhaps that's where the problem lies. I'm using a brand new JVC HR-5980 Multi-System S-VHS VCR - and I was taking the S-Video out into my ADVC-100. The VCR has internal settings for "Norm", "Edit", "Soft", and "Sharp" picture control and I had set it to "Edit".
I also have four copies of this original tape, and I had captured each copy to see of the problem was with the tapes - and the jagged edges do appear on each tape though often in different locations and in different strengths.
So I think I'll run some tests with using the RCA video out (though in theory this would give worse video quality right?) and setting my VCR differently. I'll be bummed if it turns out that the problem is inherent with the PAL DV format, since this project is pretty much why I bought the ADVC-100... :(

-LeoH:)

[frank]

It looks like a typical problem of VCR: Jitter! Caused by mechanical speed jitter of the head drum. Then the horizontal sync pulses have some jitter, and you get jagged vertical lines.

You have to set the time base correction (TBC) to ON, to start every line at the same time.
Some good VCR have that electronics built in.

[LeoH]

Hmm Frank, I hadn't even thought of that. My JVC VCR doesn't have TBC built in, do you really think this may be the source of the problem? I suppose I could buy an external TBC, but I'd like to be fairly positive that this will fix things before I spend $200 or more. Thanks,

LeoH :)

[SomeJoe]

Originally posted by trevlac
I don't think this is the problem. If you kill the chroma (like with Tweak(sat=0), I believe you would still see the artifacts.

Interesting suggestion. I will try this on Leo's footage when I get home tonight.

However, my prediction is that I will still see the artifacts, even though I still believe that this is ICP. Here's why:

The ICP appears as soon as the DV codec on the computer renders the video to RGB for handoff to any other application. On Leo's computer, here was the processing chain:

1. VCR (analog S-video, interlaced)
2. ADVC-100 (analog->digital RGB 8-bit uncompressed, interlaced)
3. ADVC-100 (downsample to 4:2:0 YUV uncompressed, interlaced)
4. ADVC-100 (compress using DCT, i.e. DV codec, interlaced)
5. 1394 transfer to computer
6. Write to DV .avi file

Then, on conversion to HuffYUV:

7. Read .avi file
8. DV codec (uncompress with DCT, interlaced)
9. DV codec (upsample 4:2:0 YUV to 4:4:4 RGB, interlaced)
10. Handoff to AVIsynth/VirtualDub
11. HuffYUV codec (convert to YUV, interlaced)
12. HuffYUV codec (lossless huffman compression)
13. Write to .avi file

Then I downloaded the clip, and did my tests.

Step 9 above is where the artifacts were introduced. Because at that point the badly interpolated chroma was mixed with luma to form RGB pixels, there is no way I can recover the non-artifact video prior to that point.

Originally posted by frank
It looks like a typical problem of VCR: Jitter! Caused by mechanical speed jitter of the head drum. Then the horizontal sync pulses have some jitter, and you get jagged vertical lines.


Another interesting possibility, although one that I somewhat looked for in my step 3 above. However, if the jitter is not consistent from line to line, they my method above won't work, and a TBC will be necessary.

Here's what I suggest to investigate both of these issues: Use a DV codec on the computer that can open and hand off the DV frames as YUV instead of RGB. The alleged chroma distortion from ICP would still be present in the frames, but then setting Tweak(sat=0) will confirm that the problem is only in the chroma channel, which would support an ICP issue.

One of the DV codecs I know that will hand off frames as YUV is the Main Concept DV codec (this is the one I use). I would suggest that Leo install this codec, and perform a conversion of one of the ADVC-100 captured DV .avi files to HuffYUV using the following script:


# With Main concept DV codec, should open as YUY2
AVISource("Test1Short.avi")
Tweak(sat=0)


If the black and white frames in the HuffYUV file here have no interlacing distortion, that proves that all distortion is in the chroma channels, and is indicative of ICP.

If the distortion remains, this means that the distortion is in the luma channel, and VCR head jitter is a possibility.

[LeoH]

Well looks like it's in the Luma channel too... :(

I converted another short clip using the MainConcept DV codec to HuffYUV and compared the images with and without Tweak(sat=0) and the jagged lines are there both times.

Source:
http://s90519770.onlinehome.us/snap1.png

with Tweak(sat=0) setting
http://s90519770.onlinehome.us/snap2.png

I also uploaded another short clip where these frames were grabbed from - this clip was converted with the MainConcept DV codec to HuffYUV.

clip2 (http://s90519770.onlinehome.us/test2.avi)

So I'm starting to think that maybe it is related to sync pulses being off and maybe I do need a TBC. Aaargh... more $$ :scared: !!

I wish I had an external broadcast monitor to check, any chance that this is related to progressive scan computer monitor not displaying correctly?

Thanks for all the help!

LeoH:)

[SomeJoe]

I tried some similar tests with the footage I already downloaded from you, and the artifacts were also visible in the luma channel.

A TBC certainly would not hurt. It may fix the problem completely. Pro VCRs with built-in TBCs are not too expensive these days, but I would definitely borrow one and test this theory before you spend the money.

Personally, I still think the ICP is the source of the artifacts. Really, the only way you could truly verify this would be to have the footage captured via a 4:2:2 codec. If you know of anyone who has a DVCPro50 or D1 deck or camcorder, this could be your test.

One other way to capture like this would be to get ahold of an old Matrox RT2000 or RT2500 editing card. They are primarily used for DV-based capture, but also had the ability to capture in 25Mb/sec MPEG-2 I-frame 4:2:2 :) . This would also be a fair test.

Probably, finding a TBC VCR to test with will be easier, but if that doesn't solve the problem, those other choices would be your next step.

[trevlac]

Originally posted by SomeJoe

here was the processing chain:

1. VCR (analog S-video, interlaced)
2. ADVC-100 (analog->digital RGB 8-bit uncompressed, interlaced)
3. ADVC-100 (downsample to 4:2:0 YUV uncompressed, interlaced)
4. ADVC-100 (compress using DCT, i.e. DV codec, interlaced)
5. 1394 transfer to computer
6. Write to DV .avi file



I don't know much about the canopus device, but I'm a bit surprized that the ADVC-100 would sample a YUV signal and then convert it to RGB. I'd think #2 should be YUV to YCbCr at 4:2:2.

Anyway ... break the pin on an svideo connector (or make a cable) to remove the chroma at the source. Even if the device converts to YUV->RGB->YCbCr there should be no error because the UV are zero to start.

Sorry ... still no help from me on the problem. But I'd look at the VCR. It does not have a line TBC. Be nice to try a different machine.