DV encoder shootout (Contributions needed)

[zambelli]

There's been a lot of talk about DV decoder differences in this forum, but I don't recall ever seeing a comparison of DV encoders, so I thought I'd start one.

Intro:
Unlike high efficiency interframe/interfield codecs such as MPEG-2, MPEG-4, VC-1, VP6, and others, standard def DV is a codec that works at a fixed bitrate (25Mbps in its MiniDV variant, aka DV25 or DVSD) and uses only intraframe/intrafield compression - I frames only. It's the most commonly used codec in video editing, which puts a high requirement on its ability to recompress the same video over and over again without noticable image degradation. While evaluating other codecs is often subject to personal opinion, evaluating DV codecs is much easier. Quite simply, a DV encoder needs to retain as much information as possible from the source video. This is easily quantifiable with a standard signal-to-noise metric such as PSNR.

DV25 exists in 2 major flavors: NTSC 720x480 29.97fps, and PAL 720x576 25fps. The former uses a 4:1:1 chroma sampling, while the latter uses a more common 4:2:0 chroma sampling. Although the DV standard defines different DCT modes for frame (progressive) vs field (interlaced) encoding, it does not define when the different modes should be used. In other words, although DV is commonly used for interlaced encoding, it is capable of encoding both progressive and interlaced content using different DCT modes at the encoder's discretion.


Sources:
In order to conduct this test, I chose a few DV videos from my own DV cameras (I have both NTSC and PAL), loaded them into Avisynth using the Cedocida decoder in YUY2 mode (I'll explain that later) and edited series of frames together into a collage that's IMHO representative of average DV footage. I then used TDeint Avisynth plugin to deinterlace both videos into progressive footage too. This produced four source videos:

DV_NTSC_Interlaced.avi (http://www.citizeninsomniac.com/video/DV_NTSC_Interlaced.avi) (35.6 MB)
DV_NTSC_Progressive.avi (http://www.citizeninsomniac.com/video/DV_NTSC_Progressive.avi) (35.4 MB)
DV_PAL_Interlaced.avi (http://www.citizeninsomniac.com/video/DV_PAL_Interlaced.avi) (47.9 MB)
DV_PAL_Progressive.avi (http://www.citizeninsomniac.com/video/DV_PAL_Progressive.avi) (47.8 MB)

Each video is encoded losslessly with HuffYUV (4cc: HFYU) in YUY2 mode. Length: 150 frames.

Why did I encode the sources as YUY2 (4:2:2) if DV is 4:2:0/4:1:1? Most software DV codecs use YUY2 as their preferred input/output format. The reason behind this is practical: in 12-bit YUV chroma mapping differs between MPEG-2 and DV, 4:2:0 and 4:1:1, progressive and interlaced, and the mapping is entirely ambiguous - there's no way for a downstream compontent to know the chroma placement coming from a DV decoder if the ouput is generic YV12. YUY2 (YUV 4:2:2) doesn't have this problem, therefore making it suitable for implementing software DV codecs without any need for user interaction. Even though Cedocida supports YV12 input/output, it is one of the few codecs that does - and this really only works for PAL. The rest mostly work in YUY2, and some even accept RGB24 only.

The Challenge
The goal of this comparison is to take each one of those source videos and encode it to DV25 using any DV encoder you might have available, then compare it to the source and calculate the PSNR. Simple! :)

The first step is encoding. Most DV codecs come in VfW flavor, making it easy to encode with VirtualDub. This also eliminates the need for any unnecessary colorspace conversion, such as the ones introduced by Adobe Premiere, for example.

In VirtualDub, the encoding would look something like this:

1. File --> Open Video File...
2. Open the source video (i.e. DV_NTSC_Interlaced.avi)
3. Audio --> No Audio
4. Video --> Fast Recompress
5. Video --> Compression...
6. Choose your DV codec (i.e. Cedocida DV codec)
7. File --> Save as AVI...

If you want to use another method (i.e. GraphEdit, NLE app, etc.), go ahead - but please note this in your post.

Do not remap luma range! The luma range on the source clips is already clamped to 16..235, so please don't compress, expand or re-clamp it in any way!

Please encode all 4 videos - PAL progressive + interlaced, NTSC progressive + interlaced.

When you're done, upload the videos to my server. They should be about 15MB each, which isn't too difficult to upload.

FTP: ftp://ftp.citizeninsomniac.com
Username: dv@citizeninsomniac.com
Password: doom9

Last step - post a message here and let me know that you've submitted new videos, and document your process. Include the name of the encoder, its version number and type (VfW, DirectShow, QuickTime, etc.). Note any special encoder options used.

The Results:
To start this comparison, I compared Cedocida DV v0.1.6 and MainConcept DV v2.4.3. I tested only the Normal Quality encoding mode of Cedocida due to reported problems with High Quality mode. I made an exception to my rule - I also tried the YV12 input/output mode, as it's somewhat exclusive to Cedocida.

These are the latest results:

|-----------------------------------------------------------------------------------------|
| | Ovrl PSNR | Min PSNR | Max PSNR | Notes |
| NTSC Interlaced: |
|-----------------------------------------------------------------------------------------|
| Cedocida v0.1.6 (VfW) | 49.3498 | 43.7395 | 55.0623 | YUY2, normal quality |
| MainConcept v2.4.3 (VfW) | 45.4076 | 39.2093 | 48.3694 | YUY2, "Fastest" OFF |
| Microsoft v6.05.2600 (DShow)| 46.5425 | 41.5732 | 51.3981 | RGB24, NTSC |
| | | | | |
| NTSC Progressive: |
|-----------------------------------------------------------------------------------------|
| Cedocida v0.1.6 (VfW) | 49.8454 | 45.3902 | 53.4945 | YUY2, normal quality |
| MainConcept v2.4.3 (VfW) | 46.7379 | 42.2356 | 48.6946 | YUY2, "Fastest" OFF |
| Microsoft v6.05.2600 (DShow)| 46.9914 | 42.0759 | 51.0454 | RGB24, NTSC |
| | | | | |
| PAL Interlaced: |
|-----------------------------------------------------------------------------------------|
| Cedocida v0.1.6 (VfW) | 46.1306 | 40.3453 | 54.7958 | YUY2, normal quality |
| Cedocida v0.1.6 (VfW) | 45.6650 | 40.1155 | 54.0237 | YV12 MPEG int, normal|
| MainConcept v2.4.3 (VfW) | 41.2780 | 35.4578 | 48.5933 | YUY2, "Fastest" OFF |
| Microsoft v6.05.2600 (DShow)| 38.9603 | 32.8893 | 52.4871 | RGB24, PAL |
| | | | | |
| PAL Progressive: |
|-----------------------------------------------------------------------------------------|
| Cedocida v0.1.6 (VfW) | 46.8568 | 41.5192 | 54.0270 | YUY2, normal quality |
| Cedocida v0.1.6 (VfW) | 46.9538 | 41.5436 | 53.9991 | YV12 MPEG prg, normal|
| MainConcept v2.4.3 (VfW) | 43.6773 | 38.2602 | 48.4947 | YUY2, "Fastest" OFF |
| Microsoft v6.05.2600 (DShow)| 39.6478 | 33.6012 | 51.9862 | RGB24, PAL |
| | | | | |
|-----------------------------------------------------------------------------------------|

Current winner: Cedocida v0.1.6

OK. Your turn. :)

[Blue_MiSfit]

I decided to give Avid a whirl. I have a copy of Avid Free DV which has Avid's DV codec. Unfortunately its a quicktime codec, and avid doesn't support importing huffyuv.

So I had to convert to an intermediate uncompressed file and import that into avid, then export using their DV25 codec. Then I used the QTInput filter to get the resulting quicktime mov into avisynth, and ran the comparison. I'm not sure how to get things nicely organized like you did - forgive me :)

I just got a big long list of frames with PSNR values. They also seem very low - 28 to 30 max. I am probably doing something wrong. Do you have suggestions on how to improve this process? I would like to try the Apple QuickTime codecs in this fashion as well.

~MiSfit

[zambelli]

I just got a big long list of frames with PSNR values. They also seem very low - 28 to 30 max. I am probably doing something wrong. Do you have suggestions on how to improve this process? I would like to try the Apple QuickTime codecs in this fashion as well.
Try commenting out the Compare() line, and replace it with an Interleave(source, DV) line. Then view the script in Avisynth and framestep through it. You should see frames from the source and DV encoded alternate and stay perfectly in sync. If they ever drift out of sync - that's going to mess up the PSNR. If everything looks good - then Compare() results should be valid.

[Blue_MiSfit]

Problem fixed. There was a colorspace conversion happening somewhere (probably within avid). I got the qtoutput function to work so I can output directly from AviSynth, decompressed the huffyuv to raw YUY2 and fed that to the Avid and Apple DV Quicktime codecs.

The comparison is much more what I expected this time. Results look competitive with MainConcept.

DV-NTSC-Interlaced.avi compared to Avid-DV-NTSC-Interlaced.mov:

Minimum Average Maximum
Mean Absolute Deviation: 0.4678 0.8294 1.2031
Mean Deviation: -0.7965 -0.5923 -0.3249
PSNR: 40.2864 45.3922 49.3904
Overall PSNR: 44.9437

The avid version looks very close to the original. The only thing that strikes me as subjectively wrong with the output is that there seems to be a hue shift from green to red. This must be due to some unforeseen colorspace conversion.

DV-NTSC-Interlaced.avi compared to Apple-DV-NTSC-Interlaced.mov:

Minimum Average Maximum
Mean Absolute Deviation: 3.5176 4.2877 4.8552
Mean Deviation: +3.1208 +4.0613 +4.5670
PSNR: 31.6977 32.6384 34.2180

Overall PSNR: 32.5749

Looks like Apple needs some real work! The resulting image looks like someone cranked up the brightness, If I recall correctly this is because the Apple DV codec applies a gamma correction every time it recompresses, but don't quote me on that. Just to be sure I did select the high quality output in quicktime player, so it wasn't throwing away half of the resolution like it does by default.

I'm really surprised how well Cedocica performs!

Source
http://home.comcast.net/~blue_misfit/Source0000.png

Avid
http://home.comcast.net/~blue_misfit/Avid0000.png

Apple
http://home.comcast.net/~blue_misfit/Apple0000.png

~MiSfit

[zambelli]

The comparison is much more what I expected this time. Results look competitive with MainConcept.
Awesome! Thank you for taking the time to do this comparison.

DV-NTSC-Interlaced.avi compared to Avid-DV-NTSC-Interlaced.mov:
DV-NTSC-Interlaced.avi compared to Apple-DV-NTSC-Interlaced.mov:
Before I add these to the results table... Is there any chance you could do the same for the other 3 sources, or at least PAL Interlaced at the least?

I'm really surprised how well Cedocica performs!
Is is indeed pretty impressive, especially when one does an A/B comparison. However, I think its PAL mode suffers from a few bugs. While the quality is impressive, I'm not sure if it's ready yet to become the one and only DV encoder.

[JohnnyMalaria]

Hi,

This is an interesting initiative. I'd like to make a couple of comments (as someone who has written a DV codec and, in my other life, as a scientist).

1. You should perform the test with different decoders, too, otherwise you may be biasing the results. There's a possibility that the Cecodica decoder makes certain assumptions that are mirrored with the matching encoder.

2. If the choice of decoder does prove to have an impact, then we'll have the possibility of needing to use the encoder half of one codec with the decoder half of another to obtain optimum results. This is difficult with the VfW architecture but can be overcome with a special wrapper (similar to the one provided with the VfW version of our DV decoder).

3. The test as currently recommended doesn't really stretch the encoders to their limits. By feeding them with previously DV-compressed frames, they shouldn't have to do much (if any) compression at all. The hardest part of writing a quality DV encoder is selecting the right QNOs etc for each macroblock within a segment. To fully test this part of the encoder, you need to feed it with uncompressed material (i.e., that's never been compressed) and such material should consist of a variety of stresses.

4. Ideally, any comparison of the encoders should be done at the encoded data level - i.e., statistically compare the DCT blocks (in the DCT domain), measure the amount of unused space in each segment (a segment is 5 macroblocks) and determine how efficiently the encoder made use of the three stages of compression (intrablock, intramacroblock and intermacroblock).

5. The comment about 4:2:0 interlaced vs. progressive isn't relevant for DV. It is only relevant for MPEG2.

John.

[zambelli]

1. You should perform the test with different decoders, too, otherwise you may be biasing the results. There's a possibility that the Cecodica decoder makes certain assumptions that are mirrored with the matching encoder.
I disagree. Any decoder that conforms to a published specification should be encoder-agnostic. I think Cedocida decoder was written first and foremost as a generic DV25 decoder. There have been several tests done that have shown Cedocida decoder has consistently high quality and doesn't suffer from luma range mapping bugs and chroma upsampling issues commonly found in other DV decoders. This was my main reason for choosing it as the preferred decoder.

The main reason for picking one decoder and sticking with it is to keep the focus of the comparison on encoders.

2. If the choice of decoder does prove to have an impact, then we'll have the possibility of needing to use the encoder half of one codec with the decoder half of another to obtain optimum results. This is difficult with the VfW architecture but can be overcome with a special wrapper (similar to the one provided with the VfW version of our DV decoder).
I agree that using one codec for encoding and another for decoding could prove to be difficult. Hmmm... Perhaps I should just have people upload their encoded DV files to my server. That will alleviate the contributor's burden and make sure the decoding and PSNR calculations are consistent.

The test as currently recommended doesn't really stretch the encoders to their limits. By feeding them with previously DV-compressed frames, they shouldn't have to do much (if any) compression at all. The hardest part of writing a quality DV encoder is selecting the right QNOs etc for each macroblock within a segment. To fully test this part of the encoder, you need to feed it with uncompressed material (i.e., that's never been compressed) and such material should consist of a variety of stresses.
I would argue that most people use DV encoders to re-encode and edit (which involves re-encoding if transitions are applied) existing DV content. Typically the first DV encoding is done in hardware, in the DV camera. This is a test of software encoders, which don't usually enter the picture until the editing process.

4. Ideally, any comparison of the encoders should be done at the encoded data level - i.e., statistically compare the DCT blocks (in the DCT domain), measure the amount of unused space in each segment (a segment is 5 macroblocks) and determine how efficiently the encoder made use of the three stages of compression (intrablock, intramacroblock and intermacroblock).
Yes, but I look at this comparison like this: if I want the best possible encoding quality for my DV editing project - which software DV encoder should I use? It's a simple question of how much image clarity is retained after a 1st re-encode. We could take the test even further and measure the image degradation after 2nd, 3rd, 4th and 5th re-encode - but I find that unnecessary. An encoder that does poorly on the 1st re-encode will not do any better on the 5th. :)

5. The comment about 4:2:0 interlaced vs. progressive isn't relevant for DV. It is only relevant for MPEG2.
Indeed, but the problem is always how the next component in the graph interprets that 4:2:0 video. Does it know it's DV 4:2:0? Or does it assume it's MPEG 4:2:0? Cedocida, for example, requires you to specify which version of 4:2:0 you're feeding the encoder if you choose to use YV12 as the input format.
Using YUY2 as both the input and ouput format just makes this a lot easier. Sure, there's the chroma downsampling and upsampling involved - but at least we don't have to deal with the ambiguity.

[zambelli]

OK, I decided to change the submission process and make it simpler for everybody: instead of calculating your own PSNR, just upload your encoded DV videos to me and I will do it.

In order to upload your videos, connect to ftp://ftp.citizeninsomniac.com and log in with this information:
Username: dv@citizeninsomniac.com
Password: doom9

Create a folder named as your Doom9 forum handle, and upload your videos there. If you're uploading videos from several encoders, make sure their filenames contain the encoder name.

[Blue_MiSfit]

I would argue that most people use DV encoders to re-encode and edit (which involves re-encoding if transitions are applied) existing DV content. Typically the first DV encoding is done in hardware, in the DV camera. This is a test of software encoders, which don't usually enter the picture until the editing process.


An EXCELLENT point. I also agree that it's important how a DV codec deals with never-been-compressed data, but much more so for the hardware codec in the camera itself... If only we could upload cedocica to our XL2s :)

I will get around to avid / apple compressing the rest of the samples. Been busy with a job search :p Later tonight!

~MiSfit

[JohnnyMalaria]

An EXCELLENT point. I also agree that it's important how a DV codec deals with never-been-compressed data, but much more so for the hardware codec in the camera itself... If only we could upload cedocica to our XL2s :)


I disagree. Whenever you make an transition or add titles or introduce any form of uncompressed footage, this is where the weaknesses of a DV encoder will show up. And, after all, this should be the time the encoder needs to be employed. The same is true of DV source material, too. Once you do anything to the decoded frame, the encoder has a lot more to do. This is where the encoders will show their merits and weaknesses.

[DSP8000]

I disagree. Whenever you make an transition or add titles or introduce any form of uncompressed footage, this is where the weaknesses of a DV encoder will show up. And, after all, this should be the time the encoder needs to be employed. The same is true of DV source material, too. Once you do anything to the decoded frame, the encoder has a lot more to do. This is where the encoders will show their merits and weaknesses.


Adding titles or transitions works differently than you think.
PAL DV 4:2:0 gets upsampled to 4:4:4 in the processing(transitions, etc) the rest of the video stays the same.
Now, this is a normal internal NLE process.
Adding transitions, titles with avisynth can be different, providing that you don't change the colorspace, you'll be just fine & the used enc/decoder will do exactly the same amount of work.

Your DV source is being decoded by your current DVSD codec on your system.Given the option to use ffdshow or Cedocida, you can choose how to decode or encode your DV footage.

Any decoder that conforms to a published specification should be encoder-agnostic. I think Cedocida decoder was written first and foremost as a generic DV25 decoder. There have been several tests done that have shown Cedocida decoder has consistently high quality and doesn't suffer from luma range mapping bugs and chroma upsampling issues commonly found in other DV decoders. This was my main reason for choosing it as the preferred decoder.

The main reason for picking one decoder and sticking with it is to keep the focus of the comparison on encoders.

100% SPOT ON

ffdshow can be used as DVSD decoder as well.
Keeping the DV source intact, using ONE enc/decoder through the editing/exporting process will keep you on the safe side & give you max quality of your video.


DSP8000

[Blue_MiSfit]

Perhaps we need to "spruce up" the source a bit with some cross dissolves? Perhaps doing it in the NLE, outputting uncompressed 4:2:2 YUY2, and then Huffying it for transfer to testers?

[JohnnyMalaria]

Adding titles or transitions works differently than you think.

Er...no.

I know EXACTLY what happens.

All conventional NLE systems will recompress the entire frame. Any part of the frame that has changed (e.g., because a title has been added) will affect the entire segment of the video frame. One segment is 5 macroblocks, one macroblock is 4 x luma 8x8 blocks and 2 chroma 8x8 blocks.

The smallest self-contained unit in DV compression is a segment.

As soon as you change just one pixel on the frame, an entire segment has to be re-encoded. Depending on the complexity of the change, the Huffman encoding length of the pre-quantized DCT coefficients may now be too large to fit into the segment space. The encoder will have to decide how much quantization to apply to the new data to fit within the confines of one DV segment. This is where the real weaknesses between encoders will show up since it is the part of the DV specification that is left open to interpretation - i.e., the selection of quantization factors is not defined. An encoder can choose the crudest way (a constant QNO for ever macroblock) or use a smart algorithm designed to maximize the distribution of compressed, Huffman codes across a segment.

This is why a simple re-encoding of a decoded frame is not a good test of the encoder.

A cross-dissolve may be reasonable test but a more stressful challenge would be a rotation (i.e., spinning) or a translation (as long as it isn't an exact multiple of 8 pixels).

FWIW, the codec system employed in our software ONLY recodes the segments that are affected, ALWAYS within the native 4:1:1 or 4:2:0 colorspace and, wherever possible, within the DCT frequency domain. This is how we are able to achieve real-time color correction, aspect ratio conversion, logo insertion etc whilst maintaining maximum quality.

[Blue_MiSfit]

Sorry I've taken so long.. the NTSC-Progressive results (Avid and Apple) are on the indicated FTP server. Do with them what you will!

PAL stuff coming up.

~MiSfit

[zambelli]

Sorry I've taken so long.. the NTSC-Progressive results (Avid and Apple) are on the indicated FTP server. Do with them what you will!
PAL stuff coming up.
Sorry for taking so long to get back to you - I was on vacation.

I'll check these out - I hope I can figure out a way to read the video streams from a MOV file via DShow.

In the meantime, I've also evaluated the Microsoft DV codec. Here are the results:


|-----------------------------------------------------------------------------------------|
| | Ovrl PSNR | Min PSNR | Max PSNR | Notes |
| NTSC Interlaced: |
|-----------------------------------------------------------------------------------------|
| Microsoft v6.05.2600 (DShow)| 46.5425 | 41.5732 | 51.3981 | RGB24, NTSC |
| | | | | |
| NTSC Progressive: |
|-----------------------------------------------------------------------------------------|
| Microsoft v6.05.2600 (DShow)| 46.9914 | 42.0759 | 51.0454 | RGB24, NTSC |
| | | | | |
| PAL Interlaced: |
|-----------------------------------------------------------------------------------------|
| Microsoft v6.05.2600 (DShow)| 38.9603 | 32.8893 | 52.4871 | RGB24, PAL |
| | | | | |
| PAL Progressive: |
|-----------------------------------------------------------------------------------------|
| Microsoft v6.05.2600 (DShow)| 39.6478 | 33.6012 | 51.9862 | RGB24, PAL |
| | | | | |
|-----------------------------------------------------------------------------------------|

Pretty decent results for NTSC, but terrible results for PAL. PAL users beware.

[zambelli]

This is why a simple re-encoding of a decoded frame is not a good test of the encoder.

A cross-dissolve may be reasonable test but a more stressful challenge would be a rotation (i.e., spinning) or a translation (as long as it isn't an exact multiple of 8 pixels).
A rotation would be good, but here's the thing: these are random frames so it's hard to say they're biased one way or the other. So far the results have been wildly different - measured in full dB points. I just don't believe that a codec that's 2-3 dB behind another codec would do much better if the source content was more challenging.

[zambelli]

I managed to get the MOVs that Blue_MiSfit sent me into AVI and then decode them with Cedocida. Here are the results:

Apple:
Overall 32.5580 dB
Min 31.7112 dB
Max 34.1999 dB

Avid:
Overall 32.6567 dB
Min 29.1042 dB
Max 35.5819 dB

Both samples were NTSC Progressive.

Those results looked spectacularly bad, so I did an AB comparison and found that in both cases luma range had been expanded. I'm guessing that HuffYUV got decoded to RGB as 16..235 to 0..255, but when the RGB video got converted to YUV DV - the range was mapped 0..255 to 0..255. That would explain the luma expansion.
Is this typical of the Apple platform or was it a user error?

[Blue_MiSfit]

Not sure... I never did colorspace conversions, but its possible that using the QTSource plugin's QTInput and QTOutput filters introduces colorspace conversions.

Could have been an error on my part as well.

[tateu]

I can't remember if it is my fault or quicktime's (probably mine) but QTOutput seems to handle the colors better if the source is RGB. Use ConvertToRGB32() on a line before QTOutput.

The same goes for QTInput. It produces better colors if "mode = 0" (RGB24) or "mode = 1" (RGB32) is used. Again, possibly due to something I screwed up.


Update:
Using:
AviSource("DV_NTSC_Progressive.avi")
ConvertToRGB32()
QTOutput("DV_NTSC_Progressive_QTOutput.mov")
produces a Quicktime mov file that is identical (PSNR = 107.04) to the one produced by loading DV_NTSC_Progressive.avi into After FX and rendering a quicktime mov file.

And the PSNR compare method (ConvertToYUY2 was used on the quicktime file because, as I said above, QTInput works better in RGB mode):
clip1 = AviSource("input.avi")
clip2 = QTInput("output.mov", color=0, quality=100, mode=0)
Compare(clip2.ConvertToYUY2(), clip1)

|-----------------------------------------------------------------------------------------|
| | Ovrl PSNR | Min PSNR | Max PSNR | Notes |
| NTSC Interlaced: |
|-----------------------------------------------------------------------------------------|
| Avid DV (Quicktime) | 46.52 | 42.13 | 50.35 | ITU-R 601 |
| | | | | |
| NTSC Progressive: |
|-----------------------------------------------------------------------------------------|
| Avid DV (Quicktime) | 47.17 | 42.03 | 50.46 | ITU-R 601 |
| | | | | |
| PAL Interlaced: |
|-----------------------------------------------------------------------------------------|
| Avid DV (Quicktime) | 35.90 | 30.92 | 37.64 | ITU-R 601 |
| | | | | |
| PAL Progressive: |
|-----------------------------------------------------------------------------------------|
| Avid DV (Quicktime) | 36.43 | 31.39 | 50.67 | ITU-R 601 |
| | | | | |
|-----------------------------------------------------------------------------------------|