smaller alternative to DV

[MatzeXXX]

Hi!

From time to time I create showreel videos for actors and actresses - it's more of a hobby which brings in some money. I am no professional though.

All those DV clips are starting to really fill up my hard drive (it's a 500GB external HDD, and I DON'T wanna buy a second one), and I am considering transcoding them to something smaller, preferably about a third of the size of DV files. I looked into MJPEG and MJPEG2000, which both seem to be suited.

It has to be frame bases encoding, and is has to be some kind of global codec (e.g. ffdshow codecs won't work in many adobe applications).

Quality and speed are both equally important. I wouldn't let my computer transcode for a week instead of a day just for a slight gain in quality...

Oh, and if the solution would be free - that's even better!

Thanks for any advice!

Matthias

[benwaggoner]

The nice thing about DV is that it's fast, I-frame only, and lossless compared to what you've already got :).

You could do something like VC-1 1-pass VBR, which does interlaced is built into Windows, and is well supported in Premiere. Keyframe-only might give you 2-3x better efficiency at a quality you can live with, but I've never tried.

To get the space savings you want, short closed GOP encoding could work better - like 15-30 frames per GOP.

Still, it's a lot of work to convert all that stuff, and stoage is pretty darn cheap these days. It might be cheaper from a time-is-money perspective to just buy a few cheap 1 TB external drives. Or, cheaper, yet, just dub back to DV tape.

[Dark Shikari]

H.264 is perfect for the job--if it has to be intra-only like DV, H.264 is currently the best intra compression format out there. Most good editing programs will accept it as input as well.

DV is so awful as a compression format that you can actually losslessly gain a lot of space (from what I recall) just by 7zipping it.

[JohnnyMalaria]

It's not so much that the compression is bad - it's because the DV stream contains additional non-AV data that, in many cases, is unused hence the ability to shrink the file by zipping it. For the video part, it's no different than JPEG.

Anyhoo, for me the best option is just keep the original tapes. I have tapes more than 11 years old and they play just like the day they were recorded. I also keep copies on an external 500GB USB2.0 hard drive - that's nearly 40 tapes. Dirt cheap, really.

[akupenguin]

For the video part, it's no different than JPEG.
You wish. DV is very constant bitrate -- not just on the frame level, but each macroblock has to have exactly the same number of bits. That cripples quality compared to even a constant-quant JPEG.

[JohnnyMalaria]

I mean in a more general term - namely quantization and Huffman entropy encoding of 8x8 DCT blocks. Once done, there's bugger all left to further compress with zip (also an entropic compression scheme) etc. The fixed bit count for a macroblock is misleading. Each encoded 8x8 block can be stored in three regions: its own assigned block, the remaining space in its parent macroblock (6 blocks - 4Y + Cr + Cb) (after the first pass of the other blocks have been stored) and then the remaining space of the superblock than contains five macroblocks. i.e., one block can be spread over 30 blocks. Each macroblock can have its own quantization level and the only space requirement is that all 30 blocks have to fit within the bit space provided by a superblock. The arrange of the macroblocks in a superblock is designed to combined portions of the image with high detail with those of lower detail. So, in effect, DV is a form of JPEG that permits variable quantization within a given image. In fact, each 16 x 16 (PAL) or 32 x 8 (NTSC) region gets its own quantization number (QNO) and each block a quantization class. That's a far cry from forcing each macroblock to have the same number of bits. The overall compression ratio for an 24-bit RGB using the DV scheme is 9.97 to 1. It is optimized for images with significant regions of low detail and others of high detail. Unfortunately, although the decoding of DV is deterministic, encoding isn't and it is up to the encoder developer to implement the selection of quantization parameters to squeeze as much into the superblock space as possible.

[Dark Shikari]

Once done, there's bugger all left to further compress with zipThis assumes the format is perfectly optimal, which it is not. Merely the fact that it requires filler bits everywhere ensures you can compress it further with ZIP; even MPEG-2 can generally be compressed another 10 percent using LZMA and twice that with PAQ. So, in effect, DV is a form of JPEG that permits variable quantization within a given image.Isn't JPEG a form of JPEG that permits variable quantization within a given image?

[JohnnyMalaria]

It can be but typically isn't. By default, a JPEG image has two quantization tables - one for luma and one for chroma. They are applied across the entire image. DV also has two tables (one for 8x8 and one for 2x4x8) but these can be scaled on a block-by-block basis. This permits regions of higher detail to receive less quantization and flatter regions to receive more.

The quantization tables for JPEG can be anything you like - as can the Huffman tables. For DV, they are prescribed.

DV and JPEG are very close relatives. DV has one further optimization for video - encoding of interlaced blocks with 2x4x8 DCTs instead of 8x8.

Regarding further compression, you may squeeze an extra 10% or so for a DV file (15% of a DV stream is non-video) unless your source is of low detail in the first place. A pure black encoded as DV will have a lot of runs of FFh bytes. But for typical real-world video, the gain may not be much and such gain is due to the non-video regions.

[Dark Shikari]

Using this extremely noisy interlaced DV source: voxnews.dv (http://samples.mplayerhq.hu/DV-raw/voxnews.dv)

I just clipped off the first ~9MB of the clip because I was too lazy to download the whole thing on my slow connection.

Original: 9148117 bytes
Gzip: 8446109 bytes (7.5% compression)
7z: 7870712 bytes (14% compression)
PAQ8: 7227960 bytes (21% compression)

[JohnnyMalaria]

I got about 8% for the whole file using the zip built in to Windows (since that meets my needs :) ).

I compressed a dozen or so of my own files (safari and underwater, ranging 30 - 800MB) at got 1.84GB for 2.02GB (9%). Based on the DV format, this is what I'd expect. The breakdown is:

Compressed video: 85%
Audio: 5%
Remaining stuff that no-one ever uses (e.g., teletext, closed captions, professional camera settings, chapters, scenes, genres): 10%

Most of the latter ends up being a long string of FF bytes (77 + 3 for DIF header).

Throughout an encoded frame, there are, of course, repeating sequences of 3 or 4 bytes at predictable positions which could be further compressed. But encoding DV needs to occur in real-time and isn't deterministic. That's why DV recorded by one camcorder isn't necessarily the same as on another. Higher quality implementations employ algorithms that require an almost trial-and-error approach to minimize the quantization within a superblock. Thankfully (if you see it that way), the constant bit rate applies to a superblock - it isn't necessary to optimize the entire image, just a superblock. Decent encoding requires 3 passes as well as (optionally) identification of strongly interlaced regions. Lesser quality encoders (hardware or software) leave unused sections of superblocks. Like with MPEG schemes, most of the effort is put in to the encoding while the decoding is fairly lightweight.

[benwaggoner]

I wonder if there's a market for a smart fast ~lossless DV storage codec. Something deeper than just file compression, but that started with bitstream and tried to apply some simple interframe compression, much better entropy coding, some context-adaptive compression for metadata versus video. Especially if you didn't need to be exactly lossless but within a small threshold.

[Dark Shikari]

I wonder if there's a market for a smart fast ~lossless DV storage codec. Something deeper than just file compression, but that started with bitstream and tried to apply some simple interframe compression, much better entropy coding, some context-adaptive compression for metadata versus video. Especially if you didn't need to be exactly lossless but within a small threshold.You should easily be able to get 30% or better compression on both DV and MPEG-1/2 simply by replacing the entropy coder with a heavily context-adaptive arithmetic one with a few hundred thousand contexts (a'la FFV1).

For example, for MPEG-1 (for which I know the syntax elements well enough to come up with a coder on the spot), you could do the following:

(let IsMBType return 0 for Inter, 1 for Intra, and 2 for Skip)
Context for intra vs inter: IsTopMBType + IsLeftMBType*3 + IsTopLeftMBType*9 + IsTopRightMBType*27
(81 total)

(let IsMBIntra return 0 for Inter, 1 for Intra, and 2 for Skip)
Context for skip: IsTopMBType + IsLeftMBType*3 + IsTopLeftMBType*9 + IsTopRightMBType*27
(81 total)

Context for CBP (for any given 8x8 dct block): IsTopBlockCoded + IsLeftBlockCoded*2 + IsTopLeftBlockCodedp*4 + IsTopRightBlockCoded*8 + IsChroma*16 + IsIntra*32
If we want more, maybe we can add the Type from above for 64*81 contexts, but that might be too many.
(64 total)

Context for motion vector: Clip(Log2( abs(MVD(Top)) + abs(MVD(Left)) ), 0, 7) + 8 * Clip(Log2( abs(MV(Top)) + abs(MV(Left)) ), 0, 7)
(note as motion vector is a symbol not a binary value, each "context" has 32 subcontexts for ffv1-style put_symbol)
(64 total)*(32 subcontexts)

BigContext for residual:

H.264 style residual coding (but better):
Significance Map: 63 contexts
Last Map: 63 contexts
Abs_level contexts: dunno, similar to H.264 but lots of them (would have to think about it more)

Which BigContext to choose would be chosen by Log2(NonzeroCountTopBlock) + 8*Log2(NonzeroCountLeftBlock) + 64*IsChromaBlock.
(128 BigContext total)

[Blue_MiSfit]

Hmm... maybe you could use H.264 for the actual video stream, FLAC or some other lossless codec for the audio (or AAC), and some other jazz for the meta.

Interesting idea!

~MiSfit

[akupenguin]

apply some simple interframe compression
H.264 Switching Pictures to the rescue. Although to make it fast you might want to just do dct-domain prediction without motion vectors.

[MatzeXXX]

H.264 is perfect for the job--if it has to be intra-only like DV, H.264 is currently the best intra compression format out there. Most good editing programs will accept it as input as well.

DV is so awful as a compression format that you can actually losslessly gain a lot of space (from what I recall) just by 7zipping it.

This is interesting! I haven't thought about this: I could just compress the drive and really gain something by it. My computer is fast enough, and there's mostly DV on that drive, which might compress ok.

Is the compression part of NTFS? Or would I run into problems using the drive with some other PC or Mac?

Matthias

[Esurnir]

This is interesting! I haven't thought about this: I could just compress the drive and really gain something by it. My computer is fast enough, and there's mostly DV on that drive, which might compress ok.

Is the compression part of NTFS? Or would I run into problems using the drive with some other PC or Mac?

Matthias

Drive compression is part of ntfs.

[MatzeXXX]

Thanks for your reply!

I think it's also worth it because USB2 is the real bottleneck when it comes to speed. The drive is slighty faster when less data is transferred over the connection, and the decompression is done by my computer in no time.

[mariush]

This is interesting! I haven't thought about this: I could just compress the drive and really gain something by it. My computer is fast enough, and there's mostly DV on that drive, which might compress ok.

Is the compression part of NTFS? Or would I run into problems using the drive with some other PC or Mac?

Matthias

ntfs compression won't give you much compression. It has quite a lot of limitations (like a certain number of consecutive sectors must be compressible otherwise everything's stored uncompressed) and compresses less than zip.