Lossless compression that works for DV?

[VonOben]

Hi.

I haven't done any testing, but I will, eventually. Perhaps someone has already tried this...

Are there any lossless compression algoritms that compress DV content to any degree? Since there are no temporal compression in DV I guess this should be possible, depending on the source.

Has anyone tried? 7z? Zip? Rar?

Edit: Did some quick testing on a random 25s VHS capture. WinRAR using compression "Best" shrinks it to 79% of the original size.

The same clip compressed using setting "Fastest" renders a file that's 81% of the original. This leads me to believe that it's DV overhead that are being compressed, not the video content itself. The compression window might be far to small far that. (Or it's not very compressable.)

Edit#2: Someone got 60% back in 2005 with 7z... http://forum.doom9.org/showthread.php?t=92590

[VonOben]

71% with Z-paq, but it's far to slow to be useful. (With my setup.)

~75% with 7-zip, various settings. Decent speed.

[LoRd_MuldeR]

DV is already compressed. In a lossy way.

As far as I know, DV compression is pretty similar to Motion-JPEG. So putting DV files into a Zip archive is like putting JPEG files into a Zip. You won't get much additional compression out of that ;)

Actually ~75% compression with 7-Zip is pretty high for an already compressed video format! It indicates that the "entropy coder" used by DV is not very good!

And of course the compression ration that can be achieved with 7-Zip can vary for different DV files. It also can vary with different 7-Zip settings - using a bigger dictionary size might help a bit.

But I think you won't get much more compression in a lossless way. Converting from the lossy DV format to a lossless video format, such as HuffYUV or FFV1, will certainly result in a much bigger file.

So the best you can do to "shrink" your file is re-encoding to another lossy format, which can retain the content of your DV file at significant smaller size with only minimal or negligible quality loss.

The H.264/AVC format, encoded with x264 (in CRF mode, with a suitable CRF value) probably is the most promising candidate here :)

[VonOben]

DV is already compressed. In a lossy way.

As far as I know, DV compression is pretty similar to Motion-JPEG.

I'm with you on this, but each frame in DV (and I guess in MJPG) are (lossy) compressed on its own. AFAIK there are no temporal lossless (or lossy for that matter) compression applied at all, that what makes them suitable for editing.

I think, or thought, that there might be a way to implement a lossless compression that looks at DV-frames mainly in the temporal way, and it would be possible to compress it a bit.

Then again, large dictionaries with 7z and the likes might do this, but I'm not sure how large their "working windows" are in kilobytes. Will they fit several frames inside RAM while trying to find patterns, or will the buffer not be large enought for that?

Anyhow, as you say, 75% isn't to shabby. After all, on a 100Gbyte file that's quite a lot of gigabytes saved. :)

(Written fast and will no spell checking. Now back to work.)

Edit: One could say that DV are compressed in X and Y direction, lossy. But no compression, not lossy nor lossless, are applied in the Z direction. (X = Horizontal, Y = Vertical, Z = Time). At least this is how I think it works.

[LoRd_MuldeR]

I'm with you on this, but each frame in DV (and I guess in MJPG) are (lossy) compressed on its own. AFAIK there are no temporal lossless (or lossy for that matter) compression applied at all, that what makes them suitable for editing.

Yes, DV is intra-frame only (very similar to MJPEG).

I think, or thought, that there might be a way to implement a lossless compression that looks at DV-frames mainly in the temporal way, and it would be possible to compress it a bit.

Sure, but not without decompressing the already-compressed frames first ;)

The data stored in a DV file is not pixel data, it is compressed data that pretty much looks like "random" bits!

So you can't do any "temporal" compression on that, unless you decompress the compressed data first - in order to retain the pixel data.

But again: Decompressing the existing lossy DV file and then re-compressing it to a lossless format (e.g. HuffYUV or FFV1) will result in much WORSE compression ratio :eek:

If you re-compress anyway, use a high-efficient lossy format, such as H.264. It will be able store your DV at much smaller size with only minimal quality loss...

Then again, large dictionaries with 7z and the likes might do this, but I'm not sure how large their "working windows" are in kilobytes. Will they fit several frames inside RAM while trying to find patterns, or will the buffer not be large enought for that?

File archivers, like 7-Zip, WinRAR and friends, can't get any significant compression on already-compressed image, video or audio data.

That's because all compressed multimedia formats (lossless or not) use entropy compression. Anything that happens before that, e.g. in the "lossy" step, is just intended to improve the final entropy compression.

JPEG uses Huffman Coding (http://en.wikipedia.org/wiki/Huffman_coding) in the last step, for example. Arithmetic Coding is supported optionally, but rarely used in reality.

After the entropy compression, you get data that look like "random" bits and therefore another entropy compression, like 7-Zip, won't be able to further compress that. Dictionary compression doesn't help either.

Anyhow, as you say, 75% isn't to shabby. After all, on a 100Gbyte file that's quite a lot of gigabytes saved. :)

Actually an additional compression ratio of 75% (with a lossless file compressor) is surprisingly high for already-compressed multimedia data!

It only shows that the entropy coder used by DV is amazingly bad :rolleyes:

(Try the same with a H.264 and stream, for example, an you will get compression ratio of no better than 99%)

[IanB]

It only shows that the entropy coder used by DV is amazingly badNot a totally fair assessment. DV is a CBR format at 28800kbps (iirc) so the encoder will stop squeezing when it achieves the bit target for the frame. Complex frames will get squeezed hard and may not compress much further. Simple frames will not get squeezed much and may compress a lot more. And as the format is DCT based the compression happens in the frequency domain so it is hard to access complexity from the images. Also 28800kbps is quite generous.

[VonOben]

After the entropy compression, you get data that look like "random" bits and therefore another entropy compression, like 7-Zip, won't be able to further compress that. Dictionary compression doesn't help either.

I'm with you on most of these things.

However, imagine a completely black stream. Compressed DV frame #1, somewhere around 125kbyte large for PAL, will contain exactly the same data as compressed frame #2.

As they both contains the same data uncompressed.

That's compressable, temporally. (And since it's CBR also non-temporal-wize, but that's out-of-scope in this context.) Perhaps not a valid real life example, but it's a way to describe how I'm thinking. :)

[Ghitulescu]

Why would you tape black frames?

[kypec]

Why would you tape black frames?
Maybe because he forgot to take off the lens cover?
Sorry for OT, I just couldn't resist to post this silliness.:p

[LoRd_MuldeR]

I'm with you on most of these things.

However, imagine a completely black stream. Compressed DV frame #1, somewhere around 125kbyte large for PAL, will contain exactly the same data as compressed frame #2.

As they both contains the same data uncompressed.

That is highly hypothetical. Even if you record in complete darkness (or with the lens cover on) there still will be some noise from the optical sensor.

This will cause the uncompressed data to be slightly different, at least. And I would assume that after the entropy coding that difference will be large enough to botch another layer of compression.

Also: How often do you record in complete darkness to generate a sequence of "black" frames ??? ;)