Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > DV

Reply
 
Thread Tools Search this Thread Display Modes
Old 1st March 2012, 15:18   #1  |  Link
VonOben
Registered User
 
Join Date: Sep 2008
Posts: 40
Lossless compression that works for DV?

Hi.

I haven't done any testing, but I will, eventually. Perhaps someone has already tried this...

Are there any lossless compression algoritms that compress DV content to any degree? Since there are no temporal compression in DV I guess this should be possible, depending on the source.

Has anyone tried? 7z? Zip? Rar?

Edit: Did some quick testing on a random 25s VHS capture. WinRAR using compression "Best" shrinks it to 79% of the original size.

The same clip compressed using setting "Fastest" renders a file that's 81% of the original. This leads me to believe that it's DV overhead that are being compressed, not the video content itself. The compression window might be far to small far that. (Or it's not very compressable.)

Edit#2: Someone got 60% back in 2005 with 7z... http://forum.doom9.org/showthread.php?t=92590

Last edited by VonOben; 1st March 2012 at 15:57. Reason: Own testing
VonOben is offline   Reply With Quote
Old 1st March 2012, 17:36   #2  |  Link
VonOben
Registered User
 
Join Date: Sep 2008
Posts: 40
71% with Z-paq, but it's far to slow to be useful. (With my setup.)

~75% with 7-zip, various settings. Decent speed.

Last edited by VonOben; 1st March 2012 at 17:51.
VonOben is offline   Reply With Quote
Old 2nd March 2012, 00:42   #3  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,248
DV is already compressed. In a lossy way.

As far as I know, DV compression is pretty similar to Motion-JPEG. So putting DV files into a Zip archive is like putting JPEG files into a Zip. You won't get much additional compression out of that

Actually ~75% compression with 7-Zip is pretty high for an already compressed video format! It indicates that the "entropy coder" used by DV is not very good!

And of course the compression ration that can be achieved with 7-Zip can vary for different DV files. It also can vary with different 7-Zip settings - using a bigger dictionary size might help a bit.

But I think you won't get much more compression in a lossless way. Converting from the lossy DV format to a lossless video format, such as HuffYUV or FFV1, will certainly result in a much bigger file.

So the best you can do to "shrink" your file is re-encoding to another lossy format, which can retain the content of your DV file at significant smaller size with only minimal or negligible quality loss.

The H.264/AVC format, encoded with x264 (in CRF mode, with a suitable CRF value) probably is the most promising candidate here
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊

Last edited by LoRd_MuldeR; 2nd March 2012 at 00:48.
LoRd_MuldeR is offline   Reply With Quote
Old 12th March 2012, 10:05   #4  |  Link
VonOben
Registered User
 
Join Date: Sep 2008
Posts: 40
Quote:
Originally Posted by LoRd_MuldeR View Post
DV is already compressed. In a lossy way.

As far as I know, DV compression is pretty similar to Motion-JPEG.
I'm with you on this, but each frame in DV (and I guess in MJPG) are (lossy) compressed on its own. AFAIK there are no temporal lossless (or lossy for that matter) compression applied at all, that what makes them suitable for editing.

I think, or thought, that there might be a way to implement a lossless compression that looks at DV-frames mainly in the temporal way, and it would be possible to compress it a bit.

Then again, large dictionaries with 7z and the likes might do this, but I'm not sure how large their "working windows" are in kilobytes. Will they fit several frames inside RAM while trying to find patterns, or will the buffer not be large enought for that?

Anyhow, as you say, 75% isn't to shabby. After all, on a 100Gbyte file that's quite a lot of gigabytes saved.

(Written fast and will no spell checking. Now back to work.)

Edit: One could say that DV are compressed in X and Y direction, lossy. But no compression, not lossy nor lossless, are applied in the Z direction. (X = Horizontal, Y = Vertical, Z = Time). At least this is how I think it works.

Last edited by VonOben; 12th March 2012 at 10:11.
VonOben is offline   Reply With Quote
Old 18th March 2012, 01:19   #5  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,248
Quote:
Originally Posted by VonOben View Post
I'm with you on this, but each frame in DV (and I guess in MJPG) are (lossy) compressed on its own. AFAIK there are no temporal lossless (or lossy for that matter) compression applied at all, that what makes them suitable for editing.
Yes, DV is intra-frame only (very similar to MJPEG).

Quote:
Originally Posted by VonOben View Post
I think, or thought, that there might be a way to implement a lossless compression that looks at DV-frames mainly in the temporal way, and it would be possible to compress it a bit.
Sure, but not without decompressing the already-compressed frames first

The data stored in a DV file is not pixel data, it is compressed data that pretty much looks like "random" bits!

So you can't do any "temporal" compression on that, unless you decompress the compressed data first - in order to retain the pixel data.

But again: Decompressing the existing lossy DV file and then re-compressing it to a lossless format (e.g. HuffYUV or FFV1) will result in much WORSE compression ratio

If you re-compress anyway, use a high-efficient lossy format, such as H.264. It will be able store your DV at much smaller size with only minimal quality loss...

Quote:
Originally Posted by VonOben View Post
Then again, large dictionaries with 7z and the likes might do this, but I'm not sure how large their "working windows" are in kilobytes. Will they fit several frames inside RAM while trying to find patterns, or will the buffer not be large enought for that?
File archivers, like 7-Zip, WinRAR and friends, can't get any significant compression on already-compressed image, video or audio data.

That's because all compressed multimedia formats (lossless or not) use entropy compression. Anything that happens before that, e.g. in the "lossy" step, is just intended to improve the final entropy compression.

JPEG uses Huffman Coding in the last step, for example. Arithmetic Coding is supported optionally, but rarely used in reality.

After the entropy compression, you get data that look like "random" bits and therefore another entropy compression, like 7-Zip, won't be able to further compress that. Dictionary compression doesn't help either.

Quote:
Originally Posted by VonOben View Post
Anyhow, as you say, 75% isn't to shabby. After all, on a 100Gbyte file that's quite a lot of gigabytes saved.
Actually an additional compression ratio of 75% (with a lossless file compressor) is surprisingly high for already-compressed multimedia data!

It only shows that the entropy coder used by DV is amazingly bad

(Try the same with a H.264 and stream, for example, an you will get compression ratio of no better than 99%)
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊

Last edited by LoRd_MuldeR; 18th March 2012 at 01:34.
LoRd_MuldeR is offline   Reply With Quote
Old 18th March 2012, 23:33   #6  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,167
Quote:
It only shows that the entropy coder used by DV is amazingly bad
Not a totally fair assessment. DV is a CBR format at 28800kbps (iirc) so the encoder will stop squeezing when it achieves the bit target for the frame. Complex frames will get squeezed hard and may not compress much further. Simple frames will not get squeezed much and may compress a lot more. And as the format is DCT based the compression happens in the frequency domain so it is hard to access complexity from the images. Also 28800kbps is quite generous.
IanB is offline   Reply With Quote
Old 26th March 2012, 08:13   #7  |  Link
VonOben
Registered User
 
Join Date: Sep 2008
Posts: 40
Quote:
Originally Posted by LoRd_MuldeR View Post
After the entropy compression, you get data that look like "random" bits and therefore another entropy compression, like 7-Zip, won't be able to further compress that. Dictionary compression doesn't help either.
I'm with you on most of these things.

However, imagine a completely black stream. Compressed DV frame #1, somewhere around 125kbyte large for PAL, will contain exactly the same data as compressed frame #2.

As they both contains the same data uncompressed.

That's compressable, temporally. (And since it's CBR also non-temporal-wize, but that's out-of-scope in this context.) Perhaps not a valid real life example, but it's a way to describe how I'm thinking.
VonOben is offline   Reply With Quote
Old 26th March 2012, 08:15   #8  |  Link
Ghitulescu
Registered User
 
Ghitulescu's Avatar
 
Join Date: Mar 2009
Location: Germany
Posts: 5,769
Why would you tape black frames?
__________________
Born in the USB (not USA)
Ghitulescu is offline   Reply With Quote
Old 26th March 2012, 15:56   #9  |  Link
kypec
User of free A/V tools
 
kypec's Avatar
 
Join Date: Jul 2006
Location: SK
Posts: 826
Quote:
Originally Posted by Ghitulescu View Post
Why would you tape black frames?
Maybe because he forgot to take off the lens cover?
Sorry for OT, I just couldn't resist to post this silliness.
kypec is offline   Reply With Quote
Old 29th March 2012, 18:17   #10  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,248
Quote:
Originally Posted by VonOben View Post
I'm with you on most of these things.

However, imagine a completely black stream. Compressed DV frame #1, somewhere around 125kbyte large for PAL, will contain exactly the same data as compressed frame #2.

As they both contains the same data uncompressed.
That is highly hypothetical. Even if you record in complete darkness (or with the lens cover on) there still will be some noise from the optical sensor.

This will cause the uncompressed data to be slightly different, at least. And I would assume that after the entropy coding that difference will be large enough to botch another layer of compression.

Also: How often do you record in complete darkness to generate a sequence of "black" frames ???
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊

Last edited by LoRd_MuldeR; 29th March 2012 at 18:19.
LoRd_MuldeR is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 21:15.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.