Some codec idea

[dimzon]

Hi!
I have some idea. It can be useful for peoples who move their RIPs onto HTPC HDD and network storage. I think it wold be usefull to recompress loseless part of MPEG4/DivX3/MPEG2/MPEG1 using most powerfull ContextModeling algorythm... It will save some space on HDD without any quality loss. And anytime back conversion is possible. What does you think?

[unskinnyboy]

Not sure what you mean by "lossless part of MPEG4/DivX3/MPEG2/MPEG1". Context Modeling is already used in H.264/AVC (CABAC).

[dimzon]

Not sure what you mean by "lossless part of MPEG4/DivX3/MPEG2/MPEG1". Context Modeling is already used in H.264/AVC (CABAC).
Just read this (http://www.google.ru/url?sa=t&ct=res&cd=23&url=http%3A%2F%2Fwww-video.eecs.berkeley.edu%2FProceedings%2FICIP2003%2Fpapers%2Fcr2137.pdf&ei=OyrMR6vPB53GnAPAjJz6Dw&usg=AFQjCNGKLc9GDV8yiFGhVLTuZP3ovuf4-w&sig2=crFXBC3f0V5aRf9uPanEXA)
And keep in ming - there are many DivX/XviD rips with, AC3/MP3 tracks and they use huffman compression. Anyway it will be good way to save some space on HDD without reencoding/quality loss.

[unskinnyboy]

Huffman in Xvid/DivX? How is that? Huffman coding is used for lossless compression, so I am not sure how and why it would apply to Xvid/DivX.

About the paper, they are presenting GRASP as a CABAC alternative. So it should be looked at as a possible option to be incorporated into an existing H.264/AVC implementation (i.e. x264), rather than into a new codec.

[Dark Shikari]

Xvid/DivX/MPEG-4 ASP use VLCs, much like Huffman...

[foxyshadis]

dizmon, look to paq8 & the paq9 previews, they can reverse the huffyuv compression in some formats and replace it with more advanced compression models (beyond cabac but also much much slower). It's a mess to perfectly losslessly reverse because there are so many variants of VLC, and any corruption throws it all for a loop, but it's there if you want to start hacking more formats into it.

[anarco]

I´d be happy if there would be a possibility to reencode mpeg2 to mpeg4 without loss. This may sound silly but when using a beamer any artefact is just too much. Atm I rip the DVDs to harddisk for the htpc but they at least consume 4GB, most 6. Mpeg4 would take ~2.
I know Harddisk are cheap but it is still of interest when you can put 3 times more movies on it.

[dimzon]

Huffman in Xvid/DivX? How is that? Huffman coding is used for lossless compression, so I am not sure how and why it would apply to Xvid/DivX.
Dear unskinnyboy, seems like You doesn't understand me (maybe problem is my poor english)
Anyway, AFAIK all lossy compression consist of 2 parts
1) Lossy transformation (like DCT + applying quatizer matrix)
2) Lossless compression of (1)
Old codecs (such MPEG4 Part2, MPEG1/2 etc) uses Huffman compression for (2). Unpacking this part(2) and recompressing it using most powerfull loseless math (wich is ContextModeling ATM) we will save additional space without ANY quality loss (i think i's possible to save up to 15-30% for MPEG2/MPEG4 part2). Keep in mind - you doesn't need to touch part(1) so you doesn't perform any quality loss!
Anytime when you need your original file back you can perform backward transformation (unpack ContextModeling and repack to Huffman or something else what used for concrete codec standart). In PC playback case it's possible to create special decoders wich can decode such compressed files directly...

[unskinnyboy]

I think I understand you better now.

The thing is - if you are using any Context Modeling based entropy coding (CABAC, GRASP etc.), then you are in H.264 (or something similarly advanced) domain, not MPEG1/2/4-ASP. Also Context Modeling is quite computationally complex to encode and decode, which isn't really acceptable for MPEG1/2/4-ASP.

[foxyshadis]

Think of it as a lossless conversion to an h.264-like format then. Obviously it isn't acceptable for the majority of uses for older mpeg format, but for personal storage, hey, why not.

You could even write a directshow filter that pre-processes the format back into its original huffman/vlc coding for ease of playback. Then sell it as a lossless divx shrinking codec for $50.

[Dethis]

I think that Dimzon's idea has already been implemented in Stuffit's lossless compression of jpeg images.
Same technique must be and Soundslimmer's lossless compression for *.mp3 & *.mp4 audio files.
http://www.soundslimmer.com/

[unskinnyboy]

The more I thought about this, the more I understood dimzon's idea (at first, I totally went off on a tangent relating Huffman strictly to lossless compression as used in a lossless codec, as opposed to Huffman/VLC/RLE, as it would apply to DCT codecs).

So anyway, while the idea is excellent, it isn't new nor can it be implemented (yet) the way dimzon thinks it can be implemented. Like Dethis pointed out, PAQ8, PackJPG, StuffIt etc. already use it for image compression and SoundSlimmer supposedly use it for audio (their research paper is in Russian, which I can't make head or tails out of). But, all these create highly compressed archives, which aren't directly usable unless decoded losslessly back to the original format. What dimzon wants is a codec which can take the DCT coefficients from the bitstream and recompress it by replacing Huffman with CM. What would this end up creating though? Will it create a playable spec-compliant MPEG-1/2/4-ASP, without breaking compatibility with existing players? I doubt it. But if dimzon wants a high-quality video archiver, that may not be far ahead in the horizon.

[nm]

So anyway, while the idea is excellent, it isn't new nor can it be implemented (yet) the way dimzon thinks it can be implemented.
Sure it can be implemented that way.

But, all these create highly compressed archives, which aren't directly usable unless decoded losslessly back to the original format.
But these files wouldn't need to be just archives. The stream format can be designed so that the video can be played back without decompressing the whole stream. Just like in any video standard that uses lossless bitstream compression.

What dimzon wants is a codec which can take the DCT coefficients from the bitstream and recompress it by replacing Huffman with CM. What would this end up creating though?
A file that can be recoded to get the original input file (or at least one in the same format as the original file, with the same video data) or decoded for playback in real time.

Will it create a playable spec-compliant MPEG-1/2/4-ASP, without breaking compatibility with existing players?
Not without recoding the lossless part again, but being compliant with existing standards was not his idea anyway.

[dimzon]

What dimzon wants is a codec which can take the DCT coefficients from the bitstream and recompress it by replacing Huffman with CM.
Yes! And remux it back to container with special magic FourCC
What would this end up creating though? Will it create a playable spec-compliant MPEG-1/2/4-ASP, without breaking compatibility with existing players?
No, I will not be MPEG-1/2/4-ASP spec-compliant but you will able to play them directly onto your PC using special DirectShow filter wich can handle special magic FourCC ;)
So it will be good solution to store your media onto HDD (in case you have many XviD rips) and you can perform back transform (for HW players etc) any time you need/want...
Actually I think it will be good for some video formats (MPEG-1/2/4-ASP) and audio formats (MP3/AC3)

Talking about decoder filters - I think it's possible to create them using libav/ffmpeg - just replace huffman unpacker by CM unpacker, isn't it?

So I propose such fourCC for this "codec" family
R000 -> DivX
R001 -> DX50
R002 -> XviD
R003 -> DIV3
R004 -> DIV4
etc...

[dimzon]

So we need something like this (http://forum.doom9.org/showthread.php?s=&threadid=85229) for "transcoding" + some set of DirectShow decoders to decode on the fly

[benwaggoner]

I wonder if there's really that big a gap between the Huffman in existing codecs and the Shannon Limit for the same content (determining maximum lossless compression). What are we talking about for real-world file savings here? My SWAG would be 10-15%, which isn't that big a difference considering the increased decode times and loss of general compatibility.

If you're going that far, why not make it visually lossless and allow some additional tweaks that aren't exactly 1:1 but increase efficiency overall.

[Dark Shikari]

Moreso, remember that the original encoder probably used some form of RD optimization which optimized for the specific entropy encoder used; this efficiency gain will probably be lost at least in part when using a different entropy encoder.

[MiroLx]

Hello.
I think dimzon idea can work. Replacing Huffman's code with some type of range coder you will never lost and you will gain in most cases. But 15-30% is too optimistic. Maybe 5% or so...

[dimzon]

). What are we talking about for real-world file savings here? My SWAG would be 10-15%, which isn't that big a difference considering the increased decode times and loss of general compatibility.
As I said before It can be useful for peoples who store their RIPs onto HTPC HDD and network storage. Talking about 2TB storage we will get 100-300 additional GB isn't it?

[akupenguin]

I think dimzon idea can work. Replacing Huffman's code with some type of range coder you will never lost and you will gain in most cases. But 15-30% is too optimistic. Maybe 5% or so...
CABAC is 10-15% better than CAVLC, and CAVLC is better than mpeg2/4's VLC, so that's a minimum bound on the improvement possible. You can even get 5% compression of mpeg2 with a generic compression format like LZMA, that's how backwards it is.
CABAC isn't the limit either, it made some compromises for speed and error resilience (e.g. limited number of contexts, and no preserving state between frames).

[dimzon]

CABAC is 10-15% better than CAVLC, and CAVLC is better than mpeg2/4's VLC, so that's a minimum bound on the improvement possible. You can even get 5% compression of mpeg2 with a generic compression format like LZMA, that's how backwards it is.
CABAC isn't the limit either, it made some compromises for speed and error resilience (e.g. limited number of contexts, and no preserving state between frames).
akupenguin, is it possible to hack libavcodec to use at least CABAC from x264 for MPEG2/MPEG4ASP streams?

[foxyshadis]

It's possible to hack lavc to do anything. It shouldn't be tremendously difficult to flip a 'reserved' flag in the bitstream for private use, and use that to send the stream to a cabac decoder. It'd probably be a lot more difficult to convert the files to use cabac in the first place.