Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > General > Newbies

Reply
 
Thread Tools Search this Thread Display Modes
Old 25th August 2012, 15:24   #1  |  Link
lulalala
Registered User
 
Join Date: Aug 2012
Posts: 11
Want to make an image archival format using x264

Summary in one sentence: I want to make an image archival format which supports random access. How can I achieve this using x264?

Long ver.:
I was looking for a format to archive a set of BMP images. The set contains many subset of very similar images. So I am looking for an archiving format which takes advantage of differential storage, but also have a good random access rate.

I was using 7z (solid mode) which compress great but does not support random access.

My conceptual format works like this. Say there are 10 similar images , A0, A1 to A9. The archive will store the whole of A0, and the difference A0-A1, A0-A2 and so on. When accessing A4, it will access A0, and then access the difference A0-A4. It will output the A4 by merging the difference.

I asked this in the compression board and they told me to investigate in modern video formats. I have self-studied a few days. Here are my questions, grouped in theory/practical /grand level.

Theoretical level:
I learned basic theory of video encoding.
I learned that x264 supports RGB and lossless.
But can h264 allow having one I-frame and have subsequent P-frames referencing to that I frame. So I won't need to read several frame before decoding one frame.
If I have several subsets, can x264 detect that and make something like IPPPPPIPPIPPP? If not what are the possible peusdo steps to achieve that(a wrapper to preprocess then feed to x264?)

On the practical level(of making such a file by hand):
I also learned that avisynth is used to have bmp source converted to raw video, and feed it to MeGUI. Is this the right path?

On the grand level(making programs to automate decoding/encoding):
I want to make a C# compression program for users. I guess this program has to generate avisynth script and pass it to megui library?
I also want to make a C# decompression program, decompressing the format to bmp or jpeg. Do I just use any h264 decoder to do that?
Is is easy to use all of these under C#?


I know the questions are a mixture of different areas/difficulties, and may not be written as clear as it can be. Sorry about that and thanks for reading this.
lulalala is offline   Reply With Quote
Old 25th August 2012, 17:57   #2  |  Link
Phantom_E
Registered User
 
Join Date: Jul 2004
Posts: 41
Quote:
Originally Posted by lulalala View Post
I was using 7z (solid mode) which compress great.
I'm somewhat surprised by the "great" comment.
My experience in lossless compression of image files gives an average of < 10% and in individual cases can actually result in a larger output file. A scan of the net shows that solid mode can do slightly better than basic PkZip, but personally I don't think the difference is worth writing home about.
Feel free to hold a different opinion.


Quote:
Originally Posted by lulalala View Post
I learned that x264 supports RGB and lossless.
Just some thought on lossless.
Is your original source digital?
Even if it was a digital source some of the following still applies.

Digitization is inherently lossy. The major issue is the Nyquist limit, but there are others involving the physical hardware employed in the processing (or capture).
Basically, If you do 10 successive scans of the same image you will get 10 different files.
If may be possible to improve compression just by averaging successive scans, but I have not tried this.

Lossless in terms of video processing only applies to how much degradation subsequent processing will cause.
If this is a one off archive then only the losses for the decoding step apply. The display device will introduce some loss.
If multiple decoding and then recoding steps apply then lossless increases in importance.

In these days of multi terabyte storage units and Blueray, do you have a need for compression at all?

Part of your research should include just how much loss you can have and still retain the level of detail required.
None is not one of your options.

Good luck with your project.
Phantom_E is offline   Reply With Quote
Old 25th August 2012, 19:08   #3  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
x264's lossless mode is not very good at intra compression. FFV1 will probably do much better.
Dark Shikari is offline   Reply With Quote
Old 26th August 2012, 03:33   #4  |  Link
lulalala
Registered User
 
Join Date: Aug 2012
Posts: 11
Quote:
Originally Posted by Phantom_E View Post
My experience in lossless compression of image files gives an average of < 10% and in individual cases can actually result in a larger output file. A scan of the net shows that solid mode can do slightly better than basic PkZip, but personally I don't think the difference is worth writing home about.
Yes my sources are digital. When difference between frames are small, and each subset contains 50+ similar images, I can compress 200mb BMP files to a 6 mb 7zip file. Solid mode acts like IPPPP..., just that it is not random accessible.

Quote:
Originally Posted by Phantom_E View Post
Is your original source digital? Even if it was a digital source some of the following still applies.
Digitization is inherently lossy. The major issue is the Nyquist limit, but there are others involving the physical hardware employed in the processing (or capture).
Basically, If you do 10 successive scans of the same image you will get 10 different files.
Yes my sources are digitally created images. Therefore scanning is irrelevant.

Quote:
Originally Posted by Phantom_E View Post
Lossless in terms of video processing only applies to how much degradation subsequent processing will cause.
If this is a one off archive then only the losses for the decoding step apply. The display device will introduce some loss.
If multiple decoding and then recoding steps apply then lossless increases in importance.
So is x264 lossless mode not really lossless? My file format is design to be decompressed by the decompressor I want to write, not viewed by video decoder. So I thought I can make it to output without quality loss.

Quote:
Originally Posted by Phantom_E View Post
In these days of multi terabyte storage units and Blueray, do you have a need for compression at all?

Part of your research should include just how much loss you can have and still retain the level of detail required.
None is not one of your options.
I plan to have true lossless, just like 7z. From your reply it seems I misunderstood x264's lossless mode. Can you give some references that I can study?

Quote:
Originally Posted by Phantom_E View Post
Good luck with your project.
Thanks. Is it okay if I move this to the x264 board? The theory questions may be answered over there?
lulalala is offline   Reply With Quote
Old 26th August 2012, 03:40   #5  |  Link
lulalala
Registered User
 
Join Date: Aug 2012
Posts: 11
Quote:
Originally Posted by Dark Shikari View Post
x264's lossless mode is not very good at intra compression. FFV1 will probably do much better.
Thanks. Well I can accept tradeoff between compression ratio to random accessibility.

FFV1 development activity seems to be stopped since 6 years ago. I'll give it a shot if x264 really didn't work out.

Since you are a developer, may I ask if you know the answer to my theory part of the questions? Thanks
lulalala is offline   Reply With Quote
Old 26th August 2012, 03:58   #6  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
There's no real advantage to compressing many unrelated images in a single archive, and similarly, inter prediction is kind of useless in that case (since it's equivalent).
Dark Shikari is offline   Reply With Quote
Old 26th August 2012, 05:52   #7  |  Link
lulalala
Registered User
 
Join Date: Aug 2012
Posts: 11
Quote:
Originally Posted by Dark Shikari View Post
There's no real advantage to compressing many unrelated images in a single archive, and similarly, inter prediction is kind of useless in that case (since it's equivalent).
The images will be divided into subsets. Each subset of images will be extremely similar to each other (Maybe only 5% of pixels will change).
lulalala is offline   Reply With Quote
Old 26th August 2012, 11:18   #8  |  Link
Ghitulescu
Registered User
 
Ghitulescu's Avatar
 
Join Date: Mar 2009
Location: Germany
Posts: 5,619
Why don't you use PNG? Is quite compressed, it's directly adressable, it's recognized by most [new] software ...
__________________
Born in the USB (not USA)
Ghitulescu is offline   Reply With Quote
Old 26th August 2012, 11:40   #9  |  Link
lulalala
Registered User
 
Join Date: Aug 2012
Posts: 11
Quote:
Originally Posted by Ghitulescu View Post
Why don't you use PNG? Is quite compressed, it's directly adressable, it's recognized by most [new] software ...
Because PNG's are individually compressed, thus 7zip solid mode won't be able to detect redundancies.

Take the 200mb BMP files as example, I converted them to 25mb PNG files, and those will compress to an 18mb 7zip file.
lulalala is offline   Reply With Quote
Old 26th August 2012, 12:48   #10  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
Compressing PNG files is silly, and I'd be shocked if you could get 25MB PNGs to compress to 18MB, let alone 24MB. Compressing an already-compressed file will almost never give any gain (unless the compressor has a special mode that internally decompresses and recompresses it).
Dark Shikari is offline   Reply With Quote
Old 26th August 2012, 15:20   #11  |  Link
Phantom_E
Registered User
 
Join Date: Jul 2004
Posts: 41
Quote:
Originally Posted by lulalala View Post
Yes my sources are digitally created images. Therefore scanning is irrelevant.
If your images are digitally created then the ultimate in compression is to store the algorithm and input parameters.
Phantom_E is offline   Reply With Quote
Old 26th August 2012, 15:23   #12  |  Link
Ghitulescu
Registered User
 
Ghitulescu's Avatar
 
Join Date: Mar 2009
Location: Germany
Posts: 5,619
Quote:
Originally Posted by Ghitulescu View Post
Why don't you use PNG? Is quite compressed, it's directly adressable, it's recognized by most [new] software ...
Quote:
Originally Posted by lulalala View Post
Because PNG's are individually compressed, thus 7zip solid mode won't be able to detect redundancies.
You didn't specify that 7z should be a mandatory part of your plan. This is why I suggested you PNG, because they are already compressed - well balanced between size and portability.
__________________
Born in the USB (not USA)
Ghitulescu is offline   Reply With Quote
Reply

Tags
avisynth, h264, x264

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 20:23.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.