CorePNG - New LossLess codec

Atamido · 29th October 2003, 17:58

Here is a new lossless codec based on PNG developed by the talented jcsston. Essentially, each frame is compressed as a PNG, so if PNG does it, this codec does too. (RGBA) This new version has the ability to write P frames and to autodetect when it should. The P frame takes the difference of the previous frame and the current frame and encodes that as a PNG.

Things to note:
1. It is slower to encode than Huffyuv.
2. It is usually slower to decode than Huffyuv.
3. File sizes are usually smaller than Huffyuv.
4. It works best with large areas of similar color like Anime/cartoons.
5. P frames take a little more power to decode because the codec must take the difference of the video after decoding the frame.
6. Encoding with Auto-Delta frames is about 50% slower because it must encode each frame twice.
7. There is a maximum interval of 1000 frames between keyframes.

Here it is:
CorePNG vfw

This really shouldn't be sitting at the bottom of another thread. Actual stats will be posted next.

Koepi · 29th October 2003, 18:04

Nice work Pamel!

Is there any chance to make that codec/PNG YUV-aware? As far as i understand it, PNG uses more or less some kind of zip/gz compression, so it shouldn't be too problematic to feed it with "any" kind of data. (I'm of course aware that there are libgz-based codecs [ok, I know of one slow one] out there, but PNG is somewhat inherantly more graphics/image based.)

Anyways, thanks for your time spent on it and for your work!

regards
Koepi

Atamido · 29th October 2003, 18:41

Whoa there speedy! This codec is ALL jcsston. I just told him to do it.

Here are some test clips to show how well certain things work. (all tests were done in RGB)

Code:

01:47.265 - 02:18.472 S03E17 - 8F13 Homer at the Bat
512x384 pixels 
31.44 seconds 
752 frames
No audio

158,184KB  Huffyuv
131,190KB  Loco
130,582KB  All  I frames Fastest
113,192KB  All  I frames Normal
109,812KB  All  I frames Best
 73,654KB    2  P frames Normal
 61,868KB  100  P frames Fastest
 50,369KB  100  P frames Normal
 49,189KB  100  P frames Best
 40,603KB  1000 P frames Best with Auto-Delta frames


StarwarsAAC5_1anamorphicXviD.mkv From the Anamorphic test.
720x432 pixels
32.60 seconds
815 frames
No audio

311,895KB  Huffyuv
325,213KB  All  I frames Normal
276,710KB  1000 P frames Best with Auto-Delta frames


3 Blocks, Simple rendered scene.
320x240 pixels
10.00 seconds
240 frames
No audio

 14,903KB  Huffyuv
  2,785KB  All  I frames Normal
  2,538KB  1000 P frames Best with Auto-Delta frames

Encoding to "All I frames Normal" is about 4-5 times slower than encoding to Huffyuv. Encoding to "Best" is about 4 times slower than "All I frames Normal". Encoding to "Fastest" is about twice as fast as "All I frames Normal". And using Auto-Delta makes the encode take 50% longer. So, if encoding a clip in Huffyuv takes 1 unit of time, then a quick estimate would be:

Code:

1  Huffyuv
2  Fastest PNG
4  Normal PNG
16 Best PNG
24 Best w/ Auto-Delta.

Those numbers will get a little more refined with some more testing.

Here is an image that shows decoding the Simpsons Clip. The first hump is Huffyuv and the second hump is "All I frames Normal". There is no decoding difference with video encoded with Fastest, Normal, or Best. The 2 P frames stayed mostly at 94-98% CPU and hits 100% every now and then. For 100 P frames, it stayed at 100% the whole time. For the Auto-Delta, the CPU usage was about halfway inbetween 2 and 100 P frames.

Koepi · 29th October 2003, 18:50

Hm, nice results - so it's definatly useful for anime/cartoon intermediate files.

My credits go out to jcsston - sorry for any confusion

(Still thakns for the tests and announcement/forcing jcsston to code that

)

Best regards
Koepi

Tommy Carrot · 29th October 2003, 19:04

Why are you compare it to Huffyuv? There are much better lossless codecs. I'm particularly curious how is it doing against ffv1 on animes or cartoons (i cannot test since i haven't got any right now).

EDIT: BTW, nice work!

Atamido · 29th October 2003, 20:11

Quote:

Originally posted by Tommy Carrot
Why are you compare it to Huffyuv? There are much better lossless codecs. I'm particularly curious how is it doing against ffv1 on animes or cartoons

I used Huffyuv because its the most common lossless codec. Almost everyone has used it, so its easier to get an idea.

I also wanted to try VBLE, but I'm not sure where to get it.

I made some tests with FFV1, but I couldn't play any of them back so I'm not sure about the numbers it produced. Although it did much better on the StarWars clip and a bit worse than Normal I frames on the Simpsons clip. If anyone can get this codec to work, please post a comparison.

jcsston · 29th October 2003, 21:05

I havn't tested againist VBLE because VBLE only works in YV12 mode. And comparing a RGB24 codec againist a YUV codec isn't much of a comparsion.

Wilbert · 29th October 2003, 22:36

@Pamel,

The vble codec can be found here:

http://forum.doom9.org/showthread.php?s=&threadid=53305

jcsston · 30th October 2003, 01:41

I say again, this codec in it's current state CANNOT be compared equally to VBLE or FFV1. Both those codecs work in YV12 colorspace, half the bits of RGB24. And neither support Alpha channel.

I don't see many people comparing 44.1Khz Monkey audio to 22.05Khz FLAC

Human_USB · 30th October 2003, 02:10

On fast mode I get 4 FPS and on Best I get 0 to 1 FPS. Also I do not have Auto-Delta on. I wish I got the speed Pamel got.

*EDIT*
Also my video is upside down.

*EDIT 2*
I find that when I encode in Fast mode my video comes out upside down.

Tommy Carrot · 30th October 2003, 02:45

Quote:

Originally posted by jcsston
I say again, this codec in it's current state CANNOT be compared equally to VBLE or FFV1. Both those codecs work in YV12 colorspace, half the bits of RGB24. And neither support Alpha channel.

I don't see many people comparing 44.1Khz Monkey audio to 22.05Khz FLAC

Well, Loco codec does support rgb colorspace, and there is an option to alpha channel support too, but i've never tried it, so i don't know if it really works.

BTW, the delta-frames are great idea! I've tried CorePNG on CGI, and the size gain is huge with delta-frames.

jcsston · 30th October 2003, 04:42

Quote:

Originally posted by Tommy Carrot
Well, Loco codec does support rgb colorspace, and there is an option to alpha channel support too, but i've never tried it, so i don't know if it really works.

I tried out the Loco codec and on my test clip it was larger than the Huffyuv version (both in RGB24 mode).

Atamido · 30th October 2003, 09:01

Quote:

Originally posted by Human_USB
On fast mode I get 4 FPS and on Best I get 0 to 1 FPS. Also I do not have Auto-Delta on. I wish I got the speed Pamel got.

*EDIT*
Also my video is upside down.

With a lower speed processor, you are probably much more able to give an idea of the speed difference between Huffyuv and CorePNG on a given clip because the processor will be the main bottleneck.

The upsidedown video is an issue with DirectShow using the VFW codec to decode. You should not see this if playing the file back in VirtualDub. This will be fixed very soon.

Dead2 · 30th October 2003, 10:24

Hmm... You say you support Alpha channel..?

Does that mean you use RGBA all the time, or is there an option to do just RGB? I guess alpha-disabled clips would benefit from that. Btw, are there any video clips with alpha yet? if so, what for?

Yuv support should/could be quite simple actually..
If I remember correctly, you could just add the 3 output yuv bitmaps together, then compress them using regular 256color png.. After all, that is what png is really good at.

Code:

_________________
|         |  u  |
|   y     |_____|
|         |  v  |
|_________|_____|

Another thing.. for .png images there is a util called "pngcrush" that basicly re-tightens the image for those 2-15% extra compression still without loosing anything.
This could be an option "Super-compression" or something. Unfortunately it tends to take time, but I would definetely use that for testclips that I have stored permanently in lossless format.

Also, if you look at the pngcrush site, there are mentions of a few tweaks they had to do to the zlib before it would be optimal. Maybe worth checking out aswell?

Good luck! We'll be watching your progress =)

-=Dead2=-

jcsston · 30th October 2003, 16:38

Quote:

Originally posted by Dead2
Hmm... You say you support Alpha channel..?

Does that mean you use RGBA all the time, or is there an option to do just RGB? I guess alpha-disabled clips would benefit from that. Btw, are there any video clips with alpha yet? if so, what for?

Yuv support should/could be quite simple actually..
If I remember correctly, you could just add the 3 output yuv bitmaps together, then compress them using regular 256color png.. After all, that is what png is really good at.

Code:

_________________ | | u | | y |_____| | | v | |_________|_____|

Another thing.. for .png images there is a util called "pngcrush" that basicly re-tightens the image for those 2-15% extra compression still without loosing anything.
This could be an option "Super-compression" or something. Unfortunately it tends to take time, but I would definetely use that for testclips that I have stored permanently in lossless format.

Also, if you look at the pngcrush site, there are mentions of a few tweaks they had to do to the zlib before it would be optimal. Maybe worth checking out aswell?

Good luck! We'll be watching your progress =)

-=Dead2=-

The Alpha channel is only compressed/used if the input is RGB32. VirtualDub only copmressed to/from RGB24 so you don't have to worry about that. The most often clips to use the Alpha channel would likely be CGI or title clips that are being used to create videos.

Thanks for the idea of having different images for YUV.

I've looked into the various things pngcrush does, but most of the things that it does to improve compression are hard-coded (need a different compile for regular compression) and it also increases decode time.

I've fixed the image flip bug with 32-bit decoding.

Download CorePNG VFW Codec v0.6

BetaBoy · 30th October 2003, 16:51

I asked jcsston to create a MAIN project page for CorePNG on www.CoreCodec.org for easier access and because it needed it

Get it here: http://corecodec.org/projects/corepng

FYI: for those looking for a nightly Source Code 'cron job' for all the CoreCodec.org projects (including Matroska and it's associated projects) go here: http://cvs.corecodec.org/nightly/

----
Dan "BetaBoy" Marlin

MfA · 30th October 2003, 16:51

It would be nice to have some idea of the comparitive performance against FFV1, how about using avisynth to produce an avi with just a single color component to do the comparison for now?

Human_USB · 30th October 2003, 17:00

Quote:

Originally posted by Pamel
With a lower speed processor, you are probably much more able to give an idea of the speed difference between Huffyuv and CorePNG on a given clip because the processor will be the main bottleneck.

The upsidedown video is an issue with DirectShow using the VFW codec to decode. You should not see this if playing the file back in VirtualDub. This will be fixed very soon.

But the sad thing is I have a 1800+ XP with 256 of 2600 DDR. Also my Hard drive has 8 MBs of cache so I have no clue what is killing my speed.

BetaBoy · 30th October 2003, 17:07

Jory just released v0.6 which fixes the flipped image bug.

http://corecodec.org/projects/corepng

Chainmax · 30th October 2003, 19:49

I second the YUV request. Also (like I asked in the original thread), could B-Frames be implemented too? What about an optional lossy mode for P-Frames (and B-Frames if they get implemented)?

29th October 2003, 17:58	#1 \| Link
Atamido Señor Member Join Date: May 2002 Location: Austin, Texas Posts: 915	CorePNG - New LossLess codec Here is a new lossless codec based on PNG developed by the talented jcsston. Essentially, each frame is compressed as a PNG, so if PNG does it, this codec does too. (RGBA) This new version has the ability to write P frames and to autodetect when it should. The P frame takes the difference of the previous frame and the current frame and encodes that as a PNG. Things to note: 1. It is slower to encode than Huffyuv. 2. It is usually slower to decode than Huffyuv. 3. File sizes are usually smaller than Huffyuv. 4. It works best with large areas of similar color like Anime/cartoons. 5. P frames take a little more power to decode because the codec must take the difference of the video after decoding the frame. 6. Encoding with Auto-Delta frames is about 50% slower because it must encode each frame twice. 7. There is a maximum interval of 1000 frames between keyframes. Here it is: CorePNG vfw This really shouldn't be sitting at the bottom of another thread. Actual stats will be posted next. Last edited by Pamel; 1st November 2003 at 01:17.

29th October 2003, 18:04	#2 \| Link
Koepi Moderator Join Date: Oct 2001 Location: Germany Posts: 4,453	Nice work Pamel! Is there any chance to make that codec/PNG YUV-aware? As far as i understand it, PNG uses more or less some kind of zip/gz compression, so it shouldn't be too problematic to feed it with "any" kind of data. (I'm of course aware that there are libgz-based codecs [ok, I know of one slow one] out there, but PNG is somewhat inherantly more graphics/image based.) Anyways, thanks for your time spent on it and for your work! regards Koepi __________________ Koepi's new media development site

29th October 2003, 18:41	#3 \| Link
Atamido Señor Member Join Date: May 2002 Location: Austin, Texas Posts: 915	Whoa there speedy! This codec is ALL jcsston. I just told him to do it. Here are some test clips to show how well certain things work. (all tests were done in RGB) Code: 01:47.265 - 02:18.472 S03E17 - 8F13 Homer at the Bat 512x384 pixels 31.44 seconds 752 frames No audio 158,184KB Huffyuv 131,190KB Loco 130,582KB All I frames Fastest 113,192KB All I frames Normal 109,812KB All I frames Best 73,654KB 2 P frames Normal 61,868KB 100 P frames Fastest 50,369KB 100 P frames Normal 49,189KB 100 P frames Best 40,603KB 1000 P frames Best with Auto-Delta frames StarwarsAAC5_1anamorphicXviD.mkv From the Anamorphic test. 720x432 pixels 32.60 seconds 815 frames No audio 311,895KB Huffyuv 325,213KB All I frames Normal 276,710KB 1000 P frames Best with Auto-Delta frames 3 Blocks, Simple rendered scene. 320x240 pixels 10.00 seconds 240 frames No audio 14,903KB Huffyuv 2,785KB All I frames Normal 2,538KB 1000 P frames Best with Auto-Delta frames Encoding to "All I frames Normal" is about 4-5 times slower than encoding to Huffyuv. Encoding to "Best" is about 4 times slower than "All I frames Normal". Encoding to "Fastest" is about twice as fast as "All I frames Normal". And using Auto-Delta makes the encode take 50% longer. So, if encoding a clip in Huffyuv takes 1 unit of time, then a quick estimate would be: Code: 1 Huffyuv 2 Fastest PNG 4 Normal PNG 16 Best PNG 24 Best w/ Auto-Delta. Those numbers will get a little more refined with some more testing. Here is an image that shows decoding the Simpsons Clip. The first hump is Huffyuv and the second hump is "All I frames Normal". There is no decoding difference with video encoded with Fastest, Normal, or Best. The 2 P frames stayed mostly at 94-98% CPU and hits 100% every now and then. For 100 P frames, it stayed at 100% the whole time. For the Auto-Delta, the CPU usage was about halfway inbetween 2 and 100 P frames. Last edited by Pamel; 29th March 2004 at 20:43.

29th October 2003, 18:50	#4 \| Link
Koepi Moderator Join Date: Oct 2001 Location: Germany Posts: 4,453	Hm, nice results - so it's definatly useful for anime/cartoon intermediate files. My credits go out to jcsston - sorry for any confusion (Still thakns for the tests and announcement/forcing jcsston to code that ) Best regards Koepi __________________ Koepi's new media development site

29th October 2003, 19:04	#5 \| Link
Tommy Carrot Registered User Join Date: Mar 2002 Posts: 747	Why are you compare it to Huffyuv? There are much better lossless codecs. I'm particularly curious how is it doing against ffv1 on animes or cartoons (i cannot test since i haven't got any right now). EDIT: BTW, nice work! Last edited by Tommy Carrot; 29th October 2003 at 19:08.

29th October 2003, 21:05	#7 \| Link
jcsston Matroska Dev Join Date: Sep 2002 Location: Texas, USA Posts: 229	I havn't tested againist VBLE because VBLE only works in YV12 mode. And comparing a RGB24 codec againist a YUV codec isn't much of a comparsion. __________________ The Matroska Effect

29th October 2003, 22:36	#8 \| Link
Wilbert Moderator Join Date: Nov 2001 Location: Netherlands Posts: 5,579	@Pamel, The vble codec can be found here: http://forum.doom9.org/showthread.php?s=&threadid=53305 Last edited by Wilbert; 29th October 2003 at 22:39.

30th October 2003, 01:41	#9 \| Link
jcsston Matroska Dev Join Date: Sep 2002 Location: Texas, USA Posts: 229	I say again, this codec in it's current state CANNOT be compared equally to VBLE or FFV1. Both those codecs work in YV12 colorspace, half the bits of RGB24. And neither support Alpha channel. I don't see many people comparing 44.1Khz Monkey audio to 22.05Khz FLAC __________________ The Matroska Effect

30th October 2003, 02:10	#10 \| Link
Human_USB Plug me in... Join Date: Aug 2003 Location: Mississippi Posts: 93	On fast mode I get 4 FPS and on Best I get 0 to 1 FPS. Also I do not have Auto-Delta on. I wish I got the speed Pamel got. EDIT Also my video is upside down. EDIT 2 I find that when I encode in Fast mode my video comes out upside down. __________________ Chaos in order or order in chaos..... which are you? Last edited by Human_USB; 30th October 2003 at 02:28.

30th October 2003, 10:24	#14 \| Link
Dead2 Registered User Join Date: Jan 2002 Location: Norway Posts: 1	Alpha Hmm... You say you support Alpha channel..? Does that mean you use RGBA all the time, or is there an option to do just RGB? I guess alpha-disabled clips would benefit from that. Btw, are there any video clips with alpha yet? if so, what for? Yuv support should/could be quite simple actually.. If I remember correctly, you could just add the 3 output yuv bitmaps together, then compress them using regular 256color png.. After all, that is what png is really good at. Code: _________________ \| \| u \| \| y \|_____\| \| \| v \| \|_________\|_____\| Another thing.. for .png images there is a util called "pngcrush" that basicly re-tightens the image for those 2-15% extra compression still without loosing anything. This could be an option "Super-compression" or something. Unfortunately it tends to take time, but I would definetely use that for testclips that I have stored permanently in lossless format. Also, if you look at the pngcrush site, there are mentions of a few tweaks they had to do to the zlib before it would be optimal. Maybe worth checking out aswell? Good luck! We'll be watching your progress =) -=Dead2=-

30th October 2003, 16:51	#16 \| Link
BetaBoy CoreCodec Founder Join Date: Oct 2001 Location: Sunny Florida Posts: 1,029	I asked jcsston to create a MAIN project page for CorePNG on www.CoreCodec.org for easier access and because it needed it Get it here: http://corecodec.org/projects/corepng FYI: for those looking for a nightly Source Code 'cron job' for all the CoreCodec.org projects (including Matroska and it's associated projects) go here: http://cvs.corecodec.org/nightly/ ---- Dan "BetaBoy" Marlin

30th October 2003, 16:51	#17 \| Link
MfA Registered User Join Date: Mar 2002 Posts: 1,031	It would be nice to have some idea of the comparitive performance against FFV1, how about using avisynth to produce an avi with just a single color component to do the comparison for now?

30th October 2003, 17:07	#19 \| Link
BetaBoy CoreCodec Founder Join Date: Oct 2001 Location: Sunny Florida Posts: 1,029	Jory just released v0.6 which fixes the flipped image bug. http://corecodec.org/projects/corepng

30th October 2003, 19:49	#20 \| Link
Chainmax Huh? Join Date: Sep 2003 Location: Uruguay Posts: 2,978	I second the YUV request. Also (like I asked in the original thread), could B-Frames be implemented too? What about an optional lossy mode for P-Frames (and B-Frames if they get implemented)?

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode