Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > New and alternative video codecs

Reply
 
Thread Tools Search this Thread Display Modes
Old 13th February 2014, 02:41   #1  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
MagicYUV - a new fast lossless video codec

Hi!

As I mentioned in the Intermediate Video Format survey thread, I've been developing a new fast lossless video codec (named MagicYUV) for Windows, it's been basically ready since a year ago, I just never got the time to release it.

But now it's here, I've fixed some remaining bugs I've known and I guess it's ready for an alpha release.

I've set up a website, you can grab it from there: http://magicyuv.com

I've also uploaded a command line tool avibench there, that can measure codec compression/decompression performance through AVI files.

Any kind of suggestions, comments, bug and crash reports, feature requests can come here. The target is to make it to 1.0 stable

Greets,
I.
Ignus2 is offline   Reply With Quote
Old 13th February 2014, 03:30   #2  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 12,908
Thank you. I will definitely give that one a try.

Do you have any graphs on how this one compares to other lossless video Codecs (HuffYUV, Lagarith, FFV1, x264) in terms of speed and compression?

BTW: Can't really see anywhere which license this is released under. So I assume it's ClosedSource, right?
__________________
There was of course no way of knowing whether you were being watched at any given moment.
How often, or on what system, the Thought Police plugged in on any individual wire was guesswork.


LoRd_MuldeR is offline   Reply With Quote
Old 13th February 2014, 03:49   #3  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Quote:
Originally Posted by LoRd_MuldeR View Post
Thank you. I will definitely give that one a try.

Do you have any graphs on how this one compares to other lossless video Codecs (HuffYUV, Lagarith, FFV1, x264) in terms of speed and compression?

BTW: Can't really see anywhere which license this is released under. So I assume it's ClosedSource, right?
Well, I do have some graphs, but it would be best if the users could test it
As a quick reference: for decompression in Dynamic mode it "should" be considerably (2-3x) faster than Ut Median with around same comp. ratio and faster (1.4x) than Ut Left with better (10%) comp. ratio. Without Disk I/O of course (measured with ramdisk/fully cached avi). At least on my machine... And I'm really curious about other machines...
Against FFV1 and x264 I didn't compare yet.

About the license: well, I'm not a lawyer, so I can't come up with licenses, so I'll just say that the version you can download from the site is free to use for whatever you want and the sources are not included

Greets,
I.
Ignus2 is offline   Reply With Quote
Old 13th February 2014, 04:17   #4  |  Link
EncodedMango
Registered User
 
Join Date: Jun 2013
Posts: 64
Seems interesting, gonna try it out.

EDIT:Okay so I just tried it using Dxtory(so 32bit version) and I must say the results seem impressive, I had(edit2) some FPS loss when recording but it didn't feel incredibly laggy. This beats Lagarith for me in this test atleast great job Ignus2.

Last edited by EncodedMango; 13th February 2014 at 11:45.
EncodedMango is offline   Reply With Quote
Old 13th February 2014, 10:56   #5  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 3,665
Recompressing a 3min, 49sec 704x384 AVI with VirtualDubMod (fast recompress). No audio. Output written to the first partition of a RAID-0 volume to take hard drive speed out the equation as much as possible. YV12 -> YV12. Total time as displayed by VirtualDubMod.
Old E6750 dual core CPU, 3.2GHz. For Lagarith and MagicYUV there's two time and CPU usage figures. Without and with multi-threading enabled. CPU usage is a fairly rough average anyway.

Huffyuv (ffdshow variant), 16 seconds, CPU 62%, 990.8MB
Huffyuv (ffdshow variant, adaptive huffman tables enabled), 16 seconds, CPU 62%, 587.2MB

FFV1, 62 seconds, CPU 58%, 432.6MB

Lagarith, 18/15 seconds, CPU 60%/73%, 503MB

MagicYUV, 17/12 seconds, CPU 60%/83%, 586.8MB

For a hard drive speed reference..... copying the largest file (990.8MB) from the location of the source file to the same location as the output files took around 6 seconds.

And for fun:

Uncompressed YV12 (Avisynth script/Direct Stream Copy), 15 seconds, CPU 45%, 2.1GB
Uncompressed RGB, 30 seconds, CPU 45%, 4.3GB

Last edited by hello_hello; 13th February 2014 at 11:33.
hello_hello is offline   Reply With Quote
Old 13th February 2014, 12:13   #6  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 12,908
Quote:
Originally Posted by Ignus2 View Post
About the license: well, I'm not a lawyer, so I can't come up with licenses, so I'll just say that the version you can download from the site is free to use for whatever you want and the sources are not included
You don't need to be a lawyer. If you ever decide to "open" your sources, you could just put them, for example, under a (L)GPL license - which would make it possible to integrate your Codec into FFmpeg (libavcodec), thus making it available in pretty much any OpenSource tool. Currently it's tied not only to the Windows platform but also to the (deprecated) VFW interface, which limits its applications, of course. And it has every user to rely on you to provide "free" decoders in the future (the last point might not be that critical for an "intermediate" Codec though)
__________________
There was of course no way of knowing whether you were being watched at any given moment.
How often, or on what system, the Thought Police plugged in on any individual wire was guesswork.



Last edited by LoRd_MuldeR; 13th February 2014 at 12:16.
LoRd_MuldeR is offline   Reply With Quote
Old 13th February 2014, 12:23   #7  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Quote:
Originally Posted by hello_hello View Post
Recompressing a 3min, 49sec 704x384 AVI with VirtualDubMod (fast recompress).
...
The codec is not yet tuned optimally for such small resolutions (on my TODO list), if you could try the same test with HD (1920x1080) or higher, that would be nice :-)

Greets,
I.
Ignus2 is offline   Reply With Quote
Old 13th February 2014, 18:23   #8  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,577
I've just updated my old test http://forum.doom9.org/showthread.php?t=165745

I don't know why but file is larger than source .y4m !?

Code:
General
Complete name                            : G:\job1.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
File size                                : 859 MiB
Duration                                 : 10s 0ms
Overall bit rate                         : 721 Mbps
Writing library                          : VirtualDub build 32842/release

Video
ID                                       : 0
Format                                   : MAGY
Codec ID                                 : MAGY
Duration                                 : 10s 0ms
Bit rate                                 : 721 Mbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 25.000 fps
Bits/(Pixel*Frame)                       : 13.903
Stream size                              : 859 MiB (100%)

Code:
General
Complete name                            : G:\job1.y4m
Format                                   : YUV4MPEG2
File size                                : 742 MiB

Video
Format                                   : YUV
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 25.000 fps
Color space                              : YUV
Scan type                                : Progressive
Compression mode                         : Lossless

Last edited by Atak_Snajpera; 13th February 2014 at 18:26.
Atak_Snajpera is offline   Reply With Quote
Old 13th February 2014, 21:32   #9  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Quote:
Originally Posted by Atak_Snajpera View Post
I've just updated my old test http://forum.doom9.org/showthread.php?t=165745

I don't know why but file is larger than source .y4m !?
That's a colorspace issue I guess (or something similar). I think the stream actually gets encoded as RGB probably.
MagicYUV actually compresses what it gets without colorspace conversion, and it is currently not possible to force or restrict what colorspace the codec accepts, so if VirtualDub is set up or decides to give MagicYUV RGB input, it will compress like that.

--
Greets,
I.
Ignus2 is offline   Reply With Quote
Old 13th February 2014, 21:47   #10  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,577
i will redo test tomorrow then.
Atak_Snajpera is offline   Reply With Quote
Old 13th February 2014, 21:49   #11  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Quote:
Originally Posted by Atak_Snajpera View Post
i will redo test tomorrow then.
That would be great, I'm very interested in the results
Ignus2 is offline   Reply With Quote
Old 13th February 2014, 22:53   #12  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 3,665
5 minute, 1920x1040 MKV with Avisynth/ffmsindex/VirtualDub (fast recompress). ffms in single threaded mode. No audio. Output written to the first partition of a RAID-0 volume to take hard drive speed out the equation as much as possible. YV12 -> YV12. Total time as displayed by VirtualDub.
Old E6750 dual core CPU, 3.2GHz. CPU usage is a fairly rough average. Encoding the first 15 to 30 seconds indicated multithreaded mode was slower for both Lagarith and MagicYUV so I disabled it.

Huffyuv (ffdshow variant), 4 min 23 sec, CPU 85%, 9542MB
Huffyuv (ffdshow variant, adaptive huffman tables enabled), 4 min 1 sec, CPU 80%, 5897MB

FFV1, 11 min 4 sec, CPU 70%, 4935MB

Lagarith, (no mulltithreading, enabling it slowed encoding speed by 2 or 3 fps), 4 min 7 sec, CPU 90%, 5451MB

MagicYUV, (no mulltithreading, enabling it slowed encoding speed by 2 or 3 fps), 4 min 23 sec, CPU 97%, 5823MB


Uncompressed YV12, (Avisynth/ffmsindex/VirtualDubMod/Direct Stream Copy), 4 min 14 sec, CPU 65%, 20572MB


I ran most of the above encodes twice to check them, given the next results weren't what I expected.......

As an experiment I thought I'd take Avisynth and ffmsindex out of the equation and reduce the CPU usage (hopefully) required for decoding. I took the second huffyuv encode and put it on the same drive as the original MKV. The 5897MB file took about 1 minute to move, so obviously hard drive speed wasn't a bottleneck. I then opened it directly with VirtualDub and re-compressed it (fast recompress). Same output file location as before. This time multithreaded mode was a little faster than single threaded mode for MagicYUV and Lagarith.

MagicYUV, (mulltithreading enabled), 4 min 02 sec, CPU 70%, 6683MB
I run the encode twice. With huffyuv as the source MagicYUV was faster. It also compressed less.

Lagarith, (mulltithreading enabled), 4 min 54 sec, CPU 65%, 5404MB
With huffyuv as the source Lagarith slowed down quite a bit. The file size decreased a tiny bit. I don't understand why. I tried again without multithreading:
Lagarith, (no mulltithreading), 5 min 29 sec, CPU 55%, 5404MB

Huffyuv (ffdshow variant, adaptive huffman tables enabled), 4 min 38 sec, CPU 55%, 5897MB
And with huffyuv as the source, huffyuv slowed down too. No file size change though. Input and output file sizes were exactly the same.

Someone will have to explain the speed/compression changes for me. Maybe the huffyuv version isn't YV12, although as best as I can tell it is. I even encoded it again using the "force input colorspace option" and setting it to YV12 and the file size after encoding (from the original video) still remained unchanged. When viewing the huffyuv encode with MPC-HC, ffdshow reports the output as NV12 (as it usually does when decoding YV12 with MPC-HC). But enough for today.....

Last edited by hello_hello; 13th February 2014 at 23:09.
hello_hello is offline   Reply With Quote
Old 13th February 2014, 23:18   #13  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Quote:
Originally Posted by hello_hello View Post
5 minute, 1920x1040 MKV with Avisynth/ffmsindex/VirtualDub (fast recompress).
...
Thanks.

What codec and size is the original MKV?

The file becoming bigger for MagicYUV from huffyuv source in fast recompress is due to (I believe) the fact, that huffyuv tends to ouput YUY2 even if it is compressed YV12. And as it is not yet possible to restrict what input MagicYUV accepts, and it doesn't do any colorspace conversion of the input, it probably actually compressed YUY2 (instead of YV12).

I probably need to an option to restrict accepted colorspaces, that would eliminate these hidden colorspace inconsistencies that others are also experiencing...

Greets,
I.
Ignus2 is offline   Reply With Quote
Old 13th February 2014, 23:42   #14  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 3,665
It was annoying me, so I tried again. Same 1920x1040 video, but I decoded it to uncompressed video using Avisynth/ffmsindex/VirtualDubMod/Direct Stream Copy. Then I compressed the 20GB version. I didn't bother with FFV1. I think it's fairly established it's very slow.

Huffyuv (ffdshow variant, adaptive huffman tables enabled), 2 min 33 sec, CPU 70%, 5897MB

Lagarith, (mulltithreading), 3 min 1 sec, CPU 80%, 5451MB

MagicYUV, (mulltithreading), 2 min 17 sec, CPU 70%, 5823MB

Now the results are what I'd expect. Faster compression using the uncompressed video as the source (compared to the original video and Avisynth). Same file sizes.
I'm not sure what that means in respect to the ffdshow version of huffyuv being lossless for YV12. I thought it was supposed to be?
hello_hello is offline   Reply With Quote
Old 14th February 2014, 01:12   #15  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Nice

BTW, what MagicYUV really should excel at is decompression speed. If you could do a decompression test, that would be nice too (dunno where to put output though, maybe use avsmeter?). Or better yet, do a recompress cycle using the same codec for input and compression.

Also, suggestions are welcome, especially about the colorspace things. Is restriction enough (simple to do) or should I implement conversion?

Greets,
I.
Ignus2 is offline   Reply With Quote
Old 14th February 2014, 01:38   #16  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 3,665
Quote:
Originally Posted by Ignus2 View Post
Thanks.

What codec and size is the original MKV?

x264, 415MB

The file becoming bigger for MagicYUV from huffyuv source in fast recompress is due to (I believe) the fact, that huffyuv tends to ouput YUY2 even if it is compressed YV12. And as it is not yet possible to restrict what input MagicYUV accepts, and it doesn't do any colorspace conversion of the input, it probably actually compressed YUY2 (instead of YV12).
I think you're correct but I'm not sure who's fault it is. I'm pretty sure the huffyuv version is encoded as YV12 and I'm pretty sure when it's opened in a player and decoded via directshow it's decoded as YV12. Create an AviSource script to open it and add Info() and it's reported as YV12. Whether it's opened by MPC-HC or VirtualDub it's YV12.
Open the AVI containing huffyuv directly with VirtualDub though and the output is YUY2. Unless..... you disable YUY2 and UVYV as output clolrspaces in the ffdshow VFW decoder configuration.

After a fair bit of playing around I'm fairly confident that's what's happening, and while I don't know why, it appears VirtualDub's Fast Recompress option is to blame. Why it seems to only want to convert YV12 to YUY2 when the input codec is huffyuv I don't know, and I don't know why disabling YUY2 and UVYV in the ffdshow list of out output colorspaces stops it from happening. It's kind of like Fast Recompress decides "when it can be decoded as YUY2, it should be".....

Fast recompress: Video is decompressed and then recompressed using the desired output codec. VirtualDub automatically chooses a intermediate video format to use between the codecs for quality and speed.

VirtualDubMod's fast recompress doesn't seem to do the same. For it, if it's huffyuv YV12 in, it's YV12 out.
hello_hello is offline   Reply With Quote
Old 14th February 2014, 15:11   #17  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,577
Test updated! http://forum.doom9.org/showthread.php?t=165745
Atak_Snajpera is offline   Reply With Quote
Old 15th February 2014, 00:29   #18  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: UK
Posts: 2,323
Did you use 32bit?
64bit MagicYuv is way faster, at least for decoding.
kolak is offline   Reply With Quote
Old 15th February 2014, 00:35   #19  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: UK
Posts: 2,323
Quote:
Originally Posted by Ignus2 View Post
Nice

BTW, what MagicYUV really should excel at is decompression speed. If you could do a decompression test, that would be nice too (dunno where to put output though, maybe use avsmeter?). Or better yet, do a recompress cycle using the same codec for input and compression.

Also, suggestions are welcome, especially about the colorspace things. Is restriction enough (simple to do) or should I implement conversion?

Greets,
I.
I think the speed is great, so we are fine here.

Next things to do:
Mac version,
Ffmpeg integration

Also as I mentioned before- 16bit is what it would make it stand out from others. Ffv1 is great, but way to slow, specially for 4K. 8bit is starting to be 'not enough' and not just for pro solution, but also for home users.

Once you have 16bit, than next thing is to add good dithering as an option .

Last edited by kolak; 15th February 2014 at 00:38.
kolak is offline   Reply With Quote
Old 16th February 2014, 00:24   #20  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 950
I did quite extensive testing of a pre-release version of Magic YUV several weeks back using his benchmark tool (no-one seems to be testing with it). I'm not sure what modifications he might have made in the interim and, since I don't really have time to repeat the tests with this alpha release, it would seem inappropriate to post the results.

Suffice it to say that when testing Magic YUV against other lossless codecs, it would pay to compare like-with-like - that is, for the Huffman-based codecs to include and compare the different prediction modes (Left and Median, at least) and not just the defaults.

Magic YUV defaults to a 'balanced' mode that is designed to automatically select (on a per-frame basis) the optimum balance of compression ratio and decompression speed - or as Ignus2 explained it to me:

Dynamic basically just selects the best it thinks per frame, but it is recommended to select it, because if median and left doesn't make a big difference in compression ratio, left is selected, which decodes faster. Dynamic makes a very quick and accurate decision, so there shouldn't be a big difference in speed between dynamic and median on the compression side, but it usually makes a big difference on the decompression side (in favor of dynamic).

If one applies the 'conventional wisdom' that median prediction generally makes for higher compression than left, then arguably 'balanced' mode, in prioritizing compression ratio, could be viewed as favoring Median prediction. Maybe Ignus2 would want to elaborate on that.

UT-Video, by contrast, defaults to Left prediction - being prioritized for decompression speed. So, to my mind, comparing only the default configurations is a bit 'apples and oranges'.

I won't comment further on the pattern of results I obtained (for the reasons stated), but it would be interesting to see what other people find when comparing the decompression and compression rates obtained with the different prediction modes, and including both YV12 and YUY2 sources. And as Ignus2 suggested, maybe look at 're-compression' rates also.

Anyhow, now that the alpha release is out, I don't think anyone could dispute that Magic YUV is indeed a very VFW fast lossless codec, at least on par with UT-Video. As Kolak has emphasized however, what would set Magic YUV apart would be the higher bit-depth support. I hope the positive reports that have been coming back encourage Ignus2 to press forward with that.
__________________
Nostalgia's not what it used to be

Last edited by WorBry; 16th February 2014 at 04:26.
WorBry is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 23:41.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.