Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
13th February 2014, 02:41 | #1 | Link |
Registered User
Join Date: Dec 2005
Posts: 250
|
MagicYUV - a new fast lossless video codec
Hi!
As I mentioned in the Intermediate Video Format survey thread, I've been developing a new fast lossless video codec (named MagicYUV) for Windows, it's been basically ready since a year ago, I just never got the time to release it. But now it's here, I've fixed some remaining bugs I've known and I guess it's ready for an alpha release. I've set up a website, you can grab it from there: http://magicyuv.com I've also uploaded a command line tool avibench there, that can measure codec compression/decompression performance through AVI files. Any kind of suggestions, comments, bug and crash reports, feature requests can come here. The target is to make it to 1.0 stable Greets, I. |
13th February 2014, 03:30 | #2 | Link |
Software Developer
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,248
|
Thank you. I will definitely give that one a try.
Do you have any graphs on how this one compares to other lossless video Codecs (HuffYUV, Lagarith, FFV1, x264) in terms of speed and compression? BTW: Can't really see anywhere which license this is released under. So I assume it's ClosedSource, right?
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊ |
13th February 2014, 03:49 | #3 | Link | |
Registered User
Join Date: Dec 2005
Posts: 250
|
Quote:
As a quick reference: for decompression in Dynamic mode it "should" be considerably (2-3x) faster than Ut Median with around same comp. ratio and faster (1.4x) than Ut Left with better (10%) comp. ratio. Without Disk I/O of course (measured with ramdisk/fully cached avi). At least on my machine... And I'm really curious about other machines... Against FFV1 and x264 I didn't compare yet. About the license: well, I'm not a lawyer, so I can't come up with licenses, so I'll just say that the version you can download from the site is free to use for whatever you want and the sources are not included Greets, I. |
|
13th February 2014, 04:17 | #4 | Link |
Registered User
Join Date: Jun 2013
Posts: 65
|
Seems interesting, gonna try it out.
EDIT:Okay so I just tried it using Dxtory(so 32bit version) and I must say the results seem impressive, I had(edit2) some FPS loss when recording but it didn't feel incredibly laggy. This beats Lagarith for me in this test atleast great job Ignus2. Last edited by EncodedMango; 13th February 2014 at 11:45. |
13th February 2014, 10:56 | #5 | Link |
Registered User
Join Date: Mar 2011
Posts: 4,829
|
Recompressing a 3min, 49sec 704x384 AVI with VirtualDubMod (fast recompress). No audio. Output written to the first partition of a RAID-0 volume to take hard drive speed out the equation as much as possible. YV12 -> YV12. Total time as displayed by VirtualDubMod.
Old E6750 dual core CPU, 3.2GHz. For Lagarith and MagicYUV there's two time and CPU usage figures. Without and with multi-threading enabled. CPU usage is a fairly rough average anyway. Huffyuv (ffdshow variant), 16 seconds, CPU 62%, 990.8MB Huffyuv (ffdshow variant, adaptive huffman tables enabled), 16 seconds, CPU 62%, 587.2MB FFV1, 62 seconds, CPU 58%, 432.6MB Lagarith, 18/15 seconds, CPU 60%/73%, 503MB MagicYUV, 17/12 seconds, CPU 60%/83%, 586.8MB For a hard drive speed reference..... copying the largest file (990.8MB) from the location of the source file to the same location as the output files took around 6 seconds. And for fun: Uncompressed YV12 (Avisynth script/Direct Stream Copy), 15 seconds, CPU 45%, 2.1GB Uncompressed RGB, 30 seconds, CPU 45%, 4.3GB Last edited by hello_hello; 13th February 2014 at 11:33. |
13th February 2014, 12:13 | #6 | Link |
Software Developer
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,248
|
You don't need to be a lawyer. If you ever decide to "open" your sources, you could just put them, for example, under a (L)GPL license - which would make it possible to integrate your Codec into FFmpeg (libavcodec), thus making it available in pretty much any OpenSource tool. Currently it's tied not only to the Windows platform but also to the (deprecated) VFW interface, which limits its applications, of course. And it has every user to rely on you to provide "free" decoders in the future (the last point might not be that critical for an "intermediate" Codec though)
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊ Last edited by LoRd_MuldeR; 13th February 2014 at 12:16. |
13th February 2014, 12:23 | #7 | Link | |
Registered User
Join Date: Dec 2005
Posts: 250
|
Quote:
Greets, I. |
|
13th February 2014, 18:23 | #8 | Link |
RipBot264 author
Join Date: May 2006
Location: Poland
Posts: 7,812
|
I've just updated my old test http://forum.doom9.org/showthread.php?t=165745
I don't know why but file is larger than source .y4m !? Code:
General Complete name : G:\job1.avi Format : AVI Format/Info : Audio Video Interleave File size : 859 MiB Duration : 10s 0ms Overall bit rate : 721 Mbps Writing library : VirtualDub build 32842/release Video ID : 0 Format : MAGY Codec ID : MAGY Duration : 10s 0ms Bit rate : 721 Mbps Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate : 25.000 fps Bits/(Pixel*Frame) : 13.903 Stream size : 859 MiB (100%) Code:
General Complete name : G:\job1.y4m Format : YUV4MPEG2 File size : 742 MiB Video Format : YUV Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate : 25.000 fps Color space : YUV Scan type : Progressive Compression mode : Lossless
__________________
Windows 7 Image Updater - SkyLake\KabyLake\CoffeLake\Ryzen Threadripper Last edited by Atak_Snajpera; 13th February 2014 at 18:26. |
13th February 2014, 21:32 | #9 | Link | |
Registered User
Join Date: Dec 2005
Posts: 250
|
Quote:
MagicYUV actually compresses what it gets without colorspace conversion, and it is currently not possible to force or restrict what colorspace the codec accepts, so if VirtualDub is set up or decides to give MagicYUV RGB input, it will compress like that. -- Greets, I. |
|
13th February 2014, 21:47 | #10 | Link |
RipBot264 author
Join Date: May 2006
Location: Poland
Posts: 7,812
|
i will redo test tomorrow then.
__________________
Windows 7 Image Updater - SkyLake\KabyLake\CoffeLake\Ryzen Threadripper |
13th February 2014, 22:53 | #12 | Link |
Registered User
Join Date: Mar 2011
Posts: 4,829
|
5 minute, 1920x1040 MKV with Avisynth/ffmsindex/VirtualDub (fast recompress). ffms in single threaded mode. No audio. Output written to the first partition of a RAID-0 volume to take hard drive speed out the equation as much as possible. YV12 -> YV12. Total time as displayed by VirtualDub.
Old E6750 dual core CPU, 3.2GHz. CPU usage is a fairly rough average. Encoding the first 15 to 30 seconds indicated multithreaded mode was slower for both Lagarith and MagicYUV so I disabled it. Huffyuv (ffdshow variant), 4 min 23 sec, CPU 85%, 9542MB Huffyuv (ffdshow variant, adaptive huffman tables enabled), 4 min 1 sec, CPU 80%, 5897MB FFV1, 11 min 4 sec, CPU 70%, 4935MB Lagarith, (no mulltithreading, enabling it slowed encoding speed by 2 or 3 fps), 4 min 7 sec, CPU 90%, 5451MB MagicYUV, (no mulltithreading, enabling it slowed encoding speed by 2 or 3 fps), 4 min 23 sec, CPU 97%, 5823MB Uncompressed YV12, (Avisynth/ffmsindex/VirtualDubMod/Direct Stream Copy), 4 min 14 sec, CPU 65%, 20572MB I ran most of the above encodes twice to check them, given the next results weren't what I expected....... As an experiment I thought I'd take Avisynth and ffmsindex out of the equation and reduce the CPU usage (hopefully) required for decoding. I took the second huffyuv encode and put it on the same drive as the original MKV. The 5897MB file took about 1 minute to move, so obviously hard drive speed wasn't a bottleneck. I then opened it directly with VirtualDub and re-compressed it (fast recompress). Same output file location as before. This time multithreaded mode was a little faster than single threaded mode for MagicYUV and Lagarith. MagicYUV, (mulltithreading enabled), 4 min 02 sec, CPU 70%, 6683MB I run the encode twice. With huffyuv as the source MagicYUV was faster. It also compressed less. Lagarith, (mulltithreading enabled), 4 min 54 sec, CPU 65%, 5404MB With huffyuv as the source Lagarith slowed down quite a bit. The file size decreased a tiny bit. I don't understand why. I tried again without multithreading: Lagarith, (no mulltithreading), 5 min 29 sec, CPU 55%, 5404MB Huffyuv (ffdshow variant, adaptive huffman tables enabled), 4 min 38 sec, CPU 55%, 5897MB And with huffyuv as the source, huffyuv slowed down too. No file size change though. Input and output file sizes were exactly the same. Someone will have to explain the speed/compression changes for me. Maybe the huffyuv version isn't YV12, although as best as I can tell it is. I even encoded it again using the "force input colorspace option" and setting it to YV12 and the file size after encoding (from the original video) still remained unchanged. When viewing the huffyuv encode with MPC-HC, ffdshow reports the output as NV12 (as it usually does when decoding YV12 with MPC-HC). But enough for today..... Last edited by hello_hello; 13th February 2014 at 23:09. |
13th February 2014, 23:18 | #13 | Link | |
Registered User
Join Date: Dec 2005
Posts: 250
|
Quote:
What codec and size is the original MKV? The file becoming bigger for MagicYUV from huffyuv source in fast recompress is due to (I believe) the fact, that huffyuv tends to ouput YUY2 even if it is compressed YV12. And as it is not yet possible to restrict what input MagicYUV accepts, and it doesn't do any colorspace conversion of the input, it probably actually compressed YUY2 (instead of YV12). I probably need to an option to restrict accepted colorspaces, that would eliminate these hidden colorspace inconsistencies that others are also experiencing... Greets, I. |
|
13th February 2014, 23:42 | #14 | Link |
Registered User
Join Date: Mar 2011
Posts: 4,829
|
It was annoying me, so I tried again. Same 1920x1040 video, but I decoded it to uncompressed video using Avisynth/ffmsindex/VirtualDubMod/Direct Stream Copy. Then I compressed the 20GB version. I didn't bother with FFV1. I think it's fairly established it's very slow.
Huffyuv (ffdshow variant, adaptive huffman tables enabled), 2 min 33 sec, CPU 70%, 5897MB Lagarith, (mulltithreading), 3 min 1 sec, CPU 80%, 5451MB MagicYUV, (mulltithreading), 2 min 17 sec, CPU 70%, 5823MB Now the results are what I'd expect. Faster compression using the uncompressed video as the source (compared to the original video and Avisynth). Same file sizes. I'm not sure what that means in respect to the ffdshow version of huffyuv being lossless for YV12. I thought it was supposed to be? |
14th February 2014, 01:12 | #15 | Link |
Registered User
Join Date: Dec 2005
Posts: 250
|
Nice
BTW, what MagicYUV really should excel at is decompression speed. If you could do a decompression test, that would be nice too (dunno where to put output though, maybe use avsmeter?). Or better yet, do a recompress cycle using the same codec for input and compression. Also, suggestions are welcome, especially about the colorspace things. Is restriction enough (simple to do) or should I implement conversion? Greets, I. |
14th February 2014, 01:38 | #16 | Link | |
Registered User
Join Date: Mar 2011
Posts: 4,829
|
Quote:
Open the AVI containing huffyuv directly with VirtualDub though and the output is YUY2. Unless..... you disable YUY2 and UVYV as output clolrspaces in the ffdshow VFW decoder configuration. After a fair bit of playing around I'm fairly confident that's what's happening, and while I don't know why, it appears VirtualDub's Fast Recompress option is to blame. Why it seems to only want to convert YV12 to YUY2 when the input codec is huffyuv I don't know, and I don't know why disabling YUY2 and UVYV in the ffdshow list of out output colorspaces stops it from happening. It's kind of like Fast Recompress decides "when it can be decoded as YUY2, it should be"..... Fast recompress: Video is decompressed and then recompressed using the desired output codec. VirtualDub automatically chooses a intermediate video format to use between the codecs for quality and speed. VirtualDubMod's fast recompress doesn't seem to do the same. For it, if it's huffyuv YV12 in, it's YV12 out. |
|
14th February 2014, 15:11 | #17 | Link |
RipBot264 author
Join Date: May 2006
Location: Poland
Posts: 7,812
|
Test updated! http://forum.doom9.org/showthread.php?t=165745
__________________
Windows 7 Image Updater - SkyLake\KabyLake\CoffeLake\Ryzen Threadripper |
15th February 2014, 00:35 | #19 | Link | |
Registered User
Join Date: Nov 2004
Location: Poland
Posts: 2,843
|
Quote:
Next things to do: Mac version, Ffmpeg integration Also as I mentioned before- 16bit is what it would make it stand out from others. Ffv1 is great, but way to slow, specially for 4K. 8bit is starting to be 'not enough' and not just for pro solution, but also for home users. Once you have 16bit, than next thing is to add good dithering as an option . Last edited by kolak; 15th February 2014 at 00:38. |
|
16th February 2014, 00:24 | #20 | Link |
Registered User
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
|
I did quite extensive testing of a pre-release version of Magic YUV several weeks back using his benchmark tool (no-one seems to be testing with it). I'm not sure what modifications he might have made in the interim and, since I don't really have time to repeat the tests with this alpha release, it would seem inappropriate to post the results.
Suffice it to say that when testing Magic YUV against other lossless codecs, it would pay to compare like-with-like - that is, for the Huffman-based codecs to include and compare the different prediction modes (Left and Median, at least) and not just the defaults. Magic YUV defaults to a 'balanced' mode that is designed to automatically select (on a per-frame basis) the optimum balance of compression ratio and decompression speed - or as Ignus2 explained it to me: Dynamic basically just selects the best it thinks per frame, but it is recommended to select it, because if median and left doesn't make a big difference in compression ratio, left is selected, which decodes faster. Dynamic makes a very quick and accurate decision, so there shouldn't be a big difference in speed between dynamic and median on the compression side, but it usually makes a big difference on the decompression side (in favor of dynamic). If one applies the 'conventional wisdom' that median prediction generally makes for higher compression than left, then arguably 'balanced' mode, in prioritizing compression ratio, could be viewed as favoring Median prediction. Maybe Ignus2 would want to elaborate on that. UT-Video, by contrast, defaults to Left prediction - being prioritized for decompression speed. So, to my mind, comparing only the default configurations is a bit 'apples and oranges'. I won't comment further on the pattern of results I obtained (for the reasons stated), but it would be interesting to see what other people find when comparing the decompression and compression rates obtained with the different prediction modes, and including both YV12 and YUY2 sources. And as Ignus2 suggested, maybe look at 're-compression' rates also. Anyhow, now that the alpha release is out, I don't think anyone could dispute that Magic YUV is indeed a very VFW fast lossless codec, at least on par with UT-Video. As Kolak has emphasized however, what would set Magic YUV apart would be the higher bit-depth support. I hope the positive reports that have been coming back encourage Ignus2 to press forward with that.
__________________
Nostalgia's not what it used to be Last edited by WorBry; 16th February 2014 at 04:26. |
Thread Tools | Search this Thread |
Display Modes | |
|
|