MagicYUV - a new fast lossless video codec

[zerowalker]

Hi!
Can we expect a higher compression? Or two passes :o > sample (https://mega.co.nz/#!EEE31ZoT!FTY2fmkgIBhJqy0UhFP-QAiDwvu5vhUVMJY-2LFMVA8)

higher compression can most likely be made one way or the other while keeping the performance. It would probably be quite minimal though as the huge optimizations has probably been made already.

2 passes is probably not something for this kind of codec, which is made for speed in both encoding and decoding (especially decoding), and i think the increase in compression efficiency is probably not worth it during these kind of works.

I personally however, would probably want it if it can decrease the size by at least 5%, while keeping the same decoding speed.

(The author is most likely to correct if i have said anything that's incorrect here).

[kolak]

I think you are correct- any compression improvements while keeping speed will be in region 5, best 10%. Quite often this is a huge effort and all what you get is this 5%. I think magyuv is really good with speed ratio/compression.

[zerowalker]

I think you are correct- any compression improvements while keeping speed will be in region 5, best 10%. Quite often this is a huge effort and all what you get is this 5%. I think magyuv is really good with speed ratio/compression.

Agree, MagicYUV is doing extremely well, it's competing at UT Video Codec, it's in the middle of the compression choices of the 2 settings (compress for decoding/size or something like that).

Which in itself is very nice, and clearly acceptable in lossless format sizes.

Decoding speed is in a different league, it's extremely fast, especially in cases were it usually is slower (much noise,grain etc).

I don't really see any "Real" reason for it to be smaller, faster is another story, there isn't a thing as too fast, but it's currently clearly enough for what it's supposed to do.


However if you need more speed, the author has shown interest in AMD Kaveri, which is a hybrid CPU and GPU, which could increase the performance, as it decreases overhead and such, how much however is another story, and we will have to wait till/if he decides to do this work.

[Ignus2]

Sorry for being absent, I have been busy with personal matters.

In the meantime, MagicYUV 1.0rc3 is out: Link (https://magicyuv.com/index.php/download).

This release includes:

Option to suggest RGB first on output when decoding in the Decompression Settings (as requested by Ghostlamer).
Fix for RGBA encoding which was completely broken. An option was added to the encoder dialog, see the tooltip on how to use it.
Fix for a very rare crash-bug when decoding. Newly encoded videos will have it fixed.
Support for some new fourccs (2vuy, HDYC, IYUV).


@WorBry Thanks and sorry for the late answer. Nice to hear that the codec finally works with the software you use :) All insights and testing is welcome. I really missed RGBA testing though, I accidentally found someone having problems on a German forum, and I only began investigating after that.

@Gravitator Not in the near future. It is hard to improve compression ratio without slowing down the codec, especially if we want to keep it intra-only (making it inter is also non-trivial). So for the foreseeable future, it will remain as it is.

Greets,
I.

[zerowalker]

Fix for a very rare crash-bug. Newly encoded videos will have it fixed.

Could you please explain this?

Does videos that are previously encoded contain a bug which causes or crash when decoded or what;S?

[Ignus2]

Could you please explain this?

Does videos that are previously encoded contain a bug which causes or crash when decoded or what;S?

Basically yes, it MIGHT happen. Actually, it was there since the launch of the codec since February (and even years before, ever since the codec started to take shape), but even I encountered it only once so far with a combination of software AND material (YUV with alpha format converted to RGBA).

So chances are low.

Greets,
I.

[zerowalker]

Okay so i don't have to worry about my current files having a chance of being corrupt?
It's basically, a crash MAY occur if conditions are met, but at the same time if you redo it, it may not crash? (Or is it constant?).

And well what i have is simply YV12 YV16/YUY2 YV24 or RGB, have no memory of ever using YUV with Alpha, didn't even know it had that option.

[Ignus2]

Okay so i don't have to worry about my current files having a chance of being corrupt?
It's basically, a crash MAY occur if conditions are met, but at the same time if you redo it, it may not crash? (Or is it constant?).

And well what i have is simply YV12 YV16/YUY2 YV24 or RGB, have no memory of ever using YUV with Alpha, didn't even know it had that option.

It's constant, not sporadic. For example, I had it happen for one specific frame of one specific file in a specific format with After Effects, while VDub and Media Player Classic and all my test tools were fine with the same file.

Furthermore, some clarification: Any file encoded with any release before 1.0 final is reached should be considered corrupt in one way or another, not specifically because of this bug but because it is beta software and as such, bugs may be present, even ones that I'm not aware of. This is a general thing for all software not being reached 1.0 stable status.

Anyway, if your files open now correctly in VDub, why worry? :)

Greets,
I.

[zerowalker]

Okay that explains it:)

Thanks

[WorBry]

@WorBry Thanks and sorry for the late answer.


No problem :)

[EncodedMango]

The biggest downside I'm finding is support for the codec for example ffmpeg doesn't recognise it. So since usually I record RGB I just convert it to Lagarith and then go about encoding to my needs, I wanted to try YUV 4:4:4 but I can't since ffmpeg won't recognise MAGY and Lagarith doesn't have YV24 but otherwise it's a pretty fast codec.

Are you planning to ever release the source, Ignus2?
If not I guess I'll have to find another lossless codec that can convert MAGY's YUV 4:4:4 to usable YV24.

[zerowalker]

I am not entirely sure this is correct but.

UT Video Codec should be closed source, and ffmpeg has made a decoding version of it as well?
They will probably make MAGY alone or with the authors help, sooner or later.

Though i am interested in if you would release the source, but i guess that's a bit sensitive as you are the maker, and releasing it could be quite controversial in a sense,
so i at the end of the day, as long as it continues to work in all it's glory, i am happy:)

[EncodedMango]

I hope they do, just asked about source because if it does happen then as far as I'm aware someone could submit the codec to ffmpeg that way. I don't know the specifics but I remember glancing over it in the ffmpeg site somewhere.

[vivan]

UT Video Codec should be closed sourceWhy? :)
It's open source, and released under gpl v2.

[zerowalker]

It is?
Was sure it was closes source and release only by the author, need to check on that.
Then my conclusion is invalid.

[Sparktank]

need to check on that.

You can download the source for each version from his site.
http://umezawa.dyndns.info/archive/utvideo/

[Ignus2]

The biggest downside I'm finding is support for the codec for example ffmpeg doesn't recognise it. So since usually I record RGB I just convert it to Lagarith and then go about encoding to my needs, I wanted to try YUV 4:4:4 but I can't since ffmpeg won't recognise MAGY and Lagarith doesn't have YV24 but otherwise it's a pretty fast codec.

Are you planning to ever release the source, Ignus2?
If not I guess I'll have to find another lossless codec that can convert MAGY's YUV 4:4:4 to usable YV24.

Not in the near future. I've prepared an SDK though (binary only) which can be used to integrate the codec into whatever software one wishes, though it's not released as of yet.

Greets,
I.

[Ignus2]

The 1.0rc4 version of the codec is out. This should be the last release before 1.0 final, and if all goes well (no new bugs are reported), this will become the final 1.0 release.

This release fixes a serious decoder bug which could cause random crashes during decoding. It works for all previously encoded files.
Decoder performance was improved by 8-10% overall (mainly for 64-bit).
Memory handling was also improved, the codec now uses much less memory.

Grab the release from the website: http://magicyuv.com (http://magicyuv.com/index.php/download/magicyuv)

Greets,
I.

[StainlessS]

Many Thanks.

[Gargamel]

Congratulations, and thank you again Ignus2 !

[zerowalker]

Nice, never had any crash issues as i recall. But then again have not used it that much i guess.
But it's really one of the top choices, only time i don't use it would be when Lagarith thrives, which are for certain content.

But even faster decoding, 8-10% is great.
How come it's mainly in 64bit. was 64bit slower than 32bit?

Also less memory usage is a good bonus optimization.

Well Done;)!

[Videogamer555]

Please add an uncompressed raw mode as well as your lossless compression mode to your codec. I need to generate some raw YUV444 frames for a specific application I'm working on. The simplest way to do so is to use the IPImageCompress API function in my program. Unfortunately this function requires that I have a codec installed for encoding in YUV. Windows comes with a YUV420 encoder only (though it comes with decoders for other YUV formats). As such, I spend hours, looking for a YUV codec that I could install that would enable me to save YUV444 frames. In my search, I finally ran across MagicYUV. Sadly, it only is capable of saving losslessly compressed YUV frames. For many video applications, lossless compression is better than raw because it saves file space, while preserving the video data that would be lost in a lossy compression. However my application specifically requires raw YUV444 frames. As such, I need to ask you to include a "full raw" mode in the next release of your codec (a mode that would allow directly outputting raw YUV 444, 422, 420, or 400 Luma grayscale frames), so that I can complete the program I'm writing.

I don't think any other codec exists that can do what I want, and yours comes closer than any to being what I need. However as my project calls for raw YUV frames, I am going to ask you to include that in your codec's next release as a settable option from your codec's control panel.

[raffriff42]

I need to generate some raw YUV444 frames for a specific application I'm working on....
I don't think any other codec exists that can do what I want...I can think of one way to generate such a video, with VirtualDub: Set output color depth (https://www.dropbox.com/s/9areszlxbpis00v/virtualdub-export-raw-yuv444-01.jpg?dl=0) to '4:4:4 planar'.
Set video codec (https://www.dropbox.com/s/thcla17749f6qva/virtualdub-export-raw-yuv444-02.jpg?dl=0) to 'uncompressed'
The resulting video is 'rawvideo YV24' per ffprobe (https://www.dropbox.com/s/btj79j8uk0qm3xn/virtualdub-export-raw-yuv444-03.jpg?dl=0).
EDIT
VirtualDub's internal 'uncompressed' codec is not available for other applications, so while my suggestion does generate 'raw YUV444 frames', it doesn't meet your desire for a codec, as pointed out by Ignus2 below. And of course, ffmpeg offers another way to create such a file.

[Ignus2]

Please add an uncompressed raw mode as well as your lossless compression mode to your codec. I need to generate some raw YUV444 frames for a specific application I'm working on. The simplest way to do so is to use the IPImageCompress API function in my program. Unfortunately this function requires that I have a codec installed for encoding in YUV. Windows comes with a YUV420 encoder only (though it comes with decoders for other YUV formats). As such, I spend hours, looking for a YUV codec that I could install that would enable me to save YUV444 frames. In my search, I finally ran across MagicYUV. Sadly, it only is capable of saving losslessly compressed YUV frames. For many video applications, lossless compression is better than raw because it saves file space, while preserving the video data that would be lost in a lossy compression. However my application specifically requires raw YUV444 frames. As such, I need to ask you to include a "full raw" mode in the next release of your codec (a mode that would allow directly outputting raw YUV 444, 422, 420, or 400 Luma grayscale frames), so that I can complete the program I'm writing.

I don't think any other codec exists that can do what I want, and yours comes closer than any to being what I need. However as my project calls for raw YUV frames, I am going to ask you to include that in your codec's next release as a settable option from your codec's control panel.

I'm not entirely sure I understand, but basically you need an RGB to YUV converter "codec"? I wouldn't include this kind of functionality in the codec, it's a bit out of scope.
The solution by raffriff42 should be fine, except if you have to do this conversion in your program.

Greets,
I.

[zerowalker]

Hmm, is there a reason why MagicYUV is not available as a filter with 64bit (GraphStudio-Next for example)?

[Ignus2]

Hmm, is there a reason why MagicYUV is not available as a filter with 64bit (GraphStudio-Next for example)?

Yes, it is currently implemented only as a VFW codec, not a DShow filter. It can join filter graphs however with the AVI Compressor and AVI Decompressor wrapper filters.
It is planned to implement it for DShow too, but it's not high priority (will come probably sometime after 1.0 final).

Greets,
I.

[zerowalker]

Ah okay, didn't know they were different (not that good at this stuff). But that explains it, good that the wrappers can be used as alternatives.

Thanks

[De-M-oN]

Agree, MagicYUV is doing extremely well, it's competing at UT Video Codec

its actually way faster than utvideo.
And thats quite impressive with its compression efficiency.


This codec gives me goddamn awesome fps rates when recording with Afterburner games.
I have a RAID 0 which can write 352 mbyte/s

This codec is able to write in 2560x1600 50fps, RGB with dynamic compression and NO lags at all, an awesome compression efficiency which makes me really wonder about the performance still being even faster than utvideo.

Thats outstanding. Awesome codec which performs hellish fast at recording games while having an excellent compression.

Thank you ! :)

I have this problem though:

http://forums.guru3d.com/showpost.php?p=4898160&postcount=1847

I don't know what could disturb Afterburner here..

[Ignus2]

its actually way faster than utvideo.
And thats quite impressive with its compression efficiency.

This codec gives me goddamn awesome fps rates when recording with Afterburner games.
I have a RAID 0 which can write 352 mbyte/s

This codec is able to write in 2560x1600 50fps, RGB with dynamic compression and NO lags at all, an awesome compression efficiency which makes me really wonder about the performance still being even faster than utvideo.

Thats outstanding. Awesome codec which performs hellish fast at recording games while having an excellent compression.

Thank you ! :)


Thanks, that's great to hear :)
Though decoding performance should be what the codec excels the most, but I'm glad to hear that encoding also performs well :)


I have this problem though:

http://forums.guru3d.com/showpost.php?p=4898160&postcount=1847

I don't know what could disturb Afterburner here..

I'll take a look at it, thanks for reporting.

Greets,
I.

[Ignus2]

I have this problem though:

http://forums.guru3d.com/showpost.php?p=4898160&postcount=1847

I don't know what could disturb Afterburner here..

I seem to have tracked it down. I will try not to be too technical, but the issue is caused by Afterburner initializing/deinitializing the codec just at the same time while loading/unloading the codec DLL (so the OS has the DLL loader lock), and this causes a deadlock inside the codec while it is initializing it's worker threads. (Or at least that is what I think is happening.)
Most probably Afterburner doesn't close the codec properly.

I consider this to be a bug inside Afterburner, and the fix would involve Afterburner to open/close the codec separately from loading/freeing the codec DLL. (Or at least properly close it.)
Though it might be worked around in the codec itself (which might introduce other bugs), but this would force me to make yet another Release Candidate :-/ (as I have the policy of making the final 1.0 release directly from an RC).
I'll think about it...

Thanks for the report BTW!

Greets,
I.

[zerowalker]

How come other codecs does this and MagicYUV doesn't, if i may ask?

[zerowalker]

A suggestion,

Not really a necessity but can't hurt, and i guess it should be fairly easy to add.

Support for: UVYU.

My card seem to have that option, though i use YUY2 as it's the "standard".
But just noticed MagicYUV wasn't listed and UT Video Codec and ffdshow were (those were the only ones).

[Ignus2]

How come other codecs does this and MagicYUV doesn't, if i may ask?

It would be a bit lengthy to explain exactly why this deadlock happens, but in short other codecs probably don't initialize threads when they are loaded (so not creating a problem at DLL unload), and MSI Afterburner only seems to incorrectly close (or not close) codecs when showing it's configuration dialog.

If you are really interested and want to be fascinated by the intricacies of DLLs and threads, here is an article to get you started (especially the "Best Practices for Synchronization" section): :)
http://msdn.microsoft.com/en-us/library/windows/desktop/dn633971%28v=vs.85%29.aspx


BTW, I already fixed/worked around the issue with MSI Afterburner, so expect a release soon (strictly bugfix only).

A suggestion,

Not really a necessity but can't hurt, and i guess it should be fairly easy to add.

Support for: UVYU.

My card seem to have that option, though i use YUY2 as it's the "standard".
But just noticed MagicYUV wasn't listed and UT Video Codec and ffdshow were (those were the only ones).

I guess you wanted to write YVYU. Yes, that could be added, but that will only make it after 1.0, as I made a feature-freeze (should have done that with RC1...) so anything new will have to wait after 1.0 is out.

Greets,
I.

[Ignus2]

@De-M-oN:
I noticed a whole lot of talking on letsplayforum.de about the codec. Thanks for the promotion ;)

First of all, I see some very good feedback coming in, but also some things to reflect on: :)


Predict Left is not necessarily the fastest option (to the contrary of common knowledge) especially when we also consider decoding speed. I wrote about it here why is that so: http://forum.doom9.org/showthread.php?p=1668769#post1668769

When compressing to YUV, the Full range YUV option should be turned off. Programs usually expect YUV to be limited range.

While encoding is also optimized to be fast, but decoding speed should show even bigger speed difference when compared to other codecs.

The decompression setting "Set Avisynth compatible" button only adjusts the checkboxes above it, nothing else. But by default, it is already set up that way, so there is no need to press it. What it basically does is to prevent conversion to YV24 from sub-sampled colorspaces.


That's some I thought about quickly. But feel free to report/ask if something doesn't work as expected.

Greets,
I.

[WorBry]

Predict Left is not necessarily the fastest option (to the contrary of common knowledge) especially when we also consider decoding speed. I wrote about it here why is that so: http://forum.doom9.org/showthread.php?p=1668769#post1668769


Oh boy, my verbosity has, once again, come back to haunt me (cringe) :o

[Ignus2]

Oh boy, my verbosity has, once again, come back to haunt me (cringe) :o

LOL :)
Well I certainly didn't thought about it this way at all, so apologies! :o

Greets,
I.

[zerowalker]

It would be a bit lengthy to explain exactly why this deadlock happens, but in short other codecs probably don't initialize threads when they are loaded (so not creating a problem at DLL unload), and MSI Afterburner only seems to incorrectly close (or not close) codecs when showing it's configuration dialog.

If you are really interested and want to be fascinated by the intricacies of DLLs and threads, here is an article to get you started (especially the "Best Practices for Synchronization" section): :)
http://msdn.microsoft.com/en-us/library/windows/desktop/dn633971%28v=vs.85%29.aspx


BTW, I already fixed/worked around the issue with MSI Afterburner, so expect a release soon (strictly bugfix only).



I guess you wanted to write YVYU. Yes, that could be added, but that will only make it after 1.0, as I made a feature-freeze (should have done that with RC1...) so anything new will have to wait after 1.0 is out.

Greets,
I.

Ah, will see if i understand it to some degree. Get the concept at least, thanks:)

Indeed, damn i even checked and searched to be sure i didn't misspell before posting, not sure how i even succeed:S

And yeah no need for it right now. No need for more RCs ;P
Also not a critical thing, just something trivial that's hopefully can get supported for the heck of it, as it should be simple to implement, as all YUV versions are the same with different positions as far as i get it.

Thanks

[WorBry]

.....so apologies! .

Absolutely none needed. ;)

Great work with codec , btw. I'm using it all the time now as an edit intermediate.

[Ignus2]

1.0rc5 is out: http://magicyuv.com/index.php/download
I REALLY hope it's the last RC before 1.0 final.

This release includes fixes and workarounds for MSI Afterburner, so it is intended only for the users of that app.
Notably, it fixes the deadlock bug, but also includes a compatibility modification for the codec configuration dialog box when the codec detects that it is called from Afterburner.
Basically it changes the OK button of the config dialog to "Save as default and Close" when running from Afterburner, as Afterburner doesn't store VFW codec settings at all, so the settings must be saved globally to the registry to take effect in this case (otherwise anything you set in the dialog would be lost).

No other changes have been made, and if no new bugs are reported, 1.0 final will be released in a week.

Greets,
I.

[zerowalker]

Ah nice. Damn bad implementation from MSI Afterburner though.
Afterburner doesn't store VFW codec settings at all, so the settings must be saved globally to the registry to take effect in this case

Active as always, good job:)

[Ignus2]

1.0 final is out :)

It is basically a re-branded 1.0rc5, so apart from a version bump, no changes have been made.

A long time have passed since I released the first 0.9alpha, and it has been a long road to get here. Thanks for everyone who helped out with suggestions/comments/bug reports!

Now that the feature freeze is lifted, features can come again :) I already have some ideas, and there are already some convenience features done for the next release, so expect a new feature release soon!

Greets,
I.

[zerowalker]

Grats for the 1.0 Release;)!

It's great to see you keeping up the good work, quite the enthusiasm you got.

Care to share any of your, ideas;P?

//Good Job!

[Gargamel]

In time ...
Big thank you, Ignus2 !

[De-M-oN]

I noticed a whole lot of talking on letsplayforum.de about the codec. Thanks for the promotion

It has good performance at recording games, so it was worth mentioning it. :)

Predict Left is not necessarily the fastest option (to the contrary of common knowledge) especially when we also consider decoding speed.

Ok interesting. It felt faster than dynamic while testing though (but just very minor) I use dynamic for the better compression.

When compressing to YUV, the Full range YUV option should be turned off. Programs usually expect YUV to be limited range.

Was that mentioned somewhere to have it enabled? Should be not the case.

While encoding is also optimized to be fast, but decoding speed should show even bigger speed difference when compared to other codecs.

Ok. But it uses not much cpu at encoding and is pretty fast as well. And thats good.
Encodingspeed while having as low cpu usage as possible is for us the more relevant thing we need for recording games.

The decompression setting "Set Avisynth compatible" button only adjusts the checkboxes above it, nothing else.
I know.

But by default, it is already set up that way, so there is no need to press it. What it basically does is to prevent conversion to YV24 from sub-sampled colorspaces.

For me all checkmarks were set. After pressing avisynth compatible the checkmarks at yuv 444 went away.

Thank you very much for fixing the Afterburner problem although it was more a fault by afterburner.

Thank you!

[Ignus2]

@zerowalker:
I'll try them out first then I'll tell about it ;-)

@De-M-On:
Thanks, you're welcome. If you encounter any other strange behaviour or bugs, just tell me.

Greets,
I.

[nekosama]

I'm having quite a hard time getting magicyuv to encode using avisynth. Everytime I try to encode my script (using a magicyuv lossless avi as source using avisource) I get 240 blank frames, hell even just plain opening the file in a mediaplayer (mpc-hc) gives 240 blan frames, only opening the script in avspmod gives me correct results.

Maybe it has something to do with NULL frames like lagarith but I don't even know if magicyuv makes use of them.

btw it's not caused by some corrupted plugin of some sort, hell even just calling source and using removegrain (or any simple filter for that matter) ends up in 240 blank frames in a mediaplayer.

[zerowalker]

I'm having quite a hard time getting magicyuv to encode using avisynth. Everytime I try to encode my script (using a magicyuv lossless avi as source using avisource) I get 240 blank frames, hell even just plain opening the file in a mediaplayer (mpc-hc) gives 240 blan frames, only opening the script in avspmod gives me correct results.

Maybe it has something to do with NULL frames like lagarith but I don't even know if magicyuv makes use of them.

btw it's not caused by some corrupted plugin of some sort, hell even just calling source and using removegrain (or any simple filter for that matter) ends up in 240 blank frames in a mediaplayer.


I have been using MagicYUV A lot with Avisynth with no problems at all. Even when null frames has been present (Which MagicYUV does Not support in itself, however it's a global feature in AVI).

Really weird issue. You sure it's not some x64/x86 thing?

[De-M-oN]

My GZDoom edited pfenh mod uses some effects which are very cpu intensive and so my recording fps seldom goes below the wanted. In this case dummy frames are inserted as well then.

Never had a problem with it though.

Avisynth reads it fine, vdub 32bit does it, mpc-hc 32bit as well (EVR-CP Renderer)

So I think for me it works.

[nekosama]

nope, Tried both x86 and x64 x264 but no luck. Maybe it has something to do with the decoder? is there something that goes between avisynth and x264 (ignoring avs4x264mod because nothing changes whether or not I use this).

[zerowalker]

I have used MagicYUV with x264 both directly and with avs2x264mod, no issues here.