Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > New and alternative video codecs

Reply
 
Thread Tools Search this Thread Display Modes
Old 17th February 2014, 01:32   #21  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Quote:
Originally Posted by WorBry View Post
If one applies the 'conventional wisdom' that median prediction generally makes for higher compression than left, then arguably 'balanced' mode, in prioritizing compression ratio, could be viewed as favoring Median prediction. Maybe Ignus2 would want to elaborate on that.
Well, that conventional wisdom can be wrong sometimes
An important thing to mention here is that decoding time doesn't only depend on the prediction method. Bad prediction can slow down other parts of the codec. This is exactly why we cannot outright say Left prediction makes the codec fast.
Dynamic combines the best of both worlds. Use the simplest prediction method giving good compression.
For a tiny bit of more work on the encoder side we can get big wins both in compression ratio and decoding speed.

This is also why I explicitly don't recommend in the codec config dialog selecting Left because the results might not be what people expect (file will be bigger AND slower to decode).
Actually I'm considering to remove the prediction method selector altogether, as it might confuse people thinking Predict Left is equal to a some kind of "Fast Mode", where in reality it isn't.

As for the immediate future, I'll probably do color space conversion built-in to the encoder side (to allow direct RGB->YUY2/YV12 compression), and after that, 10-bit might come along (or Mac, but I never coded a Mac so far...).
But again, as I already mentioned in the original Intermediate Video Format Survey thread, 10-bit is a bigger step, and that might fork into a non-free version (IF I'll ever have the time to do it). (But I will add that the current and every coming 8-bit version will always be free.)

Greets,
I.
Ignus2 is offline   Reply With Quote
Old 17th February 2014, 05:03   #22  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,187
Quote:
Originally Posted by Ignus2 View Post
Well, that conventional wisdom can be wrong sometimes
An important thing to mention here is that decoding time doesn't only depend on the prediction method. Bad prediction can slow down other parts of the codec. This is exactly why we cannot outright say Left prediction makes the codec fast.
Dynamic combines the best of both worlds. Use the simplest prediction method giving good compression.
For a tiny bit of more work on the encoder side we can get big wins both in compression ratio and decoding speed.
Fair enough, but as you will recall from the last set of test results with the pre-release version; in 32-bit mode, Magic-YUV, set to Left Prediction, did decode faster (12%) than Dynamic (sorry I called it 'Balanced' before) and Median, (which were basically on par in terms of decode rate and compression ratio) and faster (12% again) than UT-Video in default Left prediction mode. When tested in 64-bit mode, however, the Magic-YUV encodes (Left, Median, Dynamic) all decoded at similar rates, but UT-Video (Left) was faster (7%) than them all. This would reflect the improvements that had been made in 64-bit decompression rate in the last update of UT-Video. That said, UT-Video encoded with Median prediction decompressed the slowest of all, at around half the rate of UT-Video (Left) in 32-bit and 64-bit. That was using YV12 sources.

With YUY2 source, a different picture emerged. In that case, Magic-YUV Left, Median and Dynamic encodes decompressed at similar rates in 32-bit mode, with UT-Video (Left) decoding a tad faster. In 64-bit mode, UT-Video Left again was the fastest, but then Magic-YUV encoded in Dynamic mode decoded the fastest and Left prediction the slowest. So, quite the opposite of the results obtained in YV12.

As for compression rates; Magic-YUV in Dynamic mode consistently encoded the slowest of all the prediction modes, and UT-Video (both Left and Median) was consistently faster than Magic-YUV in all prediction modes. That pattern was evident in both 32-bit and 64-bit test modes and with both YV12 and YUY2 (raw) sources. All using your bench mark tool.

As for compression ratios. Yes, with YUY2 sources, Magic-YUV 'Dynamic' did yield marginally higher compression ratios than 'Median', but with YV12 sources they were basically the same. That said, with YUY2 and YV12 sources, UT-Video compressed marginally better in both Left and Median modes than the corresponding modes in Magic-YUV, and on par with HuffYuv (FFDS version).

So, where Magic-YUV did shine in comparison to UT-Video was (in my tests at least) in the decode rate of 'Median' configured encodes (in 32-bit and 64-bit), where UT-Video was significantly slower, and 'Left' configured encodes in 32-bit. But in all other respects UT-Video was faster and compressed slightly better. And yes, it did appear that the 'Dynamic' mode has a bias to Median prediction, as overall the results were not that different. And that was how I understood your explanation. If it's more complex than that, all well and good.

Granted, these tests were performed with a pre-release version on one PC system (Hex-core AMD cpu, Win7 64-bit) using your benchmark tool, and others could well obtain a different results with this alpha release, on different PC's, with different source material and using different performance test criteria - as is already evident from the the several early reports.

All I am suggesting, really, is that to make a fair comparison, one should test with UT-Video in both Left (default) and Median modes in order to best judge how Magic YUV (in default Dynamic mode) performs on the balance of speed and compression.

Quote:
Originally Posted by Ignus2 View Post
As for the immediate future, I'll probably do color space conversion built-in to the encoder side (to allow direct RGB->YUY2/YV12 compression)
Although the vfw api is an antiquated/deprecated interface, the reality is that many NLE's still use vfw with third party codecs and operate (at the edit level) in RGB color-space. Given that one potential use of Magic-YUV would be as a lossless edit intermediate in such a system, I think the integrated RGB input would be worthwhile. As you know, when I tried Magic-YUV as an edit intermediate in the (consumer level) NLE I use (Corel Video Studio "Pro" x6) it crashed, but 'magically' worked when I installed the Helix YUV codecs pack, which presumably served as an RGB to YUV interface for Magic YUV - you were going to look into that. UT-Video works perfectly.
__________________
Nostalgia's not what it used to be

Last edited by WorBry; 17th February 2014 at 16:52.
WorBry is offline   Reply With Quote
Old 17th February 2014, 09:30   #23  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,820
how does multithreading encoding work in magicyuv? does it work on frame level or does it use slices?
Atak_Snajpera is offline   Reply With Quote
Old 17th February 2014, 10:14   #24  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Quote:
Originally Posted by Atak_Snajpera View Post
how does multithreading encoding work in magicyuv? does it work on frame level or does it use slices?
Slices. VFW is one frame in one frame out, so slices are the best option.
Ignus2 is offline   Reply With Quote
Old 17th February 2014, 10:31   #25  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,820
I wonder why MagicYUV and UT VIDEO do not use more than 2 cores in my tests? RAMDISK + uncompressed source and still no more than 50% during encoding.
Atak_Snajpera is offline   Reply With Quote
Old 17th February 2014, 13:23   #26  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Quote:
Originally Posted by Atak_Snajpera View Post
I wonder why MagicYUV and UT VIDEO do not use more than 2 cores in my tests? RAMDISK + uncompressed source and still no more than 50% during encoding.
Maybe a color space conversion happening somewhere in the background on a single core?
What is the test setup/pipeline for the benchmark?
Ignus2 is offline   Reply With Quote
Old 17th February 2014, 14:39   #27  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,820
avs script has only rawsource("video.y4m")

i get 130 fps using avsmeter with this script.

virtualdub is set to fast recompress mode. according to process explorer virtualdub.exe does not use more than 2 cores ( 50% - Q6600@3GHz)
Atak_Snajpera is offline   Reply With Quote
Old 17th February 2014, 19:56   #28  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: UK
Posts: 2,370
Quote:
Originally Posted by Atak_Snajpera View Post
I wonder why MagicYUV and UT VIDEO do not use more than 2 cores in my tests? RAMDISK + uncompressed source and still no more than 50% during encoding.
How fast is your disk?
How fast is reading raw file?
These codecs should be using around 100% CPU, if not than there is some bottleneck.
The fact that both codecs are stucked at 50% means something.

Last edited by kolak; 17th February 2014 at 20:01.
kolak is offline   Reply With Quote
Old 17th February 2014, 20:57   #29  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,820
ramdisk amigo! DDR2-666 mhz
Atak_Snajpera is offline   Reply With Quote
Old 17th February 2014, 21:46   #30  |  Link
kolak
Registered User
 
Join Date: Nov 2004
Location: UK
Posts: 2,370
Try avibench tool to check if you have same results.
Vdub is good but not always that transparent.

I was able to saturate 24 threads with Utvideo, which gave 1200fps and 2 GByte/sec reading. This was raid box.
Not sure why you have only 50% CPU usage.

Last edited by kolak; 17th February 2014 at 21:53.
kolak is offline   Reply With Quote
Old 24th February 2014, 11:03   #31  |  Link
pandy
Registered User
 
Join Date: Mar 2006
Posts: 1,044
Are You consider to support YCoCg color space to support reversible RGB color space conversion?
pandy is offline   Reply With Quote
Old 4th March 2014, 17:36   #32  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Quote:
Originally Posted by pandy View Post
Are You consider to support YCoCg color space to support reversible RGB color space conversion?
I've read about that some time ago, though I don't really see the benefit.
Do you have some source that produces YCoCg?
Or do you consider RGB input and want to store it in YCoCg internally? In this case, what is the benefit (a bit more compression for RGB maybe)?
Also, to be reversible, it requires 2 extra bits.

Greets,
I.
Ignus2 is offline   Reply With Quote
Old 5th March 2014, 15:23   #33  |  Link
HulkHoganRules
Registered User
 
Join Date: Aug 2012
Posts: 28
I'm really into video game capturing so I wanted to see how it performed. I honestly got lost in this thread but I have some questions if you don't mind.

If I supply this codec a YUY2 source, will it stay YUY2 or will it convert to YV12? The reason I ask is I captured the same frame in MagicYUV and then with Lagarith YUY2 and then YV12. I ran objective video quality tests (MSU) using the MagicYUV frame against both Lagarith YUY2 and then YV12. The comparisons were 99.?% similar. If MagicYuv is in YUY2, I think this is wrong right? I.e. it should be 100% similar since it's a lossless codec?

Many months ago I remember comparing Lagarith YUY2 to x264vfw Lossless mode and that gave me a perfect 100% identical reading. I guess I can only assume that MagicYUV does convert YUY2 to YV12 slightly differently to Lagarith's YV12?

One thing that did surprise me and is really awesome about this codec. Both Lagarith and UT Video completely crumbled under my 1080p60 Xbox 360 tests. I captured with MagicYuv perfectly fine at 1080p60! Picture: http://puu.sh/7jSui.jpg

Thank you.
HulkHoganRules is offline   Reply With Quote
Old 6th March 2014, 11:06   #34  |  Link
pandy
Registered User
 
Join Date: Mar 2006
Posts: 1,044
Quote:
Originally Posted by Ignus2 View Post
I've read about that some time ago, though I don't really see the benefit.
Do you have some source that produces YCoCg?
Or do you consider RGB input and want to store it in YCoCg internally? In this case, what is the benefit (a bit more compression for RGB maybe)?
Also, to be reversible, it requires 2 extra bits.

Greets,
I.
Yes - RGB converted internally to YCoCg and back on decoding to RGB, all pros and cons but for example for screen capture especially on 4:4:4 it can be beneficial and provide fully reversible if necessary RGB back conversion. At minimal speed loss (YCoCg can be very fast - i assume perhaps faster than RGB to YCbCr) potential compressibility comparable to YCbCr (ie higher than on RGB).
I would say it can be nice as none of commonly used lossless codecs support this feature.
And as you consider 16 bit depth then perhaps 12 bit (for 10 bit RGB) is not a problem. Not sure as I've said previously - you can be first with such feature.

Big ThX and also greetz.
pandy is offline   Reply With Quote
Old 7th March 2014, 06:07   #35  |  Link
raffriff42
Retried Guesser
 
raffriff42's Avatar
 
Join Date: Jun 2012
Posts: 1,377
Harping again on this 10-bit thing, Ignus2... suppose you added 10-bit support by compressing the 8 most significant bits in the usual way, and the 2 least significant bits using a separate, simple method, eg run-length or zlib? Compression would not be "optimal," but you would have the only fast, lossless 10-bit encoder around.
raffriff42 is offline   Reply With Quote
Old 10th March 2014, 19:23   #36  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Thanks for all the suggestions!

A bit of status report:
Currently I'm in the middle of refactoring the code, the result of which will allow much easier development in the future, also adding new colorspaces and internal color space conversions will be much easier. This should also pave the way for 10bit+ (but that's a plan only for after 1.0).

Sadly this also means that I had to change the compressed stream format, which is incompatible with the current one (of 0.9alpha), so the next release will not be able to decode videos encoded with 0.9alpha. This will absolutely not happen after 1.0 is released, I do this now, as I don't intend to keep backward compatibility with alpha releases.

Stay tuned

Greets,
I.

Last edited by Ignus2; 10th March 2014 at 19:29.
Ignus2 is offline   Reply With Quote
Old 13th March 2014, 01:12   #37  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 3,503
Very interesting stuff, looking forward to it.
Asmodian is offline   Reply With Quote
Old 13th March 2014, 10:35   #38  |  Link
Gargamel
Registered User
 
Gargamel's Avatar
 
Join Date: Aug 2010
Location: Bretagne, France
Posts: 47
+1
Thanks and congratulations, maybe (we hope so) the birth of a reference codec !
Gargamel is offline   Reply With Quote
Old 21st March 2014, 00:17   #39  |  Link
Ignus2
Registered User
 
Join Date: Dec 2005
Posts: 249
Thanks!

The refactoring is now done, some smaller things remain, so a release is to be expected soon (next week at the latest probably).
Performance-wise there isn't really a big difference, though the new version might be faster in some cases

The stream format had to be changed, as I mentioned, though I'm quite satisfied with it now, so if all goes well, it will stay for 1.0 and beyond.

The next release won't see any major new features, though I added an option to restrict the kind of colorspaces the codec will accept (useful to make sure you really give the codec the colorspace you work in and expect - so conversion don't silently happen somewhere). Built-in encoder-side colorspace conversions (direct encoding from RGB to YV12 for example) are expected after this release, and I hope to include them in 1.0.

More to come soon!

Greets,
I.
Ignus2 is offline   Reply With Quote
Old 21st March 2014, 09:14   #40  |  Link
Gargamel
Registered User
 
Gargamel's Avatar
 
Join Date: Aug 2010
Location: Bretagne, France
Posts: 47
Many are waiting for... and salivating !
May the compiling Force be with you ;-)
Gargamel is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 01:43.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.