MagicYUV - a new fast lossless video codec

[Ignus2]

Hi!

As I mentioned in the Intermediate Video Format survey thread, I've been developing a new fast lossless video codec (named MagicYUV) for Windows, it's been basically ready since a year ago, I just never got the time to release it.

But now it's here, I've fixed some remaining bugs I've known and I guess it's ready for an alpha release.

I've set up a website, you can grab it from there: http://magicyuv.com

I've also uploaded a command line tool avibench there, that can measure codec compression/decompression performance through AVI files.

Any kind of suggestions, comments, bug and crash reports, feature requests can come here. The target is to make it to 1.0 stable :)

Greets,
I.

[LoRd_MuldeR]

Thank you. I will definitely give that one a try.

Do you have any graphs on how this one compares to other lossless video Codecs (HuffYUV, Lagarith, FFV1, x264) in terms of speed and compression?

BTW: Can't really see anywhere which license this is released under. So I assume it's ClosedSource, right?

[Ignus2]

Thank you. I will definitely give that one a try.

Do you have any graphs on how this one compares to other lossless video Codecs (HuffYUV, Lagarith, FFV1, x264) in terms of speed and compression?

BTW: Can't really see anywhere which license this is released under. So I assume it's ClosedSource, right?

Well, I do have some graphs, but it would be best if the users could test it ;)
As a quick reference: for decompression in Dynamic mode it "should" be considerably (2-3x) faster than Ut Median with around same comp. ratio and faster (1.4x) than Ut Left with better (10%) comp. ratio. Without Disk I/O of course (measured with ramdisk/fully cached avi). At least on my machine... And I'm really curious about other machines...
Against FFV1 and x264 I didn't compare yet.

About the license: well, I'm not a lawyer, so I can't come up with licenses, so I'll just say that the version you can download from the site is free to use for whatever you want and the sources are not included :)

Greets,
I.

[EncodedMango]

Seems interesting, gonna try it out.

EDIT:Okay so I just tried it using Dxtory(so 32bit version) and I must say the results seem impressive, I had(edit2) some FPS loss when recording but it didn't feel incredibly laggy. This beats Lagarith for me in this test atleast great job Ignus2.

[hello_hello]

Recompressing a 3min, 49sec 704x384 AVI with VirtualDubMod (fast recompress). No audio. Output written to the first partition of a RAID-0 volume to take hard drive speed out the equation as much as possible. YV12 -> YV12. Total time as displayed by VirtualDubMod.
Old E6750 dual core CPU, 3.2GHz. For Lagarith and MagicYUV there's two time and CPU usage figures. Without and with multi-threading enabled. CPU usage is a fairly rough average anyway.

Huffyuv (ffdshow variant), 16 seconds, CPU 62%, 990.8MB
Huffyuv (ffdshow variant, adaptive huffman tables enabled), 16 seconds, CPU 62%, 587.2MB

FFV1, 62 seconds, CPU 58%, 432.6MB

Lagarith, 18/15 seconds, CPU 60%/73%, 503MB

MagicYUV, 17/12 seconds, CPU 60%/83%, 586.8MB

For a hard drive speed reference..... copying the largest file (990.8MB) from the location of the source file to the same location as the output files took around 6 seconds.

And for fun:

Uncompressed YV12 (Avisynth script/Direct Stream Copy), 15 seconds, CPU 45%, 2.1GB
Uncompressed RGB, 30 seconds, CPU 45%, 4.3GB

[LoRd_MuldeR]

About the license: well, I'm not a lawyer, so I can't come up with licenses, so I'll just say that the version you can download from the site is free to use for whatever you want and the sources are not included :)

You don't need to be a lawyer. If you ever decide to "open" your sources, you could just put them, for example, under a (L)GPL license - which would make it possible to integrate your Codec into FFmpeg (libavcodec), thus making it available in pretty much any OpenSource tool. Currently it's tied not only to the Windows platform but also to the (deprecated) VFW interface, which limits its applications, of course. And it has every user to rely on you to provide "free" decoders in the future (the last point might not be that critical for an "intermediate" Codec though)

[Ignus2]

Recompressing a 3min, 49sec 704x384 AVI with VirtualDubMod (fast recompress).
...


The codec is not yet tuned optimally for such small resolutions (on my TODO list), if you could try the same test with HD (1920x1080) or higher, that would be nice :-)

Greets,
I.

[Atak_Snajpera]

I've just updated my old test http://forum.doom9.org/showthread.php?t=165745

I don't know why but file is larger than source .y4m !?

General
Complete name : G:\job1.avi
Format : AVI
Format/Info : Audio Video Interleave
File size : 859 MiB
Duration : 10s 0ms
Overall bit rate : 721 Mbps
Writing library : VirtualDub build 32842/release

Video
ID : 0
Format : MAGY
Codec ID : MAGY
Duration : 10s 0ms
Bit rate : 721 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 fps
Bits/(Pixel*Frame) : 13.903
Stream size : 859 MiB (100%)


General
Complete name : G:\job1.y4m
Format : YUV4MPEG2
File size : 742 MiB

Video
Format : YUV
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 fps
Color space : YUV
Scan type : Progressive
Compression mode : Lossless

[Ignus2]

I've just updated my old test http://forum.doom9.org/showthread.php?t=165745

I don't know why but file is larger than source .y4m !?


That's a colorspace issue I guess (or something similar). I think the stream actually gets encoded as RGB probably.
MagicYUV actually compresses what it gets without colorspace conversion, and it is currently not possible to force or restrict what colorspace the codec accepts, so if VirtualDub is set up or decides to give MagicYUV RGB input, it will compress like that.

--
Greets,
I.

[Atak_Snajpera]

i will redo test tomorrow then.

[Ignus2]

i will redo test tomorrow then.

That would be great, I'm very interested in the results :)

[hello_hello]

5 minute, 1920x1040 MKV with Avisynth/ffmsindex/VirtualDub (fast recompress). ffms in single threaded mode. No audio. Output written to the first partition of a RAID-0 volume to take hard drive speed out the equation as much as possible. YV12 -> YV12. Total time as displayed by VirtualDub.
Old E6750 dual core CPU, 3.2GHz. CPU usage is a fairly rough average. Encoding the first 15 to 30 seconds indicated multithreaded mode was slower for both Lagarith and MagicYUV so I disabled it.

Huffyuv (ffdshow variant), 4 min 23 sec, CPU 85%, 9542MB
Huffyuv (ffdshow variant, adaptive huffman tables enabled), 4 min 1 sec, CPU 80%, 5897MB

FFV1, 11 min 4 sec, CPU 70%, 4935MB

Lagarith, (no mulltithreading, enabling it slowed encoding speed by 2 or 3 fps), 4 min 7 sec, CPU 90%, 5451MB

MagicYUV, (no mulltithreading, enabling it slowed encoding speed by 2 or 3 fps), 4 min 23 sec, CPU 97%, 5823MB


Uncompressed YV12, (Avisynth/ffmsindex/VirtualDubMod/Direct Stream Copy), 4 min 14 sec, CPU 65%, 20572MB


I ran most of the above encodes twice to check them, given the next results weren't what I expected.......

As an experiment I thought I'd take Avisynth and ffmsindex out of the equation and reduce the CPU usage (hopefully) required for decoding. I took the second huffyuv encode and put it on the same drive as the original MKV. The 5897MB file took about 1 minute to move, so obviously hard drive speed wasn't a bottleneck. I then opened it directly with VirtualDub and re-compressed it (fast recompress). Same output file location as before. This time multithreaded mode was a little faster than single threaded mode for MagicYUV and Lagarith.

MagicYUV, (mulltithreading enabled), 4 min 02 sec, CPU 70%, 6683MB
I run the encode twice. With huffyuv as the source MagicYUV was faster. It also compressed less.

Lagarith, (mulltithreading enabled), 4 min 54 sec, CPU 65%, 5404MB
With huffyuv as the source Lagarith slowed down quite a bit. The file size decreased a tiny bit. I don't understand why. I tried again without multithreading:
Lagarith, (no mulltithreading), 5 min 29 sec, CPU 55%, 5404MB

Huffyuv (ffdshow variant, adaptive huffman tables enabled), 4 min 38 sec, CPU 55%, 5897MB
And with huffyuv as the source, huffyuv slowed down too. No file size change though. Input and output file sizes were exactly the same.

Someone will have to explain the speed/compression changes for me. Maybe the huffyuv version isn't YV12, although as best as I can tell it is. I even encoded it again using the "force input colorspace option" and setting it to YV12 and the file size after encoding (from the original video) still remained unchanged. When viewing the huffyuv encode with MPC-HC, ffdshow reports the output as NV12 (as it usually does when decoding YV12 with MPC-HC). But enough for today.....

[Ignus2]

5 minute, 1920x1040 MKV with Avisynth/ffmsindex/VirtualDub (fast recompress).
...


Thanks.

What codec and size is the original MKV?

The file becoming bigger for MagicYUV from huffyuv source in fast recompress is due to (I believe) the fact, that huffyuv tends to ouput YUY2 even if it is compressed YV12. And as it is not yet possible to restrict what input MagicYUV accepts, and it doesn't do any colorspace conversion of the input, it probably actually compressed YUY2 (instead of YV12).

I probably need to an option to restrict accepted colorspaces, that would eliminate these hidden colorspace inconsistencies that others are also experiencing...

Greets,
I.

[hello_hello]

It was annoying me, so I tried again. Same 1920x1040 video, but I decoded it to uncompressed video using Avisynth/ffmsindex/VirtualDubMod/Direct Stream Copy. Then I compressed the 20GB version. I didn't bother with FFV1. I think it's fairly established it's very slow.

Huffyuv (ffdshow variant, adaptive huffman tables enabled), 2 min 33 sec, CPU 70%, 5897MB

Lagarith, (mulltithreading), 3 min 1 sec, CPU 80%, 5451MB

MagicYUV, (mulltithreading), 2 min 17 sec, CPU 70%, 5823MB

Now the results are what I'd expect. Faster compression using the uncompressed video as the source (compared to the original video and Avisynth). Same file sizes.
I'm not sure what that means in respect to the ffdshow version of huffyuv being lossless for YV12. I thought it was supposed to be?

[Ignus2]

Nice :)

BTW, what MagicYUV really should excel at is decompression speed. If you could do a decompression test, that would be nice too (dunno where to put output though, maybe use avsmeter?). Or better yet, do a recompress cycle using the same codec for input and compression.

Also, suggestions are welcome, especially about the colorspace things. Is restriction enough (simple to do) or should I implement conversion?

Greets,
I.

[hello_hello]

Thanks.

What codec and size is the original MKV?

x264, 415MB

The file becoming bigger for MagicYUV from huffyuv source in fast recompress is due to (I believe) the fact, that huffyuv tends to ouput YUY2 even if it is compressed YV12. And as it is not yet possible to restrict what input MagicYUV accepts, and it doesn't do any colorspace conversion of the input, it probably actually compressed YUY2 (instead of YV12).

I think you're correct but I'm not sure who's fault it is. I'm pretty sure the huffyuv version is encoded as YV12 and I'm pretty sure when it's opened in a player and decoded via directshow it's decoded as YV12. Create an AviSource script to open it and add Info() and it's reported as YV12. Whether it's opened by MPC-HC or VirtualDub it's YV12.
Open the AVI containing huffyuv directly with VirtualDub though and the output is YUY2. Unless..... you disable YUY2 and UVYV as output clolrspaces in the ffdshow VFW decoder configuration.

After a fair bit of playing around I'm fairly confident that's what's happening, and while I don't know why, it appears VirtualDub's Fast Recompress option is to blame. Why it seems to only want to convert YV12 to YUY2 when the input codec is huffyuv I don't know, and I don't know why disabling YUY2 and UVYV in the ffdshow list of out output colorspaces stops it from happening. It's kind of like Fast Recompress decides "when it can be decoded as YUY2, it should be".....

Fast recompress: Video is decompressed and then recompressed using the desired output codec. VirtualDub automatically chooses a intermediate video format to use between the codecs for quality and speed.

VirtualDubMod's fast recompress doesn't seem to do the same. For it, if it's huffyuv YV12 in, it's YV12 out.

[Atak_Snajpera]

Test updated! http://forum.doom9.org/showthread.php?t=165745

[kolak]

Did you use 32bit?
64bit MagicYuv is way faster, at least for decoding.

[kolak]

Nice :)

BTW, what MagicYUV really should excel at is decompression speed. If you could do a decompression test, that would be nice too (dunno where to put output though, maybe use avsmeter?). Or better yet, do a recompress cycle using the same codec for input and compression.

Also, suggestions are welcome, especially about the colorspace things. Is restriction enough (simple to do) or should I implement conversion?

Greets,
I.

I think the speed is great, so we are fine here.

Next things to do:
Mac version,
Ffmpeg integration

Also as I mentioned before- 16bit is what it would make it stand out from others. Ffv1 is great, but way to slow, specially for 4K. 8bit is starting to be 'not enough' and not just for pro solution, but also for home users.

Once you have 16bit, than next thing is to add good dithering as an option .

[WorBry]

I did quite extensive testing of a pre-release version of Magic YUV several weeks back using his benchmark tool (no-one seems to be testing with it). I'm not sure what modifications he might have made in the interim and, since I don't really have time to repeat the tests with this alpha release, it would seem inappropriate to post the results.

Suffice it to say that when testing Magic YUV against other lossless codecs, it would pay to compare like-with-like - that is, for the Huffman-based codecs to include and compare the different prediction modes (Left and Median, at least) and not just the defaults.

Magic YUV defaults to a 'balanced' mode that is designed to automatically select (on a per-frame basis) the optimum balance of compression ratio and decompression speed - or as Ignus2 explained it to me:

Dynamic basically just selects the best it thinks per frame, but it is recommended to select it, because if median and left doesn't make a big difference in compression ratio, left is selected, which decodes faster. Dynamic makes a very quick and accurate decision, so there shouldn't be a big difference in speed between dynamic and median on the compression side, but it usually makes a big difference on the decompression side (in favor of dynamic).

If one applies the 'conventional wisdom' that median prediction generally makes for higher compression than left, then arguably 'balanced' mode, in prioritizing compression ratio, could be viewed as favoring Median prediction. Maybe Ignus2 would want to elaborate on that.

UT-Video, by contrast, defaults to Left prediction - being prioritized for decompression speed. So, to my mind, comparing only the default configurations is a bit 'apples and oranges'.

I won't comment further on the pattern of results I obtained (for the reasons stated), but it would be interesting to see what other people find when comparing the decompression and compression rates obtained with the different prediction modes, and including both YV12 and YUY2 sources. And as Ignus2 suggested, maybe look at 're-compression' rates also.

Anyhow, now that the alpha release is out, I don't think anyone could dispute that Magic YUV is indeed a very VFW fast lossless codec, at least on par with UT-Video. As Kolak has emphasized however, what would set Magic YUV apart would be the higher bit-depth support. I hope the positive reports that have been coming back encourage Ignus2 to press forward with that.

[Ignus2]

If one applies the 'conventional wisdom' that median prediction generally makes for higher compression than left, then arguably 'balanced' mode, in prioritizing compression ratio, could be viewed as favoring Median prediction. Maybe Ignus2 would want to elaborate on that.


Well, that conventional wisdom can be wrong sometimes :)
An important thing to mention here is that decoding time doesn't only depend on the prediction method. Bad prediction can slow down other parts of the codec. This is exactly why we cannot outright say Left prediction makes the codec fast.
Dynamic combines the best of both worlds. Use the simplest prediction method giving good compression.
For a tiny bit of more work on the encoder side we can get big wins both in compression ratio and decoding speed.

This is also why I explicitly don't recommend in the codec config dialog selecting Left because the results might not be what people expect (file will be bigger AND slower to decode).
Actually I'm considering to remove the prediction method selector altogether, as it might confuse people thinking Predict Left is equal to a some kind of "Fast Mode", where in reality it isn't.

As for the immediate future, I'll probably do color space conversion built-in to the encoder side (to allow direct RGB->YUY2/YV12 compression), and after that, 10-bit might come along (or Mac, but I never coded a Mac so far...).
But again, as I already mentioned in the original Intermediate Video Format Survey thread, 10-bit is a bigger step, and that might fork into a non-free version (IF I'll ever have the time to do it). (But I will add that the current and every coming 8-bit version will always be free.)

Greets,
I.

[WorBry]

Well, that conventional wisdom can be wrong sometimes :)
An important thing to mention here is that decoding time doesn't only depend on the prediction method. Bad prediction can slow down other parts of the codec. This is exactly why we cannot outright say Left prediction makes the codec fast.
Dynamic combines the best of both worlds. Use the simplest prediction method giving good compression.
For a tiny bit of more work on the encoder side we can get big wins both in compression ratio and decoding speed.


Fair enough, but as you will recall from the last set of test results with the pre-release version; in 32-bit mode, Magic-YUV, set to Left Prediction, did decode faster (12%) than Dynamic (sorry I called it 'Balanced' before) and Median, (which were basically on par in terms of decode rate and compression ratio) and faster (12% again) than UT-Video in default Left prediction mode. When tested in 64-bit mode, however, the Magic-YUV encodes (Left, Median, Dynamic) all decoded at similar rates, but UT-Video (Left) was faster (7%) than them all. This would reflect the improvements that had been made in 64-bit decompression rate in the last update of UT-Video. That said, UT-Video encoded with Median prediction decompressed the slowest of all, at around half the rate of UT-Video (Left) in 32-bit and 64-bit. That was using YV12 sources.

With YUY2 source, a different picture emerged. In that case, Magic-YUV Left, Median and Dynamic encodes decompressed at similar rates in 32-bit mode, with UT-Video (Left) decoding a tad faster. In 64-bit mode, UT-Video Left again was the fastest, but then Magic-YUV encoded in Dynamic mode decoded the fastest and Left prediction the slowest. So, quite the opposite of the results obtained in YV12.

As for compression rates; Magic-YUV in Dynamic mode consistently encoded the slowest of all the prediction modes, and UT-Video (both Left and Median) was consistently faster than Magic-YUV in all prediction modes. That pattern was evident in both 32-bit and 64-bit test modes and with both YV12 and YUY2 (raw) sources. All using your bench mark tool.

As for compression ratios. Yes, with YUY2 sources, Magic-YUV 'Dynamic' did yield marginally higher compression ratios than 'Median', but with YV12 sources they were basically the same. That said, with YUY2 and YV12 sources, UT-Video compressed marginally better in both Left and Median modes than the corresponding modes in Magic-YUV, and on par with HuffYuv (FFDS version).

So, where Magic-YUV did shine in comparison to UT-Video was (in my tests at least) in the decode rate of 'Median' configured encodes (in 32-bit and 64-bit), where UT-Video was significantly slower, and 'Left' configured encodes in 32-bit. But in all other respects UT-Video was faster and compressed slightly better. And yes, it did appear that the 'Dynamic' mode has a bias to Median prediction, as overall the results were not that different. And that was how I understood your explanation. If it's more complex than that, all well and good.

Granted, these tests were performed with a pre-release version on one PC system (Hex-core AMD cpu, Win7 64-bit) using your benchmark tool, and others could well obtain a different results with this alpha release, on different PC's, with different source material and using different performance test criteria - as is already evident from the the several early reports.

All I am suggesting, really, is that to make a fair comparison, one should test with UT-Video in both Left (default) and Median modes in order to best judge how Magic YUV (in default Dynamic mode) performs on the balance of speed and compression.


As for the immediate future, I'll probably do color space conversion built-in to the encoder side (to allow direct RGB->YUY2/YV12 compression)


Although the vfw api is an antiquated/deprecated interface, the reality is that many NLE's still use vfw with third party codecs and operate (at the edit level) in RGB color-space. Given that one potential use of Magic-YUV would be as a lossless edit intermediate in such a system, I think the integrated RGB input would be worthwhile. As you know, when I tried Magic-YUV as an edit intermediate in the (consumer level) NLE I use (Corel Video Studio "Pro" x6) it crashed, but 'magically' worked when I installed the Helix YUV codecs pack, which presumably served as an RGB to YUV interface for Magic YUV - you were going to look into that. UT-Video works perfectly.

[Atak_Snajpera]

how does multithreading encoding work in magicyuv? does it work on frame level or does it use slices?

[Ignus2]

how does multithreading encoding work in magicyuv? does it work on frame level or does it use slices?

Slices. VFW is one frame in one frame out, so slices are the best option.

[Atak_Snajpera]

I wonder why MagicYUV and UT VIDEO do not use more than 2 cores in my tests? RAMDISK + uncompressed source and still no more than 50% during encoding.

[Ignus2]

I wonder why MagicYUV and UT VIDEO do not use more than 2 cores in my tests? RAMDISK + uncompressed source and still no more than 50% during encoding.

Maybe a color space conversion happening somewhere in the background on a single core?
What is the test setup/pipeline for the benchmark?

[Atak_Snajpera]

avs script has only rawsource("video.y4m")

i get 130 fps using avsmeter with this script.

virtualdub is set to fast recompress mode. according to process explorer virtualdub.exe does not use more than 2 cores ( 50% - Q6600@3GHz)

[kolak]

I wonder why MagicYUV and UT VIDEO do not use more than 2 cores in my tests? RAMDISK + uncompressed source and still no more than 50% during encoding.

How fast is your disk?
How fast is reading raw file?
These codecs should be using around 100% CPU, if not than there is some bottleneck.
The fact that both codecs are stucked at 50% means something.

[Atak_Snajpera]

ramdisk amigo! DDR2-666 mhz

[kolak]

Try avibench tool to check if you have same results.
Vdub is good but not always that transparent.

I was able to saturate 24 threads with Utvideo, which gave 1200fps and 2 GByte/sec reading. This was raid box.
Not sure why you have only 50% CPU usage.

[pandy]

Are You consider to support YCoCg color space to support reversible RGB color space conversion?

[Ignus2]

Are You consider to support YCoCg color space to support reversible RGB color space conversion?

I've read about that some time ago, though I don't really see the benefit.
Do you have some source that produces YCoCg?
Or do you consider RGB input and want to store it in YCoCg internally? In this case, what is the benefit (a bit more compression for RGB maybe)?
Also, to be reversible, it requires 2 extra bits.

Greets,
I.

[HulkHoganRules]

I'm really into video game capturing so I wanted to see how it performed. I honestly got lost in this thread but I have some questions if you don't mind.

If I supply this codec a YUY2 source, will it stay YUY2 or will it convert to YV12? The reason I ask is I captured the same frame in MagicYUV and then with Lagarith YUY2 and then YV12. I ran objective video quality tests (MSU) using the MagicYUV frame against both Lagarith YUY2 and then YV12. The comparisons were 99.?% similar. If MagicYuv is in YUY2, I think this is wrong right? I.e. it should be 100% similar since it's a lossless codec?

Many months ago I remember comparing Lagarith YUY2 to x264vfw Lossless mode and that gave me a perfect 100% identical reading. I guess I can only assume that MagicYUV does convert YUY2 to YV12 slightly differently to Lagarith's YV12?

One thing that did surprise me and is really awesome about this codec. Both Lagarith and UT Video completely crumbled under my 1080p60 Xbox 360 tests. I captured with MagicYuv perfectly fine at 1080p60! Picture: http://puu.sh/7jSui.jpg

Thank you.

[pandy]

I've read about that some time ago, though I don't really see the benefit.
Do you have some source that produces YCoCg?
Or do you consider RGB input and want to store it in YCoCg internally? In this case, what is the benefit (a bit more compression for RGB maybe)?
Also, to be reversible, it requires 2 extra bits.

Greets,
I.
Yes - RGB converted internally to YCoCg and back on decoding to RGB, all pros and cons but for example for screen capture especially on 4:4:4 it can be beneficial and provide fully reversible if necessary RGB back conversion. At minimal speed loss (YCoCg can be very fast - i assume perhaps faster than RGB to YCbCr) potential compressibility comparable to YCbCr (ie higher than on RGB).
I would say it can be nice as none of commonly used lossless codecs support this feature.
And as you consider 16 bit depth then perhaps 12 bit (for 10 bit RGB) is not a problem. Not sure as I've said previously - you can be first with such feature.

Big ThX and also greetz.

[raffriff42]

Harping again on this 10-bit thing, Ignus2... suppose you added 10-bit support by compressing the 8 most significant bits in the usual way, and the 2 least significant bits using a separate, simple method, eg run-length or zlib? Compression would not be "optimal," but you would have the only fast, lossless 10-bit encoder around.

[Ignus2]

Thanks for all the suggestions!

A bit of status report:
Currently I'm in the middle of refactoring the code, the result of which will allow much easier development in the future, also adding new colorspaces and internal color space conversions will be much easier. This should also pave the way for 10bit+ (but that's a plan only for after 1.0).

Sadly this also means that I had to change the compressed stream format, which is incompatible with the current one (of 0.9alpha), so the next release will not be able to decode videos encoded with 0.9alpha. This will absolutely not happen after 1.0 is released, I do this now, as I don't intend to keep backward compatibility with alpha releases.

Stay tuned :)

Greets,
I.

[Asmodian]

Very interesting stuff, looking forward to it. :)
:thanks:

[Gargamel]

+1
Thanks and congratulations, maybe (we hope so) the birth of a reference codec !

[Ignus2]

Thanks!

The refactoring is now done, some smaller things remain, so a release is to be expected soon (next week at the latest probably).
Performance-wise there isn't really a big difference, though the new version might be faster in some cases :)

The stream format had to be changed, as I mentioned, though I'm quite satisfied with it now, so if all goes well, it will stay for 1.0 and beyond.

The next release won't see any major new features, though I added an option to restrict the kind of colorspaces the codec will accept (useful to make sure you really give the codec the colorspace you work in and expect - so conversion don't silently happen somewhere). Built-in encoder-side colorspace conversions (direct encoding from RGB to YV12 for example) are expected after this release, and I hope to include them in 1.0.

More to come soon!

Greets,
I.

[Gargamel]

Many are waiting for... and salivating !
May the compiling Force be with you ;-)

[Ignus2]

After many sleepless nights and long hours of coding, the next release 0.9.1 beta is finally here.
Actually, those small things that remained after the refactoring were not so small after all :) But now I'm really satisfied with the result.
Feature-wise, there isn't anything spectacularly new, as I mentioned I only added an option to restrict the input color space.
Performance-wise I measured a 5-10% increase in encoding and 10-15% increase in decoding speed (depending on CPU type).
Also, some compatibility issues were fixed with Windows 8/8.1 64 bit.

As I mentioned I had to change the stream format, so videos encoded with 0.9alpha cannot be decoded with this release. I guess it has reached it's final form now, so I really hope it will stay as it is for 1.0 too.

The immediate plans for/before 1.0 are:

Selectable Rec.601/Rec.709 conversion parameters for YUV input (used when doing direct RGB output)
Interlaced input option
Color space conversion on the encoder side (direct compression from RGB to YUV)


Feature requests and bug reports are always welcome.

Greets,
I.

[Gargamel]

Thank you, Ignus2, and congratulations.
But take care of your health... at least till the v.1.0 !

[Ignus2]

Thank you, Ignus2, and congratulations.
But take care of your health... at least till the v.1.0 !

Thanks, I'll try!

Some progress report since the last release:
RGB can now be compressed as YUV444, giving much better compression ratio (at the cost of the RGB->YUV conversion loss). What follows is downsampling to YUV422 and YUV420 directly. After that, I'll do interlaced video support, and that should cover the features of 1.0.

My question is what YUV444 format should I support? AYUV, YV24, IYU2?

--
Greets,
I.

[Kurtnoise]

Hi,

Is it possible to have different FourCCs ? I mean, one FourCC for YUV streams and another one for RGB ?

Thanks in advance.

[EncodedMango]

My question is what YUV444 format should I support? AYUV, YV24, IYU2?
YV24 vote here.

[kolak]

Hi,

Is it possible to have different FourCCs ? I mean, one FourCC for YUV streams and another one for RGB ?

Thanks in advance.

+1 for this.

YV24 I think is most common.

[Kurtnoise]

This is not I asked...I meant the MAGY FourCC which is in the current streams.

I would like to have a FourCC dedicated to YUV streams (let's say MAGY) and another one for RGBs (for example, MAGR).

[Ignus2]

OK, YV24 will be first then, AYUV later.

@Kurtnoise:
What would be the benefit of having a separate FOURCC for RGB and YUV streams?

[kolak]

This is not I asked...I meant the MAGY FourCC which is in the current streams.

I would like to have a FourCC dedicated to YUV streams (let's say MAGY) and another one for RGBs (for example, MAGR).

This is what I ment.

Because it's cleaner and RGB is 'different' than all YUV.
Utvideo has different fourcc for each format.

[Ignus2]

This is what I ment.

Because it's cleaner and RGB is 'different' than all YUV.
Utvideo has different fourcc for each format.

Apart from being "cleaner", any practical reasons? :)

[Kurtnoise]

to have an easy way to distinguish each streams when you parse them to retrieve some infos...:)

[Ignus2]

Hi!

I'll ponder on the MAGR fourcc.

In the meantime, a new release is out: 0.9.2 beta (https://magicyuv.com/index.php/download/magicyuv/magicyuv-0-9-2beta).
New features include:

Support for new formats: YV24, Y8/Y800/GREY, AYUV
Support for selectable YUV color matrices: Rec.601 and Rec.709. This option is used when compressing RGB --> YUV. It is also saved in the encoded stream and used when decompressing YUV --> RGB.
Added encoder-side color space conversion. Only converting to a "lower" color space is allowed. Color spaces are ordered:
RGB > YUV 4:4:4 > YUV 4:0:0 for regular formats
RGBA > YUV 4:4:4 with alpha for alpha formats.


If someone could test the alpha formats that would be nice, I couldn't really do it so far :)

The encoder side conversion list is quite limited for now but all options (including YUV 4:2:2 and 4:2:0) will come soon with the next release.
I'll do these, and support for interlaced encoding, and that should hopefully finally cover 1.0 :)

--
Greets,
I.

[kolak]

To make it even more 'full' you can add Full range and Limited range settings.
There are cases when RGB data has YUV range (or opposite), so it's good to have such an option and being able to set it manually when needed.

[benwaggoner]

To make it even more 'full' you can add Full range and Limited range settings.
There are cases when RGB data has YUV range (or opposite), so it's good to have such an option and being able to set it manually when needed.
And having it flagged in the metadata so that downstream processing knows what it's getting!

[Ignus2]

To make it even more 'full' you can add Full range and Limited range settings.
There are cases when RGB data has YUV range (or opposite), so it's good to have such an option and being able to set it manually when needed.

I see. There are some questions here.
What settings would you like to have exactly (what controls with what values and how many)? Because we have to consider 24 cases in total here. Also, there must be some global setting for decoding, as a Limited Range YUV for example could be decompressed as both Full Range RGB and Limited Range RGB (not to mention also as limited YUV and full YUV), and it wouldn't be the best to store the "how to decode this video if they request RGB (or YUV)" setting in the compressed stream. Using a decoder setting, both could be done. But decoder options are global.

I'm asking, as I don't know exactly what controls to provide so it doesn't become overly confusing.

The 24 cases to consider:

When encoding (12):
Input format -> Compressed Format
Full RGB -> Limited YUV
Full RGB -> Full YUV
Full RGB -> Limited RGB
Full RGB -> Full RGB
Limited RGB -> Limited YUV
Limited RGB -> Full YUV
Limited RGB -> Limited RGB
Limited RGB -> Full RGB
Full YUV -> Limited YUV
Full YUV -> Full YUV
Limited YUV -> Limited YUV
Limited YUV -> Full YUV

When decoding (12):
Compressed format -> Output Format
Full YUV -> Limited YUV
Full YUV -> Full YUV
Full YUV -> Limited RGB
Full YUV -> Full RGB
Limited YUV -> Limited YUV
Limited YUV -> Full YUV
Limited YUV -> Limited RGB
Limited YUV -> Full RGB
Full RGB -> Limited RGB
Full RGB -> Full RGB
Limited RGB -> Limited RGB
Limited RGB -> Full RGB


I would be interested which ones of the above should be supported and how should the user be able to specify what he wants?

EDIT: corrected. I also got confused %)

Greets,
I.

[kolak]

RGB always should be full range, untill you manually set in encoder that its limited range. YUV opposite. Settings on encoder are for manual overwrite.
On decoding stage decoder reads settings used during encoding. If someone wants RGB for YUV encoded file than you do proper conversion, taking into account infomation stored during encoding, eg that file was YUV, but full range.
Same applies to HD and SD conversion matrixes (rec709/rec601). They should be set based on resolution with possibility to overwrite manually and this info should be passed to decoder also, so if RGB<->YUV conversion is needed than its done properly.

On the encoder side you need just 4 settings:
- force limited range
- force full range
- force REC709
- force REC601

If you want you can have the same for decoder, but automatically it uses info from encoding stage.
You dont need all those cases to expose to user, this is just for you to programme ☺
These 4 options on encoding and maybe decoding are enough in my opinion- they will generate all cases which you mentioned.

[Ignus2]

RGB always should be full range, untill you manually set in encoder that its limited range. YUV opposite. Settings on encoder are for manual overwrite.
On decoding stage decoder reads setting used during encoding. If someone wants RGB for YUV encoded file than you do proper conversion, taking into account infomation stored during encodeing, eg that file was YUV, but full range.
Same applies to HD and SD conversion matrixes (rec709/rec601). They should be set based on resolution with possibility to overwrite manually and this info should be passed to decoder also, so if RGB<->YUV conversion is needed than its done properly.

So there should be 2 settings on the encoder side? One for "RGB range" and one for "YUV range"? I summarized the possibilities, but the ones marked with "???" are not clear to me:

[Input] [Compressed as] [RGB setting] [YUV setting] [Input treated as] [Decoded as RGB] [Decoded as YUV]
RGB RGB FULL FULL FULL FULL N/A
RGB RGB FULL LIMITED FULL FULL N/A
RGB RGB LIMITED FULL LIMITED FULL N/A
RGB RGB LIMITED LIMITED LIMITED FULL N/A

RGB YUV FULL FULL FULL FULL ???
RGB YUV FULL LIMITED FULL FULL ???
RGB YUV LIMITED FULL LIMITED LIMITED ???
RGB YUV LIMITED LIMITED LIMITED LIMITED ???

YUV YUV FULL FULL FULL ??? FULL
YUV YUV FULL LIMITED LIMITED ??? LIMITED
YUV YUV LIMITED FULL FULL ??? FULL
YUV YUV LIMITED LIMITED LIMITED ??? LIMITED

("N/A" means that path is not possible)

EDIT: Using the setting you mentioned (apart from Rec601/709):

[Input] [Compressed as] [Force range setting] [Input treated as] [Decoded as RGB] [Decoded as YUV]
RGB RGB NONE FULL FULL N/A
RGB RGB FULL FULL FULL N/A
RGB RGB LIMITED LIMITED FULL N/A

RGB YUV NONE FULL FULL ???
RGB YUV FULL FULL FULL ???
RGB YUV LIMITED LIMITED LIMITED ???

YUV YUV NONE LIMITED ??? LIMITED
YUV YUV FULL FULL ??? FULL
YUV YUV LIMITED LIMITED ??? LIMITED


Greets,
I.

[kolak]

Its a good question: should RGB always be delivered on decoding stage as full range? In one way this is the 'correct' way and many software expect RGB being always full range.

YUV should be always limited, but does it mean we will always clip super whites/blacks?

Iam confused now also ☺
Maybe limited/full range should be only on encoding stage and on the decoder side YUV should be always converted to limited and RGB full range. Having info from encoding stage and using correct matrix would always produce correct output data. The downside is that YUV mode would never pass super whites/blacks.
We need more opinions ☺

[Ignus2]

Great, so I'm not the only one who got a bit confused :)
What is your experience BTW? I am totally inexperienced with this stuff, I don't really know workflows and situations about what range is used and when, and what is the most common need of users.

EDIT: And another question? Is it common, that users bring RGB source, compress it as YUV and need it back as RGB again? Or bring YUV source, compress, and then want it back as RGB?
For me, RGB->YUV conversion on the encoder side is the most useful, as it allows better compression ratio but I don't know about others...

Greets,
I.

[foxyshadis]

RGB restricted to 16-235 should be considered broken and faulty, not treated as a first-class option. The only time it's ever produced is by mistake, and no systems expect or correct for it (aside from generic autolevels filters). I'd advise not to include or even acknowledge it.

On the other hand, YUV video is almost always TV range, whereas YUV still pictures are almost always full range, so either is acceptable as long as it's signaled. Some MJPEG recordings use full range, for instance, as do some advanced users.

Some people do round-trip back to RGB to do compositing and other effects, since many NLEs only work in RGB; hard drive pressure means that greater compression is sometimes more important than accumulated roundoff errors. Once converted, don't pretend the video is RGB to decoders, just offer up the YUV as-is properly signaled and let the client convert back how they want, unless the client can't understand YUV at all.

I would add an option to dither to YUV, because of the banding problems straight conversions always have and to minimize round-trip losses. Optional mainly because dithering is slower.

[kolak]

Makes sense.
So, maybe just full range for RGB and both for YUV is the way to go.

If you have RGB source, but you want compress it as YUV it's fine as long as your output is 100% the same as source.

Dithering is also good idea and very important for 8bit+ formats.

Most people deal with YUV, where RGB (444) is for high end work. Once you convert RGB to YUV you hardly ever want to go back to RGB.
I work for a company where all work is done with DPXs and than this is used to create final YUV 'delivery' master for broadcadt, web etc. If anything needs to be change it's done on DPX and converted to YUV again.
Broadcast stays away from RGB. 3D, VFX, grading, keying rather stays away from YUV ☺
For this reason good RGB <-> YUV conversion is still needed.
The problem is that so often it's done badly or even wrong.

[Ignus2]

RGB restricted to 16-235 should be considered broken and faulty, not treated as a first-class option. The only time it's ever produced is by mistake, and no systems expect or correct for it (aside from generic autolevels filters). I'd advise not to include or even acknowledge it.

On the other hand, YUV video is almost always TV range, whereas YUV still pictures are almost always full range, so either is acceptable as long as it's signaled. Some MJPEG recordings use full range, for instance, as do some advanced users.

Some people do round-trip back to RGB to do compositing and other effects, since many NLEs only work in RGB; hard drive pressure means that greater compression is sometimes more important than accumulated roundoff errors. Once converted, don't pretend the video is RGB to decoders, just offer up the YUV as-is properly signaled and let the client convert back how they want, unless the client can't understand YUV at all.

I would add an option to dither to YUV, because of the banding problems straight conversions always have and to minimize round-trip losses. Optional mainly because dithering is slower.

Great! Thanks for the explanation!

About the signaling: how do you signal when decoding YUV, that it is full range? Is it even needed? If the codec gets YUV input, it just compresses bytes, it doesn't really care if it's full range or not. Up until RGB is explicitly requested when decoding, in that case of course it has to know if the original input was full or limited (the same is true for the conversion matrix).

My question is, that when compressing RGB as YUV and then decoding as YUV, what to output: full or limited? Currently it is limited, and that is what I would expect in most cases, I'm just curious if it's a valid use-case that someone would want full range YUV decoded from an originally RGB source, which was converted to YUV by the codec during compression?

Greets,
I.

[kolak]

I would keep full range. YUV full range is 'less broken' than limited RGB in my opinion. Some people like if YUV preserve super whites/blacks.

[jmartinr]

I would keep full range. YUV full range is 'less broken' than limited RGB in my opinion. Some people like if YUV preserve super whites/blacks.

That's not a good reason. For super whites/blacks you need TV range. Also, if the compression source is rgb, would you get super whites/blacks?

[kolak]

What I want is to preserve data outside 16-235 range in YUV signal. Some solutions always clip incoming data, other pass it even if format is YUV based. As you said in case of RGB source you won't get it ( at least by my logic).
In this case the only time when you get full range YUV is when source is full range YUV.
It means for any RGB source encoded as YUV we should get limited range on output.

[kolak]

[Input] [Compressed as] [Force range setting] [Input treated as] [Decoded as RGB] [Decoded as YUV]
RGB RGB NONE FULL FULL N/A
RGB RGB FULL FULL FULL N/A
RGB RGB LIMITED LIMITED FULL N/A

RGB YUV NONE FULL FULL LIMITED
RGB YUV FULL FULL FULL LIMITED
RGB YUV LIMITED LIMITED FULL LIMITED

YUV YUV NONE LIMITED FULL LIMITED
YUV YUV FULL FULL FULL FULL
YUV YUV LIMITED LIMITED FULL LIMITED



Is this what everyone is happy with?
Everything got simplified a lot:
RGB output is always full
YUV is always limited except time when YUV source is already full range.
This also means we don't need settings for decoder stage, just on encoder side.

[Ignus2]

Is this what everyone is happy with?
Everything got simplified a lot:
RGB output is always full
YUV is always limited except time when YUV source is already full range.
This also means we don't need settings for decoder stage, just on encoder side.

Great, this is what I also had in mind finally :)

[kolak]

In the meantime UtVideo got 422 10bit support.

[Ignus2]

In the meantime UtVideo got 422 10bit support.

Great work by the developer! I hope I can do it better ;)

[kolak]

It's quite slow and single thread at the moment I think.

[WorBry]

In the meantime UtVideo got 422 10bit support.

Good news ! I thought they had reached an impasse on that.

Still, looks like MagicYUV is coming on nicely :)

[foxyshadis]

About the signaling: how do you signal when decoding YUV, that it is full range? Is it even needed? If the codec gets YUV input, it just compresses bytes, it doesn't really care if it's full range or not. Up until RGB is explicitly requested when decoding, in that case of course it has to know if the original input was full or limited (the same is true for the conversion matrix).

VFW doesn't have any built-in support for that, although some conventions might exist. DXVA, Directshow, and Media Foundation rely on DXVA2_ExtendedFormat, VIDEOINFOHEADER2, or Media attribute GUIDs (http://msdn.microsoft.com/en-us/library/windows/desktop/bb970322%28v=vs.85%29.aspx). The actual values are the same across everything (MF has one extra range restriction for xRGB), and should work with all video renderers and anything else that reads them. Best of all, they have well-defined meanings for 10-bit as well as 8-bit.

Avisynth is pretty much stuck in the VFW world, but other source plugins commonly embed the headers in the LSB of the last few pixels, which can be extracted and interesting things done. (Adding extra lines or an extra frame might make more sense for a lossless plugin.) This is already done for interlace & IVTC decisions mostly. Embedding DXVA2_ExtendedFormat makes sense for those purposes and is only 224 bits long, but how to determine when you would need to isn't simple. There isn't much a standard for how to signal or embed this in Avisynth or any VFW, sadly.

If there is no signal, you just have to pick a default, and 16-235 is the default across the entire video world.

[kolak]

About YUV full range- if we can have processing which does pass it "as it", instead of clipping to 16-235 than we may not need setting for YUV Full Range. Can this be done, so it works fine, even if than we may need RGB on output? Won't it mes up black levels, etc?

[pandy]

Perhaps there is no sense to bother with quantization range (unless it help with compression - not sure what impact is for efficiency between 0 - 255 and 16 - 235).
IMHO lossless codec can be transparent (and quantization range problem can be easily fixed in render/further processing).

[Ignus2]

I'm glad to announce, that 1.0rc1 is finally out :)
New features (among various bug fixes) are:

Encoder-side conversion to YUV 4:2:2 and 4:2:0 formats
Interlaced encoding
Full Range YUV support when converting to/from RGB

This completes the planned features for 1.0 and if no major bugs arise, this will become the final 1.0 version.

Grab it from the Download page (https://magicyuv.com/index.php/download).

Greets,
I.

[kolak]

What is your rule to use REC601?

Width<=720 or you need to set manually?

[Ignus2]

What is your rule to use REC601?

Width<=720 or you need to set manually?

Currently manual. Is automatic needed?

[kolak]

I would say so.
Automatic by default with possibility to overwrite.
I think the rule is Width<=720 use 601, otherwise 709.

[Ignus2]

Hi!

I have a problem, I would like to get a bit of help from someone who is familiar with the workings of DirectShow.

The problem I have:
Using GraphStudioNext, I noticed, that when decoding certain MagicYUV compressed files, a Color Space Converter gets inserted sometimes. The problem is, that not always, and it seems it is resolution dependent.
What makes it really strange, is that for example an RGB compressed stream with 1920x1080 resolution is fine, the codec decompresses as RGB32 and it goes straight to the Video Renderer. But if I take a 1916x1076 resolution file, decode as RGB32, I get a CSC. And the most interesting part is, that looking at either end of the CSC pins, they have EXACTLY the same type. Everything matches, resolution, RGB32, frame size, etc. So it seems like it's there to do nothing, but I still get it.
This happens for other resolutions as well.

I have no idea sadly what might be causing this, so if someone could help me out, I would be really grateful.

Thanks!

I.

[Reino]

I think the rule is Width<=720 use 601, otherwise 709.Practise shows us it's more like ≤576p: BT.601, >576p: BT.709.

[foxyshadis]

Hi!

I have a problem, I would like to get a bit of help from someone who is familiar with the workings of DirectShow.

The problem I have:
Using GraphStudioNext, I noticed, that when decoding certain MagicYUV compressed files, a Color Space Converter gets inserted sometimes. The problem is, that not always, and it seems it is resolution dependent.
What makes it really strange, is that for example an RGB compressed stream with 1920x1080 resolution is fine, the codec decompresses as RGB32 and it goes straight to the Video Renderer. But if I take a 1916x1076 resolution file, decode as RGB32, I get a CSC. And the most interesting part is, that looking at either end of the CSC pins, they have EXACTLY the same type. Everything matches, resolution, RGB32, frame size, etc. So it seems like it's there to do nothing, but I still get it.
This happens for other resolutions as well.

I have no idea sadly what might be causing this, so if someone could help me out, I would be really grateful.

Thanks!

I.

You're decoding to an unaligned stride and not responding to a stride-change request via QueryAccept. Here's MSDN's docs. (http://msdn.microsoft.com/en-us/library/dd388901%28VS.85%29.aspx) This doesn't show up in GraphStudioNext because that biWidth is only passed during the graph creation; only the real biWidth is seen after that. There may not be anything you can do about this from the AVI decompressor, I don't know VFW well.

[Ignus2]

You're decoding to an unaligned stride and not responding to a stride-change request via QueryAccept. Here's MSDN's docs. (http://msdn.microsoft.com/en-us/library/dd388901%28VS.85%29.aspx) This doesn't show up in GraphStudioNext because that biWidth is only passed during the graph creation; only the real biWidth is seen after that. There may not be anything you can do about this from the AVI decompressor, I don't know VFW well.

That was my suspicion after searching for a while, in the meantime I also found that article.
Sadly, the AVI Decompressor doesn't seem to pass down the stride info, it always passes biWidth as the real width.
Maybe it has compatibility reasons, probably VFW was never meant deal with aligned strides :/

Thanks for the help!

Greets,
I.

[foxyshadis]

Do you plan on releasing a DirectShow or Media Foundation codec? You should be able to put all the interfaces within the same dll, too, and it would be handy for some scenarios, like DShow video capture.

[Ignus2]

Do you plan on releasing a DirectShow or Media Foundation codec? You should be able to put all the interfaces within the same dll, too, and it would be handy for some scenarios, like DShow video capture.

For the near future, no. VFW is like a common base for a lot of applications. Very old, very inflexible, but for an intraframe lossless codec, it is sufficient. It would be nice to have native DShow filter, but it is not among the plans for the foreseeable future.

BTW, can't DShow capture use AVI Compressor filter for compression the same way as AVI Decompressor is used for decompression?

Greets,
I.

[zerowalker]

Does VFW in itself do any limitations to Codecs, i mean performance wise etc?
Cause i always read that it's ancient and no good, but as it's always used i can't imagine there to be anything directly wrong with it, except that perhaps it's hard to adapt to stuff, but also hard to replace as it's so common (Like DirectX in the Graphic world).

A bit off topic, but as it was mentioned, also i mean to direct this towards Lossless rather then lossy which often has it's own design.

[Ignus2]

Does VFW in itself do any limitations to Codecs, i mean performance wise etc?
Cause i always read that it's ancient and no good, but as it's always used i can't imagine there to be anything directly wrong with it, except that perhaps it's hard to adapt to stuff, but also hard to replace as it's so common (Like DirectX in the Graphic world).

A bit off topic, but as it was mentioned, also i mean to direct this towards Lossless rather then lossy which often has it's own design.

Apart from the performance problem above, like not being able to efficiently decode to specially allocated buffers, the biggest limitation of VFW is the one frame in one frame out mechanism. It is not possible to do pipelinig with that, which is not a concern for CPU processing, but could be very handy for GPU encoding.

It is easy to develop for VFW and it is widely supported, but basically those are it's only advantages.

Greets,
I.

[zerowalker]

Ah so it has performance issues, not sure how it works, but all in all that doesn't bode to well, especially as VFW is currently as common as anything can be, like mp3 for Music i guess.

Don't even know applications that use other interfaces, though i primary use Virtualdub, and then again i only know how to use VFW.

Thanks for telling, hopefully you can add other interface support sometime you think it matters, as the codec itself is probably the main focus as that increases performance/usefulness overall.


Edit:

Oh also, Lagarith is another lossless codec which you probably know of, and it has some extremely well compression when it comes to Pixelated videos, like old games, where the resolution is low, and even if you increase it, as long as it's "Nearest Neighbor" it still compresses very well.

Can MagicYUV do this as well, i doubt it can as it's probably to do with the bitstream or something, but if possible it would be great, else Lagarith itself fit's into that of course.

[zerowalker]

Noticed talk about Rec.709 etc and how some want it to go with the width < 720 = Rec.601, so would like to put a suggestion to this.

UT Video Codec has Rec.601/709 separate (it also has it's formats separate, which may be handy), and i find that pretty appropriate, as for example, there is no actual rule that says when it's Rec.601 or 709,
you normally just say, "If it's Not HD, than it must be Rec.601", this is in most cases true, but not always.

In the TV world, i think that term is always true (may be wrong), but in the PC world it's not,
if you record something on the PC (not talking capture cards here), it will always be Rec.709 as far as i know, doesn't matter the resolution.

While separate Devices, Webcams, Capture cards etc probably follow the rule more or less.

That's why it may be handy to have Rec.709 and Rec.601 separate, if you record on the PC (Gaming etc) you always want Rec.709, and it would be quite messy to always change the settings if you also use Virtualdub,
to capture Webcam or other stuff that may use Rec.601.

Well, think i went overkill with the wall of text, but hope it makes some sense, at least it does to me;P

[STaRGaZeR]

PC is RGB, not YUV, so neither 709 nor 601.

[zerowalker]

Oh, skipped some lines there.

Yes PC is RGB, i meant that when you encode it to YUV Rec.709 produces the correct color.

However, i tried MagicYUV after this, and both 601/709 look identical, so this just get's me confused.
Pretty sure that all other codecs, at least Lagarith and UT Video produces wrong colors if they are Rec.601 (Lagarith is forced at that).

[kolak]

It makes sense, but if we have rules, possibility to ovewrite and info is stored in the headers for the decoding stage than there is no need for a separate fourCC.
601/709 looks very similar and on some frames you can't really tell the difference, untill you flip between them.

[STaRGaZeR]

Yes PC is RGB, i meant that when you encode it to YUV Rec.709 produces the correct color.

It's not like you're implying. If you have RGB you have RGB, there is no "correct" way to convert it to YUV. You're probably referring to the fact that Fraps (a specific codec) converts that RGB to YUV using always 709 when encoding in YUV mode, regardless of resolution. This is true, but it's a Fraps thing, nothing general.

[zerowalker]

Hmm, i really don't get it anymore;S

I mean, sorry for a bit off topic, but, let's say i have something that's RGB, anything at all.

If both Rec.601/701 is "correct" when downsampling, why does different versions even exist?

I thought that Rec.601 is and old version, and Rec.709 is an updated, meaning it contains, more colors or information etc, and then Rec.2020(or what the name was) is the latest and greatest?
But if all achieve the same thing, and the information is only there to tell it (Decode it with this and it will look correct), it doesn't make sense to have more than one?

Am i missing something here, or can you just skip and use one of them for everything?

[ChiDragon]

Rec.601 and Rec.709 describe slightly different "triangles" in the whole space of all possible colors, and Rec.2020 describes a vastly expanded one (this page (http://www.soundandvision.com/content/podcast-45-david-reisner) has a diagram representing the gamut of 601 vs 709 -- elsewhere you can find ones comparing 709 to 2020). To see a difference between an RGB image correctly matrixed to/from 601 vs 709, I believe you would need to pick one with colors that are encompassed by only one of those two triangles and a display capable of rendering them correctly.

There is a lot of debate about the reasoning behind Rec.709 and whether the benefits outweigh the issues, but 2020 has the clear goal of being able to represent a larger color gamut.

[kolak]

601 was chosen with CRT technology in mind, where 709 for Plasmas/LCDs. Small differences are due to different nature of display technologies.
Rec2020 is very new and currently there is not a single publicly available display which can cover its whole spectrum.

[zerowalker]

But if you take an image from your PC, then convert it to YV12 or something.
The Recxxx doesn't even matter, the result will be the same as long as you decode it to RGB with the correct flag.

This is what i don't get, they look identical.

If the only way to see the differences is playing around with color triangles etc, then that doesn't make any sense, at least when it comes to simple RGB-> YUV.

EDIT:

Okay actually looked more carefully, there is a difference that is viewable, so forget the "Identical" part.
Can't say which one looks best though, both get gradients of different kinds and handle some parts differently,

But, i am guessing Rec.709 should be overall better as it's an improved Rec.601?

The point of them is to increase the colors right, meaning it get's closer to RGB?

[STaRGaZeR]

If you convert RGB to YUV with a specific rec and then back to RGB with the same rec you shouldn't see any significant difference between both RGBs. At least I am yet to meet someone that can tell which matrix was used in a conversion, or that there was a conversion in the first place, when faced with both the original RGB and the properly reconstructed RGB after the RGB->YUV->RGB conversion. If you see significant differences, then you're probably doing something wrong. Use 4:4:4 for comparisons obviously.

[zerowalker]

Yeah did that just now, and as said, the only thing that's really different, are dark areas, where gradients show, these parts will alter.
I can't however tell which is right or which is wrong, but it differs, but in bright normal colors, the difference is pretty much non-existent to what i can see.

However, as discussed before, though with Lagarith if i remember correctly, it would be interesting to see support for YCoCg, it's an interesting format, however it's never really used.
But if my memory serves correctly, it's like YUV but losslessly reversible to RGB, or something along the line?

[kolak]

But, i am guessing Rec.709 should be overall better as it's an improved Rec.601?
The point of them is to increase the colors right, meaning it get's closer to RGB?

If you watch it on LCD/Plasma than in theory yes. If you want to watch it on old CRT TV than in theory 601 is better for it. In practice difference is very small and covered by compression :)

[zerowalker]

Ah, don't get it fully, but think i at least follow the idea;P

How is Rec.2020 then, some kind of killer or pretty much the same, And will support be added for it in MagicYUV?
I know it doesn't exist a real interest except experimentation and enthusiasm, but still it never hurts to be on the edge of things, especially since it aims to be the "4K Lossless era Codec":)

[Ignus2]

Ah, don't get it fully, but think i at least follow the idea;P

How is Rec.2020 then, some kind of killer or pretty much the same, And will support be added for it in MagicYUV?
I know it doesn't exist a real interest except experimentation and enthusiasm, but still it never hurts to be on the edge of things, especially since it aims to be the "4K Lossless era Codec":)

Rec.2020 is 10-12 bit AFAIK. Though the Rec.2020 RGB coefficients could be used to convert RGB 8 bit to YUV 8 bit, but I don't know if it would have any meaning.

Greets,
I.

[zerowalker]

Oh, well that really turns of the heat. If it's meant for that i guess using it for 8bit will just cause rounding errors or something, would be quite interesting to see though.

[zerowalker]

There seem to be a bug i have encountered.

When i have YV12 content i save it with MagicYUV, it will be saved as YV24 or YUY2, or at least decoded that way in Avisynth,
Avisource = YV24, DirectShowSource = YUY2.

My settings are - As Is, so no internal conversion should be done, input-output should be the same.

Is it supposed to be like that, am i missing out something, or is this a bug?

[Ignus2]

There seem to be a bug i have encountere.

When i have YV12 content i save it with MagicYUV, it will be saved as YV24 or YUY2, or at least decoded that way in Avisynth,
Avisource = YV24, DirectShowSource = YUY2.

My settings are - As Is, so no internal conversion should be done, input-output should be the same.

Is it supposed to be like that, am i missing out something, or is this a bug?

Upscaling during compression (and downscaling during decompression) is not supported, so if the source is really YV12, it is definitely encoded like that.
However, YV12 can be decoded as both YUY2 and YV24 if explicitly requested. By default the codec will report YV12 though.
To make sure you really encoded YV12, set Input Colorspace (restrict) to YV12.
If you really encoded YV12 but get something else, that means, that something converts or requests upscaled formats explicitly.
Try the pixel_type option of AviSource and see if it helps. (Ntote: downscaling will not happen, as downscaling during decompression is not supported by the codec.)

Greets,
I.

[zerowalker]

Well that's good to hear.
I used Virtualdub Fast Recompression and made sure that it does not do any conversion, meaning MagicYUV get's feeded YV12 in this case.

I know you can restrict, but would prefer not to, as it should work anyway, but of course a safety mechanism isn't wrong.

Actually did that, and it seems to work, pretty sure i tried that before on another clip, and it reported something like "Decoder can't give that format" or something, but this time it worked.

Problem is though, if i want to feed this clip to, let's say x264, it would normally get decoded to YV24/YUY2 and then downsampled to YV12 again, unless i force the Pixel_type.

How come it doesn't decode it correct automatically?

Also, in Mediainfo there is no way to find out what pixel type it is, which is quite troublesome, as i can't figure out afterwards what the true format is, unless i know it of course.

[kolak]

I assume it will be added at some point. It makes perfect sense, specially for "4K codec". New TVs started supporting Rec2020.

http://www.theasc.com/asc_blog/thefilmbook/files/2013/10/gamut_rec_2020_dci.jpg

It shows way more colors than Rec709 (or DCI, which is used in cinemas) assuming your TV is capable of displaying all of them.

[Ignus2]

I assume it will be added at some point. It makes perfect sense, specially for "4K codec". New TVs started supporting Rec2020.

It shows way more colors than Rec709 (or DCI, which is used in cinemas) assuming your TV is capable of displaying all of them.

Rec.2020 is relevant only when doing conversion to/from RGB. Rec.601 and Rec.709 (as well as sRGB) use the same color primaries (well, almost), while Rec.2020 does not, allowing a larger color gamut. Also, Rec.2020 is for 10-12 bit.
So the question is whether it makes sense to support conversion from 8 bit RGB to YUV (Y'CbCr) using Rec.2020 coefficients, which would imply that the RGB input uses different primaries (than the usual sRGB).

EDIT: Adding the coefficients is easy, so I can add an option next to the current Rec.601 and Rec.709, but the question is whether it makes sense.

Greets,
I.

[Rumbah]

Rec. 2020 uses more than 8 bits because you'll be in danger of getting banding with 8 bits as the triangle is that large.

[kolak]

Rec.2020 is relevant only when doing conversion to/from RGB. Rec.601 and Rec.709 (as well as sRGB) use the same color primaries (well, almost), while Rec.2020 does not, allowing a larger color gamut. Also, Rec.2020 is for 10-12 bit.
So the question is whether it makes sense to support conversion from 8 bit RGB to YUV (Y'CbCr) using Rec.2020 coefficients, which would imply that the RGB input uses different primaries (than the usual sRGB).

EDIT: Adding the coefficients is easy, so I can add an option next to the current Rec.601 and Rec.709, but the question is whether it makes sense.

Greets,
I.

Nothing stop most of the panels being 10bit in a year time. It's visible trend that more and more panels support 10bit, as technology is getting cheaper and more advanced.
MagicYUV may not be only 8bit in the future neither :)
Support may show up when it makes sense, no need to be there now.

[Ignus2]

There seem to be a bug i have encountered.

When i have YV12 content i save it with MagicYUV, it will be saved as YV24 or YUY2, or at least decoded that way in Avisynth,
Avisource = YV24, DirectShowSource = YUY2.

My settings are - As Is, so no internal conversion should be done, input-output should be the same.

Is it supposed to be like that, am i missing out something, or is this a bug?

I have found the reason for this BTW. Actually it is the "fault" of AviSynth. In the AviSource docs (http://avisynth.nl/index.php/AviSource) it is stated:

"
The pixel_type parameter allows you to choose the output format of the decompressor. Valid values are "YV24", "YV16", "YV12", "YV411", "YUY2", "RGB32", "RGB24", "Y8", "AUTO" and "FULL" (default value). If omitted or set to "FULL", AviSynth will use the first format supported by the decompressor (in the following order: YV24, YV16, YV12, YV411, YUY2, RGB32, RGB24 and Y8). If set to "AUTO", AviSynth will use the old ordering: YV12, YUY2, RGB32, RGB24 and Y8. This parameter has no effect if the video is in an uncompressed format, because no decompressor will be used in that case. To put it in different words: if you don't specify something it will try to output the AVI as YV24, if that isn't possible it tries YV16 and if that isn't possible it tries YV12, etc ...
"

So AviSource doesn't ask the codec about the preferred format but always tries YV24 first. This is why you get YV24 even if the compressed form is YV12.
In my opinion, this is quite incorrect to do, but that's how it's done.

Greets,
I.

[zerowalker]

Hmm, well that really is quite problematic.
As i assume many are as me and just use Avisource and think that it will be done correctly.
I find it weird however that UT Video Codec and others are decoded correctly, but i guess that don't allow upsampling?

If that's the case, there is 2 solutions i can think of:

1: Fix Avisource (Probably most problematic as Avisynth is in the state it is).
2: Add option to prevent Upsampling on Decoding, pretty sure Lagarith has this, but i think it also introduces problems, like being unable to playback videos as they must be RGB32 for that.

hmm, do you have any ideas if it can be solved from the Codecs and like that?
As i find it quite bad (not the codec but the problem), as if i hadn't noticed it, i would have done another Downsampling losing even more Chroma for no reason.

Thanks!

[Ignus2]

Hmm, well that really is quite problematic.
As i assume many are as me and just use Avisource and think that it will be done correctly.
I find it weird however that UT Video Codec and others are decoded correctly, but i guess that don't allow upsampling?

If that's the case, there is 2 solutions i can think of:

1: Fix Avisource (Probably most problematic as Avisynth is in the state it is).
2: Add option to prevent Upsampling on Decoding, pretty sure Lagarith has this, but i think it also introduces problems, like being unable to playback videos as they must be RGB32 for that.

hmm, do you have any ideas if it can be solved from the Codecs and like that?
As i find it quite bad (not the codec but the problem), as if i hadn't noticed it, i would have done another Downsampling losing even more Chroma for no reason.

Thanks!

The reason UT "works" is because it cannot do YV24 output at all.
A solution could be to provide options on the GUI like these:
Prevent upsampling on decoding:
- Prevent YV12 -> YV24
- Prevent YV12 -> YUY2
- Prevent YUY2 -> YV24
by default all checked.

That would still allow decoding to RGB, and this way AviSource would also work correctly.

Thanks for reporting. BTW, what you encountered (getting YV24 out of YV12) is exactly the case of "this is not a bug, it's a feature" :) And now I have to workaround it...

Greets,
I.

[zerowalker]

Ah, okay, i guess it goes to YUY2, but then again it has separate codecs so it somewhat prevents "misuse", which has it's own flaws.

Hmm didn't think of that, forgot that RGB was of low priority even though you wrote it.
That indeed would solve any upsampling and playback and everything will work as expected, and making it a choice prevents cases where it may interfere.

No problem, one of those rare cases, glad it could be solved, and very thankful for you taking time investigating it, having a developer listening to it's users is worth a lot:)

Thanks!

[zerowalker]

Another thing i noticed, my Webcam can output RGB and I420, and I420 is Not supported by MagicYUV.
Now i am guessing it's because it's Interleaved and not Planar.

But even though it's rare, i think it would be handy to support that input, even if you would need to convert it to Planar (As far as i know it's lossless).

[Ignus2]

Another thing i noticed, my Webcam can output RGB and I420, and I420 is Not supported by MagicYUV.
Now i am guessing it's because it's Interleaved and not Planar.

But even though it's rare, i think it would be handy to support that input, even if you would need to convert it to Planar (As far as i know it's lossless).

Just did a quick lookup and I420 is identical to YV12, just the U/V planes are reversed. It is trivial to add support for that. Thanks for mentioning, I'll include it for the next release (rc2) along with the prevent settings, which should hopefully be released in the next days.

Greets,
I.

[zerowalker]

Splendid, grateful as always:)

[zerowalker]

Another thing i have wanted to ask, is it possible to add Null Frames in MagicYUV?
From my understanding, i guess it's not quite optimal as it uses prediction, and Lagarith(it has null frame, hence the reference) uses "Zip" on each frames if i remember correctly.

However thought it was worth asking about it, it's quite a neat thing in some rare cases.

EDIT:

Noticed something that may be important.

I noticed that MagicYUV can produce 2 identical files, while the difference in size is quite huge, like 50% more or less while still having the same settings.

I did like this.

I used Dxtory to record with MagicYUV, the settings are Default, RGB and Adaptive Dynamic etc.
The file recorded will be, let's say 1gb.

Now, i then open that file with Virtualdub and resave it with identical settings, the resulting file will be about 1.8gb (in this test), and even YV24, YUY2 won't beat it.
The files are identical in settings and looks.

I also tried to manually override the settings to see if i could produce the smaller file, but to no avail.

Perhaps it's obvious, but to me i can't conclude what the reason may be, other than MagicYUV acting differently depending on how the data to it is feeded.

Here is a link to a Direct Stream Copy of 3 sec from the file: http://www.sendspace.com/file/sj04fy

Try yourself to compress it better with MagicYUV, my tests fails only making it about 100mb.

[Ignus2]

Hmmm..., this is extremely weird. Thanks for the file, that is very helpful! I can reproduce it, I analyzed the compressed data, I can see what is happening, but I have no idea yet as to what is causing it. I'll investigate...

Greets,
I.

[zerowalker]

Great that i wasn't mistaken.

Not sure if it's a good thing or not though.
It's really interesting that it's quite smaller in size, and the decoding seems to be on par, and the looks are identical.

Logically it would only mean that normally MagicYUV isn't optimized.
And that itself doesn't even make any sense, it's like the data arrives in a way that makes the algorithm do it's work better.
But as far as i know, that shouldn't be possible except if it alters the image.

If you discover anything please tell, quite fascinated with this case.

[Ignus2]

OK, sorry, I mistook, I analyzed the file wrongly. I see what is happening. Actually dxtory is probably smart, and it automatically creates NULL frames if it sees that the codec produced the same compressed data as before (or maybe it can know that it actually fetched the same frame from the video card). As your sample has many identical frames, this is why it is happening.
You can also see in VirtualDub, that in the original compressed file, some frames are marked in the status bar as [K] (key frame) and some as [D] (null frame).
If you check "Video"->"Preserve Empty frames" and do a recompress, you will get the same size file.

This leads back to your initial question of null frames :)
The answer to that is no, the codec itself doesn't detect null frames, but as it seems, some capture software can detect it anyway. I'll think about it if it makes sense to integrate into the codec too.

Greets,
I.

[zerowalker]

Ah, both good and bad news there, explains the situation at the very least.

This leads back to your initial question of null frames

Fun fact that it indeed brings it back to my initial question which wasn't intended.
Didn't know that codecs could have null frames except if the codec itself accept it as a feature.

Please do, the only reason i see not to include it is that it can cause decoding issues in editing softwares.
But then again, that may be prevented in the decoding part, but worth noting, Lagarith does have this issue, but from my mind i know it occurs at certain cases.

For example.

You edit a file with After Effects, and in the beginning you have an image that shows for, 3 sec perhaps, and then you render it and save it with Lagarith - Null Frame.

The rendered file will be "corrupted", it will display the first frame 1 time, and then go on with the video, leaving the audio out of sync.

Now i haven't done any deep tests in this, but i have stumbled upon similar issues with other codecs, and it seems to occur if the first frame is repeated (not sure if 1 is enough).

So if you look into it, it can be worth looking into that case.


Thanks

[Ignus2]

I took a look at null frames, and based on websites and forum posts it seems it gives more trouble than advantages. Most software doesn't support it properly.
So for now I think it is not worth the trouble, I think most of the time it would be disabled anyway. Also, capture software usually detect null frames anyhow, and software which support null-frames properly (like vdub) doesn't need codec support too.
So for 1.0 final, it will definitely not be included, but if there is more need for it, I'll consider implementing it later.

Thanks for your time testing btw, if you encounter any other problems, feel free to tell me.

Also, the things we discussed about preventing upscaling of YUV formats is on the way, so stay tuned :)

Greets,
I.

[zerowalker]

Ah, true enough. It's only in rare cases it's useful, and MagicYUV isn't really aimed for that, it's usually Game captures or Photos in editing softwares etc, which you normally would encode or something, leaving the lossless part out.

Thanks for looking into it though.

Still find it hard to understand how an application can put null frames into a codec and it can be handled, even when the codec itself doesn't support it, to me it's like having Stereo sound in the middle of a Mono file;P

No problem from my part, i like playing around and testing, and if it can somehow help in your development, it's a Win/Win situation:)
So if you need some testing done, just ask;)

That's great, a feature i really need!


PS: A question i have been thinking about.

My CPU doesn't support AVX, and the latest SIMD is SSE4.2.

I am wondering, does this Codec use anything beyond that, and if so, what is the performance increase in the instructions alone?
If that can be answered at all of course.

Thanks

[Ignus2]

As for null-frames, it is a feature of the AVI, and it just tells the software to use the previous frame, whether compressed or not. So it should work with any codec.

The codec uses SSE2 only. SSE2 is nice, as almost any CPU commonly in use nowadays support that.
The next logical option would be to use AVX. I would skip anything between SSE2 and AVX. AVX might give some speed increase, but not much.
Another interesting path would be Kaveri, but I guess it's not widely popular yet.

Greets,
I.

[zerowalker]

As for null-frames, it is a feature of the AVI, and it just tells the software to use the previous frame, whether compressed or not. So it should work with any codec.
Ah, that explains it better, it's pretty much like a Monitor, if the next image hasn't come yet, it just displays the previous.

The codec uses SSE2 only. SSE2 is nice, as almost any CPU commonly in use nowadays support that.
Okay, can't believe how many applications optimize for SSE2.

The next logical option would be to use AVX. I would skip anything between SSE2 and AVX. AVX might give some speed increase, but not much.
I have no idea how good it is or anything, just find it fascinating that it's been there for a lifetime, and it's still used, while the other newer are pretty much only used in Emulators and such things.

Hmm, thought AVX would give better speed increase than that, seeing the time between the instructions and the "talk" about AVX using 128-bit or what it was that has been awaited for ages, not that it tells me much;P

Another interesting path would be Kaveri, but I guess it's not widely popular yet.
Kaveri, searched on it just to be safe, it's the AMD CPU architecture right?
Isn't that the CPU/GPU thing that AMD aims for.

Not really sure what you mean with path there, does it have something that can be used, and if so wouldn't that leave the other CPUs alone?

Thanks

[Ignus2]

AVX is good if you have lots of vectorizable code and the algorithm spends most of it's time there. The codec only spends a part of it's time in vectorizable code, and SSE2 is a very good common ground for that.

Yes, Kaveri is the new AMD hybrid CPU/GPU with unified memory. Which is very nice, as no time is spent uploading/downloading data for GPU processing (which is usually a major bottleneck of performance when doing GPGPU processing), it remains in the same place. GPU is unbeatable for vectorizable code, and the regular CPU part can be used for the rest. It's just not widespread (yet).
Implementing support for it would simply mean that for people with Kaveri CPU the codec would have better performance.

Greets,
I.

[zerowalker]

AVX is good if you have lots of vectorizable code and the algorithm spends most of it's time there. The codec only spends a part of it's time in vectorizable code, and SSE2 is a very good common ground for that.

Ah, okay, so it's like a middle ground.

Yes, Kaveri is the new AMD hybrid CPU/GPU with unified memory. Which is very nice, as no time is spent uploading/downloading data for GPU processing (which is usually a major bottleneck of performance when doing GPGPU processing), it remains in the same place. GPU is unbeatable for vectorizable code, and the regular CPU part can be used for the rest. It's just not widespread (yet).

Certainly hope it get's more well known and used with Intel etc as well, as it will benefit more of everything, GPGPU is barely used cause of the bottleneck.

Thanks for the explanation, sorry for bringing into a bit of an off topic course, happens a lot, never seems to be able to predict my own actions.

To point it towards the main goal, may i ask when you expect a release for the 1.0?

Thanks

[Ignus2]

I'm pleased to announce that MagicYUV 1.0rc2 is out.

Some minor features were added, like support for I420 format and compatibility options for Avisynth.

Decoder-side conversions can now be disabled on a per-format/per-target basis. This way YV12 can be prevented to be decoded as YV24 for example, while still keeping YV12-->RGB decoding.

This release can be considered final in terms of 1.0 features, only bug-fixes are expected (if any is found) for 1.0.

Link: http://magicyuv.com

Greets,
I.

[zerowalker]

Much Appreciated!

Good work:)!

(Decompression Settings is Grayed?)

[Ignus2]

Much Appreciated!

Good work:)!

(Decompression Settings is Grayed?)

From VirtualDub and the like, it wouldn't be persistent from there anyway (Vdub store settings temporarily).
It can be set from the start menu "MagicYUV" -> "MagicYUV codec configuration"

EDIT: Normally I would never do this, but as only you downloaded it so far, I pulled it back and uploaded a new version with a tooltip on the button to avoid confusion.

Greets,
I.

[zerowalker]

Ah, okay.


Checked it, but wondering, should it be set to YV12 -> YV12 and RGB, YUY2 -> YUY2 and RGB etc?

[Ignus2]

Ah, okay.


Checked it, but wondering, should it be set to YV12 -> YV12 and RGB, YUY2 -> YUY2 and RGB etc?

The no-conversion path (same format like YV12->YV12) are by default always on and cannot be prevented (there is no checkbox for it), as those don't count as 'conversion'. Would be quite funny if you could disable it for a lossless codec I think :-)

Greets,
I.

[zerowalker]

No didn't mean that, i meant, isn't YV12 set by default to allow upsampling to YUY2?
Or am i misreading the settings?

[Ignus2]

No didn't mean that, i meant, isn't YV12 set by default to allow upsampling to YUY2?
Or am i misreading the settings?

Ah, sorry. Yes, thats allowed, but not a problem, as avisynth tries YV12 before YUY2, so it will not upsample.

[zerowalker]

Ah, seem to have forgotten, thought it went YV24 - > YUY2 -> YV12.
But that explains it, then everything is as it should:)!

Thanks

[Gargamel]

Thank you, Ignus2 !

[Jeroi]

I would vote for null frames and delta frames as well! The main advatages are compression benefits. You will only save first or xorred image data which is full of null pixels. These are really good benefits for Lossless encoder since reduced image data affects directly to filesize. How ever there is the MSU codecs successos ScreenPressor which costs like 25$ and can compress quite good screens and game data with lossless capture.

[zerowalker]

I would like to benchmark MagicYUV in comparison to other similar codecs, is there a recommended way for this?
As playing in a media player can only do so much, would be easier with some number crunching instead.

[Ignus2]

I would like to benchmark MagicYUV in comparison to other similar codecs, is there a recommended way for this?
As playing in a media player can only do so much, would be easier with some number crunching instead.

For benchmarking codec speed, it is best to eliminate all other factors, like:
- Disk access
- Other frame processing
- Rendering
- CPU frequency scaling
etc.

So I would use a ramdisk at minimum, and preferably some command line app, like vdub with sylia script, or my avibench tool or similar. Make sure to avoid comparing apples to oranges, I mean set up all codecs the same way regarding color spaces and conversion (so to avoid situation where one codec compresses YV12 because of implicit conversion, while another YUY2, etc.)
Also, make sure to set your PC to performance mode (max CPU freq) to get consistent results.
For starters... :-)

Greets,
I.

[zerowalker]

Ah, quite much that needs to be taken into account.

Will see if i get around to trying it out, i have only done some fast testings, and with that i saw pretty much that MagicYUV as expected is very lightweighted, but doesn't waste space as to boldly put it.
This is very attractive to the point you are making for the codec to be primary for Editing software.

Thanks

[Ghostlamer]

Ignus2, could you add an option, something like this "forced decoding YUV 4.0.0, 4:2:0, 4:2:2, 4:4:4 as RGB".
There are non-linear video editors that support YUV 4.0.0, 4:2:0, 4:2:2, but convert it to rgb(on the fly), do it not very correctly, resulting in slightly incorrect brightness and color output.
Converting videos to RGB, takes up more space and time.
Forced codec decoding in RGB, in most cases will work around this problem.
Input will be RGB, and the editors will not make any color conversions.
But make it disabled by default.
Thanks.

[Ignus2]

Ignus2, could you add an option, something like this "forced decoding YUV 4.0.0, 4:2:0, 4:2:2, 4:4:4 as RGB".
There are non-linear video editors that support YUV 4.0.0, 4:2:0, 4:2:2, but convert it to rgb(on the fly), do it not very correctly, resulting in slightly incorrect brightness and color output.
Converting videos to RGB, takes up more space and time.
Forced codec decoding in RGB, in most cases will work around this problem.
Input will be RGB, and the editors will not make any color conversions.
But make it disabled by default.
Thanks.

So instead of the NLE software, you want the codec to do the conversion, always? The reason being NLE software do it incorrectly?

So this means, you want to be able to disable decompressing into native format. Or something like "Always suggest RGB when decoding"?

So if I understand correctly, you want to store material compressed always as YUV 4:x:x but edit it in RGB?

Greets,
I.

[zerowalker]

Isn't this already available since you added the "Avisynth Fix"?

[Ghostlamer]

So instead of the NLE software, you want the codec to do the conversion, always?
So this means, you want to be able to disable decompressing into native format. Or something like "Always suggest RGB when decoding"?
So if I understand correctly, you want to store material compressed always as YUV 4:x:x but edit it in RGB?
Yes, that's right.
To force the output was always RGB.
Just most nonlinear editors make conversion from YUV 4:x:x to RGB in realtime(even when the codec RGB options enabled), if the source is not the RGB.
And do it in different ways, some better some worse..

[Ignus2]

Isn't this already available since you added the "Avisynth Fix"?

No, as it currently prevents upsampling only, but you cannot disable decoding to native format of what was compressed (i.e. cannot prevent lossless decompression without conversion).

What Ghostlamer needs is the opposite: to prevent decoding into native format and allow only upsampling/conversion.

Whether this scenario makes sense for a lossless codec is another question :)

Greets,
I.

[Ghostlamer]

Certainly at the codec level it will be more correct than in the editors.

What Ghostlamer needs is the opposite: to prevent decoding into native format and allow only upsampling/conversion.
Make it optional.

Сurrent decoding options:
For player - 1 (http://imgur.com/jBqf6Lc), 2 (http://imgur.com/tul47uL).
Video editing software (Magix Video Pro X6) - useless (http://imgur.com/V28YjEs).

[Ignus2]

Certainly at the codec level it will be more correct than in the editors.


Make it optional.

Сurrent decoding options:
For player - 1 (http://6g6.eu/sih0-2257.png), 2 (http://6g6.eu/sih0-1477.png). http://simplest-image-hosting.net/
Video editing software - useless (http://6g6.eu/sih0-3175.png).

It would be quite funny making it non-optional, that would basically defeat the purpose of the codec :)

The Decompression settings currently refer to conversions only, so decoding to the native format (ex. to YV12 from compressed 4:2:0) is currently always possible.

Maybe I'll add the settings to the next release.

Greets,
I.

[Ghostlamer]

Maybe I'll add the settings to the next release.
Thanks.

[WorBry]

Hey Ignus,

It's been a while since I tested Magic-YUV, not since those early tests with your pre-release versions in fact. But I'm happy to report that version 1.0rc2 now works very well with my AVCHD and HDV camcorder clips in the NLE I use - Corel VideoStudio Pro x6.

VSPx6 is still only 32-bit and performs edit transformations in RGB colorspace. As such, any compatible third party vfw codecs need to be able to convert from and to RGB. It's pretty useless for 'native editing' of AVCHD and HDV, but the 'smart rendering' feature works well with intra-frame intermediate formats and the program has very good MT process support.

My current practice is to pre-transcode the source clips to the intermediate format via AVISynth using a suitable frame-indexing decoder (DGIndex/Dec for HDV MPEG-2 and now DGIndex/DecIM for AVCHD). VDub is set for Fast Recompress so Magic YUV should be receiving YV12 input anyway, but as a precaution (perhaps unnecessary) I've configured Magic YUV with the Input Colorspace set to YUV 4:2:0 and the Mode Conversion set likewise.

The transcodes look fine and play very well. UTVideo is also fast, but playing the files in MPC-HC, there's always a bit of stutter and unresolved offset delay. The Magic YUV encodes play very smoothly and the initial sync delay resolves very quickly. I haven't done any repeat comparative metric testing with this version but the fast decompress rate definitely shows.

So, then I load the Magic YUV (avi) encodes in VSPx6 and set the edit Project to use Magic YUV as the timeline format. You might re-call that when I tried to do this with the pre-release versions, VSPx6 always crashed and would only work if I installed an intermediary transform codec like the YUVFilters pack. Now, no such problems provided the Magic YUV input colorspace is set to Auto or RGB. Set to anything else and VSPx6 complains for lack of a suitable compressor. I have the Mode (conversion) set to YUV 4:2:0.

So configured, the Magic YUV encodes edit very smoothly on my PC at least (6-core AMD FX600 CPU) and it looks like the 'smart rendering' is working fine i.e. when set to render out in the same format as the Project (or the format of the 'first clip') it only re-encodes those frames affected by an an applied edit effect/transform and copies the unaffected frames. Unfortunately there's no 'Smart Render' analysis feature to show that definitely, but it looks to be the case just looking at the rendering progress bar.

All in all, very impressive. I'd been using UTVideo or Cineform until now (each have their pros/cons), but I could see myself using Magic YUV; I assume it's bug free?

So, great work, and no doubt assisted by the valuable feedback of those who could contribute more useful insight and advice than I.

Cheers :)

[Gravitator]

Hi!
Can we expect a higher compression? Or two passes :o > sample (https://mega.co.nz/#!EEE31ZoT!FTY2fmkgIBhJqy0UhFP-QAiDwvu5vhUVMJY-2LFMVA8)

[zerowalker]

Hi!
Can we expect a higher compression? Or two passes :o > sample (https://mega.co.nz/#!EEE31ZoT!FTY2fmkgIBhJqy0UhFP-QAiDwvu5vhUVMJY-2LFMVA8)

higher compression can most likely be made one way or the other while keeping the performance. It would probably be quite minimal though as the huge optimizations has probably been made already.

2 passes is probably not something for this kind of codec, which is made for speed in both encoding and decoding (especially decoding), and i think the increase in compression efficiency is probably not worth it during these kind of works.

I personally however, would probably want it if it can decrease the size by at least 5%, while keeping the same decoding speed.

(The author is most likely to correct if i have said anything that's incorrect here).

[kolak]

I think you are correct- any compression improvements while keeping speed will be in region 5, best 10%. Quite often this is a huge effort and all what you get is this 5%. I think magyuv is really good with speed ratio/compression.

[zerowalker]

I think you are correct- any compression improvements while keeping speed will be in region 5, best 10%. Quite often this is a huge effort and all what you get is this 5%. I think magyuv is really good with speed ratio/compression.

Agree, MagicYUV is doing extremely well, it's competing at UT Video Codec, it's in the middle of the compression choices of the 2 settings (compress for decoding/size or something like that).

Which in itself is very nice, and clearly acceptable in lossless format sizes.

Decoding speed is in a different league, it's extremely fast, especially in cases were it usually is slower (much noise,grain etc).

I don't really see any "Real" reason for it to be smaller, faster is another story, there isn't a thing as too fast, but it's currently clearly enough for what it's supposed to do.


However if you need more speed, the author has shown interest in AMD Kaveri, which is a hybrid CPU and GPU, which could increase the performance, as it decreases overhead and such, how much however is another story, and we will have to wait till/if he decides to do this work.

[Ignus2]

Sorry for being absent, I have been busy with personal matters.

In the meantime, MagicYUV 1.0rc3 is out: Link (https://magicyuv.com/index.php/download).

This release includes:

Option to suggest RGB first on output when decoding in the Decompression Settings (as requested by Ghostlamer).
Fix for RGBA encoding which was completely broken. An option was added to the encoder dialog, see the tooltip on how to use it.
Fix for a very rare crash-bug when decoding. Newly encoded videos will have it fixed.
Support for some new fourccs (2vuy, HDYC, IYUV).


@WorBry Thanks and sorry for the late answer. Nice to hear that the codec finally works with the software you use :) All insights and testing is welcome. I really missed RGBA testing though, I accidentally found someone having problems on a German forum, and I only began investigating after that.

@Gravitator Not in the near future. It is hard to improve compression ratio without slowing down the codec, especially if we want to keep it intra-only (making it inter is also non-trivial). So for the foreseeable future, it will remain as it is.

Greets,
I.

[zerowalker]

Fix for a very rare crash-bug. Newly encoded videos will have it fixed.

Could you please explain this?

Does videos that are previously encoded contain a bug which causes or crash when decoded or what;S?

[Ignus2]

Could you please explain this?

Does videos that are previously encoded contain a bug which causes or crash when decoded or what;S?

Basically yes, it MIGHT happen. Actually, it was there since the launch of the codec since February (and even years before, ever since the codec started to take shape), but even I encountered it only once so far with a combination of software AND material (YUV with alpha format converted to RGBA).

So chances are low.

Greets,
I.

[zerowalker]

Okay so i don't have to worry about my current files having a chance of being corrupt?
It's basically, a crash MAY occur if conditions are met, but at the same time if you redo it, it may not crash? (Or is it constant?).

And well what i have is simply YV12 YV16/YUY2 YV24 or RGB, have no memory of ever using YUV with Alpha, didn't even know it had that option.

[Ignus2]

Okay so i don't have to worry about my current files having a chance of being corrupt?
It's basically, a crash MAY occur if conditions are met, but at the same time if you redo it, it may not crash? (Or is it constant?).

And well what i have is simply YV12 YV16/YUY2 YV24 or RGB, have no memory of ever using YUV with Alpha, didn't even know it had that option.

It's constant, not sporadic. For example, I had it happen for one specific frame of one specific file in a specific format with After Effects, while VDub and Media Player Classic and all my test tools were fine with the same file.

Furthermore, some clarification: Any file encoded with any release before 1.0 final is reached should be considered corrupt in one way or another, not specifically because of this bug but because it is beta software and as such, bugs may be present, even ones that I'm not aware of. This is a general thing for all software not being reached 1.0 stable status.

Anyway, if your files open now correctly in VDub, why worry? :)

Greets,
I.

[zerowalker]

Okay that explains it:)

Thanks

[WorBry]

@WorBry Thanks and sorry for the late answer.


No problem :)

[EncodedMango]

The biggest downside I'm finding is support for the codec for example ffmpeg doesn't recognise it. So since usually I record RGB I just convert it to Lagarith and then go about encoding to my needs, I wanted to try YUV 4:4:4 but I can't since ffmpeg won't recognise MAGY and Lagarith doesn't have YV24 but otherwise it's a pretty fast codec.

Are you planning to ever release the source, Ignus2?
If not I guess I'll have to find another lossless codec that can convert MAGY's YUV 4:4:4 to usable YV24.

[zerowalker]

I am not entirely sure this is correct but.

UT Video Codec should be closed source, and ffmpeg has made a decoding version of it as well?
They will probably make MAGY alone or with the authors help, sooner or later.

Though i am interested in if you would release the source, but i guess that's a bit sensitive as you are the maker, and releasing it could be quite controversial in a sense,
so i at the end of the day, as long as it continues to work in all it's glory, i am happy:)

[EncodedMango]

I hope they do, just asked about source because if it does happen then as far as I'm aware someone could submit the codec to ffmpeg that way. I don't know the specifics but I remember glancing over it in the ffmpeg site somewhere.

[vivan]

UT Video Codec should be closed sourceWhy? :)
It's open source, and released under gpl v2.

[zerowalker]

It is?
Was sure it was closes source and release only by the author, need to check on that.
Then my conclusion is invalid.

[Sparktank]

need to check on that.

You can download the source for each version from his site.
http://umezawa.dyndns.info/archive/utvideo/

[Ignus2]

The biggest downside I'm finding is support for the codec for example ffmpeg doesn't recognise it. So since usually I record RGB I just convert it to Lagarith and then go about encoding to my needs, I wanted to try YUV 4:4:4 but I can't since ffmpeg won't recognise MAGY and Lagarith doesn't have YV24 but otherwise it's a pretty fast codec.

Are you planning to ever release the source, Ignus2?
If not I guess I'll have to find another lossless codec that can convert MAGY's YUV 4:4:4 to usable YV24.

Not in the near future. I've prepared an SDK though (binary only) which can be used to integrate the codec into whatever software one wishes, though it's not released as of yet.

Greets,
I.

[Ignus2]

The 1.0rc4 version of the codec is out. This should be the last release before 1.0 final, and if all goes well (no new bugs are reported), this will become the final 1.0 release.

This release fixes a serious decoder bug which could cause random crashes during decoding. It works for all previously encoded files.
Decoder performance was improved by 8-10% overall (mainly for 64-bit).
Memory handling was also improved, the codec now uses much less memory.

Grab the release from the website: http://magicyuv.com (http://magicyuv.com/index.php/download/magicyuv)

Greets,
I.

[StainlessS]

Many Thanks.

[Gargamel]

Congratulations, and thank you again Ignus2 !

[zerowalker]

Nice, never had any crash issues as i recall. But then again have not used it that much i guess.
But it's really one of the top choices, only time i don't use it would be when Lagarith thrives, which are for certain content.

But even faster decoding, 8-10% is great.
How come it's mainly in 64bit. was 64bit slower than 32bit?

Also less memory usage is a good bonus optimization.

Well Done;)!

[Videogamer555]

Please add an uncompressed raw mode as well as your lossless compression mode to your codec. I need to generate some raw YUV444 frames for a specific application I'm working on. The simplest way to do so is to use the IPImageCompress API function in my program. Unfortunately this function requires that I have a codec installed for encoding in YUV. Windows comes with a YUV420 encoder only (though it comes with decoders for other YUV formats). As such, I spend hours, looking for a YUV codec that I could install that would enable me to save YUV444 frames. In my search, I finally ran across MagicYUV. Sadly, it only is capable of saving losslessly compressed YUV frames. For many video applications, lossless compression is better than raw because it saves file space, while preserving the video data that would be lost in a lossy compression. However my application specifically requires raw YUV444 frames. As such, I need to ask you to include a "full raw" mode in the next release of your codec (a mode that would allow directly outputting raw YUV 444, 422, 420, or 400 Luma grayscale frames), so that I can complete the program I'm writing.

I don't think any other codec exists that can do what I want, and yours comes closer than any to being what I need. However as my project calls for raw YUV frames, I am going to ask you to include that in your codec's next release as a settable option from your codec's control panel.

[raffriff42]

I need to generate some raw YUV444 frames for a specific application I'm working on....
I don't think any other codec exists that can do what I want...I can think of one way to generate such a video, with VirtualDub: Set output color depth (https://www.dropbox.com/s/9areszlxbpis00v/virtualdub-export-raw-yuv444-01.jpg?dl=0) to '4:4:4 planar'.
Set video codec (https://www.dropbox.com/s/thcla17749f6qva/virtualdub-export-raw-yuv444-02.jpg?dl=0) to 'uncompressed'
The resulting video is 'rawvideo YV24' per ffprobe (https://www.dropbox.com/s/btj79j8uk0qm3xn/virtualdub-export-raw-yuv444-03.jpg?dl=0).
EDIT
VirtualDub's internal 'uncompressed' codec is not available for other applications, so while my suggestion does generate 'raw YUV444 frames', it doesn't meet your desire for a codec, as pointed out by Ignus2 below. And of course, ffmpeg offers another way to create such a file.

[Ignus2]

Please add an uncompressed raw mode as well as your lossless compression mode to your codec. I need to generate some raw YUV444 frames for a specific application I'm working on. The simplest way to do so is to use the IPImageCompress API function in my program. Unfortunately this function requires that I have a codec installed for encoding in YUV. Windows comes with a YUV420 encoder only (though it comes with decoders for other YUV formats). As such, I spend hours, looking for a YUV codec that I could install that would enable me to save YUV444 frames. In my search, I finally ran across MagicYUV. Sadly, it only is capable of saving losslessly compressed YUV frames. For many video applications, lossless compression is better than raw because it saves file space, while preserving the video data that would be lost in a lossy compression. However my application specifically requires raw YUV444 frames. As such, I need to ask you to include a "full raw" mode in the next release of your codec (a mode that would allow directly outputting raw YUV 444, 422, 420, or 400 Luma grayscale frames), so that I can complete the program I'm writing.

I don't think any other codec exists that can do what I want, and yours comes closer than any to being what I need. However as my project calls for raw YUV frames, I am going to ask you to include that in your codec's next release as a settable option from your codec's control panel.

I'm not entirely sure I understand, but basically you need an RGB to YUV converter "codec"? I wouldn't include this kind of functionality in the codec, it's a bit out of scope.
The solution by raffriff42 should be fine, except if you have to do this conversion in your program.

Greets,
I.

[zerowalker]

Hmm, is there a reason why MagicYUV is not available as a filter with 64bit (GraphStudio-Next for example)?

[Ignus2]

Hmm, is there a reason why MagicYUV is not available as a filter with 64bit (GraphStudio-Next for example)?

Yes, it is currently implemented only as a VFW codec, not a DShow filter. It can join filter graphs however with the AVI Compressor and AVI Decompressor wrapper filters.
It is planned to implement it for DShow too, but it's not high priority (will come probably sometime after 1.0 final).

Greets,
I.

[zerowalker]

Ah okay, didn't know they were different (not that good at this stuff). But that explains it, good that the wrappers can be used as alternatives.

Thanks

[De-M-oN]

Agree, MagicYUV is doing extremely well, it's competing at UT Video Codec

its actually way faster than utvideo.
And thats quite impressive with its compression efficiency.


This codec gives me goddamn awesome fps rates when recording with Afterburner games.
I have a RAID 0 which can write 352 mbyte/s

This codec is able to write in 2560x1600 50fps, RGB with dynamic compression and NO lags at all, an awesome compression efficiency which makes me really wonder about the performance still being even faster than utvideo.

Thats outstanding. Awesome codec which performs hellish fast at recording games while having an excellent compression.

Thank you ! :)

I have this problem though:

http://forums.guru3d.com/showpost.php?p=4898160&postcount=1847

I don't know what could disturb Afterburner here..

[Ignus2]

its actually way faster than utvideo.
And thats quite impressive with its compression efficiency.

This codec gives me goddamn awesome fps rates when recording with Afterburner games.
I have a RAID 0 which can write 352 mbyte/s

This codec is able to write in 2560x1600 50fps, RGB with dynamic compression and NO lags at all, an awesome compression efficiency which makes me really wonder about the performance still being even faster than utvideo.

Thats outstanding. Awesome codec which performs hellish fast at recording games while having an excellent compression.

Thank you ! :)


Thanks, that's great to hear :)
Though decoding performance should be what the codec excels the most, but I'm glad to hear that encoding also performs well :)


I have this problem though:

http://forums.guru3d.com/showpost.php?p=4898160&postcount=1847

I don't know what could disturb Afterburner here..

I'll take a look at it, thanks for reporting.

Greets,
I.

[Ignus2]

I have this problem though:

http://forums.guru3d.com/showpost.php?p=4898160&postcount=1847

I don't know what could disturb Afterburner here..

I seem to have tracked it down. I will try not to be too technical, but the issue is caused by Afterburner initializing/deinitializing the codec just at the same time while loading/unloading the codec DLL (so the OS has the DLL loader lock), and this causes a deadlock inside the codec while it is initializing it's worker threads. (Or at least that is what I think is happening.)
Most probably Afterburner doesn't close the codec properly.

I consider this to be a bug inside Afterburner, and the fix would involve Afterburner to open/close the codec separately from loading/freeing the codec DLL. (Or at least properly close it.)
Though it might be worked around in the codec itself (which might introduce other bugs), but this would force me to make yet another Release Candidate :-/ (as I have the policy of making the final 1.0 release directly from an RC).
I'll think about it...

Thanks for the report BTW!

Greets,
I.

[zerowalker]

How come other codecs does this and MagicYUV doesn't, if i may ask?

[zerowalker]

A suggestion,

Not really a necessity but can't hurt, and i guess it should be fairly easy to add.

Support for: UVYU.

My card seem to have that option, though i use YUY2 as it's the "standard".
But just noticed MagicYUV wasn't listed and UT Video Codec and ffdshow were (those were the only ones).

[Ignus2]

How come other codecs does this and MagicYUV doesn't, if i may ask?

It would be a bit lengthy to explain exactly why this deadlock happens, but in short other codecs probably don't initialize threads when they are loaded (so not creating a problem at DLL unload), and MSI Afterburner only seems to incorrectly close (or not close) codecs when showing it's configuration dialog.

If you are really interested and want to be fascinated by the intricacies of DLLs and threads, here is an article to get you started (especially the "Best Practices for Synchronization" section): :)
http://msdn.microsoft.com/en-us/library/windows/desktop/dn633971%28v=vs.85%29.aspx


BTW, I already fixed/worked around the issue with MSI Afterburner, so expect a release soon (strictly bugfix only).

A suggestion,

Not really a necessity but can't hurt, and i guess it should be fairly easy to add.

Support for: UVYU.

My card seem to have that option, though i use YUY2 as it's the "standard".
But just noticed MagicYUV wasn't listed and UT Video Codec and ffdshow were (those were the only ones).

I guess you wanted to write YVYU. Yes, that could be added, but that will only make it after 1.0, as I made a feature-freeze (should have done that with RC1...) so anything new will have to wait after 1.0 is out.

Greets,
I.

[Ignus2]

@De-M-oN:
I noticed a whole lot of talking on letsplayforum.de about the codec. Thanks for the promotion ;)

First of all, I see some very good feedback coming in, but also some things to reflect on: :)


Predict Left is not necessarily the fastest option (to the contrary of common knowledge) especially when we also consider decoding speed. I wrote about it here why is that so: http://forum.doom9.org/showthread.php?p=1668769#post1668769

When compressing to YUV, the Full range YUV option should be turned off. Programs usually expect YUV to be limited range.

While encoding is also optimized to be fast, but decoding speed should show even bigger speed difference when compared to other codecs.

The decompression setting "Set Avisynth compatible" button only adjusts the checkboxes above it, nothing else. But by default, it is already set up that way, so there is no need to press it. What it basically does is to prevent conversion to YV24 from sub-sampled colorspaces.


That's some I thought about quickly. But feel free to report/ask if something doesn't work as expected.

Greets,
I.

[WorBry]

Predict Left is not necessarily the fastest option (to the contrary of common knowledge) especially when we also consider decoding speed. I wrote about it here why is that so: http://forum.doom9.org/showthread.php?p=1668769#post1668769


Oh boy, my verbosity has, once again, come back to haunt me (cringe) :o

[Ignus2]

Oh boy, my verbosity has, once again, come back to haunt me (cringe) :o

LOL :)
Well I certainly didn't thought about it this way at all, so apologies! :o

Greets,
I.

[zerowalker]

It would be a bit lengthy to explain exactly why this deadlock happens, but in short other codecs probably don't initialize threads when they are loaded (so not creating a problem at DLL unload), and MSI Afterburner only seems to incorrectly close (or not close) codecs when showing it's configuration dialog.

If you are really interested and want to be fascinated by the intricacies of DLLs and threads, here is an article to get you started (especially the "Best Practices for Synchronization" section): :)
http://msdn.microsoft.com/en-us/library/windows/desktop/dn633971%28v=vs.85%29.aspx


BTW, I already fixed/worked around the issue with MSI Afterburner, so expect a release soon (strictly bugfix only).



I guess you wanted to write YVYU. Yes, that could be added, but that will only make it after 1.0, as I made a feature-freeze (should have done that with RC1...) so anything new will have to wait after 1.0 is out.

Greets,
I.

Ah, will see if i understand it to some degree. Get the concept at least, thanks:)

Indeed, damn i even checked and searched to be sure i didn't misspell before posting, not sure how i even succeed:S

And yeah no need for it right now. No need for more RCs ;P
Also not a critical thing, just something trivial that's hopefully can get supported for the heck of it, as it should be simple to implement, as all YUV versions are the same with different positions as far as i get it.

Thanks

[WorBry]

.....so apologies! .

Absolutely none needed. ;)

Great work with codec , btw. I'm using it all the time now as an edit intermediate.

[Ignus2]

1.0rc5 is out: http://magicyuv.com/index.php/download
I REALLY hope it's the last RC before 1.0 final.

This release includes fixes and workarounds for MSI Afterburner, so it is intended only for the users of that app.
Notably, it fixes the deadlock bug, but also includes a compatibility modification for the codec configuration dialog box when the codec detects that it is called from Afterburner.
Basically it changes the OK button of the config dialog to "Save as default and Close" when running from Afterburner, as Afterburner doesn't store VFW codec settings at all, so the settings must be saved globally to the registry to take effect in this case (otherwise anything you set in the dialog would be lost).

No other changes have been made, and if no new bugs are reported, 1.0 final will be released in a week.

Greets,
I.

[zerowalker]

Ah nice. Damn bad implementation from MSI Afterburner though.
Afterburner doesn't store VFW codec settings at all, so the settings must be saved globally to the registry to take effect in this case

Active as always, good job:)

[Ignus2]

1.0 final is out :)

It is basically a re-branded 1.0rc5, so apart from a version bump, no changes have been made.

A long time have passed since I released the first 0.9alpha, and it has been a long road to get here. Thanks for everyone who helped out with suggestions/comments/bug reports!

Now that the feature freeze is lifted, features can come again :) I already have some ideas, and there are already some convenience features done for the next release, so expect a new feature release soon!

Greets,
I.

[zerowalker]

Grats for the 1.0 Release;)!

It's great to see you keeping up the good work, quite the enthusiasm you got.

Care to share any of your, ideas;P?

//Good Job!

[Gargamel]

In time ...
Big thank you, Ignus2 !

[De-M-oN]

I noticed a whole lot of talking on letsplayforum.de about the codec. Thanks for the promotion

It has good performance at recording games, so it was worth mentioning it. :)

Predict Left is not necessarily the fastest option (to the contrary of common knowledge) especially when we also consider decoding speed.

Ok interesting. It felt faster than dynamic while testing though (but just very minor) I use dynamic for the better compression.

When compressing to YUV, the Full range YUV option should be turned off. Programs usually expect YUV to be limited range.

Was that mentioned somewhere to have it enabled? Should be not the case.

While encoding is also optimized to be fast, but decoding speed should show even bigger speed difference when compared to other codecs.

Ok. But it uses not much cpu at encoding and is pretty fast as well. And thats good.
Encodingspeed while having as low cpu usage as possible is for us the more relevant thing we need for recording games.

The decompression setting "Set Avisynth compatible" button only adjusts the checkboxes above it, nothing else.
I know.

But by default, it is already set up that way, so there is no need to press it. What it basically does is to prevent conversion to YV24 from sub-sampled colorspaces.

For me all checkmarks were set. After pressing avisynth compatible the checkmarks at yuv 444 went away.

Thank you very much for fixing the Afterburner problem although it was more a fault by afterburner.

Thank you!

[Ignus2]

@zerowalker:
I'll try them out first then I'll tell about it ;-)

@De-M-On:
Thanks, you're welcome. If you encounter any other strange behaviour or bugs, just tell me.

Greets,
I.

[nekosama]

I'm having quite a hard time getting magicyuv to encode using avisynth. Everytime I try to encode my script (using a magicyuv lossless avi as source using avisource) I get 240 blank frames, hell even just plain opening the file in a mediaplayer (mpc-hc) gives 240 blan frames, only opening the script in avspmod gives me correct results.

Maybe it has something to do with NULL frames like lagarith but I don't even know if magicyuv makes use of them.

btw it's not caused by some corrupted plugin of some sort, hell even just calling source and using removegrain (or any simple filter for that matter) ends up in 240 blank frames in a mediaplayer.

[zerowalker]

I'm having quite a hard time getting magicyuv to encode using avisynth. Everytime I try to encode my script (using a magicyuv lossless avi as source using avisource) I get 240 blank frames, hell even just plain opening the file in a mediaplayer (mpc-hc) gives 240 blan frames, only opening the script in avspmod gives me correct results.

Maybe it has something to do with NULL frames like lagarith but I don't even know if magicyuv makes use of them.

btw it's not caused by some corrupted plugin of some sort, hell even just calling source and using removegrain (or any simple filter for that matter) ends up in 240 blank frames in a mediaplayer.


I have been using MagicYUV A lot with Avisynth with no problems at all. Even when null frames has been present (Which MagicYUV does Not support in itself, however it's a global feature in AVI).

Really weird issue. You sure it's not some x64/x86 thing?

[De-M-oN]

My GZDoom edited pfenh mod uses some effects which are very cpu intensive and so my recording fps seldom goes below the wanted. In this case dummy frames are inserted as well then.

Never had a problem with it though.

Avisynth reads it fine, vdub 32bit does it, mpc-hc 32bit as well (EVR-CP Renderer)

So I think for me it works.

[nekosama]

nope, Tried both x86 and x64 x264 but no luck. Maybe it has something to do with the decoder? is there something that goes between avisynth and x264 (ignoring avs4x264mod because nothing changes whether or not I use this).

[zerowalker]

I have used MagicYUV with x264 both directly and with avs2x264mod, no issues here.

[nekosama]

Fixed the problem by swapping avisource with ffmpegsource2, found out that avisource can bug out with big files.

[zerowalker]

How big?

[nekosama]

5.2 GB, 704x450, not 'that' big though it seems like the only logical cause till something new pops up.

[zerowalker]

Indeed not that big.
I can easily have 300gb of Video without issue.
Something must be wrong on your end, probably some complications, mismatching etc.

[the_weirdo]

Fixed the problem by swapping avisource with ffmpegsource2, found out that avisource can bug out with big files.

Wait, I don't think FFMS2 can decode MagicYUV sources, at least by now. Your avi may not be a MagicYUV encode then.

[Sparktank]

I've used SD resolution with MagicYUV as an intermediate without problems.
>30GB for lossless intermediates and no issues.

[nekosama]

Yes my bad the_weirdo, made an x264 lossless in the meantime and mistook it for magicyuv. Though it's still the same problem with trying to filter a magic yuv source with avisource input.

[foxyshadis]

Indeed not that big.
I can easily have 300gb of Video without issue.
Something must be wrong on your end, probably some complications, mismatching etc.

He's got to be using Avisynth 2.5. 2.6 has lots of fixes for 4+ GB AVI files.

nekosama, try updating to 2.6a5.

[De-M-oN]

He's got to be using Avisynth 2.5. 2.6 has lots of fixes for 4+ GB AVI files.

I could load with 2.5 avis with similar sizes too though.

[zerowalker]

Have you been able to test something with GPU acceleration, HSA etc?
We talked about it before and know you were interested in it, so wondered if you have had any chance to test it, or some ideas?

Thanks

[Ignus2]

Have you been able to test something with GPU acceleration, HSA etc?
We talked about it before and know you were interested in it, so wondered if you have had any chance to test it, or some ideas?

Thanks

Hehe, well, I'd love to, but time doesn't permit :-/
Also, I don't yet have hardware to develop on, so that's also an obstacle.
It's still something for the future to try out.

Greets,
I.

[zerowalker]

Ah, well was expecting that. Not by any means saying you Should have done it by now, but asking doesn't hurt;P

Just that you have the interest in it is a good thing, as i myself find it an interesting approach, as it holds some hope.

Well will have to see what happens in the future when you got the time,will and hardware to do any sort of testing;)!

Thanks

[Mick]

Hi there,
i downloaded RC5 and 1.0 for my Win2K Pro System and the Configuration Tool for the Codecs simply Reports the Error Message "Function not supported" and quits. The MagicYUV Codec in Versions RC5/1.0 is about 20 Frames slower (!) with a higher CPU (28-35 % more) load than the RC4 that caused no trouble with Win2K Pro (SP4). Sure, i could step up to another OS but i don't want that for several reasons, so let's leave that Discussion out right now.

Typical CPU load in Virtualub (Encoding 4K YUY2) with W2K/XP:
HuffYuv -> between 15 to 35 %
Lagarith -> between 40 to 65 %
UtVideo -> between 48 to 70 %
MagicYUV -> between 28 to 40 % (RC4), 62 to 88 % (RC5/1.0)

Resolution: 4K PAL at 25 Fps with PCM Audio, 1:1 Interleaved

Note: The CPU load is about the same on a Win7 64 Bit System.

Decoding/Playback 4K PAL Windowed/Full Screen:
HuffYuv -> fluent and stable Frame rate (+)
Lagarith -> stutters, drops Frame rate down (-)
UtVideo -> almost fluent, drops down to about 20 Fps (-)
MagicYUV -> fluent, drops occasionally down few frames (+)

Note: The Playback was the same on W2K/XP/Win7 (32/64 Bit) with Intel, ATI, NVidia and Matrox Boards

Legend:
(+) -> Audio in Sync with Video
(-) -> Audio out of Sync with Video

Further, "VideoPad" from NCH rejects to load AVI's encoded with MagicYUV, regardless of the settings and FourCC's used.

You can find the Software here and test it yourself:
http://www.nchsoftware.com/videopad/index.html

Further, if the "UtVideo" Codec is also installed on W2K (SP4)/XP (SP3), then MagicYUV seem to "stutter" at Resolutions over 4K encoding/decoding Video. I noticed this with the UtVideo Codec Version 13.3.1 and believe it's the included DMO Module from UtVideo that causes Problems. When UtVideo is removed from the System, everything is back to normal with MagicYUV. Plus, MagicYUV co-exists fine with Lagarith and HuffYuv without any Problems.

Speaking of UtVideo/VirtualDub and "v210" 10 Bit: Both produce Videos that can not (!) be played on OptiBase, AJA/Kona or DVS I/O Boards. The only Codec so far is the Drastic Codec that fully conforms to the Industrial standard and Definition for 10 Bit. So, if you plan to develop in this "10 Bit Playground", chances are that it won't work with professional Equipment.

Example:
A Colleague of mine encoded 3 TB of Material with UtVideo's v210 Codec to save space and has to start all over again now because the Files can not be used on such Systems he made them for, even if the UtVideo Codec is installed on those Systems. On a DVS HD Station, only the first 3 Frames can be loaded, the rest is green without Sound and takes awfully long to load.

In VirtualDub, no Problems with Intel, ATI, NVidia and Matrox Boards and the "v210" Codec from UtVideo. also, the UtVideo v210 Codec is painfully slow for encoding with a super high CPU load on all Cores, nothing I can recommend at this point.

You can find "UtVideo" here:
http://umezawa.dyndns.info/archive/utvideo/

Still not solved:
MagicYUV RC4 still reports "RGB24" to other Codecs with VirtualDub (Version 1.6.19 - 1.10.4) no matter how the Settings are with the "Fast Re-compress" and "Normal Re-compress" Modes. HuffYuv and Lagarith use the Color-space they encoded (YV12/YUY2) the Video with.

Interlaced Mode with MagicYUV only takes the TFF (Top Field First) which is the same for all HD Modes, but in SD Resolutions for NTSC with BFF (Bottom Field First) the Image is distorted.

I work a lot with PAL, SECAM and NTSC Material from all over the World, only HuffYuv took the Fields correctly while UtVideo suffers the same Problem like MagicYUV with the TFF only and Lagarith does not handle Interlaced as well as HuffYuv.

To give you a brief Overview about Field Orders:
PAL SD/D1/HD -> Top Field First / Upper Field First
PAL SD DV -> Bottom Field First / Lower Field first
NTSC D1/HD -> Top Field First / Upper Field First
NTSC SD DV -> Bottom Field First / Lower Field First
SECAM -> same as PAL SD/D1/HD

I hope this was helpful to you. Take care :)

Cheers

Mick

[De-M-oN]

MagicYUV RC4 still reports "RGB24" to other Codecs

Sorry cant agree you with this

http://abload.de/img/unbenannt154rzuxg.png

[Mick]

Hi De-M-oN,
well, MagicYUV certainly behaves with "Top Field First" with my Typhoon Board switched to NTSC. When I select "BFF" everything is fine but MagicYUV encodes to "TFF". Your Screenshot shows "Parity: Bottom Field First", makes me wonder because the Filter for my Typhoon let's me choose the Field Order and sets it according to the Video standard, never failed me so far.

Most Material i get is the NTSC 4.43 BFF Format. Which one was yours ? NTSC M ? PAL N (PAL NTSC Playback) ?

Your Screenshot shows "YUY2", but why is RGB24 reported from MagicYUV to XviD and x264 in VirtualDub while HuffYuv reports with the same Settings YUY2 ? I really tried all the Settings in MagicYUV but still the Output is RGB24. And yes, under "Color Depth" I chosen for Output "Same as Input" which is YUY2. What are your Settings in VirtualDub ? Maybe I've overseen something.

Cheers

Mick

[De-M-oN]

My video is progressive. The information is given at 6th line : FieldBased (Seperated) Video: NO

My video is a game recording. I recorded with DXTory.

It is a simple progressive 30fps video.

[Ignus2]

Hi there,
i downloaded RC5 and 1.0 for my Win2K Pro System and the Configuration Tool for the Codecs simply Reports the Error Message "Function not supported" and quits.

That's possible. The minimum supported OS is Windows XP, the fact that it worked with anything below that was pure luck. Sorry.

The MagicYUV Codec in Versions RC5/1.0 is about 20 Frames slower (!) with a higher CPU (28-35 % more) load than the RC4 that caused no trouble with Win2K Pro (SP4). Sure, i could step up to another OS but i don't want that for several reasons, so let's leave that Discussion out right now.

There has been no change algorithm-wise between RC4 and RC5/1.0, so the cause is something else here.

Most of the time the encoded data is not what people think it is. So for example people think it's YUY2, but actually it is RGB. That could explain why it is slower (or why the file is bigger).
The next release (in the coming days) will include an icon in the notification area, which will show exactly what is being decompressed into what format and resolution etc.

Until then, to be sure that you really encoded YUY2, select YUV 4:2:2 for "Accepted colorspace" in the codec config dialog before encoding. That way the codec will reject everything that is not YUV 4:2:2 when encoding.

Further, "VideoPad" from NCH rejects to load AVI's encoded with MagicYUV, regardless of the settings and FourCC's used.

You can find the Software here and test it yourself:
http://www.nchsoftware.com/videopad/index.html

I tried the software, and it loads MagicYUV encoded files perfectly for me.

Further, if the "UtVideo" Codec is also installed on W2K (SP4)/XP (SP3), then MagicYUV seem to "stutter" at Resolutions over 4K encoding/decoding Video. I noticed this with the UtVideo Codec Version 13.3.1 and believe it's the included DMO Module from UtVideo that causes Problems. When UtVideo is removed from the System, everything is back to normal with MagicYUV. Plus, MagicYUV co-exists fine with Lagarith and HuffYuv without any Problems.

That can happen. I also noticed, that the UtVideo DMO sometimes gets inserted to do needless colorspace conversion or even encode to Ut and then back (!) for no reason.

Speaking of UtVideo/VirtualDub and "v210" 10 Bit: Both produce Videos that can not (!) be played on OptiBase, AJA/Kona or DVS I/O Boards. The only Codec so far is the Drastic Codec that fully conforms to the Industrial standard and Definition for 10 Bit. So, if you plan to develop in this "10 Bit Playground", chances are that it won't work with professional Equipment.

The raw v210 AVI produced by the latest (!) VirtualDub conforms to the v210 spec (it was buggy with earlier VDub though!).
I also had someone test my codec with v210, and it played back fine.
But we can discuss/test more about this if you want.

Still not solved:
MagicYUV RC4 still reports "RGB24" to other Codecs with VirtualDub (Version 1.6.19 - 1.10.4) no matter how the Settings are with the "Fast Re-compress" and "Normal Re-compress" Modes. HuffYuv and Lagarith use the Color-space they encoded (YV12/YUY2) the Video with.

MagicYUV does the same, it reports the colorspace that was encoded. If it always reports RGB (despite all conversion options disabled in the "Decompression Settings"), then the encoded data has to be RGB.


Interlaced Mode with MagicYUV only takes the TFF (Top Field First) which is the same for all HD Modes, but in SD Resolutions for NTSC with BFF (Bottom Field First) the Image is distorted.

Yes, the codec assumes TFF for now.
There is actually no way of knowing if the input is TFF or BFF, but I can include an option for the encoder settings.
What I would like to know is what to do on the decoder side? So for example if I encode BFF, should I also output BFF when decoding? Or always TFF, regardless of the what was encoded (so to swap the fields)? I would need some help with this.

Thanks for the comments. The next release will have the notification icon, so you can check the colorspace, until then use the "Accepted Colorspace" option. Also, if we can get to it, I can get the TFF/BFF be included as well.

Greets,
I.

[zerowalker]

I would like the Encoder to have that Option of choosing TFF or BFF (it's kinda a must, if you use that feature to play interlaced video).
How to do it, well either swap fields like you said or do it the normal way. Do what's most efficient as the end result is the same:)

Thanks

[ChiDragon]

Alright, here's a stupid question: why does an intra-only lossless codec care whether the material is TFF or BFF?

[zerowalker]

To be able to decode and deinterlace on playback, i think?

[Ignus2]

OMG. That would make for a great "lossless" codec... ;)

The only real use case I could think of is to be able to always output TFF (or always BFF) by knowing what kind the input was (and swap accordingly).

Greets,
I.

[Mick]

Hi Ignus2,
to answer some of your replies:

"VideoPad":
It chokes on Files larger than 4 GB with MagicYUV and saving a MagicYUV encoded Video from VideoPad back to MagicYUV does not work all the time. I have a older version here which causes a lot of Problems with Videos encoded with MagicYUV and maybe this is fixed now in the newer Version of VideoPad. But MagicYUV is not alone, UtVideo causes the same Problems with VideoPad while HuffYuv and Lagarith work fine. I do believe it's because of the missing Interpretation of MagicYUV and UtVideo in the FFMpeg Libraries that VideoPad uses, but, that's only a guess of mine at this point.

"v210" Issue:
I tried loading v210 Videos on a DVS and AJA System, no chance with a very long loading period until the Error Message came up. With the Drastic Codecs the Videos are loaded with a blink of an Eye on PC, Workstation or Broadcast Systems. The Reason is that "v210" is a YCbCr Format and VirtualDub (1.10.4) saves it with a RGB Header so the Decoder is initiated for RGB Conversation on the Output.

The default is to stay in YCbCr, not to convert up to RGB. The UtVideo v210 (UQY2) does the same thing like VirtualDub and even if UtVideo is installed, "v210" (UQY2) can not be used on real Workstations or Broadcast Systems. Sure, PC is fine, but Workstations or Broadcast Systems, no chance with the UtVideo/VirtualDub v210 Format. The Format definition for "v210" and other 10 Bit Formats do not use any YCbCr to RGB conversations.

But this is another Story and we should talk about this in a new Thread, okay ? For now the fact stays, both v210 don't work with YCbCr Boards while the 8 Bit (4:2:2) Formats from VirtualDub and UtVideo cause no Problems, only with 10 Bit "v210".

Interlaced:
The normal behavior is, that if you encoded with "TFF" then the Video is decoded the same way, "TFF" and vice versa. Even if all HD Formats for PAL/NTSC are now TFF, in SD Resolutions you end up with a garbled content because the Field Order is wrong. Including a Option here would really solve this Problem. The classic SD DV Format for PAL/NTSC is such a candidate, which is always "BFF".

I still don't know how, but HuffYuv encodes/decodes with the correct Field Order, no matter if it's PAL/SECAM or NTSC. I checked the settings again and my VCR's use BFF for NTSC and TFF for PAL/SECAM. NTSC is only using TFF for D1 and PAL D1 when selected. When I select DV for PAL/NTSC then both are "BFF", fully conforming to the standards.

But there is also a "evil" Trap: Some simple USB Devices always use "BFF" because they are shipped with the NTSC Standard where other "better" USB Devices change the Field Order according to the Standard used. In other Words, if the User does not know the Field Order of the Device, it ends up to a "Trial and Error" thing.

The second "evil" Trap is the VCR used for Playback. Some VCR's switch the Field Order if the norm changes, others don't. Good example are simple European VCR's with NTSC Playback. With those VCR's the Field Order is not changed and the NTSC Video is played back as "PAL 60", a mix of PAL and NTSC with "TFF". More professional VCR's from JVC, Panasonic or Sony switch the Field Order for NTSC to "BFF" and back to "TFF" for PAL/SECAM.

Same as my Multi-Format Typhoon Board and AVP Video Mixer, the Field Order changes to the right Order and uses NTSC 4.43, not PAL 60 or NTSC-M. Again, HuffYuv encodes the YUY2 Materials with the right Field Order without changing anything. If it's not in the Source code, then it must be my Typhoon Board that takes care of this and would explain, why MagicYUV Material is distorted with NTSC and BFF if it only encodes "TFF" like UtVideo. BTW, NTSC with BFF is also distorted with UtVideo so you're not alone with this Problem, okay ;)

The "RGB" Output:
I am sure that it's not RGB and really YUY2 so that can be excluded. Plus, HuffYuv and Lagarith decode in YUY2 to Xvid (1.3.2) and x264 with no Problem. AviSynth reports YUY2 with HuffYuv and Lagarith when the Captures are opened with a Script where MagicYUV reports RGB24 with AviSynth (2.5.8). I never use RGB, only YUY2 and occasionally I420 or YV12 to avoid any color conversation.

So, that's quite a bit of Information for you and i hope this was helpful to you. Anyway, MagicYUV is a great Codec and i never seen a perfect Take-Off from a new Codec and it's normal that it takes some time until everything is how it should be.

Take care and keep up that great work, okay ? :)

Cheers

Mick

@ChiDragon
It also costs Quality by skipping one Filed or blending (De-interlace) them if the Source is interlaced, even with Lossless Intra-Frame Codecs. As you can't play any Lossless encoded Material on a DVD/BluRay you need to encode them to the final Format which is, right, interlaced, at least 90 % of it and would mean that you would have to either go down to a progressive resolution or re-interlace the Material with a huge loss of quality.

In Mastering Studios the default Rule is to uphold the Quality:
If it's interlaced, leave it interlaced.
If you used a Lossless Codec, stay with it until the Master is saved.
When the Lossless Master is done, then (!) encode it to the final Format.

[Ignus2]

Thanks for the detailed answer.

First of all, it would be best if you could send me a sample MagicYUV-compressed file, which reports RGB despite being compressed YUY2 for me to take a look at it.

The same goes for the interlaced thing. Can you send me a raw uncompressed sample, and the same encoded with HuffYUV and MagicYUV to show the problem?

There really is no other way for me to fix the problem, as I cannot reproduce it.

Greets,
I.

[Mick]

Hi Ignus2,
you can reproduce it very easy: Open VirtualDub and make a Capture. Once with the Option "Swap Fields" disabled and one with the "Swap Field" Option enabled, of course with MagicYUV as a compressor.

I look for a short Sequence that i can upload here that is not critical because i transfer Material from Company's and Broadcast Stations around the World and is meant for Archives, not for being used for Samples. I have a look and upload a short Clip in the next Days, okay ? It will be with MagicYUV RC4, just to let you know.

BTW, why did you change the minimum OS that for the newer Versions of MagicYUV ? Was there a specail reason for it ?

I can do that with the 2 Versions, HuffYuv and MagicYuv, but then you would need the HuffYuv Version i use, which is the 2.1.1 CCESP Patch 0.2.5 I attached to this post. I don't know if the installer will work on your System. Plus, the versions of HuffYuv included in ffdshow work differently to this version.

Cheers

Mick

[Ignus2]

Hi Ignus2,
you can reproduce it very easy: Open VirtualDub and make a Capture. Once with the Option "Swap Fields" disabled and one with the "Swap Field" Option enabled, of course with MagicYUV as a compressor.

Well, I have no capture card, so I cannot make a capture :)


I look for a short Sequence that i can upload here that is not critical because i transfer Material from Company's and Broadcast Stations around the World and is meant for Archives, not for being used for Samples. I have a look and upload a short Clip in the next Days, okay ? It will be with MagicYUV RC4, just to let you know.

I'm most interested in the YUY2 reported as RGB situation. You don't have any YUY2 material on your machine by chance?

BTW, why did you change the minimum OS that for the newer Versions of MagicYUV ? Was there a specail reason for it ?

Who said I changed? It was WinXP from the beginning (take a look at the homepage, I never mentioned Win2K) :)
Again, the fact that it worked with anything below WinXP was pure chance!

Greets,
I.

[Mick]

Hi Ignus2,
no Capture Card ? No Problem :) Open VirtualDub (1.10.4) and select under "Tools" the "Create Test Video" Option and scroll down the List to the one with "Top Field First" and afterwards the other one for "Bottom Field First".

Use MagicYUV as a Compressor and under "Color Depth" choose "4:2:2 YUY2" (YUVY) Color space instead of "Same as Input". This will give you a short Test clip which does the same like a Capture. As i wrote before, I look up some YUY2 Material and upload it here soon for you. You see, the Material I have is not meant for the Public at this point and is owned by the Company's or Broadcasters I do the Transfer for. Plus, I would violate my Contracts if I would load up something that's not mine.

Again, when I have a good Moment with Time, then I either make a short Clip or look up something that's not causing me any trouble about Copyrights, okay ? :)

Not to forget the Mystery: Why does HuffYuv and Lagarith report "YUY2" in the "Fast recompress" and "Normal recompress" Modes in VirtualDub to XviD (1.3.2) while MagicYUV reports "RGB24" to XviD with the same settings ? I tried Lagarith with YV12 and XviD compressed to YV12, just like the YUY2 Videos before.

And, if i was "lucky" with the RC4 on Win2K, which is also really meant for XP, could that be the Reason why MagicYUV reports RGB24 instead of YUY2 ? I doubt it from my point of view at this time because the RC4 Release never caused any Errors under Win2K (SP4) and worked fine.

Cheers

Mick

[Ignus2]

OK, the sooner you send the files, the sooner I can make the new release :)


Not to forget the Mystery: Why does HuffYuv and Lagarith report "YUY2" in the "Fast recompress" and "Normal recompress" Modes in VirtualDub to XviD (1.3.2) while MagicYUV reports "RGB24" to XviD with the same settings ?

I need a MagicYUV-compressed file which exhibits this behavior for you, otherwise I cannot answer this question, as it works correctly for me (and others as well), ie. reports the originally encoded colorspace correctly (YUY2 in case it was YUY2, etc.).


And, if i was "lucky" with the RC4 on Win2K, which is also really meant for XP, could that be the Reason why MagicYUV reports RGB24 instead of YUY2 ?

No. There has been no change regarding the algorithms and colorspace reporting between RC4 and RC5/1.0.

Greets,
I.

[kolak]

Hi Ignus2,
to answer some of your replies:


"v210" Issue:
I tried loading v210 Videos on a DVS and AJA System, no chance with a very long loading period until the Error Message came up. With the Drastic Codecs the Videos are loaded with a blink of an Eye on PC, Workstation or Broadcast Systems. The Reason is that "v210" is a YCbCr Format and VirtualDub (1.10.4) saves it with a RGB Header so the Decoder is initiated for RGB Conversation on the Output.

The default is to stay in YCbCr, not to convert up to RGB. The UtVideo v210 (UQY2) does the same thing like VirtualDub and even if UtVideo is installed, "v210" (UQY2) can not be used on real Workstations or Broadcast Systems. Sure, PC is fine, but Workstations or Broadcast Systems, no chance with the UtVideo/VirtualDub v210 Format. The Format definition for "v210" and other 10 Bit Formats do not use any YCbCr to RGB conversations.

But this is another Story and we should talk about this in a new Thread, okay ? For now the fact stays, both v210 don't work with YCbCr Boards while the 8 Bit (4:2:2) Formats from VirtualDub and UtVideo cause no Problems, only with 10 Bit "v210".



MagicYUV 10bit output was tested with many software- AE, Premiere, Edius, Nuke etc and it worked fine.

MagicYUV decoder output has been also tested directly with BM diretchsow filters and this also worked fine for v210 (also r210 format).

If you use Vdub to save v210 than set Color Depth (input/ output) to 10bit uncompressed and maybe later change fourcc so it's the same as AJA codec sets.

Aja control room won't load files which are not supported and magicyuv is definitely one of them. If you have AJA directshow filters installed than you can build a chain in graphedit and play 10bit magicyuv files (v210 or R10k) diretcly to AJA, like I have done to BM card. This should work fine.

If you want to capture directly to magicyuv at 10bit (422 or 444) with BM or AJA than this maybe possible soon :)
Please note that not many software can read 10bit from a codec through vfw or directshow, so 10bit pipe needs to be done through SDK or other ways.

You can always use vapoursynth to read magicyuv files and present them as fake uncompressed avi files through vsfs and these should work with eg. AJA control room.

[ChiDragon]

@ChiDragon
It also costs Quality by skipping one Filed or blending (De-interlace) them if the Source is interlaced, even with Lossless Intra-Frame Codecs.
I never mentioned deinterlacing. I was asking how the codec could care about field order.

My understanding is that the Interlaced modes of Huffyuv, Ut Video, and MagicYUV function like SeparateFields() -> compress / decompress -> Weave(). In that case, knowing the field order would be important if inter-frame compression were done. But since it isn't, it makes no difference if the assumed field order is wrong. All that matters is that the same assumption has to be made throughout.

Again, HuffYuv encodes the YUY2 Materials with the right Field Order without changing anything. If it's not in the Source code, then it must be my Typhoon Board that takes care of this
AVI has no way of flagging field order, so I think it must be your board.

[kolak]

AVI has no way of flagging field order, so I think it must be your board.

Yes, at leas no standard way.
Some manufactures do implement it using private AVI headers, but this is not a "standard" way. For example GV Edius software uses its own flagging and it works as long as you stay in Edius software (same for timecode).

[Mick]

@Kolak
thanks for the Information about v210 and just found out today that the AJA/Kona QuickTime for Windows Codecs decode v210 Videos from VirtualDub. The AJA/Kona QuickTime for Windows Codecs transform the YCbCr Source to RGB24 and save in RGB24 with a QuickTime Pro version.

I use the Drastic YCbCr Codecs for VfW and QuickTime. These Codecs stay in YCbCr and don't transform to RGB24 just like the Optibase and VideoPump Codecs, which is the normal behavior for YCbCr Systems. The Videos from VirtualDub (1.10.4) can't be played with the Drastic or OptiBase Codecs, not even with QuickTime. I don't know the Blackmagic Codecs and after reading your Reply, i do believe they work the same Way, YCbCr up to RGB, just like the AJA/Kona QuickTime Codecs do.

On the DVS HD Station and OptiBase Systems, which are pure YCbCr Systems, such Videos can't be used that where saved with the VirtualDub v210 Format while the Drastic and OptiBase v210 encoded Videos load instantly, even on the older AJA/Kona Systems.

So, one obviously has to make a difference here between Systems that only use YCbCr and others that always transform to RGB. I only work with YCbCr and pass that Material back to the Clients, unless they ordered a MPEG-2/MPEG-4 or Lossless Format after the transfer.

And the UtVideo v210 (UQY2) Codec does not work either with a installed UtVideo Codec Suite, only the other 8 Bit Formats from UtVideo. A Colleague of mine tested this for me on his DVS System and leaves the Question to me: Does one "really" need 10 Bit Codecs if the Format is going so many different Directions ? One Side says "YCbCr stays YCbCr" while the other Side says "YCbCr transforms to RGB".

I mean, one of the most used Formats in Mastering Labs is I420 12 Bit RAW (4:2:0) only because MPEG-2 and H264 see this as the default Standard and is most recommended to prevent unnecessary Color Transitions. Many Cameras use the same Color Space for MP4 and MTS Recordings and Color Transitions are not Lossless and can cost a lot of Image Quality.

If a Client of mine wants a 4:2:2 or 4:2:0 Format, then i stay with it until the end, means from Capture/Transfer over Editing to rendering the Final after Grading. If the Client wants a Lossless Format, HuffYuv for Example, then i stay with HuffYuv until the end of the Transfer, same with other Formats. MPEG-2/MPEG-4 comes at the very end, when the Master is finished and only if the Client wants a MPEG-2/MPEG-4 encoded Version.

This is one of the Basic Rules to uphold the Quality:
- Never change the Format until you're done.
- Never change the Codec.
- Once compressed, always compressed.
- Once Interlaced, always Interlaced.

@ChiDragon
You're right, it's my Typhoon Board that sets the correct Field Order when i change the TV Norm and HuffYuv encodes it that Way. But, why does MagicYUV have trouble with it ? "Ignus2" wrote, that MagicYUV is only conforming to HD Field Orders and they are all "Top Field First" for PAL and NTSC now. I still use Win2K Pro (SP4) and the last MagicYUV version that worked for me is RC4.

For me that is still the Riddle, why the captures with HuffYuv are clean and have the right Field Order while MagicYUV gives me a distorted and noisy Image with "Bottom Field First" Field Orders with NTSC 4.43 Formats. Don't get me wrong, i really like MagicYUV, great Codec, stable, reliable and fast, but at lower SD NTSC Resolutions not usable, only for Progressive Content, but no Client demanded Progressive Transfers so far.

I have to deal with Material from all over the World, all kinds of TV Norms where not even NTSC is a reliable "Bottom Field First" (DV/D1) Candidate and if a Client demands a "Lossless" Transfer, then I give them the Choice between HuffYuv and Lagarith. Those Clients either take HuffYuv or RAW YCbCr, Lagarith only occasionally. I would like to offer them MagicYUV too, but I simply can't at this point because the Image is distorted, even if those Videos are played back on other Systems with "Normal" Video Boards like Intel, ATI, NVidia and Matrox on newer Windows Versions.

So, somehow there must be a difference between HuffYuv and MagicYUV when it comes to Interlaced Material and how it's encoded internally because UtVideo suffers the same Symptoms and Problems like MagicYUV. Sure, i could load those captures with MagicYUV in VirtualDub and reverse the Field Order and encode it again with MagicYUV but, this makes it even worse, tried that already and did not solve the Problem.

@Ignus2
A Friend of mine tried MagicYUV (Final 1.0) with Windows 8.1 64 Bit and a USB Grabber who wants to Capture some of his old VHS Family Tapes, all NTSC. Result was, that he had the same Problem with MagicYUV because of the "Bottom Field First" from his VCR (Hitachi with S-Video) and VirtualDub (1.10.4). The Image was distorted and came clean when he used the De-Interlace Filter in VirtualDub during the Capture.

He gave me these Specs from the Manual for his USB Grabber over the Phone:
Resolution: 720x480 (Maximum)
Frame Rate: 29.970 Fps
Aspect Ratio: 4:3
Field Order: Bottom Field First
Color Space: YUY2 or I420 (Other not available)
Composite Video: Yes
S-Video: Yes
Audio: 48.000 kHz, 16 Bit, Stereo, PCM (Maximum)

His Idea was to capture the Tapes with a Lossless Codec to encode those later, after editing them, to a nice DVD for the Family. After I told him that Progressive Material works only on BluRays in HD well, he made the first captures in RAW YUY2, later in I420 upon my recommendation for a MPEG-2 DVD as he does not want a BluRay. HuffYuv still has no reliable Installer for 64 Bit Systems, Lagarith did not work for some Reason on his System, and ffDShow he found to "confusing" to use. Knowing he would have the same Problems with UtVideo, i did not mention UtVideo because it would have caused the same Problems.

So, i do believe it's a good Idea to include a Option for the Field Order in MagicYUV because there are many like my Friend in Florida out there that want to save the Family Moments for the digital Future. And at this Point I have a "Bullet Proof" Idea:

Why don't you include Profiles for SD Resolutions with the correct Field Order ? For Example, SD DV at 720x480 (NTSC) and DV at 720x576 (PAL), both with a fixed "Bottom Field First" ? All the User needs to do is to select the SD Resolution he wants and the correct Field Order is automatically set. HD is always "Top Field First", it's just some SD Resolutions that cause Problems when the Source is Interlaced and the User wants to save it Interlaced.

Just a Idea and in case you want to use it: It's free, okay ? ;)

Cheers

Mick

[Ignus2]

From the codec point of view, TFF or BFF doesn't make any difference at all. If we don't take in-codec conversions into account (true lossless mode), then whatever came into the codec as input when compressing, the output on decompressing will be EXACTLY the same.

That is why I don't understand what you experience by certain codecs getting the field order right, others wrong. None of the codecs have any way to know about the field order in VFW. But even if they did, it would make no difference at all, as they would still output exactly what they got (I refer to lossless codecs here).

Again, samples would help here, raw yuy2 material, compressed with both huffyuv, and magicyuv, and show where it is wrong or distorted.
And while we are at samples, don't forget the "always RGB reported" sample too.

I tried the VDub test videos, both TFF/BFF encoded with huffyuv(ccesp)/magicyuv/utvideo, then used avisynth to Compare() them, and the decompressed output from all of them were bit-equal (for the given TFF or BFF case).

Greets,
I.

[kolak]

The problem is not raw data, but container headers.
Codec has nothing to do with it, but application which saves avi file.
As it was said, avi has no standard way of storing field order, so it's all pure luck.

Maybe field order could be stored in 'codec data' same as matrix and than maybe it could work, thought I am not sure as reading software reads container data, not deep codec data.
It's all down to old avi container which is quite limited in this area ( same as it does not store timecode).

[kolak]

@Kolak
thanks for the Information about v210 and just found out today that the AJA/Kona QuickTime for Windows Codecs decode v210 Videos from VirtualDub. The AJA/Kona QuickTime for Windows Codecs transform the YCbCr Source to RGB24 and save in RGB24 with a QuickTime Pro version.

I use the Drastic YCbCr Codecs for VfW and QuickTime. These Codecs stay in YCbCr and don't transform to RGB24 just like the Optibase and VideoPump Codecs, which is the normal behavior for YCbCr Systems. The Videos from VirtualDub (1.10.4) can't be played with the Drastic or OptiBase Codecs, not even with QuickTime. I don't know the Blackmagic Codecs and after reading your Reply, i do believe they work the same Way, YCbCr up to RGB, just like the AJA/Kona QuickTime Codecs do.

On the DVS HD Station and OptiBase Systems, which are pure YCbCr Systems, such Videos can't be used that where saved with the VirtualDub v210 Format while the Drastic and OptiBase v210 encoded Videos load instantly, even on the older AJA/Kona Systems.

So, one obviously has to make a difference here between Systems that only use YCbCr and others that always transform to RGB. I only work with YCbCr and pass that Material back to the Clients, unless they ordered a MPEG-2/MPEG-4 or Lossless Format after the transfer.

And the UtVideo v210 (UQY2) Codec does not work either with a installed UtVideo Codec Suite, only the other 8 Bit Formats from UtVideo. A Colleague of mine tested this for me on his DVS System and leaves the Question to me: Does one "really" need 10 Bit Codecs if the Format is going so many different Directions ? One Side says "YCbCr stays YCbCr" while the other Side says "YCbCr transforms to RGB".

I mean, one of the most used Formats in Mastering Labs is I420 12 Bit RAW (4:2:0) only because MPEG-2 and H264 see this as the default Standard and is most recommended to prevent unnecessary Color Transitions. Many Cameras use the same Color Space for MP4 and MTS Recordings and Color Transitions are not Lossless and can cost a lot of Image Quality.

If a Client of mine wants a 4:2:2 or 4:2:0 Format, then i stay with it until the end, means from Capture/Transfer over Editing to rendering the Final after Grading. If the Client wants a Lossless Format, HuffYuv for Example, then i stay with HuffYuv until the end of the Transfer, same with other Formats. MPEG-2/MPEG-4 comes at the very end, when the Master is finished and only if the Client wants a MPEG-2/MPEG-4 encoded Version.

This is one of the Basic Rules to uphold the Quality:
- Never change the Format until you're done.
- Never change the Codec.
- Once compressed, always compressed.
- Once Interlaced, always Interlaced.


Why don't you include Profiles for SD Resolutions with the correct Field Order ? For Example, SD DV at 720x480 (NTSC) and DV at 720x576 (PAL), both with a fixed "Bottom Field First" ? All the User needs to do is to select the SD Resolution he wants and the correct Field Order is automatically set. HD is always "Top Field First", it's just some SD Resolutions that cause Problems when the Source is Interlaced and the User wants to save it Interlaced.

Just a Idea and in case you want to use it: It's free, okay ? ;)

Cheers

Mick

Well- if you want to use any v210 codec just to have data decoded to 8bit (YUYV or RGB) than there is no point to have this 10bit in first place. Idea of v210 is that reading application plugs in into raw 10bit data, so there is no need for codec. BM, AJA, Drastic codecs exist only to enable people to play v210 files in "home" users players. For professional use you want applications which read this 10bit data directly/natively.
And yes, if you want this to be decodec to 8bit, than use codec which stays in YUV- going to RGB is bad.

MagicYUV 10bit data will be decoded back to v210 (or r210/R10k) depending if the source was 10bit 422 or 444. There is no colorspace conversion for 10bit implemented as far as I know, so you will never get 10bit decoded to RGB data (at least not now).

Your idea about presets for SD with predefined field order probably won't work.
When you drag avi file to eg. Premiere it looks for header informations (container info not actual codec) which it does understand and one of them is field order. Adobe uses its own flagging (different then eg. GrassValey) for avis, so when you use avi files from some other software than Premiere this info does not exist. Premiere will assume some field order (maybe the one which project is set to). You can overwrite it in file properties and this is what you should always check- if source file is interpreted properly. Avis are poor with metadata, as there is no standard for storing all these additional information eg. filed order, timecode etc. This information is a container info set by the writing application, not actual codec data. FFmpeg was/is also famous for not flagging ProRes MOV files as interlaced (and some other metadata), but yet again this is not a ProRes codec problem, but ffmpeg (MOV headers). Ffmbc, which is used by BBC has this fixed :)

Ingus2 can add this info into "codec data" based on user choice, but I'm not sure if this will help.

Solution is simple (well not for magicyuv)- use MOV or MXF container which have standard ways of storing all metadata, so all software can understand and read it properly, This is the difference between broadcast solutions and "home" solutions. Broadcast app will rather always set these metadata and that's why 75% of them use MOV or MXF as containers. AVI is not used much at all because of poor metadata support and problematic support for higher bit depths.

[Mick]

@Ignus2
i did not forget about the Samples, just very busy at this point, that's why i haven't done it yet. Very interesting about your comparison of HuffYuv/MagicYUV and UtVideo. Well, if they are "Bit Identical", even with Interlaced Material, where could be the Problem ? My Typhoon uses a "Strict Header" for the AVI's and switches the Drastic Codecs from Top Field to Bottom Field for NTSC Material and back to Top Field for PAL/SECAM. Could that have something to do with it why MagicYUV and UtVideo cause such Problems ?

The native Windows Codecs for I420, IYUV, UYVY, YUY2 and YVYU use a DE-interlacing Routine to Weave both Fields together on Playback because they where designed for PC use. The Drastic Codecs don't and stay Interlaced, what i can control on my connected Sony Multi-Norm Television Set and my other Monitors.

@Kolak
The Drastic Codecs offer v210, 012v and auv2 for 10 Bit YCbCr for Compression and Decompression along with all the other 8 Bit Formats for VfW and QuickTime for Windows, means, i can setup my Board for 10 Bit and capture or transfer this way without any transcoding. But so far no Client wanted 10 Bit, only 8 Bit 4:2:2 or 8 Bit 4:2.0 in YCbCr for the Transfer or Lossless HuffYuv/Lagarith and very seldom Motion JPEG 2000. I don't know the Black Magic Codecs and never tried them, only the ones from AJA for QuickTime which transcode under QuickTime for Windows YCbCr to RGB24 and from that what i've been reading the Black Magic do the same thing.

About my "SD Presets" Idea:
Makes Sense what you wrote and was just a Idea. If the Lossless Codecs are Bit Identical, no matter if TFF or BFF is used, then it makes no sense in offering such Option. As i wrote earlier, my Typhoon uses a "Strict Header" for the AVI's and i can Imagine that has something to do with it. My Aist/Cinegy Moviepack Pro works the same Way like Adobe, at least from my Time back with Premiere 6.5, and correctly reads the Field Order from Transfers or Captures from my Board.

About AVI/MOV/MXF Containers:
I only have 3 Clients that demand YCbCr MOV Files, all the others are always asking for AVI, believe it or not. We are talking about Company's and TV Stations from around the World, big and small ones, not any private "John or Jane Doe" in the Neighborhood. Further, almost all Clients want the Captures or Transfers on Hard Disk in YCbCr or Lossless. I even have some, that wanted the Transfers in 2GB Slices but no one ever wanted MXF, makes me wonder.

Anyway, up to this Day, never a Reclamation or complain from any of my Clients because they tell me what and how they want it and that's what they get back along with the Material they send me via Parcel Service.

I know that AVI is not the "Ideal" Container Format but the Clients demand for AVI's is still very high and plays the major Role. My best guess at this point is, that they do the Rest themselves when i send them the Hard Drives back. One thing they all have in common: They want the best Quality, either YCbCr or Lossless. Going by that, I do believe that the AVI Container is not so easily replaced by better ones in the near Future.

Cheers

Mick

[kolak]

Sorry, but you are the first person who talks about avis used in broadcast.
I don't mind avi, as long as headers info is passed between apps.
MOV has its own problems also ( some very annoying), where MXF can be over complicated.
When I worked at DVD/BD authoring house my workflow was setup over avisynth and avis, but it was very different compared to other places. Every delivery related to broadcast was never done with avi files. Now I work for big VFX house and it's the same- no one asks about avi as delivery option. It's all done as DPX, v210 Mov, ProRes, AVC-I 100 or legacy SD 50mbit mpeg2.

V210 and 012V is the same thing. It's about little v big endian coding.

[rec]

Hello Ignus2,

I tested MagicYUV and it's great! I love it. I do have a request, though. Files encoded with MagicYUV don't show thumbnails in Windows. Is there any solution for this problem? Suggestion: Some third party thumbnail generators work via FFMPEG. Have you considered getting the FFMPEG people to incorporate MagivYUV? I know they did it with UT.

Thanks again for this great codec.

[zerowalker]

He is at some point from my understanding. But it's going to be some kind of closed source release that they can work with.

[Mick]

@Kolak
i can only tell you what my Clients demand when they send me Material along with the Paperwork (Technical Riders) and that's AVI. Maybe it's because it's mostly SD Material from Tapes that I get.

@Ignus2
Well, you're ready ? Here are the Test clips you wanted with kind permission of Polygram Video for private use only (!). Captured from a Sony VCR with VirtualDub (1.10.4), no Filters, straight once to HuffYuv and again with MagicYUV, means, no Transfer from HuffYuv to MagicYUV for example. In VirtualDub i cut out the same Sequence for both Videos after the Capture and used the "Direct Stream Copy" for Video and compressed the Audio from PCM to MP3. I also included Screen Shots of my Settings for HuffYuv and MagicYUV.

Technical Rider:
Color Space: YUY2
Format: NTSC 4.43
Resolution: 720 x 480
Field Order: Bottom Field First / Lower Field First (Even Field)
Pixel Type: Square
Frame Rate: 29.970 Fps
Audio: 48.000 kHz 16 Bit Stereo (MP3 Compression 192 Kb/Sec)

You have to uncompress the Videos with 7Zip (9.20). The HuffYuv Video is called "HfyuTest.7z" (HfyuTest.avi) and the MagicYUV Video is called "MagyTest.7z" (MagyTest.avi).

Further, to make sure the Drastic YCbCr Codecs are not causing this Problem, i did the same Capture on my Laptop with a PCMIA Capture Card using the native Windows Codecs. The Result was the same so the Drastic Codecs are fine on my Workstation and don't cause this Problem with MagicYUV.

I hope this was helpful to you and now you can see it for yourself what i mean. So, it's obvious that the Field Order does matter, even with Lossless Codecs. ;)

Cheers

Mick

[Mick]

@Ignus
I tried to uploading the two Videos but Doom kicks me out every time and cancels the Upload. I have to look for a File-sharing Server and link them from there to Doom for Download. Both Files are less than 200 MB each. Sorry for that but could not find a Solution for this Problem now.

Cheers

Mick

Correction: Found a Solution. You can find the Download Link in my last Post :)

[kolak]

I hope this was helpful to you and now you can see it for yourself what i mean. So, it's obvious that the Field Order does matter, even with Lossless Codecs. ;)

Cheers

Mick

It doesn't. As long as pixel format is the same as source decoded video will be 100% the same. It can go wrong when your video is stored as yuv, but you want rgb output, but I assume this is not the case here.
Field order is just a metadata. Lossless codec works like zip or rar. Did you care about field order when you zipped your files?

Your issue is not related strictly to codec itself, but more to container and whole workflow. Maybe there is a way of fixing it, but it won't be related to changes in actual encoding algorithm.
Here is atest for you.
Get some interlaced source file, compress to magicyuv keeping original pixel format. Than load both file to eg. Premiere ( we assume codec sends original pixel format), interpreate field order for both and apply difference filter. Do you see any difference?
Yet again, your issue are headers/metadata. It's a seperate issue, but maybe Ingus2 can do something to improve it.

[Mick]

@Kolak
I mainly work with Interlaced Content for my Clients, but you can see this now yourself, just download the Videos then you see what i mean :) Anyway, normally this should not matter, but somehow it does with MagicYUV. All Settings for Capturing and saving the two Videos where exactly the same. Plus, why should I write something that is not true ? I'm just trying to help the Author of MagicYUV and it's obvious that there is a Problem when it comes to NTSC with BFF instead of TFF. And yes, My NLE triggers the right Field Order with HuffYuv but not with MagicYUV and when i set the Field Order manually for MagicYUV, then the Image is still distorted and Jitters on Playback. I get the same distortion and Jitter if i set HuffYuv to the wrong field order and no, i don't want RGB when i work with YCbCr. :) Thanks anyway for your informative Reply's ;)

@Ignus2
Here is the Link to download the 7Zip packed AVI's that I mentioned earlier. If you can find the Problem or the cause, let me know, okay ?

->Wetransfer Link expired<-

Cheers

Mick

P.S.: Just checked the Link and worked. Please note that "Wetransfer" will remove the Videos from the Server in the next 7 Days from Today on, so they won't be available forever.

[kolak]

Do I have to login?
There are no files there.
Put it on https://www.wetransfer.com and share download link.

So you can't set field order manually to have proper playback?

Maybe issue is with the way how data is passed to codec or eg. specific frame size.

Does the data match uncompressed one? If it does than it's some even different issue.

You can also compare them in avisynth:

a=avisource(uncompressed.avi)
b=avisource(magicyuv.avi)
compare(a,b)

Pixel format have to be the same for both files. PSNR should be something like 114dB if they are identical.

[Sparktank]

P.S.: Just checked the Link and worked.

Not that I'm interested in checking the files out, just the link.

It shows up blank on my end.
It probably works for you since you're signed into your own account.

What I see. (click to open in new tab in full size)
http://i.imgur.com/8gWK95Vs.png (http://imgur.com/8gWK95V)

[Mick]

@Kolak
No, you don't have to Login :) Yes, i can set the Field Order manually in my NLE, for Import, Capture, Playback and Export. I made the Captures with VirtualDub (1.10.4) directly from my VCR, not with my NLE without any Filters or other modifications. Many thanks for the Hint with "Wetransfer" :) I uploaded the Files there.

@Sparktank
Sorry, Ge.tt somehow is overloaded at the Moment and a new Link through "Wetransfer" is available now :) Plus, it's no Problem if you want to check the Files out.

@Ignus
The new Link for the Videos:
->Download Link expired<-

Both Files are packed with 7Zip (9.20) so you need to unpack them first befor you can watch them. The Download Link is good for 7 Days from Today on and then deleted according to "Wetransfer", so download them soon :) The Download contains both Videos, about 320 MB total.

Cheers

Mick

[ChiDragon]

Well, I see the problem in your sample. But if I recompress your Huffyuv file with MagicYUV in Interlaced mode, there is no difference between them. Win2K issue?

Files encoded with MagicYUV don't show thumbnails in Windows.
Mine do on Win7 x64 (and Huffyuv files don't, for me).

[kolak]

There are some capture problems with your files. Are they captured from DV or digibeta? They are 480, shouldn't they be 486?

One is upper field, other is lower, but this is not the real problem.
If you interpret huffyuv as lower and magicyuv as upper than fields are aligned, but there is 2 pixels vertical shift. Even after this shift is adjusted videos still don't match 100%. It there are cropping going on at some point in your capture chain? Is it a crop from 486?

At the moment answer is (if huffyuv matches DV or uncompressed capture): magicyuv does not work with your capture solution.
How was it captured? Why video is shifted vertically? Magicyuv will losslessy compress whatever it gets on input, so something is happening before video hits codec.

If you feed magicyuv with correct input data than it won't do anything to it and in this case filed order is just a metadata and can be manually changed.

I don't see any field info stored in neither of them and when I drag files to Edius than they are both interpreted as Lower Field, as per projects settings. Because Edius does not see any info about field order which it can understand it applies order set in project settings, so both files behave exactly the same.

There may be soon professional capture solution which will work with BM, AJA and some other cards with direct capture to magicyuv (including 10bit). It won't be based on vfw or diretchsow, as these are not very reliable technologies and not used in pro solutions.

I don't know what to suggest- try other way of capturing when using magicyuv? Fact that huffyuv works fine (if it really does) does not mean solution is reliable. Your problem does not really prove any problem with magicyuv codec itself, but it does not mean that there is no problem at all neither.

Ingus2: what does happen if magicyuv get file with eg.483 height and it's in YUY2 mode? Is it passed as is?

ps. I can't register 32bit huffyuv codec (64bit is fine), anyone knows why?

[Ignus2]

Thanks for the videos. I would have needed raw uncompressed YUY2 too as reference. Can you capture that as well?

This way I can only tell, that the input for the two codecs was different. The codec compresses what it gets, and neither huffyuv, nor magicyuv does any shifting around to the fields by default.

So this way I can only say, that most certainly the two cases are not equal, the codecs get different data. The question really is why?

Also: I looked at the fields with VirtualDub deinterlace filter and used "Duplicate fields" and "Double Framerate TFF/BFF" then single stepped and verified the following:
- The huffyuv coded sample is BFF and spatially correct
- The magicyuv coded sample is indeed TFF (the top field is really the first TEMPORALLY!) but spatially incorrect, the fields are swapped.

I'll do some more analysis of them in the evening.

@kolak: 483 height YUY2 passed as-is, just like anything else. If interlaced is ticked however, it is rejected (cannot compress). Only even height material is accepted as interlaced.

Greets,
I.

P.S.: @Mick: Some questions out of interest: Why do you have RGBA compression enabled and decompression as YUV 4:4:4 all enabled? Do you have special need for these? As normally turning these on just cause trouble.

[kolak]

Field order swapping may be related to specific capture device- huffyuv has a special setting which allows to swap fields.

There is clearly some issues between capture hardware/software and codec. That's why for pro use you need verified and more closed solutions which don't break because, eg you have installed ffdshow filter :)

[Mick]

@ChiDragon
The HuffYuv Capture is how it comes from my VCR's, BFF for NTSC, that's why MagicYUV compresses it fine but also makes me wonder. Why does it not work that Way with a direct Capture to MagicYUV or UtVideo without being distorted ?

The MagicYUV Capture is TFF on Playback instead of BFF and my VCR's use BFF for NTSC 4.43 and from what I understand, TFF is the default Field Order for MagicYUV and UtVideo because all Interlaced HD Material is now TFF, no matter what TV Norm. Somehow HuffYuv encodes this different to MagicYUV and UtVideo, how, I don't know at this point.

It can't be a Win2K Issue because the same thing happened with XP (SP3, 32 Bit) and Win7 (64 Bit). My Friend in Florida reported the same Problem to me over the Phone when he asked me for my advise, also NTSC from a Hitachi VCR with S-Video and changing the Input from S-Video to Composite did not change anything either with his KWorld USB Grabber.

@Kolak
I used 720 x 480 without Scaling or Cropping conforming to the NTSC DV Standard except instead of "Non Square" i used "Square" Pixel for the Capture because "Non Square" would only be Important for a straight DV Capture with Firewire (IEEE 1394 or iLink). D1 NTSC 720 x 486 I only use with my Capture Module from my NLE and only if demanded. Plus, some 486 Material can be TFF instead of BFF, the Aurora Igniter D1 Board is such a Candidate that uses TFF for D1 NTSC.

The Material was Captured from a VCR (S-VHS) and S-Video. I tried my Sony's, JVC's and Panasonic's, all the same. Then i used a PCMIA Capture Card and a older USB Grabber on my Laptop, just to make sure that it's not my Typhoon Board. Further, i used the default native Windows Codecs and VirtualDub (1.10.4) to exclude that the Drastic YCbCr Codecs might be responsible for this and I'm glad to say they're not.

Using the Composite Input did not make a difference, except a lower Quality. Again, this was a straight Capture, twice the same, directly once to HuffYuv and once again to MagicYUV, no Filters, straight from the VCR's to the Devices to the Codecs, no Mixing Console. I used the Direct Show Drivers for the Devices and checked it back by using the WDM Capture Interface in VirtualDub, no difference, same result. Plus i tried the same Setup with the UtVideo ULY2 Codec (13.3.1) and the captures where the same like MagicYUV, distorted with Jitter.

I used the same Settings for both Codecs in VirtualDub and after both captures, i opened them in VirtualDub, selected the same Sequence in both captures and saved them via "Direct Stream Copy" from VirtualDub to the final Files i Uploaded to "Wetransfer". BTW: Thanks for the helpful Tip ;) Only the Audio was compressed from PCM to MP3 to save some Space, the rest is untouched and is really as it came from the Capture from the VCR for both Files. "ffdshow", "Matroska", "Haali", "Morgan" or other additional Filters are not installed on my Systems, so this can be excluded as well ;)

@Ignus
I left the Settings for MagicYUV to their defaults, except "Interlaced" and "RGBA". Disabling the "YUV 4:4:4" and the "RGBA" Options never changed anything, never caused any Problems when left on and when i load the Captures or Transfers in my NLE, then i often use Titling that uses the Alpha Channel for Blending, would not work without this Option enabled and always worked with HuffYuv.

I can upload some pure YUY2 but I would have to shorten it a bit and leave the Audio out, if you don't mind, to keep it "downloadable" to a reasonable Size. Anyway, i make it available again over "Wetransfer" soon, not today because last Night was long enough :)

So, from my Point of View i can exclude:
- My Typhoon Board and Workstation because it was the same on my Laptop with Intel Chipset and Graphics
- The Drastic YCbCr Codecs because i used the default native Windows Codecs from DirectX9 (latest Version 904)
- My Aist/Cinegy MoviePack Pro because I did not use it and used VitrualDub for capturing/saving the Samples
- The Direct Show and WDM Drivers, same Result
- My VCR's because the Image did not change by changing them
- My PCMIA or other USB Grabbers, same Image
- Even with my old Dazzle DVC100, same thing
- Changing the Resolution from 720 x 480 down to 704 x 480 or 640 x 480 had also no effect, same thing
- Same Result with other Devices from AFA, Empia, Typhoon, QSonic, Logitech and SynTech

I really hope this was helpful to you and could you please notify the Author of "UtVideo" that his Codec has the exact same Problem like MagicYUV ? Believe me, I tested really hard to exclude everything that could cause this Problem and at this point, i really don't know what it could be. All I know is, that HuffYuv does it right, why and how I don't know.

Again, as long as the Field Order is TFF for PAL/SECAM and Interlaced HD Resolutions, MagicYUV and UtVideo work fine, just like HuffYuv, but as soon as the Content is BFF on the Input the Problem arises with MagicYUV and UtVideo while HuffYuv stays "clean".

Ignus2, keep that great work up because MagicYUV is really a very good Codec and hope you find the Problem, so hang in there and have another look in the Source Code, maybe you overseen something ;) So, that's all what I can do from my end and as I was "lucky" with RC4 I won't benefit from a new Release that fixes this Problem, which is okay, don't worry, I can live with it and I'm glad to help you out :)

@All
Many thanks to all of you for the fantastic Reply's in this Thread :)

Cheers

Mick

[kolak]

Do you have swap fields option ticked in huffyuv codec?

None of the lossless codecs will alter incoming video- it encodes what it gets and this will stay lossless.
In your case issues is before data hits codec. Problem is between app and codec, something goes wrong as there is even shift in the video. Yet again, this is not really magicyuv problem. Maybe huffyuv people made special work around it, so it works fine.

[Mick]

@Kolak
I used the Settings for HuffYuv as seen in the Screen Shot i posted, means, no "Swap Fields" Option enabled. I even tried VirtualDub-MPEG2 1.6.19 (ffcHandler) instead of VirtualDub 1.10.4, same Result. I agree, normally this should not happen but it does somehow and the only Software involved is VirtualDub, the Capture Devices and the Codecs, that's all i used for the Samples, no funny Codec Packs installed that could mess up something.

Win2K, XP, Win7, all with MagicYUV RC4 and UtVideo 13.3.1 along with VirtualDub gives the same results with NTSC and BFF Input, not just for me, for others too like my Friend in Florida who had the same Problem with MagicYUV. As i wrote earlier, i really don't know where the Problem could be and maybe Ben Roudiak Gould has a Routine in HuffYuv that takes care of this Situation when BFF is used on the Input.

So i ask you, where could be the Problem between VirtualDub, the different Capture Devices and the Codecs ? I practically grew up with VirtualDub but this is the first Time that i see a Codec behave like this. If it would be a Problem along with a exotic Software with funny Interfaces, yes, but VirtualDub ?

I did notice in VirtualDub 1.9/1.10 that some Filters handle the Field Order wrong. In the Manual it says for Example that "Field Order A" (Now TFF) handles the EVEN Field (Lower Field) under "Preview-Options", which is wrong, it handles the ODD Field (Upper Field) and "Field Order B" (now BFF) the ODD Field (Upper Field) but really is the EVEN Field (Bottom Field) for Interlaced Preview Mode.

VDub Manual "Field Order A" = Even Field, Truth: Odd Field (TFF)
VDub Manual "Field Order B" = Odd Field, Truth: Even Field (BFF)

This is also the case for the new Interlace Filter, Odd/Even Field is the other Way around. You can verify that easily with a proper DV Encoded Video which is always BFF for PAL and NTSC, for a long Time a reliable Standard. Load it in VirtualDub and select the Preview Option to "Weave TFF" and let it play in VirtualDub, afterwards to "Weave BFF" and you see the difference.

So, is it possible that there is a Error about the Fields in VirtualDub when using the Capture Mode ? And if so: For how long, if i get the same Result with Version 1.6.19 ? On the other hand, if there "is" a Error in the Field Order in VirtualDub, how come no one ever saw or noticed it before ? Millions of Users use VirtualDub since it came out and I'm the first one to discover this ? You must agree, that would be very strange, don't you think so ?

Going by Occam's Rule:
"The principle in philosophy and science that assumptions introduced to explain a thing must not be multiplied beyond necessity, and hence the simplest of several hypotheses is always the best in accounting for unexplained facts."

Or Occam's Razor:
"A rule in science and philosophy stating that entities should not be multiplied needlessly. This rule is interpreted to mean that the simplest of two or more competing theories is preferable and that an explanation for unknown phenomena should first be attempted in terms of what is already known. Occam's razor is named after the deviser of the rule, English philosopher and theologian William of Ockham (1285?-1349?)."

Source:
http://www.thefreedictionary.com/Occam%E2%80%99s+Rule

So, we all must ask ourselves at this point: Could it be that VirtualDub causes this Problem, that no one ever noticed before besides me, or the Codecs MagicYUV and UtVideo ? If it would be just me with this Problem, yes, could be something wrong with my Setup and VirtualDub and the Codecs are Innocent, but others too ? Same Problem with different OSes, Machines, Setup's and other Capture Devices ? Coincidence ?

Quote Kolak:
"None of the lossless codecs will alter incoming video- it encodes what it gets and this will stay lossless. In your case issues is before data hits codec. Problem is between app and codec, something goes wrong as there is even shift in the video."
Quote end

I agree with you and you're right, it should not happen, but somehow it does, can't explain why and how but you saw that yourself with the Samples i linked to the Forum.

Anyway, this whole thing is a Mystery for me as all the other captures I've done in all these Years with VirtualDub worked fine without any Problems and maybe it's because i mainly used HuffYuv the whole time. At this Point i really don't know what it could be and really puzzles me.

Cheers

Mick

[kolak]

Try this:

http://www.videohelp.com/tools/Stoik-Video-Capture

just to check something else.


I don't rate Vdub as a reliable capture tool. It's great tool, but Iam not sure about capture module.
My another simple advice is to capture DV as is ( no recompression) and than convert it to other codec. With todays PC it takes not much time. This way should be way more reliable.

[Ignus2]

Sorry being absent, was quite busy this week.

My suggestion would also be to try another capture program, anything else than VDub, and see what happens. I would also be interested in raw YUY2 capture.

About the compressed YUY2 being decoded as RGB always situation: do you have a sample which demonstrates this behaviour for you? I'd like to look at that as well.

About Win2k: The problem is that Win2k is missing a lot of Windows API functions that are needed to avoid the DLL shutdown deadlocks that was discovered after RC4, among other things. That OS is simply just too old.

Greets,
I.

[Mick]

@Kolak
I knew the Stoik Capture Tool and stopped using that a long time ago because it's not very reliable in keeping the Audio in Sync with the Video, but thanks anyway :)

@Ignus
About the RGB Output:
When i loaded a MagicYUV Capture in VirtualDub and selected the "Fast recompress" Mode with XviD as a Target Codec, no matter which VirtualDub Version (1.6.19 to 1.10.4), showed "RGB888" as a Target Format for XviD instead of YUY2. Same thing with a HuffYuv encoded File in VirtualDub, but shows "YUY2" as a target Format for XviD, not "RGB888".

By looking in the File Details Description of a XviD Video in the Windows Explorer, encoded with MagicYUV the XviD File says "24 Bit" while a HuffYuv/Lagarith encoded Video to XviD shows "16 Bit" under Color Depth.

I also tried the YUV Capture Module from MoviePack Pro using YUY2 for NTSC 4.43, saved in RAW YUY2 (!), opened that Capture in VirtualDub, used "Fast recompress", selected once the HuffYuv Codec, then the MagicYUV Codec. Result: Same distortion in the Image with MagicYUV while HuffYuv was clean.

The Capture Settings where:
Resolution: 720x480
Format: NTSC 4.43
Frame Rate: 29.97 Fps
Color Space: YUY2 4:2:2 16 Bit Uncompressed
Interlaced: Yes
Field Order: Bottom Field First (Lower Field / Even Field First)
Audio: 48.000 kHz, 16 Bit Stereo PCM
A/V Interleave: 1:1 (Every 1 Frame)

Same Settings with the simple Microsoft "Amcap" Tool, no difference. As soon as MagicYUV or UtVideo is used for Material that is NOT captured "TFF", then you get the Result that I made available where HuffYuv encodes the capture correct. I compared a direct re-compress with VirtualDub from the HuffYuv encoded Sample to YUY2, same as the Capture, no difference between RAW YUY2 and re-encoded from HuffYuv.

Note:
I tried the Multithreaded Version of HuffYuv (huffyuv_mt.dll), perfect, just like the HuffYuv CCESP Patch 0.2.5 that i use. Further: Lagarith (Version 1.3.23 for Win2K) also keeps track of the correct Field Order. Lagarith is one of the derivative Codecs of the HuffYuv Source Code.

About Win2K:
Yes, it's old, not the newest, but works fine and reliable for me what i can't say from the 2 Years I tried XP, nothing but Trouble and Problems with it. Booting Win2K is a matter of 20 seconds and my Machine is up and running, ready to go. With XP i went down to the Kitchen, made some fresh Coffee and when i came back after about 5 minutes it was just about done. No thanks, no need for a OS like that. BTW: The Win XP (SP2) came from the Dell Support, "optimized" for my Workstation and the later Dell update CD for SP3 made things even worse.

Further, Dell, who customized the Workstation for my needs, strongly recommended Win2K for being far more stable, faster and reliable. Guess what, they where right. Plus, for my kind of work i need something that does not hang every 5 Minutes. And yes, i even tried Win7, skipped the "fat" Vista and "Playtool" 8. A alternative ? Certainly not for me and takes even longer until everything is ready, far away from a RTOS like Win2K.

I refuse to work with a OS that takes several hours for a 1 Hour Documentation when Win2K does exactly the same thing in 12 Minutes. My Family would kill me if i spend my Time with watching for Hours the Screen until it's done. And my Clients would look for somebody else if i tell them that they get the Material back in 2 Weeks instead of a couple of Days.

Win2K works for me without any Problems, stable, reliable, fast, without Blue Screens or System hangs and nothing on this Planet will make me change to another Windows OS. If it works for you and all the others, that's fine with me, it did not work for me. But please, let's leave this excursus about OSes and hope you understand why i stick with Win2K, okay ?

Just to let you know: For the moment I uninstalled MagicYUV and UtVideo from my System because of this Problem. MagicYUV and UtVideo are real great Codecs, but still seem to have some Problems and HuffYuv served me reliably well over the Years without any Problems. I really hope you and the Author of UtVideo can find the Problem what's causing this.

So, there is nothing i can do any more from my end to help you solve this Problem, I wrote and described you everything that I know and observed. Still, don't give up, sometimes it's those small simple things that can cause a big Headache filled with Question marks ;)

Cheers

Mick

[Ignus2]

Now this is very interesting.
Could you send that raw YUY2 sample as well? I would be really interested and would like to try it on my machine to see the issue, and that would help a LOT, as I might have a chance to reproduce the problem on my machine.

Also, can you send the sample which reports RGB888 to xvid?

So, there is nothing i can do any more from my end to help you solve this Problem, I wrote and described you everything that I know and observed. Still, don't give up, sometimes it's those small simple things that can cause a big Headache filled with Question marks

You can :)
By sending the raw YUY2 sample and the other MagicYUV compressed sample which reports RGB888, that would allow me at least a chance to fix the problem.
I do not have any sort of capture hardware, so without those samples, it is simply impossible.

So, if you have a little more time, I would really appreciate those samples.

Greets,
I.

[kolak]

And another thing. Invest money into proper capture tool.
I would suggest GrassValley Edius.
There is 30 days fully working demo.

[Mick]

@Kolak
i have a very good Capture Module with my Movie Pack Pro from Aist Cinegy that keeps everything in perfect Sync, including the access to all the Settings for all TV Norms. Edius is nice but not that what I need for my Work. Thanks anyway for the Tip :)

Plus, I really don't think that it's a VirtualDub Capture Problem. I tried it again with the Stoik Capture Tool, the Microsoft WMV9 Amcap (original, not the commercial AMCAP), the Debut Capture Tool from NCH Software, the Logitech Capture Program and the NeroVision Capture Module (Nero Suite), all resulted with the same Field Order in NTSC 4.43 (BFF), no difference for I420, YUY2, YV12, UYVY or other Colorspaces.

I even tested Paul Glaga's "CaptureFlux" Tool without any difference what so ever:
http://paul.glagla.free.fr/index_en.htm

Other Tools tested: Grabshow, Video View, Syntech, Empia and QSonic Capture Tools, no difference, means from my point of view that VirtualDub works fine what capturing concerns. ;)

All Capture tests with the different Tools where done once with the native Windows YUV Codecs without the Drastic YCbCr Codecs and once again with the Drastic YCbCr Codecs installed, no difference, just to exclude that the Drastic YCbCr Codecs might be the Problem, but they are not. All NTSC 4.43 captures where BFF conforming to the norm with all Tools.

@Ignus
You already have a MagicYUV File that Reports RGB24 (RGB888) to XviD: The Sample you downloaded :) Since I tried MagicYUV, all Files reported RGB24 (RGB888) to XviD, never YUY2, regardless what Output Settings i used in the MagicYUV Config. Again, I left the Video Input at "Auto Detect" and the Video Output at "Same as Input", trying it with "Fast recompress" and "Normal recompress".

HuffYuv/HuffYuv_MT and Lagarith always reported in both recompress modes YUY2 to Xvid where MagicYUV and UtVideo always reported RGB24 (RGB888) to Xvid. A Win2K Issue ? No, because i tried the same thing with a small short Video on a Laptop with XP (SP3) using HuffYuv, Lagarith, MagicYUV and UtVideo, same result like under Win2K. HuffYuv/Lagarith reported YUY2 to Xvid, MagicYUV/UtVideo reported RGB24 (RGB888) to XviD.

And the RAW YUY2 ? It's really simple, open the HuffYuv Sample you downloaded in VirtualDub, select YUY2 for Output and use under Compression "Uncompressed YCbCr/RGB". This is the same as any other RAW YUY2 File that I would upload, verified with a uncompressed NTSC 4.43 Capture, both RAW YUY2 where exactly the same.

If you do really, really need another RAW YUY2 Sample, then I see when I get down to it and upload it via "Wetransfer", okay ? But this Time without Sound and in 640x480 instead of 720x480 to keep the Size down a bit. Would 10 seconds be enough ?

I really hope this helped you a bit further. BTW, did you or anybody else inform the Author of UtVideo about this Field Problem ?

Cheers

Mick

[Ignus2]

I don't see the problems you mention. The MagicYUV file gets correctly reported as YUV 4:2:2. I get the following log for Fast Recompress of the MagicYUV file you sent with 32 bit VirtualDub to XVID:


AVI: Opening file "E:\_vid\bugreport\MagyTest.avi"

Beginning dub operation.
[i] Dub: Fast recompress mode started with format: UYVY.

MagicYUV can decode compressed YUV 4:2:2 to both YUY2 or UYVY, whichever is requested.

For Normal Recompress with output set to "Same as decompression format" and input to "Autodetect" I get this for MagicYUV:

AVI: Opening file "E:\_vid\bugreport\MagyTest.avi"

Beginning dub operation.
[i] Dub: Recompressing using format: RGB888.

and this for HuffYUV:

AVI: Opening file "E:\_vid\bugreport\HfyuTest.avi"

Beginning dub operation.
[i] Dub: Recompressing using format: RGB888.

So in Normal Recompress BOTH report RGB. In Fast Recompress BOTH report YUV correctly.

I also tried to create the raw YUY2 file from HuffYUV. It looks correct. I then compressed the raw YUY2 file with MagicYUV: it also looks correct, moreover compared with Avisynth it is bit-by-bit identical to the HuffYUV encode.

Greets,
I.

[kolak]

Don't trust Vdub blindly.
Set color depth as desired instead of hoping it that everything will work as needed.

[ceth]

It seems there is a bug in your v1.0 codec install, I had to change my TMP folder to another or I get an access denied error (maybe due to non-english win7)

[Ignus2]

It seems there is a but in your v1.0 codec install, I had to change my TMP folder to another or I get an access denied error (maybe due to non-english win7)

Strange, what language? It should install on any language. I tried on various systems I had access to: English, Central-European, East-Asian OSes from Win XP to Win 8.1.

Greets,
I.

[ceth]

French

[Ignus2]

Anyone having any luck rendering 10-bit? I'm choosing the "UtVideo Pro YUV422 10bit VCM" option and have tried a few resolutions and frame rates but rendering in Vegas Pro it always tells me "The selected codec does not support the current render settings." In VirtualDub it says "Couldn't find compatible format. Possible reasons: *Codec may only support YUV *Codec might be locked *Codec might be decompression-only".

This is a reply to a post in the Ut Video codec thread, but I answer here, as the discussion would be off-topic there.

Hi!

I noticed your discussion on the Vegas Pro forums, and I'm facing the exact same problem with my codec's 10-bit variant. Basically what happens is that Vegas is presenting ONLY RGB32 for VFW codecs, period. It doesn't even ask anything about possible supported formats by the codec, I tried setting the project settings to 32-bit float, loaded an uncompressed v210 AVI, but still, the encoder gets RGB32 no matter what.

There are other problems as well, namely while Vegas can open uncompressed v210 AVIs, but if a codec reports that it can decode to v210, Vegas refuses to open it.
The Sony 10-bit YUV "codec" is actually what is doing v210 (hence the possibility to open uncompressed v210 AVI), but it isn't really a codec, all it does that it packs/unpacks the uncompressed data into v210, that is all.
However, the Sony 10-bit YUV codec is hard-coded into the avi output module of Vegas (you can verify this by looking into the avi output DLL file in the vegas folder - don't remember now where it is precisely - with Total Commander lister and search for Sony 10-bit, you'll find that, which means it is hard-coded), so it is probably able to get the native internal representation of Vegas, and not just RGB32.

So I have no idea how 10-bit or higher bit-depths can be supported by Vegas using VFW.
I asked Sony support about the issue, but they didn't answer, and I would be surprised if they did. Maybe you can try to convince them? ;)

EDIT: I also noticed some confusion on the Vegas forums about MagicYUV being a 422 codec, etc. MagicYUV can compress RGB, as well as various YUV formats, but in it's default mode, it compresses what it gets. In the next release I implemented a task-bar icon which will show exactly what the codec is doing (what input is it getting and how it is compressing it) to be clear what is happening. If you want to make sure you compress RGB, select RGB in the "Accepted colorspace" drop-down in the codec settings, this will make the codec accept only RGB and reject anything else. And if you want to be absolutely sure, press "Save as default" to save the settings to the registry. Note, that this will make the setting global for all apps, so remember that!

Greets,
I.

[nhope]

Ignus2, thank you for the very helpful explanation.

That's sad :( I will try to bring it to Sony's attention.

I wonder if Cineform's 10-bit VFW codec (in GoPro Studio Premium) is supported in Vegas Pro. I don't own it but I could download the trial to test.

[Mick]

@Ignus2
You have unread PN's :)

Cheers
Mick

[kolak]

Ignus2, thank you for the very helpful explanation.

That's sad :( I will try to bring it to Sony's attention.

I wonder if Cineform's 10-bit VFW codec (in GoPro Studio Premium) is supported in Vegas Pro. I don't own it but I could download the trial to test.
As far as I remember it does as Vegas has native support for CF (if you have license). Latest Adobe Media Encoder has native CF exporter included now, no need for seperate license (but MOV only).

[Mick]

@Ignus2
You still have unread important Pn's ;)

Cheers

Mickey

[Orivaa]

Soooo... what exactly is the point of this codec? I know it's fast and such, but what can one even use it for? You can't edit it because editing or re-encoding programs don't recognize it (At least when I tried), and you can't upload it to YouTube because it can't re-encode it.

[De-M-oN]

You can't edit it because editing or re-encoding programs don't recognize it

They do.

[Orivaa]

They do.

I couldn't get Handbrake nor Xmedia Recode to recognize it. If they couldn't, I don't see why editing programs would be able to.

[De-M-oN]

Handbrake uses ffmpeg for decoding only. Maybe XMR does the same.

Every program, which can access DirectShow will be able to decode this codec. Means there is a decoder in your system. But there is some software like VLC, Handbrake etc which ignore that and only use their internal decoder.
So of course Windows Media Player would be able to decode it.
MeGUI can decode it (better said avisynth)
Premiere, Vegas, etc can do as well, while the worst NLE camtasia cant though.

[zerowalker]

Ignus2 , how is it going with the ffmpeg implementation.
You started or is it not in the works?

Thanks

[Ignus2]

Ignus2 , how is it going with the ffmpeg implementation.
You started or is it not in the works?

Thanks

No, and it's unlikely in the near future. VLC plugin is more likely (I already have a working prototype). Also there are other very interesting stuffs in the pipe!

Greets,
I.

[zerowalker]

Ah ok, when it appears will Encoding be possible (why i ask is cause OBS will use ffmpeg which only supports UTVideo as a good Lossless Codec).
(What do you mean with VLC plugin, you mean a decoder inbuilt in it?)

Oh, can't wait to see those interesting things, you just got my anticipation up for more news;P

Thanks!

[Ignus2]

Not built-in, but as a dll/so binary plugin.

Greets,
I.

[zerowalker]

Yeah what supposed to say that.
Well that's great:)

[Orivaa]

So question. How well would this work for recording? I generally use Lagarith, then shrink the file size with Handbrake (A sad necessity).
So 1: Does this codec have less of a performance impact than Lagarith
And 2: How would one go about re-compressing it with h264 afterwards? Can't get it to work with Handbrake, after all.

[De-M-oN]

So 1: Does this codec have less of a performance impact than Lagarith

by far.
In my case a i7 3770k @ 4,1 ghz cpu, 1600 mhz cl7 RAM, RAID 0 with 352 mbyte/s - afterburner shows a frame compression time of about 19 to 22ms for Lagarith (2560x1600, yv12, multithreading) and for magicYUV about 8ms (2560x1600, 4:2:2 (same with YV12, maybe 1ms difference)

[zerowalker]

To answer the second, yes you do re-compress or rather Encode it with x264 afterwards.

Uploading Lossless is just impossible and not realistic in any scenario to Youtube.
MagicYUV works with x264 just as any codec should as long as you got a decoder.

[De-M-oN]

Uploading lossless is possible if you have the connection. Some not so complex videos I compressed with x264 via qp=0 which is by far smaller and was doable to upload with a 12 mbit uploadspeed.

[zerowalker]

Yeah it's possible, but it's not really something you should aim for:P
But i for that matter upload all Retro 2D games in x264 CRF=0 YV24 just for the heck of it.

[De-M-oN]

CRF 0 is not lossless. Use qp 0 (constant quantizer @ 0)

[zerowalker]

Wasn't both the same. Thought i read when CRF reached 0 it was the same thing.
Pretty sure i did some difference tests comparing Original Converted to YV24 with the x264 encoded version as well just to make sure.

[De-M-oN]

CRF 0 is not lossless. I assure you that. The encoder still writes b-frames and so on. All Lossy techniques are still used (which of course is also much slower than lossless)

[Information] [18.01.2015 20:18:00] Job commandline: "C:\Program Files (x86)\MeGUI\tools\x264_10b\avs4x264mod.exe" -L "C:\Program Files (x86)\MeGUI\tools\x264_10b\x264-10b_64.exe" --crf 0 --keyint infinite --min-keyint 1 --colorprim bt709 --transfer bt709 --colormatrix bt709 --sar 1:1 --output "E:\Lets Play Blood\gzdoom_2015_01_14_12_19_11_653.264" "E:\Lets Play Blood\gzdoom_2015_01_14_12_19_11_653.avs"
[...]
[Information] [18.01.2015 20:18:00] Standard error stream
-[Information] [18.01.2015 20:18:00] raw [info]: 320x240p 1:1 @ 60/1 fps (cfr)
-[Information] [18.01.2015 20:18:00] x264 [info]: using SAR=1/1
-[Information] [18.01.2015 20:18:00] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
-[Information] [18.01.2015 20:18:00] x264 [info]: profile High 10, level 2.1, 4:2:0 10-bit
-[Information] [18.01.2015 20:18:08] x264 [info]: frame I:1 Avg QP:11.58 size: 60801
-[Information] [18.01.2015 20:18:08] x264 [info]: frame P:727 Avg QP:16.59 size: 25621
-[Information] [18.01.2015 20:18:08] x264 [info]: frame B:306 Avg QP:19.37 size: 16797
-[Information] [18.01.2015 20:18:08] x264 [info]: consecutive B-frames: 42.8% 52.6% 1.5% 3.1%
-[Information] [18.01.2015 20:18:08] x264 [info]: mb I I16..4: 0.0% 25.3% 74.7%
-[Information] [18.01.2015 20:18:08] x264 [info]: mb P I16..4: 0.2% 8.8% 15.5% P16..4: 15.7% 16.0% 20.8% 0.0% 0.0% skip:23.1%
-[Information] [18.01.2015 20:18:08] x264 [info]: mb B I16..4: 0.0% 2.0% 5.6% B16..8: 13.8% 17.8% 20.2% direct:19.2% skip:21.3% L0:30.4% L1:27.0% BI:42.6%
-[Information] [18.01.2015 20:18:08] x264 [info]: 8x8 transform intra:34.6% inter:23.9%
-[Information] [18.01.2015 20:18:08] x264 [info]: coded y,uvDC,uvAC intra: 99.7% 99.9% 99.8% inter: 63.6% 69.3% 66.1%
-[Information] [18.01.2015 20:18:08] x264 [info]: i16 v,h,dc,p: 8% 9% 62% 21%
-[Information] [18.01.2015 20:18:08] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 9% 7% 38% 7% 7% 8% 7% 8% 9%
-[Information] [18.01.2015 20:18:08] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 12% 8% 19% 9% 10% 11% 9% 11% 10%
-[Information] [18.01.2015 20:18:08] x264 [info]: i8c dc,h,v,p: 66% 10% 12% 11%
-[Information] [18.01.2015 20:18:08] x264 [info]: Weighted P-Frames: Y:16.9% UV:16.5%
-[Information] [18.01.2015 20:18:08] x264 [info]: ref P L0: 80.9% 10.4% 8.1% 0.6%
-[Information] [18.01.2015 20:18:08] x264 [info]: ref B L0: 87.9% 12.1%
-[Information] [18.01.2015 20:18:08] x264 [info]: ref B L1: 100.0% 0.0%
-[Information] [18.01.2015 20:18:08] x264 [info]: kb/s:11060.91
-[Information] [18.01.2015 20:18:08] encoded 1034 frames, 136.68 fps, 11060.91 kb/s

[zerowalker]

Remember 10bit is never lossless if the source is 8bit.
If it's the same still with the 8bit version, i am at a loss of words.

Doesn't make sense that the images are lossless in difference.
Really must look into it the next time i do that, was 100% sure it was lossless, i must have done something wrong with my tests.

Thanks for bringing it up.

EDIT:

I got a file recorded with CRF=0 as far as i can remember, it says 0 bframes and not even crf=0 with MediaInfo.

https://trac.ffmpeg.org/wiki/Encode/H.264 - here it says crf=0 is lossless :S

[De-M-oN]

With qp 0 the encoder doesnt use b-frames and the quantizer values stay at zero. qp 0 works with 10bit as well.

[zerowalker]

10bit can't be lossless on 8bit sources.
It's like YV12 to RGB isn't lossless.
It's just larger files for no use.

10bit lossy is a gain on 8bit, but lossless it should just get larger sizes as it must use 10bit instead of 8bit and can't do any clever compression.

[De-M-oN]

--[Information] [01.02.2015 21:43:05] resolution: 1920x1200
--[Information] [01.02.2015 21:43:05] frame rate: 60/1
--[Information] [01.02.2015 21:43:05] aspect ratio: 8:5 (1.600)
--[Information] [01.02.2015 21:43:05] custom command line: --output-csp i422
--[Warning] [01.02.2015 21:43:05] --min-keyint bigger as --keyint/2+1. Lowering --min-keyint to max value: 1
--[Information] [01.02.2015 21:43:05] Job commandline: "C:\Program Files (x86)\MeGUI\tools\x264_10b\avs4x264mod.exe" -L "C:\Program Files (x86)\MeGUI\tools\x264_10b\x264-10b_64.exe" --qp 0 --keyint infinite --min-keyint 1 --colorprim bt709 --transfer bt709 --colormatrix bt709 --output-csp i422 --sar 1:1 --output "D:\XVideos\Lets Play Need For Speed SE\WT-NFSSE-Vertigo Ridge.264" "D:\XVideos\Lets Play Need For Speed SE\WT-NFSSE-Vertigo Ridge.avs"
--[Information] [01.02.2015 21:43:05] Process started
--[Information] [01.02.2015 21:43:05] Standard output stream
---[Information] [01.02.2015 22:06:43] avs [info]: AviSynth 2.60, build:Sep 28 2013 [15:09:12]
---[Information] [01.02.2015 22:06:43] avs [info]: Video colorspace: YV16
---[Information] [01.02.2015 22:06:43] avs [info]: Video resolution: 1920x1200
---[Information] [01.02.2015 22:06:43] avs [info]: Video framerate: 60/1
---[Information] [01.02.2015 22:06:43] avs [info]: Video framecount: 37550
---[Information] [01.02.2015 22:06:43] avs4x26x [info]: "C:\Program Files (x86)\MeGUI\tools\x264_10b\x264-10b_64.exe" - --qp 0 --keyint infinite --min-keyint 1 --colorprim bt709 --transfer bt709 --colormatrix bt709 --output-csp i422 --sar 1:1 --output "D:\XVideos\Lets Play Need For Speed SE\WT-NFSSE-Vertigo Ridge.264" --frames 37550 --fps 60/1 --input-res 1920x1200 --input-csp i422
--[Information] [01.02.2015 21:43:05] Standard error stream
---[Information] [01.02.2015 21:43:05] raw [info]: 1920x1200p 1:1 @ 60/1 fps (cfr)
---[Information] [01.02.2015 21:43:05] x264 [info]: using SAR=1/1
---[Information] [01.02.2015 21:43:05] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
---[Information] [01.02.2015 21:43:05] x264 [info]: profile High 4:4:4 Predictive, level 5.0, 4:2:2 10-bit
---[Information] [01.02.2015 22:06:43] x264 [info]: frame I:94 Avg QP: 0.00 size:512519
---[Information] [01.02.2015 22:06:43] x264 [info]: frame P:37456 Avg QP: 0.00 size:338061
---[Information] [01.02.2015 22:06:43] x264 [info]: mb I I16..4..PCM: 58.1% 0.0% 41.9% 0.0%
---[Information] [01.02.2015 22:06:43] x264 [info]: mb P I16..4..PCM: 13.1% 0.0% 8.6% 0.0% P16..4: 2.5% 1.6% 1.6% 0.0% 0.0% skip:72.6%
---[Information] [01.02.2015 22:06:43] x264 [info]: 8x8 transform intra:0.0% inter:21.5%
---[Information] [01.02.2015 22:06:43] x264 [info]: coded y,uvDC,uvAC intra: 89.6% 89.4% 89.3% inter: 3.8% 5.7% 5.6%
---[Information] [01.02.2015 22:06:43] x264 [info]: i16 v,h,dc,p: 22% 78% 0% 0%
---[Information] [01.02.2015 22:06:43] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 37% 51% 3% 3% 2% 1% 1% 1% 1%
---[Information] [01.02.2015 22:06:43] x264 [info]: i8c dc,h,v,p: 9% 30% 61% 0%
---[Information] [01.02.2015 22:06:43] x264 [info]: Weighted P-Frames: Y:0.9% UV:0.9%
---[Information] [01.02.2015 22:06:43] x264 [info]: ref P L0: 66.5% 11.9% 21.5% 0.0%
---[Information] [01.02.2015 22:06:43] x264 [info]: kb/s:162478.88
---[Information] [01.02.2015 22:06:43] encoded 37550 frames, 26.48 fps, 162478.88 kb/s

qp 0

I somehow doubt that 10bit changes the quality to negative at lossless.

[zerowalker]

Done some tests before, 8bit vs 10bit on 8bit sources.
10bit was always greatest in size, and it wasn't lossless (most likely as it has to be dithered down to 8bit, in 10bit it's probably identical if you have such gpu/monitor).

Do a fast recording on a 2D game, like just 1 sec.
Encode it with 8bit CRF=0 and compare original to encoded version (of course in same colorspace).
You can also try qp=0 and 10bit on both.

EDIT:

Interesting.

"Use of -qp is recommended over -crf for lossless because 8-bit and 10-bit x264 use different -crf values for lossless."

[De-M-oN]

Indeed interesting ^^

But 10bit has absolutely no loss to me compared to my 8 bit lossless capture.

But on the other hand if lossless mode is active the x264 8bit shouldnt produce banding if the lossless codecs dont either.
But if I use 10bit for x264 I thought I can be safe then.^^

[zerowalker]

Both of us seems to have been wrong.

10bit can do lossless on 8bit (still confused how it works. For me it's like storing YV12 in RGB), but it works it's just that i have failed using lossless (crf=0).

And you thought crf=0 wasn't lossless cause it wasn't in 10bit and i was using 8bit.

Quite funny, learned something. Must try 10bit and compare filesize sometime, if it's less then 8bit i am at a loss:P

[Orivaa]

by far.
In my case a i7 3770k @ 4,1 ghz cpu, 1600 mhz cl7 RAM, RAID 0 with 352 mbyte/s - afterburner shows a frame compression time of about 19 to 22ms for Lagarith (2560x1600, yv12, multithreading) and for magicYUV about 8ms (2560x1600, 4:2:2 (same with YV12, maybe 1ms difference)
So if used to record, the performance impact would be less, yes? Aaaaalso, whenever I use a recording program that supports External Codecs like MagicYUV, I can only use the 32 bit version. I imagine that would have different performance from the 64 version.

To answer the second, yes you do re-compress or rather Encode it with x264 afterwards.

Uploading Lossless is just impossible and not realistic in any scenario to Youtube.
MagicYUV works with x264 just as any codec should as long as you got a decoder.

I can't find a decoder, though. Neither Handbrake nor XMedia Recode recognizes it, and I was never able to render x264 with Sony Vegas. Know any programs (Preferably not editors, but programs like Handbrake) that supports both MagicYUV, and allows it to re-encode it as h264?

[Bloax]

10bit can do lossless on 8bit (still confused how it works. For me it's like storing YV12 in RGB)
Bits are not colorspaces; you can easily store 8 bits of information in 10 bits of information (the simplest thing to do would be leaving the excess two bits empty), but you can't store YUV values in RGB.

[Ignus2]

So if used to record, the performance impact would be less, yes? Aaaaalso, whenever I use a recording program that supports External Codecs like MagicYUV, I can only use the 32 bit version. I imagine that would have different performance from the 64 version.

That's because the program itself you use is 32 bit. If you use a 64 bit program, then it will use the 64 bit version.
And yes, the 64 bit version of the codec is much faster (especially for decoding).


I can't find a decoder, though. Neither Handbrake nor XMedia Recode recognizes it, and I was never able to render x264 with Sony Vegas. Know any programs (Preferably not editors, but programs like Handbrake) that supports both MagicYUV, and allows it to re-encode it as h264?
The reason why they don't "recognize" it is because they probably use ffmpeg, which is a monolithic collection of codecs, and obviously MagicYUV is not in it.
I might write a reference decoder sometime in the future for ffmpeg, though that sometimes could lead to confusion, like currently in the case of UT looking like it is slow, while in fact sometimes the installed inferior ffmpeg implementation (through Lav filter for example) does the decoding instead of the optimized official decoders.

In the meantime, you can use Avisynth with "AVISource()" to open MagicYUV encoded files and feed that to an h264 encoder.

Greets,
I.

[zerowalker]

Bits are not colorspaces; you can easily store 8 bits of information in 10 bits of information (the simplest thing to do would be leaving the excess two bits empty), but you can't store YUV values in RGB.

Yeah know. But thought those extra two bits would ruin things.
But if they can be empty then i guess the file is only larger by the address of that space or something, which should be next to nothing.

Thanks

[zerowalker]

And yes, the 64 bit version of the codec is much faster (especially for decoding).

Damn had no idea about this, knew it would be faster but thought like 1-5% at best. Guess it's more?

[De-M-oN]

Like said. Use MSI Afterburner and you can access the 64bit encoder.

http://www.abload.de/gallery.php?key=jRBPGPYE

with this settings (fps and codec of course your choice - the screen is older - but the rest of the options are still up to date.)

You'll still see MagicYUV's 32bit config, but Afterburner will use the 64bit encoder when you record. It does via the dedicated encoder server. An extra process for the encoding. This way it can use the 64bit encoder.

[zerowalker]

Thing i hate about MSI Afterburner is that the sync is off by a different way compared to Dxtory (non of them is good as both are unsync, Afterburner seems to be better though).
Other is that the record using 16bit audio which causes a bit of a noise floor which disturbs me. (Asked them to use float and convert to 16bit at the end to no avail, but well only i that care about it;P).

Other than that it's an excellent software, didn't really think about the 64bit option even though i have seen it.

[De-M-oN]

Afterburner is perfectly fine in sync - no asynch at all. Else story for dxtory over time :( And DXTory got so goddamn slow on OpenGL, while it was in the past fast there :/

There is only one thing you have really to take attention for: Afterburner closes Audio a lot milliseconds (up to 800ms) earlier than the video track. If you append several avi files without paying attention for that - of course it will get asynch.

Thats one of the reasons why we at letsplayforum use SSM. It can extract the audio while syncing it to the videotrack - if the track is too short it will fill up with silence and if its too long it will cut the rest.

This way you have a perfect synchronization with audio and video (if its not asynch within the track) and for me at Afterburner its perfectly accurate in sync.

The 16bit audio had never been an issue to me to be honest. (Rode NT2-A + PreSonus Audiobox 22VSL)

SSM : http://www.letsplayforum.de/index.php?page=Thread&threadID=110277 (its basically a avisynth script GUI with widened options for youtube videos compared to megui, allows an audio preview (and also for cutting) and has the said audio extracter - if you extract audio after making the trim entries it will even consider for the extraction that, means audio is directly cutted then. No clt list for megui necessary then.

And dont worry, you can choose english if you are not german.

[zerowalker]

In my tests it's not (recorded for 24 hours), was some time ago but haven't seen any difference in the changelog.
Remember there is a huge difference per system, it all depends on Audio Clock / PC Clock. Even though Afterburner seems to do something as it's not offsync in the order it should be, which is the difference between the two clocks.
So they are using something to sync the tracks, i don't know how. But it's not perfect as it gets offsync but on a much smaller scale (may have to do a new test as it was quite a while ago).

Hmm didn't know about that it closes Audio earlier that much, but did now it recorded every track independently, dxtory doesn't (All tracks are always the same as Video track, even if it's offsync, so something is wrong in the coding there).

16bit sound noise floor is not something a sane man would complain about. It's only noticeable when it's no sound at all, then there is this Hiss. (I am not sane;P).
(It won't matter on a Mic as there is no such thing as Silence with that).

Nice setup though, i myself got a USB Mic, but would like to have a external Sound Card and a Mic + Mic amplifier. All to prevent dependence and also internal interference (PC Chassis is a noisy place).

Wait so what does it do, append videos the correct way to prevent offsync?
Cause i don't really append recorded videos, i Trim and add intro/outro in Avisynth and with that it's always correct as long as i don't mess something up.

[Orivaa]

That's because the program itself you use is 32 bit. If you use a 64 bit program, then it will use the 64 bit version.
And yes, the 64 bit version of the codec is much faster (especially for decoding).


The reason why they don't "recognize" it is because they probably use ffmpeg, which is a monolithic collection of codecs, and obviously MagicYUV is not in it.
I might write a reference decoder sometime in the future for ffmpeg, though that sometimes could lead to confusion, like currently in the case of UT looking like it is slow, while in fact sometimes the installed inferior ffmpeg implementation (through Lav filter for example) does the decoding instead of the optimized official decoders.

In the meantime, you can use Avisynth with "AVISource()" to open MagicYUV encoded files and feed that to an h264 encoder.

Greets,
I.

Noob question (As if the previous ones weren't, lol.) but does decoding play a factor when recording at all?

Also, I'm not very good with the very technical stuff like Avisynth (NOOOOOOOB), so is there a program with an interface that I could use instead?




Also, different question. I tried seeing if recording with Afterburner could be an option, but it also only showed the 32-bit version of MagicYUV. So I went and checked, and, as far as I can tell from the install location, I'm using a 32-bit version of Afterburner. However, I can't find a 64 bit version of the newest version anywhere, as it keeps wanting to install it to my 86 bit folder.
It may be that the newest version simply includes both 32 and 64 bit versions, but then how come only the 32 bit version of MagicYUV shows up?

[De-M-oN]

Also, I'm not very good with the very technical stuff like Avisynth (NOOOOOOOB), so is there a program with an interface that I could use instead?

If you can live with their shitty encoders and almost no settings - yes.

Also, different question. I tried seeing if recording with Afterburner could be an option, but it also only showed the 32-bit version of MagicYUV. So I went and checked, and, as far as I can tell from the install location, I'm using a 32-bit version of Afterburner. However, I can't find a 64 bit version of the newest version anywhere, as it keeps wanting to install it to my 86 bit folder.
It may be that the newest version simply includes both 32 and 64 bit versions, but then how come only the 32 bit version of MagicYUV shows up?

It isnt necessary that the whole program runs at 64bit.

A seperate process for encoding is enough - which the checkmark for dedicated encoding server does (look at my image gallery).

[zerowalker]

Just did a test with Afternburner (though it fails almost all the time, but was able to save a long file).

Me and the Audio is in sync with each other as far as i can tell.
However The video is out of sync with the audio.

Here is a file that shows it: https://www.sendspace.com/file/6uluwg - 97.5MB LAGS

EDIT: Made a topic at the right place: http://forums.guru3d.com/showthread.php?p=5008541#post5008541

[zerowalker]

If i decode MagicYUV using Avisource in Avisynth 64bit it gets corrupted (The image is distorted, like it's stretched diagonal).
Any idea what's going on?

[Ignus2]

If i decode MagicYUV using Avisource in Avisynth 64bit it gets corrupted (The image is distorted, like it's stretched diagonal).
Any idea what's going on?

I've encountered a similar bug once, and it was on Avisynth's side, somewhere it got the stride wrong, but I forgot to file a bug-report, and now I don't remember exactly how to reproduce it. I think it was resolution dependent.
What resolution?
Does the video open correctly in VirtualDub?
Do you have a sample of the video?

Greets,
I.

[zerowalker]

Well i get it with pretty much every resolution, cause i first thought it was the problem.

Resolution i tried after 2 weird ones was 640x480 (which should be as normal as it can get?).
All failed.

It was RGB32(or 24, no Alpha, not sure what you save as).

I tried opening it with x264 first and it failed, then virtualdub and it failed, then MPC-HC x64 and it failed.
Failed = it's corrupted.

Also noticed it's not just MagicYUV, happens with Lagarith as well, so may be just RGB that's the problem?

Here is a file: https://www.sendspace.com/file/ohpp7o

This file will fail if decoded with Avisynth 64bit.

[Ignus2]

Yes, this is a bug in Avisynth, at certain resolutions decoding to RGB32/RGB24 it gets the stride wrong. It's wrong for both 32/64 bit editions BTW.

Greets,
I.

[zerowalker]

32bit works for me.
Though 32bit is 2.6 and 64bit is Avisynth+.

[Orivaa]

I did some testing with Afterburner, and the codec would work wonderfully for the entire recording. (Solid 60 fps when not recording, and solid 50 when recording. This is on decently high settings and 1080p, 60fps.)

The video works fine, right up to the moment I decide to actually move my character, in which case the player crashes.

[zerowalker]

Wait, Afterburner fails for you?
Player crashes, do you mean the Media Player or EncoderServer?

[Orivaa]

I don't know if it's Afterburner or the codec.

I use MPC-BE, and it works fine until I move my character. All other plays couldn't play the files at all, and the program I used to re-encode my last MagicYUV file (XviD4PSP 5) failed to read the file.

[Ignus2]

I don't know if it's Afterburner or the codec.

I use MPC-BE, and it works fine until I move my character. All other plays couldn't play the files at all, and the program I used to re-encode my last MagicYUV file (XviD4PSP 5) failed to read the file.

What do you mean by moving the "character"?
Does VirtualDub open the file?
Can you send a sample, which exhibits the failure?

Greets,
I.

[Orivaa]

Ok, I tried again with a different game, and the same thing happened. However, I then tried again using the AVI container option instead of the MKV, and now it works.

[Orivaa]

What do you mean by moving the "character"?
Does VirtualDub open the file?
Can you send a sample, which exhibits the failure?

Greets,
I.

I was doing it on Far Cry 4, and when I started the recording, I'd stand still for a while and observe the framerate. However, once I actually began to move my character, the video would fuck up. The game showed no sign, but it was the same on all my videos.


And no, Virtualdub wouldn't open the file. Said it was unsupported, but that might be because of the MKV container.

As I said in my post above, changing the container to AVI fixed the problem.

[zerowalker]

Still weird, MKV should work as well.

[Mystery Keeper]

Any hope for P010 and P016 support? All essential VapourSynth plugins already can work with 8-16 bit planar video, and we need a codec for intermediate encoding.

[Ignus2]

Still weird, MKV should work as well.

I'd be surprised if it did. But then again, miracles can happen ;)

@Orivaa: Good to hear you got it working.

Greets,
I.

[Ignus2]

Any hope for P010 and P016 support? All essential VapourSynth plugins already can work with 8-16 bit planar video, and we need a codec for intermediate encoding.

Mochiron, it would be almost trivial to implement it in the current architecture of the codec. But note that P016 would also be only up to 10 bits precision.
However, above-8-bit compression is unreleased as of yet. I can't give dates yet (there's just mountains of work to do nowadays), but that part of the codec will not be free. Is that suitable for you?
BTW, how does VapourSynth interface with codecs? VFW/DShow/other(custom)?

Greets,
I.

[Mystery Keeper]

But note that P016 would also be only up to 10 bits precision.Then why do it at all?
but that part of the codec will not be free. Is that suitable for you?Depends on price.
BTW, how does VapourSynth interface with codecs? VFW/DShow/other(custom)?It doesn't. It is a library that outputs raw uncompressed frames, which can be passed anywhere. Since VirtualDub can open VapourSynth scripts, they can be encoded with VfW codecs.

[Ignus2]

P016 would only have a meaning if P010 is not recognized or the whole internal processing is 16-bit, then it's better if the codec provides P016 (needs no further conversion), but in the case of VapourSynth, P010 would probably be fine.
Another case would be if the codec encodes at 12 bit precision (planned), that can be encoded/decoded to/from P016.

I assume you would need to both encode output from VapourSynth and decode input for VapourSynth. How does video enter VS? AviSynth has AVISource(), DirectShowSource(), etc., what is the equivalent in VS?
Also, can VirtualDub handle P016/P010?

Greets,
I.

[Mystery Keeper]

How does video enter VS? AviSynth has AVISource(), DirectShowSource(), etc., what is the equivalent in VS?
AviSource, FFMS2, other source plugins.
Also, can VirtualDub handle P016/P010?Looks like it can't =/ I'm going to implement VfW encoding in VapourSynth Editor when I have time. Also should request high depth handling in VD.

[foxyshadis]

Even in the absence of real support, there's always the stack16 hack that works just as well with MagicYUV as anything else. Stack16 isn't really interoperable to other editors, but tools like avs4x264mod and Avs2YUV will convert back to normal 16-bit for command-line encoders' sake.

[Are_]

Long time ago I did try to encode to lossless using stack16 and the encoder crashed. Did you try to actually encode to lossless with it and succeed or is it just a wild guess (was I unlucky?)?

[Ignus2]

After a long hiatus, the 1.1 version of the codec is here!
The major new feature of the codec is that it now includes a QuickTime encoder/decoder component and VLC decoder plugins and adds support for Mac OS X (through QuickTime and VLC) and Linux (through VLC). The QuickTime and VLC plugin is also available in the Windows version.
Get to the website for details and download: http://magicyuv.com

The QuickTime and VLC plugins are considered beta, though I've made the best effort to make them as stable as possible. So if you encounter any crashes or glitches, reports are welcome.

Greets,
I.

[zerowalker]

Great work as always!

[ceth]

Thank you !

[kolak]

Codec uses about 2x less CPU during playback than ProRes :)

[Gargamel]

Thank you again, Ignus2 !

[TCmullet]

I just discovered the existence of this codec today, and am very glad! Thank you for it!

I've installed it twice. On one system, installation bombs with the message, "Internal error: Cannot install files to 64-bit locations on this version of Windows."

When I first saw that I said to myself, "That's okay, this is not a 64-bit Windows anyway, and I was already planning on running 32-bit editions of everything. Surely the 32-bit installation must have finished." But when I went to encode in Vdub, MagicYUV does NOT appear in the list of available codecs!

I have a 64-bit hardware, but have to run 32-bit Windows XP for the time being.

I really really need to get this running asap. Any chance you can get it fixed quickly, hopefully TODAY??

[Ignus2]

I just discovered the existence of this codec today, and am very glad! Thank you for it!

I've installed it twice. On one system, installation bombs with the message, "Internal error: Cannot install files to 64-bit locations on this version of Windows."

When I first saw that I said to myself, "That's okay, this is not a 64-bit Windows anyway, and I was already planning on running 32-bit editions of everything. Surely the 32-bit installation must have finished." But when I went to encode in Vdub, MagicYUV does NOT appear in the list of available codecs!

I have a 64-bit hardware, but have to run 32-bit Windows XP for the time being.

I really really need to get this running asap. Any chance you can get it fixed quickly, hopefully TODAY??

As a workaround, try to install without the VLC component, I think that's the culprit.

Greets,
I.

[Ignus2]

Revision 1 for Windows is out which fixes the installer bug for 32-bit editions of Windows, which prevented installation on 32-bit Windows if the VLC component was selected.

Also, the download links for 1.1 have been modified, so anyone who hard-linked the site should update the links.

Greets,
I.

[Ignus2]

Revision 2 for Windows and Revision 1 for Mac OS X have been released, adding VLC 2.2.x plugins. There were no other changes.

Greets,
I.

[benwaggoner]

When do you anticipate a public release of 10-bit support?

[kolak]

This won't be free for public.

10bit in AVI has almost 0 support in applications. Only things like avs, vapoursynth can make some use of 10bit in this codec.

10bit in MOV already works well on Mac. MOV on PC is different and needs some work.
The best is to use API for 10bit.

[Ignus2]

When do you anticipate a public release of 10-bit support?

As kolak said.

10-bit is mainly done now through QuickTime, but you have to be super careful and know what you're doing to really retain 10-bit (and not dithered 8-bit or similar).

Also, plugins for various apps are currently under development, that should work better than QuickTime.

Greets,
I.

[TCmullet]

As a workaround, try to install without the VLC component, I think that's the culprit.

I unchecked the VLC component, and the installer worked without errors. Thanks.

I did try running the codec on the different machine here where the full install worked. I left all settings at their defaults. The output file was huge, much bigger than I expected. Is there some combination of settings that will minimize the file size?

[Ignus2]

I unchecked the VLC component, and the installer worked without errors. Thanks.

I did try running the codec on the different machine here where the full install worked. I left all settings at their defaults. The output file was huge, much bigger than I expected. Is there some combination of settings that will minimize the file size?

The rev1 and rev2 Windows installers should work on 32-bit machines with VLC too.

You can expect a compression ratio around 2:1 - 4:1 (depending on the material), there isn't much you can do about it.
But it depends a lot on what you are trying to do, what color space is your source, etc.

Greets,
I.

[kolak]

I unchecked the VLC component, and the installer worked without errors. Thanks.

I did try running the codec on the different machine here where the full install worked. I left all settings at their defaults. The output file was huge, much bigger than I expected. Is there some combination of settings that will minimize the file size?

It lossless codec, so you can't predict final size.
If source is clean, full of flat areas it may compress it 8x, but if source has eg. loads of random noise than it may not compress at all, so your file may be as big as uncompressed.

The one thing which you known that after decoding your video has 100% the same quality as original (if no 422, 420 etc conversion is happening).

As Ignus2 said- typical value for compression are between 2-4x.

[TCmullet]

True, you can't predict final size. But compared to Lagarith, it was much larger (but less buggy). I was able to get it smaller via x264vfw-lossless for now. Will keep MagicYuv in mind for other things.

[kolak]

Was it the same color sampling: 4:4:4, 4:2:2 or 4:2:0 for both cadecs?
Was it normal footage or some with repeated frames?
There shouldn't be massive difference.

[TCmullet]

Was it the same color sampling: 4:4:4, 4:2:2 or 4:2:0 for both cadecs?
Was it normal footage or some with repeated frames?
There shouldn't be massive difference.
If you're talking to *me*, it's 4:2:0 (had to figure out MediaInfo). "Normal" footage, not a slide show. In fact, it's so normal that in Lagarith I'm inclined not to bother allowing null frames. It's video camera footage of sports.

[kolak]

And what is the size difference between magicyuv and Lagarith ?

If you can post 10sec samples fro both- cut in Vdub.

[benwaggoner]

10bit in AVI has almost 0 support in applications. Only things like avs, vapoursynth can make some use of 10bit in this codec.
The Adobe products like After Effects and Premiere can use 10-bit in AVI and QuickTime on both Windows and Mac. Internally they can render at up to 32-bit float, and output as "Trillions of colors" to a codec that supports >8-bit per channel.

There really aren't any good free interoperable compressed 10-bit codecs today that support a wide range of frame sizes. ProRes is okay, but only can encode via QuickTime on Mac. DNxHD has a 1920x1080 frame size cap. You need to pay for the Cineform encoder.

For my internal workflows I can make it all work, but for collaboration between companies/teams having a high quality, flexible, compressed 10-bit codec would be a big deal.

[kolak]

AE will never ask for 10bit pixel format in AVI when you import file. Only v210 is supported, but this is hard coded.
If you want to do it you have write a plugin, like Cineform use to have.

Magicyuv may have very good support in Adobe software if all goes well.
Mov already works well, but native/direct support in Adobe engine or over plugin would be even better.

The ”issue” is that this is lossless codec, so saves bits, but not as much as eg. Cineform.

[De-M-oN]

Please add RGB24 conversion.

"as is" would record pc video capture like games as RGB32 - but RGB32 has no use for pc game capture, there is RGB24 enough.

Sure I could use YUV 4:4:4 but thats not supported by the Spline100 Scaler.

[foxyshadis]

Please add RGB24 conversion.

"as is" would record pc video capture like games as RGB32 - but RGB32 has no use for pc game capture, there is RGB24 enough.

Sure I could use YUV 4:4:4 but thats not supported by the Spline100 Scaler.

Unless you have serious memory pressure, RGB32 is usually faster than RGB24, and it won't be any larger. The alpha plane is empty, so it'll be compressed to no more than a byte or two.

[Ignus2]

Please add RGB24 conversion.

"as is" would record pc video capture like games as RGB32 - but RGB32 has no use for pc game capture, there is RGB24 enough.

Sure I could use YUV 4:4:4 but thats not supported by the Spline100 Scaler.

The codec supports RGB24 as both an input and output format, so I don!t really understand what do you mean by "RGB24 conversion".
From the compressed data point of view, there is no such thing as RGB32 or RGB24. There is 3-channel RGB or 4-channel RGBA. Both RGB32 and RGB24 INPUT is compressed as 3-channel RGB (for RGB32: if "Enable RGBA compression" is disabled), and presented when decoding as either RGB24 or RGB32 (depending of what the decoding app asks).

So recording "as-is" results in compressed 3-channel RGB for both RGB24 and RGB32 input (if "enable alpha" is disabled), so the compressed data will be the same.

(As a side note: YUV 4:4:4 can also be presented by the codec as either RGB24 or RGB32 btw.)

Greets,
I.

[benwaggoner]

Unless you have serious memory pressure, RGB32 is usually faster than RGB24, and it won't be any larger. The alpha plane is empty, so it'll be compressed to no more than a byte or two.
The main problem with having a "blank" alpha channel is that a compositor doesn't have any way to know if the channels are blank. So it might wind up doing per-pixel transparency calculations for every frame even though it all winds up as a NOOP.

I've seen it happen, with a deadly perf hit.

I only turn on an alpha channel when there's actually alpha data to avoid downstream confusion. Even if the codec acts just the same with every forth byte zeroed out.

[zeropc]

there seams to be a gamma shift upon export to magicyuv using quicktime 7 pro on mac

[Ignus2]

there seams to be a gamma shift upon export to magicyuv using quicktime 7 pro on mac

That's very much possible, QT is riddled with gamma shift issues.

Sorry for being absent for a long time, things in life kept me quite busy.
Anyway, there'll be a new release of the codec soon!

Greets,
I.

[De-M-oN]

MagicYUV is somehow very strict with the Chroma Subsampling conversion.

See this post:

http://www.letsplays.de/forum/index.php/Thread/77996-Videoqualit%C3%A4t-und-Encodierverfahren-Sammelthread-Startpost-beachten/?postID=2031592&highlight=magicyuv%2Bungerade#post2031592

Gerade = even
Ungerade = odd


If every 2nd pixel is without color results text being partially completely colorless. The winamp playlist had only partially color left in its text.
Or at the game doom 2 the automap where the red lines of the walls are 1 pixel width - they switch color especially when you scroll the map. Really doesnt look good.

With UTVideo the 2nd pixels are only weaker colored, but not colorless like you can see on the images. This way text and automap and so on still have color.

[zerowalker]

That's very much possible, QT is riddled with gamma shift issues.

Sorry for being absent for a long time, things in life kept me quite busy.
Anyway, there'll be a new release of the codec soon!

Greets,
I.

Don't think about it, the anticipation is real now;P!

[Ignus2]

MagicYUV is somehow very strict with the Chroma Subsampling conversion.

See this post:

http://www.letsplays.de/forum/index.php/Thread/77996-Videoqualit%C3%A4t-und-Encodierverfahren-Sammelthread-Startpost-beachten/?postID=2031592&highlight=magicyuv%2Bungerade#post2031592

Gerade = even
Ungerade = odd


If every 2nd pixel is without color results text being partially completely colorless. The winamp playlist had only partially color left in its text.
Or at the game doom 2 the automap where the red lines of the walls are 1 pixel width - they switch color especially when you scroll the map. Really doesnt look good.

With UTVideo the 2nd pixels are only weaker colored, but not colorless like you can see on the images. This way text and automap and so on still have color.

Right, I can include an option for interpolation in the next release.

Greets,
I.

[Mick]

"MagicYUV" and "UtVideo" are good lossless Codecs, no doubt, but "HuffYUV" still rules for SD/HD/UHD Content and "Lagarith" for SD with YV12. Many Colleagues of mine around the World in Mastering Labs tested MagicYUV and UtVideo with the result that they stay with HuffYUV (CCESP Patch 0.2.5/MT and 2.1.1 64 Bit). They found MagicYUV and UtVideo "interesting" but not for professional use for Mastering.

In Short:
MagicYUV and UtVideo can get painfully slow on decompression if Dolby Surround, Digital (5.1) or DTS is added. Encoding with MagicYUV and UtVideo can end in a "freeze" for several Seconds if 2K is used and both freeze fast with higher Resolutions than 4K. Tested on 32/64 Bit Hig-End Systems and 32/64 Bit Codec Versions.

With HuffYUV never a Problem and BTW, HuffYUV uses much less CPU than all the other lossless Codecs while encoding/decoding Videos. (32/64 Bit Versions)

Here you can compare yourself:

Codec Setups:
https://yadi.sk/d/9NzZJc3Hcumhx

Test Videos: (A short Sequence with kind permission of PolyGram)
https://yadi.sk/d/X5r8s7HVcurL6

The Test Videos are in NTSC SD DV with BFF (Bottom Field First). You will notice that HuffYUV has the best Field Separation and is clearer than Lagarith. MagicYUV and UtVideo could not handle this Field Order, only TFF (Top Field First).

TFF = Odd/Upper Field first
BFF= Even/Lower Field first

Please note that the Videos have the same Sequence/Length and uses the same Settings inside the NLE to get a reliable Result for comparison.

The "To-do" List for MagicYuV and UtVideo:
The "Chroma" Problem
The Gamma Problem with QuickTime (Win uses 2.2, MAC 1.8)
The Field Order Problem (only the "Top Field" is considered)
Both Codecs don't support RAW-VBI
Dual Image 3D not well supported

UtVideo: v210 leaves a Black/Green Screen and stutters inside professional NLE's with a YCbCr Card and plays only the first 10 Frames before the Playback freezes inside the NLE's.

I do believe that both, MagicYUV and UtVideo can be a excellent choice if some Issues are finally fixed by the Authors.

@Ignus: You still have unread PN's, please read, just trying to help, okay ? ;)

Cheers
Mick

[Ignus2]

BTW, HuffYUV uses much less CPU than all the other lossless Codecs while encoding/decoding Videos.

Could you back up these claims by showing your testing methodology and results, especially detailing how you measured codec-only performance by eliminating other factors, such as Disk I/O, color space conversion, etc?
(To be clear, saying "I loaded up into VirtualDub and looked at the FPS" is not a relevant testing methodology.)


Here you can compare yourself:

Codec Setups:
https://yadi.sk/d/9NzZJc3Hcumhx

Test Videos: (A short Sequence with kind permission of PolyGram)
https://yadi.sk/d/X5r8s7HVcurL6

The Test Videos are in NTSC SD DV with BFF (Bottom Field First). You will notice that HuffYUV has the best Field Separation and is clearer than Lagarith. MagicYUV and UtVideo could not handle this Field Order, only TFF (Top Field First).

TFF = Odd/Upper Field first
BFF= Even/Lower Field first

I've tried to explain to you many times calmly in PMs, that neither of the codecs have anything to do with field ordering. Period. I've even detailed it to you by hours of dissecting the material you sent to me in PM. The videos you sent at that time had a problem called reversed field dominance (AND swapped fields as well), which is described here:
http://rationalqm.us/reverse/reverse.html
Try to understand what is written there, and also what I wrote to you in PM. It has much more, than just simple TFF/BFF.
For some reason, your compression setup ends up sending wrong field dominance frames to both MagicYUV and UT, which is beyond the codecs, and I don't know why that is.


Both Codecs don't support RAW-VBI

Neither do HuffYUV AFAIK, but someone more knowledgeable about HuffYUV correct me if I'm wrong.

Greets,
I.

[De-M-oN]

professional NLE's

Dont forget though that NLEs all have very bad decoder and encoder.

Especially Premiere and Vegas and so on. Such a disaster. They even dont have full h.264 support. I mean: Their H.264 decoder cant decode 10bit H.264, they cant decode High 4:4:4 profile and the list goes on. Since when exists H.264? Since about 2003? And they still have an decoder which doesnt support h.264 fully?
What is their h.264 decoder doing? Apparantly not its job.

Oh and all the problems premiere had with lagarith in the past and still has ..

Blame the NLEs, not the codecs..

[Mick]

@Ignus2,
not only me but many of my Colleagues monitored this and leaves VirtualDub here out as a Measurement. RAW-VBI is supported by HuffYUV because my Capture Card supports it, disabling it does make a big difference with HuffYUV.

What they did was that they took a short Sequence with different Audio Streams and rendered it once with MagicYUV and once with UtVideo in 2K and 4K where 8K failed, but neither MagicYUV nor UtVideo claimed to handle that Resolution.

My Cinegy Process Monitor from my NLE also showed a much higher CPU load and lower Frame Rate (Rendering) with MagicYUV and UtVideo than with HuffYUV, Lagarith I leave out for the moment. And the different Field Orders are okay in MagyicYUV and UtVideo as long as it's TFF, only BFF creates the Problem and should not be ignored.

HuffYUV offers a Option to switch the Field Order if it got upside down. I don't know either why HuffYUV works with both Field Orders but it is Important because there was never a real standard for this, besides PAL/NTSC SD DV that is always BFF and PAL/NTSC HD which always uses TFF.

Don't get me wrong, I am not trying to make MagicYUV or UtVideo "look bad" to push HuffYUV, it is just what has been discovered, nothing else. So don't go mad with me, okay ?

Cheers
Mickey

[Ignus2]

For completeness' sake, I'll copy-paste the answer I wrote in PM to Mick dissecting the videos he sent (the same as now), so others who might want to jump into a many-hour endeavour to find out what's wrong with them can start with something I've found out:


First of all, I analyzed the videos you sent with VirtualDub using the "deinterlace filter" with "Unfold fields side-by-side" to figure out the field order (by eye, but it was extremely tricky). HuffYUV is indeed Bottom Field First, as expected.
However, MagicYUV (and UT) is strange. There are TWO problems actually. First, the spatial field order is wrong, second the field dominance is ALSO wrong!
Here is the order of the fields of the frames in time (each column is a frame in the video and the numbers represent the order when the given field should appear on screen):
HuffYUV:
Top: 2 4 6 8 10 ...
Bottom: 1 3 5 7 9 ...

MagicYUV:
Top: 1 3 5 7 9 ...
Bottom: 4 6 8 10 12 ...

To correct this, two things has to be done. First, use the "field delay" filter in "Top-field-first -> Bottom-field-first" mode to fix the temporal ordering. This will pair up the fields correctly. THEN use the "field swap" filter to fix the spatial ordering.

MagicYUV:
Top: 1 3 5 7 9 ... [field delay] 3 5 7 9 11 ... [field swap] 4 6 8 10 12 ...
Bottom: 4 6 8 10 12 ... ---------------> 4 6 8 10 12 ... --------------> 3 5 7 9 11 ...


You will loose 2 fields this way, but actually "field 2" wasn't even there in the original MagicYUV video!

Go ahead and try this! It really brings back the clip to it's original splendor!

There is some info here about the "field delay" and why it's different from field swap: http://neuron2.net/reverse/reverse.html

Now, as to why this happens? NO IDEA!
But let me put one thing straight: the codec (neither MagicYUV, nor UT) has NOTHING to do with this. You would get the SAME result if you disable interlaced encoding in the codecs. This messed up field ordering is ACTUALLY WHAT THE CODEC GETS! VirtualDub gives this crap to the codec to encode. Whose fault is this? Again, no idea. It could be the driver, it could be VDub, but one thing is sure: not the codec!
(Actually, it is impossible for the codec to do this kind of field trickery, as it would require it to do inter-frame shuffling, which is impossible for a keyframe-only codec!)


I personally like the HuffYUV Version best because it is "clearer" than Lagarith but it is well known that Lagarith does not handle Interlaced very well.

They should be the same (we're talking about mathematically lossless codecs here). Only if you make the codec convert to RGB might you see some difference (but I doubt).


There where no Filters or Conversions from YUY2, straight Capture to the Codecs. And i activated "Swap Fields" in VirtualDub, now guess what happened: Yes, MagicYUV and UtVideo looked less distorted but HuffYUV and Lagarith looked awful. So, there must be a Field Issue otherwise this would not happen.

Yes, swap fields only fixes half of the problem.


The funny thing: If i compress via "Fast Recompress" a HuffYUV NTSC Capture or DV-AVI to MagicYUV or UtVideo, the Field Order is correct and not reversed. The Field Order is only reversed when MagicYUV and UtVideo is used for a direct NTSC SD (BFF) Capture, PAL/SECAM SD (TFF) is fine.

Of course! Recompressing a correct HuffYUV video to MagicYUV will produce a correct MagicYUV video. MagicYUV does no trickery to the fields (neither UT for that matter).


I also tried to disable Interlaced in all Codecs except Lagarith. The Result with MagicYUV and UtVideo was that the Audio (PCM) was out of Sync with the Video and looked very poor. With HuffYUV it was a higher Bitrate and "Softer", in Sync with the Audio.

The interlaced setting has nothing to do with audio, so it was pure chance I guess. The interlaced setting is completely internal to all codecs, NOTHING can see it from the outside (neither VDub or any program, AVI has NO interlaced flag, only the user knows if a material is interlaced).


By the Way, it is not a good Idea to switch a YUY2 to UYVY on decompress because the Codec should always decompress in the Format it was compressed with even if they are similar. Every Color conversation costs Quality and after some Generations it is visible. You are right, MagicYUV decompresses in UYVY in "Fast Recompress" but it should not. One of the oldest Rules is to stay in the same Color Space until you are done to uphold the Quality.

YUY2 and UYVY are identical formats, only the byte-order is swapped around, but the Y/U/V values are the same to the bit. The link you sent about UYVY explains this.


AND now onto the raw VBI stuff: I have again completely no idea why turning it off fixes the problem. Just curious: does it fix the problem completely (like the fix I described above)?
Where is this VBI setting BTW? In the capture driver?

My very wild guess: something happens when VDub connects HuffYUV to the capture driver, which doesn't happen for MagicYUV. But sadly, I have no idea what it is.

Again, thanks for your extensive testing. I would be very interested why the VBI setting caused this, but it seems it'll be a long time before I find out... :(

Greets,
I.


Greets,
I.

[Ignus2]

Don't get me wrong, I am not trying to make MagicYUV or UtVideo "look bad" to push HuffYUV, it is just what has been discovered, nothing else. So don't go mad with me, okay ?


I'm not mad because any of that, but you seem to completely ignore and/or refuse to read what I write, and what I wrote in PMs as well.
- It's not a TFF/BFF issue, but a field dominance issue AND a field swap issue. Please read the link I posted, and the detailed dissection in my previous post.
- Neither codecs have to do anything with TFF/BFF (apart from HuffYUV optionally allowing to swap the fields).

And about the performance "measurement" you did: you still not detailed what programs you used, what colorspace, where did (if any) color space conversion happen (or didn't happen), etc.

Also, both UT and MagicYUV can handle any resolution, moreover, you didn't explain what is the exact problem by saying "neither MagicYUV nor UtVideo claimed to handle that Resolution".

Greets,
I.

[Mick]

@De-M-on
I am not talking about the normal NLE's here, I am talking about real Workstations based on Ingest Systems, not comparable with Adobe or other typical NLE candidates. These Guys work in Hollywood for the "big ones", other for big Film Studios and Company's around the World transferring and coding Material, do I need to say more ?

@Ignus2
Again, the Tests where NOT done with VirtualDub, my Colleagues monitored the same behavior on their Workstations, leave VirtualDub out at this time. Somewhere must be a Field-Bug if HuffYUV does it right and MagicYUV and UtVideo don't, right ?

So, let's cool down and I ask my Friends if they have a detailed Report and if so I publish it here, okay ? Maybe this clears this Mystery, Deal ?

Cheers
Mick

[Ignus2]

@Ignus2
Again, the Tests where NOT done with VirtualDub, my Colleagues monitored the same behavior on their Workstations, leave VirtualDub out at this time. Somewhere must be a Field-Bug if HuffYUV does it right and MagicYUV and UtVideo don't, right ?

So, let's cool down and I ask my Friends if they have a detailed Report and if so I publish it here, okay ? Maybe this clears this Mystery, Deal ?


It probably won't unless you thoroughly understand what the real problem is (reverse field dominance and field swap) and are willing to invest the time to investigate it.

EDIT:
Again: the codecs cannot by their nature affect field dominance. So the problem you see is beyond the codecs.
Also, I think it's simply not professional to make claims (like this is faster than that) and then not being able to present supporting reports/evidence to back it up.

Greets,
I.

[WorBry]

@Ignus2,
HuffYUV offers a Option to switch the Field Order if it got upside down. I don't know either why HuffYUV works with both Field Orders but it is Important because there was never a real standard for this, besides PAL/NTSC SD DV that is always BFF and PAL/NTSC HD which always uses TFF.


"At the mouth of two or three witnesses...."

http://forum.videohelp.com/threads/235270-Little-confused-about-huffyuv?p=1379263&viewfull=1#post1379263

Quote:

"HuffYUV encodes interlaced and progressive frames exactly the same. So it doesn't need to know if the frames are interlaced or progressive. The "swap fields" option is only there to correct a bug where some old cards captured with the scanlines swapped -- it isn't to be used to specify field order"

Plain enough.

[kolak]

"MagicYUV" and "UtVideo" are good lossless Codecs, no doubt, but "HuffYUV" still rules for SD/HD/UHD Content and "Lagarith" for SD with YV12. Many Colleagues of mine around the World in Mastering Labs tested MagicYUV and UtVideo with the result that they stay with HuffYUV (CCESP Patch 0.2.5/MT and 2.1.1 64 Bit). They found MagicYUV and UtVideo "interesting" but not for professional use for Mastering.

In Short:
MagicYUV and UtVideo can get painfully slow on decompression if Dolby Surround, Digital (5.1) or DTS is added. Encoding with MagicYUV and UtVideo can end in a "freeze" for several Seconds if 2K is used and both freeze fast with higher Resolutions than 4K. Tested on 32/64 Bit Hig-End Systems and 32/64 Bit Codec Versions.

With HuffYUV never a Problem and BTW, HuffYUV uses much less CPU than all the other lossless Codecs while encoding/decoding Videos. (32/64 Bit Versions)

Here you can compare yourself:

Codec Setups:
https://yadi.sk/d/9NzZJc3Hcumhx

Test Videos: (A short Sequence with kind permission of PolyGram)
https://yadi.sk/d/X5r8s7HVcurL6

The Test Videos are in NTSC SD DV with BFF (Bottom Field First). You will notice that HuffYUV has the best Field Separation and is clearer than Lagarith. MagicYUV and UtVideo could not handle this Field Order, only TFF (Top Field First).

TFF = Odd/Upper Field first
BFF= Even/Lower Field first

Please note that the Videos have the same Sequence/Length and uses the same Settings inside the NLE to get a reliable Result for comparison.

The "To-do" List for MagicYuV and UtVideo:
The "Chroma" Problem
The Gamma Problem with QuickTime (Win uses 2.2, MAC 1.8)
The Field Order Problem (only the "Top Field" is considered)
Both Codecs don't support RAW-VBI
Dual Image 3D not well supported

UtVideo: v210 leaves a Black/Green Screen and stutters inside professional NLE's with a YCbCr Card and plays only the first 10 Frames before the Playback freezes inside the NLE's.

I do believe that both, MagicYUV and UtVideo can be a excellent choice if some Issues are finally fixed by the Authors.

@Ignus: You still have unread PN's, please read, just trying to help, okay ? ;)

Cheers
Mick


Your problem is that your are trying to use codecs in ingest systems, which were never tested with them. This is most likely to have many problems. Just stick with uncompressed capture or get something which is tested if you want reliability. You are trying to match some legacy/quite strict systems with custom made codecs. If you want this than talk to Drastic- they can make you proper integration in MediaNXS. You mentioned Cinegy- it's enough that their system doesn't support pixel format which magicyuv requests, so if there is a need for some conversion (and this conversion is not optimised) you end up with big penalty in speed. Talk to Cinegy. Problem with pro systems is that they are very limited, but this is due to reasons. If there is one bit more open than don't expect it to work perfectly with all codecs out there. Maybe Cinegy did some work to make HuffYUV working well. Cinegy won't talk to magicyuv directly but through e.g. vfw. This is not what you want in pro system.
MagicYUV is veeeeery fast and you can easily build 4K ingest system on low spec machine, but this requires proper integration on low level (over SDK)- not over QT or vfw/diretcshow.
MagicYUV is the only codec which can natively take 10bit YUV/RGB based pixel formats used by BlackMagic and AJA cards and compressed them. This is 100% reliable and than the same can be done on the decoding stage, so you end up with 100% same data. Any problems with field order, chroma conversion etc are bit different story.

Pro companies don't touch these sort of codecs- they work with DPXs, v210 etc, so not sure who are your colleagues around the world. Sometimes they use compressed formats, mainly ProRes these days, but this is over properly integrate systems, so they never end up with problems like yours.
If you really want to use losses codecs try talking to James at Drastic or with someone at Cinegy.

[Mick]

@WorBry
Quote:
"HuffYUV encodes interlaced and progressive frames exactly the same. So it doesn't need to know if the frames are interlaced or progressive. The "swap fields" option is only there to correct a bug where some old cards captured with the scanlines swapped -- it isn't to be used to specify field order"

This Statement is simply wrong because HuffYUV considers more than 288 Lines as Interlaced. For Progressive Content like 1280x720 this needs to be set to 720 instead of 288. Did you ever read the Codec Description ? And the "Swap Field" option is for broken Capture Drivers ONLY (!) that use the Field Order upside-down. Checking this Option corrects this behavior.

Quote:
"At the mouth of two or three witnesses...."

This Statement is simply sad, very sad, no further comment....

@kolak
I use the Drastic YCbCr Codecs now for a long time and i know the Guys at Drastic and Cinegy well, don't worry about that.

Quote:
"Pro companies don't touch these sort of codecs- they work with DPXs, v210 etc, so not sure who are your colleagues around the world."

I use RAW (Uncompressed) just like my Colleagues and it is irrelevant here who my Friends are, they lift the Leg with the Big Guys, are well known, often credited in Movies and I certainly don't publish their Names here. Some use lossless Codecs for different purposes. No hard feelings, okay ?

@Ignus2
Quote:
"It probably won't unless you thoroughly understand what the real problem is (reverse field dominance and field swap) and are willing to invest the time to investigate it. Again: the codecs cannot by their nature affect field dominance. So the problem you see is beyond the codecs."

I do understand the Problem very well and the Solution to it, don't worry. The Codecs can't AFFECT (!) the Field Order, right, but the AVI Container does. I suggest you get the Whitepapers for the AVI Container Format where everything is clearly defined and explained.

AVI1 (VfW) does NOT save the Field Order and is limited to 2GB, AVI2 (OpenDML) DOES save the Field Order and I work and transfer with OpenDML and my Capture Module from Cinegy uses this Information written in the AVI2 Header.

Quote:
"Also, I think it's simply not professional to make claims (like this is faster than that) and then not being able to present supporting reports/evidence to back it up."

Okay, now read closely: This is something what my Colleagues and me monitored. I've spent hours on the Phone, my Workstation and invested my Money and my Time being apart from my Family trying to help you to share what I found out.

And now you're sort of saying that I am a stupid Idiot ? Well let me tell you that I am doing this for a long Time for major Company's and TV Stations around the Globe, I am not a "newbie" in this Field and if this is your way of saying "thank you", well bless you too.

Go on, keep ignoring facts and be happy to have the "fastest" codec around but I am out at this point. I thought Doom9 was different, until today. I've sent you everything backed up with Links to the Web in your PN, many times and most of them you never read. I suggest you don't ask for help or suggestions next time you work on a Project like "MagicYUV".

And once again: It was NEVER my Intention to make "MagicYUV" or "UtVideo" look bad ! Both Codecs have a great potential but still have some problems I wanted to share with you and other Users, that is all and I hope we are clear on this now ! This Chapter is closed for me and I am not willing to invest anything any further because I have plenty other Things to do.

Case closed.

Mick

[kolak]

I've done 1000s of DVDs and 100s of Blu-ray projects using lossless codec as an intermediate format, so I agree with you that there are different needs. You just have to understand your workflow and make sure it's robust. This what you complaining is not directly related to codec itself. You have to understand this.

[Mick]

@kolak
I am not "complaining" about the Codecs "MagicYUV" or "UtVideo", you got me wrong in this aspect, I was trying to help because "MagicYUV" and "UtVideo" can be very good alternatives to other lossless Codecs for certain purposes when needed, once the little things are fixed. A robust Workflow ? Oh yes, very important and underestimated by many :)

Cheers

Mick

[Ignus2]

@WorBry
Quote:
"HuffYUV encodes interlaced and progressive frames exactly the same. So it doesn't need to know if the frames are interlaced or progressive. The "swap fields" option is only there to correct a bug where some old cards captured with the scanlines swapped -- it isn't to be used to specify field order"

This Statement is simply wrong because HuffYUV considers more than 288 Lines as Interlaced. For Progressive Content like 1280x720 this needs to be set to 720 instead of 288. Did you ever read the Codec Description ?

No, WorBry's statement is correct, if there is no in-codec color-space conversion involved, you get back the same result whether interlaced encoding is turned on or not.
Make a test encode or read the codec source code if you have doubts.


And now you're sort of saying that I am a stupid Idiot ?

I haven't said that, so don't put words in my mouth that I didn't say.


Go on, keep ignoring facts and be happy to have the "fastest" codec around but I am out at this point. I thought Doom9 was different, until today. I've sent you everything backed up with Links to the Web in your PN, many times and most of them you never read. I suggest you don't ask for help or suggestions next time you work on a Project like "MagicYUV".

I get plenty of help from other people here on doom9 who are truly willing and meaning to help.
Still awaiting your facts though...


Case closed.

The best thing to do when you run out of arguments...

Greets,
I.

[Mick]

@Ignus2
I wrote:
"And now you're sort of saying that I am a stupid Idiot ?"

Your reply:
"I haven't said that, so don't put words in my mouth that I didn't say."

Right, you did not say that but reading between the Lines of your replies it comes over like that to me. Correct me if I am wrong in this, okay ?

You wrote: (About HuffYUV Swap Field)
"No, WorBry's statement is correct, if there is no in-codec color-space conversion involved, you get back the same result whether interlaced encoding is turned on or not."

Wrong, please read for yourself:

HuffYUV Documentation:

Field Threshold
When I added the YUY2 stuff described above, I discovered a hardcoded threshold that will trigger progressive or field based compression. Something that was not acceptable, as NTSC has a threshold of 240, Progressive Frames have none at all. I simply added another entry in huffyu.ini and added a way to edit it in the configuration dialog. You can change the value to any number between 1 and 16384.

If you're processing NTSC video, you should set it to 240, so that material with more than 240 lines will use interlaced compression. For PAL the original value of 288 is right.

If you are doing progressive material, setting it to 480 or 576 lines will compress even video at this resolution progressive. This should result in better compression.

If you set this option correctly you will get properly separated fields, if not, your fields will be blended, or you will have lower quality.

Swap fields on decompress
Some capture drivers are broken and get the field order backwards. If you're stuck with one of these, you can compensate by checking this option--proving once again that two wrongs make a right.
Source: HuffYUV Documentation

I wrote:
"Case closed."

You wrote:
"The best thing to do when you run out of arguments..."

Well, I don't have "Arguments" but some facts from some Whitepapers:

Raw VBI (Whitepaper)

Raw Video Data Output:
The vertical blank interval (VBI) data processor (VDP) slices various data services like wide screen signaling (WSS) to the hardware and compressor.

These services are acquired by programming the VDP to enable a standard(s) in the VBI. The results are stored in a FIFO and/or registers. (ITU-R BT 601)

Available VBI lines are from line 6 to line 27 of both field 1 and field 2. Theoretically, each line can be any VBI mode because the VDP processes VBI data in a line by line. When changing modes, the VDP must allow the current transaction to complete through the delays of the VDP before switching the line mode register’s contents. It must also complete the loading of the line mode registers before the next line starts processing.

This format is also used to store any VBI data into the FIFO. The size of FIFO is 512 bytes.

Sliced VBI data can be output as ancillary data in the video stream in ITU-R BT.656 mode. VBI data is output during the horizontal blanking period following the line from which the data was retrieved.

Raw VBI Synchronization Signals
Non-data stream embedded syncs are provided via the following signals:
VSYNC (vertical sync)
FID/VLK (field indicator or vertical lock indicator)
GPCL/VBLK (general-purpose IO or vertical blanking indicator)
PALI/HLK (PAL switch indicator or horizontal lock indicator)
HSYN (horizontal sync)
AVID (active video indicator)

In hardware, VSYN, FID, PALI, and VBLK are software set and double buffered to an independent programmable SCLK pixel count. This allows any possible alignment to the internal pixel count and line count.
Sources: Texas Instruments Whitepaper, Phillips Whitepaper, Empia Whitepaper


AVI2 (OpenDML Whitepaper)

60 Fields Per Second vs. 30 Frames Per Second; or 24 Frames Per Second For Film:
Professional Video Applications require the ability to sequence individual fields of the video data for certain playback rates and video effects. For example, in order to do a slow motion effect, it is not sufficient to just repeat the frames; the fields must be individually repeated.

For example, as sequence of fields in a file numbered 123456... should be played at half speed as 11223344... rather than 12123434.... As such, the individual fields should be accessible. AVI2 requires that frames are stored per data chunk.

The extended AVI2 format allows access to individual fields. In addition, the format will allow for storage of 24 frame based files, such that file readers can properly convert between 24 frame-based data storage and 25 or 30 frame PAL and NTSC playback.

AVI2 Field Index Chunk:
The AVI2 Field Index Chunk is the same as the Standard Index Chunk except that it contains the locations of each field in the frame.

The AVI2 spec lists that heights less than 288 are for single frames (one chunk) and greater than 288 are for interleaved fields (interlaced frames).

The biHeight parameter refers to the raw height of the frame (interleaved or not). OpenDML codecs that operate in field by field modes will be passed this value divided by 2 to get the field height.

AVI2 Field Indexing:
In order to support field accurate indexing while still retaining AVI compatibility, the video data must be stored one frame per ##dc ‘movi’ chunk, while an index must be present that gives the locations of both fields (if they exist) within the frame. Applications could then use field-based codecs to perform field-based effects (e.g., slow motion, etc.).

Improved AVI2 Tombstone and Header Information:
Professional video applications need extended information such as starting timecode and reel ID to be contained inside the AVI2 file along with more advanced information such as the location of timecode discontinuities, wide screen, etc.
Sources: Matrox Whitepaper, Microsoft Whitepaper, Empia Whitepaper


You wrote:
"Still awaiting your facts though..."

Here are some more facts:

I've spend the past hours on the Telephone conferencing with my Guys and noted down that what they found out about MagicYUV and UtVideo. They authorized me to publish the following Information, not the rest, sorry, but there are confidential rules.

Source Format:
YCbCr 4:2:2 YUY2 16 Bit Uncompressed
Video: NTSC 4.43, 29.970 Fps at 720x480 with BFF
Audio: 48.000 Hz, 16 Bit, Stereo, PCM
Source: S-VHS via Y/C (BNC)

Export with MagicYUV and UtVideo, no FX, BFF, same as source.
Input Monitor: YUY2 (Source)
Output Monitor: YUY2 (MagicYUV/UtVideo)
CPU Load: Average at 45-62 % for both Codecs (8 Threads)

Re-Import MagicYUV/UtVideo Video to Timeline, same Setup:
Input Monitor: UYVY with TFF (!), not BFF
Playout Monitor to VTR: UYVY, TFF (!)
Original Source to VTR: YUY2, BFF

I420, IYUV: MagicYUV decodes them as UYVY, NOT I420/IYUV,TFF (!)
YV12: MagicYUV/UtVideo decode this at UYVY, NOT YV12, TFF (!)

Export with HuffYUV/Lagarith, same setup:
HuffYUV/Lagarith decode YUY2 back to YUY2 with BFF.
Lagarith decodes YV12 back to YV12 with BFF.
CPU Load with HuffYUV (MT): around 9-20 % (8 Threads)
CPU Load with Lagarith: around 55-72 % (8 Threads)

My Guys verified this behavior with VirtualDub (1.10) on a Win7 Ultimate (64 Bit) using "Fast recompress" using YCbCr (no Codec) with the 64 Bit versions of MagicYUV/UtVideo.

Result: Saved as UYVY with UtVideo and MagicYUV instead of I420,IYUV,YV12 or YUY2 with TFF instead of BFF. Lagarith decoded as YV12/YUY2 with BFF and HuffYUV (2.1.1 64 Bit) as YUY2 with BFF. All tests where done with the same setup, frame rate and resolution, same conditions on all codecs for Import and Export.

So, fact is: MagicYUV and UtVideo unify to UYVY on the Output rather than decoding the source Color Space with TFF, not BFF which brings me to this Section.

Section "UYVY is the SAME as YUY2"

VirtualDub Help says:
4:2:2 YCbCr (UYVY)
This is a format which uses the YCbCr color space (luma, chroma-blue, chroma-red), which is closer to the way color images are perceived by the human brain. It averages only 16 bits per pixel with similar perceptual quality to 24-bit RGB by only storing color information at half horizontal resolution, producing slight color bleeding but only taking two-thirds as much space.

This format, as do all other YCbCr formats listed below, encodes luminance (Y) with a range of [16, 235] and chroma (Cb/Cr, or U/V), with a range of [16, 240].

UYVY is accepted directly by many video codecs. Since many video codecs internally use color spaces similar to YCbCr, using this format with video codecs can speed up rendering.

4:2:2 YCbCr (YUY2) (!)
This is the same as 4:2:2 YCbCr (UYVY), except for a shuffling of data bytes. It has the same quality and performance advantages as UYVY.

Correction
(!) = This is NOT WRONG ! Sorry, my fault.

Microsoft says:
The YCbCr YUY2 format uses a slightly different byte order for data and chroma compared to the YCbCr UYVY format. All trans-codes between YUY2 and UYVY are done with a new chroma byte and data order which results in no losses in quality over time BUT TO RGB if done too often, even if both formats are very similar.
Source: Microsoft YUV Format Whitepaper

I talked about this with my Guys as well and they totally agree that this is a total misbehavior of MagicYUV and UtVideo. "Lossless" means something else to me. They also downloaded the most recent Versions of MagicYUV and UtVideo and made no changes in the result.

The best results where made with MagicYUV and UtVideo using UYVY as a Source with TFF, but for NTSC Archives not really useable. They also said that all Options for "RGB Output" where disabled on all Codecs because it's a YCbCr Input/Output setup.

So, before you go up the Wall, these are just facts, nothing else, okay ? I have nothing against you or the Author of UtVideo nor I am trying to make MagicYUV ot UtVideo look bad, but there are some serious Bugs that are responsible for this behavior.

Plus, HuffYUV and Lagarith work as expected and don't change the Color Space on the decoding part nor the Field Order under the same (!) conditions, they decode that what they been fed with including the correct Field order. Why they work with the right Field order and Color Space, I don't know.

Here you have it, all you need and please share this with the Author of UtVideo because all these things are important if both of you want your Projects to work. I wish you both a big Mountain of Luck. This is all I can do at this end and hope you find out why the Codecs behave this way.

Over and out.

Mick

P.S.: They also downloaded my Samples I linked here as well. They verified that my Samples behave exact the same Way like their own Samples.

[vivan]

Microsoft says:
The YCbCr YUY2 format uses a slightly different byte order for data and chroma compared to the YCbCr UYVY format. All trans-codes between YUY2 and UYVY are done with a new chroma byte and data order which results in slight losses in quality over time if done too often, even if both formats are very similar.
Source: Microsoft YUV Format WhitepaperWow, this is golden. Apparently moving bytes too much destroys them!

P.s. jesus christ, learn how to quote.

[poisondeathray]

@Mick -
I'm sure everyone appreciates that you're trying to add feedback to improve the software, but those are mostly observations, not facts. Facts are independently verifiable.

It looks like a problem or incompatibility with your hardware card and software setup - this is almost certain

When used properly, the both magic yuv and ut video are bit for bit lossless . This is easy to prove and independenly verifiable over "n" generations (That is a fact, not obseration. ) . Many people can independently verify that

You can interpret the field order as you want in most software. Or is that your issue ?

It's definitely a software issue because your software is upscaling the chroma on I420 . With other software it decodes correctly. Some NLE's actually hand off RGB from UT or magicyuv from YUV input, so it's not only your software that is a problem, other titles can have issues as well

If you cannot name the software used due to NDA concerns , or if it's in-house software title not much anyone can do to help





Do you have a link to the MS whitepaper ? There is no quailty loss between YUY2 and UYVY conversions. The original whitepaper from 2002 said nothing of the sort. I can find no reference about quality loss either. Test it yourself as I have, over "n" iterations there is no loss if you do it properly.

(Technically, there is no "transcoding" between YUY2 and UYVY . The term is used loosely now, but transcoding, by definition involves requantizing DCT coefficients at a lower precision to save bits. It's always lossy process.)

[Ignus2]

Right, you did not say that but reading between the Lines of your replies it comes over like that to me. Correct me if I am wrong in this, okay ?

You are wrong in this.
However there are many things you are getting wrong and are confused about. This is not stupidity. But you should consider the possibility that you are wrong about things. Others have also began pointing this out in your messages.
This is not a problem.
However, there are things others know better than you, even though you have many years of experience. So don't stick to your guns, take a step back and try to see where you are wrong.


You wrote: (About HuffYUV Swap Field)
"No, WorBry's statement is correct, if there is no in-codec color-space conversion involved, you get back the same result whether interlaced encoding is turned on or not."

Wrong, please read for yourself:

HuffYUV Documentation:
...

There is no need to read the docs, as I can read the source code as well. What I have said is:
"If there is no in-codec color-space conversion involved, you get back the same result whether interlaced encoding is turned on or not."
Which is verifiable by reading the HuffYUV sources. HuffYUV does no blending of fields whatsoever if you give it YUY2 and ask YUY2 back. It will give you back the same result if you encode and decode in the same colorspace whether interlaced encoding is turned on or not.
You can verify this by doing an encode-decode cycle with both settings and check the result.


If you set this option correctly you will get properly separated fields, if not, your fields will be blended, or you will have lower quality.

This statement is in effect ONLY if HuffYUV does color-space conversion, for example when decoding compressed YUY2 data as RGB or compressing RGB as YUY2.


Raw VBI (Whitepaper)
...

Again: neither HuffYUV, MagicYUV or UT codecs' VFW versions don't have any sort of ability to access VBI information. This is again verifiable by source code and the fact that VFW simply has no way of feeding this info to the codecs.


AVI2 (OpenDML Whitepaper)
...

What you quoted, unfortunately has no additional information to the problem at hand.


...
So, fact is: MagicYUV and UtVideo unify to UYVY on the Output rather than decoding the source Color Space with TFF, not BFF which brings me to this Section.

This is an observation, not a fact. The codecs output a format what they are asked to output by the application. They can also propose a format, but ultimately it is the application which decides what to ask from the codec. In your case, the app asks for UYVY for whatever reason.
Also, the big difference in the CPU load clearly indicates, that there is some color-space conversion (or something else) going on in the background. This is the source of all problems, but unfortunately, it originates from someplace else than the codecs.
Again: the codecs do what they are asked to do.


Microsoft says:
The YCbCr YUY2 format uses a slightly different byte order for data and chroma compared to the YCbCr UYVY format. All trans-codes between YUY2 and UYVY are done with a new chroma byte and data order which results in slight losses in quality over time if done too often, even if both formats are very similar.
Source: Microsoft YUV Format Whitepaper

I don't know what whitepaper is this, and I'm sorry to say this, but this is just plain stupid.
Do a google search on "UYVY vs YUY2" and you'll find out for yourself.


So, before you go up the Wall, these are just facts, nothing else, okay ? I have nothing against you or the Author of UtVideo nor I am trying to make MagicYUV ot UtVideo look bad, but there are some serious Bugs that are responsible for this behavior.

And these bugs are most certainly in the setup (which I would very much like to find out as well), and not in the codecs.


Plus, HuffYUV and Lagarith work as expected and don't change the Color Space on the decoding part nor the Field Order under the same (!) conditions, they decode that what they been fed with including the correct Field order. Why they work with the right Field order and Color Space, I don't know.

Again, MagicYUV and UT also don't change colorspace unless they are asked to do so.
BTW: MagicYUV specifically has options to prevent conversions on the output ("Decompression settings"). Disable all the checkboxes there, and there will be no conversions, apart from the lossless byte/plane-swapping variants.

I'm a bit tired now, so sorry if I forgot to reflect on something.

Greets,
I.

P.S.: The next version of MagicYUV will have logging support, so then we'll be able to see what it gets and what it is asked to output.

[kolak]

Mick, as said before- it's all down to your specific setup and not much to do with codecs itself.

Lossless codec in the proper/best case scenario takes data as is (in on of the supported pixel formats: YUY2, v210 etc), compresses it and that's it. Because it's lossless compression when you decompress it (to the same pixel format which it was compressed from) than result is always 100% the same as source data! It's the same as zipping file.

Your software/hardware is responsible for proper field order interpretation and this is outside of the codec.

This is why I said talk to Drastic/Cinegy. They can do proper magicyuv implementation, so whole transcoding chain can be done at cards native pixel formats (magicyuv support all needed ones) and whole chain will be robust.

VBI is past and again- has nothing to do with codec itself. Still used by old payout systems, but in most cases it's not needed anymore (can be blank). It's basically unsupported in new capture cards: AJA, BM etc.

Not sure why you can't understand that problem is not inside magicyuv or utvideo, but in the integration between your ingest system and codecs.

[kolak]

Did you say that you have 45-62% CPU load on realtime encoding to magicyuv for NTSC on 8 core system?
Or is it during export?
Realtime encode or playback for NTSC on 8 threads system should have <5% CPU load, not 50% (or your system is ancient).

[Mick]

@Ignus2
I am glad that we cleared this point between us so no hard feelings, okay ? ;)

About HuffYUV: Sorry, but we are talking about different things here, so let me try to clarify this to you.

Your Quote about Interlaced:
"If there is no in-codec color-space conversion involved, you get back the same result whether interlaced encoding is turned on or not."

HuffYUV Documentation:
"Field Threshold
When I added the YUY2 stuff described above, I discovered a hardcoded threshold that will trigger progressive or field based compression. Something that was not acceptable, as NTSC has a threshold of 240, Progressive Frames have none at all. I simply added another entry in huffyu.ini and added a way to edit it in the configuration dialog. You can change the value to any number between 1 and 16384.

If you're processing NTSC video, you should set it to 240, so that material with more than 240 lines will use interlaced compression. For PAL the original value of 288 is right.

If you are doing progressive material, setting it to 480 or 576 lines will compress even video at this resolution progressive. This should result in better compression.

And this is the same behavior in AVI2 OpenDML:
"The AVI2 spec lists that heights less than 288 are for single frames (one chunk) and greater than 288 are for interleaved fields (interlaced frames)."

Sometime reading a Documentation can help ;)

Correction about UYVY vs YUY2: I WAS WRONG, Sorry ! What I posted related to converting UYVY/YUY2 Footage to RGB and back to UYVY/YUY2 which was later described in the Whitepaper and I've overseen, so, sorry for that. There are no losses converting between UYVY to YUY2 and back. Again, sorry, my fault.

Here is a interesting Link about YUV Formats from Microsoft:
https://msdn.microsoft.com/en-us/library/windows/desktop/dd206750%28v=vs.85%29.aspx

About RawVBI where you wrote:
"Again: neither HuffYUV, MagicYUV or UT codecs' VFW versions don't have any sort of ability to access VBI information. This is again verifiable by source code and the fact that VFW simply has no way of feeding this info to the codecs."

Well, I use AVI2 (OpenDML, and NOT AVI1 VfW ;) because OpenDML offers the implementation for this. The Codecs don't "access" the Information, the Signals are transferred WITH the Codecs and placed in the AVI2 OpenDML Header, it's a big difference. You see, I use a Play-In/Play out Ingest System that works in Real Time from Tape to Disk or direct to the Timeline with Synchronization (LE/NLE) and then out to Disk or Tape.

Remember the good old Days of DV and Firewire ? ;)

I give you and the others here a rough Idea of what I mainly do: Many Company's and TV-Stations around the World send me, for example Tapes (VHS) that never made it on DVD which they want to sell now as a DVD or for their Archives on Disk, TV-Stations send me their S-VHS Archives via Courier/Parcel Service to preserve them for the Digital Century in the Format they want. I have to deal with all kinds of Material and TV-Norms from all over the World from different Country's.

I transfer, edit and master the Audio/Video Material for them and a Courier/Parcel Service picks the Tapes along with the Disks or DVD's up when I'm done, depends on the Client and what they ordered. So you see, I depend very much on proper synchronization and all my Multiform Decks offer this, that's why I still use RawVBI along with the other Signals involved. Even on VHS/S-VHS the Wide Screen Flag (WSS) exists and many have been distributed and sold with it.

And you're right, these are observations and experiences of my Colleagues and me but still can't be simply wiped of the Table or ignored.

@kolak
You wrote:
"Not sure why you can't understand that problem is not inside magicyuv or utvideo, but in the integration between your ingest system and codecs."

and poisondethray wrote:
"I'm sure everyone appreciates that you're trying to add feedback to improve the software, but those are mostly observations, not facts. Facts are independently verifiable.

It looks like a problem or incompatibility with your hardware card and software setup - this is almost certain"

Thanks for the Tip Guys, you where right because I took a look in the Cinegy Manual and the Section for "Supported Codecs". Now guess what I found there:

"There is no support for a free-form codec selection for AVI export except the ones listed below."

HuffYUV is on the List and with Lagarith I got "lucky" because Lagarith is not on the List, but works fine.

@kolak
You wrote:
"Mick, as said before- it's all down to your specific setup and not much to do with codecs itself."

I understand that and know how lossless Codecs work, but there are some things that really puzzle me because I really like MagicYUV and UtVideo and observed some things I can't find a solution or Answer for. The only thing I found out was, disabling RawVBI with MagicYUV and UtVideo solves the Problem of their strange behavior compared to HuffYUV and Lagarith. But since RawVBI is no longer supported since Vista, it's obvious that this never came up and no one noticed.

RawVBI on:
MagicYUV = TFF with NTSC (SD)
UtVideo = TFF with NTSC (SD)
HuffYUV = BFF with NTSC (SD)
Lagarith = BFF with NTSC (SD)

RawVBI off:
MagicYUV = BFF with NTSC (SD)
UtVideo = BFF with NTSC (SD)
HuffYUV = BFF with NTSC (SD)
Lagarith = BFF with NTSC (SD)

With "RawVBI off" the Image was a bit softer with MagicYUV and UtVideo compared to HuffYUV and Lagarith where HuffYUV has the best Field separation and sharper Image compared to Lagarith. All Tests where done with the same (!) Footage and Conditions/Settings for Export to Disk.

Then you wrote:
"Lossless codec in the proper/best case scenario takes data as is (in on of the supported pixel formats: YUY2, v210 etc), compresses it and that's it. Because it's lossless compression when you decompress it (to the same pixel format which it was compressed from) than result is always 100% the same as source data!"

Right, but that does not happen with MagicYUV and UtVideo. Here is what we observed:

YUY2 is decoded as UYVY with MagicYUV (all decoding options off) and UtVideo, basically the same, no loss but still, HuffYUV and Lagarith decode to YUY2.

I420/IYUV is decoded to YV12 by MagicYUV and not as I420/IYUV. Here is a swap in the planar color space. YV12 is decoded as UYVY with UtVideo, means, up-sampled from 4:2:0 to 4:2:2 where Lagarith decodes YV12 if the source was YV12. v210 is decoded as RGB and not as YCbCr, same with VirtualDub exporting a Video with uncompressed v210. UtVideo and VirtualDub show on some Systems a Black Screen, on others a Green Screen before they hang after a long pause.

MagicYUV and UtVideo negotiate with other Codecs (Xvid/x264vfw) UYVY, even if I420/IYUV/YV12/YUY2 was used as a source to compress the Video (fast recompress) while HuffYUV uses YUY2 and Lagarith YUY2/YV12.

All this also happens on any "normal" System (Win7 x86/x64) with VirtualDub (1.6 to 1.10) or other Consumer/Semi-Pro/Pro NLE's installed, not just on our Systems because we tried that too with RAW 4:2:2/4:2:0 Videos to be sure ;) The Drastic Codecs are innocent because this happens with the native Windows Codecs too (I420/IYUV/UYVY/YUY2) and using the native Intel I420/IYUV Codec results in a RGB24 Output and Playback.

And the high CPU Load ? No Idea but it's the truth. Here are some Images a Colleague from Germany provided me with and is about the same what we saw on our Systems. The Source was a 5 Minute 4K Footage on a Win7 x64 Ultimate System. "Green" is the CPU Load, "Red" the Kernel Load on the Diagrams (7 Images), you can download the File "CPU-Load.7z" here:

http://ge.tt/2N8ZQhJ2/v/0?c

"HY" is HuffYUV (MT), "MY" is MagicYUV, "UT" is UtVideo and Image 07 is the CPU-Load of the Input Uncompressed. "Enc" means "Encoding" and "Dec" means "Decoding". He also said that these are the absolute MAXIMUM Measurements he monitored, depending on Movement, taking off that maximum about 20-30 percent gives you a good average Overview of the CPU Load. The Diagrams only show 2 of 8 Threads used because the other Threads had the same CPU Load as the first two.

@All
So, this is really all I can do to help and hope it helps to figure out the strange behavior of MagicYUV and UtVideo. And again, I really think that MagicYUV and UtVideo are great codecs and I am not trying to make them look bad because this is not the case at all, just trying to help, nothing else. I also would like to thank everybody here in this Thread for their patience, tips and helpful thoughts.

Cheers
Mick

P.S.: It's way to hot at the moment here in France... Puuhhh...

[kolak]

You are getting closer and more and more confirms that problem is integration between Cinegy and codecs (not tested/optimised).

Vdub has no problems at all with magicyuv or utvideo. If you set correct pixel format in Vdub settings than it will talk to both codecs through YUY2 or even v210. If you see RGB it means that you have wrong (most likeky default) settings. If your source is magicyuv (encoded from v210) and you see black/green picture in Vdub than you have wrong setting in decompression format (magicyuv has no internal conversion between 10 and 8bit). It has to be set to v210.
Both magicyuv and utvideo work very well in Edius NLE if files were encoded as 422 YUV. This is because Edius engine 'likes' YUY2 pixel format. Many software will ask codec to provide RGB, so you end up with conversion.

Magicyuv 25p 4K playback through v210 pixel format directly to madVR renderer in mediaplayer classic was using <20% of 6 cores 3.2 GHz E5 Xeon (HP Z420). SD files play at tiny CPU load on such a machine. Key point is to avoid any pixel format conversion. Ideally in pro solutions you want direct integration through codec SDK, not over WIN system vfw etc.

UYVY and YUY2 are “the same“ and conversion between them is trivial, so even if you see this, it's 100% lossless and fast. There is NO picture quality difference whatsoever, so if you see leess sharp footage than there are some serious problems with decoding chain.

As I said- write to Cinegy, so maybe they can debug and add proper support for magicyuv/utvideo.
If you want to test pure speed of the codec use tool provided by Ignus2 on his website. Most NLEs etc. will give totally inaccurate results.

[poisondeathray]

Then you wrote:
"Lossless codec in the proper/best case scenario takes data as is (in on of the supported pixel formats: YUY2, v210 etc), compresses it and that's it. Because it's lossless compression when you decompress it (to the same pixel format which it was compressed from) than result is always 100% the same as source data!"

Right, but that does not happen with MagicYUV and UtVideo. Here is what we observed:

YUY2 is decoded as UYVY with MagicYUV (all decoding options off) and UtVideo, basically the same, no loss but still, HuffYUV and Lagarith decode to YUY2.

I420/IYUV is decoded to YV12 by MagicYUV and not as I420/IYUV. Here is a swap in the planar color space. YV12 is decoded as UYVY with UtVideo, means, up-sampled from 4:2:0 to 4:2:2 where Lagarith decodes YV12 if the source was YV12. v210 is decoded as RGB and not as YCbCr, same with VirtualDub exporting a Video with uncompressed v210. UtVideo and VirtualDub show on some Systems a Black Screen, on others a Green Screen before they hang after a long pause.

MagicYUV and UtVideo negotiate with other Codecs (Xvid/x264vfw) UYVY, even if I420/IYUV/YV12/YUY2 was used as a source to compress the Video (fast recompress) while HuffYUV uses YUY2 and Lagarith YUY2/YV12.

All this also happens on any "normal" System (Win7 x86/x64) with VirtualDub (1.6 to 1.10) or other Consumer/Semi-Pro/Pro NLE's installed, not just on our Systems because we tried that too with RAW 4:2:2/4:2:0 Videos to be sure ;) The Drastic Codecs are innocent because this happens with the native Windows Codecs too (I420/IYUV/UYVY/YUY2) and using the native Intel I420/IYUV Codec results in a RGB24 Output and Playback.


Thanks for reporting back and providing more info

It's true, not all codecs will decompress to the identical input pixel arrangement, but it does not matter. This is true for all codecs, not just ut video and magicyuv. You can change it to whatever pixel arrangement you want, e.g if you start with YUY2 , and a certain decoder or application decodes to UYVY, or Y422, or UYNV, or HDYC, or YVYU , etc.. - if doesn't matter what arrangement you use (ie. 4:2:2 is 4:2:2, it doens't matter if it's planar or packed or what order - this has no effect on quality) . There is no loss if you convert between them correctly - that was already discussed. Huffyuv is no different here - for example, if you give it HDYC, it won't give you HDYC back. I think it would be a lot of work to keep track of all possible input/output pixel arrangements

There are some circumstances where a certain fourcc is desireable - in the case of certain host applications, there are "preferred" fourcc's and pixel arrangements which get preferential treatment and pass through, but this is slightly off topic

However, your UT video observation is cause for worry , because an up or downsample would cause quality loss. Also, if UYVY is always used to negotiate with other codecs, that would also cause quality loss from any 4:2:0 input. If it were true, then the decoded output would be detected as different than the original (ie. NOT lossess) by any method e.g. PSNR, SSIM, Amplifie, Difference or Subtract testing etc... I think this might be related to some workflow issues on your end, because I cannot reproduce it. I just tested it (again). (And if you think about it, someone would have reported especially the 4:2:0 case, because codecs like ut, magicyuv are used by thousands of non professional users as well. )

[Mick]

@kolak
I never said that VirtualDub has Problems with MagicYUV or UtVideo, I was relating to the Ingest Systems only. And v210 was only tested with UtVideo and the Uncompressed Output from VirtualDub using v210, not with MagicYUV. Loading these Files with v210 from UtVideo or VirtualDub to the Timeline to the Ingest System, THEN comes either a "Black" or "Green" Screen after a long Pause before everything hangs. Using the Drastic Codec with v210 is no Problem. I get back with the Guys at Cinegy and let you all know here how it turned out and what they said, okay ?

One thing was discovered: Installing the AJA QuickTime Codecs enables the Import to the Timeline with Ingest Systems using the QuickTime Interface when importing v210 from VirtualDub. But: Instead of v210 YCbCr it's converted to RGB24 and is not as fluent as v210 YCbCr, not the best Solution but works.

@poisondeathray
You should be able to verify that. Here is what we tried:
New Win7, no NLE's, just VirtualDub 1.10 along with Xvid (1.3.2) and x264vfw (Version 41), HuffYUV, Lagarith, MagicYUV and UtVideo, nothing else, means no extra Filter etc.. Then we imported a RAW 4:2:0 (I420/IYUV/YV12) and a RAW 4:2:2 (UYVY/YUY2) Video in VirtualDub. After loading the Video in VirtualDub, we selected "fast recompress", left PCM Audio with "Direct Stream copy" and selected Xvid first, then x264.

The result was that Xvid and x264vfw used the Source Color space that was given by the RAW Files and decoded it like that on the Ingest System, means, I420/IYUV/YV12/UYVY/YUY2 was decoded correctly, same with x264vfw.

Now we fed MagicYUV with I420/IYUV and opened the MagicYUV Video in a new Instance of VirtualDub. Same setup with RAW to Xvid but instead of I420/IYUV MagicYUV decoded as YV12 to Xivd and x264vfw. We fed UtVideo with a Raw YV12 Video and opened a new Instance of VirtualDub, same thing, but UtVideo used UYVY for Xvid and x264vfw instead of YV12. The compressed Raw Sources with UYVY/YUY2 where both decoded by MagicYUV and UtVideo as UYVY to Xvid and x264vfw.

HuffYUV used YUY2 for decoding and Lagarith used YUY2 and YV12 for decoding to Xvid and x264vfw. So the "as is" does not seem to work as expected with MagicYUV or UtVideo. We even tried different Systems with ATI, NVidia and Intel Graphics, no change. Changing the decoding Options in MagicYUV made no difference.

The only thing we noticed that, if all Options are enabled in the Decoding Setting, MagicYUV even uses RGB24 instead of UYVY (all Decoding Options off).

BTW: With x264vfw I also got "lucky" because it's also not on the Cinegy List, only Xvid.

There is no doubt at all that MagicYUV and UtVideo are great codecs and "professional", that word can be seen in different Ways because I've seen "professionals" that had absolutely no Clue of what they are doing. It's not a matter if someone uses these Codecs as a "pro" or not, the result counts, nothing else.

Then you wrote:
"Huffyuv is no different here - for example, if you give it HDYC, it won't give you HDYC back."

I heard rumors that there is a patched HuffYUV for HDYC floating around, based on the CCESP Patch 0.2.5, don't know if it works or solves this.

You wrote:
"There are some circumstances where a certain fourcc is desireable..."

I prefer the FourCC that was used and noticed for example, when exporting as MPEG-2 (DVD) with 4:2:2 instead of 4:2:0 it's slightly slower than staying in 4:2:0 Color Space. When I have do to Color Corrections and they're done in 4:2:0 like the transferred Source, you get a complete different result if you do the Color Corrections in 4:2:2 to be exported to Disk.

If a Client wants MPEG on the Disk, i use 4:2:0 for the cleanest encoding. If a Client wants it as a Archive, then I use YUY2/2vuy or v210 for the Disk. Some Clients request HuffYUV, others Motion JPEG or JPEG 2000 but the majority either RAW YCbCr or HuffYUV.

But maybe it's just me because I would like to see the Color Space used on the Output like it was processed with. There are Limits if you convert 4:2:0 to 4:2:2 or vice versa and my Goal is to keep the Quality at it's best because my Clients expect that from me.

@All
My Attachment is still not "approved" ? No Problem, here is the external Link where you can download the Images from my Colleague from Germany:

CPU-Load.7z:
http://ge.tt/2N8ZQhJ2/v/0?c

You all take care and happy "sweating" with this Heat ;) Many thanks for the responsive and informative Posts.

Cheers
Mick

[poisondeathray]

@poisondeathray
You should be able to verify that. Here is what we tried:
New Win7, no NLE's, just VirtualDub 1.10 along with Xvid (1.3.2) and x264vfw (Version 41), HuffYUV, Lagarith, MagicYUV and UtVideo, nothing else, means no extra Filter etc.. Then we imported a RAW 4:2:0 (I420/IYUV/YV12) and a RAW 4:2:2 (UYVY/YUY2) Video in VirtualDub. After loading the Video in VirtualDub, we selected "fast recompress", left PCM Audio with "Direct Stream copy" and selected Xvid first, then x264.

The result was that Xvid and x264vfw used the Source Color space that was given by the RAW Files and decoded it like that on the Ingest System, means, I420/IYUV/YV12/UYVY/YUY2 was decoded correctly, same with x264vfw.

Now we fed MagicYUV with I420/IYUV and opened the MagicYUV Video in a new Instance of VirtualDub. Same setup with RAW to Xvid but instead of I420/IYUV MagicYUV decoded as YV12 to Xivd and x264vfw. We fed UtVideo with a Raw YV12 Video and opened a new Instance of VirtualDub, same thing, but UtVideo used UYVY for Xvid and x264vfw instead of YV12. The compressed Raw Sources with UYVY/YUY2 where both decoded by MagicYUV and UtVideo as UYVY to Xvid and x264vfw.

HuffYUV used YUY2 for decoding and Lagarith used YUY2 and YV12 for decoding to Xvid and x264vfw. So the "as is" does not seem to work as expected with MagicYUV or UtVideo. We even tried different Systems with ATI, NVidia and Intel Graphics, no change. Changing the decoding Options in MagicYUV made no difference.

The only thing we noticed that, if all Options are enabled in the Decoding Setting, MagicYUV even uses RGB24 instead of UYVY (all Decoding Options off).




How are you making those determinations of what color model/ pixel arrangements are being output exactly? . Let's just take 1 case at a time. It seems the UT Video 4:2:0 is the one that is causing you problems, because upsampling would cause quality loss if your observation was correct.

I'm saying the output lossless x264vfw (4:2:0) from utvideo (4:2:0) input is the same (no quality loss, as determined by those various methods mentioned earliy), and the decoded utvideo is the same as input uncompressed 4:2:0 . By logic , the original uncompressed 4:2:0 is the same as x264vfw, but testing them directly also confirms that.

When you have uncompressed 4:2:0 input loaded directly, check what vdub file=>file information says . Repeat at each stage

The end result is what is important, I think you might either be misreading the output pixel formats, or something else in your workflow is causing the problem. Or test the decoded input and output files directly. If they are not the same using one of the methods mentioned earlier then something else is wrong with your workflow

(I dislike using the way you are using the term "RAW" here. RAW has a different meaning in the professional world - RAW is a term usually reserved for undebayered sensor data. It's completely different than what you are using)



Then you wrote:
"Huffyuv is no different here - for example, if you give it HDYC, it won't give you HDYC back."

I heard rumors that there is a patched HuffYUV for HDYC floating around, based on the CCESP Patch 0.2.5, don't know if it works or solves this.



Yes, there are specific builds, the HDYC was just an example to illustrate the point. What if I started with and needed "YV16"? There are many more configurations for each pixel type... The point is you don't want to install 121 different huffyuv or "codec xyz" versions .



You wrote:
"There are some circumstances where a certain fourcc is desireable..."

I prefer the FourCC that was used and noticed for example, when exporting as MPEG-2 (DVD) with 4:2:2 instead of 4:2:0 it's slightly slower than staying in 4:2:0 Color Space. When I have do to Color Corrections and they're done in 4:2:0 like the transferred Source, you get a complete different result if you do the Color Corrections in 4:2:2 to be exported to Disk.



Of course you stay in the original color sampling and color model. That is NOT the same thing as using different pixel arangements for the same color sampling and color model

If you downsample chroma or convert to some other color model like RGB - of course there is quality loss - we are not talking about that . There is no quality loss between UYVY and YUY2 for example - we already covered that! Because they are both 4:2:2 . If you converted to IYUV (4:2:0) of course there would be loss incurred. Similarly , there is no loss when starting with and converting between YV12 or IYUV or NV12 or I420 because they are all 4:2:0

[kolak]

@kolak
I never said that VirtualDub has Problems with MagicYUV or UtVideo, I was relating to the Ingest Systems only. And v210 was only tested with UtVideo and the Uncompressed Output from VirtualDub using v210, not with MagicYUV. Loading these Files with v210 from UtVideo or VirtualDub to the Timeline to the Ingest System, THEN comes either a "Black" or "Green" Screen after a long Pause before everything hangs. Using the Drastic Codec with v210 is no Problem. I get back with the Guys at Cinegy and let you all know here how it turned out and what they said, okay ?

One thing was discovered: Installing the AJA QuickTime Codecs enables the Import to the Timeline with Ingest Systems using the QuickTime Interface when importing v210 from VirtualDub. But: Instead of v210 YCbCr it's converted to RGB24 and is not as fluent as v210 YCbCr, not the best Solution but works.

Cheers
Mick

This is exactly what you don't want to happen- v210 being decoded to RGB. It breakes whole point of having v210 in the beginning.

Magicyuv won't convert any of the 10bit formats to 8bit and this is done 'sort of' on purpose. If your software can't established vfw pipe over v210 pixel format (which is case for 95% software) than you will have black/green video or crash. For some software it will work and than you have clean 10bit chain (like Vdub, madVR renderer). By installing codecs which convert 10bit to 8bit you are making your system unpredictable and worthless.
Installing Drastic or BM/AJA codec to make v210 (or 10bit magicyuv/utvideo) working is 'vary bad' and it actually tells you that your pipe is only 8bit. What you want is your software to be able to accept v210 pixel format after magicyuv/utvideo decoder natively, so pipe stays 'native'.
Problem is that almost not a single software can accept v210 on decoding pin over vfw. You will be more lucky with MOV, but this has its own problems also.
Yet again- you need a proper integration between the app and codec on SDK level not through vfw/directshow/QT etc. If you think you will have perfect integration over vfw etc between some custom codec and app than think twice. If you want the same for 10bit+, than you may forget, as this is basically guaranteed not to work without proper integration. If you keep instaling 'other' codecs: Drastic, ffdshow, etc than this veeeeery bad idea and not the way to get reliable workflow.

[poisondeathray]

You should definitely test end to end what kind of results you are getting. For 10bit tests, using a gradient makes it easy to see the problem

I usually start on the ends (input and final output), and if something is wrong, work towards the middle tracing each step to find out where something went wrong. For some software / host applications there is no solution except to use their "preferred" uncompressed format (for some applications, they do not handle huffyuv, lagarith, ut video , magic yuv , etc... as lossless YUV ,or at the correct bit depth). That' s not really the codec developers fault IMO


I prefer the FourCC that was used ...


Maybe you missed, it but the point of my earlier "offtopic" statement was that some software handle certain fourcc's better than others. I would argue it matters less what you prefer - I would argue you should use what works better in that specific workflow or software title. In windows software , for YUV 4:2:2, UYVY is almost always the "preferred" format . YUY2 almost always gets converted to RGB internally. Similarly for 4:2:0 , IYUV (and sometimes I420) is usually the preferred format . YV12 almost always gets converted to RGB internally

[Ignus2]

I now have just a very little time to answer, but I write as the discussion begins to touch on too many subjects concerning various codecs and YUV formats in ingest systems, which does not directly concern codec development, but rather workflow and system setup, so I think it is getting slightly offtopic.

May I suggest to start a new thread in some other category on doom9 and continue the discussion there? That way maybe more people would be able to provide feedback as well!

I also would very much like to reflect upon the recent posts, but unfortunately I cannot do that now, as I'm not at home for a few weeks.

Thanks!

Greets,
I.

[easy2Bcheesy]

Hi,

Just bought a Blackmagic Intensity 4K for capturing PS4, Xbox One etc at 1080p60.

Annoyingly the only way to get a picture via DirectShow is using the r210 colourspace - which does not seem to be supported by any codec! Would it be possible to incorporate downsampling support for r210 into MagicYUV?

[raffriff42]

r210 colourspace - which does not seem to be supported by any codec!ffmpeg supports it -
C:\ffmpeg\bin>ffmpeg -codecs | find "r210"
ffmpeg version N-70664-ga47c2a1 Copyright (c) 2000-2015 the FFmpeg developers
[...]
DEVI.S r210 Uncompressed RGB 10-bit

C:\ffmpeg\bin>(and therefore probably ffmpegsource for avisynth & ffmpeg input for virtualdub)

[easy2Bcheesy]

Would that work for real-time capture though?

[ChiDragon]

Annoyingly the only way to get a picture via DirectShow is using the r210 colourspace
Sadly, a quote from Blackmagic support indicates (https://forum.blackmagicdesign.com/viewtopic.php?f=12&t=33412) that lack of any RGB -> YUV conversion is a deliberate part of the architecture, so it seems they will never support it. Sure would be nice if they added a note somewhere in the specs that they removed a popular feature (available with nearly every other HDMI capture device) as part of this "upgrade" from the Intensity Pro.

If you are capturing from HDMI, please note that the behaviour of the Intensity Pro 4K is a bit different to the Intensity Pro.
DeckLink 4K devices (which include the Intensity Pro 4K) require an application to explicitly set the pixel format for capture over HDMI that matches the video source. If you specified YUV for capture and the feed was RGB, the older Intensity Pro would automatically convert the pixel format.
However the Intensity Pro 4K does not. So if you have a RGB HDMI video source, you need to set capture for RGB (bmdFormat10BitRGB) with the Intensity Pro 4K, however you can get away with selecting 8-bit YUV (bmdFormat8BitYUV) or 10-bit YUV (bmdFormat10BitYUV) with the Intensity Pro.

The ridiculous thing about this is that the card actually does support conversion from RGB HDMI sources if the resolution is 480p/i! Probably 576p/i too though I haven't tested. Media Express lets you select every format with no problem when resolution is "NTSC", while the DirectShow interface actually forces you to capture "NTSC RGB" in YUV as there is no driver support for RGB capture at NTSC size.

However, there is good news. I discovered yesterday that the v10.4.1 driver/firmware update added support for "A RGB32" if you use the "Blackmagic WDM Capture" interface instead of "Decklink Video Capture" -- so thank goodness, we no longer have to use 10-bit RGB to capture 8-bit RGB sources... (v10.4.2 was released on Tuesday but I haven't tried it yet.)

It's still a bit of a pain in the ass to get the RGB32 capture working in VirtualDub. The Capture Pin for this driver version has a bug where the Frame Rate for HDYC, A RGB32, and r210 reverts to 25.000 no matter what you pick. The workaround is this:


Video -> Capture filter.
Video Format -> Override Video Mode.
Manually set Display Mode to the exact resolution/frame rate and Pixel Format to "8 bit BGRA".
Video -> Capture Pin.
Manually type in the exact frame rate and hit Apply or OK.
At this point, for me there is no often no image at all. Toggle from Video -> Overlay to Video -> Preview or vice-versa to restore the image.
Now you can set to Overlay or Preview mode, whichever you prefer, and capture with any codec that supports RGB32, including those that convert to YUV.
The overridden Video Mode is not saved, but all other settings are. When you close and reopen VirtualDub's capture mode, you will need to manually change the settings in Capture filter again, but you should then immediately get the image without any more steps.


I'm still trying to figure out which lossless codec and Disk I/O buffer settings might actually allow me to capture to my regular 5400RPM 4TB SATA drive without dropped frames when there are complex scenes.

[easy2Bcheesy]

Lossless 4:2:2 can hit a max (in my experience) 140MB/s, which may well be beyond your 5400rpm drive.

I am on Blackmagic 10.4.2 and alas I don't see any Blackmagic WDM capture interface in either VirtualDub or (my preferred tool), AmaRecTV.

I'm on Windows 8.1 if that makes any difference.

EDIT: 10.4.1 has the correct filter - giving it a go!

EDIT 2: Okay, so it works in VirtualDub, which is OK. I don't have to change the capture FPS either, it defaults to 59.94. However, turning on real-time audio preview kills the video preview in all modes. I don't suppose you could take a look at AmaRecTV?

[ChiDragon]

I seem to be averaging only 77MB/sec when capturing RGB -> lossless 4:2:2 but still getting random frame drops. Lossless RGB is more in the 90-100 range and giving me more "consistent" drops. The drive has been benchmarked (http://hdd.userbenchmark.com/Seagate-Desktop-HDD-4TB/Rating/1598) with a min write speed of 83.2 and average 122 (it's actually 5900RPM, not that it makes a difference). It could understand if it was suddenly dropping off because it was hitting the far edge of the disk, but it's the random nature that confuses me.

AmaRecTV showed all the various pixel types so it seemed like it would be easier, but when I tried to capture it just failed with some error message. I didn't bother to troubleshoot.

They really removed it with 10.4.2? Ridiculous. That's like how RGB capture was broken completely with 10.4 after working okay with 10.3.7. Glad I didn't hop on the update bandwagon or I would have missed seeing this feature entirely.

EDIT 2: Okay, so it works in VirtualDub, which is OK. I don't have to change the capture FPS either, it defaults to 59.94. However, turning on real-time audio preview kills the video preview in all modes.
Hmm, there is no difference for me when I "Enable audio playback" (they both display with a very poor frame rate for me when fed 1080p). Audio is set to "Capture device" and the format is 16-bit stereo. I'm way back at VirtualDub v1.9.10 but I doubt that gives me any advantage.

[Bloax]

IThe drive has been benchmarked (http://hdd.userbenchmark.com/Seagate-Desktop-HDD-4TB/Rating/1598) with a min write speed of 83.2 and average 122 (it's actually 5900RPM, not that it makes a difference). I could understand if it was suddenly dropping off because it was hitting the far edge of the disk, but it's the random nature that confuses me.
The random nature is likely explained by the framedrops occuring when you hit a slow sector on the drive.
And yes, sectors are not equal on a drive - especially later in their life.

You can take a look at it by running a "read" surface test using HDDScan (http://hddscan.com/).

[benwaggoner]

This is definitely a job for RAID.

[kolak]

Hi,

Just bought a Blackmagic Intensity 4K for capturing PS4, Xbox One etc at 1080p60.

Annoyingly the only way to get a picture via DirectShow is using the r210 colourspace - which does not seem to be supported by any codec! Would it be possible to incorporate downsampling support for r210 into MagicYUV?

Magicyuv has native support for r210 pixel format (and also AJA's R10k which is almost the same), but not in public/free version. This is one of the pro features which I convinced Ignus2 to implement.

You could try to force BM to output RGB 8bit instead of 10bit.

[ChiDragon]

There is no need to read the docs, as I can read the source code as well. What I have said is:
"If there is no in-codec color-space conversion involved, you get back the same result whether interlaced encoding is turned on or not."
Which is verifiable by reading the HuffYUV sources. HuffYUV does no blending of fields whatsoever if you give it YUY2 and ask YUY2 back. It will give you back the same result if you encode and decode in the same colorspace whether interlaced encoding is turned on or not.
Going back a bit in the thread, but it occurred to me that since [the non-libavcodec versions of] Huffyuv only operate with YUY2 and RGB (no YV12), the presence of interlacing shouldn't even affect colorspace conversion for them.

[easy2Bcheesy]

Magicyuv has native support for r210 pixel format (and also AJA's R10k which is almost the same), but not in public/free version. This is one of the pro features which I convinced Ignus2 to implement.

You could try to force BM to output RGB 8bit instead of 10bit.

Does the pro version offer the ability to process r210 input but capture in RGB24, YUY2 or YV12? The BM device is literally padding those two extra bits per channel as the consoles don't output 10-bit RGB.

ChiDragon's technique for forcing 8-bit RGB works, but it is not ideal.

[kolak]

No, there is no 10<->8bit conversion and this is done sort of on purpose. Decoder will decode r210 back to r210 or rbg64a/rgb48, so only to "better" pixel formats.

[pandy]

ffmpeg supports it -
C:\ffmpeg\bin>ffmpeg -codecs | find "r210"
ffmpeg version N-70664-ga47c2a1 Copyright (c) 2000-2015 the FFmpeg developers
[...]
DEVI.S r210 Uncompressed RGB 10-bit

C:\ffmpeg\bin>(and therefore probably ffmpegsource for avisynth & ffmpeg input for virtualdub)

And FFV1 can be used instead MagicYUV also ffmpeg can provide conversion from 10 to 8 bits (with dithering?) - so why not try to capture with ffmpeg ?

I've try few times and it works (capture)...

[ChiDragon]

FFV1 is much, much too slow for 1080p60 in realtime.

[kolak]

But new huffyuv in ffmpeg not :)

[ChiDragon]

It only converts to YUY2/YV12, not RGB. :\

[kolak]

Ffmpeg version supports up to 12bit RGB and with -vf pix_fmt you can convert between maaany pixel formats.
You probably talking about 'old' huffyuv in ffmpeg, not new one- 'ffvhuff'.

[ChiDragon]

[Sorry for all the OT, Ignus2.] How can you use that with VirtualDub or another capture app? The newest ffdshow-tryouts version that I found only offers those two formats for "FFVH".

[Ignus2]

Hi everyone!

It has been some time since I last visited here, a mountain of real-life-stuff got on top of me (resulting mild symptoms of sleep-deprivation :) ), so I couldn't do things as I wanted.

Anyway, I just wanted to announce version 1.2 of the codec has been released. And this time, it comes with a small twist. It might mean no change for some, and it might mean something for others. Everything is explained at the Download page (http://magicyuv.com/index.php/download), so head over there to see what I mean.

Anyway, some of the highlights of this release (the full change log is on the website):
- The null-frame decoding bug reported on the German letsplay forums has been fixed, encoded video with null-frames should decode now correctly.
- Added optional interpolation when downsampling/converting the input during encoding. This fixes the color conversion issue reported by De-M-oN
- Improved encoding performance by 10-20% by further optimizing some parts of the encoder.
- Added logging to logfiles, logging communication with the host application and timing of frame processing (encode/decode).

As always, read the tooltips in the config dialog!

Thanks for all who tested the codec and reported bugs, further reports/requests are welcome as always!

Greets,
Balázs Oroszi
- Ignus

[Bloax]

I'm not really seeing why you would hide it behind a one-time-per-nose payment-of-down-to-0$ choice while offering an "authentic" (crippled) trial version.
All that will be good for is making people that wouldn't mind throwing money at it but don't actually possess enough funds to do so (*cough* *cough*) at the moment not do anything at all.

[Ignus2]

I'm not really seeing why you would hide it behind a one-time-per-nose payment-of-down-to-0$ choice while offering an "authentic" (crippled) trial version.
All that will be good for is making people that wouldn't mind throwing money at it but don't actually possess enough funds to do so (*cough* *cough*) at the moment not do anything at all.

It seemed to be a good idea at the time. Maybe it isn't (I'm no expert on this). Time will tell...

Greets,
I.

[zerowalker]

Personally i like the idea of "paying what it's worth", for me it feels like you get more motivation to actually care so to speak.
I think that if someone is using it and actually think it's making a difference for them, they will be willing to pay for it:)

I have nothing to say about the trial thing though, it feels weird when i read about it but then again i can't think of a better way.

Other than that thanks for the update!
I would like to see those "color aliasing artifacts" in games though, seems like i have missed something here?
And that kind of downsampling to prevent that, is those normally used (say avisynth)?

[Ignus2]

Thanks!


I would like to see those "color aliasing artifacts" in games though, seems like i have missed something here?
And that kind of downsampling to prevent that, is those normally used (say avisynth)?

It was reported here, and is now fixed:
http://www.letsplays.de/forum/index.php/Thread/77996-Videoqualit%C3%A4t-und-Encodierverfahren-Sammelthread-Startpost-beachten/?postID=2031592#post2031592

Basically if you compress pixelated stuff (like Mario or Doom) in YUV 4:2:2 or 4:2:0, you'll now get better results.

Greets,
I.

[De-M-oN]

I would like to see those "color aliasing artifacts" in games though, seems like i have missed something here?


If every 2nd pixel is without color it is very noticable..

Record a Desktop Capture in 4:2:0 and watch out for fonts. Small fonts and especially when you scroll - looks very ugly when the pixel switch from color to non-color and also at a static image you of course see it, when every 2nd pixel is without color. This results in small font being almost colorless.

Just read this post:

http://forum.doom9.org/showpost.php?p=1727986&postcount=352

[Ignus2]

@De-M-oN: Could you test the new version?

Greets,
I.

[djonline]

I can export movie from Adobe Premiere using MagicYUV, but there is no Magic YUV in Sequence Preview settings. Why ? But I can see UT-video in this tab.

[TheSkiller]

If every 2nd pixel is without color it is very noticable..This is not entirely related to the chroma subsampling itself but mainly to the sampling of it.

If you have an RGB desktop recording and then convert it to 4:4:4 YUV (YV24 in AviSynth) and need the final video to be in 4:2:0, it makes a difference whether you "delete" every other chroma pixel or if you average them, in other words: PointResize vs. [any other Resizer]. If the 4:2:0 chroma pixels are sampled properly, there are no pixels without color, and the chroma is not pixelated either – it is just not as sharp as the RGB source (but I admit this can still be quite obvious on small colorful text).

[De-M-oN]

@De-M-oN: Could you test the new version?

It does its job fine. Thank you.

@TheSkiller : It was entirely the problem of missing interpolation.

The new option fixed it entirely.

[Ignus2]

I can export movie from Adobe Premiere using MagicYUV, but there is no Magic YUV in Sequence Preview settings. Why ? But I can see UT-video in this tab.

I don't have Premiere, so can't directly help. Maybe you could attach some screenshots of what you're missing, and also what codec settings you use.

Greets,
I.

[De-M-oN]

Maybe you can help here too ignus?

http://forum.doom9.org/showthread.php?t=172647

:)

[RRD]

When updating to a newer version should I uninstall my current version of MagicYUV beforehand or install the new version directly?

[Ignus2]

When updating to a newer version should I uninstall my current version of MagicYUV beforehand or install the new version directly?

You can just install directly. The installer since 1.1 basically does an uninstall first before performing the actual copying of the new files.

Greets,
I.

[djonline]

I don't have Premiere, so can't directly help. Maybe you could attach some screenshots of what you're missing, and also what codec settings you use.

Greets,
I.
Can't show anything, because there is no MagicYUV in codec selection for sequence preview in Premiere (CC 2014 or CC 2015). But UT Video, ffdshow and others VFW is here.
You can install trial version of Premiere to debug this problem.
I don't want to use UT Video anymore :)
http://s008.radikal.ru/i306/1511/d4/8f6cb5cb3b87.png

[poisondeathray]

Interestingly, the Quicktime/MOV variant of MagicYUV shows up as an option for the sequence preview in PP

[benwaggoner]

Interestingly, the Quicktime/MOV variant of MagicYUV shows up as an option for the sequence preview in PP



I believe Premiere only exposes VfW encoders, not DirectShow or MediaFoundation. 90's throwback!



Adobe Media Encoder, which you can export to straight from Premiere, may be more flexible.

[Ignus2]

Can't show anything, because there is no MagicYUV in codec selection for sequence preview in Premiere (CC 2014 or CC 2015). But UT Video, ffdshow and others VFW is here.
You can install trial version of Premiere to debug this problem.
I don't want to use UT Video anymore :)

Strange. Does it show up in VirtualDub?

Greets,
I.

[[SLiM]D12]

stupid question here -- but is there a way to use ffmsindex to index these and ffvideosource in avisynth? I get a "video codec not found" error.

thanks!

[vivan]

No, this codec is closed-source and is not supported by ffmpeg. Just use avisource instead.

[[SLiM]D12]

No, this codec is closed-source and is not supported by ffmpeg. Just use avisource instead.

dss2 works as well right?

[De-M-oN]

I highly recommend to use AVISource.

[[SLiM]D12]

I highly recommend to use AVISource.

Great thanks for the recommendation.

[foxyshadis]

DirectshowSource would just use Directshow to call back to VFW, Avisource does the exact same thing without all the overhead.

Only reason to use DSS2 is if you have VFR video.

[djonline]

Strange. Does it show up in VirtualDub?

Greets,
I.
Of course yes! Also it shown in File->Export in Premiere.

[WorBry]

No problem here. I've just been running a trial of Premiere Pro CC (64-bit) and MagicYUV is there listed with the other installed 64-bit AVI codecs. Same goes for other NLE's I've tried out that support third-party VFW codecs. And I've been using it for some time now in Corel VideoStudio Pro as both the edit (Project) and output format, with Smart Rendering....smooth as butter. I'm still using MagicYUV version 1.1 btw.

[De-M-oN]

I'm still using MagicYUV version 1.1 btw.

You really really should update to 1.2. 1.1 has no chroma interpolation which results for everything below 4:4:4 color being pixels without color instead of averages.

[WorBry]

...1.1 has no chroma interpolation which results for everything below 4:4:4 color being pixels without color instead of averages.

I'm not sure I fully understand what that means.... for me. Let me explain my basic workflow.

To prepare AVCHD (1080/30PsF) mts clips from my camcorder for editing, I transcode to MagicYUV (with WAV audio) using DGIndexIM and AVISynth via VDub (32-bit).

If I know I'm just going to doing straightforward cut-edits, with maybe some inter-cut titles and transitions, I'll encode (with Fast Recompress) as YV12 i.e. MagicYUV set with YUV 4:2:0 as the accepted colorspace and "Compress as is". Everything else default. That way the original YV12 luma range (effective 16-255) is preserved.

Normally, I join my clips together before transcoding to produce one (large) input lossless avi file. VideoStudio Pro (x8.5) has a nice Scene/Content Change Detection feature, so I can easily segment the clips again on the timeline

In VSPro, I set up the Project format with MagicYUV configured to accept RGB(RGB32,RGB24) and YUV 4:2:0 for the "conversion mode". VSPro is limited to 24-bit RGB processing.

After editing, I render out to MagicYUV as per the Project Settings, but with smart rendering. That way only those frames changed by an edit transform are re-encoded, bottom-up from RGB to YV12. This does mean that any re-encoded or newly created frames have a clipped 16-235 luma range, but that's usually inconsequential - I rarely use title overlays or burned-in subtitles and transitions are usually limited to an abrupt fade to black - so any luma changes are masked.

If however I know I'm going to be doing more extensive editing with effects, then I may first pre-convert (with AVISynth) the clips to RGB32 and transcode to MagicYUV set to accept RGB24/32 and 'Compress As Is". And in the VSPro Project Settings, I will configure MagicYUV likewise. So that's even larger RGB files in and out, but that's OK. I can still apply the smart-rendering feature, but in this way marked fluctuations in luminance in the course of a scene are avoided.

In the light of all that, could you explain what implications your statement may or may not have for my workflow and the quality of my edits whilst using MagicYUV 1.1 ?

Cheers.

[WorBry]

OK, I've looked back through the thread and found your post about this:

http://forum.doom9.org/showpost.php?p=1727986&postcount=352

and Ignus2's response:

http://forum.doom9.org/showpost.php?p=1728013&postcount=354

and the changelog for version 1.2:

http://magicyuv.com/index.php/change-log

Quote:

NEW: Core: Added optional interpolation when downsampling/converting during encoding (default: ON). This fixes the color conversion issue reported when recording certain games
...and I still don't understand what implications this may or may not have for my workflow. Plus the link you provided in that first post was to a forum thread in German which I do not speak. Please explain.

[WorBry]

Wonderful, phases me with this interpolation thing and then doesn't follow it up with an explanation !

[De-M-oN]

just read. It is described in this forum and in Lets Play Forum as well.

If you dont interpolate Chroma Downconversion then every 2nd pixel is colorless. This isnt limited to games like your quote says - it is a problem for every video.

[WorBry]

Just READ.

... Plus the link you provided in that first post was to a forum thread in German which I do not speak.

i.e. the Lets Play Forum thread that you referred to:

http://www.letsplays.de/forum/index.php/Thread/77996-Videoqualit%C3%A4t-und-Encodierverfahren-Sammelthread-Startpost-beachten/?postID=2031592&highlight=magicyuv%2Bungerade#post2031592

and


and the changelog for version 1.2:

http://magicyuv.com/index.php/change-log

Quote:
NEW: Core: Added optional interpolation when downsampling/converting during encoding (default: ON). This fixes the color conversion issue reported when recording certain games


It wasn't my quote, it is what is stated in the changelog on the official MagicYUV webpage.

Anyhow, I'll run some tests. Thanks for your help.

[WorBry]

OK, if I understand the issue correctly, the only point at which this could possibly impact my editing workflow is:

...In VSPro, I set up the Project format with MagicYUV configured to accept RGB(RGB32,RGB24) and YUV 4:2:0 for the "conversion mode". VSPro is limited to 24-bit RGB processing.

After editing, I render out to MagicYUV as per the Project Settings, but with smart rendering. That way only those frames changed by an edit transform are re-encoded, bottom-up from RGB to YV12

i.e. in the conversion/downsampling of RGB(supplied by the VSPro processing engine) to YV12 by MagicYUV.

I've just run a few tests with some sample mts clips comparing:

Clip1. Round trip conversion from source YV12 to RGB and back, encoded to MagicYUV configured to accept YUV 4:2:0 and compress "as is". Script:

DGSourceIM("Path.....\Test mts clip.dgi",silent=true).AssumeFPS(30000,1001)
ConvertToYV24(chromaresample="point", interlaced=false)
MergeChroma(PointResize(width, height, 0, 1))
ConvertToRGB32(matrix="rec709", interlaced=false)
ConvertToYV12(chromaresample="point",matrix="rec709")

with Clip2. Source converted to RGB and encoded to MagicYUV configured to accept RGB(RGB24,RGB32) and compress to YUV 4:2:0. Script:

DGSourceIM("Path.....\Test mts clip.dgi",silent=true).AssumeFPS(30000,1001)
ConvertToYV24(chromaresample="point", interlaced=false)
MergeChroma(PointResize(width, height, 0, 1))
ConvertToRGB32(matrix="rec709", interlaced=false)


In all cases, ColorYUV(analyze=true) and mt_MakeDiff(clp1,clp2,U=3,V=3).ColorYUV(analyze=true) report no difference between the two outputs.

So where's the chroma interpolation error - or am I missing something ?

[WorBry]

@Ignus2,

...
So where's the chroma interpolation error - or am I missing something ?

Could you please clarify this matter? Is this interpolation issue specifically related to the recording of certain games, as indicated in your change-log for 1.2, or is it something that pertains to all video, as De-M-oN would suggest? And if the latter case, why am I not seeing evidence of it when transcoding RGB video to YV12 with MagicYUV 1.1?

Irrespective of whether I upgrade or not, I would hate to think that the quality of all of my video editing using MagicYUV to date (as De-M-oN's statement would imply) has in some way been compromised.

Thanks.

[Ignus2]

In-codec RGB to YUV 4:2:2/4:2:0 conversions' chroma downsampling was done simpler (nearest neighbor) in v1.1 and earlier, which resulted in visible aliasing artifacts for pixelated games with fine (1 pixel width) colored lines for example.
It doesn't affect other types of material much, so don't worry. Also, it only was the case when the codec did the conversion. If it compressed YV12 and the like to begin with, this doesn't apply.

BTW: why not use 1.2, it's been out since months?

Greets,
I.

[De-M-oN]

It doesn't affect other types of material much, so don't worry

Nah if every 2nd pixel is colorless I see it as a serious problem everywhere. I highly recommend the interpolation.

[Ignus2]

Nah if every 2nd pixel is colorless I see it as a serious problem everywhere. I highly recommend the interpolation.

This statement is incorrect and misleading, even with interpolation you throw away 3/4 of the color data (for 4:2:0), only the effects of it are different (less aliasing).

Greets,
I.

[WorBry]

It doesn't affect other types of material much, so don't worry

There you go.

Also, it only was the case when the codec did the conversion. If it compressed YV12 and the like to begin with, this doesn't apply.

But is that not the case in the instance I went on to test i.e. where I am using MagicYUV YV12 as the edit intermediate in VideoStudio Pro and an edit transform is applied (in RGB colorspace) requiring MagicYUV to convert the RGB back to YV12. Yet, simulating that in the test I did above, round-tripping YV12 to RGB and back, there was no evidence of any chroma defecit.



BTW: why not use 1.2, it's been out since months?


Oh, just coz...I saw no compelling need really. As you can see, my needs are quite straightforward and version 1.1 was plenty fast enough.

Truth be told though, I was curious to see what happens at the next update, and generally what direction you see MagicYUV going in. Is it simply a case of bug fixes as they arise? Do you have intent to merge the now private/evaluation 10-bit version, or keep it separate? And then, for those people who have already passed through the "pay what you think it is worth" update registration, come the next update do they automatically receive a new license or will everyone be obliged to register all over? That type of thing. Reasonable questions, no ? Perhaps you could elaborate ?

Tell you what I would like to see though, at some point down the road. I've been dabbling over on the 'L-Side' with KDenLive of late - a bit of Cinelerra also, once I can get it to work properly. But the options for intra-frame edit intermediates are still very limited. Well there's DNxHD, which is top-notch, but it's not lossless and there's the YV12/YUY2 inter-conversion hit to account for. UTVideo has been in FFMPEG for years now, but is still not an option in KDenLive. Otherwise what- stable but heavyweight HuffYUV.

I see you've produced a trial version for Linux which, from the description appears to be only a decoder for VLC? Is it possible that we could see MagicYUV in FFMPEG proper at some point or no-go?

Bet you wish you hadn't asked now :D

Anyhow, I'm glad to see that MagicYUV is gaining due recognition as a fast, lossless AVI option in other Windows NLE's also.

Cheers.

Don't you two start squabbling now ;)

[Ignus2]

PWYW is just a way to support development for those who find the codec useful, and those who do receive priority. If you are concerned, just enter 0 and download it, you'll get the same thing.
The Linux version is not just trial, but it only contains a VLC decoder so far.
ffmpeg integration is unlikely at the moment.

Greets,
I.

[De-M-oN]

This statement is incorrect and misleading, even with interpolation you throw away 3/4 of the color data (for 4:2:0), only the effects of it are different (less aliasing).


every 2nd pixel is colorless. With interpolation its just every 2nd pixel has the average color of the neighbor pixel. Huge difference.

And thats also why I made the thread about my capture card. Every 2nd pixel colorless is awful.

Its already problematic if you have texts. Not just a map in doom. Every 2nd pixel colorless is obviously very bad.

With interpolation the pixels stay colored - huge difference. Really.

[Ignus2]

every 2nd pixel is colorless. With interpolation its just every 2nd pixel has the average color of the neighbor pixel. Huge difference.


You are confusing up- and downsampling here. In v1.1 during upsampling the missing chroma value was also interpolated from neighboring pixels, only downsampling was incorrect.
Interpolated downsampling is obviously better with regard to aliasing.

Greets,
I.

[WorBry]

PWYW is just a way to support development for those who find the codec useful, and those who do receive priority. If you are concerned, just enter 0 and download it, you'll get the same thing .

I well appreciate that and donating was not a concern per se, although, as you will recall I did put a lot of time into beta testing your codec when it was first announced and people were a bit skeptical about the efficiency claims.

With all respect, I suppose I am always a bit wary about giving a private email address away, to anyone, but I assume that goes into a secure Gumroad merchant account database, and not one on a PC somewhere.

Anyhow, I've just updated.

ffmpeg integration is unlikely at the moment.

Figured as much but thought I'd ask.

You didn't comment on the 10-bit aspect though. I have no burning need for 10-bit at this time, but I'm sure there are many who will be looking to it. Where MagicYUV has come up on a number of video editing forums I have seen there has always a query about that.

Cheers.

[Ignus2]

The email address goes to me only actually (gumroad doesn't use it for their stuff afaik), so (I think) I would be the only one capable of sending enlargement pill ads and such... :)
And yes, I just noticed, and thank you very much for your support! :)

As for 10-bit: 10-bit is über-tricky. Even with the relatively simple and widely supported 8-bit formats there is still so much confusion on the forums, 10-bit would just simply make people even more confused (and probably drive my nuts trying to explain).
10-bit works in a relatively small amount of NLEs, only through QuickTime, and even then some of them has to be manually tweaked by editing obscure undocumented config files (Adobe AE for example). Worst is that QuickTime loves to do "magic" things, so if you mess up the config, it will "appear" to work, but silent dithering and other nasty stuff will happen without telling you (or the codec).

Greets,
I.

[WorBry]

..... so (I think) I would be the only one capable of sending enlargement pill ads and such... :)


I'll look forward to that ;)

Just had a brief "Oh, No" moment testing version 1.2 in VideoStudio Pro x8.5. Configured the updated codec, as it appeared in the Project Settings AVI codec options, for accepting RGB(RGB24,RGB32) and compressing to YUV 4:2:0 (with the interpolation option ON), as usual. Loaded some source YV12 clips, made some cuts and applied a few effects, and smart rendered out to MagicYUV, as per the project settings. The outputs wouldn't play (crashed MPC-HC and VDub) and wouldn't load in AVISynth. However, after re-launching VSPro a couple of times, it appears to have righted itself. Weird. Hope it doesn't recur.

[Ignus2]

That's weird. What does "smart" render do actually?

[WorBry]

As it applies to AVI files - basically as I described in my edit workflow above.


To prepare AVCHD (1080/30PsF) mts clips from my camcorder for editing, I transcode to MagicYUV (with WAV audio) using DGIndexIM and AVISynth via VDub (32-bit).

If I know I'm just going to doing straightforward cut-edits, with maybe some inter-cut titles and transitions, I'll encode (with Fast Recompress) as YV12 i.e. MagicYUV set with YUV 4:2:0 as the accepted colorspace and "Compress as is". Everything else default. That way the original YV12 luma range (effective 16-255) is preserved.

Normally, I join my clips together before transcoding to produce one (large) input lossless avi file. VideoStudio Pro (x8.5) has a nice Scene/Content Change Detection feature, so I can easily segment the clips again on the timeline

In VSPro, I set up the Project format with MagicYUV configured to accept RGB(RGB32,RGB24) and YUV 4:2:0 for the "conversion mode". VSPro is limited to 24-bit RGB processing.

After editing, I render out to MagicYUV as per the Project Settings, but with smart rendering. That way only those frames changed by an edit transform are re-encoded, bottom-up from RGB to YV12. This does mean that any re-encoded or newly created frames have a clipped 16-235 luma range, but that's usually inconsequential - I rarely use title overlays or burned-in subtitles and transitions are usually limited to an abrupt fade to black - so any luma changes are masked.

If however I know I'm going to be doing more extensive editing with effects, then I may first pre-convert (with AVISynth) the clips to RGB32 and transcode to MagicYUV set to accept RGB24/32 and 'Compress As Is". And in the VSPro Project Settings, I will configure MagicYUV likewise. So that's even larger RGB files in and out, but that's OK. I can still apply the smart-rendering feature (to the lossless RGB file) , but in this way marked fluctuations in luminance in the course of a scene are avoided.


Basically it treats the MagicYUV files (encoded as YV12 or RGB) as though they were DV. In fact to configure the Project Settings to use other accessible AVI codecs as the edit video format you have to go into the DV Profile and edit it to create a Custom Profile and adjust the frame resolution/aspect ratio and field order as appropriate.

Then in the Output settings I choose an option to use the Project Settings as the render profile with or without Smart Rendering. Without Smart Rendering the entire edit stream is re-encoded.

Works perfectly well with MagicYUV and UTVideo. An exception is Cineform which cannot be smart-rendered. According the Cineform (now GoPro, of course) CTO, David Newman, that is only possible with custom-tailored builds.

It's a really nice feature in this context and the main reason I continue to use VSPro.

Sony Vegas Movie Studio and Premiere Elements can be configured to output to third-party AVI formats, but not to use them as the Project video format. Can't quite recall if Cyberlink PowerDirector can output custom AVI profiles, but I think it does do Smart Rendering where the format permits.

[Ignus2]

So smart rendering copies some compressed frames as-is in binary form and changed ones are re-encoded, so effectively the output file will have some old compressed frames and some freshly compressed ones?
If that's the case, it's a problem, the codec is not prepared for that. In fact, it's a miracle that it even works. 1.1 and 1.2 frames are compatible, but 1.0 is different for example. So while newer versions of the codec can always decode older versions' full videos, it is not supported on the frame level (ie. a video is not expected to have a mix-and-match of frames from various versions).
So 1.2 can decode anything from earlier versions, as long as they are fully from a previous version. If frames from 1.0 and 1.1 are mixed in a video file, it will likely crash.
Or it might work (1.1 and 1.2 frames are compatible), but this scenario is not tested at all.

I don't know if smart rendering actually means this or something else, if it means something different, then the above does not apply.

Greets,
I.

[WorBry]

So smart rendering copies some compressed frames as-is in binary form and changed ones are re-encoded, so effectively the output file will have some old compressed frames and some freshly compressed ones?

That's basically how I understand it, but I'm not exactly sure what happens at the interface of the editors RGB processing engine and MagicYUV's encoder, except that it will only accept RGB(RGB24,RGB32), so it's certain that the engine itself is not supplying YV12, or even YUV 4:4:4.


If that's the case, it's a problem, the codec is not prepared for that. In fact, it's a miracle that it even works. 1.1 and 1.2 frames are compatible, but 1.0 is different for example. So while newer versions of the codec can always decode older versions' full videos, it is not supported on the frame level (ie. a video is not expected to have a mix-and-match of frames from various versions).
So 1.2 can decode anything from earlier versions, as long as they are fully from a previous version. If frames from 1.0 and 1.1 are mixed in a video file, it will likely crash.
Or it might work (1.1 and 1.2 frames are compatible), but this scenario is not tested at all.

I don't know what to say. I've gone on to do some more tests and 1.2 appears to work OK with input files previously encoded with 1.1. And it surely is smart-rendering, not just re-encoding the lot. Looking at the luma levels of the outputs, those frames that are 'passed-through' retain the full 16-255 range, whereas the re-encoded frames are clipped to 16-235. And you can see it in rendering progress bar also; fast as it passes through the smart-rendered segments, slowing down when it hits the parts to be re-encoded.

Generally though, once I'm done editing and encoding to the usual delivery formats, I delete the intermediate files as they are too large to keep in the long term. I don't think I have anything that was previously encoded with version 1.0 to test with 1.2.

[Ignus2]

1.1 and 1.2 frames are compatible, so it could work. I'll take this smart rendering into account for new releases.
16-255 range is strange, limited range Y is 16-235, full should be 0-255.

Greets,
I.

[WorBry]

The effective 16-255 luma range is pretty standard for consumer-level digital camcorders and the way it has been since the DV camcorder days. Basically the black point is set at 16 but overshoot is allowed at the top end for 'super-whites' up to 255. It complicates YUV levels adjustment a bit, but for all other AVISynth processing in YV12 (where there is a need to decide between "PC" and "TV" level parameters) I have always processed as full 0-255 (PC range) luma.

One consideration when converting to RGB however, is whether it is preferable to clip the super-whites (or let the Rec709 conversion do it) or pre-compress to 16-235 to preserve any additional detail or specular highlights that may be present in the 'super-white' range, at the expense of loss in contrast. And that's something you have to judge on a case-by-case basis.

In case you are wondering - no, I don't select the 'Full Range YUV' option in MagicYUV. That would bring the black point down from 16 to 0 and darken the image.

[Ignus2]

Well, as long as the codec gets YUV and outputs YUV and doesn't do conversions to/from RGB, it doesn't matter what YUV Range is selected in the codec, you'll get back the same numbers as you gave the codec, so the codec does no explicit clipping of YUV data (that is done only when converting to/from RGB).

Greets,
I.

[sausuke]

can anyone know how to have thumbnail on windows when recording using magicYUV, it doesn't have thumbnail on raw files thanks

[Ignus2]

can anyone know how to have thumbnail on windows when recording using magicYUV, it doesn't have thumbnail on raw files thanks

Thumbnail in explorer for MagicYUV compressed avi files should just work without doing anything special (apart from having the codec installed).

Greets,
I.

[[SLiM]D12]

So i've capped an avi with MagicYUV -- how does one go about extracting the audio out of the file? I've not found a way to demux the audio out.

thx

[De-M-oN]

Virtualdub

[[SLiM]D12]

Virtualdub

doesn't let me extract the wav, comes out to 0 bytes.

[De-M-oN]

Then your system is broken. It definitely works.

[[SLiM]D12]

Then your system is broken. It definitely works.

kinda hard to believe that. it's a brand new vm with nothing on it really -- and it acts this way on several machines, so nope.

[Sparktank]

Alternatively, you can use ffmpeg:

ffmpeg -i source.avi -map 0:1 -acodec pcm_s24le output.wav

change -map to the actual audio stream (can run it blank in ffmpeg or use ffprobe). and change pcm format if needed (pcm_s16le).

[WorBry]

Bumped by Spartank's post but was going to ask:

Does the source MagicYUV-WAV avi file even play OK, with audio, in VDub?

Are you sure the audio is WAV? What does File Information report?

If yes to both, it's difficult to understand why there would be a problem.

Otherwise, like Spartank said.

[[SLiM]D12]

Alternatively, you can use ffmpeg:

ffmpeg -i source.avi -map 0:1 -acodec pcm_s24le output.wav

change -map to the actual audio stream (can run it blank in ffmpeg or use ffprobe). and change pcm format if needed (pcm_s16le).

thanks this worked!

[Mick]

@WorBry
"Smart Rendering" or "Intelli-Rendering" prevents the NLE to re-compress Audio/Video Material (Streams) if they are in the same (!) Format and Frame Rate.

Example:
Video has NTSC DV (dvsd) as a Codec and Audio is PCM. If you load the Video in the NLE, make the cuts needed and save it again without changing anything else then the NLE makes a "Direct Stream" copy to uphold the quality, just like VirtualDub does in "Direct Stream copy" mode. Every re-compression costs quality and that is where "Smart/Intelli-Rendering" kicks in.

BUT: If you take a Sequence to resize it or to save it with a different Frame Rate, THEN Smart/Intelli-Rendering does NOT work and the NLE re-encodes the whole Material or selected Sequence. In the Parts where Titles or Overlays are added, Color grading and so on, then the same thing happens, the Material gets re-encoded.

If you do NOT Scale or change the Frame Rate and Colors, then ONLY the Parts with Titles and/or Overlays are re-encoded, the rest is untouched with Smart/Intelli-Rendering.

Main thing is, Smart/Intelli-Rendering saves you a heck amount of Time and a 2 hour Movie for example is done is a couple of Minutes. Without Smart/Intelli-Rendering it can take up to Days, depending on your material and complexity.

In a proper NLE/Ingest Setup you use the same Codec for your Previews too. If you load a Video encoded with MagicYUV then select MagicYUV for your Previews too. Why ? Simple: That way you can track errors and is the fastest way to display your Material in a NLE without the need to re-encode the Previews.

You wrote:
"The effective 16-255 luma range is pretty standard for consumer-level digital camcorders and the way it has been since the DV camcorder days."

That's wrong, it's never been 16-255, always 16-235 for Luma and 16-240 for Chroma by definition. 16-255 does not exist, at least as far i know and would not make any sense.

@Ignus2
During my tests with the MagicYUV RC4, Smart/Intelli-Rendering worked absolutely fine if the Source was TFF and encoded with MagicYUV/UtVideo and was exported with MagicYUV/UtVideo. Hate to mention this again but BFF did not work and caused an unusable File with MagicYUV/UtVideo at the end when Smart/Intelli-Rendering was used.

Lagarith and HuffYUV caused no problems with TFF/BFF here and also worked fine with Smart/Intelli-Rendering.

Cheers
Mick

[WorBry]

@WorBry
"Smart Rendering" or "Intelli-Rendering" prevents the NLE to re-compress Audio/Video Material (Streams) if they are in the same (!) Format and Frame Rate.

Example:
Video has NTSC DV (dvsd) as a Codec and Audio is PCM. If you load the Video in the NLE, make the cuts needed and save it again without changing anything else then the NLE makes a "Direct Stream" copy to uphold the quality, just like VirtualDub does in "Direct Stream copy" mode. Every re-compression costs quality and that is where "Smart/Intelli-Rendering" kicks in.

BUT: If you take a Sequence to resize it or to save it with a different Frame Rate, THEN Smart/Intelli-Rendering does NOT work and the NLE re-encodes the whole Material or selected Sequence. In the Parts where Titles or Overlays are added, Color grading and so on, then the same thing happens, the Material gets re-encoded.

If you do NOT Scale or change the Frame Rate and Colors, then ONLY the Parts with Titles and/or Overlays are re-encoded, the rest is untouched with Smart/Intelli-Rendering.

Main thing is, Smart/Intelli-Rendering saves you a heck amount of Time and a 2 hour Movie for example is done is a couple of Minutes. Without Smart/Intelli-Rendering it can take up to Days, depending on your material and complexity.

In a proper NLE/Ingest Setup you use the same Codec for your Previews too. If you load a Video encoded with MagicYUV then select MagicYUV for your Previews too. Why ? Simple: That way you can track errors and is the fastest way to display your Material in a NLE without the need to re-encode the Previews.

Yes, that's what I do and why I do it.

You wrote:
"The effective 16-255 luma range is pretty standard for consumer-level digital camcorders and the way it has been since the DV camcorder days."

That's wrong, it's never been 16-255, always 16-235 for Luma and 16-240 for Chroma by definition. 16-255 does not exist, at least as far i know and would not make any sense.

Please don't ask me to defend that statement as the discussion would go on forever. A cursory web search along the lines of "YV12 16-255 luma scale" would reveal that the 16-255 species does indeed exist and has been a subject of discussion for at least the last 10 years up to present....not least in the context of NLE's and their respective tendencies to interpret and treat the luma levels correctly or incorrectly.

A popular opinion is that camcorder engineers, from all of the major companies, exploited allowances in the Rec.601 and then Rec.709 standards for 8-bit encoding in the 236-254 range to accommodate filter overshoot, transient signals and specular highlights, and used this as a way of gaining a little extra dynamic range and headroom for grading. To what extent that's true or simply that full scale YV12 decoding reveals these "super-white" excursions, I'm not sure.

I do not own a DSLR myself, but, from what I understand on some cameras that record HD video there is even a option to select 16-255 scaling.

@Ignus2
During my tests with the MagicYUV RC4, Smart/Intelli-Rendering worked absolutely fine if the Source was TFF and encoded with MagicYUV/UtVideo and was exported with MagicYUV/UtVideo. Hate to mention this again but BFF did not work and caused an unusable File with MagicYUV/UtVideo at the end when Smart/Intelli-Rendering was used.

Lagarith and HuffYUV caused no problems with TFF/BFF here and also worked fine with Smart/Intelli-Rendering.


What NLE was this with, out of interest?

[Mick]

@WorBry
About Smart/Intelli-Rendering: I don't want to live without it anymore, a real time-saver :)

About YV12 16-255: To be honest, this is new to me and never heard anything about that. Do you have any References (Links) about that ? I certainly did not try to make you look "stupid" here and i apologize if it came over like that.

About my NLE: I use a Cinegy Ingest System on my Workstation and Aist/Cinegy MoviePack Pro and Serif MoviePlus on my Laptop. Occasionally i also use Ulead Media Studio. For the problematic Stuff i use VirtualDub and AviSynth when i can't import Material properly.

Cheers
Mick

[WorBry]

@WorBry
About YV12 16-255: To be honest, this is new to me and never heard anything about that. Do you have any References (Links) about that ? I certainly did not try to make you look "stupid" here and i apologize if it came over like that.

No problem, my self esteem is still intact....just ;)

As for references - well I don't have any definitive reference on file if that's what you mean. But, as I said, a cursory web search would pull up a plethora of forum posts where this has been the topic of discussion, or comes up in the course of. Just a few examples picked at random:

http://forum.videohelp.com/threads/293041-Any-harm-in-using-ConvertToRGB24%28%29-by-default?p=1790620&viewfull=1#post1790620http://

http://forum.doom9.org/showthread.php?t=88512

http://forum.doom9.org/showthread.php?t=165003

http://forum.videohelp.com/threads/296018-DVD-to-DVD-YV12-to-RGB-to-YV12-color-problem?p=1808200&viewfull=1#post1808200

http://forum.doom9.org/showthread.php?p=1540659#post1540659


About my NLE: I use a Cinegy Ingest System on my Workstation and Aist/Cinegy MoviePack Pro and Serif MoviePlus on my Laptop. Occasionally i also use Ulead Media Studio. For the problematic Stuff i use VirtualDub and AviSynth when i can't import Material properly.


I'm not familiar with the first three. Cinegy is broadcast level software right? Ulead Media Studio is what Video Studio Pro was before Corel took it over, a good few years back. Still uses the same old Ulead H264 encoder -one of the reasons why I prefer to do my encoding outside of VSPro with x264.

So, which is the NLE that you tested with MagicYUV and Smart/Intelli Rendering ?

[WorBry]

Pending your reply:


During my tests with the MagicYUV RC4, Smart/Intelli-Rendering worked absolutely fine if the Source was TFF and encoded with MagicYUV/UtVideo and was exported with MagicYUV/UtVideo. Hate to mention this again but BFF did not work and caused an unusable File with MagicYUV/UtVideo at the end when Smart/Intelli-Rendering was used.

I don't normally shoot 60i AVCHD on my Canon HF-G10, but I've just tested this myself with VideoStudio Pro x8 and found that 'smart rendering' does not work at all with 60i MagicYUV or UTVideo YV12 transcodes, only progressive.

I wondered if the reason for this is that the 'smart rendering' process expects BFF, as it would for DV. So I prepared a MagicYUV YV12 transcode from a test 60i MTS clip with the field order changed from TFF to BFF (using the AVISynth Reverse Field Dominance plugin). Loaded that into VSPro x8, re-configured the Project profile settings (for MagicYUV ) to Field Type: Lower Field First. Then split the clip on the timeline to apply an effect (Rotate 180o) to one of the segments and rendered out to MagicYUV as per the Project settings, with Smart Rendering applied.

The rendered avi file emerged with the BFF field order preserved and was perfectly usable except that (as in the first TFF 60i test ) the original 16-255 luma range was clipped to 16-235, indicating that the entire clip had been re-encoded and not 'smart rendered'.

Repeated the test using UTVideo, in like manner, with the same result.

So it's interesting you say that 'smart rendering' worked in your tests using MagicYUV and UTVideo encodes of TFF interlaced sources - and also that:

Lagarith and HuffYUV caused no problems with TFF/BFF here and also worked fine with Smart/Intelli-Rendering.


I suppose I could go on to test that myself but I wouldn't be that interested in using Lagarith or HuffYuv in any case.

Which HuffYuv was that by the way - I'm assuming the FFDShow (ffmpeg) variant in YV12 mode ? And again, in what NLE did you do your tests ?

[Mick]

@WorBry
"Smart Rendering" is used with Ulead Software (now Corel) and "Intelli-Rendering" with Aist/Cinegy Software. And yes, Cinegy is the Broadcast Level Software, including a NLE and Transfer Modules. Aist MoviePack Pro is the NLE Variant of the Cinegy Software.

I still use the Ulead Media Studio 6 on my Laptop, not a Corel Version. As long as TFF was used, MagicYUV and UtVideo worked and caused Problems with BFF when Smart/Intelli-Rendering was used. HuffYUV (CCESP Patch 0.2.5) and Lagarith caused no Problems with TFF or BFF.

Progressive Content is never a Problem using Smart/Intelli-Rendering and with MagicYUV/UtVideo only BFF caused the Problems while TFF worked fine. Many of my Colleagues in Mastering Labs and Studios monitored the same behavior and Problems.

With the Aist/Cinegy Software i can exchange the Render Engines (Direct3D, OpenGL, OpenCL, ProPro, Intel M3 and M4) for Intelli-Rendering and MagicYUV/UtVideo caused Problems with all the Engines (BFF) while HuffYUV/Lagarith worked fine (TFF/BFF).

About YV12 16-255:
I also found some Links and find it rather curious, really, because to me it makes no Sense, but that is another Story. Thank you anyway for the Links and you're right, we should leave this out in this Thread, okay ?

About your 60i Material:
Well, it looks like Smart Rendering did not work if the Clips have 16-235 after the Export instead of 16-255 and is rather strange. How long was the Time to Export the Video ? If it was just a couple of minutes then it might be that MagicYUV clipped the 255 itself to 235. At this point I have no Clue about the behavior of UtVideo you monitored. Did you encode your Material with your NLE or another Tool like VirtualDub to MagicYUV/UtVideo ?

You are on a new Windows Version like 7/8/10 ? If so, then seek the Net for "dxcpl.exe" from Microsoft. It's a little Tool from Microsoft to re-activate the Hardware Acceleration for your Audio/Video Hardware. God only knows why Microsoft deactivated the Hardware Acceleration and the Options from DirectX and the Chances are good that Smart-Rendering then will work properly.

BE CAREFUL: You might find some Registry Hacks to re-activate the Hardware Acceleration in Windows. DO NOT USE THESE HACKS ! The Tool from Microsoft works reliably while the Registry Stuff can really screw up your System, okay ?

The Tool is also included in the "DXSDK_Jun2010" Package from Microsoft. You also find the Tool on certain DVD's of Windows.

Here is a Link: (in German)
http://www.3dcenter.org/download/directx-control-panel

Useful Information about the Tool: (in German)
http://www.wintotal.de/tipparchiv/?id=1804

Microsoft Link:
http://www.microsoft.com/en-us/download/details.aspx?id=6812

Cheers
Mick

[WorBry]

"Smart Rendering" is used with Ulead Software (now Corel)...
I still use the Ulead Media Studio 6 on my Laptop, not a Corel Version.

Wow..Ulead Media Studio 6 was one of the early versions from way back in 2002. Does HD material even run on it? You must surely be editing SD ?

Things have come a long way since then, not least in performance - hardware acceleration and MT support. Corel VideoStudio Pro x7, released in 2013, introduced full 64-bit support and the latest version x8.5 brought further performance boosts, making editing HD with MagicYUV just like DV in the old days. And that's all running on multi-core CPU clout; I only have an MB-integrated Radeon HD3000 graphics unit.


About YV12 16-255:
I also found some Links and find it rather curious, really, because to me it makes no Sense, but that is another Story. Thank you anyway for the Links and you're right, we should leave this out in this Thread, okay ?

Okay.

...it might be that MagicYUV clipped the 255 itself to 235.

Well yes, although its not "clipping" per se- for those frames that are re-encoded MagicYUV is simply taking the 0-255 RGB output and applying Rec709 conversion to yield YV12 with a 16-235 luma range. Whereas those 'smart rendered' frames that are 'passed through' retain the original 16-255 luma. Simple as that. That's why I mentioned above that I do not select the 'Full YUV Range' option in MagicYUV, as then the luma of the re-encoded frames is expanded to 0-255 i.e. with the black point shifted from 16 to 0, making the image darker and I don't want that.

... Did you encode your Material with your NLE or another Tool like VirtualDub to MagicYUV/UtVideo ?

As per my workflow that I outlined above:


....To prepare AVCHD (1080/30PsF) mts clips from my camcorder for editing, I transcode to MagicYUV (with WAV audio) using DGIndexIM and AVISynth via VDub (32-bit).



...
You are on a new Windows Version like 7/8/10 ? If so, then seek the Net for "dxcpl.exe" from Microsoft. It's a little Tool from Microsoft to re-activate the Hardware Acceleration for your Audio/Video Hardware. God only knows why Microsoft deactivated the Hardware Acceleration and the Options from DirectX and the Chances are good that Smart-Rendering then will work properly


Thanks for the info, but I'm really not inclined to meddle with what, for me, "ain't broken". I'm not that interested in smart-rendering of interlaced material or figuring out why it didn't work in this case, as I rarely, if ever shoot interlaced these days.

Cheers.

[Mick]

@WorBry
Ulead Media Studio 6 goes up to 800x600 and I mainly use it for SD Material and Ulead can not deal with HD content. Aist MoviePack Pro goes up to 4K, just like the Cinegy Ingest System. Ulead, Aist and Cinegy use Multi-Processor, Intel IPL, IPP and LeadTools Library's. I use Ulead Media Studio and Aist MoviePack Pro on my Laptop for my own Films when I am abroad to edit, view or to prepare my recordings. For my Clients I use the Cinegy Ingest System.

I specialized myself for VHS/S-VHS Tape Transfers for TV Stations and Film Studios from around the World. I have to deal with many different formats all the time, that's why I had problems with MagicYUV and UtVideo when Clients wanted NTSC with BFF just like the original Content of the Tapes.

The current Format "Hit-List" looks like this:
Uncompressed 4:2:2 16 Bit (YUY2) (Drastic)
JPEG-2000 (MainConcept)
Motion-JPEG (MainConcept)
DV50 (MainConcept)
HuffYUV (YUY2)
HDV MPEG-2 (HonesTech)
DVSD (MainConcept)

Audio is always 48.000 Hz, Stereo/Dual Mono, 16 Bits PCM. The Video Container is either AVI (OpenDML) or QuickTime MOV, both Interlaced.

These are the most demanded Formats of my Clients when I send the Material back to them along with the Tapes after I transferred them. Many Clients tried MagicYUV and UtVideo upon my request but rejected them and wanted HuffYUV back instead. Only a few Clients ask for Lagarith instead of HuffYUV and I can count them on one Hand.

Don't get me wrong, MagicYUV and UtVideo are great codecs but still have some Problems and I am sure they will be solved one day by the Authors. I am not trying to make these codecs look "bad", not at all, they have great potential. At the Moment they just can't make it in the professional Field and need further development, that's all.

Once all the little Bugs are solved, both could replace DNxHD, ProRes or HQX easily one Day. None of my Clients ever wanted DNxHD, ProRes or HQX, not even H264/AVC1 in a MP4 Container. It's funny and makes me wonder if these codecs are really the "Cream of the Crop" as many others claim.

My Goal and the demand of my Clients is to uphold the Quality as much as possible. That's why I use YUY2 in 80 percent of the cases and compress at the very end when everything is done, Grading, Edit, Intro, Outro, Time-codes, Subtitles, Master File and so on, not before.

About Smart/Intelli-Rendering: I forgot the Cinegy MX Renderer, also did not work with MagicYUV/UtVideo in BFF Mode.

About new Systems: Sure, the Systems got faster but better ? Well, that is another Story that we don't dig deeper here into this Sphere, okay ?

About Hardware Acceleration: If it works fine for you being deactivated then don't fiddle with it. God only knows why Microsoft deactivated this great Feature by default. ;)

Cheers
Mick

[WorBry]

Out of curiosity I've run some tests with HuffYUV.

A few years back, before MagicYUV came along, I did some tests with the FFDShow (ffmeg) variant of HuffYuV (in YV12) and it didn't smart render in VideoStudio Pro. Now, with a 64-bit system, the VFW component of 64bit FFDShow (Official and CLSID build) doesn't even show up in the list of available VFW codecs in the VSPro project settings. Why, I'm not that bothered to find out. So that's a non-starter.

Tried instead using the 64-bit version of 'classic' HuffYUV (v2.1.1). Encoded a HuffYUV YUY2 input file and set-up HuffYUV as the Project video format, with default settings. Predictably, the rendered file (with or without Smart Rendering selected) came out RGB. Tried setting the RGB compression option to "Convert to YUY2" and VSPro would have none of it and rendered a corrupt, unplayable file.

So based on that, should I assert that HuffYUV is not fit for professional use and is need of further development simply because it did not behave the way I wanted it to in the particular system I use? Of course not.

I don't know about your Cinegy set-up, but "Smart-Rendering" in VSPro was designed to work with DV. The fact that it works at all with some AVI codecs is fortunate. Cineform doesn't smart-render at all in the VFW environment, so does that mean it's also unfit for professional use? Hardly.

Get my point? It's "horses for courses". I really don't understand what you hope to achieve with these statements, or, more to the point, are expecting from MagicYUV. The fact that it happens to be smart-renderable in one set of circumstances and not another is neither here nor there, and has absolutely no bearing on it's quality and value as a lossless vfw codec. Same goes for any third-party codec.

If 'smart rendering' is so important to you, yet problematic, why not edit in DV - you might find even DV50 works also, as I recall it did when I tested it way back.

nuff said?

Here - I've just been running a trial of Sony Movie Studio 13 Platinum. These are the formats that it smart renders:

http://postimg.org/image/fzph5fak9/

Tried MagicYUV and UTVideo in YV12 mode (input and output) and they definitely don't 'smart render'. Unlike Corel VideoStudio Pro, there's no provision to use third party codecs as the Project video medium, so that's not surprising. But get this - if I configure MagicYUV or UTVideo (YV12) as the output format (in place of default Sony YUV), in both cases the luma gets clipped, not to 16-235, but to 32-235; not good. But, if I instead select the "Full YUV Range" option in MagicYUV, lo and behold, the source 16-255 luma range is re-instated. Magic, huh? So for me, in that system, MagicYUV would be a viable option.

Serves to emphasize the point - different system, different behavior and find the one that works for you.

[Mick]

@WorBry
About Smart Rendering: Ulead and Aist/Cinegy work fine with HuffYUV (CCESP patch 0.2.5) and Lagarith. Right, DV was "the" Format those Days but Smart Rendering is not specially designed for the DV Format. The main thing is that the Input and Output Settings are the same, means Resolution, Field Order and Frame Rate. Changing one of those disables Smart Rendering already.

You seem to misunderstand me, Smart/Intelli-Rendering is no problem on my Systems and is not bound to a specific Codec. MagicYUV and UtVideo just did not work for me if the Field Order is BFF instead of TFF, that's all.

You wrote:
"Serves to emphasize the point - different system, different behavior and find the one that works for you."

That's right and is true. It certainly took me a long time to find my Setup that is stable, fast and reliable. Wouldn't it be great if you could by Hardware and Software that causes no Problems ? ;) As you said... "Different System, different behavior" :)

Cheers
Mick

[WorBry]

About Smart Rendering: Ulead and Aist/Cinegy work fine with HuffYUV (CCESP patch 0.2.5) and Lagarith. Right, DV was "the" Format those Days but Smart Rendering is not specially designed for the DV Format. The main thing is that the Input and Output Settings are the same, means Resolution, Field Order and Frame Rate. Changing one of those disables Smart Rendering already.

Oh boy, this is going round in circles. I was specifically referring to VideoStudio Pro where in order to set-up a project with a third party avi codec, one has to go into a DV profile preset and set-up a custom profile. Once the codec has been selected (from the drop down list of installed codecs) one can then change the Resolution, Field Order and Frame Rate accordingly, in as far as those parameters are permitted by the codec in question. Like so:

http://postimg.org/image/veoyrx4xl/

http://postimg.org/image/h9j5q3vw9/

And then render out as per the Project settings with Smart Rendering:

http://postimg.org/image/ueynw7prt/

None of the other video format presets allow that (the Project configuration, that is).

You seem to misunderstand me, Smart/Intelli-Rendering is no problem on my Systems and is not bound to a specific Codec. MagicYUV and UtVideo just did not work for me if the Field Order is BFF instead of TFF, that's all.

So why even bring it up if BFF interlace editing is not important to you? And if it is, your second statement negates the first.

Wouldn't it be great if you could by Hardware and Software that causes no Problems ? ;)

Depends what you want to do and how much you are prepared to pay. For budget-constrained persons such as I, the choices are limited.

And whilst I'm posting images, this is what the mysterious 16-255 the luma range looks like:

http://postimg.org/image/wxkcwwbi1/

OK, enough now. :o

[[SLiM]D12]

Hi,

Trying to cap with AmaRecTV and i'm getting "Can not support colorspace by video codec"

The only format that lets me see my card in Amarec is

1920x1080 @59.94 fps fcc=r210, bit=30

Any ideas or advice as to how i can cap this with MagicYUV?

thx in advance!

[kolak]

Your card pipes only r210?
This is good: 10bit 444, but not many losses codecs supports it.
MagicYUV does but not in free version.

[[SLiM]D12]

Your card pipes only r210?
This is good: 10bit 444, but not many losses codecs supports it.
MagicYUV does but not in free version.

Ahh and how do you get the paid version? What amount of donation gets you the paid version?

[WorBry]

D12;1753455']Ahh and how do you get the paid version? What amount of donation gets you the paid version?

LOL Good question.

[kolak]

http://magicyuv.com/index.php/contact

ask here.

[kolak]

D12;1753455']Ahh and how do you get the paid version? What amount of donation gets you the paid version?

If you are using Decklink card than switch it to YUV 8bit pixel format.

[De-M-oN]

There is a problem with the auto detection of input with the TMPGEnc 6 NLE.

It always detects 4:2:0 from TMPGEnc and so it is encoded in this color subsampling by MagicYUV, while if I choose UTVideo RGB or any other codec which lets me choose RGB directly, it encodes to RGB and the output video has RGB quality.

[[SLiM]D12]

If you are using Decklink card than switch it to YUV 8bit pixel format.



If I do this, the colors become all washed out. 10 bit has proper color.

[kolak]

It means data is not properly converted from RGB range to YUV range. You should also have 8bit RGB based pixel format, no?

[Ignus2]

D12;1753455']Ahh and how do you get the paid version? What amount of donation gets you the paid version?

None of the released versions have 10-bit encoding. Application support for 10bit+ compressed formats through the general codec interfaces is very weak, a few apps work with it and even there you have to be very careful. It would create too much confusion and give false impressions, that's the main reason it's not included.
Regarding donations, everything is explained on the codec website and the gumroad page.

Greets,
I.

[Ignus2]

There is a problem with the auto detection of input with the TMPGEnc 6 NLE.

It always detects 4:2:0 from TMPGEnc and so it is encoded in this color subsampling by MagicYUV, while if I choose UTVideo RGB or any other codec which lets me choose RGB directly, it encodes to RGB and the output video has RGB quality.

Could you explain your setup here (what kind of input do you encode)?
You can force the codec to RGB by selecting RGB as "Accepted colorspace" in the config dialog. That will restrict the codec to accept RGB only.

Greets,
I.

[[SLiM]D12]

It means data is not properly converted from RGB range to YUV range. You should also have 8bit RGB based pixel format, no?

I think, but i'm not sure how i can force that in Amarectv -- i only get video appear when i set it as i stated above, and i get that error -- maybe i'm doing something wrong?

[kolak]

Set it in the BM directshow filter- you should have access to it in Amarectv. There should be a list of pixel formats to choose from.

[kolak]

None of the released versions have 10-bit encoding. Application support for 10bit+ compressed formats through the general codec interfaces is very weak, a few apps work with it and even there you have to be very careful. It would create too much confusion and give false impressions, that's the main reason it's not included.
Regarding donations, everything is explained on the codec website and the gumroad page.

Greets,
I.

He is using BlackMagic card which sends r210 pixel format, so this wold work well with MagicYUV as it has native support for this format. I've tested it and it was working well when connected to BM directshow filter.

[[SLiM]D12]

Set it in the BM directshow filter- you should have access to it in Amarectv. There should be a list of pixel formats to choose from.

Hrm i'll have to check it out -- i think you mean under "Device Setting" option, but i tried to change that to different settings and none of em worked, i always got the same error about wrong colorspace.

[[SLiM]D12]

He is using BlackMagic card which sends r210 pixel format, so this wold work well with MagicYUV as it has native support for this format. I've tested it and it was working well when connected to BM directshow filter.

Hi,

I've still not been able to figure this out -- any insight or help you can provide would be appreciated.

Thanks,

[djonline]

I still not have MagicYUV in Premiere cc 2015 ->Sequence Settings -> Video Preview -> Codec.

[Ignus2]

I still not have MagicYUV in Premiere cc 2015 ->Sequence Settings -> Video Preview -> Codec.

I'm in the dark here, as I don't have Premiere. Maybe it has a cache and refuses to search for new codecs? Maybe these links can be of some help:
https://forums.adobe.com/message/4595051
https://helpx.adobe.com/premiere-pro/kb/features-presets-missing-premiere-pro.html
Though I doubt this is the problem, but might worth browsing through those posts...

Greets,
I.

[Ghostlamer]

Ignus2, MagicYUV sometimes crash Adobe Premiere Pro CC 2015, premiere crash occurs when magicyuv.dll_unloaded (in windows events).
For example, while sony vegas keeps magicyuv.dll always loaded, Adobe Premiere Pro periodically load (when i play video at timeline)/unload (when playback is stopped) magicyuv.dll.

[wiak]

i find this codec to be twice as fast as lagarith on my 4-core amd laptop
thats the good thing, but the bad thing is that it would been nice if there was a open source decoder* that can be compiled into ffmpeg
currently i dont think this codec will be useful for archiving, what will happend down the line when the homepage is gone? and we cant download the decoder?

please release a open source decoder on github etc

[[SLiM]D12]

So today i've got a weird problem. On a machine that i cap on, i'm getting severe banding, but if i move to another machine, there is no banding at all. My avs only has avisource and the spline36resize command -- so i'm looking for guidance as to what could be the reason for banding on one pc and no banding on the other?

http://puu.sh/nb9yw/867bc04d02.jpg

http://puu.sh/nb9zf/817105e63d.jpg


Edit: I just tried this on a couple more pc's -- installed the Magic codec and no banding there -- so not sure what's going on my capping/encoding box -- hopefully somebody can point me in the right direction-- thx!


I got it sorted -- seems my rdp session to this remote pc wasn't set to 32bit -- oops.

[Ignus2]

Ignus2, MagicYUV sometimes crash Adobe Premiere Pro CC 2015, premiere crash occurs when magicyuv.dll_unloaded (in windows events).
For example, while sony vegas keeps magicyuv.dll always loaded, Adobe Premiere Pro periodically load (when i play video at timeline)/unload (when playback is stopped) magicyuv.dll.

Which version of the codec?

Greets,
I.

[Ignus2]

please release a open source decoder on github etc

There are plans about this, but I am currently swamped with other work. But unless a bus hits me till that time, the format will be released at some point :)

Greets,
I.

[Ghostlamer]

Which version of the codec?
Latest - 1.2.

[Ignus2]

Latest - 1.2.

There was a bug in the 1.1 release that specifically occured when the codec DLL was unloaded and then reloaded, but it should have been fixed in 1.2.
As I don't have Premiere I don't know when/if I'll be able to debug it.
In any case if you can provide steps to reproduce that would be helpful.

Greets,
I.

[[SLiM]D12]

very excited for the 10-bit encoding for this codec -- i've emailed for the private release, is there any other process or do i just wait? PM here perhaps? i wasn't 100% sure.

thx!!

[ChiDragon]

I've finally paid for this fantastic piece of software! After freeloading for 2 years. ;)

A couple feature suggestions, since the Gumroad page asks for them and the text box there looks small. Plus I didn't think of the first one until now. :o

Add Auto mode selection for YUV color matrix setting, based on SD vs HD resolution (picture height) of input. Or keep the existing dropdown as it is, and add a checkbox to override that selection with Auto when the input exactly matches a standard: [720|640]x480, [720|768]x576, 1280x720, 1920x1080, 3840x2160, 7680x4320. I frequently switch between doing SD/ED and HD captures using the same capture device, so I have to go into the Compression menu merely to toggle this option.
Tossing in my support for the previous "open source decoder" desires echoed by several people. If you take a look at forum discussions of lossless codecs, people tend to frown upon MagicYUV due to its lack of FOSS decoding.

[Ignus2]

Thanks for your support! :)
1. Auto mode for YUV matrix makes sense. Can you give an exact list when it should be SD?
2. Yes, it is on my list. My only problem right now is time, I am simply too overwhelmed with various work right now, which I hope will consolidate on a more manageable level in the near future.

Greets,
I.

[zerowalker]

The standard i think is that anything < 720p (16/9 or 4/3) is considered SD. But it kinda differs, but i think that's the general consensus from my experience.

[vivan]

I believe the consensus is width bigger than 1024 or height bigger than 576 means HD:
- 1024x576 is 16:9 PAL DVD
- 960x720 is 4:3 720p
- 1280x544 is 16:9 720p

[zerowalker]

That's indeed more correct than the vague standard i put forward.
Then again there aren't really any resolutions in between in most cases.

[Asmodian]

That's indeed more correct than the vague standard i put forward.
Then again there aren't really any resolutions in between in most cases.

I think significant difference is that there might be cropped 720p content but there should be virtually nothing SD above 1024x576. ;)

[zerowalker]

Cropped 720p content, do you mean that it's below 720p then?
Or well from what i first wrote that must be the case.

And that's pretty much what i think is better with that 1024x576 max as well.
Cause there might be "PC Encoded material" to put it lightly, that doesn't follow the "standard" so to speak.
But as for Real videos, there won't be. But hey, world isn't that fancy now is it;P

[ChiDragon]

1280x544 is 16:9 720p
Well... "was" 16:9 (with bars), since it's obviously 2.35:1 cropped.

[EncodedMango]

Is there any way I can pipe a MagicYUV file into ffmpeg to encode to whatever I want(H264/VP8)? So far I have never bothered to find out and just used Virtualdub to edit(if required) and then change MAGY to raw uncompressed since and then throw it into ffmpeg for encoding but it's time consuming and tedious and seems like redundant effort.

Usually I record with MAGY and at the end I require the final to be H264.

[zerowalker]

With Avisynth it works, don't think otherwise cause ffmpeg can't decode it.

[EncodedMango]

With Avisynth it works, don't think otherwise cause ffmpeg can't decode it.
What would be the chain in that case, process the .avs file with ffmpeg?

Can you tell me what would be the lines inside the .avs? Thanks.

[zerowalker]

Well it would just be.

Avisource("X:\something\whateverMagicYUV.avi")

As far as i know:)

[raffriff42]

Is there any way I can pipe a MagicYUV file into ffmpeg to encode to whatever I want(H264/VP8)?
You can go direct to ffmpeg and/or x264 using VirtualDub's External Encoder feature:
http://forum.doom9.org/showthread.php?t=171141

[EncodedMango]

Well it would just be.



As far as i know:)
Needed to update AviSynth+ but this worked. Thanks.

You can go direct to ffmpeg and/or x264 using VirtualDub's External Encoder feature:
http://forum.doom9.org/showthread.php?t=171141
The scripts are bit complex for what I consider a very mundane task but it works and that's great, thank you.

[EncodedMango]

On a separate note. I can't seem to select the 64bit variant of MagivYUV in MSI Afterburner and as far as I'm aware MSI AB supports 64bit codecs. I ahve the dedicated encoder server selected and set it it up to prefer 64 bit codecs but it just doesn't show in the list.

Dxtory however shows the 64bit codec in the list.

[Ignus2]

On a separate note. I can't seem to select the 64bit variant of MagivYUV in MSI Afterburner and as far as I'm aware MSI AB supports 64bit codecs. I ahve the dedicated encoder server selected and set it it up to prefer 64 bit codecs but it just doesn't show in the list.

Dxtory however shows the 64bit codec in the list.

It depends on the application itself if it is 32 bit or 64 bit.
I assume the MSI application where you do the selection is 32 bit. During actual capture though it uses probably the one depending on the game running (32/64 bit).
The Dxtory app itself is probably 64 bit, so that's why it shows 64 bit. But even for Dxtory, if the game is 32 bit, it will still use the 32 bit codec during recording even if the selection dialog shows 64 bit.

To see actually which variant of the codec is used, turn on logging in the MagicYUV settings dialog during codec setup, do an encoding session, then look in the logfiles in the log dir. At the top it will show the EXE of the game and the exact variant of the codec used.

Greets,
I.

[EncodedMango]

To see actually which variant of the codec is used, turn on logging in the MagicYUV settings dialog during codec setup, do an encoding session, then look in the logfiles in the log dir. At the top it will show the EXE of the game and the exact variant of the codec used.

Greets,
I.
Yeah I turned on logging and it does indeed use the x64 variant when recording. That's nice, thank you for that.

On checking the recording log btw, it says:
Input format: RGB24 (bottom-up)

What does bottom up mean? The image certainly isn't inverted for me.

[Ignus2]

Yeah I turned on logging and it does indeed use the x64 variant when recording. That's nice, thank you for that.

On checking the recording log btw, it says:
Input format: RGB24 (bottom-up)

What does bottom up mean? The image certainly isn't inverted for me.

:)
It's historical. On Windows, RGB images are most often stored in memory upside down or bottom-up. They can be top-down too, which is indicated by a negative height (though I don't write it to the log like that, would be too confusing, instead I just indicate the orientation).

You can read more about it here if you're interested:
https://msdn.microsoft.com/en-us/library/windows/desktop/dd407212(v=vs.85).aspx

It's implementation detail, so no concern for users, but good to be in the log for debugging purposes.

Greets,
I.

[Ignus2]

I guess it can be called a historical moment (at least for me): MagicYUV has begun to make it's way into ffmpeg :)
This is good news for all people using tools based on ffmpeg as their decoder (HandBrake comes to mind), allowing easier transcoding (for uploading to YouTube for example).

The implementation was done by an ffmpeg developer essentially by reverse-engineering the bitstream of the codec. Though I think it would have been easier to ask me about it, but I guess that would have taken out the fun of reversing. Since then I sent him a quick description of the headers and format, etc. (though he already figured out the bulk of it).
I certainly wouldn't have had the time to do this by myself (not being familiar with ffmpeg sources at all). As time permits though I'll try to make sure the decoder in ffmpeg works correctly for new releases as well.

On a sentimental side of this, it is a great feeling having something done initially as a hobby turning worthy for investing the time and effort to reverse it and integrating it into the mostly widely used tool in the video coding field.

Greets,
I.

[poisondeathray]

ffmpeg support is great news!

I know it's early, but some questions"

1) has ffmpeg implemented decoding support only, or encoding as well, or is encoding planned for later ?

2) any early benchmarks /numbers for speed compared to VFW version ?

[Ignus2]

ffmpeg support is great news!

I know it's early, but some questions"

1) has ffmpeg implemented decoding support only, or encoding as well, or is encoding planned for later ?

2) any early benchmarks /numbers for speed compared to VFW version ?

This is all very early, the commit is 2-3 days old, no benchmarks whatsoever.
Decoding only.

Greets,
I.

[richardpl]

And decoding only 8-bit depth samples. Unless someone provide >8 bit ones.

[zerowalker]

ffmpeg decoding, awesome.

What can one expect from the performance, worse, same, better?

[richardpl]

Try it and report back. I'm still looking for 10-bit samples

[Ignus2]

Sorry for being late with the response, I'll reply to all requests soon, I'm just a bit preoccupied right now...

Greets,
I.

[wiak]

magicyuv 1.2 seems to crash games when used in msi afterburner :/

[De-M-oN]

No it doesnt.

[Jalkion]

It does though, it works fine in 32 bit games but as soon as you try to record in a 64 bit game the game insta crashes.

It obviously has no problems when you use dedicated encoder server.

[wiak]

It does though, it works fine in 32 bit games but as soon as you try to record in a 64 bit game the game insta crashes.

It obviously has no problems when you use dedicated encoder server.
No it doesnt.
yes it does, my hd 7970 playing the division crashes, playing unreal tournament (reboot) also crashes, and this is even after a windows reinstall

FYI yeah will try the encoder server, i didnt think of 32-bit vs 64-bit, but who runs 32-bit games now? when a game can use like 7GB by it self


FX-8350
32GB DDR3
HD 7970 3GB
MagicYUV 1.2
Afterburner 4.3 beta (same with non-beta)

[De-M-oN]

I always use dedicated encoder server. its faster and better.
And use framebuffer, not viewport..
I never had crashes with MagicYUV.

[wiak]

I always use dedicated encoder server. its faster and better.
And use framebuffer, not viewport..
I never had crashes with MagicYUV.
works fine for me with the encoder sever, didnt know that had that big of affect, viewport? you mean full frame?

[De-M-oN]

Look at RTSS Settings, there are settings for your capture:

I would use framebuffer instead viewport (much more compatible to games to read from framebuffer)
You can set to use the 64bit encoder of your codecs (which also only works by using the dedicated encoder server) (config dialogs will be still 32bit, but afterburner will use 64bit encoder then. MagicYUV's tray symbols will confirm that.
You can set "show own statistics" to on to have data displayed while capture (filesize counter, fps, compression ratio, compression time)

[zerowalker]

Try it and report back. I'm still looking for 10-bit samples

Didn't do any deep testing, just a quick playback when i was doing some stuff before, and MagicYUV decoding on ffmpeg seems to be quite a bit slower.

I just looked as CPU Usage so not much to go after as they were fairly close anyway though.
Maybe it's around 20% slower, but don't take my word for it, at least i am pretty sure it's not faster;P!

But for some reason ffmpeg pretty much never is faster.

[richardpl]

How you can tell it is not faster because it use less CPU? Your logic is flawed. Also last sentence is crap.
Looking for real numbers and not some claims.

[zerowalker]

How you can tell it is not faster because it use less CPU? Your logic is flawed. Also last sentence is crap.
Looking for real numbers and not some claims.

Hence why i said i hadn't done deep testing,
it's what you call "take it with a grain of salt" :)

As for the ffmpeg being slower sentence,
that's from my experience with Lagarith and Utvideo.
At least when i did the tests both were quite a lot slower and used a lot more CPU.

How i can tell it's faster if it uses less CPU is simple,
play a video, if it uses less CPU for that framerate then the conclusion is you will be able to push a higher framerate before hitting the limit.
Pretty sure that's valid, at least if both are limited to the same amount of threads.

[kolak]

You test it with ffmpeg this way:

Make RAM disk (https://www.softperfect.com/products/ramdisk/), copy source there and run:

ffmpeg -i "source" -f null -

this will force decoding to be as fast as it can be.

Ignus2 use to have testing tool for avi source which would test different levels of decoding: reading, pure decoding speed (from RAM) etc.
I don't think ffmpeg decoding is anywhere near as fast as native decoder.

[richardpl]

Why people keep thinking instead of providing numbers. FFmpeg decoder have SIMD optimizations too.

[kolak]

Last time I tested it was no near as fast.

[raffriff42]

I imagine the first priority is correct decoding, without crashing -- which users (this means you) can help move forward, by testing. Optimization will come later.

[poisondeathray]

quick test observations

1920x1080p29.97, 4:2:0, 1800 frames, ramdisk, 3 runs

x86 VFW decoder, tested though avisynth avisource(), avsmeter
~680-700FPS

ffmpeg x64 -i input.avi -c:v rawvideo -an -f null NUL
~220-230FPS

avs through ffmpeg32 (ie x86 VFW decoder, through avisynth, then ffmpeg)
ffmpeg32 -i input.avs -c:v rawvideo -an -f null NUL
~420-430FPS

[richardpl]

Something is really fishy about your results. Make sure that your build use multiple threads and is build with yasm.

[poisondeathray]

Something is really fishy about your results. Make sure that your build use multiple threads and is build with yasm.

I used a recent ffmpeg binary from zeranoe
https://ffmpeg.zeranoe.com/builds/

manually specifying -threads 4,8, or 16 (on a 4C/8T CPU) didn't make much difference , +/- a few fps

[richardpl]

And if using -threads 1 gives similar fps than something is seriously broken on your side.

[kolak]

Something is really fishy about your results. Make sure that your build use multiple threads and is build with yasm.

For me looks fine (had similar results). ffmpeg code is no near as fast as original.

[kolak]

I used a recent ffmpeg binary from zeranoe
https://ffmpeg.zeranoe.com/builds/

manually specifying -threads 4,8, or 16 (on a 4C/8T CPU) didn't make much difference , +/- a few fps

Did you put -threads in front of -i ?

[richardpl]

It is unfair to compare if threads and SIMD are disabled. Make sure also that your build is with yasm enabled.

[poisondeathray]

Did you put -threads in front of -i ?

yes.




I think threads are "auto" that's why it doesn't make a difference on zeranoe's . I don' t know how it's compiled or if yasm is enabled

I don't know how to compile ffmpeg manually - I only know how to use the auto build script. The last time I did it was before the magicyuv commit. I don't have time to build it right now

Other people are welcome to post test observations

[poisondeathray]

And -threads is definitely doing something with that ffmpeg build , because -threads 1 yields ~40 FPS, -threads 2 ~75FPS

[kolak]

It does an it proves that auto does good job also (that's why you had not much of a difference in your tests).
Your results are in line with mines, even if I've done it some time ago.

[poisondeathray]

It does an it proves that auto does good job also (that's why you had not much of a difference in your tests).
Your results are in line with mines, even if I've done it some time ago.

Yes, but it could be that there are problems with the ffmpeg build or implementation . ~1/3 as fast is significantly slower so there might very well be other issues. To put things into perspective the ffmpeg UT Video implementation is about 2/3 as fast as the native VFW decoder. I would have expected the magicyuv ffmpeg implementation to be similar

I don't trust myself to build ffmpeg from scratch with the ideal switches or how to do it properly, and the autobuild script has just a few options. If some ffmpeg expert gets significantly different trends then please chime in

However, this zeranoe distributed build is very popular with Windows users, so those observations are roughly what they can expect on Windows

[kolak]

I can build ffmpeg myself, but only based on instructions :)

I see zeranoe's builds as "reference" and don't think you can simply add such a speed boost by "compiling in a different way". I assume you need to improve code to get such a speed difference. Different compilers may give 5-10% speed difference but not 50% or so (I think).

[richardpl]

Its funny that in your case 64 bit version is about 50% slower than 32 bit one.

[poisondeathray]

Its funny that in your case 64 bit version is about 50% slower than 32 bit one.

You can't really draw that conclusion from those observations, because the VFW decoder tested was 32bit one, not 64bit.

Only 64bit ffmpeg was compared for the ffmpeg magicyuv implementation, because decoder was not present in 32bit ffmpeg version , at least in zeranoe's version

A proper comparision would compare 64bit VFW and 32bit VFW , 64bit ffmpeg and 32bit ffmpeg

The only conclusion that is supported from those observations is the 64bit ffmpeg implementation is markedly slower than 32bit VFW native, at least on windows

[poisondeathray]

And another test with 32bit l-smash (compiled with libav sources) avisynth, avsmeter ~230FPS . This is a directly comparable "apples to apples" comparison to the AVISource VFW avisynth, avsmeter results. So it suggests libav / ffmpeg implementation not as optimized as native VFW

[raffriff42]

Testing decoders at 250-450 fps is invalid IMHO, as there are probably unknown bottlenecks at that speed, such as RAM bandwidth. But as long as you're doing these tests, why not compare MagicYUV to some other fast codecs, say, YUV uncompressed and HuffYUV.

[poisondeathray]

Testing decoders at 250-450 fps is invalid IMHO, as there are probably unknown bottlenecks at that speed, such as RAM bandwidth. But as long as you're doing these tests, why not compare MagicYUV to some other fast codecs, say, YUV uncompressed and HuffYUV.

I think it's perfectly valid, and there is very strong correlation with those synthetic numbers (at least the trend) with actual usage in a NLE or a final encoding codec. ie. Something at say 600fps will have a higher final speed and faster operations than something at 400fps.

I've posted bencharks / observations on UT, MagicYUV, Huffyuv before (either here or videohlep), as well as the effect on actual encoding speed on a final encoder like x264 , not just synthetic benchmarks. The summary is magicyuv (VFW) is the fastest by a hair, overtaking UT Video

There is absolutely no reason to use Huffyuv. UT Video is more compressed, more colorspace support (huffyuv original didn't even support 4:2:0, and the final encoding speed with x264 was about 5-10% faster). Stability is high for UT video in host applications like NLE's, compositing programs, but I cannot say the same for MagicYUV only because it's not as extensively tested in NLE's etc... It simply hasn't been around long enough yet

The one issue windows users will be having is QT support is removed. QT for windows is no longer developed. The QT MOV versions for both magicyuv and utvideo are actually more compatible in some professional programs, because AVI has been depreciated out, and higher depth support is problematic. But now MOV is being phased on (at least the QT implementation, some programs have their own native QT implementation that doesn't rely on Apple QT)

[raffriff42]

Agree, HuffYUV is not the way to go any more. I meant to say, compare them on speed only, as a way to cancel out (as much as possible) everything outside the decoder. They have roughly similar compression ratios, therefore roughly similar data throughput.

[Music Fan]

I've posted bencharks / observations on UT, MagicYUV, Huffyuv before (either here or videohlep), as well as the effect on actual encoding speed on a final encoder like x264 , not just synthetic benchmarks. The summary is magicyuv (VFW) is the fastest by a hair, overtaking UT Video
For a capture in real time (analog to digital), do you advise UT, MagicYUV, or it doesn't matter and both are good for this ?

[wiak]

For a capture in real time (analog to digital), do you advise UT, MagicYUV, or it doesn't matter and both are good for this ?
most lossless ones are good, eerm they are lossless, you should probobly test on your content

there can be some big size diff and performance diff based on content

[poisondeathray]

For a capture in real time (analog to digital), do you advise UT, MagicYUV, or it doesn't matter and both are good for this ?

Both are good.

But depending on what programs you are using afterwards , or your specific workflow is - that might influence which you use.

[Music Fan]

Ok. I need avi container (easier to use with Virtual Dub and others) : I generally make some denoising with avisynth and sometimes editing with Vegas.
For the capture in real time, I believe the least possible CPU (and maybe RAM) use is better.
And I need 4:2:2. Until now I used to capture in Lagarith yuy2 but I read that it could bug sometimes.
Ut seems ok, I never tried MagicYUV.

[poisondeathray]

Ok. I need avi container (easier to use with Virtual Dub and others) : I generally make some denoising with avisynth and sometimes editing with Vegas.
For the capture in real time, I believe the least possible CPU (and maybe RAM) use is better.
And I need 4:2:2. Until now I used to capture in Lagarith yuy2 but I read that it could bug sometimes.
Ut seems ok, I never tried MagicYUV.

I would use MagicYUV if Vegas is somewhere in the workflow

Because the full range option in MagicYUV gets treated as "Studio RGB" in vegas . Other lossless codecs in YUV mode get clipped in Vegas , because they get treated as "Computer RGB"

[Music Fan]

Ok, interesting.
For captures I already did in Lagarith, I guess I can convert them in MagicYUV yuy2 if I need to edit them in Vegas.
By the way, why are they treated differently by Vegas ? Isn't there any trick to avoid clipping ?

[kolak]

Agree, HuffYUV is not the way to go any more. I meant to say, compare them on speed only, as a way to cancel out (as much as possible) everything outside the decoder. They have roughly similar compression ratios, therefore roughly similar data throughput.

If you test pure decoding speed in RAM then there should be no bottlenecks and definitely not at 400fps for HD frame (this is just 1.6GB/sec). I had speeds way above 1000fps on dual Xeon machine.

[poisondeathray]

Ok, interesting.
For captures I already did in Lagarith, I guess I can convert them in MagicYUV yuy2 if I need to edit them in Vegas.
By the way, why are they treated differently by Vegas ? Isn't there any trick to avoid clipping ?

If you adjust your captures at the source to Y = 16-235 , you'll be "safe" using other lossless YUV codecs. It's only if you have "superbrights" or "superdarks" that you'll lose data in those regions

I don't know why it's treated differently; vegas does all sorts of things differently. Generally, native camera sources get studio RGB treatment, everything else gets computer RGB treatment, including most lossless YUV codecs. You can read about it more in the Glenn Chan article series or do some testing

[Music Fan]

If you adjust your captures at the source to Y = 16-235
How to do this ?
With Lagarith, there is only the choice between RGB, yv12 and yuy2, I chose yuy2.
With Ut there is rgb, rgb 10 bit, 4:2:0 BT 601, 709 ... I would select 4:2:2 BT 601 for VHS captures.

And with MagicYUV I don't know how to select it, but I just saw it was not free thus I probably won't test it, I make too few captures to buy it.

You can read about it more in the Glenn Chan article series
Where is it ?

[poisondeathray]

How to do this ?


Adjust your capture hardware setup so everything is in "legal range". Otherwise it will get clipped when using YUV lagarith or ut video in vegas




Where is it ?

Search for "Glenn Chan" and "Vegas"

http://www.glennchan.info/articles/articles.html

[Music Fan]

http://www.glennchan.info/articles/articles.html[/url]
Thanks.

Adjust your capture hardware setup so everything is in "legal range".
I believe all I can do is to select the appropriated colorspace (4:2:2) in the capture software, there is nothing about 16-235 and 0-255.

[ChiDragon]

He means you need to manually ensure the levels are set so that nothing you care about reaches outside 16-235, using the capture device's proc amp, an external proc amp, etc.

[Music Fan]

Ok, but I don't know how to do this, it is maybe not possible with my hardware. Anyway I guess that the card does only 16-235 and adapt the signal for this when digitizing the analog signal.

[Micheal813]

I just found MagicYUV today. I'm using Avisynth with VirtualDub. I captured video and compressed it with MagicYUV as YUY2. At a point in my workflow I'm trying to convert the YUY2 to RGB24 using MagicYUV. I know I can not upsample directly so I add ConvertToRGB24 to the end of my script. VirtualDub is set to Fast Recompress and the Color Depth output format to compressor is set to same as decompression format. I also have compress as is selected in MagicYUV.

The saved file shows RGB32. Now I'm no expert, so this may be completely normal. Is RGB32 & RGB24 the same as far as MagicYUV is concerned? Am I misunderstanding something?

[mariush]

In Virtualdub, you can choose the decompression format and the format sent to the compressor. Go in the Video menu, select Normal Recompress (or Full Processing Mode, but at least Normal Recompress is required) and from there, select 24bit RGB (888).
If you know the source format, in some cases you may want to select the decompression format there instead of leaving it on autoselect.
In rare cases, some codecs have quirks and compress yuy2 content but when an application like virtualdub opens the file, the same codec enumerates several output formats (for example rgb24, yv12 and last yuy2) and by default they do an extra color space conversion to give Virtualdub RGB24 or something like that. By changing the autoselect, you may sometimes speed decoding by forcing the codec to not do a color space conversion before passing the raw video frames to Virtualdub.

Of course, you could also set YUY2 at both decompression format and output to compressor, so the frames just pass through Virtualdub and configure MagicYUV to do the conversion from YUY2 to RGB24 inside.

You should also keep in mind that most modern codecs like h264 (with x264.exe as most known encoder) or hevc (x265.exe being most known encoder) default to using YV12 and Youtube and other video playback sites default to YV12 so whatever you'd upload would be reconverted. So it makes no sense to keep videos around in RGB24 unless they're for your own use, for archival, some specialized cases like old video game captures etc

[Micheal813]

In Virtualdub, you can choose the decompression format and the format sent to the compressor. Go in the Video menu, select Normal Recompress (or Full Processing Mode, but at least Normal Recompress is required) and from there, select 24bit RGB (888).

Thanks for the response, but this did not work. I want it to be RGB24 for Neat Video denoiser. Neat Video works in and outputs RGB24. I know it can just do the conversion itself, but I like to take care of the conversions myself when possible.

My script is:

Avisource("YUY2Video.avi")
AssumeTFF()
ConvertToRGB24()

I open this with Virtualdub and select the above settings. If I add Info() to the script it show Colorspace as RGB24. I save the avi. Then to check it, I use this script:

Avisource("NewVideo.avi")
AssumeTFF()
Info()

It shows RGB32 as the colorspace.



Of course, you could also set YUY2 at both decompression format and output to compressor, so the frames just pass through Virtualdub and configure MagicYUV to do the conversion from YUY2 to RGB24 inside.

I thought MagicYUV could not do upsampling. Isn't YUY2 to RGB24 considered upsampling? Am I misunderstanding? I don't see any setting to make it convert the YUY2 to RGB24.



You should also keep in mind that most modern codecs like h264 (with x264.exe as most known encoder) or hevc (x265.exe being most known encoder) default to using YV12 and Youtube and other video playback sites default to YV12 so whatever you'd upload would be reconverted. So it makes no sense to keep videos around in RGB24 unless they're for your own use, for archival, some specialized cases like old video game captures etc

Yes, after I convert it to RGB24, and process it with Neat Video's VDub filter, I plan on converting to YV12 and compressing with x264 to save a lot of space. I have lots of video to process.

[Music Fan]

You can put converttorgb32() instead of ConvertToRGB24(), this way you are sure the conversion to rgb32 is done by avisynth, that's what I do when using Neat.
And you don't have to create any lossless video before to compress in x264, you can use Neat in your avisynth script (look at Neat's user guide).

[Micheal813]

You can put converttorgb32() instead of ConvertToRGB24(), this way you are sure the conversion to rgb32 is done by avisynth, that's what I do when using Neat.
And you don't have to create any lossless video before to compress in x264, you can use Neat in your avisynth script (look at Neat's user guide).

So going from YUY2 to RGB32 will be just a good as YUY2 to RGB24? I'm trying to keep as few color conversions a possible.

Neat works in RGB24. Is capturing with MagicYUV in YUY2 ----> RGB32 for Neat ---> YV12 for x264 just the same as MagicYUV in YUY2 ---> RGB24 for Neat ---> YV12 for x264?

I'm really just not understanding why it is not being saved as RGB24 when it shows up as RGB24 in Vdub when I add Info() to my script.

I really don't want to bother learning how to use Neat with Avisynth. I'm more comfortable with the Neat GUI in VDub. I convert to x264 with MeGUI or Handbrake.

[kolak]

If you see RGB24 in Source Information this means nothing.
What goes to codec depends on your setting in out section for Color Depth settings:

http://forum.videohelp.com/attachment.php?attachmentid=12456&d=1337168317

If you have it set this way (or RGB32) than MagicYUV should use RGB mode for encoding. MagicYUV may require RGB32, I'm not sure, but you can try.

[Micheal813]

If you see RGB24 in Source Information this means nothing.
What goes to codec depends on your setting in out section for Color Depth settings:

http://forum.videohelp.com/attachment.php?attachmentid=12456&d=1337168317

I have tried that but it still ends up as RGB32 according to VLC Media Player and VDub.

[kolak]

This is fine- I think this is how MagicYUV works. Most likely original data is RGB24, but it's decoded to RGB32 (RGB24+ dummy/empty alpha channel) for better compatibility with apps. Nothing to worry about.

[Music Fan]

I really don't want to bother learning how to use Neat with Avisynth. I'm more comfortable with the Neat GUI in VDub.
I do both :
-I open my script (without Neat function) in Vdub,
-configure Neat,
-save the settings (dnp and nfp files),
-close Vdub,
-add the Neat lines in the script (referring to these dnp and nfp files),
-open the script in the x264 encoder (I use Hybrid).

[djonline]

Still can't use MagicYUV as preview codec format in Adobe Premiere, like described in http://forum.doom9.org/showpost.php?p=1746640&postcount=415
Yes, i see it in vdub

[Ignus2]

Still can't use MagicYUV as preview codec format in Adobe Premiere, like described in http://forum.doom9.org/showpost.php?p=1746640&postcount=415
Yes, i see it in vdub

I'll try to take a look.

[Ignus2]

Still can't use MagicYUV as preview codec format in Adobe Premiere, like described in http://forum.doom9.org/showpost.php?p=1746640&postcount=415
Yes, i see it in vdub

OK, here is the solution:

Open this file with a text editor: c:\Program Files\Adobe\Adobe Premiere Pro CC 2015.3\Settings\EncoderPresets\SequencePreview\9678AF98-A7B7-4bdb-B477-7AC9C8DF4A4E\Microsoft AVI.epr
Search for the line: <PresetComments>...</PresetComments>
For me it looks like this:
<PresetComments>IsConstrained,dsv2,dsvn,dsvp,VUYI,ELRM,CVSM,YVYU,012v,BID,SGAL,2YLU,GRLU,ARLU,g01R,k01R,k01r,yuV2,yuv2,yvyu,DHFC,dsvd,52vd,1hvd,05vd,SEMM,407M,107M,507M,207M,201M,401M,101M,301M</PresetComments>
I think you are getting the clue by now...
Remove the content between those tags, so it looks like this: <PresetComments></PresetComments>

After this, all the VFW codecs should show up.

Now, whether they will work or not, that's a different question. I don't know, and as I haven't got the foggiest at all how to use Premiere, I didn't test, but if you do, please report back your findings.

rant:
Did I mention I LOVE this type of hackery? It's the same madness you have to do for 10bit+ QuickTime codecs to show up in AE. Only this time, it's a hard-coded list in the app directory.
Next time we'll be hex-editing DLLs...
Whoever comes up with these brilliant ideas of hard-coded FCC lists should be awarded something...

Greets,
I.

[Rotareneg]

In Sony Movie Studio (and Vegas) 13, the software can pass frames directly without recompression if they haven't been changed, which can really speed things up. This works fine with the Ut Video codec, but I cannot get this to work with MagicYUV. Are there any particular settings in MagicYUV that I might be able to change to make it work?

[Ignus2]

In Sony Movie Studio (and Vegas) 13, the software can pass frames directly without recompression if they haven't been changed, which can really speed things up. This works fine with the Ut Video codec, but I cannot get this to work with MagicYUV. Are there any particular settings in MagicYUV that I might be able to change to make it work?

Not in particular. I don't know how MS/Vegas does smart rendering so I'm not sure what would be needed to make it work. It would be nice to know what do those programs look at to make the decision to smart render.
Maybe they too have a hard-coded list of fourccs (I wouldn't be surprised) :)

Greets,
I.

[Rotareneg]

Ah, I think I figured out what the problem might be: I'm recording from games using Dxtory, which outputs 32 bit RGB data (RGBA compression is unchecked in MagicYUV) If I re-compress that video as 24 bit RGB smart rendering will work fine in MS. Videos recorded from Dxtory with MagicYUV set to convert to YUV420 won't smart render either.

It would seem MS/Vegas smart rendering does not like it when a codec supports multiple color spaces on the same fourcc. I was able to get MS to spit out a corrupted magicYUV .avi by re-compressing a RGB24 video through MS with MagicYUV converting to YUV420, and then trying to smart render that file. The MagicYUV tray icon will show it's converting RGB24 to RGB while MS is smart rendering the YUV420 video, which crashes anything, including the windows explorer when it trying to build a preview for it.

Here's a link to the corrupted avi, packed in a 7zip archive: https://drive.google.com/file/d/0BzS6i2JCDwdteUJoVFZuTmdkcXM/view?usp=sharing

Beware, it'll crash explorer.exe if it tries to build a thumbnail for the file, it's that messed up!

[Ignus2]

Great, thanks for the info! I might have some ideas here.
Could you also share:
- a sample which is recorded directly with dxtory and refuses to smart render (either compressed RGB or YUV420 or both)
- a recompressed sample of the above which actually smart-renders
That would help a lot, as I could look at what the differences are.

About the crash: yes, as there are mixed frames in the container AVI, the codec barfs. It should check for this at least and either refuse to decode the mixed-up frames or do something about it. So it can be considered a bug also in the codec.

In any case, the next release will have separate fourccs, the current solution is problematic for multiple reasons (the alpha checkbox being one prime example, among others).

Greets,
I.

[Ignus2]

On a side note: Are you the one, who reported the following issues with dxtory?
http://forum.exkode.com/index.php?topic=1793.0
http://forum.exkode.com/index.php?topic=2165.0

In fact, I remember someone writing to me about similar strange issues, and we couldn't track it down. I tried writing to the dxtory devs to look into the issue, but no answer so far.
Were you able to find out the cause?

Greets,
I.

[Rotareneg]

Yes, I was the one who reported that. I can only imagine it's an issue with with Dxtory itself as I've not seen that problem happen when testing MagicYUV or Lagarith with other software.

Here's a 7zip with three videos, one direct from dxtory, one recompressed with msp13, and the final smart rendered from the previous with msp13: https://drive.google.com/open?id=0BzS6i2JCDwdtLWdFT3hBZ19LdzA

[Ignus2]

About Dxtory: Does the issue still persist? Do you still get storage bottleneck? BTW, how did you measure disk queue length and drive active time?

About the videos: Thanks. The immediate difference I see between the Dxtory record and the recompressed video is the global AVI bitmap info header field "biSizeImage".
Hmmm... My guess would be Vegas/MS probably checks this global header of the original video and the newly compressing video, and if it differs in any way, then refuses to smart-render as maybe it thinks the format is not exactly the same.
The funny thing about this value however is that it's not relevant for compressed formats in the GLOBAL AVI header, as each frame has a different size (obviously, each frame is compressed to different sizes, duh). But unfortunately it must have some global value, whatever that is. The codec doesn't care.

The reason it works "sometimes" is that currently I simply pass the raw input value in there. So as Dxtory gives RGB32, this value gets to be: ( width * height * 4 ) in the compressed AVI.
BUT, when an app (like Vegas) begins to smart render, it passes the codec RGB24, so the value gets to be: ( width * height * 3 ) (well, precisely it is round_to_4(width*3)*height, but anyway...).
So now we have a difference in this irrelevant global AVI header field, and my guess is Vegas detects it as difference!
The reason it works in the recompressed case is that the recompressor also passed RGB24 to the codec, so now this irrelevant field ended up being the same that Vegas will produce in the smart rendering case.

So it seems I should set this field to something consistent, so apps like Vegas can smart-render.

Greets,
I.

[Rotareneg]

Yes, the disk usage bug is still present in Dxtory. You can monitor the disk queue length and other similar info in Windows using the resource monitor, accessible through the appropriately named "resource monitor" button in the performance tab of the task manager.

[Ignus2]

I see. I just tried, but I can't reproduce. I have the same value in Resource Monitor "Disk Activity" (the small green block below the process list there) as the bitrate reported by Dxtory, and "Highest Active Time" corresponds accordingly: 7-10% for 32 MB/s.
Do you also have a higher value for "Disk Activity" in MB/s compared to the bitrate reported by Dxtory?

Greets,
I.

[Rotareneg]

Here's two screenshots showing the difference between recording with utvideo:

http://i.imgur.com/4tEtD2a.png

And MagicYUV:

http://i.imgur.com/vNnmqIi.png

When recording utvideo, it was writing about 133 megabytes/sec with a drive activity of about 55%. With MagicYUV it was writing only 111 megabytes/sec, but with a drive activity of 64%.

Now that I've got both those screenshots up next to each other I can see something else odd: when recording utvideo Dxtory had very little read activity, about 700 bytes/sec, but with MagicYUV it was reading around 250000 bytes/sec, a significant difference.

[Ignus2]

Indeed, that Read is abnormal, I can reproduce. It's zero for Ut, but well around 200K for MagicYUV. It also happens for x264vfw and Lagarith.
This very well could be a reason, but I have no idea about the cause yet...

[Ignus2]

OK, I think I tracked it down.
It happens if the codec produces an odd number of bytes as output. If I round up the compressed data to be an even number (pad with a byte when it's odd), the Read is zero.
I'd say this is mildly insane...

Greets,
I.

[Asmodian]

OK, I think I tracked it down.
It happens if the codec produces an odd number of bytes as output. If I round up the compressed data to be an even number (pad with a byte when it's odd), the Read is zero.
I'd say this is mildly insane...

Greets,
I.

I wonder if it is related to sector sizes? If you write less than the sector size to a drive it has to read that data back when writing the next bit of the file because it fills up the partial sector with the new data and can only write complete sectors.

Is there a 4K aligned write buffer in MagicYUV?

[Ignus2]

I wonder if it is related to sector sizes? If you write less than the sector size to a drive it has to read that data back when writing the next bit of the file because it fills up the partial sector with the new data and can only write complete sectors.

Is there a 4K aligned write buffer in MagicYUV?

The destination buffer is given by the application using the codec (which is large enough for the worst case), the codec simply writes there the compressed data and specifies how many useful bytes have been written to that buffer.
And it still wouldn't explain why only odd sizes are affected, and a single byte of padding (to be even) fixes it.

So what I think might happening instead, is that AVI chunks must be padded to a 2 byte boundary, and Dxtory probably has something like:
write(compressed_data, compressed_size)
if compressed_size is odd:
write(0,1) // write a single byte

So I assume it makes a separate write call to pad with that 1 byte, which could cause the read-back you describe.

EDIT: This also explains the 200K read-back, as at 60 FPS you write 60 AVI frame chunks, and if you have all buffers odd that would be 60 * 4k = 240k. For me it fluctuates between 120k-240k, as some buffers end up being even.

Greets,
I.

[Ignus2]

1.2rev1 is out which includes the workaround, and it seems to fix the disk activity issue with Dxtory.

It also has some changes which might help with smart-rendering. It needs testing though, so if someone (or you Rotareneg) can check that out that'd be very helpful.

Big kudos to Rotareneg for pointing in the right direction to nail down the issue!

Greets,
I.

[Rotareneg]

The smart rendering works in Movie Studio 13 Platinum now, with a few catches. If you render an RGB video to YUV420 or 422 it properly re-renders all frames, but if you render to YUV444 it smart renders it and you get a video that's blank but has the file size of a proper video. If you render a YUV420 video with MagicYUV set to compress as-is it also creates a blank video that's still full sized. If you set MagicYUV's "Mode (conversion)" setting to YUV420 it will properly smart render a valid video. Here are the YUV420 files if you want to take a look at 'em: https://drive.google.com/open?id=0BzS6i2JCDwdtNXYwMW5SWWdqb3c

I wouldn't be surprised if these odd behaviors are just MS13 being a bit buggy, in normal use everything's working great now, many thanks! :)

[Ignus2]

Thanks. Looking at it "Untitled_420.avi" and "Untitled_420_convert_to_420.avi" are identical, and they are correct.

However "Untitled_420_compress_as_is.avi" in it's global AVI header indicates that it is compressed RGB, however it contains compressed YUV420 frames. This is why it turns out blank, as the codec now checks this and errors out (instead of crashing, as it did before).
The reason it's RGB in the global header is that MS can only send RGB frames for compression, so the resulting video gets compressed "as-is", as RGB, BUT, being "smart"-rendered, the original YUV420 frames are simply copied, hence you get a discrepancy between the global header (RGB) and the actual frames (YUV420).

Now, MS cannot seem to detect that the compressed output will be different than the input, as it only checks a few parameters, which seem to be:
- frame size (width, height)
- codec fourcc
- bit count

The problem is those values are the same for both compressed RGB and compressed YUV420:
- codec fourcc: MAGY
- bit count: 24

MagicYUV, and as I've seen with other codecs as well, bit count is set to 24 for non-alpha and 32 for alpha compressed formats. I'm reluctant to modify this, as it might break other apps expecting it to be this way.

Having separate fourccs for the different compressed color space/subsampling variations could help further with smart-rendering (which 1.3 will have), but not fully, as even in this case, the various other differences in codec settings (full-range/limited-range YUV, color matrix, etc.) would not be detected (MS cannot reliably check for these), making the resulting compressed file broken.

Conclusion:
So as it stands now, smart-rendering is a slippery terrain, it can work, but you have to know exactly ALL the parameters the input file was encoded with, and absolutely make sure that the new compression settings match those exactly.

Greets,
I.

[Sparktank]

:thanks: Had just enough to buy the Ultimate version (after currency exchange). Thanks!

[Ignus2]

:thanks: Had just enough to buy the Ultimate version (after currency exchange). Thanks!

Yeah, I saw, thank you very much! :thanks:
BTW, you can get over to the MagicYUV forums if you wish, I'd like to know how do you make use of the codec (especially the 10bit+ variants).

Greets,
I.

[De-M-oN]

I never knew that Median is the best method for game capture. Thank you for the much better quickinfo text and its really awesome that you even consider game capture at codec development and giving us even tips for best settings to do it (the median compression tip was very good it works indeed very fast and more smooth than dynamic. Thank you!)

Predict Left is gone as a setting or fully gone? I dont miss it. I'm just interested what happened to this.

maybe you can try to add a mode for maximum compression in the region like Lagarith has? Lagarith has an insane good compression and I think that your codec may be faster than lagarith, especially because lagarith' last update was some years ago and I would assume that it doesnt support AVX instruction and so on.
Sometimes maximum compression can be useful. :) This time I mean in general, not as game capture. There I'll stay at median of course.

[zerowalker]

Lagarith shines on Pixelated data, i guess it maybe is cause it uses soem ZIP algorithm and those have easier time with stuff like that?

Well no real clue but i think i read about it and it was something like that.
So Lagarith is insane on stuff that's pixelated, basically where there is simple colors and in blocks.

In short, low resolution or PointResized to higher.

Other than that i can't say personally i find Lagarith that good,
it's a bit better in compression, but not that much i think,
and it's obviously very much slower.

[De-M-oN]

I dont remember which game I saw the big difference. I didnt use it too long.
But I'll test it then again.

[Ignus2]

I never knew that Median is the best method for game capture. Thank you for the much better quickinfo text and its really awesome that you even consider game capture at codec development and giving us even tips for best settings to do it (the median compression tip was very good it works indeed very fast and more smooth than dynamic. Thank you!)

Thanks!
As quite a lot of the codec users are gamers, it's natural not to ignore them :)


Predict Left is gone as a setting or fully gone? I dont miss it. I'm just interested what happened to this.

Yes, I removed that as an option, as it was giving the false impression of being the "fastest" option, which is not the case by far (in fact could even be slower, and compress terribly). There's a reason for this, but I don't think the details are interesting...


maybe you can try to add a mode for maximum compression in the region like Lagarith has? Lagarith has an insane good compression and I think that your codec may be faster than lagarith, especially because lagarith' last update was some years ago and I would assume that it doesnt support AVX instruction and so on.
Sometimes maximum compression can be useful. :) This time I mean in general, not as game capture. There I'll stay at median of course.
You mean develop new algorithms to compress better :)
Yes, I've got quite a few ideas of what to experiment with, I just wanted to push out 2.0 (before it spiraled out to development/feature hell). At 1.2 I thought I maxed out speed, but I discovered a few tricks to make it better, the 30-50% increase in speed is real, see the video codec benchmark post on the website.
It is a good starting point for better compression I believe.

From time-to-time I read the posts on the german letsplay forum in the MagicYUV thread too (with the help of my very basic German and Google translate :) ), to see what people are struggling with or concerned about. It's sometimes very interesting to see the guesswork, not just there but also on reddit, and other places, how people try to find out what does what and how the codec behaves, etc.
That's one of the main reason I set up the forum, and an easy contact form, so if anyone has a question, it's easy to ask.

Also, I saw you do game recording differently than you did a year ago or so. I read about nvidia locking down fast APIs for frame grabbing, that was noted by the MSI Afterburner dev guy (that's interesting btw, but not surprising sadly), and did I get it correct that you now use a capture card somehow hooked up to the output of you video card?
I'd be very interested in how you do your captures.
I just set up a Workflow section on the forum, if you want you can write there too (or here, as you prefer).

Greets,
I.

[De-M-oN]

I use now a Datapath Vision DVI-DL capture card. This way I dont have to hook into the game, which saves a lot of fps.

The only downside on the card is that it downsamples chroma via nearest neighbor. I did a thread about it here, you should remember. I wanted 60fps at 2560x1600, but in RGB Mode it gets only 51fps. But I have to live with it, because having every 2nd pixel without color is very ugly. I was really surprised that such an expensive card uses no interpolation on chroma downsampling..

You mean develop new algorithms to compress better

Yes. I remember Lagarith having very strong compression. Obviously slower, but if you dont capture video it may be a good option to have the possibility to maximize the compression. I think magicyuv with newest cpu instructionsets supported and so on will be much faster than lagarith even with similar compression.

[dipje]

you mean you get aliasing issues on the chroma channels? I know in game-footage chroma-issues are more easily noticed but still.
You could always apply a little Gaussian blur to the chroma channels to sort it out. Maybe with an edge mask but I don't know how that would look in the end.

If it really is the downsample method of the chroma you have issues with (not the fact that your chroma is subsampled which is more normal of course) that means you have aliasing, in other words 'too sharp chroma at edges'. That's why I think blurring it slightly might be a good way.

[De-M-oN]

@dipje : Here my thread about it:

http://forum.doom9.org/showthread.php?t=172647

nearest neighbor for chroma downscaling is a terrible idea. And shockingly bad surprise with such an expensive card. Even shadowplay doesnt use nearest neighbor lol.

[raffriff42]

@dipje : Here my thread about it:

http://forum.doom9.org/showthread.php?t=172647

nearest neighbor for chroma downscaling is a terrible idea. And shockingly bad surprise with such an expensive card. Even shadowplay doesnt use nearest neighbor lol.I don't know if it's practical, but slightly pre-blurring the chroma with a shader (sweetfx, reshade.me) should get rid of the worst aliasing artifacts.

[kolak]

Is this official or some reverse engineering?

https://www.ffmpeg.org/doxygen/3.1/magicyuv_8c_source.html

[Ignus2]

Seems you missed it: http://forum.doom9.org/showthread.php?p=1769315#post1769315
:)

[shekh]

I noticed one difficult thing with recent FFMPEG:
For example I take M0Y2 avi (422P10) and do "-c:v copy" to another avi.
Here happens magic: FFMPEG knows something and instead of keeping the same tag creates new avi with MAGY tag.
Now this new avi is useless for vfw use: it uses generic decoder and cannot extract 422P10; also impossible to infer that it has alpha (if I start with M8YA).

[wiak]

I noticed one difficult thing with recent FFMPEG:
For example I take M0Y2 avi (422P10) and do "-c:v copy" to another avi.
Here happens magic: FFMPEG knows something and instead of keeping the same tag creates new avi with MAGY tag.
Now this new avi is useless for vfw use: it uses generic decoder and cannot extract 422P10; also impossible to infer that it has alpha (if I start with M8YA).
we're in 2017 now, everybody should use .mkv

avi is like flash, old and outdated

did you try muxing it into mkv and use a mkv vfw demuxer?

[shekh]

Same story with mkv:
Codec ID : V_MS/VFW/FOURCC / MAGY

Sorry, false report for M0Y2, but there is still a problem with M8YA.
It seems there are more tags supported in mov container (including M8YA).
Any idea why MagicYUV has vfw encoder to M8YA but FFMPEG does not include it in riff tag list?

[Ignus2]

Same story with mkv:


Sorry, false report for M0Y2, but there is still a problem with M8YA.
It seems there are more tags supported in mov container (including M8YA).
Any idea why MagicYUV has vfw encoder to M8YA but FFMPEG does not include it in riff tag list?

That's probably because in previous versions, there was only one fourcc for vfw/avi: MAGY, and I believe ffmpeg is not aware of the new ones yet, or not fully (or it's buggy). Mov/QT had more from the beginning. But I'm not familiar with the ffmpeg implementation, so it's just my guess. I don't know how does ffmpeg decide/recognise what is the codec in an avi, and how does it select a tag for output.
So I think maybe you have to specify the tag explicitly now, or find a way/switch of ffmpeg which transfers the original tag.

Greets,
I.

[richardpl]

So you mean there are no more MAGY fourcc used for AVI or what?

I can add other fourcc to riff file very easily, its piece of cake, I just was not sure that such files are really valid.

[Ignus2]

MAGY is the umbrella fcc, but the 2.0 codec doesn't produce it anymore, only decode. (But it's still valid to have any format variant under it.)

I can add other fourcc to riff file very easily, its piece of cake, I just was not sure that such files are really valid.
What made you think they are not valid?

Greets,
I.

[dipje]

Seeing there is some more MagicYUV development happening (I last checked over a year ago) and I'm happily seeing the 10bit+ versions being offered on sale now with a price _and_ a trial. Perfect!

Knowing the sh*tstorm that is 10bit support in 'pro' editing applications decided to trial if I can get the Windows 10 bit versions working in + out from After Effects.

Good news, the 10 bit RGB worked pretty easily after adding lines to MediaCoreQTCodecRulesCC. Noticeable quicker than UTVideo :).
Loading a b64a file /script into VDFilterMod and encoding it to avi with MagicYUV 10bit RGB also worked (after manually setting the decode + encode formats the 64bit rgb+a).
Loading the .avi directly into After Effects doesn't work or doesn't give deep color. Muxing the .avi with ffmpeg (-vcodec copy -acodec copy) gives a .mov file that is loaded in After Effects with 'trillions of colors' no problem. It suddenly reports an alpha channel though, but it seems no issue and you can also interpret the file as 'ignore alpha channel'.

To bad for the remuxing part if I want to go from vapoursynth / avisynth+ / vdfiltermod to magicyuv -> adobe, but so be it. If someone is willing to write a commandline quicktime encoder with deep color support (And y4m input or .vpy input.. or heck, even .avi input but with support for v210 / v410 / b64a) I would be internally grateful :P.


I then tried to get the 10bit 4:2:2 version working in After effects, editing MediaCoreQTCodecRulesCC with the codec M0Y2 with v210 as the delivery-format. The 'trillions of colors' option became available while encoding but it still errors out at / after the first frame. Does someone know a working setup for this or should I just be happy with the deep color RGB(A) versions working fine? :P.

PS: I tried opening a b64a .avi file with Quicktime Player Pro, and then I thought I could do 'file->export' to make a quicktime movie with MagicYUV. I selected the 10bit RGB variant, but I could only select 'millions of colors' or 'millions of colors+' in the quicktime export options. I picked the colors+ option (I think) but the export actually worked and it created a file with the correct filesize (the same as exported through after effects). Can I assume that this is now indeed 10bit or did it get sampled down somewhere at some point? (Or to rephrase my question, does the 10bit RGB MagicYUV codec _reject_ anything less than 10bit as input?)

Is there someone who can fill me in with some missing info here? Or is the Adobe suite 'that weird thing that nobody knows what it exactly does internally' :)

(Little PS I discovered after writing this: If I add the b64a .avi to after effects, it loads and is recognized as b64a, but it seems to be reported as 'millions of colors', so no deep color. Truncating / dithering down to 8bit somewhere in the After Effects pipeline?.
Now, if I add a codec 'b64a' to the MediaCoreQTCodecRulesCC to be delivered as 'b64a' (yes, I tell After Effects to open 'b64a' as 'b64a' which seems redundant :P) it now suddenly opens b64a .avi files as trillions of colors. Together with the AVFS module of Vapoursynth that means I can no 'frameserver' RGB48 .vpy scripts into After Effects. Which means I can encode it to MagicYUV 10bit RGB or the native Cineform encoder built into AE or the native DNX encoder built into AE. Now I can finally go lossless out from AE into vapoursynth back into AE again.. _quickly_ (exr/dpx/utvideo isn't really quick. Cineform / dnxhd isn't really lossless). I've been looking quite a while for this :P).

[Ignus2]

Wow, I'm very happy to see someone taking the time to do all this by himself :)
Yes, the deep color path is tricky to get going. One additional thing in general I discovered is that AE seems to "remember" how a file was opened last time, so for example if you forget to add the QT codec rules, open a 10-bit MagicYUV .mov it will be reported as Millions of Colors (dithered by QT), then add the rules and open again the same file, it's still just Millions. BUT if you rename the file, it opens now as Trillions :) Talking about quirks...

Loading the .avi directly into After Effects doesn't work or doesn't give deep color. Muxing the .avi with ffmpeg (-vcodec copy -acodec copy) gives a .mov file that is loaded in After Effects with 'trillions of colors' no problem. It suddenly reports an alpha channel though, but it seems no issue and you can also interpret the file as 'ignore alpha channel'.

I'm not entirely sure how Adobe decides about alpha, there are multiple ways, and it might be that remuxing the avi doesn't preserve some info correctly. I'll try some debugging with remuxed avi->mov, I did a lot of that before.

I then tried to get the 10bit 4:2:2 version working in After effects, editing MediaCoreQTCodecRulesCC with the codec M0Y2 with v210 as the delivery-format. The 'trillions of colors' option became available while encoding but it still errors out at / after the first frame. Does someone know a working setup for this or should I just be happy with the deep color RGB(A) versions working fine? :P.

:) I would also be very happy if someone could figure this out. When I was trying this, even though I set v210 as you, it just presented some variant of RGB, but never v210. Same for decoding, it never asked or accepted v210.

PS: I tried opening a b64a .avi file with Quicktime Player Pro, and then I thought I could do 'file->export' to make a quicktime movie with MagicYUV. I selected the 10bit RGB variant, but I could only select 'millions of colors' or 'millions of colors+' in the quicktime export options. I picked the colors+ option (I think) but the export actually worked and it created a file with the correct filesize (the same as exported through after effects). Can I assume that this is now indeed 10bit or did it get sampled down somewhere at some point? (Or to rephrase my question, does the 10bit RGB MagicYUV codec _reject_ anything less than 10bit as input?)

I could never make Qucktime Player do Trillions. I'm not even sure it has that option. The deep color MagicYUV variants don't accept 8bit data. The 10bit variant also accepts r210/R10k, but Quicktime doesn't deal with those, so if it went through with encoding then the codec definitely got b64a or b48r.
BUT unfortunately I think it did dithering somewhere. I wouldn't be surprised that the insanity of "b64a input -> dither to 8bit -> convert to b64a for codec" happened inside the QT engine.
The only surefire way to test what happened is to make smooth gradients with known values and check the result, not by eye, but by scope or looking at the numbers.

Is there someone who can fill me in with some missing info here? Or is the Adobe suite 'that weird thing that nobody knows what it exactly does internally' :)

It's the latter :) So consider yourself as a pioneer :)

(Little PS I discovered after writing this: If I add the b64a .avi to after effects, it loads and is recognized as b64a, but it seems to be reported as 'millions of colors', so no deep color. Truncating / dithering down to 8bit somewhere in the After Effects pipeline?.
Now, if I add a codec 'b64a' to the MediaCoreQTCodecRulesCC to be delivered as 'b64a' (yes, I tell After Effects to open 'b64a' as 'b64a' which seems redundant :P) it now suddenly opens b64a .avi files as trillions of colors. Together with the AVFS module of Vapoursynth that means I can no 'frameserver' RGB48 .vpy scripts into After Effects. Which means I can encode it to MagicYUV 10bit RGB or the native Cineform encoder built into AE or the native DNX encoder built into AE. Now I can finally go lossless out from AE into vapoursynth back into AE again.. _quickly_ (exr/dpx/utvideo isn't really quick. Cineform / dnxhd isn't really lossless). I've been looking quite a while for this :P).
Now this is interesting. So you say that opening a b64a AVI file was first Millions, but after adding the rules meant for the QuickTime engine, it opened the AVI as Trillions? I ask because that would mean it opened the AVI with the QuickTime engine then?

I'm happy to hear your progress, I'd be glad if you could describe your workflow more end-to-end and share any other thoughts you might have.

Greets,
I.

[dipje]

Great, typed a little thing on my phone. 'token expired' and now it's gone. Let's try again.

As far as I know, AE uses the Quicktime engine for avi loading by default, but it must be installed. These days, they're moving away from the Quicktime engine (making the .mov writer and more codecs built in) and removing the 'no Quicktime warning' during startup.

Thinking about it, I didn't have Quicktime installed before, but I did install it for testing your codec. So getting 'deepcolor b64a' in AE might've come through my xml edit, or through installing Quicktime.. or both :P. To be honest, it's working now so I don't really care.

Quicktime 'remembering settings' per file is new to me (I have to 'interpret movie as' times and times again on certain files, if only... ). The Quicktime 'engine' is handled in a separate process though, and upon exiting an Adobe application that Quicktime process stub has the habit of staying around, specially if an error occurred. In my findings if you make a change to an ini, xml or registry you need to really check that you end all Adobe-related processes before starting the application up again. Maybe this is the cause of the inconsistent file behavior you encountered.

I can now render a comp in After Effects to a .mov with Magic YUV 10bit RGB (didn't try the even higher bitdepth ones to be honest). That .mov I can open in a Vapoursynth script with the latest ffms2 (2.23.1 or something). It's ffmpeg is recent enough to contain the deepcolor RGB MagicYUV support / codecs. Which means I have the AE data lossless in Vapoursynth (and in a single file with audio still intact, helps a lot sometimes) as 'RGB30' (10bpp RGB, as it should be). I can filter there what I want, and in the end I need to make sure I 'upsample' it to RGB48 (16bpp RGB). That .vpy script I can 'mount' as an b64a avi file with AVFS included in Vapoursynth. It creates a virtual AVI file with raw b64a data. I can load that into After Effects and there encode it to MagicYUV again (or use it directly, but it will do all the filtering real time then, not efficient). After the file is encoding with MagicYUV I can 'unmount' the virtual AVI file and have my Vapoursynth output lossless and fast back into AE.

Before this I would be using DPX or EXR sequences. EXR's piz-compression made it pretty OK filesize-wise.

I also use some Vapoursynth scripting (in case of interlacing QTGMC but even for progressive stuff always a bit of chroma-reconstructing / better chroma-upsizing, noise reduction with KNLMeansCL and debanding with f3kdb) and this way I can pass the data lossless on to AE. I hate using image sequences to maintain data, and it also helps to keep everything in a single file _with audio_ to keep the audio sync during a lot of processing stuff.

In other parts during 'fixing sequences' I start in After Effects (tracking, stabilizing, fixing rolling shutter, applying LUTs, etc..) but I want to do some final filtering / sharpening in Vapoursynth. In this case I use After Effect's 'pre-render comp' function to make a render of a comp. It is meant to be a 'temporary render', to use it as 'a cache' of some sorts. But I use it to render a file-quality MagicYUV quicktime file. I then process that file with Vapoursynth and encode it back to MagicYUV but in my After Effects project I keep the reference to the (new filtered) .mov file. This way I have a composition in AE and I can see all the stuff I did with it in AE, but when scrubbing along the timeline (or embedding the comp into another composition) it uses the MagicYUV .mov file, which has Vapoursynth processing done to it.
For bigger projects this helps to keep track of what I did (in case I need to change something, redo it or reapply it to something else) so it helps in organizing bigger projects. But in the end AE uses the MagicYUV file so scrubbing, editing and doing other stuff goes fast and I can sneak some AVS+ / Vapoursynth processing in there.

This all was already possible with image sequences.. but I don't like those. And it was already all possible through the use of Cineform, DNXHD or Prores. But since they aren't true lossless, I don't always 'trust it' (it's a gut feeling thing, I know). Having something with sharpening and film-grain applied and then throwing it through intermediate files which aren't pure lossless gives me a bad feeling somewhere. Probably unjustified, but still.

UTVideo reached this point before, but 1) He removed Quicktime support (After finally finishing it and making it work in Adobe stuff) because of the whole 'Quicktime player is unsupported by Apple' thing (*sigh*), and 2) it was a lot slower compared to the near-lossless codecs or plain DPX image sequences.

MagicYUV keeps the speed (if not faster), specially when the resolution goes up for certain source files it helps a lot (making me do my processing on 4K files in 4K, and only downsizing it to project-size 1080p at the end of the pipeline instead of at the start pure because of editing speed). It trades filesize for 'pure lossless' basically compared to other near-lossless codecs, while being just as fast if not faster. For the small amount of money you ask for it, it's an easy choice to me :).

edit: PS: I know you had a 10bit codec before, but your website stated 'contact me for...'. And apparently that's enough to let my mind go 'ok nevermind'. While seeing an actual price and a buy button makes me consider it. Weird, but it helps.

[dipje]

bleh, spoke to soon maybe. Both in opening my b64a .avi file in Quicktime Player and exporting it to 10bit RGB Magic YUV and opening the b64a .avi file in After Effects and rendering it to 10bit RGB MagicYUV gives a colorcast. Which is weird since we're talking about RGB everywhere, not a single conversion to YUV or something in sight...

Testing with a 16bit gradient as source. Rendering it in After Effects to 10bit MagicYUV (which gives some truncating to the 10bit depth, and then trying to open that magicyuv .mov file in Vapoursynth and quicktime player.. somewhere there's a colorcast there :(

[Ignus2]

Yeah, token expired sucks, that's why I always copy to clipboard all my reply before hitting preview or submit... just in case :)

AVI is certainly loaded through VFW by default. But maybe in some cases it uses QT then. I'll try to investigate.
Well, maybe this is not fully true either, I've once read AE uses a chain, where plugins get a first chance look (in some order), and if they reject, then comes VFW/QT.

A quick question about the xml rules: what did you use for gammatag and cvbuffertag? I've tried both true/false and 0/2.2, didn't see any difference :/

If you use the ffmpeg decoder, it's not as fast as the official, but I don't know if vapoursynth can use VFW/QT codecs for decoding. Does it? If it could, it would be even faster.

Thanks for the info BTW, it's really helpful!

bleh, spoke to soon maybe. Both in opening my b64a .avi file in Quicktime Player and exporting it to 10bit RGB Magic YUV and opening the b64a .avi file in After Effects and rendering it to 10bit RGB MagicYUV gives a colorcast. Which is weird since we're talking about RGB everywhere, not a single conversion to YUV or something in sight...

Testing with a 16bit gradient as source. Rendering it in After Effects to 10bit MagicYUV (which gives some truncating to the 10bit depth, and then trying to open that magicyuv .mov file in Vapoursynth and quicktime player.. somewhere there's a colorcast there

I believe QT Player cannot do it correctly no matter what, but in the AE case I believe it is when loading the b64a avi.
What I know works for sure is make a gradient, encode in AE to MagicYUV 10-bit RGB .mov, open again in AE, and all should be correct.
So maybe opening b64a avi in AE is not correct.

Greets,
I.

[dipje]

New info. If I load the vapoursynth script through b64a into AE I get perfect 1-1 mapping _if I load MagicYUV through avisource in Vapoursynth_.
If I re-encode that vapoursynth-b64a-avi file in AE to MagicYUV again I still get a perfect 1-1 mapping.

It seems the colorcast I talked about is introduced by my ffms2 loading.. bug in their decoder maybe? I'll look some more.
This means I have to re-mux the MagicYUV MOV files AE made to load them as .avi into Vapoursynth. Then through virtual b64a avi files back into AE. So close to having a pretty perfect in-and-out of Adobe situation going on, grr :).

Hey something else, I'm having trouble playing back the MagicYUV .mov files in MPC-HC. Probably because the .mov extension makes it use LAVDecoder which doesn't seem to have a deepcolor MagicYUV decoder yet (I'll check out the nightlies to be sure). I really have to make it an .avi file to let the VFW decoders kick in, right? sucks :).

I'm actually wishing AE could do deep-color avi export now :(.

[Ignus2]

So vapoursynth can load through VFW, that's great. If you have the codec tray icon enabled (the default), you should see a green [M] icon in the notification area (this is ONLY for VFW, the QT decoder doesn't create tray icon yet), and if you hover over or click it, it shows how it's decoding. Logfiles are also helpful.

MPC should work from .mov too, it shouldn't need to use QT, because actually it should be able to demux the .mov and then use the VFW decoder (wrapped for DirectShow) to actually decode. GraphEdit can be useful here, I know I could make this work before.

Also, you can use VLC, MagicYUV has a plugin for that and it should decode and display 10-bit RGB just fine.

Yes, the re-wrap step would be nice to be avoided somehow. If something can demux the .mov and then use the VFW MagicYUV decoder, it should work. Maybe DirectShowSource?

Greets,
I.

[dipje]

yay fixed it (sorry for spam-posting).

In Vapoursynth I was doing this:
c = core.ffms2.Source(source = r'Comp 1.mov', alpha = False, format = vs.RGB48)

And I think ffms2 uses ffmpeg's libavfilter (or whatever it is these days) to scale/filter it from RGB30 to RGB48. Causing a colorcast.

If I do it like this:
c = core.avisource.AVISource(r'Comp 1.avi')

I get RGB48 directly from MagicYUV's VFW decoder (but requiring remuxing to .avi)

If I do it like this:
c = core.ffms2.Source(source = r'Comp 1.mov')
c = core.fmtc.bitdepth(c, bits = 16)

FFMS2 gives me back RGB30 (which is the unscaled unaltered MagicYUV data) and then I use fmtconv to scale it to 16bpp, making it RGB48. -> No more color cast!!


I dropped this virtual b64a AVI file in After Effects and re-encoded it to MagicYUV again, and also dropped it in Quicktime Player and exported it to MagicYUV.
Both work fine (yes, the Quicktime Player generated file contains the exact same RGB values as the AE generated file in a 16bit project).

Just to make sure I'm not crazy. I start in a _16bit_ AE project and I create a gradient there. The gradient goes from pure black (through gray) to pure white.
If I take a certain area of pixels I get these values of pixels (R,G,B are the same): 1485, 1502, 1519, 1536, 1553, 1570, 1587, 1604, 1621, 1638

If I let AE encode that to 10bit MagicYUV RGB and I take a look at the values in that file I get: 1473, 1473, 1505, 1538, 1538, 1570, 1570, 1602, 1602, 1634
These seem like the correct files when you truncate something from 16bit to 10bit. (Well, if you truncate something 15bit to 10bit. After Effects' 16-bit projects run from 0 to 32767, don't know why).


Then I do my Vapoursynth trick on that MagicYUV mov file and open the resulting virtual b64a in AE: Same values. I re-encode it to MagicYUV 10bit RGB -> same values.
Here comes the kicker, that virtual b64a avi file in Quicktime Player exported to MagicYUV -> same values. Even though Quicktime player says 'millions of colors+', apparently it does use deep color somewhere.

Going to try the same test from start to end again to be sure, and this time use a very subtle gradient (A gradient that goes from RGB 15423 to 17343. In a 1920 wide image that gives me one color pixel per width of the image, in 15bit scale).

[Ignus2]

Thanks, this sounds great!

I was not aware the QT Player could do this, I was certain it couldn't.

By color cast, what do you mean exactly? Can you show some comparison images how it looks wrong and how it looks right?
I'll be interested in the new gradient test too.

A note about Adobe: it would be nice if they supported deep color through VFW, but it's unlikely to happen, they all move to built-in codecs, which is unfortunate. Standard interfaces are slowly disappearing and there is no replacement. Anyway, whining a little at Adobe is still an option, so if you have the time, write to them to add deep color support through VFW, the more people do, maybe they'll take notice.

Geets,
I.

[dipje]

I think I'm getting into the limits of all the scaling between 10bit, 15bit and 16bit that things give slightly off values.

First to be clear: I _am_ getting 10bits of precision out of my magicyuv -> vapoursynth -> virtual b64 avi -> after effects route. I think there's no denying there.
If I look at the color picker in 10bit mode, I'm getting 32 pixels width of the value '510', then 32 pixels width of the value '511', etc.. If the color values go in single step in a 10bit color picker, I'm getting 10bits of precision.

But you gotta be real careful if programs like to do colorspace conversion or something, because that can easily put the values off by 1 or something like that. I get a 1-on-1 mapping now.
I'm also guessing that I'm now basically doing 15bit -> 10bit -> 16bit -> 15bit. (After Effects starts at 15bit, I encode to a 10bit MagicYUV file, I open that and scale it back to 16bit for b64a, then After Effects opens that and uses 15bit again) so I wouldn't be surprised if some rounding errors snug in somewhere, but for now it's OK apparently. The workflow does seems to be picky for the small things though :).

(Example: There is a slight difference in the values (still 10 bit precision, the values are off by 1) in the b64a virtual avi file directly in After Effects. But if I encode that file back to 10bit RGB I get my original 10bit values again. So the upscaling seems to give some rounding errors, but if you go down to 10bit again (I'm testing a 10bit codec let's not forget) you get a perfect 1-on-1 again.

Also, After Effects doesn't seem to dither _at all_ when going from 16bit / 15bit to 10bit. I was led to believe it did, but I think it only dithers when going from 'deepcolor' (10bit + ) to 8bit... and the dithering is ugly anyway.

Back to your comments:
Vapoursynth uses the VFW decoder if you use 'avisource' (which requires an avi file) and it always gives b64a apparently. If I 'mount' a Vapoursynth script with AVFS, the tray icon only shows that 'AVFS' is using MagicYUV and the VFW decoder is delivering b64a to the AVFS app. Which is no surprise. The big question is / was if opening that virtual b64a avi file if programs like Quicktime and others don't go sneakily truncating down to 8bit (with dither) and then somewhere scaling it back up again. That the tray-icon says AVFS is receiving b64a doesn't mean much, the question is what is After Effects receiving / using :). But as stated before, as long as you're careful and make sure ffmpeg itself doesn't start scaling stuff you'll be OK.

My MPC-HC is 64bit, is it possible that the 'VFW decoder inside DirectShow' is not available in 64bit? Looking through the config I see no other problem to be honest.
And yes, I've noted the VLC plugin during install but let's just keep it at "I am not a VLC user" :).

I can drag and drop the .mov files onto Virtualdub FilterMod to preview anyway (another reason why I'd rather have a single large file instead of a folder with an image sequence.

[Ignus2]

Googling for "after effects 16 bit 32768" gives results :)
https://forums.adobe.com/thread/583452
So yes, 16bit for Adobe means max 32768. It's technically using the 16th bit (15bit would be 32767 max), so it's "correct"... in a way :)

About MPC: they very well can use both 32/64bit VFW decoder (both can be wrapped in DShow), they just cannot render the result. MagicYUV supports the following output formats through VFW for 10-bit:
RGB 10-bit: r210,R10k,G3[0][10],GBAL
RGB 12-bit: G3[0][12]
RGB 14-bit: G3[0][14]
RGB 16-bit: b48r,b64a,BGR[48],BRA[64],G3[0][16],GBFL
They should take their pick :)

Thanks for the info again.

About image sequences: you said you tried DPX. I once made a prototype filesystem-based on-the-fly DPX compressor/decompressor (though Linux only), what would be your opinion if something like that existed?

Greets,
I.

[kolak]

I think I'm getting into the limits of all the scaling between 10bit, 15bit and 16bit that things give slightly off values.

First to be clear: I _am_ getting 10bits of precision out of my magicyuv -> vapoursynth -> virtual b64 avi -> after effects route. I think there's no denying there.
If I look at the color picker in 10bit mode, I'm getting 32 pixels width of the value '510', then 32 pixels width of the value '511', etc.. If the color values go in single step in a 10bit color picker, I'm getting 10bits of precision.

But you gotta be real careful if programs like to do colorspace conversion or something, because that can easily put the values off by 1 or something like that. I get a 1-on-1 mapping now.
I'm also guessing that I'm now basically doing 15bit -> 10bit -> 16bit -> 15bit. (After Effects starts at 15bit, I encode to a 10bit MagicYUV file, I open that and scale it back to 16bit for b64a, then After Effects opens that and uses 15bit again) so I wouldn't be surprised if some rounding errors snug in somewhere, but for now it's OK apparently. The workflow does seems to be picky for the small things though :).

(Example: There is a slight difference in the values (still 10 bit precision, the values are off by 1) in the b64a virtual avi file directly in After Effects. But if I encode that file back to 10bit RGB I get my original 10bit values again. So the upscaling seems to give some rounding errors, but if you go down to 10bit again (I'm testing a 10bit codec let's not forget) you get a perfect 1-on-1 again.

Also, After Effects doesn't seem to dither _at all_ when going from 16bit / 15bit to 10bit. I was led to believe it did, but I think it only dithers when going from 'deepcolor' (10bit + ) to 8bit... and the dithering is ugly anyway.

Back to your comments:
Vapoursynth uses the VFW decoder if you use 'avisource' (which requires an avi file) and it always gives b64a apparently. If I 'mount' a Vapoursynth script with AVFS, the tray icon only shows that 'AVFS' is using MagicYUV and the VFW decoder is delivering b64a to the AVFS app. Which is no surprise. The big question is / was if opening that virtual b64a avi file if programs like Quicktime and others don't go sneakily truncating down to 8bit (with dither) and then somewhere scaling it back up again. That the tray-icon says AVFS is receiving b64a doesn't mean much, the question is what is After Effects receiving / using :). But as stated before, as long as you're careful and make sure ffmpeg itself doesn't start scaling stuff you'll be OK.

My MPC-HC is 64bit, is it possible that the 'VFW decoder inside DirectShow' is not available in 64bit? Looking through the config I see no other problem to be honest.
And yes, I've noted the VLC plugin during install but let's just keep it at "I am not a VLC user" :).

I can drag and drop the .mov files onto Virtualdub FilterMod to preview anyway (another reason why I'd rather have a single large file instead of a folder with an image sequence.

If you want to work with AE than you rather want MOV container, as AVI is not robust when it comes to 8bit+. MOV files should work well over b64a in AE.
This is why we need to make virtual file system which uses not AVI, but either MOV or e.g. image sequence: DPX or EXR.

You can do this. Load your virtual AV file to QT7 PRO, save as QT ref and use this as a source in AE. This is double virtualisation and it did work with e.g. v210 AVI files (regardless of AE supporting AVI v210 natively, which is a special case in AE) when I tried. Should work with b64a also. This is bit overkill workaround thought :)

AE is not that bad in reading at high bit depths, but as long as you use MOV.
Vapoursynth can read some 8bit+ pixel formats from avisource which was kindly implemented by Myrsloik as well as v210 virtual mount (AE won't read things like P210). v210 has best support when it comes to 10bit YUV. RGB is "easier"- all can be presented as b64a if needed and it's also quite widely supported.

[dipje]

Load your virtual AV file to QT7 PRO, save as QT ref and use this as a source in AE.

Tried that already before, doesn't work. Reference .mov files don't open in AE.

Since a lot of the 'free / opensource / doom9'-related video projects which work with deepcolor started using b64a, b64a avi is fine for me.

Virtual mov I'm not opposed to, but I guess it's a lot harder to do then AVI, since the mov file format is way more complex. And to be honest, for me with b64a avi files I don't see the need for the time being.

AVI would be fine for most intermediate stuff if people just read up on the deep color formats in avi :). Everyone (like, big application coders like Adobe) just assume DirectShow and VFW are 8bpp or less only, just not implementing anything else.

So when a cool codec comes along like MagicYUV, UTVideo, Cineform AVI, FFV1, etc.. you can't do sh*t with it because applications just don't _think_ about feeding it with b64a / y210 / y410 or any of the other 'lesser known pixel formats'. It's frustrating, above all.
The pro world embracing MOV and the free/opensource world embracing MKV and both leaving AVI behind (kinda understandable) that's just asking for 'not compatible'-listings :(.

Aanyway.. @Ignus2: Virtual DPX filesystem. I'm already very interested :). Can you elaborate on how it would work? (I'm a linux 'pro', just blurt out how you got it :P).

You would mount an existing video file or target a new file, and it would create a virtual folder. If random applications write files in those folders in the lines of %05d.dpx or something you take that raw DPX data, encode it MagicYUV (or whatever) and change the corresponding frame number in the videofile? And the other way around, mounting an existing file creates a virtual folder filled with DPX files which represent the frames in the video file?

Especially writing support, makes it sound awesome. In the Windows world I guess it would be 'easy' with the Pismo mounting library (same stuff AVFS uses, also possible to mount .zip files as folders, mont .iso files, encrypted folders, etc... Closest thing to a userspace filesystem driver like fuse but in the Windows world).

For example, staying in the MagicYUV world: Creating a file would require setting some MagicYUV settings (at least the bitdepth required :P). Maybe I'm already thinking a few steps ahead but I would love support for colorspace conversion, YUV conversion and dithering in it (include zimg for example? Don't know if that's possible license wise).
Because DPX is RGB only, so to make the MagicYUV 422 / 444 codecs a possibility you would need to convert it yourself somewhere. Also would present a fix for ugly dithering in a lot of 'pro' applications. Let Adobe render 16-bit DPX files, select 10-bit 422 MagicYUV in your output, specifying dither algo, resizing kernel for the chroma and the colormatrix. Takes the YUV conversion + dithering-down away from 'pro' apps and leaves it up to you (or zimg in my example) which is always a big win :P.

DPX is also not a format as clear cut as an uncompressed BMP :). It supports (officially) random value ranges to allow for different gamma's, log stuff, having some sort of 'overbrights' and 'underblacks' support... just stuff to think about, I'd be more than satisfied with 'regular' DPX files which just use full range from 0-to-max, with 8bpp, 10bpp, 16bpp. Done. :). I never used the other possibilities of DPX, but I know there's a lot in it (cineon stuff comes to mind, timecodes, ...).


Only 'pro' software I'm used to is the Adobe suite + Davinci Resolve. I know ffmpeg reads/writes them, Imagemagick supports it (and the imagereader/imagewriter in Vapoursynth) and Mocha (a plane tracker as standalone and plugin version). That's already a very wide palette of support you have then, in both 'pro' world and 'free/opensource' world.


I'm all for it! (Can I help? :P). Just thinking about it, it would solve a lot of frameserving problems :).

[kolak]

If you use mainly open source tools then you are fine.
Going to pro tools over AVI won't work. MOV is as you said way more complicated, so would need a lot of development time. I think CODEX has such a solution.
Other way is to have native support for the codec, but this is veeery difficult. Adobe maybe easiest, but for many other pro tools will rathe never happen.

[dipje]

Going to pro tools over AVI won't work Everybody his thing. Works fine here and this way I can combine the worlds. Unless some virtual .mov or virtual .dpx system comes along, this will do just fine (finally!!).

[LexSfX]

MagicYUV in DxTory worked amazingly for recording lossless RGB 1920×1200 @ 60 fps with DxTory. However, I can't figure out how to use it with OBS-Studio. It doesn't seem to be in OBS' ffmpeg video codecs dropdown, and UtVideo can't handle it (makes the game lag immensely). I'd like to record with other sources overlaid, which is what makes OBS so nice.

I used version 1.2rev2 because it's free. I can't justify buying it if it can't work with OBS.

[De-M-oN]

OBS needs vfw support. Suggest it them. They need to implement vfw.

[LexSfX]

I did suggest it in #obs-dev. Their response is basically that an encoder for it should be added to ffmpeg. Here's the entire exchange, quoted directly from IRC.

22:39:21 <LexSfX> MagicYUV codec (using version 1.2rev2) allows me to realtime encode lossless RGB 1920×1200@60Hz while playing a game on the same CPU with >10% CPU to spare, and UtVideo can't manage it (can't even manage 30Hz). is MagicYUV on the wishlist for OBS somehow? like maybe through DirectShow oslt? or is there a DirectShow plugin? i guess i should google that first
22:41:03 <c3r1c3_Linux> LexSfX: The real question is can you get it into ffmpeg. if so, you're set.
22:41:26 <LexSfX> yes, but that seems unlikely, considering MagicYUV has gone from freeware to pay-what-you-want to payware
22:41:41 <LexSfX> always proprietary
22:41:52 <LexSfX> unfortunately :(
22:45:15 <LexSfX> here's the youtube transcode of the video content i'm talking about (original uploaded to youtube is lossless RGB), which looks to have codec poisonous content, which may be why UtVideo can't keep up: https://www.youtube.com/watch?v=J_RRsdW2NSY

[De-M-oN]

AVI is fastest on VfW. You see how slow ffmpeg method is.

They need to implement vfw. Simple. I dont get why they still didnt work on it...

[LexSfX]

I've posted on the OBS project forums now.

https://obsproject.com/forum/threads/magicyuv-vfw-and-lossless-recording-performance.62524/

By the way, what's slow about the "ffmpeg method"? I thought it was just that the codecs I was trying (UT Video, libx264rgb, Lagarith) were slow.

[osgZach]

Given the nature of how OBS is built, it's not unrealistic for them to say it needs to be added to ffmpeg. Their entire software is centered around ffmpeg and always has been. That's the encoder they use. They don't independently support codecs, your options are either ffmpeg, hardware NVENC, and that's about it (not sure about AMD).

I don't think they're being unrealistic about it.

[LexSfX]

I agree with you. It would be best to be available via ffmpeg. The vast majority of video software for all platforms supports ffmpeg.

[osgZach]

I'm actually intrigued by this codec, however. I do a lot of work with intermediate files before final encoding, and also it would be interesting to see how it performs when recording games and what compression efficiency is like. Although I use a capture card so I'm not restricted to OBS or DXTory.
I typically avoid lossless codecs for recording due to the write bandwidth required and file size though.

[richardpl]

I could write ffmpeg magicyuv encoder in few minutes, but I would feel bad about it considering how its current implementation is payware. Like stealing food from someone's mouth.

[LexSfX]

If that's the case, why not write a different, FOSS codec with a similar level of efficiency? You'd have to write the decoder too in that case, I guess.

[Ignus2]

How about adding encoding through vfw to ffmpeg then? I'm sure I've found a discussion about it once, and quite understandably, it was quickly rejected.
Though I think it would be possible to do technically...

Greets,
I.

[LexSfX]

That would be cool. FFmpeg would become even more useful. Is this the discussion you found? https://ffmpeg.org/pipermail/ffmpeg-devel/2008-December/044516.html

If so, it looks like michaelni is interested if someone else does it.

[shekh]

How would you setup options for vfw codec wrapped in ffmpeg?

[LexSfX]

You'd have set them up before using it. The codec settings are saved on the system.

[osgZach]

I could write ffmpeg magicyuv encoder in few minutes, but I would feel bad about it considering how its current implementation is payware. Like stealing food from someone's mouth.

Could it not be setup so its required to have the DLL or something? Kind of like some programs don't distribute certain binaries.

I mean there IS a demo version, too. So its not like simply having that DLL would take away from sales. The demos are watermarked, etc.

Personally I am probably going to buy a license when I get my next paycheck (I have $10 in my bank account lmao) because from what I have tested its pretty nice.

I mean holy crap the editing speed /scrubbing even when working with huge AVI files is amazing. It renders 1080p60 like a dream

[shekh]

You'd have set them up before using it. The codec settings are saved on the system.

It means depending on random garbage. Settings on system can change any time. (afaik MagicYUV does not maintain them anyway)

[LexSfX]

Obviously you would set the settings before you encode. Why do something technical without thinking about what you're doing? Also, "random garbage"? It would be the same settings you set last time. Who else would change your MagicYUV settings but you?

[Ignus2]

That would be cool. FFmpeg would become even more useful. Is this the discussion you found? https://ffmpeg.org/pipermail/ffmpeg-devel/2008-December/044516.html

If so, it looks like michaelni is interested if someone else does it.
Yes, that was the one. I wouldn't say he's particularly interested, but rather if someone takes the effort to do it and maintain it, he wouldn't object to it :)

How would you setup options for vfw codec wrapped in ffmpeg?
I also think that would be tricky to do, if not impossible. Possible options could be: provide no settings and assume the codec must be set up globally, or provide random binary garbage blobs by user. Not nice. I think global setup could work initially (no runtime setup), and think about a solution later.

Greets,
I.

[LexSfX]

Huh? FFmpeg would just load the usual settings, wouldn't it? I mean the settings configured here: http://lex.clansfx.co.uk/dump/749597a9d965de5c3e194ed7203a9fcc/000006FA.png

[Ignus2]

Huh? FFmpeg would just load the usual settings, wouldn't it? I mean the settings configured here: http://lex.clansfx.co.uk/dump/749597a9d965de5c3e194ed7203a9fcc/000006FA.png

Those settings are stored differently for every codec. MagicYUV stores it in the registry, but there is also a so-called "per-app config" or "client config" or "runtime config". Whatever it's called, the way it works is that the app using the codec queries the codec about it's settings (after asking it to show it's config dialog) through a VFW call, and the codec returns it's settings as a binary blob/data, transparent to the app. So the app has no idea what it means, just saves it, and the next time it uses the codec, simply hands it back to the codec to read it.
This is what shekh referred to as random garbage (I assume).

EDIT: And what I said was basically to skip the above, and let the codec use it's default/global settings.

Greets,
I.

[shekh]

To explain "random garbage" idea:

Imagine you have setup a background job to convert 1000 files.
In the middle you forget about it and run foreground capture, and change encoding options for it.
Now the background job will do something wrong (because the settings do not belong to it).

[osgZach]

Seems to me the solution is simple then, just require the flags to be specified by the user, or it will use a default preset, like every other ffmpeg codec does.

[LexSfX]

Besides, why forget what you are doing? Encoding is a conscious decision that requires lucid thought. Changing global options that are currently being read repeatedly during a multi-file encoding job seems like a bad idea that nobody who has the mental capacity to configure a multi-file encoding job in the first place would execute. You're right that it's a bad design decision to rely on global settings, but it's a good halfway point if a parameter like "-vfwopts" similar to "-x264opts" ffmpeg already exposes is too much design and implementation work up front for any one volunteer.

FFmpeg has never protected its users against potential disasters. It goes for the "KISS" approach, aka "keep it simple, silly!"

[Ignus2]

In any case, if we ignore the settings stuff for now, we would just need to get someone actually try to do basic VFW encoding through ffmpeg to see how/if it works.

Greets,
I.

[osgZach]

In any case, if we ignore the settings stuff for now, we would just need to get someone actually try to do basic VFW encoding through ffmpeg to see how/if it works.

Greets,
I.

I could write ffmpeg magicyuv encoder in few minutes, but I would feel bad about it considering how its current implementation is payware. Like stealing food from someone's mouth.


Assuming you meant doing the above, richard, how much would this cost.

[dipje]

It means depending on random garbage. Settings on system can change any time. (afaik MagicYUV does not maintain them anyway)


As far as I know, you can set up a structure with VfW settings (present a dialog to the user with codecs and a config button for the codec). That selection is put into a memory object / structure and returned to the programmer. You can dump this information to disk.

Then, while using this 'ffmpeg vfw interface', you can give the settings file in the command line options and it would use that.

This way you can present a GUI to the user for selecting VfW stuff, but still use those settings in a batch job. Also, it would prevent the issue that the settings are 'just whatever the state of the system is in that moment', because I do agree that isn't the proper way to go.

(Makes debugging and helping users a b*tch if you just can't guarantee stuff isn't changed by some other VfW program somewhere on the system). Also, having those settings in a file makes it easier to debug or pass settings along, since you can share the file (so that way the codec developer can see exactly what your settings were)

[shekh]

VirtualDub is keeping these settings as base64 encoded string. Example for MagicYUV "ygAAAAQAAAABAAAABAAAAAIAAAD/////AAAAAAIAAAAAAAAAAQAAAAAAAAA="
I actually have various pieces of very similar problems, but not desire to make a universal wrapper for ffmpeg.
The amount of wrapping required for simple (intra) codec is very small.

[dipje]

since we're talking about enabling magicyuv in ffmpeg workflow (while keeping the encoder closed source) I think simple-intra-only would suffice for people in this thread.
Supporting the deep-color versions is more important otherwise there would be little use I guess.

I'm just hoping for that virtual-folder-compressed-DPX-on-the-fly from Ignus2 :P

[LexSfX]

If the source was open, so much more could be done with MagicYUV, and it could be really respected. As a closed-source proprietary project, it doesn't improve the standards of video encoding. It's very sad that it's closed-source, considering the performance level I've seen with it. Why not open the source (at least for older revisions) and start a Patreon instead?

[richardpl]

But that contradicts current payware policy.

[LexSfX]

Obviously, hence the phrasing of "Why not[...]?".

[WorBry]

@Ignus2

But that contradicts current payware policy.

So let me get this right. Just over a year ago I made a donation when updating to v1.2 to 'support development':

https://forum.doom9.org/showthread.php?p=1748491#post1748491

And now I'd have to pay a fee just to update to v2.0 (standard) ?

https://www.magicyuv.com/

I would have thought that those who made a donation before this pricing was introduced at least would be honoured with a free update to v2.0.

[Ignus2]

@Ignus2
I would have thought that those who made a donation before this pricing was introduced at least would be honoured with a free update to v2.0.
They were. Everyone who made a donation, even 0.99c, got a free upgrade to 2.0. They all received an email in December about it. If you donated, that includes you. Maybe check your spam folder, or tell me the email address you used for the donation and I can resend the email.

Greets,
I.

[WorBry]

Yes, I've found it there in the spam folder. My apologies :o

[hajj_3]

UT 17.4.0 is out now, you should update the benchmarks on your site comparing MagicYUV to UT v17.4.0.

[mcjordan]

Dear Ignus, I've a donation for your Magic ;-) from december, 16 (my e-mail: mcjordan@abv.bg) How to get update to version 2.0 RC2?
P.S. Your automail sender never give me a password or password form for forum registration (may be some wrong with my mail provider?)

[Ignus2]

Dear Ignus, I've a donation for your Magic ;-) from december, 16 (my e-mail: mcjordan@abv.bg) How to get update to version 2.0 RC2?
P.S. Your automail sender never give me a password or password form for forum registration (may be some wrong with my mail provider?)

Hmmm, that is strange. Anyway, I saw your registration, will write an email to you shortly.

Greets,
I.

[mcjordan]

Thank you very much Ignus, I received your email.

[dipje]

Came around to testing 2.0rc2, but I had to revert back to 2.0rc1.

Files saved through Virtualdub VDFilterMod to avi (with 4:2:2 10bit and RGB 10bit, so M0Y2 and M0RG) opened fine when I loaded them back into VDFiltermod, but no matter what I tried I couldn't get them to work in After Effects. It would jus always show a black frame. When trying to play the files in Quicktime Player they would work sometimes, but they would also crash Quicktime a lot.
Recent VDFilterMod versions have an option to save directly to .mov, that way I can skip the 'remux avi to mov with ffmpeg' step. But the .mov files saved by VDFilterMod would or show black frame, _or_ trigger After Effects in saying the .mov was corrupt.

Closing any Quicktime and Adobe processes, reverting back to 2.01rc1 and suddenly the M0RG codec works just fine again in After Effects. I can load a RGB48 .vpy file in VDFilterMod, select XRGB64 as decode format and as format delivered to the compressor, select MayivYUV 10 bit RGB, and save to .mov. That .mov opens fine in After Effects as 'trillions of colours'.

Also, _encoding_ from After Effects failed with the rc2 version, but works fine with rc1.

Of course, the YUV variant (M0Y2) is not behaving well in After Effects, but that has always been the case (After Effects and YUV just don't mix, kind of understandable).

[Ignus2]

Came around to testing 2.0rc2, but I had to revert back to 2.0rc1.

Files saved through Virtualdub VDFilterMod to avi (with 4:2:2 10bit and RGB 10bit, so M0Y2 and M0RG) opened fine when I loaded them back into VDFiltermod, but no matter what I tried I couldn't get them to work in After Effects. It would jus always show a black frame. When trying to play the files in Quicktime Player they would work sometimes, but they would also crash Quicktime a lot.
Recent VDFilterMod versions have an option to save directly to .mov, that way I can skip the 'remux avi to mov with ffmpeg' step. But the .mov files saved by VDFilterMod would or show black frame, _or_ trigger After Effects in saying the .mov was corrupt.

Closing any Quicktime and Adobe processes, reverting back to 2.01rc1 and suddenly the M0RG codec works just fine again in After Effects. I can load a RGB48 .vpy file in VDFilterMod, select XRGB64 as decode format and as format delivered to the compressor, select MayivYUV 10 bit RGB, and save to .mov. That .mov opens fine in After Effects as 'trillions of colours'.

Also, _encoding_ from After Effects failed with the rc2 version, but works fine with rc1.

Of course, the YUV variant (M0Y2) is not behaving well in After Effects, but that has always been the case (After Effects and YUV just don't mix, kind of understandable).

This is odd, but thanks for the report! I'll try to check it out, but there really shouldn't be such issues, there isn't any fundamental change since 2.0rc1, except being more robust when decoding corrupt data.
I believe you do have the AE QTCodecRules XML modified, correct?

Greets,
I.

[dipje]

Yes. But in a working setup with .rc1 so shouldn't be any difference.

I'll try on my other PC just to be sure. Any tips or pointers to help debugging or figuring out what's wrong ?

Maybe something changed in AE, not impossible of course.

[dipje]

Hmm.. maybe, 'nevermind'. I don't really know.

I can load a b64a .avi file into After Effects (neatly detected as trillions of colours+), and from AE save it to MagicYUV 10bit RGB in .mov. File loads again just as fine (and yes, still detected as trillions...).
If I then install rc2 this still works, so with that part of the quicktime codec -> nothing wrong.


But if I open that b64a .avi file into VDFilterMod and from there encode an .mov with MagicYUV 10bit RGB, that file won't open correctly in AE. Sometimes it shows just black frames, sometimes it gives errors parsing / reading the .mov. Since the .mov writing of VDFilterMod is somewhat new (and probably 'hacky'), this is nothing to worry about.

But if I save an .avi with MagicYUV 10bit RGB and then remux that .avi into .mov with ffmpeg, that file also won't load anymore and will give the same black frames. I'm pretty sure this worked before.
But to be honest, now that I'm thinking about, that is an .avi file created with newer builds of VDFilterMod and it's remuxed now with newer builds of ffmpeg, so god knows what changed in those that might break it.

Loading (virtual) b64a AVI into AE and encoding and decoding from there still works, and I guess that's the most important for me.


But mounting a simple Vapoursynth script (that reads a file and outputs it as b64a / RGB48) as a virtual AVI file and then encoding it from within AE takes 2 minutes. That same Vapoursynth script encoded to MagicYUV through VDFilterMod (and I'm guessing the VfW codecs) is done in just under 30 seconds. So quite a speed difference. But I guess I have to deal with that :).

[shekh]

That is sad. Writing mov function is basically ffmpeg remuxing plugged in.
As I understood: ffmpeg.exe cannot remux avi to mov correctly when codec is rc2? Is the avi in turn correct?
Any idea for simpler way to detect corrupt avi/mov other than import it to AE? :)

[kolak]

I don't think AE will work well with AVI for codecs with require decoding to something different than RGB 8bit.
Use MOV with edited xml file rules.

[dipje]

@kolak
You keep saying that, and I keep saying that
a) that's impossible, there is no 'virtual MOV' or anything to frameserve something into deep-colour mov, and
b) it's working fine with b64a AVI files

@sekh
remuxing with ffmpeg seemed to work before, and since your recent builds basically use the same stuff and the same ffmpeg (libavcodec/libavformat) as base it's understandable they create the same output :).

I don't think this has _anything_ to do with .rc1 or .rc2 of MagicYUV. I think ffmpeg changed it's .mov wrappiong or something that causes this to stop working.
Like I said, the .avi files created by VDFilterMod with .rc2 seem to work fine when opening them in ffmpeg or back into VDFilterMod (or into Vapoursynth with their AVISource)

Only .movs that AE seems to open are the ones created by the QuicktimePlayer or by AE itself :).

Just as a test, the prores_ks codec from ffmpeg wrapped into .mov seems to open just fine like it always did, and I even think your prores encoder UI in the newer VDFilterMod builds work fine.

Something changed with the whole 'remuxing VfW codecs into .mov' in ffmpeg it seems. Not sure though.

[Ignus2]

Actually, if you encode to MOV with rc1 and rc2 (using the same version of VDFM) you should get exactly the same output, could you check if this is the case?

Regarding AE reading MOVs: I had some interesting results with MOVs saved by BlackMagic Fusion and imported into AE. Even with modified XML, the MOVs saved by Fusion were not always interpreted correctly by AE. ALSO: AE seems to remember something about files it opened once, so if you save a file with the same filename/path that AE once opened before, then it'll most likely mess up loading it again if something changed in it. This was very weird. Please read through this thread: https://www.magicyuv.com/community/help-support-troubleshooting/magicyuv-not-encoding/#post-38

Also, AE might put some extra metadata into MOVs. But one thing is for sure: for AE to load MagicYUV 10bit+ RGB correctly as "Trillions of colors" you must modify the XML codec rules, and rename the file if it was once opened by AE!

Greets,
I.

[dipje]

Like I said, in and out from AE work fine. Yes, it required modifying the XML file but that is somehow expected.

I can recheck the rc1/rc2 output again, but when I did an encode with each from AE I can confirm they are exactly the same size to the byte. Didn't actually compare the contents byte for byte :).

I think I messed up when I tried rc2 for the first time that I had 'millions of colors' selected when exporting from AE, and then MagicYUV borks because it receives 8bit input for an 10bit codec. Retrying with 'trillions' and rc2 exports fine.
Files exported that way open up fine in recent ffmpeg builds, and also ffms2 / ffms2000 in vapoursynth.

The only thing that doesn't work anymore is remuxing MagicYUV avi to MagicYUV mov and I think that has nothing to do with rc1/rc2 but with changes in the last months or so to ffmpeg.

edit: and about the 'AE remembers something about the file opened' I say the same as a few months before: If you want to make sure the file is properly 'reread', go to the 'edit' menu where you have the option to 'purge' the ram and disk media cache. If you made changes to the Quicktime XML file, make sure that the Qt32 stub-process is killed (and restarted) to the file is actually reread. Working like this, I _never_ had the experience that 'settings' are saved for a filename. Still a pain to have to remember though :).

AE / Premiere are the only 'pro' tools I have experience with (well, and Davinci but never tried to export anything different than DPX-sequence or DNXHR from Davinci Resolve) so can't talk about fusion <=> AE.
Converting a b64a AVI through the Quicktime Player to MagicYUV 10bit / 12bit RGB both work fine though.

[hajj_3]

MagicYUV 2.0.0 is out now: https://www.magicyuv.com/change-log/

[Stereodude]

Any chance for YUV420P10 support? There's no option for 10-bit YUV 4:2:0 with MagicYUV 2.0. There is 10-bit VUY 4:2:2, but that option won't accept 4:2:0.

[Ignus2]

Any chance for YUV420P10 support? There's no option for 10-bit YUV 4:2:0 with MagicYUV 2.0. There is 10-bit VUY 4:2:2, but that option won't accept 4:2:0.

Actually, it's been done for a while in the betas (not released yet), I'll make a release shortly.

I.

[preludium975]

How can i use DaVinci Resolve with MagicYUV? I need compression, the lossless uncompressed video is very very big.

[WorBry]

You can't, nor UTVideo. DaVinci Resolve doesn't support third-party VFW codecs per se. The only system-supported, 'near/visually lossless' AVI codecs are Cineform and Grass Valley HQ/HQX. Otherwise the same in Quicktime (MOV) format, along with DNxHD, DNxHR and ProRes. DNxHD and DNxHR in MXF format also. Take your pick.

[preludium975]

You can't, nor UTVideo. DaVinci Resolve doesn't support third-party VFW codecs per se. The only system-supported, 'near/visually lossless' AVI codecs are Cineform and Grass Valley HQ/HQX. Otherwise the same in Quicktime (MOV) format, along with DNxHD, DNxHR and ProRes. DNxHD and DNxHR in MXF format also. Take your pick.

What is the best encoder in Resolve for lossless encoding? The size is the point.

[WorBry]

Thought I'd been clear. The only option for 'lossless encoding', whether transcoding for input (using Resolve's own Transcode function) or rendered output, is uncompressed. What is your video source material, out of interest ?

BTW, ProRes encoding is only available in the Mac version, but the Windows version will accept ProRes input.

Edit: I see the Studio (purchased license) version of Resolve 15 now supports Kakadu-based lossless JPEG 2000 encoding.

[foxyshadis]

If you're in production, you should give up the idea of a lossless intermediate at all stages. The storage and throughput requirements will absolutely kill you, even if you did use software that gave you the flexibility. Grass Valley (Canopus) HQX Superfine is an amazingly good codec that not only holds up to repeated re-encoding, but to extensive transformation between re-encodings. Cineform is great too, but HQX Superfine gives you double the bitrate for the perfectionist in you. The occasional off-by-one, especially in 10-bit, really isn't worth 10x the storage space.

What you really need to worry about is your HDR format and coverage. That's what's going to get you.

[Cary Knoop]

What the best encoder in Resolve for lossless encoding? The size is the point.
Apart from uncompressed, there is only one, the Kakadu JPEG 2000 encoder.

[kolak]

If you're in production, you should give up the idea of a lossless intermediate at all stages. The storage and throughput requirements will absolutely kill you, even if you did use software that gave you the flexibility. Grass Valley (Canopus) HQX Superfine is an amazingly good codec that not only holds up to repeated re-encoding, but to extensive transformation between re-encodings. Cineform is great too, but HQX Superfine gives you double the bitrate for the perfectionist in you. The occasional off-by-one, especially in 10-bit, really isn't worth 10x the storage space.

What you really need to worry about is your HDR format and coverage. That's what's going to get you.

HQX at best setting is below Cineform FS2.

Cineform's "issue" is that sometimes pre-compressed sources are getting encoded with fairly low bitrate, even for best setting. Good part- it keeps PSNR very steady, so all frames have relatively same quality.

These ares results for never compressed shot on film STEM DCI test footage (going from v210, back to v210 without any RGB on the way):

Cineform FS2:
PSNR y:56.9 u:57.0 v:58.5
Average: 57.3 dB
Min: 56.6 dB
Avg bitrate: 226Mbit

Cineform FS1:
PSNR y:50.6 u:50.9 v:54.1
Average: 51.3 dB
Min: 49.9 dB
Avg bitrate: 136Mbit

ProResHQ
PSNR y:55.1 u:55.7 v:57.7
Average: 55.8 dB
Min: 46.5 dB
Avg bitrate: 172Mbit

GV HQX (Q=0, S=100)
PSNR y:56.0 u:53.9 v:58.3
Average: 55.8 dB
Min: 52.6 dB
Avg bitrate: 219Mbit

DNxHR HQ
PSNR y:54.0 u:53.9 v:55.5
Average: 54.3 dB
Min: 45.2 dB
Avg bitrate: 176Mbit

DNxHD HQ
PSNR y:53.3 u:50.7 v:53.9
Average: 52.6 dB
Min: 43.5 dB
Avg bitrate: 176Mbit

XAVC-I 200
PSNR y:53.2 u:51.2 v:54.3
Average: 52.8 dB
Min: 44.3 dB
Avg bitrate: 183Mbit

ProResXQ 444 (converted back to 10bit 422)
Avg y:62.5 dB
Min y:57.6 dB
Avg bitrate: 378Mbit

[WorBry]

The Grass Valley HQX encoder implementation in Resolve is 8-bit only though.

[kolak]

It's not. It's 10bit.
It's native implementation over codec SDK, not anymore over QT engine.

[WorBry]

That's interesting. Just ran some Cineform (Best=FilmScan 2) and GV HQX (Best) transcodes, both AVI and MOV, from a 10-bit 422 source using Resolve's transcode function. Indeed, Resolve reports all four transcodes to be 10-bit, according to Clip Details. Yet, MediaInfo reports 8-bit for the GV HQX AVI transcode (and no bit depth at all for the other 3).

[kolak]

Bit depth report for Cineform or HQX is not reliable in mediainfo. It's more a guess than real info from headers. Don't rely on it. It's not a proper way of validating it anyway.

[WorBry]

Good to know GV HQX is really 10-bit anyway.

[kolak]

Yep, it's proper integration (also on Linux if I'm correct), but Resolve and Edius are about the only apps where you can import/export HQX at 10bit.
There is workaround for AE also (+decoding in ffmpeg). Not big choice of apps though.

You better of using DNxHR which is also good and has RGB and YUV modes for 4:4:4. It also supports 12bit for 4:4:4. Cineform 4:4:4 mode is always RGB and ProRes 4:4:4 is always (95% sure) YUV+ 12bit is not really kept. It's more like 11.5 bits (at least in my test in AE and Resolve).

[preludium975]

I still didn't get a specific answer to my question. You've written a lot of encoders.

So what is the best, nearly/real lossless encoder with 10 bit support? I have 2TB free space for export the movies.

[kolak]

Don't ask for best or you will get banned.

It's your choice- what works for you. DNxHR or ProRes depending on the OS are good choices.
Do you need near lossless? What is your source, what you do with it in Resolve and what are you going to do with it after export? Lossless or intermediate codecs are used to provide high quality end master which is later used further eg. passed to 3rd parties for further transcoding for web etc.

[preludium975]

Don't ask for best or you will get banned.

It's your choice- what works for you. DNxHR or ProRes depending on the OS are good choices.
Do you need near lossless? What is your source, what you do with it in Resolve and what are you going to do with it after export? Lossless or intermediate codecs are used to provide high quality end master which is later used further eg. passed to 3rd parties for further transcoding for web etc.

Firtst, I tonemap HDR -> SDR, then I going to export this in 10bit.
Then I load it to AvsPmod, then i will write the filterchain (Convertbits 10bit -> 8bit, dirty.lines fix, crop, resizemod, etc..).
Then encode it with x264.

[kolak]

DNxHR HQX or Cineform (best quality setting) will be good enough.
Is Resolve HDR to SDR conversion any decent? I think you may get as good results natively in avs or vs and avoid whole Resolve route.

[WorBry]

Good to know GV HQX is really 10-bit anyway.

Interesting observation here also:

https://forum.blackmagicdesign.com/viewtopic.php?f=21&t=78557&p=434432#p434432

I can see I've going to have to install the Grass Valley VFW codecs again to satisfy my curiosity about this.

[kolak]

Not much to satisfy- HQX VFW codec is 8bit on input/output and that's about it :)
It has been like this from the beginning and unlikely to change. GV is not really interested to provide new version.
Edius is actually dying.

[preludium975]

I tried DNxHR HQX and GoPro CineForm. All of them, after exporting, at 30 seconds-1 minute intervals duplicate frames. Like fps drop, but not fps drop, its just sometimes duplicate frames.

Here is a test encode for example, at 00:43-00:47. here (https://mega.nz/#!wt9CQKRC!52jmzGquuod0cZsE2cn87gjLLKsSfl6egHXjO9HJbAY)

Who has an idea?

[poisondeathray]

I tried DNxHR HQX and GoPro CineForm. All of them, after exporting, at 30 seconds-1 minute intervals duplicate frames. Like fps drop, but not fps drop, its just sometimes duplicate frames.

Here is a test encode for example, at 00:43-00:47. here (https://mega.nz/#!wt9CQKRC!52jmzGquuod0cZsE2cn87gjLLKsSfl6egHXjO9HJbAY)

Who has an idea?


Did you check at what stage of the process that it occurred ? e.g. is it in your source, in resolve after decoding , after export, or after decoding the export then encoding to your final format ?

What about other export formats ? eg. uncompressed . If that has problem too then that suggests problem with your settings (maybe fps mismatch) or problem with source decoding in resolve

Maybe a mod should move these last few posts , because they have nothing to do with magicyuv

[WorBry]

The 'test encode' is a 720x302, 23.976, AVC.mkv file - there's no way it came out of Resolve.


Maybe a mod should move these last few posts , because they have nothing to do with magicyuv

True.

@Preludium975. I see you registered on the Blackmagic Forum:

https://forum.blackmagicdesign.com/viewtopic.php?f=21&t=78574&p=434386&hilit=HDR#p434386

Probably best to address all of your Resolve-related issues there. I'd suggest you give more contextual information though, if you want answers.

[preludium975]

The 'test encode' is a 720x302, 23.976, AVC.mkv file - there's no way it came out of Resolve.

Yes, i encoded it with x264. But in the source file also there is the lag. I just don't want to send a 500GiB file.

[WorBry]

Yes, i encoded it with x264. But in the source file also there is the lag. I just don't want to send a 500GiB file.

I'm assuming by that you mean the Resolve export (DNxHR, Cineform) that you used as source for the x264 encoding, not the original (HDR) source? If so then it would point to some FPS-mismatch (likely settings related) going on in Resolve.

Quite honestly if that is your target x264 format and you are only using Resolve for the HDR>SDR conversion, why jump through these hoops, generating large intermediate files in the process - you could do it all with AVISynth(+) or VapourSynth, like Kolak suggested?

Is Resolve HDR to SDR conversion any decent? I think you may get as good results natively in avs or vs and avoid whole Resolve route.

[preludium975]

I'm assuming by that you mean the Resolve export (DNxHR, Cineform) that you used as source for the x264 encoding, not the original (HDR) source? If so then it would point to some FPS-mismatch (likely settings related) going on in Resolve.

Quite honestly if that is your target x264 format and you are only using Resolve for the HDR>SDR conversion, why jump through these hoops, generating large intermediate files in the process - you could do it all with AVISynth(+) or VapourSynth, like Kolak suggested?

The best Tone Mapper for now the DaVinci Resolve. Yes, I tone mapped HDR->SDR, then I exported it to DNxHR HQX.
I need the Resolve, because this is the best solution.

The last thing, what need to be works, the DNxHR HQX. Because this is a visually lossless encoder. Cineform are great too, but this is also duplicating frames...

[WorBry]

Well if there's a FPS mismatch going on it will appear in the Resolve export whatever the format. Like Poisondeathray said, to pin-point where that is occurring means going through your Resolve project set-up. Really this is not the place for that, but if it brings closure on this issue (and this segment of the thread gets moved anyway):

1. What is the format and FPS of your original HDR source, as declared (Clip Details or Clip Attributes) by Resolve on import ?

2. What is the declared project timeline frame-rate after you have loaded your clips on the timeline. Check in Project Settings > Master Settings > Timeline Format. Resolve defaults to 24FPS. If your source frame-rate is different, when you import the first clip you will be given the option to change the project frame-rate to match the source or leave at default 24FPS, in which case it will be 'locked' at 24FPS - can't be changed subsequently. Otherwise you should set-up the project Timeline Format beforehand. Your problem could be right there. 23.976p source in 24p project maybe ?

3. I'm assuming you didn't mess with Clip Speed/Duration/Retime on the timeline ?

4. What was the render (Delivery) frame-rate setting on export to Cineform, DNxHD ?. Default is the project frame-rate, but there may be other options. A 24 FPS project timeline for example will give the option of exporting at 24, 23.976 or 30 (3:2) FPS. Potential for FPS discontinuity there also.

With that advice you should be able to figure out where FPS discontinuity occurred, if it did. Don't bring it back here though. If you still have issues, address it on the Blackmagic Forum.

The best Tone Mapper for now the DaVinci Resolve.

Based on what ? Again though, not a point for discussion in this thread.

Edit: Incidentally where exactly (frame sequence position) are the duplicate frames occurring in your 'Test.mkv' file ? Started manually frame stepping through the file with VDub2 but got fed up about a third of the way .

[Stereodude]

According to the e-mail I got earlier in the week v2.1 is out now.

2.1.0
Released: 2019.07.11.
NEW: Added v308, v408 pixel format support (YUV 4:4:4).
NEW: Added 10-bit YUV 4:4:4 compressed format.
FIX: Core: Fixed a bug that prevented encoding of 16-bit planar RGB pixel formats.

[TomArrow]

2.2.0

Released: 2019.10.30.

NEW: MagicYUV MediaCore plugin for Adobe Premiere Pro, Media Encoder and After Effects.
Other minor fixes and improvements.

^ This finally caused me to buy this. Great stuff. Finally a way to export losslessly at higher bit depths that is very comfortable to use.

If the programmer is still reading this thread, I still have one plea: Please make a native After Effects plugin. I know I can use MediaCore plugins via Media Encoder and Media Encoder can encode After Effects projects, but you lose control over the more detailed render settings, most importantly the export color space, which Media Encoder incomprehensibly doesn't let you set. So it's impossible to, for example, export a floating point linear composition as in HDR PQ color space, which After Effects can do with native export plugins but not with Media Encoder. As such I still end up having to export an image sequence from AE in order to finally encode the video with MagicYUV.

Anyway, great codec, thanks for making this!

[Ignus2]

@TomArrow

Yes, I occasionally read this thread. Glad you can make use of the plugins.

The MediaCore plugins were a major pain in the rear to write in the first place for a number of reasons. But it adds the greatest coverage for Adobe tools, albeit as you say After Effects export through Media Encoder has it's limits.
The problem is that After Effects uses a completely different API/SDK for its plugins, and I haven't yet looked into it that much.
An important thing to note though is that the plugins don't really do anything with color spaces for higher bit depths. That means that through MediaCore I get 16-bit RGB values and encode them without any transformation (except bit depth reduction to 10/12/14). On import the project must be set up manually to properly interpret the footage (I presume).
I'll look into the After Effects SDK and see how much effort would it take to make a dedicated plugin for it.

[TomArrow]

@TomArrow

Yes, I occasionally read this thread. Glad you can make use of the plugins.

The MediaCore plugins were a major pain in the rear to write in the first place for a number of reasons. But it adds the greatest coverage for Adobe tools, albeit as you say After Effects export through Media Encoder has it's limits.
The problem is that After Effects uses a completely different API/SDK for its plugins, and I haven't yet looked into it that much.
An important thing to note though is that the plugins don't really do anything with color spaces for higher bit depths. That means that through MediaCore I get 16-bit RGB values and encode them without any transformation (except bit depth reduction to 10/12/14). On import the project must be set up manually to properly interpret the footage (I presume).
I'll look into the After Effects SDK and see how much effort would it take to make a dedicated plugin for it.

Thanks for your answer!

I really don't know anything about how it works behind the scenes, but I would wager that After Effects already delivers the RGB values in the correct color space to the plugin that is exporting it, so you wouldn't have to deal with that yourself. Only with Media Encoder there is no way to set the output color space whereas with After Effects there is. I would then guess that the RGB values you get from Premiere/MediaCore are Rec709 or sRGB by default, but I really don't know to be honest.

And I totally feel your pain there, I also once wanted to inform myself about how to write some Premiere/AE plugin and I gave up quickly after glancing over the SDK, being the complete C++ beginner that I am. :)

[tormento]

I'd like to give MagicYUV a try before buying.

Which is the best way to read/write MagicYUV compressed video with AVS+?

[hajj_3]

@Ignus2 PLEASE can you update the benchmarks on your website as it compares MagicYUV v2.0 with UTVideo v17.1.0 instead of the latest MagicYUV v2.2.0 and UTVideo v21.3.0.

[foxyshadis]

I'd like to give MagicYUV a try before buying.

Which is the best way to read/write MagicYUV compressed video with AVS+?

MagicYUV is a plain ol' VFW codec, so all you need is AVISource if it's in AVI, or some other source reader if it's in another container.

[tormento]

MagicYUV is a plain ol' VFW codec, so all you need is AVISource if it's in AVI, or some other source reader if it's in another container.



Picture we have a MKV. What is a good frame accurate VFW reader?

What a pity they did not develope a AVS/VS plugin.

[hajj_3]

v2.3.0 Released: 2020.09.21.

NEW: MagicYUV plugin for Vegas Pro 17/18.
NEW: New codec variant: 10-bit YUV 4:2:0.
FIX: Fixed incorrect video length of certain AVI files when imported into Adobe software through the plugin (particularly files recorded by vMix).
FIX: Minor Adobe plugin improvements: better error handling/reporting (like disk full, etc.).

[GAP]

Are the default settings good for lossless RGB videos?

[PCU]

Please support FLAC for audio format & MKV container for Adobe plugin!

[PCU]

Update: MagicYUV is no longer free and you have to pay to use this codec!

[richardpl]

or use ffmpeg

[hajj_3]

Update: MagicYUV is no longer free and you have to pay to use this codec!

v1.2 is free.

[PCU]

v1.2 is free.

Latest release (2.3.0)

That free version is obsolete now!

[hajj_3]

2.4.0 Released: 2023.12.06.

NEW: DaVinci Resolve Studio encoder plugin (Ultimate Edition and Windows only)
Vegas Pro plugin: Fixed various issues
Vegas Pro plugin: Added support for Vegas Pro 19/20/21
Vegas Pro plugin: Improved decoding performance for certain pixel formats