MagicYUV - a new fast lossless video codec

[Rotareneg]

In Sony Movie Studio (and Vegas) 13, the software can pass frames directly without recompression if they haven't been changed, which can really speed things up. This works fine with the Ut Video codec, but I cannot get this to work with MagicYUV. Are there any particular settings in MagicYUV that I might be able to change to make it work?

[Ignus2]

Originally Posted by Rotareneg

In Sony Movie Studio (and Vegas) 13, the software can pass frames directly without recompression if they haven't been changed, which can really speed things up. This works fine with the Ut Video codec, but I cannot get this to work with MagicYUV. Are there any particular settings in MagicYUV that I might be able to change to make it work?

Not in particular. I don't know how MS/Vegas does smart rendering so I'm not sure what would be needed to make it work. It would be nice to know what do those programs look at to make the decision to smart render.
Maybe they too have a hard-coded list of fourccs (I wouldn't be surprised)

Greets,
I.

[Rotareneg]

Ah, I think I figured out what the problem might be: I'm recording from games using Dxtory, which outputs 32 bit RGB data (RGBA compression is unchecked in MagicYUV) If I re-compress that video as 24 bit RGB smart rendering will work fine in MS. Videos recorded from Dxtory with MagicYUV set to convert to YUV420 won't smart render either.

It would seem MS/Vegas smart rendering does not like it when a codec supports multiple color spaces on the same fourcc. I was able to get MS to spit out a corrupted magicYUV .avi by re-compressing a RGB24 video through MS with MagicYUV converting to YUV420, and then trying to smart render that file. The MagicYUV tray icon will show it's converting RGB24 to RGB while MS is smart rendering the YUV420 video, which crashes anything, including the windows explorer when it trying to build a preview for it.

Here's a link to the corrupted avi, packed in a 7zip archive: https://drive.google.com/file/d/0BzS...ew?usp=sharing

Beware, it'll crash explorer.exe if it tries to build a thumbnail for the file, it's that messed up!

[Ignus2]

Great, thanks for the info! I might have some ideas here.
Could you also share:
- a sample which is recorded directly with dxtory and refuses to smart render (either compressed RGB or YUV420 or both)
- a recompressed sample of the above which actually smart-renders
That would help a lot, as I could look at what the differences are.

About the crash: yes, as there are mixed frames in the container AVI, the codec barfs. It should check for this at least and either refuse to decode the mixed-up frames or do something about it. So it can be considered a bug also in the codec.

In any case, the next release will have separate fourccs, the current solution is problematic for multiple reasons (the alpha checkbox being one prime example, among others).

Greets,
I.

[Ignus2]

On a side note: Are you the one, who reported the following issues with dxtory?
http://forum.exkode.com/index.php?topic=1793.0
http://forum.exkode.com/index.php?topic=2165.0

In fact, I remember someone writing to me about similar strange issues, and we couldn't track it down. I tried writing to the dxtory devs to look into the issue, but no answer so far.
Were you able to find out the cause?

Greets,
I.

[Rotareneg]

Yes, I was the one who reported that. I can only imagine it's an issue with with Dxtory itself as I've not seen that problem happen when testing MagicYUV or Lagarith with other software.

Here's a 7zip with three videos, one direct from dxtory, one recompressed with msp13, and the final smart rendered from the previous with msp13: https://drive.google.com/open?id=0Bz...WdFT3hBZ19LdzA

[Ignus2]

About Dxtory: Does the issue still persist? Do you still get storage bottleneck? BTW, how did you measure disk queue length and drive active time?

About the videos: Thanks. The immediate difference I see between the Dxtory record and the recompressed video is the global AVI bitmap info header field "biSizeImage".
Hmmm... My guess would be Vegas/MS probably checks this global header of the original video and the newly compressing video, and if it differs in any way, then refuses to smart-render as maybe it thinks the format is not exactly the same.
The funny thing about this value however is that it's not relevant for compressed formats in the GLOBAL AVI header, as each frame has a different size (obviously, each frame is compressed to different sizes, duh). But unfortunately it must have some global value, whatever that is. The codec doesn't care.

The reason it works "sometimes" is that currently I simply pass the raw input value in there. So as Dxtory gives RGB32, this value gets to be: ( width * height * 4 ) in the compressed AVI.
BUT, when an app (like Vegas) begins to smart render, it passes the codec RGB24, so the value gets to be: ( width * height * 3 ) (well, precisely it is round_to_4(width*3)*height, but anyway...).
So now we have a difference in this irrelevant global AVI header field, and my guess is Vegas detects it as difference!
The reason it works in the recompressed case is that the recompressor also passed RGB24 to the codec, so now this irrelevant field ended up being the same that Vegas will produce in the smart rendering case.

So it seems I should set this field to something consistent, so apps like Vegas can smart-render.

Greets,
I.

[Rotareneg]

Yes, the disk usage bug is still present in Dxtory. You can monitor the disk queue length and other similar info in Windows using the resource monitor, accessible through the appropriately named "resource monitor" button in the performance tab of the task manager.

[Ignus2]

I see. I just tried, but I can't reproduce. I have the same value in Resource Monitor "Disk Activity" (the small green block below the process list there) as the bitrate reported by Dxtory, and "Highest Active Time" corresponds accordingly: 7-10% for 32 MB/s.
Do you also have a higher value for "Disk Activity" in MB/s compared to the bitrate reported by Dxtory?

Greets,
I.

[Rotareneg]

Here's two screenshots showing the difference between recording with utvideo:



And MagicYUV:



When recording utvideo, it was writing about 133 megabytes/sec with a drive activity of about 55%. With MagicYUV it was writing only 111 megabytes/sec, but with a drive activity of 64%.

Now that I've got both those screenshots up next to each other I can see something else odd: when recording utvideo Dxtory had very little read activity, about 700 bytes/sec, but with MagicYUV it was reading around 250000 bytes/sec, a significant difference.

[Ignus2]

Indeed, that Read is abnormal, I can reproduce. It's zero for Ut, but well around 200K for MagicYUV. It also happens for x264vfw and Lagarith.
This very well could be a reason, but I have no idea about the cause yet...

[Ignus2]

OK, I think I tracked it down.
It happens if the codec produces an odd number of bytes as output. If I round up the compressed data to be an even number (pad with a byte when it's odd), the Read is zero.
I'd say this is mildly insane...

Greets,
I.

[Asmodian]

Originally Posted by Ignus2

OK, I think I tracked it down.
It happens if the codec produces an odd number of bytes as output. If I round up the compressed data to be an even number (pad with a byte when it's odd), the Read is zero.
I'd say this is mildly insane...

Greets,
I.

I wonder if it is related to sector sizes? If you write less than the sector size to a drive it has to read that data back when writing the next bit of the file because it fills up the partial sector with the new data and can only write complete sectors.

Is there a 4K aligned write buffer in MagicYUV?

[Ignus2]

Originally Posted by Asmodian

I wonder if it is related to sector sizes? If you write less than the sector size to a drive it has to read that data back when writing the next bit of the file because it fills up the partial sector with the new data and can only write complete sectors.

Is there a 4K aligned write buffer in MagicYUV?

The destination buffer is given by the application using the codec (which is large enough for the worst case), the codec simply writes there the compressed data and specifies how many useful bytes have been written to that buffer.
And it still wouldn't explain why only odd sizes are affected, and a single byte of padding (to be even) fixes it.

So what I think might happening instead, is that AVI chunks must be padded to a 2 byte boundary, and Dxtory probably has something like:

write(compressed_data, compressed_size)
if compressed_size is odd:
    write(0,1) // write a single byte

So I assume it makes a separate write call to pad with that 1 byte, which could cause the read-back you describe.

EDIT: This also explains the 200K read-back, as at 60 FPS you write 60 AVI frame chunks, and if you have all buffers odd that would be 60 * 4k = 240k. For me it fluctuates between 120k-240k, as some buffers end up being even.

Greets,
I.

[Ignus2]

1.2rev1 is out which includes the workaround, and it seems to fix the disk activity issue with Dxtory.

It also has some changes which might help with smart-rendering. It needs testing though, so if someone (or you Rotareneg) can check that out that'd be very helpful.

Big kudos to Rotareneg for pointing in the right direction to nail down the issue!

Greets,
I.

[Rotareneg]

The smart rendering works in Movie Studio 13 Platinum now, with a few catches. If you render an RGB video to YUV420 or 422 it properly re-renders all frames, but if you render to YUV444 it smart renders it and you get a video that's blank but has the file size of a proper video. If you render a YUV420 video with MagicYUV set to compress as-is it also creates a blank video that's still full sized. If you set MagicYUV's "Mode (conversion)" setting to YUV420 it will properly smart render a valid video. Here are the YUV420 files if you want to take a look at 'em: https://drive.google.com/open?id=0Bz...XYwMW5SWWdqb3c

I wouldn't be surprised if these odd behaviors are just MS13 being a bit buggy, in normal use everything's working great now, many thanks!

[Ignus2]

Thanks. Looking at it "Untitled_420.avi" and "Untitled_420_convert_to_420.avi" are identical, and they are correct.

However "Untitled_420_compress_as_is.avi" in it's global AVI header indicates that it is compressed RGB, however it contains compressed YUV420 frames. This is why it turns out blank, as the codec now checks this and errors out (instead of crashing, as it did before).
The reason it's RGB in the global header is that MS can only send RGB frames for compression, so the resulting video gets compressed "as-is", as RGB, BUT, being "smart"-rendered, the original YUV420 frames are simply copied, hence you get a discrepancy between the global header (RGB) and the actual frames (YUV420).

Now, MS cannot seem to detect that the compressed output will be different than the input, as it only checks a few parameters, which seem to be:
- frame size (width, height)
- codec fourcc
- bit count

The problem is those values are the same for both compressed RGB and compressed YUV420:
- codec fourcc: MAGY
- bit count: 24

MagicYUV, and as I've seen with other codecs as well, bit count is set to 24 for non-alpha and 32 for alpha compressed formats. I'm reluctant to modify this, as it might break other apps expecting it to be this way.

Having separate fourccs for the different compressed color space/subsampling variations could help further with smart-rendering (which 1.3 will have), but not fully, as even in this case, the various other differences in codec settings (full-range/limited-range YUV, color matrix, etc.) would not be detected (MS cannot reliably check for these), making the resulting compressed file broken.

Conclusion:
So as it stands now, smart-rendering is a slippery terrain, it can work, but you have to know exactly ALL the parameters the input file was encoded with, and absolutely make sure that the new compression settings match those exactly.

Greets,
I.

[Sparktank]

Had just enough to buy the Ultimate version (after currency exchange). Thanks!

[Ignus2]

Originally Posted by Sparktank

Had just enough to buy the Ultimate version (after currency exchange). Thanks!

Yeah, I saw, thank you very much!
BTW, you can get over to the MagicYUV forums if you wish, I'd like to know how do you make use of the codec (especially the 10bit+ variants).

Greets,
I.

[De-M-oN]

I never knew that Median is the best method for game capture. Thank you for the much better quickinfo text and its really awesome that you even consider game capture at codec development and giving us even tips for best settings to do it (the median compression tip was very good it works indeed very fast and more smooth than dynamic. Thank you!)

Predict Left is gone as a setting or fully gone? I dont miss it. I'm just interested what happened to this.

maybe you can try to add a mode for maximum compression in the region like Lagarith has? Lagarith has an insane good compression and I think that your codec may be faster than lagarith, especially because lagarith' last update was some years ago and I would assume that it doesnt support AVX instruction and so on.
Sometimes maximum compression can be useful. This time I mean in general, not as game capture. There I'll stay at median of course.