Loss of details in gradients when I convert YUV 4:2:2 to YUV 4:2:0

[SuperLumberjack]

Hello :)

Today I tried to do a capture of the game "Rayman Origins" in 60 fps with Dxtory, just to see if it works (I haven't any SSD).

I got good results with the codec "Ut Video YUV422 BT.709 VCM" with a stable framerate. Strangely, it was less stable with "Ut Video YUV420 BT.709 VCM" :confused:

So, I need to use the YUV 4:2:2 format. But in output from Sony Vegas, I will use the codec "Ut Video YUV420 BT.709 VCM", because I want to encode my video with Handbrake.

And now is the problem ! :rolleyes:

I noticed when I converted YUV 4:2:2 to YUV 4:2:0, that there is a big degradation of the gradients. I know it's normal that there is a little loss, but here it's a bit too much :(

This is a screenshot from the capture with the codec "Ut Video YUV422 BT.709 VCM" :

http://image.noelshack.com/fichiers/2016/44/1478020695-rayman-origins-dxtory-ut-video-yuv422-bt-709-vcm.png

With the "Ut Video YUV420 BT.709 VCM" :

http://image.noelshack.com/fichiers/2016/44/1478020762-rayman-origins-dxtory-ut-video-yuv420-bt-709-vcm.png

And after I converted the video in YUV 4:2:2 to YUV 4:2:0 with "Ut Video YUV420 BT.709 VCM" :

http://image.noelshack.com/fichiers/2016/44/1478020837-rayman-origins-dxtory-ut-video-yuv422-bt-709-vcm-converti-en-yuv420.png

Compared to the original YUV 4:2:0 capture, it's worse !

My only possibility should be to use the codec Lagarith in the Sony Vegas output, but one day I use Lagarith and the colors vere wrong with Handbrake. It's why I used the codec Ut Video, because it's flagged with the good color matrix.

How can I do to avoid this big loss of quality in the gradients do you think ?

Is it possible with AviSynth before I put the sequence in Sony Vegas ? Or have you another solution please ?


Thanks for you help ! ;)

(sorry for my bad English)

[Mounir]

Try to disable smart resample in sony vegas (look on youtube-> disable smart resample, sony vegas)

[SuperLumberjack]

Hi :)

I always disable the smart resample in Sony Vegas ;) But here, I converted my video with VirtualDub. But it's the same with Vegas.

[SuperLumberjack]

I tried with my video in YUV 4:2:2, to export it with the codec Lagarith in YV12.

It's a bit better :

http://image.noelshack.com/fichiers/2016/44/1478032273-rayman-origins-dxtory-ut-video-yuv422-bt-709-vcm-vers-lagarith-yv12.png

But it's not as good as the original YUV 4:2:0.

The other problem is with Handbrake :

http://image.noelshack.com/fichiers/2016/44/1478032396-rayman-origins-resultat-encodage-handbrake-lagarith-yv12.png

As you can see, reds looks orange. I think Handbrake think that the video use the Rec.601 color matrix, and then convert it to Rec.709.

It's maybe because there is no flag for the color matrix with Lagarith.

So it's not the solution :(

Any idea ?

Thanks ! :p

[FranceBB]

@SuperLumberjack... why don't you save your video in 4:2:2 Ut Video, then convert it to 4:2:0 using avisynth Converttoyv12(interlaced=false) and then save it to lossless Ut Video? Besides, you could use flash3kyuu to deband your clip after Converttoyv12() and then save it to 4:2:0 Ut Video. ;)

[SuperLumberjack]

Thanks for you answer :)

I tried with AviSynth to convert the YUY2 to YV12, but I found that it was a bit blurry. But I will try again with your suggestions tomorrow ;)

Before, I tried with Fraps too instead of Dxtory, and I can capture a video in 60 fps very well.

The problem, as you see at 4 min 46 in this video, is that the videos in YUV have wrong colors compared to RGB :(

https://youtu.be/dn3umK0gir0?t=4m46s

[FranceBB]

Hang on.
If RGB has the correct colours, then why don't you save it in RGB and then convert it with avisynth to yv12 and convert the color matrix to BT709 using colormatrix in order to avoid having the wrong colours once you have the YUV clip?

[SuperLumberjack]

Since I noticed these wrong colors in YUV with Fraps, I use RGB.

The problem is that it's impossible to have smooth videos with this in 60 fps without have a SSD. So I'm limited with 30 fps with RGB.

Fraps in YUV has clearly the advantage to be very reactive. Even more that Dxtory I find.

[raffriff42]

For the color shift, try adding this in the handbrake x264 extra options textbox (I have not tested this)
colormatrix=bt709:colorprim=bt709:transfer=bt709

For the gradients, it looks like the captured source has minor banding issues (not unusual for a video game); color conversion in 8-bit arithmetic is making it more noticeable. Try adding a very small amount of dither (called add noise in Vegas)

[SuperLumberjack]

Thanks raffriff42 ;)

I will try. But why is there more banding when I convert YUV 4:2:2 to YUV 4:2:0, if I compare to the original YUV 4:2:0 ?

Shouldn't it look almost identical ?

Another question : do you understand why the colors are wrong in YUV with Fraps compared to RGB ? It's too bad, because the videos in YUV with Fraps are very smooth :(

Why anybody noticed this ? Although I contacted the Fraps team and talked to them about the problem, but they didn't really understand :rolleyes:

And I show them with pictures like that :p :

http://image.noelshack.com/fichiers/2016/37/1473697820-fraps-rgb-vs-fraps-yuv-vs-yuv420-rec-709.png

That's a shame because Fraps is a good product. Without this default, it could be even better !

[raffriff42]

> But why is there more banding when I convert YUV 4:2:2 to YUV 4:2:0
Because the color channels are being resampled, and this always introduces errors, especially-noticeable with 8-bit video, and super-especially-noticeable on shallow gradients.

>why the colors are wrong in YUV with Fraps compared to RGB
They are not wrong if they are decoded properly and transcoded properly. It uses an unusual format: full range "Rec.709."

I wrote a program (FRAFS Test Pattern (https://sourceforge.net/projects/frafstestpatt/)) to help screen-capture users like you and me spot color problems right away, at any point in the signal chain -- from screen to the end viewer. Do you worry if Fraps (or other computer video capture tool) is giving you accurate colors? Do you wish you had some kind of standard, recordable (DirectX 9) source with known colors? Well, that's what this is, so you're in luck.

[SuperLumberjack]

You're the boss raffriff42 ! ;)

I will see this later, because I need to sleep. It's 2h37 in France ! :p

[SuperLumberjack]

Very good test raffriff42 ! ;)

I tried with Fraps in RGB, and no problem :

http://image.noelshack.com/fichiers/2016/44/1478100014-fraps-rgb.png

But colors are different in YUV as I said :

http://image.noelshack.com/fichiers/2016/44/1478100196-fraps-yuv.png

I noticed a little difference with the Ut Video codecs in YUV too, but it's more correct :

http://image.noelshack.com/fichiers/2016/44/1478100285-dxtory-ut-video-yuv422-bt-709-vcm.png

You can see this with the color picker in MS Paint.

So, what should I do ? :)

[SuperLumberjack]

I came back on the story of YUV 4:2:2 converted to YUV 4:2:0 :p

I wanted to try to encode directly the YUV 4:2:2 in YUV 4:2:0 with Handbrake without any convertion before. I find the result is very good ! :)

The MKV x264 from Handbrake in YUV 4:2:0 (YUV 4:2:2 input) :

http://image.noelshack.com/fichiers/2016/44/1478107979-rayman-origins-dxtory-ut-video-yuv422-bt-709-vcm-encode-en-yuv420-par-handbrake.png

The original YUV 4:2:2 :

http://image.noelshack.com/fichiers/2016/44/1478020695-rayman-origins-dxtory-ut-video-yuv422-bt-709-vcm.png

Maybe the compression helps a bit too (but we lost a bit the depth I found) ;)

So, it's the best solution and the simplest !

What do you think ? Is it possible to do better ? I tried with AviSynth, but the reds became more red :rolleyes:

But with Fraps YUV, how can I correct the colors ?

[raffriff42]

Glad you like my test! But how are the colors bad? Yes, the RGB bars all measure 191 exactly, and some of the YUV bars measure 189, but I consider that small deviation to be acceptable.


If you mean the loss of fine color detail, that comes with the YV12 format.

[SuperLumberjack]

Yeah, good test ! :p

No, the problem is not the loss of details. But yes, I can accept a small deviation like with the Ut Video. The problem is for the Fraps YUV. It's more visible I think, especially on this picture :

http://image.noelshack.com/fichiers/2016/37/1473697820-fraps-rgb-vs-fraps-yuv-vs-yuv420-rec-709.png

The third image was the Fraps RGB converted in YUV 4:2:0 with the codec Ut Video. The Ut Video is nearer to the RGB I find.