Loss of details in gradients when I convert YUV 4:2:2 to YUV 4:2:0

SuperLumberjack · 1st November 2016, 19:09

Hello

Today I tried to do a capture of the game "Rayman Origins" in 60 fps with Dxtory, just to see if it works (I haven't any SSD).

I got good results with the codec "Ut Video YUV422 BT.709 VCM" with a stable framerate. Strangely, it was less stable with "Ut Video YUV420 BT.709 VCM"

So, I need to use the YUV 4:2:2 format. But in output from Sony Vegas, I will use the codec "Ut Video YUV420 BT.709 VCM", because I want to encode my video with Handbrake.

And now is the problem !

I noticed when I converted YUV 4:2:2 to YUV 4:2:0, that there is a big degradation of the gradients. I know it's normal that there is a little loss, but here it's a bit too much

This is a screenshot from the capture with the codec "Ut Video YUV422 BT.709 VCM" :

With the "Ut Video YUV420 BT.709 VCM" :

And after I converted the video in YUV 4:2:2 to YUV 4:2:0 with "Ut Video YUV420 BT.709 VCM" :

Compared to the original YUV 4:2:0 capture, it's worse !

My only possibility should be to use the codec Lagarith in the Sony Vegas output, but one day I use Lagarith and the colors vere wrong with Handbrake. It's why I used the codec Ut Video, because it's flagged with the good color matrix.

How can I do to avoid this big loss of quality in the gradients do you think ?

Is it possible with AviSynth before I put the sequence in Sony Vegas ? Or have you another solution please ?

Thanks for you help !

(sorry for my bad English)

Mounir · 1st November 2016, 19:34

Try to disable smart resample in sony vegas (look on youtube-> disable smart resample, sony vegas)

SuperLumberjack · 1st November 2016, 19:59

Hi

I always disable the smart resample in Sony Vegas

But here, I converted my video with VirtualDub. But it's the same with Vegas.

SuperLumberjack · 1st November 2016, 21:53

I tried with my video in YUV 4:2:2, to export it with the codec Lagarith in YV12.

It's a bit better :

But it's not as good as the original YUV 4:2:0.

The other problem is with Handbrake :

As you can see, reds looks orange. I think Handbrake think that the video use the Rec.601 color matrix, and then convert it to Rec.709.

It's maybe because there is no flag for the color matrix with Lagarith.

So it's not the solution

Any idea ?

Thanks !

FranceBB · 1st November 2016, 23:41

@SuperLumberjack... why don't you save your video in 4:2:2 Ut Video, then convert it to 4:2:0 using avisynth Converttoyv12(interlaced=false) and then save it to lossless Ut Video? Besides, you could use flash3kyuu to deband your clip after Converttoyv12() and then save it to 4:2:0 Ut Video.

SuperLumberjack · 1st November 2016, 23:58

Thanks for you answer

I tried with AviSynth to convert the YUY2 to YV12, but I found that it was a bit blurry. But I will try again with your suggestions tomorrow

Before, I tried with Fraps too instead of Dxtory, and I can capture a video in 60 fps very well.

The problem, as you see at 4 min 46 in this video, is that the videos in YUV have wrong colors compared to RGB

https://youtu.be/dn3umK0gir0?t=4m46s

FranceBB · 2nd November 2016, 00:24

Hang on.
If RGB has the correct colours, then why don't you save it in RGB and then convert it with avisynth to yv12 and convert the color matrix to BT709 using colormatrix in order to avoid having the wrong colours once you have the YUV clip?

SuperLumberjack · 2nd November 2016, 00:34

Since I noticed these wrong colors in YUV with Fraps, I use RGB.

The problem is that it's impossible to have smooth videos with this in 60 fps without have a SSD. So I'm limited with 30 fps with RGB.

Fraps in YUV has clearly the advantage to be very reactive. Even more that Dxtory I find.

raffriff42 · 2nd November 2016, 01:38

For the color shift, try adding this in the handbrake x264 extra options textbox (I have not tested this)
colormatrix=bt709:colorprim=bt709:transfer=bt709

For the gradients, it looks like the captured source has minor banding issues (not unusual for a video game); color conversion in 8-bit arithmetic is making it more noticeable. Try adding a very small amount of dither (called add noise in Vegas)

SuperLumberjack · 2nd November 2016, 02:01

Thanks raffriff42

I will try. But why is there more banding when I convert YUV 4:2:2 to YUV 4:2:0, if I compare to the original YUV 4:2:0 ?

Shouldn't it look almost identical ?

Another question : do you understand why the colors are wrong in YUV with Fraps compared to RGB ? It's too bad, because the videos in YUV with Fraps are very smooth

Why anybody noticed this ? Although I contacted the Fraps team and talked to them about the problem, but they didn't really understand

And I show them with pictures like that

:

That's a shame because Fraps is a good product. Without this default, it could be even better !

raffriff42 · 2nd November 2016, 02:22

> But why is there more banding when I convert YUV 4:2:2 to YUV 4:2:0
Because the color channels are being resampled, and this always introduces errors, especially-noticeable with 8-bit video, and super-especially-noticeable on shallow gradients.

>why the colors are wrong in YUV with Fraps compared to RGB
They are not wrong if they are decoded properly and transcoded properly. It uses an unusual format: full range "Rec.709."

I wrote a program (FRAFS Test Pattern) to help screen-capture users like you and me spot color problems right away, at any point in the signal chain -- from screen to the end viewer.

Quote:

Do you worry if Fraps (or other computer video capture tool) is giving you accurate colors? Do you wish you had some kind of standard, recordable (DirectX 9) source with known colors? Well, that's what this is, so you're in luck.

SuperLumberjack · 2nd November 2016, 02:37

You're the boss raffriff42 !

I will see this later, because I need to sleep. It's 2h37 in France !

SuperLumberjack · 2nd November 2016, 16:40

Very good test raffriff42 !

I tried with Fraps in RGB, and no problem :

But colors are different in YUV as I said :

I noticed a little difference with the Ut Video codecs in YUV too, but it's more correct :

You can see this with the color picker in MS Paint.

So, what should I do ?

SuperLumberjack · 2nd November 2016, 18:44

I came back on the story of YUV 4:2:2 converted to YUV 4:2:0

I wanted to try to encode directly the YUV 4:2:2 in YUV 4:2:0 with Handbrake without any convertion before. I find the result is very good !

The MKV x264 from Handbrake in YUV 4:2:0 (YUV 4:2:2 input) :

The original YUV 4:2:2 :

Maybe the compression helps a bit too (but we lost a bit the depth I found)

So, it's the best solution and the simplest !

What do you think ? Is it possible to do better ? I tried with AviSynth, but the reds became more red

But with Fraps YUV, how can I correct the colors ?

raffriff42 · 2nd November 2016, 21:58

Glad you like my test! But how are the colors bad? Yes, the RGB bars all measure 191 exactly, and some of the YUV bars measure 189, but I consider that small deviation to be acceptable.

If you mean the loss of fine color detail, that comes with the YV12 format.

SuperLumberjack · 2nd November 2016, 22:10

Yeah, good test !

No, the problem is not the loss of details. But yes, I can accept a small deviation like with the Ut Video. The problem is for the Fraps YUV. It's more visible I think, especially on this picture :

The third image was the Fraps RGB converted in YUV 4:2:0 with the codec Ut Video. The Ut Video is nearer to the RGB I find.

1st November 2016, 19:09	#1 \| Link
SuperLumberjack French Registered User Join Date: Aug 2016 Location: France Posts: 333	Loss of details in gradients when I convert YUV 4:2:2 to YUV 4:2:0 Hello Today I tried to do a capture of the game "Rayman Origins" in 60 fps with Dxtory, just to see if it works (I haven't any SSD). I got good results with the codec "Ut Video YUV422 BT.709 VCM" with a stable framerate. Strangely, it was less stable with "Ut Video YUV420 BT.709 VCM" So, I need to use the YUV 4:2:2 format. But in output from Sony Vegas, I will use the codec "Ut Video YUV420 BT.709 VCM", because I want to encode my video with Handbrake. And now is the problem ! I noticed when I converted YUV 4:2:2 to YUV 4:2:0, that there is a big degradation of the gradients. I know it's normal that there is a little loss, but here it's a bit too much This is a screenshot from the capture with the codec "Ut Video YUV422 BT.709 VCM" : With the "Ut Video YUV420 BT.709 VCM" : And after I converted the video in YUV 4:2:2 to YUV 4:2:0 with "Ut Video YUV420 BT.709 VCM" : Compared to the original YUV 4:2:0 capture, it's worse ! My only possibility should be to use the codec Lagarith in the Sony Vegas output, but one day I use Lagarith and the colors vere wrong with Handbrake. It's why I used the codec Ut Video, because it's flagged with the good color matrix. How can I do to avoid this big loss of quality in the gradients do you think ? Is it possible with AviSynth before I put the sequence in Sony Vegas ? Or have you another solution please ? Thanks for you help ! (sorry for my bad English)

2nd November 2016, 02:01	#10 \| Link
SuperLumberjack French Registered User Join Date: Aug 2016 Location: France Posts: 333	Thanks raffriff42 I will try. But why is there more banding when I convert YUV 4:2:2 to YUV 4:2:0, if I compare to the original YUV 4:2:0 ? Shouldn't it look almost identical ? Another question : do you understand why the colors are wrong in YUV with Fraps compared to RGB ? It's too bad, because the videos in YUV with Fraps are very smooth Why anybody noticed this ? Although I contacted the Fraps team and talked to them about the problem, but they didn't really understand And I show them with pictures like that : That's a shame because Fraps is a good product. Without this default, it could be even better ! Last edited by SuperLumberjack; 2nd November 2016 at 02:07.

2nd November 2016, 18:44	#14 \| Link
SuperLumberjack French Registered User Join Date: Aug 2016 Location: France Posts: 333	I came back on the story of YUV 4:2:2 converted to YUV 4:2:0 I wanted to try to encode directly the YUV 4:2:2 in YUV 4:2:0 with Handbrake without any convertion before. I find the result is very good ! The MKV x264 from Handbrake in YUV 4:2:0 (YUV 4:2:2 input) : The original YUV 4:2:2 : Maybe the compression helps a bit too (but we lost a bit the depth I found) So, it's the best solution and the simplest ! What do you think ? Is it possible to do better ? I tried with AviSynth, but the reds became more red But with Fraps YUV, how can I correct the colors ? Last edited by SuperLumberjack; 2nd November 2016 at 21:12.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

1st November 2016, 19:34	#2 \| Link
Mounir Registered User Join Date: Nov 2006 Posts: 773	Try to disable smart resample in sony vegas (look on youtube-> disable smart resample, sony vegas)

1st November 2016, 19:59	#3 \| Link
SuperLumberjack French Registered User Join Date: Aug 2016 Location: France Posts: 333	Hi I always disable the smart resample in Sony Vegas But here, I converted my video with VirtualDub. But it's the same with Vegas.

1st November 2016, 21:53	#4 \| Link
SuperLumberjack French Registered User Join Date: Aug 2016 Location: France Posts: 333	I tried with my video in YUV 4:2:2, to export it with the codec Lagarith in YV12. It's a bit better : But it's not as good as the original YUV 4:2:0. The other problem is with Handbrake : As you can see, reds looks orange. I think Handbrake think that the video use the Rec.601 color matrix, and then convert it to Rec.709. It's maybe because there is no flag for the color matrix with Lagarith. So it's not the solution Any idea ? Thanks !

1st November 2016, 23:41	#5 \| Link
FranceBB Broadcast Encoder Join Date: Nov 2013 Location: Royal Borough of Kensington & Chelsea, UK Posts: 2,903	@SuperLumberjack... why don't you save your video in 4:2:2 Ut Video, then convert it to 4:2:0 using avisynth Converttoyv12(interlaced=false) and then save it to lossless Ut Video? Besides, you could use flash3kyuu to deband your clip after Converttoyv12() and then save it to 4:2:0 Ut Video.

1st November 2016, 23:58	#6 \| Link
SuperLumberjack French Registered User Join Date: Aug 2016 Location: France Posts: 333	Thanks for you answer I tried with AviSynth to convert the YUY2 to YV12, but I found that it was a bit blurry. But I will try again with your suggestions tomorrow Before, I tried with Fraps too instead of Dxtory, and I can capture a video in 60 fps very well. The problem, as you see at 4 min 46 in this video, is that the videos in YUV have wrong colors compared to RGB https://youtu.be/dn3umK0gir0?t=4m46s

2nd November 2016, 00:24	#7 \| Link
FranceBB Broadcast Encoder Join Date: Nov 2013 Location: Royal Borough of Kensington & Chelsea, UK Posts: 2,903	Hang on. If RGB has the correct colours, then why don't you save it in RGB and then convert it with avisynth to yv12 and convert the color matrix to BT709 using colormatrix in order to avoid having the wrong colours once you have the YUV clip?

2nd November 2016, 00:34	#8 \| Link
SuperLumberjack French Registered User Join Date: Aug 2016 Location: France Posts: 333	Since I noticed these wrong colors in YUV with Fraps, I use RGB. The problem is that it's impossible to have smooth videos with this in 60 fps without have a SSD. So I'm limited with 30 fps with RGB. Fraps in YUV has clearly the advantage to be very reactive. Even more that Dxtory I find.

2nd November 2016, 01:38	#9 \| Link
raffriff42 Retried Guesser Join Date: Jun 2012 Posts: 1,373	For the color shift, try adding this in the handbrake x264 extra options textbox (I have not tested this) colormatrix=bt709:colorprim=bt709:transfer=bt709 For the gradients, it looks like the captured source has minor banding issues (not unusual for a video game); color conversion in 8-bit arithmetic is making it more noticeable. Try adding a very small amount of dither (called add noise in Vegas)

2nd November 2016, 02:37	#12 \| Link
SuperLumberjack French Registered User Join Date: Aug 2016 Location: France Posts: 333	You're the boss raffriff42 ! I will see this later, because I need to sleep. It's 2h37 in France !

2nd November 2016, 16:40	#13 \| Link
SuperLumberjack French Registered User Join Date: Aug 2016 Location: France Posts: 333	Very good test raffriff42 ! I tried with Fraps in RGB, and no problem : But colors are different in YUV as I said : I noticed a little difference with the Ut Video codecs in YUV too, but it's more correct : You can see this with the color picker in MS Paint. So, what should I do ?

2nd November 2016, 21:58	#15 \| Link
raffriff42 Retried Guesser Join Date: Jun 2012 Posts: 1,373	Glad you like my test! But how are the colors bad? Yes, the RGB bars all measure 191 exactly, and some of the YUV bars measure 189, but I consider that small deviation to be acceptable. If you mean the loss of fine color detail, that comes with the YV12 format.

2nd November 2016, 22:10	#16 \| Link
SuperLumberjack French Registered User Join Date: Aug 2016 Location: France Posts: 333	Yeah, good test ! No, the problem is not the loss of details. But yes, I can accept a small deviation like with the Ut Video. The problem is for the Fraps YUV. It's more visible I think, especially on this picture : The third image was the Fraps RGB converted in YUV 4:2:0 with the codec Ut Video. The Ut Video is nearer to the RGB I find.