Conversion of YV12 to RGB

[WorBry]

Hi,

I've just got VapourSynth up and running in linux (Kubuntu 15.10) and I'm now looking at the possibility of replicating some of the scripted AVISynth routines I use.

For conversion of my YV12 (HD AVC) sources to RGB and back I've been using what I call "Gavino's approach" as per:

http://forum.doom9.org/showthread.ph...88#post1571388

So:

ConvertToYV24(chromaresample="point", interlaced=false)
MergeChroma(PointResize(width, height, 0, 1))
ConvertToRGB32(matrix="rec709", interlaced=false)

...for further processing in RGB32 and then:

ConvertToYV12(chromaresample="point", matrix="rec709")

Is there a way the same can be done in VapourSynth?

[Myrsloik]

I have to ask why you need to convert to RGB and back at all in the first place. You probably shouldn't do that.

[WorBry]

Well, because:

Originally Posted by WorBry

...for further processing in RGB32...

And by that I mean encoding to intermediate RGB formats suitable/desirable for further processing in other linux applications (KDenLive, Cinelerra, Natron etc).

Basically, I want to examine the pros and cons of 'direct' transcoding of my YV12 sources to various RGB formats (UTVideo, Raw, PNG Image Sequences etc) with FFMPEG versus pre-conversion to RGB using Vapoursynth and piping out to FFMPEG for encoding. And then of course converting back to YV12 for encoding out to deliverable formats.

That's just one of the scenarios. The peculiarities of the "Progressive Segmented Frame" (PsF) AVCHD.mts format that some of my sources are recorded in and the way they are handled weigh into that evaluation. And another thing I want to look at is compression of full scale 0-255 sources to 16-235 before conversion to RGB. So there are good reasons.

I just would like to know if the above method that I use for converting YV12 to RGB with AVISynth has an equivalent in VapourSynth.

[TheFluff]

Don't process in RGB if your source is YUV. It is almost certainly avoidable. That said, the builtin resizers should do the right thing if you really do need RGB, there's no need to try to manipulate chroma sample location manually.

As an aside, I'm really curious about what peculiarities these sources of yours have.

[WorBry]

Well I didn't really want to go off on a tangent on this. But there are circumstances where it may be more desirable to pre-convert to RGB, and have control over the way that conversion is carried out.

KDenLive, for example, handles the YUV ranges of different source formats in different ways and depending on what effects are applied. In some cases the source YUV scaling is preserved, in others it is clamped or compressed (as in the case of full scale 0-255 sources) to limited 16-235 ranges. All in turn related to the way MLT, and thence FFMPEG, sets and respects the pix_format flagging. A whole subject in itself that I don't want to get into here. Suffice it to say that there are situations where using RGB input (and render) formats is expedient, especially when is it certain that the edit workflow will involve internal up-sampling to RGB and Rec709 conversion back to YUV anyway.

The reason why I use the above method for YV12<>RGB inter-conversion in AVISynth is precisely to minimize quality loss.

As for the "peculiarities" of the 1080/30PsF format that my AVCHD camcorder records in. I was merely referring to the fact that some NLE's still interpret this format incorrectly i.e. they treat the progressive YV12 chroma subsampling pattern as being interlace pattern, resulting in CUE artifacts. Also some source filters, like ffms2 (in AVISynth and VapourSynth) output the YV12 at double-frame rate (as duplicated frames) unless the correct fps is specified by the fpsden and fpsnum parameters. That's all I meant by "peculiarities"...."nuances to be aware of" might be a better term.

In answer to my question then - which is tantamount to "what is the most precise way to convert progressive YV12 to RGB32 with VapourSynth ?" - your advice is that I best stick to the "built-in resizer"? So is this referring to fmtconv, which I'll have to get my head around:

http://forum.doom9.org/showthread.php?t=166504

....or "something else" ?

Edit: So according to the fmtconv readme that would be:

YUV 4:2:0 to RGB:

c = core.fmtc.resample (clip=c, css="444")
c = core.fmtc.matrix (clip=c, mat="709", col_fam=vs.RGB)
c = core.fmtc.bitdepth (clip=c, bits=8)

and for RGB to YUV 4:2:0

c = core.fmtc.matrix (clip=c, mat="709", col_fam=vs.YUV, bits=16)
c = core.fmtc.resample (clip=c, css="420")
c = core.fmtc.bitdepth (clip=c, bits=8)

Correct ?

Edit2 - Ah, OK having tested that it appears that the only way to pipe to FFMPEG is via yuv4mpegpipe which would require conversion of the RGB back to YUV, which is clearly a redundant exercise. So I'm probably just as well using FFMPEG to transcode to RGB then or still doing it with AVISynth and vfw codecs?

What I might want to look at though is using VapourSynth for compressing full scale 0-255 YV12 sources (DSLR HD AVC clips) to 16-235 range. This is how KDenLive handles native full-scale sources, but I'm seeing a lot of luma "banding" in the MLT compressed YV12 and YUY2 renders

[Myrsloik]

You can pipe raw RGB to ffmpeg just fine. You just have to manually specify width, height, pixfmt and all the other stuff it would otherwise read from the y4m header.

You can also simply use the built in core.resize.Bicubic(format=vs.RGB24, matrix_in_s="709") to do more or less the same thing.

Just try the resizers without tricks and see if they look good enough.

[age]

The conversion of that script should be :

c = core.fmtc.resample (clip=c, css="444",kernel="point")                   #ConvertToYV24(chromaresample="point", interlaced=false)

                                                                              
r=core.fmtc.resample (clip=c,kernel="point",sw=0, sh=-1)                    #MergeChroma(PointResize(width, height, 0, 1))
c=core.std.ShufflePlanes(clips=[c,r], planes=[0, 1, 2], colorfamily=vs.YUV) #


c = core.fmtc.matrix (clip=c, mat="709", col_fam=vs.RGB)                    #ConvertToRGB32(matrix="rec709", interlaced=false)


c = core.fmtc.matrix (clip=c, mat="709", col_fam=vs.YUV)                    #
c = core.fmtc.resample (clip=c, css="420",kernel="point")                   #ConvertToYV12(chromaresample="point", interlaced=false)
c = core.fmtc.bitdepth(clip =c, bits=8)                                     #

For compressing full scale YV12 sources using expr:

bit=2**c.format.bits_per_sample-1         
c=core.std.Expr(clips=c, expr=[" {bit} x {bit} / 0.85882352941 * 1 / 0.06274509803 + * ".format(bit=bit),"  {bit} x {bit} / 0.50196078431 - 0.87843137254 * 1 / 0.50196078431 + * ".format(bit=bit),"  {bit} x {bit} / 0.50196078431 - 0.87843137254 * 1 / 0.50196078431 + * ".format(bit=bit)])

[Reel.Deel]

Originally Posted by WorBry

The reason why I use the above method for YV12<>RGB inter-conversion in AVISynth is precisely to minimize quality loss.

Might be better to stick with traditional methods:

Originally Posted by cretindesalpes

YV12 <-> RGB conversions are not lossless whatever the bitdepth, unless you’re resampling the chroma planes with a specific kernel in order to make YV12 -> RGB -> YV12 lossess, but this may reduce the overall quality of each conversion.

[WorBry]

Originally Posted by Myrsloik

You can pipe raw RGB to ffmpeg just fine. You just have to manually specify width, height, pixfmt and all the other stuff it would otherwise read from the y4m header.

OK thanks, that's good to know. Whilst searching for information I came across this post and so just assumed it was a non-starter:

http://forum.doom9.org/showpost.php?...85&postcount=5

Could you point me to any documentation that explains/shows how to do that? Trying to find pertinent information in the FFMPEG labyrinth is often like searching for a needle in a field of haystacks - a snippet here, a bug ticket there.

[WorBry]

Originally Posted by age

The conversion of that script should be :

c = core.fmtc.resample (clip=c, css="444",kernel="point")                   #ConvertToYV24(chromaresample="point", interlaced=false)

                                                                              
r=core.fmtc.resample (clip=c,kernel="point",sw=0, sh=-1)                    #MergeChroma(PointResize(width, height, 0, 1))
c=core.std.ShufflePlanes(clips=[c,r], planes=[0, 1, 2], colorfamily=vs.YUV) #


c = core.fmtc.matrix (clip=c, mat="709", col_fam=vs.RGB)                    #ConvertToRGB32(matrix="rec709", interlaced=false)


c = core.fmtc.matrix (clip=c, mat="709", col_fam=vs.YUV)                    #
c = core.fmtc.resample (clip=c, css="420",kernel="point")                   #ConvertToYV12(chromaresample="point", interlaced=false)
c = core.fmtc.bitdepth(clip =c, bits=8)                                     #

For compressing full scale YV12 sources using expr:

bit=2**c.format.bits_per_sample-1         
c=core.std.Expr(clips=c, expr=[" {bit} x {bit} / 0.85882352941 * 1 / 0.06274509803 + * ".format(bit=bit),"  {bit} x {bit} / 0.50196078431 - 0.87843137254 * 1 / 0.50196078431 + * ".format(bit=bit),"  {bit} x {bit} / 0.50196078431 - 0.87843137254 * 1 / 0.50196078431 + * ".format(bit=bit)])

Brilliant. Thanks for that.

[WorBry]

Originally Posted by Reel.Deel

Seems like contradiction in terms, at face value. Still, once I've got the RGB to FFMPEG piping sorted, it would be interesting to see if there is anything to be gained in terms of quality preservation. It's how we learn isn't it....test/evaluate.

[jackoneill]

Brilliant. Thanks for that.

Maybe brilliant, but needlessly complicated.

Don't let ffmpeg (swscale) do any format conversions. VapourSynth's resizers (zimg) or fmtconv will do it better.

As for converting to RGB and back like in your initial post:

rgb24 = core.resize.Point(yv12, format=vs.RGB24, matrix_in_s="709", chromaloc_in_s="top_left")

# rgb processing

yv12 = core.resize.Point(rgb24, format=vs.YUV420P8, matrix_s="709", chromaloc_s="top_left")

There is no need to shift the chroma like in Gavino's code because resize.Point to yv24 followed by resize.Point back to yv12 is a no-op, provided you use the same chroma location in both calls.

If your yv12 input is full range, add range_in_s="full". You don't have to convert it to limited range yv12 first. But if you must convert to limited range yv12:

yv12_limited = core.resize.Point(yv12_full, range_in_s="full", range_s="limited")

[WorBry]

Ah you're probably right. You guys are the experts on this stuff, and I wouldn't want to discard the advice given to me...again

"Sufficient today is the getting the RGB FFMPEG pipe to work thereof"

[jackoneill]

Originally Posted by WorBry

"Sufficient today is the getting the RGB FFMPEG pipe to work thereof"

You can pipe planar RGB and change the plane order for ffmpeg's "gbrp" pix_fmt:

clip = core.resize.Point(clip, format=vs.RGB24, <options>)
clip = core.std.ShufflePlanes(clip, planes=[1,2,0], colorfamily=vs.RGB)
clip.set_output()

vspipe script.py - | ffmpeg -f rawvideo -pix_fmt gbrp -s 1920x1080 -r 24 -i pipe: -vcodec utvideo utvideo.mkv

Alternatively, you can pipe packed RGB and use ffmpeg's "bgr0" pix_fmt:

clip = core.resize.Point(clip, format=vs.COMPATBGR32, <options>)
clip.set_output()

vspipe script.py - | ffmpeg -f rawvideo -pix_fmt bgr0 -s 1920x1080 -r 24 -i pipe: -vcodec utvideo utvideo.mkv

The latter should be slightly slower due to the packing step. (ShufflePlanes is practically free.)

[age]

Reading the discussion linked in first post,that script is based on the assumption that avisynth's point resize can does "lossless" yv12->yv24->yv12 conversion only shifting the chroma, at least the point resize found in fmtc is the same.
From the documentation:
"point" Nearest neighbour interpolation. Same as Avisynth’s PointResize.

I admit that I don't know very well how to use zimg,
can I ask here how to upsampling 4:2:0 to 4:4:4 with the internal resizer?

[TheFluff]

The moral of this story: Avisynth has a lot of old cruft and weird workarounds for odd problems that has gathered over the years since it hasn't had any development of note in many years and almost everything needs to be backwards compatible all the way to 2001. VS endeavors to do the Right Thing and doesn't hesitate to rethink things. Don't assume that just because you have to work around an issue in Avisynth that that same problem exists in VS (in this case, the VS resizers support custom chroma location settings).

[WorBry]

Originally Posted by jackoneill

You can pipe planar RGB and change the plane order for ffmpeg's "gbrp" pix_fmt:

clip = core.resize.Point(clip, format=vs.RGB24, <options>)
clip = core.std.ShufflePlanes(clip, planes=[1,2,0], colorfamily=vs.RGB)
clip.set_output()

vspipe script.py - | ffmpeg -f rawvideo -pix_fmt gbrp -s 1920x1080 -r 24 -i pipe: -vcodec utvideo utvideo.mkv

Alternatively, you can pipe packed RGB and use ffmpeg's "bgr0" pix_fmt:

clip = core.resize.Point(clip, format=vs.COMPATBGR32, <options>)
clip.set_output()

vspipe script.py - | ffmpeg -f rawvideo -pix_fmt bgr0 -s 1920x1080 -r 24 -i pipe: -vcodec utvideo utvideo.mkv

The latter should be slightly slower due to the packing step. (ShufflePlanes is practically free.)

Thanks for that. Much appreciated.

[WorBry]

Originally Posted by TheFluff

The moral of this story: Avisynth has a lot of old cruft and weird workarounds for odd problems that has gathered over the years since it hasn't had any development of note in many years and almost everything needs to be backwards compatible all the way to 2001. VS endeavors to do the Right Thing and doesn't hesitate to rethink things. Don't assume that just because you have to work around an issue in Avisynth that that same problem exists in VS (in this case, the VS resizers support custom chroma location settings).

LOL. Thanks. I'll bear that in mind as I haul my little row boat up onto the beach of this brave and enchanting new world.

[WorBry]

Regarding:

Originally Posted by jackoneill

You can pipe planar RGB and change the plane order for ffmpeg's "gbrp" pix_fmt:

clip = core.resize.Point(clip, format=vs.RGB24, <options>)
clip = core.std.ShufflePlanes(clip, planes=[1,2,0], colorfamily=vs.RGB)
clip.set_output()

vspipe script.py - | ffmpeg -f rawvideo -pix_fmt gbrp -s 1920x1080 -r 24 -i pipe: -vcodec utvideo utvideo.mkv

Alternatively, you can pipe packed RGB and use ffmpeg's "bgr0" pix_fmt:

clip = core.resize.Point(clip, format=vs.COMPATBGR32, <options>)
clip.set_output()

vspipe script.py - | ffmpeg -f rawvideo -pix_fmt bgr0 -s 1920x1080 -r 24 -i pipe: -vcodec utvideo utvideo.mkv

The latter should be slightly slower due to the packing step. (ShufflePlanes is practically free.)

I've had no problem running these scripts and the pipe commands with 'native' progressive HD AVC sources, but I've got an issue with AVCHD.mts clips from my Canon HFG10 camcorder which I mentioned earlier records in 1080/30PF format:

Originally Posted by WorBry

As for the "peculiarities" of the 1080/30PsF format that my AVCHD camcorder records in. I was merely referring to the fact that some NLE's still interpret this format incorrectly i.e. they treat the progressive YV12 chroma subsampling pattern as being interlace pattern, resulting in CUE artifacts. Also some source filters, like ffms2 (in AVISynth and VapourSynth) output the YV12 at double-frame rate (as duplicated frames) unless the correct fps is specified by the fpsden and fpsnum parameters. That's all I meant by "peculiarities"...."nuances to be aware of" might be a better term.

Applying the above methods for converting to RGB32 or RGB24:

import vapoursynth as vs
core = vs.get_core()
clip = core.ffms2.Source("Path...../TestHFG10.mts", fpsnum=30000,fpsden=1001)
clip = core.resize.Point(clip, format=vs.COMPATBGR32, matrix_in_s="709", chromaloc_in_s="top_left")
clip.set_output()

import vapoursynth as vs
core = vs.get_core()
clip = core.ffms2.Source("Path...../TestHFG10.mts", fpsnum=30000,fpsden=1001)
clip = core.resize.Point(clip, format=vs.RGB24, matrix_in_s="709", chromaloc_in_s="top_left")
clip = core.std.ShufflePlanes(clip, planes=[1,2,0], colorfamily=vs.RGB)
clip.set_output()

When I load the scripts in VapourSynth Editor and open Preview, the Preview screen is blank and there is the log error message:

Error getting the frame number 0:
Resize error: Resize: field-based video not supported

Running just the bare source script:

import vapoursynth as vs
core = vs.get_core()
clip  =core.ffms2.Source("Path..../TestHFG10.mts",fpsnum=30000,fpsden=1001)
clip.set_output()

The preview shows a lot of "interlace-like" and block artifacts:

http://i.imgur.com/VlRyN8l.jpg

I recall having similar issues, although not as severe as this, when using ffms2 in AVISynth, and I switched to using L-Smash for a short while before DGDecIM arrived on the scene.

So, unless there's remedy for this, a better option might be to try L-Smash source? Unfortunately, the "extra plugins" package that came with VapourSynth and other required packages from here:

https://launchpad.net/~djcj/+archive/ubuntu/vapoursynth

....didn't include the L-Smash plugin.

I'd be really grateful if someone could advise me where to source and how to install the L-Smash plugin on my system.

As I mentioned in the first post I'm running Kubuntu 15.10 (64 AMD) and now also Mint KDE 17.3 (64 AMD). In both VapourSynth installations the plugins got placed in /usr/lib/x86_64-linux-gnu/vapoursynth

Cheers.

[TheFluff]

You didn't post a video sample so my first recommendation would be to 1) read and understand error messages and 2) field-separate it and see if it's actually interlaced or not (it might be MBAFF for all I know). If it is interlaced then well, ya gotta deinterlace it. If it's not, figure out how to weave it properly, I guess, because FFMS2 evidently figures it to be field-based and outputs it as such.