What is wrong with this YUV => RGB => YUV roundtrip ?

[poisondeathray]

Is 16bits not enough ? Or did I make simple error in testing diff? (I made sure to change the MakeDiff inputs when doing other tests), or other user error I hope

Let's say the YUV420P10 is the "source" to test against. 8bit RGB test image is here .

https://www.mediafire.com/file/l57xqaol4lwk4f5/testchart720.png
or
http://www99.zippyshare.com/v/IVqs58H0/file.html


1) If the "RGB intermediate" step uses RGBH or RGBS instead of RGB48 - it works, no difference. This would suggest 16bit not enough. But supposedly 12bit would be enough for 10bit starting point

2) No chroma subsampling works (ie. if you test YUV444P10 => RGB48 => YUV444P10). Even with RGB30 instead of RGB48 works (10bit444 YUV => 10bit RGB => 10bit444 YUV) , which is unexpected. Granted this are very limited test values, but something smells fishy here; I would have expected some error at the same bit depth

3) Testing up/down scaling chroma with point/nearest neighbor alone works as expected (ie. YUV444P10 <=> YUV420P10 works with Point. Or other bitdepths work too)

4) Since individually , duplicating/discarding chroma samples , and YUV444 <=> RGB seems to work, I thought how about splitting out the processes individually into steps instead of 1 command?
(ie. chroma upsample to YUV444P10 => RGB48 => YUV444P10 => YUV420P10) .

But it actually gives worse results . And various combinations of splitting steps gives worse results

5) fmtconv gives similar results (but slightly different diff results)

6) Is there anyway to export valid RGBH or RGBS from vpy, vspipe ? IM plugin doesn't seem to preserve when exporting to EXR (int), unless there was an issue with linearizing with transfer_in_s and transfer_s with srgb and linear




clip = core.lsmas.LWLibavSource(r'PATH\testchart720.png')
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, matrix_s="709")

clip2 = core.resize.Point(clip, format=vs.RGB48, matrix_in_s="709", dither_type="none", range_in_s="full")
clip3 = core.resize.Point(clip2, format=vs.YUV420P10, matrix_s="709", dither_type="none", range_s="full")
clip3 = core.std.SetFrameProp(clip3, prop="_ColorRange", intval=1) # Mark video as limited range.

d = core.std.MakeDiff(clip, clip3)
da = core.std.Levels(d, min_in=511, max_in=513, gamma=1, min_out=0, max_out=1023, planes=[0,1,2])

da.set_output()

[feisty2]

Is there anyway to export valid RGBH or RGBS from vpy, vspipe ?
sure, u can output the raw binary of any pixel type with vspipe, the real problem is how u gonna load it after that
I hacked vsrawsource for GrayS support like eons ago, you could load the binary with that in vs, I didn't make the extra support for RGBS, cuz I always do my thing on luminance

to export the binary:
vspipe xxx.vpy Sequence.bin -p
to import the binary back in vs:
clp = core.raws.Source("Sequence.bin", width, height, src_fmt="GRAYS")
load the binary as a numpy array for research and academic stuff:
def LoadRawBinaryGrayscaleSequence(Address, Width, Height, Length, PixelType='float32'):
Sequence = numpy.memmap(Address, dtype=PixelType, mode='r', shape=(Length, Height, Width))
Sequence = numpy.array(Sequence, dtype='float64')
return Sequence

Sequence = LoadRawBinaryGrayscaleSequence("Sequence.bin", Width, Height, Length)

[feisty2]

and u can of course always split ur RGBS stuff into 3 GrayS clips

[poisondeathray]

thanks f2, but I meant for use in other programs . A common reason to go into RGB land is for other completing operations in other programs . So standardized format is the most likely candidate . EXR should be the ticket - it supports 32bit float - but there is problem with IM exporter in keeping that

Did you look at the other problem ? In theory, that roundtrip should be possible with only 12bit RGB - so is it a problem with zimg, or some other issue (like hopefully user error)

[feisty2]

numpy is one of the "other programs", and there're maybe millions of python based research libraries that take numpy arrays as the input
I say numpy array is a standardized format

[poisondeathray]

yeah ok...you got me there

But I don't mean geeky:D scientific programs, I mean grading or compositing programs like resolve, nuke, ae . Or even video editors like pp, mc

[poisondeathray]

Or if there is one of those million "bumpy numpy" :D programs that can take that input and output a valid EXR ?

Maybe openimageio , which has python support
https://github.com/OpenImageIO/oiio

I started playing with an oiiotool binary that someone else compiled for Win but can't get it to accept pipe
http://www.nico-rehberg.de/tools.html

[feisty2]

https://github.com/jamesbowman/openexrpython/blob/master/intro.rst

[poisondeathray]

It would be nice if it was integrated in vapoursynth (hint hint :) )




But I think I found a temp workaround . You can export 16bit int like a PNG with the IM exporter (or probably pipe to ffmpeg but I didn't test specifically) . Reimport with IM . Then convert to RGBS, and continue as before and it works (at least on this limited test) . Although the IM importer "says" it's RGBS, it's actually int . vspipe --info at that stage even says it's RGBS (but it's not), so that explicit step is necessary, then it works

Some bizzare and unexpected observations findings though ....

[feisty2]

It would be nice if it was integrated in vapoursynth (hint hint :) )

theoretically you should be able to map a GrayS or RGBS vaporsynth video clip to a 4D numpy tensor (array of (FrameCount, Height, Width, ChannelCount)) without going through vspipe, Myrsloik said it was possible, and I'm not sure how,,, then you use openexrpython to make ur numpy tensor an EXR sequence

[poisondeathray]

theoretically you should be able to map a GrayS or RGBS vaporsynth video clip to a 4D numpy tensor (array of (FrameCount, Height, Width, ChannelCount)) without going through vspipe, Myrsloik said it was possible, and I'm not sure how,,, then you use openexrpython to make ur numpy tensor an EXR sequence

If you're not sure how, then I have about a 0.00000% chance figuring it out. But it sure sounds cool :cool:

[feisty2]

If you're not sure how, then I have about a 0.00000% chance figuring it out. But it sure sounds cool :cool:

u could ask Myrsloik to do something for conversions between video clips and numpy arrays maybe, I did it a while ago and he didn't seem to be very interested, maybe he will be more motivated if there're 2 requesting the same thing :)

[poisondeathray]

u could ask Myrsloik to do something for conversions between video clips and numpy arrays maybe, I did it a while ago and he didn't seem to be very interested, maybe he will be more motivated if there're 2 requesting the same thing :)

A lot of that scientific testing stuff is above my head. It would be more convincing if I actually understood that stuff - then you could make a more convincing argument. I'm just a simple user

So I'm like adding a 0.1 voice :D. So that's 1.1 people requesting :D

[poisondeathray]

But if you can demonstrate how numpy arrays can achieve results, like a valid float EXR , then I'll cheat and vote twice :)

That's what I'm interested in - results. It's just the actual path and details in between that is a bit murky for me

[AzraelNewtype]

The cause of the round-trip failure is the initial statement to convert to YUV 4:2:0. The reason a 4:4:4 version of the test chart is able to round-trip is that it contains only a limited number of shades with rounding-friendly U/V characteristics. The bicubic filter introduces ringing along the test pattern lines and therefore no longer round-trips. Using a floating-point intermediate clip reduces the number of rounding stages, as the colorspace operations are implemented in floating-point. There is no guarantee in z.lib that any operation has a bit-exact round-trip, except for point sampling and certain bit depth conversions.

I have nothing to add but he's going to delete this.

[poisondeathray]

I have nothing to add but he's going to delete this.


Originally Posted by Stephen R. Savage View Post
The cause of the round-trip failure is the initial statement to convert to YUV 4:2:0. The reason a 4:4:4 version of the test chart is able to round-trip is that it contains only a limited number of shades with rounding-friendly U/V characteristics. The bicubic filter introduces ringing along the test pattern lines and therefore no longer round-trips. Using a floating-point intermediate clip reduces the number of rounding stages, as the colorspace operations are implemented in floating-point. There is no guarantee in z.lib that any operation has a bit-exact round-trip, except for point sampling and certain bit depth conversions.





Thanks to both of you

Where does the "conventional wisdom" that you can round trip with +2 bits go ? Shouldn't 16bits be enough for 8 or 10bit sources ? Is that only possible with for limited situations like synthetic test patterns that are "perfect" with limited colors? What if you start with a "non perfect" 444 source? You can have native recorded 444 footage from cameras with aliasing, ringing, noise. Even raytraced CG renders can have those issues

[feisty2]

Thanks to both of you

Where does the "conventional wisdom" that you can round trip with +2 bits go ? Shouldn't 16bits be enough for 8 or 10bit sources ? Is that only possible with for limited situations like synthetic test patterns that are "perfect" with limited colors? What if you start with a "non perfect" 444 source? You can have native recorded 444 footage from cameras with aliasing, ringing, noise. Even raytraced CG renders can have those issues

depending on the matrix u use to convert between RGB and YUV, an integer matrix like YCgCo is always lossless with 2 extra bits, and the widely used matrixes (I know "matrices" is the standard plural form but I fucking hate irregular shit) Rec601 and Rec709 r floating point matrixes, and such arithmetic operations will introduce precision loss (rounding errors) cuz the precision of floating point values is limited, but that's not the end of the story

note ur untouched 8-bit YUV444 clip as A
the clip gotta be converted to floating points before the conversion to RGB cuz the matrix, Rec601 or 709 or whatever has floating point values, note the floating point version of ur clip just before any actually conversion as B
then u convert B to RGB, call the result C
then u convert C back to YUV444 with the inverse matrix, call the result D
then u convert D, a floating point clip back to an integer clip, call the result E

now, A and E are uint8 clips, B, C, D r floating point clips, mathematically speaking, B and D should be the exact same, but they r not cuz floating point arithmetic introduced some precision loss, and...uh, here comes the fun part
E is converted from D, and u got many options to do that, and if E is directly rounded down from D, A and E will be the exact same!

concretely, like (I actually made up all the values just to show u how it works)
A: 1, 1, 1
B: 1.0, 1.0, 1.0
C: 4.2, 5.5, 0.1
D: 0.997, 1.001, 0.994
E: 1, 1, 1

[poisondeathray]

Ok thanks got it - Rec floating point "matrices/matrixes :D"; But my impression was some math models predicted +2 bits was enough , even using the standard "Rec" matrices . Not talking about YCgCo or reversible transforms. Have I misunderstood or misread what is being said ?

It's just when some people say 16bit is "overkill" blah blah... BUT the specifics and steps in between are important. As they say "The devil :devil: is in the details"

But it's nice to be able to actually "concretely" do it with something (as in actual files, actual workflow), even if you need RGBS in vpy.

[feisty2]

major point: precision loss introduced by floating point arithmetic will be magically canceled out by rounding back to integers,,,
now I wanna talk a bit more about that 2 extra bits, say ur conversion trip is RGB->YUV444>RGB, and the intermediate YUV444 result was not stored in floating point but rounded down to an integer representation, the aforementioned magic sticks around long as that intermediate result has at least 2 bits (theoretically at least 1 more bit with a cherry picked matrix) more precision than that RGB clip

[poisondeathray]

major point: precision loss introduced by floating point arithmetic will be magically canceled out by rounding back to integers,,,
now I wanna talk a bit more about that 2 extra bits, say ur conversion trip is RGB->YUV444>RGB, and the intermediate YUV444 result was not stored in floating point but rounded down to an integer representation, the aforementioned magic sticks around long as that intermediate result has at least 2 bits (theoretically at least 1 more bit with a cherry picked matrix) more precision than that RGB clip

The other direction should be "easier" , since YUV covers all of RGB at the same bit depth if looking at the color cube representation. But the opposite way produces illegal negative results

[poisondeathray]

I could swear there was another post by S.Savage just right now. But POOF it vanished . Not ninja quick enough to quote. Reflexes too slow.

[poisondeathray]

I looked into this a bit more carefully. What is actually happening is unrelated to numerical precision, as is shown by the results with a RGBH intermediate. The problem is that the subsampling operation when creating the YUV version of the test pattern causes ringing that maps to out-of-range RGB values. As a result, when you store the intermediate RGB in an integer format, it gets clipped, causing a relatively high degree of error in the round-trip. You can see this by storing the intermediate RGB as a limited-range clip, which allows some headroom for out-of-range values and reduces the number of places with round-trip errors.

See this thread for information about invalid YUV values: http://forum.doom9.org/showthread.php?t=154731



Yes, that thread comes up often, again and again.

"The problem is that the subsampling operation when creating the YUV version of the test pattern causes ringing that maps to out-of-range RGB values." - this I can understand, you have "illegal" values, negative RGB values when solving. But +2 higher bit depth for the "standard" matrices was supposed to solve that, irrespective of full. vs. limited range ...

Also those observations suggest "clipping" cannot be the entire story, because reimporting that RGB48 integer intermediate (I tested with physical files) then converting to RGBS/H fixes everything. If it was clipped, those values would have been discarded. Granted this is a limited test scenario with limited colors only...



What is actually happening is unrelated to numerical precision, as is shown by the results with a RGBH intermediate.

But doesn't that suggest the opposite, that it IS related to precision - because the RGBS/H has higher precision, than say RGB30 or RGB48. All the math and operations done while in RGBS/H have higher precision don't they ?

[feisty2]

seriously, u should stop using bullshit like RGB or YUV (with or without chroma subsampling) if u could do ur thing with floating point precision
CIE colorspaces r a million times more mathematically exquisite than lesser beings like RGB or YUV
take CIE XYZ if u want an RGB-like (LMS (https://en.wikipedia.org/wiki/LMS_color_space) based) colorspace, take CIE Lab if u wanna YUV-like (luminance-chrominance) colorspace

[poisondeathray]

seriously, u should stop using bullshit like RGB or YUV (with or without chroma subsampling) if u could do ur thing with floating point precision
CIE colorspaces r a million times more mathematically exquisite than lesser beings like RGB or YUV
take CIE XYZ if u want an RGB-like (LMS (https://en.wikipedia.org/wiki/LMS_color_space) based) colorspace, take CIE Lab if u wanna YUV-like (luminance-chrominance) colorspace


Sure, in your fairy world :D :devil:

But I live in reality, with real files, real programs , real workflows that use real conventional color models like YUV and RGB.

And most of the time, it actually doesn't matter if there is a bit of error on the trip, this is just academic

[feisty2]

Sure, in your fairy world :D :devil:

But I live in reality, with real files, real programs , real workflows that use real conventional color models like YUV and RGB.

And most of the time, it actually doesn't matter if there is a bit of error on the trip, this is just academic

photoshop supports CIE Lab if I remembered correctly, u could trick the program anyways, tag ur XYZ images as RGB and Lab as YUV444, ur good long as u don't color grade them

[poisondeathray]

photoshop supports CIE Lab if I remembered correctly, u could trick the program anyways, tag ur XYZ images as RGB and Lab as YUV444, ur good long as u don't color grade them

Yes, many post production programs support CIE Lab. But it' s not common for usage

Increasingly common now is ACES .

[feisty2]

Yes, many post production programs support CIE Lab. But it' s not common for usage

Increasingly common now is ACES .

I did some googling and it seems like an LMS based color space, so a competitor to CIE XYZ, not exactly comparable to CIE Lab,,, anyways, u should pick one of these colorspaces, RGB and YUV r a huge mess, just take a look at how many different kinds of RGB and YUV colorspaces we got out there, they suck ass cuz they r hardware-dependent, which is the main reason they have soooooooo many variants

all images and videos should be stored in a standard hardware independent colorspace like CIE XYZ or CIE Lab or ACES
the display device should take the standard CIE or ACES signal as the input and convert the signal to its own hardware dependent RGB colorspace implicitly within, and the whole colorspace thing would stay transparent to the user and any software

[TheFluff]

I could swear there was another post by S.Savage just right now. But POOF it vanished . Not ninja quick enough to quote. Reflexes too slow.

He's paranoid (or maybe just obnoxious, it's sometimes very hard to tell what is just his internet persona speaking) and deletes his own posts after a few hours. You need to quote them immediately when you see them - there's usually good info in there that's worth keeping around.

feisty2, when you grow up you will hopefully realize that regardless of your opinions (technical, political or otherwise), you have to live in the world we actually have, not the one you wish we lived in. There are many things that are bad and that we can attempt to improve, but for purely practical reasons we cannot turn reality on its head overnight. (This should not be misconstrued as a defense of cynicism. We can make the world better, but FULL COMMUNISM NOW is not a practically possible option, for example.)

[poisondeathray]

deletes his own posts after a few hours. You need to quote them immediately when you see them - there's usually good info in there that's worth keeping around.

Yes I know...

And f2 was no help in ninja quoting even with those "youthful" reflexes (What? not enough video game practice ? :D click click click)





More nuggets please!

I'm just trying to make sense of the world

[poisondeathray]

Also those observations suggest "clipping" cannot be the entire story, because reimporting that RGB48 integer intermediate (I tested with physical files) then converting to RGBS/H fixes everything. If it was clipped, those values would have been discarded. Granted this is a limited test scenario with limited colors only...




This is wrong - I double checked. This part was user error

I used the wrong "input" clip when converting to RGBS .

So there is no workaround :(



So I'm back to : how to export a valid RGBS/H into a physical file format for use in another program, or pipe into something that can.

And I am still questioning the status of the "magical" +2 sufficiency, when +6 or 8 bits couldn't preserve all values, or at least when using this method

[TheFluff]

And I am still questioning the status of the "magical" +2 sufficiency, when +6 or 8 bits couldn't preserve all values, or at least when using this method

I think you're overthinking it. If we start out with an 8-bit clip we have 256 distinct possible input values. The arithmetic we want to do can produce many other distinct values, so we have to round the results, which may end up giving us values that are off by +/-1 compared to the originals. If we do the arithmetic with one more bit of precision, we now have 512 distinct values instead, but the new ones are exactly halfway in between the old ones, which is ambiguous. We cannot with certainty tell if the 9-bit result 257 should be rounded down to 128 or up to 129 when converting back to 8-bit.

However, if we have two more bits of precision (1024 distinct values), we're fine. If we started out with 128 in 8-bit, scaled it to 512, did our arithmetic and got a result that ended up getting rounded to 513, that's no problem - it'll get scaled back to 128.

The issues you're having here are not actually related to numeric precision, as Steven said. Read his post again.

[poisondeathray]

I think you're overthinking it. If we start out with an 8-bit clip we have 256 distinct possible input values. The arithmetic we want to do can produce many other distinct values, so we have to round the results, which may end up giving us values that are off by +/-1 compared to the originals. If we do the arithmetic with one more bit of precision, we now have 512 distinct values instead, but the new ones are exactly halfway in between the old ones, which is ambiguous. We cannot with certainty tell if the 9-bit result 257 should be rounded down to 128 or up to 129 when converting back to 8-bit.

However, if we have two more bits of precision (1024 distinct values), we're fine. If we started out with 128 in 8-bit, scaled it to 512, did our arithmetic and got a result that ended up getting rounded to 513, that's no problem - it'll get scaled back to 128.

The issues you're having here are not actually related to numeric precision, as Steven said. Read his post again.


This makes sense, thanks

Higher bit depths, increased precision won't cover those values that don't have a valid representation in RGB , for this direction (RGB=>YUV=>RGB should be able to be covered)

So basically it's not possible using "standard" matrices. Unless you use RGBS/H to store those invalid values . (Again, I'm leaving out YCgCo or the other various lossless transforms out of the discussion for now.)


So I'm back to : how to export a valid RGBS/H into a physical file format for use in another program, or pipe into something that can. :D

[feisty2]

feisty2, when you grow up you will hopefully realize that regardless of your opinions (technical, political or otherwise), you have to live in the world we actually have, not the one you wish we lived in. There are many things that are bad and that we can attempt to improve, but for purely practical reasons we cannot turn reality on its head overnight. (This should not be misconstrued as a defense of cynicism. We can make the world better, but FULL COMMUNISM NOW is not a practically possible option, for example.)

I wasn't talking about imaginary stuff in my head, CIE colorspaces are real things that exist now in this world, it's a matter of choice if u wanna use them or go with low level hardware dependent basic bitches like RGB or YUV

[feisty2]

And I am still questioning the status of the "magical" +2 sufficiency, when +6 or 8 bits couldn't preserve all values, or at least when using this method

I think ur misunderstanding the prerequisites of that 2 more bits magic, it works for:
RGB->YUV444->RGB
the point is 2 bits more precision is enough to cancel out errors introduced by 2 matrix multiplications
now u wanna bring resampling to the game and that's so much more than just 2 matrix multiplications, sure shit happens cuz u broke the prerequisites

[poisondeathray]

Yes, thanks for the clarification . We don't wanna gosh darn broke dem stuff do we ? :)



But even if you could export some RGBS format from vapoursynth... there is no way of currently importing it back in is there? IM plugin currently at 16bit (even though it will report RGBS) , just like ffmpeg ?

[feisty2]

But even if you could export some RGBS format from vapoursynth... there is no way of currently importing it back in is there? IM plugin currently at 16bit (even though it will report RGBS) , just like ffmpeg ?

u can, just output the raw binary via vspipe and load it with vsrawsource, assuming "back in" = "back in vaporsynth"

[poisondeathray]

RGBS isn't listed on the format list for vsrawsource . Is there another version ?
https://github.com/chikuzen/vsrawsource/blob/master/format_list.txt

[feisty2]

RGBS isn't listed on the format list for vsrawsource . Is there another version ?
https://github.com/chikuzen/vsrawsource/blob/master/format_list.txt

my hacked version (https://github.com/IFeelBloated/vsrawsource/releases) has support for GrayS

should work for RGBS with some tricks
output (wrap RGBS as GrayS x3):

r = core.std.ShufflePlanes(clp, 0, vs.GRAY)
g = core.std.ShufflePlanes(clp, 1, vs.GRAY)
b = core.std.ShufflePlanes(clp, 2, vs.GRAY)
clp = core.std.Interleave([r,g,b])


input (reassemble RGB channels):

r = core.std.SelectEvery(clp, 3, 0)
g = core.std.SelectEvery(clp, 3, 1)
b = core.std.SelectEvery(clp, 3, 2)
clp = core.std.ShufflePlanes([r,g,b], 0, vs.RGB)

[poisondeathray]

Thanks, I'll test it out. But like you said earlier, the other problem is getting that workaround imported into other programs, let alone exporting something that vapoursynth can handle in 32bit float

So apparently there is no way of currently importing other 32bit float formats like EXR properly (or even half ? )

(It's not just for little YUV=>RGB=>YUV test , EXR is probably the most common format for VFX post production, CG work)

[feisty2]

Thanks, I'll test it out. But like you said earlier, the other problem is getting that workaround imported into other programs, let alone exporting something that vapoursynth can handle in 32bit float

So apparently there is no way of currently importing other 32bit float formats like EXR properly (or even half ? )

(It's not just for little YUV=>RGB=>YUV test , EXR is probably the most common format for VFX post production, CG work)

I guess raw binary (especially grayscale raw binary, cuz color binaries r ambiguous, they could either be packed or planar) should be the universal solution here, vaporsynth recognizes it, numpy recognizes it, and I think production programs might as well recognize it...

if u insist on EXR sequence, ur best shot is EXR <-> numpy tensor <-> vaporsynth

[poisondeathray]

I guess raw binary (especially grayscale raw binary, cuz color binaries r ambiguous, they could either be packed or planar) should be the universal solution here, vaporsynth recognizes it, numpy recognizes it, and I think production programs might as well recognize it...

if u insist on EXR sequence, ur best shot is EXR <-> numpy tensor <-> vaporsynth



It's not that I "insist" on EXR ; I'll use whatever works :)

The raw binary isn't really usable in other (non scientific) programs as-is . I'll keep on playing with it to see if I can get it working

EXR is clearly the "best" open format for high bit depth / visual FX production workflows. Large support from both commerical and open source programs. I don't think there is anything else remotely close in terms of functionality, compatibility/interoperability, standardization . I don't think I have to "sell" this - it's pretty much fact.

I'm exploring if vapoursynth can be used in parts of certain workflows, and it definitely can. If it could import/export EXR properly (not necessarily all the features, just the float/half, ability to read 1 layer with the common compression formats used is good enough) it would be a big plus.

My understanding is that there are IM variants that can be compiled with HDR/ 32bit float support. Not sure on the status of that

I think OpenImageIO might be a reasonable alternative to IM, but I don't know enough about python bindings or the API to determine if it's suitable . For the "open" programs like blender and natron, I think that's how they navigate through this situtation. The ffmpeg libraries only work at 16bit int.

[feisty2]

Any luck with numpy tensor <-> EXR?

[poisondeathray]

Any luck with numpy tensor <-> EXR?

I haven't had a good crack at it yet, but I'm pretty much a python newbie. So I don't have my hopes up

You could probably figure it out 10x faster ... (hint hint :D )