Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
11th January 2022, 12:08 | #1761 | Link |
Registered User
Join Date: Jul 2018
Posts: 1,067
|
It is good to add to wiki about RGB and YUV formats (http://avisynth.nl/index.php/Color_spaces ): the digital YUV colorspace is more limited in precision in compare with digital RGB colorspace. So for comparable quality it is about RGB 8bit and YUV 10bit. One consequence mean using YUV8 (YUV 4xx 8bit, typical storage) stage for converting to RGB8 (RGB24, typical display) cause more digital distortions (banding).
YUV cover 'wider' colour gamut in compare with RGB but this cause precision lost with 'natural' colour gamut with same storage integer bits per value. So the conversions RGB<->YUV with same bitdepth are not lossless even with YUV444. The error is about 1 LSB but it visible on smooth colour gradients with 8bit. At first I think it is an issue in Convert() functions. May be it is already somewhere covered in AVS wiki ? Last edited by DTL; 11th January 2022 at 12:20. |
11th January 2022, 22:41 | #1762 | Link |
German doom9/Gleitz SuMo
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,781
|
Your conclusion is not completely correct. The precision reduction has two possible reasons: "TV range" (but full range YUV exists too) and float arithmetics in the YCbCr conversion formulas (avoidable in the YCgCo model). If you don't need to convert between RGB and YUV, then full range YUV has the same precision of 8 bits per component as RGB. Thus, YUV is not generally worse; it is only worse for RGB based devices.
|
12th January 2022, 01:17 | #1763 | Link |
Registered User
Join Date: Jul 2018
Posts: 1,067
|
""TV range" (but full range YUV exists too)"
Here is the test: taking RGB $808080 something around mid-grey colour (no-colour grey) and slowly increase saturation of green Code:
BlankClip(color=$FF808080) ConvertToYUV444(matrix="PC.601") ConvertToRGB32(matrix="PC.601") ScriptClip ("""Subtitle (String (AverageR()) + " " + String (AverageG()) +" "+ String (AverageB()))""") Input $FF808180 - out 129 129 129 < ??? - luma raised, no green colour = error in colour tone and colour saturatrion Input $FF808280 - out 128 130 127 < ??? - green colour + some minus-blue colour = error in colour tone Input $FF808380 - out 129 131 128 < ??? - green colour + some plus-red colour = error in colour tone The only known way to fix - use >8 bit processing of RGB->YUV->RGB: Code:
BlankClip(color=$FF808080) ConvertBits(16) ConvertToYUV444(matrix="PC.601") ConvertToRGB32(matrix="PC.601") ConvertBits(8) ScriptClip ("""Subtitle (String (AverageR()) + " " + String (AverageG()) +" "+ String (AverageB()))""") Input $FF808180 - out 128 129 128 = OK Input $FF808280 - out 128 130 128 = OK Input $FF808380 - out 128 131 128 = OK Last edited by DTL; 12th January 2022 at 01:25. |
12th January 2022, 02:54 | #1764 | Link | |
Registered User
Join Date: Sep 2007
Posts: 5,374
|
Quote:
That is the expected result for 8bit RGB=>YUV=>RGB round trip. +/- 3 value errors Not just precision and 8bit rounding errors, but multiple 8bit YUV values can map to the same RGB value, which produces more "banding" instead of smooth gradient In general, 10bit YUV or greater is required for lossless round trip from 8bit RGB => YUV => 8bit RGB, if implemented properly. But you can demonstrate cases where 10bit is not sufficient with internal Convert(), yet 10bit is sufficient with AvsResize/zimg. ( 10bit also sufficient with other programs like NLE's) . In this example, 12bit with internal convert is required , but 10bit with AvsResize/zimg or other programs works ok https://forum.doom9.org/showthread.p...86#post1897686 This is pinterf's explanation for the difference using internal convert https://forum.doom9.org/showthread.p...83#post1902783 |
|
12th January 2022, 08:41 | #1765 | Link |
German doom9/Gleitz SuMo
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,781
|
(matrix="PC.601") ... please read: Wikipedia: YUV, specifically chapters 2.1: SDTV with BT.470 (oh, look, fractional matrix coefficients with 3 decimal places) and 3: Numerical approximations (okay, small integers here, but with rounding errors, and they cause most of the banding, especially because they need to be applied forth and back).
A YCoCg matrix has much simpler coefficients, halves and quarters, nice to handle in binary form. |
12th January 2022, 14:51 | #1766 | Link |
Registered User
Join Date: Jul 2018
Posts: 1,067
|
"3: Numerical approximations (okay, small integers here, but with rounding errors, and they cause most of the banding, especially because they need to be applied forth and back)."
May be good to copy to Avisynth wiki (and may be Convert() functions note). May be possible to develop some 'pre-distortions' for 8bit format conversions to minimize colour tone shift (that is more visible ?) ? Currently the distribution of rounding errors is equal for colour tone, luma and saturation errors ? I see typical solutions is add dithering. Unfortunately 8bit still widely used and in end-users displays it is the only supported with h.264 encoding. So using of h.265 with 10bit decreases number of possible users of encoded content. |
14th January 2022, 08:39 | #1767 | Link |
Registered User
Join Date: Jan 2014
Posts: 2,314
|
Avisynth 3.7.2 test 1
Avisynth+ 3.7.2 test 1 (20220113) Code:
20220113 3.7.2-WIP ------------------ - Fix: Attempt to resolve deadlock when an Eval'd (Prefetch inside) Clip result is used in Invoke which calls a filter with GetFrame in its constructor. (AvsPMod use case which Invokes frame prop read / ConvertToRGB32 after having the AVS script evaluated) Remark: problem emerged in 3.7.1test22 which is trying to read frame properties of the 0th frame in its constructor. A similar deadlock situation was already fixed earlier in Neo branch and had been backported but it did not cover this use case. - Fix: Histogram AudioLevels half character upshift (regression since v3.6) - Bump Copyright year to 2022 |
14th January 2022, 11:06 | #1768 | Link | |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,904
|
Nice one, Ferenc!
But... it doesn't work. Quote:
With 3.7.1 Stable: And I had to get rid of frame properties to make it work: With 3.7.2 Test 1 it's the same thing: and once I get rid of frame properties... Source: Code:
General Complete name : \\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf Format : MXF Commercial name : XDCAM HD422 Format version : 1.3 Format profile : OP-1a Format settings : Closed / Complete File size : 5.50 GiB Duration : 14 min 58 s Overall bit rate : 52.6 Mb/s Package name : Source Package Encoded date : 2022-01-12 10:16:12.076 Writing application : Avid Technology, Inc. Avid MediaProcessor Plug-In 1.0.54.10052.1 Writing library : MXF::SDK (4.7.8) on Win64 4.7.8.10137.1 Video ID : 512 Format : MPEG Video Commercial name : XDCAM HD422 Format version : Version 2 Format profile : 4:2:2@High Format settings : CustomMatrix / BVOP Format settings, BVOP : Yes Format settings, Matrix : Custom Format settings, GOP : Variable Format settings, picture structure : Frame Format settings, wrapping mode : Frame Codec ID : 0D01030102046001-0401020201040300 Duration : 14 min 58 s Bit rate mode : Constant Bit rate : 50.0 Mb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate : 25.000 FPS Standard : PAL Color space : YUV Chroma subsampling : 4:2:2 Bit depth : 8 bits Scan type : Interlaced Scan order : Top Field First Compression mode : Lossy Bits/(Pixel*Frame) : 0.965 Time code of first frame : 00:00:00:00 Time code source : Group of pictures header Stream size : 5.23 GiB (95%) Color range : Limited Color primaries : BT.709 Transfer characteristics : BT.709 Matrix coefficients : BT.709 Audio #1 ID : 768 Format : PCM Format settings : Little Format settings, wrapping mode : Frame (AES) Codec ID : 0D01030102060300-0402020101000000 Duration : 14 min 58 s Bit rate mode : Constant Bit rate : 1 152 kb/s Channel(s) : 1 channel Sampling rate : 48.0 kHz Frame rate : 25.000 FPS (1920 SPF) Bit depth : 24 bits Stream size : 123 MiB (2%) Locked : Yes Audio #2 ID : 1024 Format : PCM Format settings : Little Format settings, wrapping mode : Frame (AES) Codec ID : 0D01030102060300-0402020101000000 Duration : 14 min 58 s Bit rate mode : Constant Bit rate : 1 152 kb/s Channel(s) : 1 channel Sampling rate : 48.0 kHz Frame rate : 25.000 FPS (1920 SPF) Bit depth : 24 bits Stream size : 123 MiB (2%) Locked : Yes Other #1 ID : 1-Material Type : Time code Format : MXF TC Frame rate : 25.000 FPS Time code of first frame : 10:00:00:00 Time code settings : Material Package Time code, striped : Yes Title : Timecode Other #2 ID : 0-Source Type : Time code Format : MXF TC Frame rate : 25.000 FPS Time code of first frame : 10:00:00:00 Time code settings : Source Package Time code, striped : Yes Other #3 Type : Time code Format : SMPTE TC Muxing mode : SDTI Frame rate : 25.000 FPS Time code of first frame : 10:00:00:00 Using VirtualDub, it works, regardless of frame properties: |
|
14th January 2022, 11:29 | #1769 | Link |
Registered User
Join Date: Jan 2014
Posts: 2,314
|
This issue must be different, there is no Prefetch here.
EDIT: Can you put a ConvertToRGB32() at end of the script? (to check if it fails; AvsPMod (as far as I saw in the source) does two additional things after evaluating our original AVS script: reads frame properties and converts the clip to rgb32 in order to display it. My fix was intended to heal only the "hang" issue. Last edited by pinterf; 14th January 2022 at 14:33. |
14th January 2022, 17:41 | #1770 | Link |
Registered User
Join Date: Oct 2018
Location: Germany
Posts: 999
|
As soon as FranceBB has it in his hands, he breaks it.
Try this version, no problems for me. 32bit: https://www.mediafire.com/file/ar8qp...6-32).zip/file 64bit: https://www.mediafire.com/file/uvtp8...6-64).zip/file
__________________
Live and let live |
14th January 2022, 18:25 | #1771 | Link | |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,904
|
Quote:
Same error as before. Please note that this happens only with XDCAM files muxed in .mxf I can share a sample if you want, 'cause I've just tried with a ProRes file muxed in .mov and it works just fine. Same goes for an H.264 file in .mp4. eheheheh nah, it's more like: "literally no one except FranceBB uses weird FULL HD yv16 50 Mbit/s 25i TFF MPEG-2 files muxed in mxf anymore, so no one ever checked" xD Interesting error, it fails with this: however this is a bit weird given that it's a simple yv16 with the standard MPEG-2 chroma placement (after all it IS an MPEG-2 eheheheh). Adding propclearall() fixes it and allows me to use ConverttoRGB32(): Code:
video=LWLibavVideoSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf") ch1=LWLibavAudioSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf", stream_index=1) ch2=LWLibavAudioSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf", stream_index=2) audio=MergeChannels(ch1, ch2, ch1, ch2, ch1, ch2, ch1, ch2) AudioDub(video, audio) propClearAll() ConverttoRGB32() Code:
video=LWLibavVideoSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf") ch1=LWLibavAudioSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf", stream_index=1) ch2=LWLibavAudioSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf", stream_index=2) audio=MergeChannels(ch1, ch2, ch1, ch2, ch1, ch2, ch1, ch2) AudioDub(video, audio) propDelete("_ChromaLocation") Success! Getting rid of the Chroma Location property makes it work! So now we know what happened! Basically ConverttoRGB32() fails due to the Chroma Location frame property, therefore when I try to use AVSPmod, it tries to convert to RGB32 invoking such a function, but since such a function fails, it won't display anything and fail instead!! |
|
14th January 2022, 18:50 | #1773 | Link | |
Registered User
Join Date: Oct 2018
Location: Germany
Posts: 999
|
Quote:
But the frame properties (matrix) are read from the original source clip.
__________________
Live and let live |
|
14th January 2022, 18:55 | #1774 | Link | |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,904
|
Quote:
Looks like the indexer is reporting another kind of Chroma Location, 'cause it should be _ChromaLocation = 0 left (mpeg2) and not _ChromaLocation = 2 top_left... interesting... This is another XDCAM file and it has the very same problem: and ffprobe seems to agree with the indexers: [STREAM] index=0 codec_name=mpeg2video codec_long_name=MPEG-2 video profile=4:2:2 codec_type=video codec_tag_string=[0][0][0][0] codec_tag=0x0000 width=1920 height=1080 coded_width=0 coded_height=0 closed_captions=0 film_grain=0 has_b_frames=1 sample_aspect_ratio=1:1 display_aspect_ratio=16:9 pix_fmt=yuv422p level=2 color_range=tv color_space=bt709 color_transfer=bt709 color_primaries=bt709 chroma_location=topleft This is yet a different XDCAM file I received from A&E: index=0 codec_name=mpeg2video codec_long_name=MPEG-2 video profile=4:2:2 codec_type=video codec_tag_string=[0][0][0][0] codec_tag=0x0000 width=1920 height=1080 coded_width=0 coded_height=0 closed_captions=0 film_grain=0 has_b_frames=1 sample_aspect_ratio=1:1 display_aspect_ratio=16:9 pix_fmt=yuv422p level=2 color_range=tv color_space=unknown color_transfer=bt709 color_primaries=unknown chroma_location=topleft and this is a movie I've got from Notorious Pictures and it says top left too: [STREAM] index=0 codec_name=mpeg2video codec_long_name=MPEG-2 video profile=4:2:2 codec_type=video codec_tag_string=[0][0][0][0] codec_tag=0x0000 width=1920 height=1080 coded_width=0 coded_height=0 closed_captions=0 film_grain=0 has_b_frames=1 sample_aspect_ratio=1:1 display_aspect_ratio=16:9 pix_fmt=yuv422p level=2 color_range=tv color_space=unknown color_transfer=bt709 color_primaries=unknown chroma_location=topleft Ok, either there's something fishy and totally broken here, or I've lost my certainties... |
|
14th January 2022, 19:03 | #1775 | Link |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,904
|
Ok, guys, for the glory and my mental health, what's the REAL chroma location of this file: https://we.tl/t-MhjJWTj0lQ
and is the metadata just wrong or is it FFMpeg and the indexers that are not getting it right? (link available for 7 days) |
14th January 2022, 19:16 | #1777 | Link | |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,904
|
Quote:
I mean, let's think this through logically. Top Left is Type 2, so the one generally found in H.265 UHD HDR PQ BD. Now, those files are MPEG-2 50 Mbit/s BT709 SDR in FULL HD and 4:2:2 8bit planar. Sony made the XDCAM standard in 2003 and the XDCAM-50 is from 2006 or something like that if I remember correctly. Back then, Type 2 top left didn't even exist, did it? Besides, I would find hard to believe that such a standard uses top left instead of left, given that it's an MPEG-2, right? So we can assume that indexers and ffprobe are all wrong and that it's actually Type 0, left, MPEG-2, right? If someone is willing to download the masterfile and check I would really appreciate it. Also 'cause if it's actually Type 0 and indexers + FFMpeg say "Type 2", this is a BIG BIG PROBLEM and we need a fix soon. |
|
14th January 2022, 22:18 | #1778 | Link |
Registered User
Join Date: Nov 2009
Posts: 2,361
|
Same error trying to display the clip, I'm stable 3.7.1 and AvspMod 2.6.7.9 with "read from source script" disabled. propclearall() fixes it. I will try to check chroma placement, haven't seen an empirical method to find out so.
EDIT: Yes, looks MPEG2 to me
__________________
i7-4790K@Stock::GTX 1070] AviSynth+ filters and mods on GitHub + Discussion thread Last edited by Dogway; 15th January 2022 at 01:22. |
15th January 2022, 05:17 | #1779 | Link | |
Registered User
Join Date: May 2018
Posts: 184
|
Quote:
https://ffmpeg.org/contact.html#IRCChannels https://ffmpeg.org/contact.html#MailingLists |
|
15th January 2022, 11:42 | #1780 | Link | |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,904
|
Quote:
|
|
Thread Tools | Search this Thread |
Display Modes | |
|
|