HD Colorimetry "Blues"

[stabilimenta]

so i'm using DGAVCIndex for HD (AVC) rips, and have a significant problem. the issue lies in the fact that certain movies have no colorimetry declared for the video stream, and in these cases, DGAVCIndex selects a default scheme:

--from DGAVCIndex manual--
Note that if the stream does not declare the colorimetry, then ITU-R BT.709* is reported for HD video, and ITU-R BT.470-2* is reported for SD video. The * character indicates that the stream did not declare the colorimetry. The actual value read from the stream is shown in brackets. (in this instance, DGAVCIndex reports "BT.709* [2]")

this was the case for my last Blu Ray, and when attempting to rip w/ Gordian Knot, it became painfully apparent to me that this default setting was assigning incorrect color values to the .dga file generated (see preview sample image below). it acually appears that the green component has been omitted altogether. has anyone encountered this, and is there any way to circumvent the problem, or fix the .dga file after the fact?

http://k.min.us/il5tvy.jpg
*i realize the AR is off*


if anything is unclear, i’ll be very happy to further elucidate or provide additional information as needed.

i'd be much obliged for the help, peoples.

[Guest]

Post a link to an original stream sample (before encoding) and explain how you ripped and encoded the video.

[stabilimenta]

that image is from the original stream (within the .m2ts container) before any encoding. it's a capture of the preview generated by GK (the virtualdub preview also looked like this) from the .dga file generated by DGAVCIndex. i'd done no encoding yet at that point, so the problem lies in the indexing and the way that GK is instructed to interpret the images (far as my feeble knowhow can tell me). here's a link to a clip from the raw h264 stream (and also a .dga file generated by DGAVCIndex from its indexing of the .m2ts clip:

http://www.mediafire.com/?qfmd4iotbqicern


FYI - the way i feed the .dga file to Gordian K is through the following simple, no-frills script:

# PLUGINS
LoadPlugin("C:\Program Files\Video Processing Tools\dgavcdec\DGAVCDecode.dll")

# SOURCE
AVCSource("C:\Documents and Settings\PATH\FILENAME.dga")


this method has always worked for me before, both with standard def VOBs and other HD films (that's right, i have used DGAVCIndex before with h264 streams, to proper effect).


thx again

[hello_hello]

I downloaded your sample and it behaved perfectly normally for me. Colors looked as expected. There was definitely no green component missing.
I couldn't open your dga file directly though without the original m2ts file, so I had to create a new one.

By the way, the problem you're experiencing probably hasn't got anything to do with BT.709 v BT.470-2 (BT.601) colorimetry. The difference is nowhere near that drastic, if it's noticeable at all. The effect of going from HD to SD without color converting is usually to make reds a little darker and greens and blues a little brighter (if memory serves me correctly). Aside from skin tones, which can look a little too dark (red), it's often hard to notice the difference unless you're comparing the original and the encode side by side.

Assuming you are encoding from HD to SD, use the colormatrix plugin to correct the colors. I don't think DGAVCIndex actually does anything in relation to colorimetry aside from reporting which is used. HD is virtually always BT.709 and SD is virtually always BT.601. Just add something like this to your script:

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\ColorMatrix.dll")
ColorMatrix(mode="Rec.709->Rec.601")

When encoding without any RBG conversion taking place (which is where the colorimetry conversion comes into it) the colorimetry is kind of irrelevant. The encoder will just be encoding YUV video without caring which colorimetry should be used for playback. It's generally decided then which colorimetry to use, and it's generally BT.709 for HD and BT.601 for SD, so you normally only have to think about converting the colors when encoding from one type to the other. Chances are your last HD to SD encode actually plays back using the wrong colorimetry, only you just didn't notice. This time, I think something else is wrong, but I don't know what that might be.

PS. Windows... or the renderer... seems to be a little dumb when picking which colorimetry to use for playback, although the rule is still basically BT.709 for HD and BT.601 for SD. Colorimetry in the video stream is generally ignored. I couldn't remember exactly, so I found an old post of mine and copied this from it. It's how, using XP, Windows seems to choose colorimetry on playback.

"I've tentatively concluded that Windows will always display using R.601 if the width is less than 1200, regardless of the height, and it'll always display using R.601 if the height is less than 578, regardless of the width.
Or to put it another way, for video to display using R.709, width must be equal to or greater than 1200 AND height must be equal to or greater than 578."

Good luck with the green problem... or lack thereof.

[stabilimenta]

i knew that the .dga file wouldn't be much good to you w/out the original .m2ts, but i wanted you to see, firsthand, the file that DGAVCIndex produced, in case it held any clues.

so i realize that it's not a BT.709 v BT.470-2 (BT.601) issue. sorry if you feel like you've covered this already, but it really seems to me to be a difficulty that DGAVCIndex is having reading the color scheme. i'm sticking to this deduction because the only difference between the successful rip i did and this one was that for the successful one, the program simply reported BT.709, with none of that asterisk and brackets stuff. when i loaded the successful one into GK, the preview, and the resultant rip, looked perfect color-wise (and everything-wise). both were cut in size to a horizontal aspect greater than or equal to 1200, but again, the problem appears before the editing and encoding even get started. it really seems to be all about that DGAVCIndex indexing operation.

and a quick question about that script. if i want to just command avisynth to use BT.709, how would it read (if that's even possible)? i tried to just remove the "->Rec.601" portion of the script, but GK didn't accept that. not surprising since i really have no idea what i'm doing with the command line stuff. ;)

i emphasize that you're dealing with a real novice, here. sorry to prove such a hopeless charity case, but i really do appreciate the help.

thx again

[hello_hello]

and a quick question about that script. if i want to just command avisynth to use BT.709, how would it read (if that's even possible)? i tried to just remove the "->Rec.601" portion of the script, but GK didn't accept that. not surprising since i really have no idea what i'm doing with the command line stuff.

You can't force BT.709 the way you're suggesting, only convert the colors. Colorimetry only comes into it when converting the YUV video to RGB, so unless you're using an RGB filter when encoding, the only time RGB conversion will take place is during playback. It's only then that "should I be using BT.709 or BT.601?" comes into it.
When encoding, the encoder takes the original YUV video and re-encodes it, so the colorimetry shouldn't be effected. If you're encoding a HD BT.709 video, BT.709 will be used on playback for both the original and the encode. If you're reducing it to SD, BT.601 will be used on playback, so then you need to convert the colors while encoding.

The problem could be with dgavcindex, but I really don't think it's anything to do with BT.709 v BT.601 as such. I looked at the dga file (you can open it with notepad) and nothing looked out of the ordinary to me, but I'm no expert there. You might have to wait for someone else to look.

I recently tried to encode a similar video using AutoGK and had a similar but different problem. No matter how I opened it for encoding, the colors looked separated. For want of a better description R, G & B seemed to be separated giving me three different images which weren't exactly laid over the top of each other (sort of ghosting). I tried 300 different ways to open it and in the end remuxing it into an MP4 before encoding solved the problem (MKV had the same issue). I never worked out why.

So until a DGAVCIndex expert comes along, try opening it in a different way. Maybe remux the m2ts file as an MKV (MKVTooloNix) and/or try opening it again, using DirectShowSource or FFmpegSource. For the former it's just opened via DirectShow so there's no indexing, for the latter an index file will be created the first time you run the script. http://code.google.com/p/ffmpegsource/
Maybe also try extracting the raw video stream and muxing it into an MKV rather than working with the M2ts file directly.

Using MeGUI it seems to be recommended not to work with m2ts files directly, but I'm not 100% sure why as being a GUI kind of guy I'm lazy about remembering certain things. I think I've only ever used GKnot once in my life, so I don't know if it's causing the problem. Generally when ripping the disc with MeGUI's HD streams extractor the video is saved as an MKV and I'm fairly sure, indexed using FFmpegSource.
I assume you've tried indexing the raw video stream yourself, given you supplied a raw sample in your download rather than a section of the m2ts file? I can't tell if I'd have the same problem working with the m2ts file.

Sorry I can't be more specific, but it's some ideas until an expert comes along.

PS. Not having been GKnot user I can't say much about it, but I do use MeGUI regularly and quite like it. Maybe give it a whirl instead. Wouldn't GKnot be a bit outdated these days?

[Guest]

I'm a DGAVCDec expert. :)

stabilimenta is not providing what I need and asked for ("original stream sample"). I need a fragment of the m2ts file. Without that I cannot help properly. You can cut it with DGSplit.

My guess is that the issue is caused by GK. There's really no reason to have it in your processing chain. Just open your no-frills AVS script in VirtualDub -- are the colors OK? If so, then DGAVCDec is exonerated.

Maybe GK is treating it as I420 instead of YV12?!

[hello_hello]

I wondered the same thing. Might be a bit hard to tell while indexing the raw video stream rather than the m2ts file.
While you're here, is there any reason why working with m2ts files directly would be a bad idea?

[Guest]

While you're here, is there any reason why working with m2ts files directly would be a bad idea? No, there's no reason; that's why I asked him to explain his processing, but he didn't account for this. So I wanted to try my own chain beginning with the source M2TS file.

[stabilimenta]

i realize you're the expert, and you're actually exactly who i was hoping would offer some help here. so thanks much for getting involved. i've had nothing but success with your tools so far, so i'm sure we can get this worked out somehow.

so that stuff being said... i'm pretty sure that my entire chain of operations leading up to encoding is accounted for above. also, i did mention that i ran into the exact same problem when circumventing GK and going directly to virtualdub. preview produced same color issues (w/ absolutely no GK involvement). i was unaware of the DGSplit tool for taking segments of the m2ts. i'll try it and get that to you. before now, i thought i had provided what you needed with the portion of the raw video stream i uploaded. sorry about that.

i'll get back to you soon.


here's a 20MB clip:

http://www.mediafire.com/?l6e8d2fgr02z2rl

hopefully this will suffice for your purposes.

[hello_hello]

I used MeGUI to open it and index it with DGAVCIndex and the colors were fine.
I opened the script MeGUI created using both VirtualDub and VirtualDubMod and the colors were still fine.

LoadPlugin("C:\Program Files\MeGUI\tools\dgavcindex\DGAVCDecode.dll")
AVCSource("D:\Stream.Clip.dga")
#deinterlace
#crop
#denoise

Here's the dga file, if you'd like to compare, or neuron2 would like to convert it to people speak for us. :)

DGAVCIndexFile7

D:\Stream.Clip.m2ts
C:\Program Files\MeGUI\tools\dgavcindex\

STREAM 1
PKTSIZ 192
VPID 4113
FO 2
YUVRGB 1
LUMFILT 0 0
CLIP 0 0 0 0
RANGE 0 20971519
AUDIO

SPS 2668258
PPS 2668313 0
LPOC 0
FRM 5 1
PIC 0
FRM 6 0
PIC 0
FRM 5 2
PIC 0
FRM 6 1
PIC 0
FRM 5 3
PIC 0
FRM 6 2
PIC 0
FRM 5 4
PIC 0
FRM 6 3
PIC 0
FRM 5 5
PIC 0
FRM 6 4
PIC 0
FRM 5 6
PIC 0
FRM 6 5
PIC 0
FRM 5 7
PIC 0
FRM 6 6
SPS 2668258
PPS 2668313 0
PIC 0
RPS 2668355
IDR 2668363
FRM 7 0
PIC 0
FRM 5 1
PIC 0
FRM 6 0
PIC 0
FRM 5 2
PIC 0
FRM 6 1
PIC 0
FRM 5 3
PIC 0
FRM 6 2
PIC 0
FRM 5 4
PIC 0
FRM 6 3
PIC 0
FRM 5 5
PIC 0
FRM 6 4
PIC 0
FRM 5 6
PIC 0
FRM 6 5
PIC 0
FRM 5 7
PIC 0
FRM 6 6
PIC 0
FRM 5 8
PIC 0
FRM 6 7
PIC 0
FRM 5 9
PIC 0
FRM 6 8
PIC 0
RPS 6320195
I 6320203
FRM 7 10
PIC 0
FRM 6 9
PIC 0
FRM 5 11
PIC 0
FRM 6 10
PIC 0
FRM 5 12
PIC 0
FRM 6 11
PIC 0
FRM 5 13
PIC 0
FRM 6 12
PIC 0
FRM 5 14
PIC 0
FRM 6 13
PIC 0
FRM 5 15
PIC 0
FRM 6 14
PIC 0
FRM 5 16
PIC 0
FRM 6 15
PIC 0
FRM 5 17
PIC 0
FRM 6 16
PIC 0
FRM 5 18
PIC 0
FRM 6 17
PIC 0
FRM 5 19
PIC 0
FRM 6 18
PIC 0
RPS 10131395
I 10131403
FRM 7 20
PIC 0
FRM 6 19
PIC 0
FRM 5 21
PIC 0
FRM 6 20
PIC 0
FRM 5 22
PIC 0
FRM 6 21
PIC 0
FRM 5 23
PIC 0
FRM 6 22
PIC 0
FRM 5 24
PIC 0
FRM 6 23
PIC 0
FRM 5 25
PIC 0
FRM 6 24
PIC 0
FRM 5 26
PIC 0
FRM 6 25
PIC 0
FRM 5 27
PIC 0
FRM 6 26
PIC 0
FRM 5 28
PIC 0
FRM 6 27
PIC 0
FRM 5 29
PIC 0
FRM 6 28
PIC 0
RPS 13875203
I 13875211
FRM 7 30
PIC 0
FRM 6 29
PIC 0
FRM 5 31
PIC 0
FRM 6 30
PIC 0
FRM 5 32
PIC 0
FRM 6 31
PIC 0
FRM 5 33
PIC 0
FRM 6 32
PIC 0
FRM 5 34
PIC 0
FRM 6 33
PIC 0
FRM 5 35
PIC 0
FRM 6 34
PIC 0
FRM 5 36
PIC 0
FRM 6 35
PIC 0
FRM 5 37
PIC 0
FRM 6 36
PIC 0
FRM 5 38
PIC 0
FRM 6 37
PIC 0
FRM 5 39
PIC 0
FRM 6 38
PIC 0
FRM 5 39
PPS 18293081 0
PIC 0
RPS 18293123
IDR 18293131
FRM 7 0
PIC 0
FRM 5 1
PIC 0
FRM 6 0
PIC 0
FRM 5 2
PIC 0
FRM 6 1
PIC 0
FRM 5 3
PIC 0
FRM 6 2
PIC 0
FRM 5 4
PIC 0
FRM 6 3
PIC 0
FRM 5 5
PIC 0
FRM 6 4
PIC 0
FRM 5 6
PIC 0
FRM 6 5
PIC 0
FRM 5 7
SIZ 1920 x 1080
FPS 240000 / 10010
CODED 108
PLAYBACK 108

[stabilimenta]

okay, so same BS on my end, and nothing on yours. let's try this another way.

here's a link to a folder with an x264 encode (in .MKV form) of that clip that i just did with GK:

http://www.mediafire.com/?i9k0tly96783sl3

it includes all avs, log, and settings files, so you should be able to see all portions of my operation clearly. that is my hope.



p.s. my .dga file looks identical to the one posted by hello_hello above, so i guess it ain't that.

p.p.s. as for using MeGUI (which i've been told is the program to use for HD encodes), i've tried to install it before and encountered the following error each time:

--------------------------------------------------------------------------------------

Preliminary log file only. During closing of MeGUI the well formed log file will be written.

-[Error] Unhandled error
--[Information] [8/21/2011 3:43:19 PM] Exception message: Configuration system failed to initialize
-[NoImage] at System.Configuration.ClientConfigurationSystem.EnsureInit(String configKey)
-[NoImage] at System.Configuration.ClientConfigurationSystem.PrepareClientConfigSystem(String sectionName)
-[NoImage] at System.Configuration.ClientConfigurationSystem.System.Configuration.Internal.IInternalConfigSystem.GetSection(String sectionName)
-[NoImage] at System.Configuration.ConfigurationManager.GetSection(String sectionName)
-[NoImage] at System.Configuration.PrivilegedConfigurationManager.GetSection(String sectionName)
-[NoImage] at System.Diagnostics.DiagnosticsConfiguration.GetConfigSection()
-[NoImage] at System.Diagnostics.DiagnosticsConfiguration.Initialize()
-[NoImage] at System.Diagnostics.DiagnosticsConfiguration.get_SwitchSettings()
-[NoImage] at System.Diagnostics.Switch.InitializeConfigSettings()
-[NoImage] at System.Diagnostics.Switch.InitializeWithStatus()
-[NoImage] at System.Diagnostics.Switch.get_SwitchSetting()
-[NoImage] at System.Diagnostics.BooleanSwitch.get_Enabled()
-[NoImage] at System.Xml.Serialization.TempAssembly.LoadGeneratedAssembly(Type type, String defaultNamespace, XmlSerializerImplementation& contract)
-[NoImage] at System.Xml.Serialization.XmlSerializer..ctor(Type type, String defaultNamespace)
-[NoImage] at System.Xml.Serialization.XmlSerializer..ctor(Type type)
-[NoImage] at MeGUI.core.util.Util.XmlDeserializeOrDefault[T](String path)
-[NoImage] at MeGUI.core.details.JobControl.loadJobLists()
-[NoImage] at MeGUI.core.details.JobControl.loadJobs()
-[NoImage] at MeGUI.MainForm.setGUIInfo()
-[NoImage] at MeGUI.MainForm..ctor()
-[NoImage] at MeGUI.MainForm.Main(String[] args)
--[Information] [8/21/2011 3:43:20 PM] Stacktrace
-[Error] [8/21/2011 3:43:20 PM] Exception message: Unrecognized configuration section system.serviceModel. (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Config\machine.config line 134)
-[NoImage] at System.Configuration.ConfigurationSchemaErrors.ThrowIfErrors(Boolean ignoreLocal)
-[NoImage] at System.Configuration.BaseConfigurationRecord.ThrowIfParseErrors(ConfigurationSchemaErrors schemaErrors)
-[NoImage] at System.Configuration.BaseConfigurationRecord.ThrowIfInitErrors()
-[NoImage] at System.Configuration.ClientConfigurationSystem.EnsureInit(String configKey)
-[Error] [8/21/2011 3:43:20 PM] Stacktrace
-[Error] [8/21/2011 3:43:20 PM] Inner exception: null
--[Error] [8/21/2011 3:43:20 PM] Inner exception

--------------------------------------------------------------------------------------

[Guest]

so that stuff being said... i'm pretty sure that my entire chain of operations leading up to encoding is accounted for above. You never mentioned demuxing or how you did it. Anyway, it's moot now.

also, i did mention that i ran into the exact same problem when circumventing GK and going directly to virtualdub. preview produced same color issues (w/ absolutely no GK involvement). Just as hello_hello reports, simply indexing your script and serving it into VirtualDub with a no-frills AVCSource() call looks perfect.

I don't know where to go from here. Are you seeing green from other clips?

EDIT: Just saw your new post. I'll look at it. Meanwhile, did you try just indexing and serving with a simple script using AVCSource() (not avisource())? Your "same BS" is a bit ambiguous. :) Just answer the question yes or no.

[Guest]

Again, if you serve Stream.Clip.avs (no-frills) into VirtualDub what do you see?

Well, I'm not going to debug GK for you, because I have never used it, know nothing about it, and have no time for it. You'll have to go to the GK forum for that. But this looks goofy to me and I would never do anything like this:

avisource("C:\Documents and Settings\Jed\Desktop\Fist-Clip\Stream.Clip.avs")

Try serving a script with just that line and it will tell us if that strange access method is at fault. You may need to force the color space when doing goofy stuff like this.

So, to repeat, what is the result when serving with just AVCSource()? And what is the result when serving with just the avisource() line?

[stabilimenta]

ha! that was the key. GK wanted me to serve up the new .avs script that it had created, but the default scripting language had directions to the wrong source (avisource) and the wrong .dll function (DGDecode.dll). i do believe we just cracked it, my good people.

lest my celebration be premature, i intend to perform a complete encode beginning tonight, and i will report back to you upon completion.


**and i think that the color issue in the GK preview (using MPC) is mostly unrelated... though i know that the first full encode i attempted actually produced an MKV that played w/ that incorrect color scheme. either way it seems solved, and i'm very pleased (at least tentatively).

[hello_hello]

It all sounds good now...

I'd maybe post your MeGUI problem in the MeGUI bug reports thread. Only thing I could guess at... do you have the latest .net crap service packs/updates installed. I'm pretty sure version 2 is all you need, but maybe it's not the latest version of .net version 2 which you have installed. :)

[Guest]

@stabilimenta

You continue to ignore my direct and repeated questions, so I will leave this thread. Good luck.

[stabilimenta]

i'm really sorry to have perturbed you so much; it wasn't at all my intention. i was hoping you'd be happy that i'd solved my problems. and i really don't think i ignored any questions or directions until the very last post, at which point (and through your guidance) i figured out the problem. if you are still interested in finding answers to your previous questions for software development purposes, please let me know through PM. i'd really not care to leave things on these terms and spoil the possibility of any future contact.

thank you again, and my sincere apologies.


and my thanks to hello_hello as well, of course.

[Guest]

Your solution explanation still appears incoherent. E.g., you attribute it partially to the script loading DGDecode.dll, but that is irrelevant because mpeg2source() is not invoked. Anyway, DGDecode cannot open an index file from DGAVCDec. And your reference to a "default scripting language" is baffling. If you answer my questions we would have a hope of clarifying everything.

[stabilimenta]

Q: Meanwhile, did you try just indexing and serving with a simple script using AVCSource() (not avisource())? Your "same BS" is a bit ambiguous. Just answer the question yes or no."
A: yes, i said that in my very first post.

Q: Again, if you serve Stream.Clip.avs (no-frills) into VirtualDub what do you see?
A: i'm pretty sure that it produced the same results at the time (i also stated this at the beginning). did not happen this time around though, so perhaps i did something differently (and incorrect) the first time around.

Q: (actually a request) Post a link to an original stream sample (before encoding) and explain how you ripped and encoded the video.
A: provided the sample. used GK to make edits to size, cropping etc., and then fed the modified script to GK and set the x264 parameters. let it rip.

Q: You never mentioned demuxing or how you did it. Anyway, it's moot now.
A: honestly, you never asked. DGAVCIndex seemed to run into problems with this film in .m2ts form, so i used TSRemux to remux it to a .ts file with the sole audio stream that i needed.

Q: Are you seeing green from other clips?
A: no. all the same.

Q: Try serving a script with just that line and it will tell us if that strange access method is at fault. You may need to force the color space when doing goofy stuff like this.
A: did it just this way, and reported back with my success

[Guest]

Thread re-opened after some time. I'm happy you've resolved your issue and I stand ready to seek the root cause and clarification if you need or desire any more analysis.