MJPEG AviSynth Color Clipping

[kevina]

I recently discovered a rather disturbing thing, the color gets clipped. That is very dark colors and very light colors get crushed together and the other colors are spread out. After searing this form I discovered there is a filter mjpegcorrect which can help with this problem. I also figured out that using:
AVISource("file.avi", pixel_type="RGB32")
ConvertToYuy2()
Fixes this problem.

My question is, what is causing this?

I also have a request, could you please make a note of this in the manual under the AVISource section. By not using pixel_type="RGB32" color information is lost. However, the effects of this are not easy to notice, and thus the casual user may not notice until it is too late.

This is especially disturbing to be since I carefully adjust the levels to get the full range after I capture the video. Thus important color information is at both ends of the spectrum and will thus get lost if using Avisource with mjpeg.

[sh0dan]

MJPEG is stored in YUV - so what you are getting is actually the correct colors.

I however depends on whether the MJPEG codec upscales the YUV colors, before sending them on. Some have this effect, which is quite undesirable.

Instead of adding a YUV -> RGB32 -> YUV colorspace conversion simply try:

ColorYUV(levels="PC->TV") - this will give you the correct CCIR 601 compliant output.

[unaffiliated note]
I can personally recommend buying Morgan MJPEG - it provides correct luma range, and it even has YV12 in their latest beta.
[/unaffiliated note]

[kevina]

Instead of adding a YUV -> RGB32 -> YUV colorspace conversion simply try: ColorYUV(levels="PC->TV")
I tried that and it works. However the histogram is not as smooth as given by VirtualDub. And since VirtualDub converts back to RGB it semas that YUV->RGB->YUV->RGB gives better results than YUV->Fix->RGB. Of course if I drop the ConvertToYUY2 I get the best results.

Anyway a mention of this in the AVISynth manual would be very helpful.

[sh0dan]

The reason is looks more "smooth" is because of the errors introduced by YUV->RGB->YUY conversion . This is not without loss. Trust me - the result is better, if you avoid doing the conversions, being more true to the original.

[waka]

If you can encode and decode in the same colorspace, preferably YUV, there shouldn't be any problems. If you are capturing to mjpeg in YUV, don't open the mjpeg directly with virtualdub to check the levels. It would be decoded to RGB and what you see in virtualdub is incorrect.

If you are re-encoding to mjpeg, do all your filtering in avisynth so you can use fast recompress in virtualdub. Otherwise you will encode from RGB and likely decode to YUV, again screwing things up.

[kevina]

waka wrote: If you can encode and decode in the same colorspace, preferably YUV, there shouldn't be any problems.

The problem has nothing to do with colorspace conversion. It has to do with the fact that some "MJPEG codec upscales the YUV colors, before sending them on", as sh0dan pointed out.

[waka]

For me, the issue does seem to be caused by colorspace conversions performed by the codec. Everything is fine with zero or two(letting the codec fix itself) conversions. With one conversion, either encoded RGB/decoded YUV or encoded YUV/decoded RGB, there are problems.

Using blankclip(color=$0000f0,pixel_type="yuy2"), starting clip has YUV values of 39, 234, 111. Picvideo mjpeg encoded with fast recompress for YUV or full processing for RGB. Resulting mjpeg opened with avisynth using pixel_type="yuy2" or "rgb32". Values checked with coloryuv(analyze=true), converttoyuy2() as necessary. Last two used coloryuv(levels="tv->pc") or "pc->tv" for correction.


EncodeYUV/DecodeYUV Y:39 U:234 V:111
EncodeRGB/DecodeRGB Y:39 U:234 V:111
EncodeYUV/DecodeRGB Y:50 U:221 V:113
EncodeRGB/DecodeYUV Y:27 U:248 V:108
EncYUV/DecRGB tv->pc Y:40 U:234 V:111
EncRGB/DecYUV pc->tv Y:39 U:233 V:111


I think sh0dan and myself are more or less describing the same thing. I hope I didn't give him the wrong idea in a previous discussion. I could also be totally wrong.

[sh0dan]

Silly - I guess the only thing I can learn from that is that Input colorspace and output colorspace from Picvideo MJPEG should be the same. Great test!

Could be funny to crosstest with Morgan to see what it actually stores. ;)

[kevina]

Sh0dan, I removed PicVideo and installed Morgan MJPEG v 3.0.0.9 and I have the exact same problem.

With your codec can you please try the waka test using your codec. Thanks.

[FulciLives]

I capture using PICVideo MJPEG and then I use AviSynth 2.5x and open that in CCE

Do I need to do anything?

I tried the AviSynth script line:

ColorYUV(levels="PC->TV")

and in CCE selected 0-255

The result seemed pleasing to my eyes.

BTW this was with a USA NTSC capture and my capture card was set to NTSC_M

Any advice on this would be most welcomed :)

- John "FulciLives" Coleman

[trevlac]

I wanted to comment on this mystery. I'm not sure this behavior is a bad thing. It's just that one should know what is going on to properly use the tools.

The differences are due to mapping from YUV to RGB. This means going from a 16-235 range to a 0-255 range (or not).

I've done a bunch of reading, and it seems that 'video people' think mappping is a bad thing, because 1) it removes headroom, 2) introduces more rounding errors. We PC people do not (as much).

Here (http://www.ajavideo.de/technic/colorspacing.html) is a link to comments from a serious 'video person'. :)


Originally posted by waka


1. EncodeYUV/DecodeYUV Y:39 U:234 V:111
2. EncodeRGB/DecodeRGB Y:39 U:234 V:111
3. EncodeYUV/DecodeRGB Y:50 U:221 V:113
4. EncodeRGB/DecodeYUV Y:27 U:248 V:108
5. EncYUV/DecRGB tv->pc Y:40 U:234 V:111
6. EncRGB/DecYUV pc->tv Y:39 U:233 V:111




The possible steps in this test were:

Create YUV clip in Avisynth
PC default YUV codec converts to RGB for VDub
PIC MJpeg Encodes RGB or YUV
PIC MJpeg Decodes and may convert RGB->YUV or YUV->RGB
Convert2YUV may be used to measure with ColorYUV()
TV to PC or PC to TV


I show the paths below. A (+) indicates that the range 16-235 was changed to 0-255, and a (-) indicates that the range 0-255 was changed to 16-235. The different results are because PIC does not map.


1. (a)YUV -> (b)n/a -> (c)YUV -> (d)YUV -> (e)n/a -> (f)n/a Y:39
2. (a)YUV -> (b)RGB+ -> (c)RGB -> (d)RGB -> (e)YUV- -> (f)n/a Y:39
3. (a)YUV -> (b)n/a -> (c)YUV -> (d)RGB -> (e)YUV- -> (f)n/a Y:50
4. (a)YUV -> (b)RGB+ -> (c)RGB -> (d)YUV -> (e)n/a -> (f)n/a Y:27
5. (a)YUV -> (b)n/a -> (c)YUV -> (d)RGB -> (e)YUV- -> (f)YUV+ Y:40
6. (a)YUV -> (b)RGB+ -> (c)RGB -> (d)YUV -> (e)n/a -> (f)YUV- Y:39


As you can see, if you do an xtra down size you get Y:50. If you do an xtra upsize, you get Y:27. Also, all the converting can cause a small shift.

What I thought was most interesting was the shift of the U:221. This is due to the PIC conversion from YUV to RGB. This conversion requires 4:4:4, and YUV here has 4:2:2. When pic upconverts, they may do more than just double the values, and this causes a shift.

-----------------------
John,

I'd say CCE expects RGB data. If you choose the 0-255 option, CCE will not remap it to 16-235. If you pic 16-235, it will. This means the following would be a working process from capture to encode.

1. Cap PIC YUV 16-235 (Correct for video/Bad for PC viewing)
2. CCE requests RGB from PIC. PIC gives RGB without mapping (Still correct for Video)
3. CCE option 0-255. CCE takes RGB, changes to YUV and does not map (Still correct for video)

So, Cap PIC YUV feed to CCE 0-255 should be golden. No RGB/YUV conversions in the middle.

Trev

:D

[trevlac]

Originally posted by FulciLives
BTW this was with a USA NTSC capture and my capture card was set to NTSC_M


BTW:

Although it seems people think there is a difference between PAL,NTSC, and NTSC_J; for digital YCbCr, there is none.

For each, 16=Black and 235=White. When you capture, the card converts from YUV or YIQ to YCbCr taking into account if black starts at 7.5IRE or 0IRE. When you get YCbCr there is no difference.

In addtion, it is my understanding that something like a dvd player converts from the 601 standard back to your TV standard for you. The digital data does not differentiate. I have read that some dvd players (like the sony playstation) do not set the black level to 7.5IRE.

[FulciLives]

Originally posted by trevlac
-----------------------
John,

I'd say CCE expects RGB data. If you choose the 0-255 option, CCE will not remap it to 16-235. If you pic 16-235, it will. This means the following would be a working process from capture to encode.

1. Cap PIC YUV 16-235 (Correct for video/Bad for PC viewing)
2. CCE requests RGB from PIC. PIC gives RGB without mapping (Still correct for Video)
3. CCE option 0-255. CCE takes RGB, changes to YUV and does not map (Still correct for video)

So, Cap PIC YUV feed to CCE 0-255 should be golden. No RGB/YUV conversions in the middle.

Trev

:D
Well according to ADAM (a mod at dvdrhelp.com) CCE uses YUV and not RGB

Also ... according to Adam ... you pick 0-255 in CCE if your capture is YUV 16-235 otherwise you pick 16-235 in CCE

How true this all is? I don't know!

But assuming Adam is correct and CCE works in YUV and PICVideo MJPEG works in YUV then I guess you don't have to worry about it ... except this thread is about how some MJPEG codecs (I am unclear if this includes PICVideo MJPEG) need the AviSynth line:

ColorYUV(levels="PC->TV")

To correctly "decode" to the encoder.

So I'm still confused hehehe

- John "FulciLives" Coleman

[trevlac]

Well,

According to this doc:

http://www.cinemacraft.com/files/doc/ccl_264e.pdf

CCE 1st looks for YUY2. If it finds it, then there is no mapping to do so 0-255 nor 16-235 should not have any effect. If it finds RGB then the 0-255 option means do no mapping. If you pick 16-235 then it says map to 16-235. Seems backwards, but that's what that doc says.

So in your case, if you have PicVideo send YUY2 to CCE, there should be no problem. If you have PicVideo convert the YUY2 to RGB32 then you should pic 0-255. This is because pic does not expand the YUV 16-235 to 0-255 when it converts, so CCE should not contract the range back when it converts to YCbCr 4:1:1.

If you post the link to what adam said, I'll comment on it.


PS: I have very little doubt about how the general process works. As far as what a given piece of software does, hard to say. :)

[FulciLives]

Well since Trevlac got me using the BTwincap driver I am now capturing at 368x480 using PICVideo MJPEG at the 19 quality setting.

So here is the latest AviSynth script I've been using:

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec3.dll")
LoadPlugin("convolution3d.dll")
SegmentedAviSource("D:\capture\capture.avi")
Trim(801,157603)
SeparateFields()
crop(0,2,0,-6)
AddBorders(2,4,2,4)
even = SelectEven(last).Convolution3D (0, 6, 10, 6, 8, 2.8, 0)
odd = SelectOdd(last).Convolution3D (0, 6, 10, 6, 8, 2.8, 0)
Interleave(even, odd)
LanczosResize(352, 240)
weave()
ColorYUV(levels="PC->TV")

Since my crop command is after SeparateFields() I am actually cutting 12 from the bottom and 4 from the top. I'm thinking maybe I should just change the crop to:

crop(0,4,0-12)

And put it and the AddBorders command BEFORE the SeparateFields() command. This would change AddBorders to: AddBorders(2,8,2,8)

But really the thing I am most worried about is if I need that ColorYUV line or not.

Again I'm using CCE not TMPGEnc which is ASS slow on my computer (hence my switch to CCE as it is MUCH faster).

- John "FulciLives" Coleman

P.S.
My AddBorders command (after my crop command) brings my capture from 368x480 to 372x480 which sounds odd I know but if you capture at 712x480 and padd to 720x480 then resize to 352x480 you get about 2 pixels of black on the side so I am just trying to "mimic" that with my 368x480 capture. I know we have that other thread going on dvdrhelp.com about this but this is how I've decided to handle it ... even though I know you (i.e., trevlac) are saying to do it differently ;)

[trevlac]

John,

Do what makes sense (and works) for you. ;)

On the color space thing. If you cap in YUY2, I don't see any need for PC->TV.

Trev

[stickboy]

Originally posted by trevlac
On the color space thing. If you cap in YUY2, I don't see any need for PC->TV.Isn't that exactly what kevina is talking about?

[trevlac]

Originally posted by stickboy
Isn't that exactly what kevina is talking about?

I'm not clear what you mean. PC-TV and TV-PC are never a good thing. :)

The point I was trying to make was that the results of kevina's tests really were no mystery. The MJPeg codec does not change the nominal range when it does a color space conversion. Avisynth does.

Personally, I think no change is better than making the change.

The reason is that (contrary to popular belief) if you do not change the range, you do not loose any luma info due to rounding, when converting from YUY2 to RGB. This is not true for chroma, but I think ~ 200 of over a possible 1 million colors are effected.

The general calculation for the conversion (assuming black and white) is:

R = Y
G = Y
B = Y

now go back

Y = .299R + .587G + .114B


Plug any number between 0-255 into Y and there will be no rounding loss in the result when you hit RGB. If you then convert back, you will get the same Y. Not so if you convert and map to a different nominal range. Because to go from a range of 219 steps to a range of 255 steps requires multiplication by 255/219 (better known as 1.164).

The full equations for color are here: http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html#RTFToC31

They are directly derived from Spec 601, which states the numbers I gave above.



---------
What was kevina's point? Don't use RGB because it's confusing ? What do you think?

Cheers,

Trev

[stickboy]

kevina wasn't using RGB at all except as a workaround. The problem he's reporting is that even though the MJPEG data is stored in YUV, even when YUY2 frames are requested through PicVideo's MJPEG decoder, colors at the end get crushed together.

[FulciLives]

Originally posted by stickboy
kevina wasn't using RGB at all except as a workaround. The problem he's reporting is that even though the MJPEG data is stored in YUV, even when YUY2 frames are requested through PicVideo's MJPEG decoder, colors at the end get crushed together.
Here is what I've learned so far from this thread:

1.) Some MJPEG codecs may "crush" the luma range. With MJPEG codecs that do this the workaround is the AviSynth line:

ColorYUV(levels="PC->TV")

2.) PICVideo MJPEG does not "crush" the luma when you encode YUV and decode in YUV

I wasn't so sure about point 2 but after reading waka's post this now seems more clear to me. I'm still not 100% though if PICVideo MJPEG does this "crush" or not but now it seems like it doesn't.

Damn I'm still confused!

- John "FulciLives" Coleman

[trevlac]

Originally posted by stickboy
kevina wasn't using RGB at all except as a workaround. The problem he's reporting is that even though the MJPEG data is stored in YUV, even when YUY2 frames are requested through PicVideo's MJPEG decoder, colors at the end get crushed together.

OK,

"MJPEG codec upscales the YUV colors, before sending them on"

But this does not match waka's test results. Or does it?

Would "EncodeYUV/DecodeYUV Y:39 U:234 V:111" not have come out as Y:27... if it up scaled?

I don't mean to be talking out of my arse here :) I can get PIC and do some tests if you like. I was basing my comments on waka's tests.

I was not clear where kevina was seeing a histogram. Another thread ?

Cheers


Edit I did some quick tests with PIC. If i get it to decode to RGB, it clearly does not clamp. If I use avisynth/huffyuv to convert to RGB, it clamps at 235.

Avery says (at vdub forum) it is a bug in PIC that they admit to, and that it is due to how they mjpeg encode yuy2. This implies that there is a problem with encode Yuy2/ decode yuy2. I did not test this, but waka's test do not show this. Perhaps using morgan to decode would show a problem.

[FulciLives]

Originally posted by trevlac
I did some quick tests with PIC. If i get it to decode to RGB, it clearly does not clamp. If I use avisynth/huffyuv to convert to RGB, it clamps at 235.

Avery says (at vdub forum) it is a bug in PIC that they admit to, and that it is due to how they mjpeg encode yuy2. This implies that there is a problem with encode Yuy2/ decode yuy2. I did not test this, but waka's test do not show this. Perhaps using morgan to decode would show a problem.
Well know we are finally getting somewhere ,,, don't suppose you would want to test more including YUV to YUV :)

- John "FulciLives" Coleman

[trevlac]

Anything for you John,

:)


blankclip(width=480,height=480,pixel_type="YUY2",color=$FDFDFD)
ColorYUV(Levels="TV->PC")
ColorYUV(Analyze=true)


1) Above clip creates a YUY2 file with ranges outside of what is normal. 250 for everything.

2) Open in vdub and make sure by reading the numbers

3) Remove the last line, open in vdub, fast recompress with PIC (latest version from their site)


blankclip(width=480,height=480,pixel_type="YUY2",color=$FDFDFD)
ColorYUV(Levels="TV->PC")



4) Now open this in vdub. The numbers should still be 250.

Avisource("Pic.avi", pixel_type="YUY2")
ColorYUV(Analyze=true)




conculsions
Pic encodes the full range from YUY2 source. It decodes the full range from YUY2 and from RGB. Other converters do not. kevina did not provide enough info to determin what was his problem. Was his source out of range? How did he know it clipped? If he opened it in VDub, something did a conversion.


Why do I care?

Well, I wrote a color correction plugin for VDub. To do that one has to know what is going on. When I saw this thread, it was appearent, although I had never tested PIC.

Here is another test: using this (http://trevlac.us/clrtools.vdf)


#Avisource("Pic.avi",pixel_type="RGB32")
Avisource("Pic.avi",pixel_type="YUY2")


5) Open in vdub, use histograms in my vdub filter (keep all defaults). Your default YUY2 codec (unlikely to be pic) will convert to RGB. When it does this, it will most likely clamp the 250 number to 255. If you switch the lines above, PIC will do the conversion and it will keep the 250.

[WARNING: Bold statement]
PIC does everything correctly. Most other converters are wrong.
[/WARNING: Bold statement]

To back off a bit, maybe everything should give an option.


Who's your daddy? :)

[FulciLives]

Trevlac you are a godsend sometimes (ok most times) :)

This is what I did:

I created an AVS script ala your example above:

blankclip(width=480,height=480,pixel_type="YUY2",color=$FDFDFD)
ColorYUV(Levels="TV->PC")
ColorYUV(Analyze=true)
When I open this in VirtualDub I get this:

LumaY = 253
ChromaU = 128
ChromaV = 128

I get this "across the board" for Average, Minimum, Maximum, etc.

I then adjusted the script as you said and deleted the last line so it looks like this:

blankclip(width=480,height=480,pixel_type="YUY2",color=$FDFDFD)
ColorYUV(Levels="TV->PC")

I open this in VirtualDub and do a FAST RECOMPRESS using PICVideo MJPEG codec which I set up the following way:

Quality = 19
Which sets Luminance Quality to 5 and Chrominance Quality to 6

Subsampling = 4/2/2

Checked = 2 fields if more than (240) Lines
Checked = Primary MJPEG Codec
Checked = Codec Enabled

Unchecked = Swap Decompress Fields
Unchecked = Force YUY2 Output

I created a file that I called testtvpc.avi

I then loaded this file into VirtualDub with this AviSynth script:

Avisource("D:\testtvpc.avi", pixel_type="YUY2")
ColorYUV(Analyze=true)
Again the values were as before:

LumaY = 253
ChromaU = 128
ChromaV = 128

So I guess this matches what you got ???

On a side note ...

I did a capture the other night of a 60 minute TV show that ended up at about 44 minutes with commercials cut out.

I encoded the same clip (captured with PICVideo MJPEG using the settings as above) using AviSynth (clips imported from VirtualDubMod) but with encode A I used ColorYUV(levels="PC->TV") but with encode B I deleted that part.

So now I have two different MPV video files from CCE (which is the encoder I used ... AviSynth AVS files loaded into CCE direct).

Can I load these MPV video files into VirtualDub using an AviSynth script to check the Luma and Chroma levels and if so what would the script be?

I loaded them direct into VirtualDub and noticed that the one with the ColorYUV(levels="PC->TV") line is slightly brighter overall (too bright) and less rich in color than the encode where I did not use ColorYUV(levels="PC->TV")

Which reminds me ... I used "PC->TV" for this TV encode but in your example (which I just repeated) I used TV->PC as you did. Yet I used PC->TV in my encode because if you look at the first post made by sh0dan this is what he said to use (he said use PC->TV)

So I decided to do your test over again (as I did at the start of this thread) but this time I used "PC->TV"

The results?

LumaY = 216
ChromaU = 128
ChromaV = 128

Again across the board. First loading the script and then loading the file created with the script.

So I guess when using PICVideo MJPEG the AviSynth command:
ColorYUV(levels="PC->TV") or ColorYUV(levels="TV->PC")
is not needed.

Correct?

- John "FulciLives" Coleman

[trevlac]

@Fulci

Sorry for that small bit of confusion. I should have said 253 not 250. I changed my mind in mid post and did not update the hex. You got it right.

I would never use the following:
ColorYUV(levels="PC->TV") or ColorYUV(levels="TV->PC")

The 1st crushes everything down and you then loose about 15% tonality.
The 2nd streatches out from the middle and crushes the ends.

If your video is out of wack (because you had the brightness to high when you cap'd for example), I'd say you need to look at it and decide what to do. The ColorYUV commands do an 'auto fix'.

For your case, If you cap in YUV and CCE reads YUV (as it appears to), you are set.


PS: I'm putting a guide together on this stuff, to go with my vdub plugin. I'll pm you at vcdhelp. Maybe you can test for me.

Trev

[trevlac]

Originally posted by FulciLives

Which reminds me ... I used "PC->TV" for this TV encode but in your example (which I just repeated) I used TV->PC as you did. Yet I used PC->TV in my encode because if you look at the first post made by sh0dan this is what he said to use (he said use PC->TV)



BTW:

sh0dan knows what he is talking about. He probably wrote the code for PC->TV. He suggested this because the orignial poster complained that the YUY2 from PIC was streatched out. PC->TV squishes it back. Problem is, the original poster didn't do enough testing. Waka's tests showed that kevina may have been wrong.

So, wrong problem, wrong cure.

Could also be that the version of PIC used by kevina was different than what we tested. :)

[FulciLives]

Well I'm glad we finally got this mess sorted out!

I'll test whatever you want ... PM me on that there ummm other site I'm not on doom9 all that often.

I just come here from time to time to get confused about tech issues :p

- John "FulciLives" Coleman

[Coon_Dog]

I am having this problem too. I changed from Huffy to Pic video and I stopped getting blue aliens for family members. But the colors are a bit off at times.

I am getting light green bars (like window blinds) in my video.

The big problem is everything you guys are saying is Greek to me. Is there one answer to the proper way to take in an MJPEG file in AviSynth?

Thanks

[trevlac]

@Coon_Dog

There are many ways to skin a cat. :)

Describe your process and I will see if I can pinpoint the problem.

For example:

Capture yuy2 with huffy
open in vdub using this avs script
save to xxx
Encode with x


The blue color problem sounds like the u & v are swapped. There is a swapuv command in avisynth to fix this.

If you stick with yuy2 in Pic, you should not have a problem. Are the colors off on a PC or TV?

[Coon_Dog]

The colors are off a Cannon A70 digital camera. And PIC solved the blue switch, but causes the green bars.

Get file from Cannon A70
make and AVS script
Open with Nic's Windows Media Encoder to encode to WM9

Thats when I get green bars in the output. Is there a way to tell what kind of color is in my cameras MJPEG files?

Jim

[trevlac]

@Dog

I did a quick search and it turns out you have a still camera that makes mjpeg files. Presumably you use usb to get them to your computer. In the manual, it appears that quicktime is recommended. From my limited understanding; avi is a container file, mjpeg is a compression, what exactly is compressed into the container file may have to be known by the decompressor, so that it can output it correctly. For example, PICmjpeg can compress rgb and yuy2 video.

So, I can't really say more about your camera, other than maybe: try using quicktime to decompress the movie files.

Sorry