XVID-VDubMod and color space

[gatormac]

Let me start off by saying that this is not a bug report. I am extremely satisfied with XVID RC3 and feel it is ready for final regarding my needs....thanks to all the developers for this great piece of software. I just have an issue regarding color space that I don't know how to resolve.

I have recently made a backup of School of Rock, the results look great with the exception of a few seconds in the concert scene near the end of the movie. The scene in question has a very dark background with bright blue flourescent-ish lights. It doesn't help that they have filled the stage with fog/dry ice. The edges/lines of everything that falls within the blue lines has a stair-step or pixelated effect. I have done a search regarding this and found that XVID works in the YV12 color space and that some information is lost in the process--I tried some of Crusty's suggestions regarding improving poor upsampling, but the results were the same. I also found that Geforce 2 cards have overlay problems in the YV12 colorspace and I do have a Geforce 2 card....I have the latest Nvidia driver and don't know how to fix that issue without just going out and buying a better card.

Some of you may not need (or want!) to read any further to know what the problem is, but in order to paint the best picture I will further detail my process of encoding.

I keep things very simple and encode at moderately high bitrates. I start by ripping the DVD with Robot4Rip(0.5). I then use Gknot(0.28.7) to auto crop and set the video to MOD16 WxH. I do a compression test (even though it is debated on these boards whether or not XVID RC builds work this way yet) and set the bitrate to 80% of the compression result. I make no other changes, I note the video file size and save the avisynth script. I then open the .avs file in VDubMod(1.5.10.1) select fast recompress and adjust my XVID RC3 settings:


AS@L5
No AQ, Interlace, Qpel or GMC
B-VOPs 2, 1.50, 1.00 with closed GOV, but no packed bitstream
AR at default (square pixel)

Discard 1st pass, defaults on second pass (curve compression)

Set my desired filesize

MSP 6, VHQ 1, chroma motion enabled, Quants 2-31 with trellis enabled

All else defaults


I take the resulting video and interleave it with the MP3 using direct streams in Nandub(1.0rc2). The final .avi is about 1.4GB and looks great, with the aforementioned exception.

I looked at the DVD source, it looked fine.
I looked at the .d2v in Gknot, it looked fine.
I looked at the .avs in VDubMod, aha! Pre-XVID stair-stepping!

Having no idea what I was doing I added ConvertToRGB32() to the end of my avisynth script, loaded it into VDubMod, aha! No stair-stepping! I thought my problem was solved, but after encoding it with ConvertToRGB32() I had the same stair-stepping results. Apparently, ConvertToRGB32() fixed the issue for VDubMod, but once XVID started encoding it forced it back into YV12 (at least this is my simplistic rationalization), thus the resulting stair-stepping.

I use the latest version of MPC and only use the XVID RC3 encoder (DIVX has been removed from my machine).

As I said before, this is in no way a bug report or a complaint. I am very satisfied with XVID and its supporting programs. However, I am somewhat of a perfectionist and would like to solve this issue if I can. Any help would be greatly appreciated. Forgive me if I have unknowingly broken any forum rules....I don't think I have.

I would have included some captures, but my captures are all very large and shrinking them down to a reasonable size lost a good bit of the detail that I would have wanted you to see.

[Koepi]

Check the xvid decoder filter settings. You can force the output colour spae. It defaults to YUV2 now, but to make sure, try all of them.

The effect should be very visible with every MPEG codec btw., even if you reencoded what you have not to XVid/MPEG4 but as well as mpeg2!

Maybe this helps.

Regards
Koepi

[gatormac]

Thanks for the fast response Koepi. I've forced the possible colorspace selections in the XVID decoder (YV12, YUY2, RGB24, RGB32) and each make very slight differences, but none solve the problem of stair-stepping. I have also tried the different post-processing options with no success.

Maybe it is just my Geforce 2 card and there is nothing I can do about it. Has anybody else encoded this movie? What were the results of frames 136377 to 142320, stair-stepping from frames 140280-140370 most noticeable?

[LigH]

If you face a lack of bilinear filtering, sometimes the reason is that DirectX is not installed, and therefore no DirectShow filters and no fast output technologies (e.g. Overlay) are used - therefore only pixel doubling occurs for resized videos.

And even if DirectX is installed, there might be more reasons that no DirectShow filter was used to decode the video, but the VfW codec instead.

For the case that I misunderstood you, and the effect does not belong to "no bilinear filtering", you may have to describe the stairs more precisely...

[Soulhunter]

Read this... (http://forum.doom9.org/showthread.php?s=&postid=390365#post390365) ;)

Bye

[gatormac]

Thanks for the responses LigH and Soulhunter.

@LigH: DirectX is installed and DirectShow/overlay is being used, but that is thinking outside of the box and I appreciate your response. I would love to show a capture of the stair-stepping, but all my captures are large (>500kb)....I haven't found a way to compress them without losing the quality to show the problem adequately.

@Soulhunter: The only problem is that I have avoided using ffdshow due to the reports of jerky playback and such, not to mention the ease of use and great playback quality (up to this point) of MPC. I did see your post during my search and tried the VMR 9 Renderless mode in MPC (also in Crustys FAQ), but it didn't work either.

I guess the solution would be to buy a new graphic card, but it really isn't that big a deal, but hopefully this won't be an issue with standalones (whenever they get around to making a decent one).

[LigH]

Indeed, you shall use PNG format for captures; maybe just a smaller rectangular selection is enough to show the effect?

[gatormac]

As I said before, the .avs file looked stair-stepped pre-XVID within VDubMod, but I managed to make it look good in VDubMod by adding ConvertToRGB32() to the end of my avisynth script. This is how it looked in VDubMod:

http://home.hiwaay.net/~dthiot/140310_RGB32_full.png

And a closer look:

http://home.hiwaay.net/~dthiot/140310_RGB32.png

Here is how the resulting .avi looked in MPC after encoding with the aforementioned settings:

http://home.hiwaay.net/~dthiot/140310_full.png

And a closer look:

http://home.hiwaay.net/~dthiot/140310.png

Look familiar to anyone? Of course it looks worse when the video is running.

[crusty]

Just a few quick questions:

-does this happen with every encode or just with this one?

-could you post the avs scripts you tried?

-is your desktop color depth 32 bit?

-I know they don't do much, but have you tried adjusting the overlay settings of your display driver?

-are you working on a TFT monitor? Some models might have issues with very subtle color variations.

-have you tried encoding that scene separately? If all else fails, you might try to encode the problematic clip with Quantizer 1 or 2, with a different avs script, and then paste it together to the rest of the movie (that process will work as long as you don't alter the resolution and certain XviD settings). You could also try with zones.

EDIT: looking at the pics icannot see the issue. They look a tiny bit different, but I cannot see the stair-stepped effect. Could you highlight it somehow (with drawing an arrow in Paint or something like that)?

[Soulhunter]

@gatormac

Yeah, its definitive caused through wrong chroma upsamling... ;)

But why have you chosen this blue area for the "closer look" ???

For me its much more visible in thus red light in the middle !!!

Maybe we have just a different color sensitivity... :confused:
Originally posted by crusty
Just a few quick questions:

-does this happen with every encode or just with this one?

-could you post the avs scripts you tried?

-is your desktop color depth 32 bit?

-I know they don't do much, but have you tried adjusting the overlay settings of your display driver?

-are you working on a TFT monitor? Some models might have issues with very subtle color variations.

-have you tried encoding that scene separately? If all else fails, you might try to encode the problematic clip with Quantizer 1 or 2, with a different avs script, and then paste it together to the rest of the movie (that process will work as long as you don't alter the resolution and certain XviD settings). You could also try with zones.

EDIT: looking at the pics icannot see the issue. They look a tiny bit different, but I cannot see the stair-stepped effect. Could you highlight it somehow (with drawing an arrow in Paint or something like that)?
Ehm... Can you really not see the difference ???

Really, no pre-processing and no encoder settings would help to solve his problem !!!

It can only be solved while de-coding... ;)


Bye

[Inc]

@ Gatormac

Look at the last post of that thread, there you can see that chroma stepps issue as its generally very noticable in at red edges:

http://forum.doom9.org/showthread.php?s=&threadid=68925&perpage=20&highlight=420&pagenumber=2

[sh0dan]

Originally posted by gatormac
As I said before, the .avs file looked stair-stepped pre-XVID within VDubMod, but I managed to make it look good in VDubMod by adding ConvertToRGB32() to the end of my avisynth script. This is how it looked in VDubMod:

As mentioned in the post Souldhunter linked to, this is due to chroma upsampling.

- When you convert from YV12 to RGB in AviSynth chroma is interpolated.

- When video is converted from YV12 to RGB (for vdub preview) by XviD it is NOT interpolated.

Thus chroma will look slightly worse when decoded by XviD for vdub preview with the same YV12 data.

Your display card works in YUV mode, so it doesn't matter how XviD converts to RGB, as it is never used. It is up to your graphic card to display this data as good as possible. ATI does this well - nVidia does not.

So this is not a real bug in XviD. Even if XviD changed it's 4:2:0 YUV upsampling when converting to RGB, it would look the same in your player. If you want vdub to look better, open the XviD file in avisynth using AviSource, and convertToRGB32().

[gatormac]

@Crusty

-does this happen with every encode or just with this one?
Just this one. All others look sweet.
-could you post the avs scripts you tried?
I didn't post the scripts because I thought I was clear in that I hadn't changed the final Gknot script, but that assumption is my fault, so here is the script (without #lines being shown):


LoadPlugin("c:\progra~1\gordia~1\mpeg2dec3.dll")
Mpeg2Source("school of rock.d2v")
crop(2,6,716,464)
undot()
LanczosResize(688,368)
ConvertToRGB32()


-is your desktop color depth 32 bit?
Yep.
-I know they don't do much, but have you tried adjusting the overlay settings of your display driver?
I've tried. The Nvidia driver adjustments are very simplistic and inadequate.
-are you working on a TFT monitor? Some models might have issues with very subtle color variations.
I'm working on a Samsung SyncMaster 955DF 19" CRT @ 1024x768.
-have you tried encoding that scene separately? If all else fails, you might try to encode the problematic clip with Quantizer 1 or 2, with a different avs script, and then paste it together to the rest of the movie (that process will work as long as you don't alter the resolution and certain XviD settings). You could also try with zones.
I'm glad you asked that, because I already pick out what I anticipate to be a difficult scene and zone it with a Q of 1. This scene was encoded with a Q of 1.
EDIT: looking at the pics icannot see the issue. They look a tiny bit different, but I cannot see the stair-stepped effect. Could you highlight it somehow (with drawing an arrow in Paint or something like that)?
I don't see how in the world you can't see the stair-stepping, but as everyone knows, different eyes see different things! Try looking along the outline or edges of the boys face, arms and hands in both cropped images. Also look at the edges of the fog machine. Very different from one image to the next to my eyes.

@Soulhunter

Wow. People really do see things differently! In my mind the blue has a much worse pixelated edge...the red looks fine to me :eek:

@Incredible

Thats interesting, but for some reason I am only seeing problems in blue. I must live in Bizzarroworld :)

@Shodan

Yeah, thats why I tried to make it clear that I wasn't reporting a bug...XVID is ready for final IMO. VDub already looks great as I have added the ConvertToRGB32(), but it just looks funky on playback in MPC (my favorite), and WMP9 for that matter (just to check ;) ).

[jang0]

You can't be that weird because I also think it's much more visible in the blue part than in the red.

[gatormac]

Originally posted by jang0
You can't be that weird because I also think it's much more visible in the blue part than in the red.

Thank you, thank you, thank you! Now I don't have to go to the eye doctor:D

I suppose that it is all just a limitation of my Geforce 2 card, so I can live with it for now until I get a better one. Thanks to everyone that tried to help me out.

Peace,

gatormac

[crusty]

Enlarging with irfanview I can see the difference now. But even then, I would not consider this a large issue, it looks more to me like 'normal' edge pixelation than stairstepping in pure color areas (If you get my drift). In the original picture I can't see it at all. Perhaps it is more noticable in the avi but I'd certainly consider this good quality.

Perhaps my eyes just aren't that sensitive to blue.

[Soulhunter]

Originally posted by gatormac
Wow. People really do see things differently! In my mind the blue has a much worse pixelated edge...the red looks fine to me :eek:

Originally posted by jang0
You can't be that weird because I also think it's much more visible in the blue part than in the red.

Hmm, thats interesting !!!

So far...

- 2 said -> Red !!!

- 2 said -> Blue !!!

- 1 said -> See nothing...

Could someone start a poll with a sample pic to test this ???


Bye

[Soulhunter]

Ok, Ive started a poll by my self... -> HERE !!! (http://forum.doom9.org/showthread.php?s=&threadid=72803)

Btw, Ive done a upsampling compare between AviSynth and my Radeon... -> HERE !!! (http://forum.doom9.org/showthread.php?s=&postid=459560#post459560)


Bye