Getting bad jaggies, how do I fix?

[Hotpocketdeath]

I've started using x264 w/ MeGUI and every video I have encoded has these jaggies coming up I can't seem to get rid of.

http://www.malamber.org/gaw/tmp/Capture9-20-2007-10.31.10%20PM9-20-2007-10.31.30%20PM11-23-2007-3.31.13%20AM11-23-2007-3.31.53%20AM11-23-2007-3.33.46%20AM.jpg

I've tried color conversions and some other tricks, but nothing is helping.

Is there a way to get rid of them?

[nm]

Looks like poor deinterlacing. I don't know what kind of filters MeGUI uses by default, but you can probably override them with better (and heavier) AviSynth filters or scripts. Or, if your source is not interlaced, no deinterlacers should be used.

[Hotpocketdeath]

Source is progressive and no de-interlacer is turned on. At least that's how it gets scanned by MeGUI

[foxyshadis]

Might be poor deinterlacing on the authoring house's side, sadly. Have a short sample of the source you can post?

[nm]

On closer inspection, seems that there are jaggies in both horizontal and vertical edges (a bit hard to tell from this sample image though). Maybe you are using a bad resizing method?

[Hotpocketdeath]

I've used a couple of resizers. Lancos, Bicubic, bilinear. All of them have the same results.

And I have tried out a couple of de-interlacers and all them have the same results.

I'll see about getting a short clip together.

And btw, that picture is zoomed in. It seems that the jaggieness is happening on a pixel level. And to NM, yah, the issues isn't just on horizontal surfaces. In the red and green areas, it seems to do it vertically which seems to tell me it's not an interlacing issue.

The main reason why I started noticing this was I have been playing the movies through my PS3 on a 47inch 1080p TV. Blowing up the video to that size made it noticeable.

Edit: Looks like this isn't just in h.624. I took a closer look at some of my divx and xvid movies and they have the same issues. just that it was covered up more by artifacts within divx and xvid so it wasn't as noticeable.

Looks like unless I encode at high resolutions, this is going to appear in all the encodes no matter what filters I use.

[nm]

Ok, now I can see where the problem is: bad resizing on playback. PS3 probably uses bicubic (or maybe just bilinear) scaling, which is good enough for most people, but when resizing SD to a large HD screen, better methods (lanczos) should be used.

As you said, if you want to use PS3 for playback and don't run Linux on it, the only way to avoid this problem is to resize to HD resolution before encoding, by using good resizers and maybe sharpening filters. This will eat a lot of extra bitrate, but after some tinkering, you can get very good results. Search for Didée's LimitedSharpen and SeeSaw, for example.

When using a PC for playback, you should use at least lanczos resizing with ffdshow or MPlayer (which also works on PS3/Linux).

Finally, make sure you're feeding the TV at its native resolution; otherwise it will also resize and the methods used in some TVs are quite primitive. I've been under the impression that PS3 actually has a pretty good scaler for SD video (http://www.firingsquad.com/hardware/autumn_2007_video_shootout/page3.asp) (although aliasing problems are mentioned in this shootout too), so it could also be your TV that causes the jaggies.

[Hotpocketdeath]

I don't think it's the PS3 or the TV. I can see the same thing on my PC w/ monitor.

The shot I took is of the source video using the AVIsynth preview with all the filters plugged in. Before I encoded the video.

So the problem is coming up long before the playback stage. And like I said, I have used every resize filter in MeGUI's arsenal and all of them show the jaggies.

I'm almost convinced this is something that is within the source videos. As the only resizing I am doing is only to correct the Aspect ratio.

Edit: Ok, I just found that is in fact coming from the source. So looks like I was doing everything right, just that the sources had the problems.