Filtering MPEG2 video with "artefact ghosting", checkerboard patterns

[cihub]

I have an interlaced MPEG2 video and after IVTCing it there's some noticeable "artefact ghosting" (blocks from edges of previous frames lie on top of consequent as well as previous frames), checkerboard patterns etc. What could be the source of these artefacts and how to deal with them? At first I thought that that ghosting effect could've been caused by the bad IVTC, but ATM I'm not that sure. As to patterns, I've seen them even on BDs.

It seems like MVTools 2 can be handy here, but it smoothes important image texture as well.

Image sample (~150 KB): http://www.mediafire.com/view/pgsva8b8gs4g9gy/patterns.png
M2V video sample (~6.5 MB): http://www.mediafire.com/download/jp4frlj9ma7bmym/video-sample-for-doom9-filtering-thread.7z

[Asmodian]

How did you do the IVTC? Can you post a small sample of the original which shows the problem?

[Blue_MiSfit]

try fft3dfilter. I've seen it do amazing things with this kind of stuff.

Play with the different sigma parameters one at a time until you find the "knob" that kills the artifact in question.

[TheSkiller]

These artefacts (especially well visible in the top left corner of your first image) occur whenever the motion estimation of the MPEG encoder got confused and made a bad choice, even worse if the bitrate is rather limited.
This happens especially at fades and some encoders seem to be very prone for it while others don't seem to exhibit it at all.

Basically this is major damage on the video, DCT-aware deblocking will smooth it but not remove it.

[cihub]

Asmodian, it was AnimeIVCT(mode=3), nothing more. I choose it because with mode=1 sometimes deinterlacing was wrong. To be honest, I'm noob in this whole interlacing stuff (because of described artefacts I wanted to read more on the matter, but if TheSkiller is right it won't do much of a difference), so I've used the first filter that worked for me.
As to sample, I guess there's no use for it after TheSkiller's comment. If I'll not forget, and there will be any success in filtering these artefacts, I'll post .avs script.

Blue_MiSfit, thank you for advice. I tried not to use it, but since it works on per-block basis, it might resolve some of the issues, esp. checkerboard patterns.

TheSkiller, that's what I thought... It seems that my only hope here will be MVTools.

Update #1
So, after all, it seems that some of "artefact ghosting" is in fact wrong deinterlacing. After I've used AnimeIVTC with mode=2 (Field-Blended) it disappeared (everything became more obvious after spotting this frame (http://imageshack.us/a/img7/1628/ghosting.png)).

[manono]

So, after all, it seems that some of "artefact ghosting" is in fact wrong deinterlacing.
Which is exactly why you should always provide a sample when asking questions about processing something. Asmodian shouldn't even have had to ask. All those pictures were next to useless

[cihub]

manono, I wasn't sure whether it's legal etc. Updated first post with the sample and removed attachments since they was loading every time someone visits this thread.

[manono]

manono, I wasn't sure whether it's legal etc.
Uploading small samples is perfectly OK under Fair Use laws.

There's nothing difficult about that one. It doesn't need AnimeIVTC. This works perfectly well on the sample, after making a D2V project file using Honor Pulldown Flags in DGIndex:

TFM(D2V="Test.d2v")#adjust for how you name your D2V
TDecimate(Mode=1)

You shouldn't be using DirectShowSource. You may as well learn how to do it right - use DGIndex to make a project file before using the DGDecode.dll and MPEG2Source:

http://www.videohelp.com/tools/DGMPGDec

The included docs are outstanding and will be enough to get you going. If you have more questions, just ask.

[cihub]

>>>There's nothing difficult about that one. It doesn't need AnimeIVTC.
I've just tried your method and... well, if one doesn't use that debugging filter that I used in patterns.png etc., then it seems so. Yet, under more thorough examination it's obvious that there's still some residual interlacing left.

1st frame filtered with TFM: http://www.mediafire.com/view/6xjlbaqijzdl4lo/filtered-frame1.png
2nd frame filtered with TFM: http://www.mediafire.com/view/61n76x1xw29g918/filtered-frame2.png
1st frame filtered with AnimeIVTC (mode=2, degrain=3, aa=1, bbob=4): http://www.mediafire.com/view/4yfyfq8751ytjip/animeivtc-mode2-frame1.png
2nd frame filtered with AnimeIVTC (mode=2, degrain=3, aa=1, bbob=4): http://www.mediafire.com/view/h41h1xh0ws0cu11/animeivtc-mode2-frame2.png

That's exactly why I was so confused in the first place. Also, there's a difference in one frame between two clips (at the time these snapshots were made), which might explain something to those who know something about this stuff.

As for DirectShowSource + M2V, I used them because I couldn't figure out how to make D2V sample that will work w/o VOBs.

Anyhow, I really appreciate your help.

[manono]

Yet, under more thorough examination it's obvious that there's still some residual interlacing left.
Are those pics from the sample? If not then how is anyone supposed to help with anything? That sample had only one place (right before the grey writing appeared in the middle of the screen) that was interlaced video, while the rest was film that didn't need any IVTC, by AnimeIVTC or anything else.

When testing that script I gave, did you use a D2V file and MPEG2Source? I'm unclear on that.
I couldn't figure out how to make D2V sample that will work w/o VOBs.
I don't get it. Your source is a DVD, right? So you have the VOBs, right? That's right, it doesn't work with anything but MPEG-1 and MPEG-2 video. Even if the source isn't a DVD, DirctShowSource shouldn't be used except as a last resort. I usually use FFVideoSource for other stuff.

[cihub]

>>>That sample had only one place (right before the grey writing appeared in the middle of the screen) that was interlaced video [...]
Sorry for that. As I've said -- I'm noob in this stuff. I thought that any piece of that video will suffice. Here's proper sample: http://www.mediafire.com/download/5s9stat5am0vy46/video-sample2-for-doom9-filtering-thread.7z

>>>When testing that script I gave, did you use a D2V file and MPEG2Source?
Yes.

[paradoxical]

I ran your sample with simply:

tfm(mode=3,pp=0)
tdecimate(mode=1)

Only issue I saw was compression artifacts already present in the source because they were also visible on the non-IVTCed version.

[cihub]

Hm... That's weird. No matter what I do (use original VOBs with MPEG2Source or M2V with DSS) I still get something like filtered-frame1.png @ mediafire.

[paradoxical]

Unless you can see the issue during normal playback without enhancing the artifacts, is it really worth being so obsessive about? I didn't notice anything beyond some edge noise due to compression artifacts while actually playing the video.

[manono]

That second sample is better as it's all hard telecine and includes the frame you wanted us to look at, but it still IVTC's easily. I zoom in on the frame you showed in luma (?) and it looks just fine. TIVTC has no trouble with the sample and field-matches perfectly. As paradoxical mentioned, about the only things out of the ordinary are the compression artifacts that look like mosquito noise to me, and I don't see any residual interlacing.
well, if one doesn't use that debugging filter that I used in patterns.png etc., then it seems so.
I, for one, refuse to examine video under whatever debugging filter you use. If you can't see it without putting it under an X-Ray, it's not there.

[cihub]

You two are right, but I want to remove any unnecessary detail that will make file size bigger than it should. The less there will be artefacts, the higher CRF (in x264) I could use.

And, yes, manono, that was luma.

P.S. I've just tried to use AnimeIVTC(mode=2, degrain=1, aa=0, bbob=1) and the results were somewhat close to tfm(mode=3,pp=0).tdecimate(mode=1), though interlacing is still much less pronounced. Nevertheless, AnimeIVTC suits me best (ATM I don't know how to deal with those artefacts otherwise).

[paradoxical]

Just use a deblocker or noise remover that will take care of those invisible artifacts you are so worried about.