Proper deinterlacing of badly encoded movie

[Dust Signs]

Hi,

I recently purchased a DVD of a movie I was looking for a long time, and as I backup all my movies using AviSynth and x264 I also tried doing so with this one.
The movie itself is somewhat old, but - much worse - the quality of the DVD is catastrophic. It looks as if some intern recorded the VHS version of the movie using a one click VHS to DVD program.
Anyways, I wanted to backup the movie, but I either don't get the deinterlacing right, or some sort of stuttering about once per second. I am aware of telecine and interlacing, but the main problem is that the DVD itself is encoded as 25fps PAL (progressive) content, with those awful combing artefacts inside.
I decrypted the movie and indexed it with DGIndex, resulting in a picture like this on slight movement in the middle of the movie:
http://temp.dustsigns.de/original.jpg
After applying the following filters (I experimented around a bit) I get the following result which removes a lot of the interlacing artefacts, but is still blurry and quite jerky about once per second. Here is the script and the same picture as before, in its filtered version:
MPEG2Source("F:\TEST\VIDEO_TS\VTS_01_1.d2v")
SmoothDeinterlace()
Crop(18,136,-14,-128)
fft3dfilter(sigma=1.5, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=2.0)
Lanczos4Resize(688,288)

http://temp.dustsigns.de/filtered.jpg
A lot of cropping was necessary as on the left and right there was some kind of gray shade which I only know from old VHS recordings. The sharpening of FFT3D might seem quite high, but if I lower the value, the picture is way too blurry.
I also tried differenent deinterlacing filters (yadif, tdeint,...) but it seems as if the deinterlacing artefacts get worse when the blurring and stuttering gets better and vice verse, depending on the filter I am using. Does anyone know how to deal with this kind of bad quality material in a reasonable way? I already wrote an email to the production studio but they don't seem to care at all about the **** they encoded here. I would be thankful for any kind of help.

Dust Signs

[neuron2]

If it's progressive PAL with field shifts, then just use a field matcher by itself, such as Telecide(). If it is field blended, things are trickier. For a definitive answer, post a link to an unprocessed source sample that shows the problem. Set a range in DGIndex, then Save Project and Demux Video. Upload the M2V to www.mediafire.com and post the link here. 50MBytes is fine.

[Dust Signs]

As I'm unsure about the legal situation of uploading video content from a consumer DVD I uploaded a part of the end credits - as the quality and severity of artefacts does not vary throughout the movie I hope this is ok. The file should be finished within the next minutes: http://www.mediafire.com/?sharekey=69cc812d0f34637b0de4fc1039a01674136f2dcb5931f8da5be6ba49b5870170

Dust Signs

[Inspector.Gadget]

Dust Signs, often the credits are mastered differently than the rest of the film and so have a different field order, blending, etc.

[neuron2]

Small samples are allowed under the fair use clause of the Copyright law. Upload a part from the main movie that shows the problem.

Your credits upload is useless. There's not even any video in it, just scrolling credits on a black background.

[Dust Signs]

Thanks for the note about the copyright law, neuron2. Should be finished in about 10 minutes: http://www.mediafire.com/?sharekey=69cc812d0f34637b0de4fc1039a01674e83903202efedbd4c95965eaa7bc68bc

Dust Signs

[neuron2]

It's field-blended garbage from a poor standards conversion. I don't mess with stuff like that, so you'll have to hope someone that does sees this.

You could do a search on "blended fields".

[Dust Signs]

It's field-blended garbage from a poor standards conversion. I don't mess with stuff like that, so you'll have to hope someone that does sees this.

You could do a search on "blended fields".

Thanks for the hint. I hope someone can help me with that, as I looked for years for this (unrated) version of the DVD and paid 30 EUR for it. So I hope to get the picture quality to a level I can view

Dust Signs

[audyovydeo]

interesting case. I'm no expert, but let's try this : if it's a non-deinterlaced video turned into progressive, then you can apply an interlacing function (without anything else) and the artifacts *should* (*might* ?) get correctly separated across fields. You could then try to deinterlace and see what gives.

I'd be curious to know the result.

cheers
a/v

[Dust Signs]

@audyovydeo: Isn't that what I did in the filtered image in my first post? Or do I miss something?

Dust Signs

[scharfis_brain]

get the avisynth function srestore() as well as a good bob-deinterlacer, like
tdeint() or yadif() [yadifmod()]

then set up the deblending.
I haven't looked at the sample yet.

[Dust Signs]

@scharfis_brain: Hm... thanks for the hint, but the stuttering is getting really bad now. Here is the script I'm currently using:

MPEG2Source("F:\Test\VIDEO_TS\VTS_01_1.d2v")
LoadCplugin("C:\Users\MyName\AppData\Local\StaxRip\Applications\Yadif\yadif.dll")
Yadif()
Crop(18,136,-14,-128)
fft3dfilter(sigma=1.0, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=1.5)
Lanczos4Resize(688,288)


I hope I got it right.

Dust Signs

[scharfis_brain]

you haven't read my previous post entirely.
1) use bob (fullrate) deinterlacing
2) use srestore()

edit:
yep. simple stupid fieldblending from telecined NTSC film to PAL.

solution:

loadplugin("c:\x\mt_masktools.dll")
loadplugin("c:\x\yadifmod.dll")
loadplugin("c:\x\nnedi.dll")
import("c:\srestore.avs")

mpeg2source("VTS_01_1.demuxed.d2v")

i=crop(18,136,-14,-128,align=true)

i.yadifmod(mode=1, edeint=i.nnedi(field=-2))
srestore(dclip=i.bob().reduceby2())
lanczosresize(688, 288)

[audyovydeo]

@audyovydeo: Isn't that what I did in the filtered image in my first post? Or do I miss something?



The initial premise was wrong :

... but the main problem is that the DVD itself is encoded as 25fps PAL (progressive) content, with those awful combing artefacts inside.

hence my suggestion is not applicable.


cheers
audyovydeo

[Dust Signs]

:eek: Amazing, scharfis_brain. Thank you very much. This removes the stuttering and the combing artefacts. The video quality in total is not really good, but at least the most annoying artefacts could be removed. Thank you very much :)

Dust Signs

[laserfan]

:eek: Amazing, scharfis_brain. Thank you very much. This removes the stuttering and the combing artefacts.Thanks for posting-back Dust Signs--I have a bad internet connection or would have dl'ed your clip and tried this myself.

In my view the stuff scharfis_brain comes-up with is often sufficiently advanced to be considered "magic"!

[Dust Signs]

Unfortunately AviSynth crashes at about half the way when encoding the whole movie and I can't figure out why. I tried to narrow down the scene in which the error occurs but that I can't as the error only occurs if I encode the whole movie - 20% out of the middle (where the error occurs) work fine. Maybe there is an error in my current script? Removing the FFT3D line does not change anything - that's what I was able to test until now.

MPEG2Source("F:\Test\VIDEO_TS\VTS_01_1.d2v")
i=Crop(18,136,-14,-128,align=true)
i.yadifmod(mode=1, edeint=i.nnedi(field=-2))
srestore(dclip=i.bob().reduceby2())
fft3dfilter(sigma=0.5, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=0.5)
Lanczos4Resize(688,288)

I downloaded the latest versions of all the plugins involved. Anyone any idea?

Dust Signs

[manono]

Anyone any idea?
Encode it in sections (perhaps as lossless, Lagarith, maybe) and join them afterwards?

[Dust Signs]

As I just noticed there seems to be an absurdly high amount of memory to be leaking when using the AviSynth script posted above:
http://temp.dustsigns.de/memleak.jpg
I didn't notice this before as I have 8 GB of RAM in my machine, so 2 missing GB don't hurt that much. Any idea how to narrow that down?

Dust Signs

[neuron2]

Add the filters successively and see which one causes it. That is:

A

then

A
B

then

A
B
C

etc.

[Dust Signs]

Removing the line with srestore the memory leak disappears.

Dust Signs

[Leak]

I didn't notice this before as I have 8 GB of RAM in my machine, so 2 missing GB don't hurt that much.
Oh, but it does hurt you - because that's why it's crashing mid-encode: your 32-bit encoding app has exhausted the 2 GB of address space that 32-bit process can use but still wants to allocate more memory and gets killed by Windows (or at least the runtime library).

np: Autechre - Nine (Amber)

[Lyris]

Eugh, standards converted field-blended content.
I wonder why they chose to master it this way - incompetence, or is the film music-based?

[Dust Signs]

Oh, but it does hurt you - because that's why it's crashing mid-encode: your 32-bit encoding app has exhausted the 2 GB of address space that 32-bit process can use but still wants to allocate more memory and gets killed by Windows (or at least the runtime library).

I know that. What I meant was that I didn't notice it in terms of free memory in total ;)
@Lyris: I already compained about that, but the producers don't seem to care at all. Consequence for me: don't ever buy DVDs produced by them any more.

Dust Signs

[unix_sansei]

add /3GB to your boot.ini file
reboot

[Dust Signs]

add /3GB to your boot.ini file
reboot

Besides the fact that I doubt that all libraries and executables involved can handle the 3GB (as an extra flag has to be set in the PE header in order to actually use the extra addresses) modifying boot.ini won't help as I'm using Vista 64 Bit. Of course, there is bcdedit and the increaseuserva value, but wouldn't it be more meaningful to resolve this bug by sending a bug report to whoever is responsible (the author of srestore? I'm not sure)? Even if the 3 GB are used, around 70% of the movie will be encoded before the encoding process crashes. 100% would be nice(r) ;)

Dust Signs