Wavy pattern in a PAL DVD

[helge79]

So, I have this PAL DVD that displays a strange wavy pattern throughout. You can see it clearly, for example, looking at the vertical parts of the black frame of the photograph hanging on the wall in the video frame below:

http://img120.exs.cx/img120/9317/woainiinterlace8xk.jpg

I admit it may seem rather mild, but it draws my attention very much either in playback or in freeze-frame, and I would like to get rid of it. Of course, downsizing the video slightly makes the pattern less visible, but I wonder if anyone knows a more intelligent solution.

The DVD shows as PAL Interlaced in the DGIndex, but I haven't been able to find any genuine interlacing looking through the frames. On the other hand, the wavy pattern seems to appear in every frame. Is it some sort of interlacing artifact, or is its origin completely different?

[zettai]

Isn't that dot crawl?

[KaiserS]

Originally posted by zettai
Isn't that dot crawl?

Looks like that to me, but without seeing a source clip I can't tell just from the screenshot alone.

[helge79]

Thank you very much for the responses, zettai & KaiserS!
Here is a 1Mb VOB sample:

http://www.freefilehosting.net/file.cfm?id=24365BC2-CF1F-D571-ADD0780FA087D18A

Hopefully, it will help you to form a definitive opinion. :)

[helge79]

One other thing, the pattern that I am trying to get rid of
does not move. So it doesn't seem to match the descriptions
of the "dot crawl"...

:confused:

[scharfis_brain]

I don't know of which origin this weavy pattern derives from.
(maybe IVTC+Speedup NTSC->PAL conversion of some jittery (fields horizontally misaligned?!?) NTSC content? )

I found a solution to blur only the affected image areas, so that the weaves visually are gone if you do not search fro them with your magnification glasses:


loadplugin("d:\x\dgdecode.dll")
loadplugin("d:\x\masktools.dll")

#load mpeg
input = mpeg2source("helge.d2v")

#build mask of vertical edges
mask = input.DEdgemask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5",setdivisor=true,divisor=10,y=3,u=1,v=1)

#alter the mask to only cover the jaggy vertical edges
mask = mask.levels(10,1,40,0,255).greyscale().blur(1)

#prepare a vertical blurred clip
blurred= input.blur(0,1) #.blur(0,1)

#blur only vertical edges
maskedmerge(input, blurred, mask, y=3, u=1, v=1)

[kassandro]

Originally posted by helge79

The DVD shows as PAL Interlaced in the DGIndex, but I haven't been able to find any genuine interlacing looking through the frames. On the other hand, the wavy pattern seems to appear in every frame. Is it some sort of interlacing artifact, or is its origin completely different?
DGIndex is right. Though the material may be progressive, it was encoded as interlaced, i.e. the two fields were encoded separately. This results in inferior compression and inferior color quality. Probably the analog-digital conversion has also been done sepearately for both fields and then artifacts like in the above picture may easily happen.

Edit: Actually you can visualise this artificial interlace by very strong sharpening (it really should look ugly) of the video. Then will see the difference between both fields nearly everywhere.

[helge79]

Thank you so much, scharfis_brain! Here is the same frame processed with your script:

http://img229.exs.cx/img229/8448/sharfiswoaini906360cu.jpg

Looks much better, doesn't it?!! Maybe I should invest effort into developing the corresponding masking solution for your avatar. :D

I think it is unlikely that the pattern originated from an NTSC->PAL conversion. The movie is Chinese and the DVD has been made in China, so it's hard to imagine how NTSC would get into this. I guess then it must be some improper transfer from the analog source, just like kassandro says. :)

[helge79]

Originally posted by scharfis_brain

fields horizontally misaligned?!?


Yeah, I've been wondering if it's just that. I should probably try to shift the fields horizontally a little bit and see if it is getting me anywhere...

EDIT: OK, this is not gonna work. Not naively, at least. Here is a (4x) magnified detail from the unprocessed frame:

http://img136.exs.cx/img136/152/woaini90636detail4mr.jpg

It seems apparent that naive field-shifting will not yield any improvement...

[scharfis_brain]

Did you read my post entirely especially about what NOT to do?

maybe you want to try to include the commented 2nd blur(0,1) again. this should effectively knock out all weaves, but may leave some artifacts behind.

[KaiserS]

Originally posted by helge79
I think it is unlikely that the pattern originated from an NTSC->PAL conversion. The movie is Chinese and the DVD has been made in China, so it's hard to imagine how NTSC would get into this.

It would because China uses NTSC. Look for some R3 Chinese DVDs and you will see they are all NTSC. :)

[scharfis_brain]

China uses PAL for broadcasting.

Maybe there isn't spend much effort in converting the videos from NTSC to PAL. so they are leaved as NTSC !?

[KaiserS]

Originally posted by scharfis_brain
China uses PAL for broadcasting.

Maybe there isn't spend much effort in converting the videos from NTSC to PAL. so they are leaved as NTSC !?

Go to yesasia.com and look up some Chinese DVDs and alot of the ones I've looked at are NTSC. Heres one (http://us.yesasia.com/en/PrdDept.aspx/pid-1003940861/code-c/version-all/section-videos/did-106/) and another (http://us.yesasia.com/en/PrdDept.aspx/pid-1003954634/code-c/version-all/section-videos/did-106/) for example. Its rather confusing that they would have PAL for broadcast yet use NTSC on DVDs. Unless its just that HK uses NTSC while the mainland doesn't. And maybe that movie was made in Hong Kong?

[scharfis_brain]

http://www.lhvm.org/vidstand.html

[KaiserS]

So then my question is, if HK is PAL why are DVDs released as NTSC? That just doesn't make any sense. The publishers of the DVDs I linked are HK companies.

[scharfis_brain]

I don't know.
ask the producers.

Maybe Japanese influence?!?

[KaiserS]

Or maybe to be like Taiwan and South Korea. Either way, its just weird.

[kassandro]

I don't know what you want to do with the video, but if you want to recompress it, I would only apply blur(0,1). It certainly removes the field differences and you loose only a very moderate amount of detail. It's fast as well. If you blur with respect to an edge mask, then you have a lot of transitions between blurred and unblurred, which introduces new artifacts and results in poor compression. If you only want to capture a single frame with highest visual quality I would go along the lines of scharfis_brain.

[scharfis_brain]

I edited my script above.
It now blurs the mask even more. so there is no chance for transition artifacts.
But I didn't see any artifacts. Kassandro, did you?

also blur(0,1) doesn't seem to satisfy helge79

[kassandro]

@scharfis_brain:
When I read your script, the blur(1) was missing. By adding it you clearly avoid any visual transition artifacts and spatially your script is always superior over a simple blur(0,1). However, new temporal artifacts, i.e. flicker, are still possible, though they are also reduced by blurring the mask. Thus compression will be inferior compared with a simple blur(0,1).
Now if one wants achieve a fixed size by compression, then your script requires a higher average quantiser, which results in additional blurring and amplifies blockiness. The higher quantiser effect has always to be kept in mind, when designing scripts. It often leads to better overall results by trading some input detail in favor of a lower average quantiser.
If I read his postings correctly, then helge79 only objects downsizing, which only works well if one does a brutal 2:1 downsizing, which clearly is much more lossy than blur(0,1).

[helge79]

The DVD has been purchased from a web-store in the mainland China (www.dangdang.com), and my understanding is that all the DVDs there are PAL. On the other hand, all the HK and Taiwan DVDs I have are NTSC (just like those at yesasia.com). So I think the pattern is clear.

I am going to recompress the video, but I think I'd rather not blur the whole frames. Scharfi's blurred mask sounds like a very good idea. And it's true, I cannot see any artifacts anyway. :)

[helge79]

Originally posted by scharfis_brain

also blur(0,1) doesn't seem to satisfy helge79

Oh, I am sorry, I must have been unclear. The magnified detail I posted is from the unprocessed frame, and I was just thinking about the other possible solutions different from the edge mask blurring.

I think the very first script you posted does a good job, and blurring the mask will probably make it even better. :)

[KaiserS]

Just as an aside, what movie is this exactly?

[helge79]

Originally posted by KaiserS
Just as an aside, what movie is this exactly?

The movie is called I Love You (http://dvdtimes.co.uk/content.php?contentid=6274) (it's a story of troubled marriage set in the modern Chinese urban environment) and it has been made by Zhang Yuan (http://www.zhang-yuanfilms.com/EN/index.htm), one of the most prominent Chinese independent directors. For all I know, this mainland PAL DVD with a poor transfer is the only one that exists. :(

[helge79]

Originally posted by kassandro
I don't know what you want to do with the video, but if you want to recompress it, I would only apply blur(0,1). It certainly removes the field differences and you loose only a very moderate amount of detail. It's fast as well. If you blur with respect to an edge mask, then you have a lot of transitions between blurred and unblurred, which introduces new artifacts and results in poor compression.

I am not sure I can quite see this point. Blurring with respect to an edge mask will certainly produce a less compressible stream than blur(0,1). However, this stream will still be more compressible than the original one (since blurring with respect to an edge mask removes detail rather than introduces detail).

I can't see any difference then between what you're saying and suggesting to blur every video a little bit to improve its compressibility, which is not what you would usually want to do, since haziness tends to be more noticeable than a slight increase in blocking... Is there anything I am getting wrong here?

:confused:

[kassandro]

Originally posted by helge79
I am not sure I can quite see this point. Blurring with respect to an edge mask will certainly produce a less compressible stream than blur(0,1). However, this stream will still be more compressible than the original one (since blurring with respect to an edge mask removes detail rather than introduces detail).

I can't see any difference then between what you're saying and suggesting to blur every video a little bit to improve its compressibility, which is not what you would usually want to do, since haziness tends to be more noticeable than a slight increase in blocking... Is there anything I am getting wrong here?

:confused:
Compression depends on temporal and spatial continuity of the source. At least with a blurred mask spatial continuity should be improved by scharfis_brain's script. That is certainly positive as far as compression is concerned. On the other hand, it may easily happen that on one frame an edge is detected and on the subsequent frame the same edge is not detected, because ultimately edge detection depends on whether certain thresholds are exceeded or not. Consequently, on one frame the edge is blurred and on the subsequent frame it is not blurred. If this phenomenon is strong enough, it shows up as flicker. Thus temporal discontinuity may be increased by scharfis_brain's script. Because temporal continuity is more significant for compression than spatial continuity, it is theoretically possible that scharfis_brain's script worsens compression. I certainly don't expect this. But you may do tests on your own. Simply select single pass encoding in XviD, but instead of a fixed bitrate choose a fixed quantiser (I suggest a quantiser between 4 and 8) and encode without audio, say 5000, frames with scharfis_brain's original script, then with the blurred mask, then with blur(0,1) and finally without filtering. The smaller the resulting avi the more compression friendly the input. I expect the blur(0,1) avi to be clearly the smallest, followed by the avi obtained from the blurred mask, then the unfiltered avi and finally the avi obtained from scharfis_brain's original script. Of course you must always take the same 5000 frames and these 5000 frames should be typical for the entire video.
I prefer single pass encoding with a fixed quantiser over two pass encoding not only because it is a lot faster, but also because I have better control over the encoding process. If something goes wrong then usually the avi size explodes and you can make changes to the script and restart the encoding. With two pass encoding, you only realise a problem after finshing the encoding process, when the video is simply too blurry and too blocky.
Having encoded well over a thousand long videos with a quantiser 5 for I- and P-frames I certainly got a good feeling for compressability of a video.

[scharfis_brain]

ultimately edge detection depends on whether certain thresholds are exceeded or not.

this is not the case.
I do not create a binary mask.

[rotflol]

KaiserS, mainland China is PAL and region 6. HK is NTSC and region 3.

[KaiserS]

Originally posted by rotflol
KaiserS, mainland China is PAL and region 6. HK is NTSC and region 3.

Actually HK is PAL according to the site that scharfis_brain gave, but it doesn't really matter anymore since that was completely OT. :)

[helge79]

OK, here are the file sizes (10000 frames, XviD with quantizer 4):

unfiltered: 36,777,984 bytes
sharp mask with single blur(0,1): 35,969,024 bytes
soft mask with double blur(0,1): 35,592,192 bytes
blur(0,1): 34,949,120 bytes
I guess I'll go with some variant of the soft edge mask solution.

Once again, many thanks to everybody for discussing this matter. It has been practically useful for me, as well as educational. :)

[rotflol]

Originally posted by KaiserS
Actually HK is PAL according to the site that scharfis_brain gave, but it doesn't really matter anymore since that was completely OT. :)
OK, I meant HK DVDs. But for TV and video, sure, that's PAL. A lot of their DVDs are sourced from PAL video.

[KaiserS]

Originally posted by rotflol
OK, I meant HK DVDs. But for TV and video, sure, that's PAL. A lot of their DVDs are sourced from PAL video.

Yeah, I had already pointed that out in posts above. Once again there's not much point in still discussing it since it's completely OT.

[kassandro]

Originally posted by helge79

unfiltered: 36,777,984 bytes
sharp mask with single blur(0,1): 35,969,024 bytes
soft mask with double blur(0,1): 35,592,192 bytes
blur(0,1): 34,949,120 bytes[/size]
I guess I'll go with some variant of the soft edge mask solution.


I agree that the compression advantage of blur(0,1) is not big enough.

[scharfis_brain]

what about adding limitedsharpen with some moderate values?

eg:
limitedsharpen(wide=true,strength=70)

maybe you don't like it, because it also enhances the artifacts again.

[kassandro]

Originally posted by scharfis_brain
what about adding limitedsharpen with some moderate values?

Actually, you can sharpen a little bit. If the above 10000 frames are typical, then you have only an average 3500 bytes per frame with the high quality quantiser 4. Thus for 90 minutes you need only 400-500 MB for high quality encoding. The video must be pretty smooth. Thus there is some room for wasting compression with a sharpener.

[helge79]

Yeah, this clip is only so compressible because the image is blurry. So some sharpening would be a good idea. Thanks for the suggestion about which filter to use.

Also, to make the image size divisible by 16 after cropping and to correct for the rectangular shape of the PAL DVD pixels, I will have to downsize the video slightly (a few percent). It's not that bad, since it removes any remaining wavy artifacts. So, my understanding is that the workflow should be like this: cropping -> scharfi's soft edge mask blurring -> resizing -> limitsharpen. Are there any flaws with such an approach?

PS: I couldn't see the wavy artifacts come back after sharpening, so the entire procedure seems to be coherent.