Correcting colors after deblocking (using Masks?)

[redfordxx]

I started the discussion in another thread, but this was definitely off topic, so I made new thread of it.
OK, here is my problem... real challenge.

I use MSU Deblocker for deblocking my LOTR TTT EE (I have a special blocky version:(). As MSUDeblocker is VDMod plugin, I am forced to go to RGB32:
deblocked=orig.ConvertToRGB32(matrix="pc.601").MSUDeBlock(1,1,1).ConvertBackToYUY2(matrix="pc.601").ConvertToYV12()

MSUDeblocker works mostly perfectly, but sometimes produces ugly artefacts, as follows...

I show you pictures 3 frames with color analysis.
Here are the expressions explained:
o=original
d6=deblocked with MSU in RGB via 601
d7=deblocked with MSU in RGB via 709
s6=Subtract(o,d6)
s7=Subtract(o,d7)
s67=MergeChroma(s7,s6,0.5)


Frame with badly processed areas:
Matrix (how the captures are placed):
...o d6 d7 (YUV)
s67 s6 s7 (contrastU=4096)
s67 s6 s7 (contrastV=4096)
s67 s6 s7 (contrastY=4096)
http://img175.exs.cx/img175/566/frame101900210hz.th.png (http://img175.exs.cx/my.php?loc=img175&image=frame101900210hz.png)
http://img175.exs.cx/img175/566/frame101900210hz.png
As you see mostly on sky are big areas of totally disorted color.
Interesting is that when I merge pc.601 and pc.709 converted frames, some disorted areas are anihilated. (Visible especially on the V-channel)


Frame with badly processed areas:
.o d7 (YUV)
s6 s7 (contrastU=4096)
s6 s7 (contrastV=4096)
s6 s7 (contrastY=4096)
http://img34.exs.cx/img34/7205/frame103300350mc.th.png (http://img34.exs.cx/my.php?loc=img34&image=frame103300350mc.png)
http://img34.exs.cx/img34/7205/frame103300350mc.png
Here is interesting that artefacts in the sky in U(s6) reappear in V(s7) and Y(s7) in the same shape...


Correctly processed frame:
.o d7 (YUV)
s6 s7 (contrastU=4096)
s6 s7 (contrastV=4096)
s6 s7 (contrastY=4096)
http://img34.exs.cx/img34/7234/frame442034220il.th.png (http://img34.exs.cx/my.php?loc=img34&image=frame442034220il.png)
http://img34.exs.cx/img34/7234/frame442034220il.png
Now I want to find a way how to detect the bad areas a replace them. I suppose to use MaskTools, I have only to figure out the exact way. One thing I know for sure, it will be slowww.


Well, don't know theter it is clear. Just wanna keep deblocking effect and remove artefacts.

[rfmmars]

I don't know if there is a solution to the problem. In processing 60 year old film I soon came to the conclusion that I have to expect some artifacts. A good example is the old ACD-see "AutoLevels"

For the few times it's misses the mark, I am rewarded with very useable film, and that what my customer wants. If it is perfection your after, you will be disappointed many times. Hand editing those bad frames is the solution

richard
photorecall.net

[redfordxx]

New findings:
On the clip with differences, e.g. s6:
1. the deblocked areas are characterized by values around 128 (+-cca 3 for very blocky spots, usually +-1 or 2)
2. the bad areas have value either 127 or 129, in U,V colors (or is it called plane?). That is +1 or -1 from grey.

So, the task is reduced to finding out which areas with color 127 or 129 are the wrong ones.

The idea came to me was to find where the color is concentrated, create mask, then expanding corrupted areas one block in all directions to make sure no edges will remain.

[redfordxx]

First idea...
you don't need to read this;) I have just few questins in next post, for finetuning this and optimize this. So this FYI.

LoadPlugin("C:\PROGRA~1\MULTIM~1\GORDIA~1\AviSynthPlugins\MaskTools.dll")
LoadPlugin("C:\PROGRA~1\MULTIM~1\GORDIA~1\AviSynthPlugins\dgdecode.dll")
LoadVirtualdubPlugin("C:\Program Files\Multimedia\GordianKnot\VirtualDubMod\plugins\deblock.vdf","MSUDeBlock")

#producing original and deblocked video
mpeg2source("D:\record\rip\lotr tt\1\tt1.d2v")
o=crop(8,80,704,416)
d6=o.ConvertToRGB32(matrix="pc.601").MSUDeBlock(1,1,1).ConvertBackToYUY2(matrix="pc.601").ConvertToYV12()

#finding differences
s6=Subtract(o,d6)

#producing mask for colors 129 and 127 occurence, 4xdownsizing, so one 2x2 pixel represents common block size in full resolution (8x8) (does not apply to Y, where the pixel size is 1x1 --- then 8xdownsize needed)
color1=129
color2=129
x1=SubTract(s6.Binarize(color1-1,u=1,v=3,y=1,upper=false),s6.Binarize(color2,u=1,v=3,y=1,upper=false)).ReduceBy2.ReduceBy2
color1=127
color2=127
x2=SubTract(s6.Binarize(color1-1,u=1,v=3,y=1,upper=false),s6.Binarize(color2,u=1,v=3,y=1,upper=false)).ReduceBy2.ReduceBy2
#joining masks together (subtract or merge?) Now create mask which shows amount of colors 127 and 129 with it’s intensity
x3=SubTract(x1,x2)


#selecting areas with colors between some threshold values. These areas have correct deblocking
color1=50
color2=255-color1
x4=SubTract(x3.Binarize(color1-1,u=1,v=3,y=1,upper=false),x3.Binarize(color2,u=1,v=3,y=1,upper=false)).Binarize(128,u=1,v=3,y=1,upper=false)

#expanding areas with color=0 (=bad areas) and resizing to full video size
x5=x4.ConvertToYUY2.SpatialSoften(1,256,256).ConvertToYV12()
x6=x5.Binarize(230,u=1,v=3,y=1,upper=false).Pointresize(704,416)

#proving on blank clip the result. Areas evalulated as bad are taken from blank clip, othervise deblocked clip is taken
b=BlankClip (length=21000, width=704, height=416, pixel_type="YV12",
fps=25.000, color_yuv=$808080)
fin=MaskedMerge(b,s6,x6,u=1,v=3,y=1)


#Showing the result and steps to achieve it, to look at in VDM
StackHorizontal(StackVertical(d6,fin.ColorYUV(cont_v=8192),s6.ColorYUV(cont_v=8192)),\
StackVertical(x6,x5.Pointresize(704,416),x4.Pointresize(704,416)),\
StackVertical(x3.Pointresize(704,416),x2.Pointresize(704,416),x1.Pointresize(704,416)))

[redfordxx]

Is there more efficient way to create mask based on occurence of a range of colors in a frame? Special case could be one color only (then color1=color2). Now I usecolor1=127
color2=129
x1=SubTract(s6.Binarize(color1-1,u=1,v=3,y=1,upper=false),s6.Binarize(color2,u=1,v=3,y=1,upper=false))
I want to expand areas in the mask. Examples:

oooo
oxoo
oooo
oooo

make into (different based on some threshold)

xxxo oxoo oooo
xxxo xxxo oooo
xxxo or oxoo or oooo
oooo oooo ooooI tried to useclip=clip.ConvertToYUY2.SpatialSoften(1,256,256).ConvertToYV12().Binarize(threshold,u=1,v=3,y=1,upper=false)But it gave wierd results. Either it is subjective or it doesnot do what I expect.
Maybe I need spatial average weighted like this or similar.
121
232
121