Deinterlacing PAL gives wave artifacts

[StainlessS]

Here, removed YAD (Brightness diff metric), and added frames file writing for singlerate, and doublerate even and doubrate odd scene change.

Voting thresholds not tweaked [I only got a single scene change clip].


Avisource("clip3.mkv.avi")
o=last
assumetff()

### Config ###
LPD_Th = 10 # RT_LumaPixelsDifferent() Threshold, if corresponding pixels more different than this then is counted
# Voting Thresholds, if above these then VOTE is counted
V_YD_TH = 30.0
V_YCD_TH = 100.0
V_YPD_TH = 100.0
###
SUBS=True
WRITE=True
SFrames = ".\SingleRate.txt"
DFramesE = ".\DoubleRateEven.txt"
DFramesO = ".\DoubleRateOdd.txt"
############################
############################
############################

if(WRITE) {
RT_fileDelete(SFrames)
RT_fileDelete(DFramesE)
RT_fileDelete(DFramesO)
}

COUNT = 0 # COUNT valid only on forward ONLY scan
SSS="""
n = current_frame
YD = RT_LumaDifference(Last,Last,n=n,y=0,n2=n,y2=1,interlaced=true) # Average Per Pixel Diff betwwen Even and Odd fields
YC = RT_LumaCorrelation(Last,Last,n=n,y=0,n2=n,y2=1,interlaced=True) # Correlation Between Even and Odd fields
YCD = (1.0-YC)*255.0 # Difference by Correlation [0.0->255, max poss 512.0=-1.0 -ve correlation]
YPD = RT_LumaPixelsDifferent(Last,Last,n=n,y=0,n2=n,y2=1,interlaced=true,Thresh=LPD_Th) # Population [0->255.0] where pixels different by LPD_Th or more
VOTE_YD = (YD > V_YD_TH) ? 1:0
VOTE_YCD = (YCD > V_YCD_TH) ? 1:0
VOTE_YPD = (YPD > V_YPD_TH) ? 1:0
VOTE_CNT = VOTE_YD + VOTE_YCD + VOTE_YPD
SC = VOTE_CNT >= 2
COUNT = (SC) ? COUNT + 1 : COUNT
(SUBS)?RT_Subtitle("%d]\nYD =%7.3f\nYCD=%7.3f\nYPD=%7.3f\nVOTE_YD =%.1s\nVOTE_YCD=%.1s\nVOTE_YPD=%.1s\nVOTE_CNT=%d\nSceneChange=%s (COUNT=%d)",
\ n,YD,YCD,YPD,VOTE_YD!=0,VOTE_YCD!=0,VOTE_YPD!=0,VOTE_CNT,SC,COUNT) : NOP
if(WRITE && SC) {
RT_WriteFile(SFrames,"%d",n,Append=True)
RT_WriteFile(DFramesE,"%d",n*2,Append=True)
RT_WriteFile(DFramesO,"%d",n*2+1,Append=True)
}
Return Last
"""
Scriptclip(SSS)
#StackHorizontal(o,Last)
Return last

EDIT: Slight mods to comments

Frame 103
https://i.postimg.cc/50NyXWVW/f103.jpg (https://postimages.org/)

Frame 104
https://i.postimg.cc/Rh2Zp3nv/f104.jpg (https://postimages.org/)

Frame 105
https://i.postimg.cc/LXb9hqDj/f105.jpg (https://postimages.org/)

.\SingleRate.txt
104
.\DoubleRateEven.txt
208
\DoubleRateOdd.txt
209

Could just use result of BOB() [Maybe SelectEven/Odd if singlerate] to fix scene change frame using ClipClop(cmd=Frames) after your dienterlace pass.

[zorr]

mirror of both clips
https://www.mediafire.com/file/7ewphbxi0h82ebr/atlantis_wave_artifact.zip/file

Also maybe look at
http://www.avisynth.nl/users/vcmohan/FFTQuiver/F1Quiver.htm
http://www.avisynth.nl/users/vcmohan/FFTQuiver/F2Quiver.htm

Thanks, those links work. I will try the Quivers too but I'd like to start with a simpler filter just to see how it performs.

Annoyingly I cannot get even Defreq to work, it's the classic "FFT3DFilter: Can not load FFTW3.DLL". It used to work with 32bit Avisynth but not with the 64bit one which I'm using now. I have updated fftw3.dll in Windows/System32 to a 64bit version but I suspect I still may have a wrong version. Could someone provide a link to a version that is known to work with 64bit Windows 10 and 64bit Avisynth? Here's the log from AvisynthInfoTool if it helps:

[OS/Hardware info]
Operating system: Windows 10 (x64) (Build 19041)

CPU: AMD Ryzen 9 3900X 12-Core Processor / Ryzen 9 (Matisse)
MMX, MMXEXT, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, SSE4A, FMA3, RDSEED, ADX, AVX, AVX2
12 physical cores / 24 logical cores


[Avisynth info]
VersionString: AviSynth+ 3.7.0 (r3382, 3.7, x86_64)
VersionNumber: 2.60
File / Product version: 3.7.0.0 / 3.7.0.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\WINDOWS\SYSTEM32\avisynth.dll
Avisynth.dll time stamp: 2021-01-11, 21:36:39 (UTC)
PluginDir2_5 (HKLM, x64): E:\AviSynth\AVSPLUS370_x64\plugins
PluginDir+ (HKLM, x64): E:\AviSynth\AVSPLUS370_x64\plugins


[C++ 2.5 Plugins (64 Bit)]
E:\AviSynth\AVSPLUS370_x64\plugins\defreq.dll [2007-05-22]

[C++ 2.6 Plugins (64 Bit)]
E:\AviSynth\AVSPLUS370_x64\plugins\AddGrainC.dll [1.8.3.0]
E:\AviSynth\AVSPLUS370_x64\plugins\avsresize.dll [0.0.0.0]
E:\AviSynth\AVSPLUS370_x64\plugins\avstimer.dll [2018-09-11]
E:\AviSynth\AVSPLUS370_x64\plugins\ConvertStacked.dll [2021-01-11]
E:\AviSynth\AVSPLUS370_x64\plugins\DePan.dll [2.13.1.6]
E:\AviSynth\AVSPLUS370_x64\plugins\DePanEstimate.dll [2.10.0.4]
E:\AviSynth\AVSPLUS370_x64\plugins\DirectShowSource.dll [2021-01-11]
E:\AviSynth\AVSPLUS370_x64\plugins\EEDI2.dll [1.0.0.0]
E:\AviSynth\AVSPLUS370_x64\plugins\eedi3.dll [0.9.2.3]
E:\AviSynth\AVSPLUS370_x64\plugins\ffms2.dll [2020-08-22]
E:\AviSynth\AVSPLUS370_x64\plugins\fft3dfilter.dll [2.8.0.0]
E:\AviSynth\AVSPLUS370_x64\plugins\ImageSeq.dll [2021-01-11]
E:\AviSynth\AVSPLUS370_x64\plugins\KNLMeansCL.dll [2021-04-19]
E:\AviSynth\AVSPLUS370_x64\plugins\masktools2.dll [2.2.26.0]
E:\AviSynth\AVSPLUS370_x64\plugins\mvtools2.dll [2.7.44.0]
E:\AviSynth\AVSPLUS370_x64\plugins\neo-dfttest.dll [1.0.0.0]
E:\AviSynth\AVSPLUS370_x64\plugins\neo-fft3d.dll [1.0.0.0]
E:\AviSynth\AVSPLUS370_x64\plugins\nnedi3.dll [0.9.4.57]
E:\AviSynth\AVSPLUS370_x64\plugins\RawSourcePlus.dll [1.0.0.0]
E:\AviSynth\AVSPLUS370_x64\plugins\RgTools.dll [1.1.0.0]
E:\AviSynth\AVSPLUS370_x64\plugins\SangNom2.dll [0.6.0.0]
E:\AviSynth\AVSPLUS370_x64\plugins\Shibatch.dll [2021-01-11]
E:\AviSynth\AVSPLUS370_x64\plugins\TDeint.dll [1.8.0.0]
E:\AviSynth\AVSPLUS370_x64\plugins\TimeStretch.dll [2021-01-11]
E:\AviSynth\AVSPLUS370_x64\plugins\VDubFilter.dll [2021-01-11]
E:\AviSynth\AVSPLUS370_x64\plugins\yadifmod2.dll [0.2.7.0]

[Scripts (AVSI)]
E:\AviSynth\AVSPLUS370_x64\plugins\AiUpscale.avsi [2020-07-25]
E:\AviSynth\AVSPLUS370_x64\plugins\colors_rgb.avsi [2020-03-02]
E:\AviSynth\AVSPLUS370_x64\plugins\GMSD.avsi [2021-04-06]
E:\AviSynth\AVSPLUS370_x64\plugins\IQA_downsample.avsi [2021-03-27]
E:\AviSynth\AVSPLUS370_x64\plugins\MDSI.avsi [2021-03-27]
E:\AviSynth\AVSPLUS370_x64\plugins\mtmodes.avsi [2021-04-07]
E:\AviSynth\AVSPLUS370_x64\plugins\QTGMC.avsi [2021-04-19]
E:\AviSynth\AVSPLUS370_x64\plugins\RoundHalfToEven.avsi [2021-03-27]
E:\AviSynth\AVSPLUS370_x64\plugins\Zs_RF_Shared.avsi [2021-04-19]

[Uncategorized DLLs (64 Bit)]
E:\AviSynth\AVSPLUS370_x64\plugins\libfftw3-3.dll [2021-03-27]
E:\AviSynth\AVSPLUS370_x64\plugins\libfftw3f-3.dll [2021-03-27]

[Uncategorized files]
E:\AviSynth\AVSPLUS370_x64\plugins\ffms2.lib [2020-08-22]
E:\AviSynth\AVSPLUS370_x64\plugins\ffmsindex.exe [2020-08-22]



[Plugin errors/warnings]
____________________________________________________________________________________________

Function duplicates:

"undefined" : "[InternalFunction]"
"Undefined" : "E:\AviSynth\AVSPLUS370_x64\plugins\Zs_RF_Shared.avsi"

____________________________________________________________________________________________

'E:/AviSynth/AVSPLUS370_x64/plugins/KNLMeansCL.dll' cannot be used as a plugin for AviSynth.
____________________________________________________________________________________________

[StainlessS]

Zorr,
Some plugs use the same dll via different names,
fftw3.dll
libfftw3f-3.dll

And a few months back, another plug used yet another name, I dont recall which plug it was.

EDIT: Here [used by KPassFilterCL, Passfilter.dll]:- https://forum.doom9.org/showthread.php?p=1929800#post1929800

Some more recent plugs can use either of the two first mentioned names.

[zorr]

Zorr,
Some plugs use the same dll via different names,
fftw3.dll
libfftw3f-3.dll


Yes it's probably something like that. Because for example FFT3DFilter is working and it's using the same fftw3 library.

Just to be clear, I have downloaded the latest available (v3.3.5) 64 bit dlls from http://www.fftw.org/install/windows.html. There's source available for 3.3.9 but seems there are no binaries for that. In the package there are 3 dll files:
libfftw3-3.dll
libfftw3f-3.dll
libfftw3l-3.dll


I took libfftw3-3.dll and renamed it to fftw3.dll and put it into windows/System32. Are you suggesting that I should take the same file, rename it to libfftw3f-3.dll and put it into System32 also? Note that the package itself contains a file with that name, so it must be a different dll right?

EDIT:
The only link to 64bit Defreq I was able to find was in this post (https://forum.doom9.org/showthread.php?p=1006019#post1006019).
Interestingly the package claims to contain x64 version of fftw3.dll but it seems to be x86 - I get "Avisynth open failure: Cannot load a 32 bit DLL in 64 bit Avisynth..." when it's placed into Avisynth plugins folder. If I place it only to System32 the error message is the original "Can not load FFTW3.DLL". I think this means Defreq is not looking for it at System32 at all, and instead looking for it in the plugins folder.

EDIT2:
I can't find x64 Quivers either, does anybody have a link?

EDIT3:
Ah, the Quivers are within ManyPlus plugin. And x64 version is available. :)

[Atlantis]

I see there are interest in this. So for research and testing purposes I post a longer clip for people who want to do academic stuff!
2 minutes, includes wave patterns, clean cuts and bad combed cuts.

https://www.mediafire.com/file/nk1gfnek2orz5nd/clip.mkv/file

[StainlessS]

by zorr
libfftw3-3.dll and renamed it to fftw3.dll
Wrong one, remove it.

libfftw3-3.dll
libfftw3f-3.dll # THIS ONE, Copy as is to x64 system32 and make another copy and rename to fftw3.dll also in system32
libfftw3l-3.dll


EDIT: Remove this lot from plugins, [except ffmsindex.exe, if you use it via command line from plugins directory(ffms2 indexer)]

[Uncategorized DLLs (64 Bit)]
E:\AviSynth\AVSPLUS370_x64\plugins\libfftw3-3.dll [2021-03-27]
E:\AviSynth\AVSPLUS370_x64\plugins\libfftw3f-3.dll [2021-03-27]

[Uncategorized files]
E:\AviSynth\AVSPLUS370_x64\plugins\ffms2.lib [2020-08-22]
E:\AviSynth\AVSPLUS370_x64\plugins\ffmsindex.exe [2020-08-22]

[StainlessS]

Bit more tinker, Pre-scan on short bit of whole clip, to judge best thresholds to use for full clip.


#Avisource("clip3.mkv.avi") # Original Short clip
Avisource("clip.mkv.avi") # Longer one, but still short piece of whole clip.
o=last
assumetff()

### Config ###
SUBS = True
WRITE = True
LPD_Th = 10 # RT_LumaPixelsDifferent() Threshold, if corresponding pixels more different than this then is counted
SHOWMINMAX = True # Show Metric minimums where scenechange detected, AND metrics Maximums where NO Scenechange detected [tweak thresholds]
# Voting Thresholds, if above these then VOTE is for SceneChange.
V_YD_TH = 32.0 # I've made these the average of MAX when NO Scenechage and MIN when SceneChange, as shown when SHOWMINMAX=True.
V_YCD_TH = 107.0 # Ditto
V_YPD_TH = 162.0 # Ditto, # If you change LPD_Th, then re-scan and re-calc V_YPD_TH as Ave of YPD_MIN and YPD_MAX
# Intent, allow choose best setting for metric pre-scan on shorter piece of your clip. Then do full scan to write files.
# Can use Vdub2 Video Analysis pass for pre-scan and scan proper.
###
SFrames = ".\SingleRate.txt"
DFramesE = ".\DoubleRateEven.txt"
DFramesO = ".\DoubleRateOdd.txt"
############################
############################
############################

if(WRITE) {
SFrames = SFrames.RT_GetFullPathName
DFramesE = DFramesE.RT_GetFullPathName
DFramesO = DFramesO.RT_GetFullPathName
RT_fileDelete(SFrames)
RT_fileDelete(DFramesE)
RT_fileDelete(DFramesO)
}
YD_MIN = 255.0
YCD_MIN = 512.0
YPD_MIN = 255.0
YD_MAX = 0.0
YCD_MAX = 0.0
YPD_MAX = 0.0

COUNT = 0 # COUNT valid only on forward ONLY scan
SSS="""
n = current_frame
YD = RT_LumaDifference(Last,Last,n=n,y=0,n2=n,y2=1,interlaced=true) # Average Per Pixel Diff betwwen Even and Odd fields
YC = RT_LumaCorrelation(Last,Last,n=n,y=0,n2=n,y2=1,interlaced=True) # Correlation Between Even and Odd fields
YCD = (1.0-YC)*255.0 # Difference by Correlation [0.0->255, max poss 512.0=-1, -ve correlation]
YPD = RT_LumaPixelsDifferent(Last,Last,n=n,y=0,n2=n,y2=1,interlaced=true,Thresh=LPD_Th) # Population [0->255.0] where pixels different by LPD_Th or more
VOTE_YD = (YD > V_YD_TH) ? 1:0
VOTE_YCD = (YCD > V_YCD_TH) ? 1:0
VOTE_YPD = (YPD > V_YPD_TH) ? 1:0
VOTE_CNT = VOTE_YD + VOTE_YCD + VOTE_YPD
SC = VOTE_CNT >= 2
COUNT = (SC) ? COUNT + 1 : COUNT
(SUBS)?RT_Subtitle("%d]\nYD =%7.3f\nYCD=%7.3f\nYPD=%7.3f\nVOTE_YD =%.1s\nVOTE_YCD=%.1s\nVOTE_YPD=%.1s\nVOTE_CNT=%d\nSceneChange=%s (COUNT=%d)",
\ n,YD,YCD,YPD,VOTE_YD!=0,VOTE_YCD!=0,VOTE_YPD!=0,VOTE_CNT,SC,COUNT) : NOP
if(WRITE && SC) {
RT_WriteFile(SFrames,"%d",n,Append=True)
RT_WriteFile(DFramesE,"%d",n*2,Append=True)
RT_WriteFile(DFramesO,"%d",n*2+1,Append=True)
}
if(SHOWMINMAX) {
if(SC) {
YD_MIN = ( YD < YD_MIN) ? YD : YD_MIN
YCD_MIN = (YCD < YCD_MIN) ? YCD : YCD_MIN
YPD_MIN = (YPD < YPD_MIN) ? YPD : YPD_MIN
} else {
YD_MAX = ( YD > YD_MAX) ? YD : YD_MAX
YCD_MAX = (YCD > YCD_MAX) ? YCD : YCD_MAX
YPD_MAX = (YPD > YPD_MAX) ? YPD : YPD_MAX
}
RT_Subtitle(" SC: VD_MIN = %7.3f : VCD_MIN = %7.3f : VPD_MIN = %7.3f\n" + \
"!SC: VD_MAX = %7.3f : VCD_MAX = %7.3f : VPD_MAX = %7.3f\n" + \
"\a!SET: V_YD_TH= %7.3f : V_VCD_TH= %7.3f : V_VPD_TH= %7.3f",
\ YD_MIN,YCD_MIN,YPD_MIN,YD_MAX,YCD_MAX,YPD_MAX,
\ Round((YD_MIN+YD_MAX)/2.0),Round((YCD_MIN+YCD_MAX)/2.0),Round((YPD_MIN+YPD_MAX)/2.0),
\ align=1)
}
Return Last
"""
Scriptclip(SSS)
#StackHorizontal(o,Last)
Return last


Metrics on Final frame
https://i.postimg.cc/ZqPDKPCB/Last-Frame.jpg (https://postimages.org/)

EDIT: NOTED SceneChanges are of course mid frame (bottom field) scene changes. [the other scene changes are ignored, not detected]
(just for anybody thinking that this is a standard scene change detector, it aint)

EDIT: And the last of 3 detected mid frame scene changes in 3002 frames.
https://i.postimg.cc/LXHQqq1y/Frame1744.jpg (https://postimages.org/)

SingleRate.txt

492
581
1744


DoubleRateEven.txt

984
1162
3488


DoubleRateOdd.txt

985
1163
3489

[Atlantis]

I have to share another clip for completeness.

https://www.mediafire.com/file/hat1gxxndxppgmw/clip2.mkv/file

This is a dark clip where bad scene change detection is more difficult. This script did not work with this part of the video:

o=last

o
assumetff()
bwdif(field=3, edeint=nnedi3(field=3))
deint=last

deint
merge(selecteven, selectodd)
deintblend=last

deintblendhalf = deint.selecteven()

ConditionalFilter(deintblend, deintblendhalf, deintblend, "YDifferenceFromPrevious>19 && YDifferenceToNext>19")

[wonkey_monkey]

Had a stab at a hunch, more for fun than anything else, not intended to be a real solution (left is QTGMC output, right is my "filter"):

https://i.imgur.com/LiWKTe8.png

[Atlantis]

Not enough! :)

[zorr]

libfftw3-3.dll
libfftw3f-3.dll # THIS ONE, Copy as is to x64 system32 and make another copy and rename to fftw3.dll also in system32
libfftw3l-3.dll



SUCCESS! Defreq (and FFT3DFilter) works! :thanks:

I already have some promising results, stay tuned.

[StainlessS]

OK, dropped two of the 3 voting metrics, just kept the difference by correlation metric as it is less affected by dark sequences.
I've mixed it into PDR's script, also shows metrics so is easier to mod if necessary. [Scriptclip is a lot more flexible than ConditinslFilter].


LoadPlugin(".\BWDif.dll")
##########
V_YCD_TH = 107.0 # Detect Threshold
SHOW = True # Metrics
STACK= True
##########
A=Avisource("clip3.mkv.avi") # Short clip
B=Avisource("clip.mkv.avi") # Longer one
C=Avisource("clip2.mkv.avi") # third one
A ++ B ++ C # All 3 clips [in supplied order]
Assumetff()
#########
o=Last

deint=bwdif(field=3, edeint=nnedi3(field=3))
deintblend=merge(deint.selecteven, deint.selectodd)
deinteven = deint.selecteven()

SSS= """
n = current_frame
YC = RT_LumaCorrelation(Last,Last,n=n,y=0,n2=n,y2=1,interlaced=True) # Correlation Between Even and Odd fields (ORIGINAL SOURCE CLIP)
YCD = (1.0-YC)*255.0 # Difference by Correlation [0.0->255, max poss 512.0=-1, -ve correlation]
SC = (YCD > V_YCD_TH)
(SC) ? deinteven : deintblend
(SHOW) ? RT_Subtitle("%d] %.3f %s",n,YCD,(SC)?Chr(7)+"!*** SCENECHANGE ***":"") : NOP
return Last
"""

Scriptclip(o, SSS) # Detect on o, o = original src, not deinterlaced

Return (STACK) ? StackHorizontal(o,Last) : Last


EDIT:
And you can maybe use this to pre-scan longer clip to estimate best V_YCD_TH threshold.
I've modified the SingleRate.txt output to be ClipClop format.

A=Avisource("clip3.mkv.avi") # Short clip
B=Avisource("clip.mkv.avi") # Longer one
C=Avisource("clip2.mkv.avi") # third one

A ++ B ++ C # All 3 clips [in supplied order]
assumetff()

o=last

### Config ###
SUBS = True
WRITE = True
SHOWMINMAX = True # Show Metric minimums where scenechange detected, AND metrics Maximums where NO Scenechange detected [tweak thresholds]
V_YCD_TH = 107.0 #
###
SFrames = ".\SingleRate.txt"
############################

if(WRITE) {
SFrames = SFrames.RT_GetFullPathName
RT_fileDelete(SFrames)
}
YCD_MIN = 512.0
YCD_MAX = 0.0

COUNT = 0 # COUNT valid only on forward ONLY scan
SSS="""
n = current_frame
YC = RT_LumaCorrelation(Last,Last,n=n,y=0,n2=n,y2=1,interlaced=True) # Correlation Between Even and Odd fields
YCD = (1.0-YC)*255.0 # Difference by Correlation [0.0->255, max poss 512.0=-1, -ve correlation]
SC = (YCD > V_YCD_TH)
COUNT = (SC) ? COUNT + 1 : COUNT
(SUBS)?RT_Subtitle("%d]\nYCD=%7.3f\nSceneChange=%s (COUNT=%d)",n,YCD,SC,COUNT) : NOP
if(WRITE && SC) { RT_WriteFile(SFrames,"1 %d",n,Append=True) } # Write Frame file in Clipclop style.
if(SHOWMINMAX) {
if(SC) { YCD_MIN = (YCD < YCD_MIN) ? YCD : YCD_MIN }
else { YCD_MAX = (YCD > YCD_MAX) ? YCD : YCD_MAX }
RT_Subtitle(" SC: VCD_MIN = %7.3f\n!SC: VCD_MAX = %7.3f\n\a!SET: V_VCD_TH= %7.3f",
\ YCD_MIN,YCD_MAX,Round((YCD_MIN+YCD_MAX)/2.0),align=1)
}
Return Last
"""
Scriptclip(SSS)
Return last


EDIT:
And here a Clipclop script, one advantage is that you can remove or add frame numbers to the clipclop SingleRate.txt
file, so easily modded if problem frames found. ClipClop needs frames in this format [where 1 is FIXED, and 6699 is frame number
where mid frame scenechange lives].
1 6699

Clipclop script [USES SingleRate.txt generated by 2nd script]

# Requires ClipClop

LoadPlugin(".\BWDif.dll")
##########
SHOW = True
STACK = true

Frames = ".\SingleRate.txt"
Frames = Frames.RT_GetFullPathName

A=Avisource("clip3.mkv.avi") # Short clip
B=Avisource("clip.mkv.avi") # Longer one
C=Avisource("clip2.mkv.avi") # third one
A ++ B ++ C # All 3 clips [in supplied order]
Assumetff()
#########
o=Last
deint=bwdif(field=3, edeint=nnedi3(field=3))
deintblend=merge(deint.selecteven, deint.selectodd)
deinteven = deint.selecteven()

ClipClop(deintblend,deinteven,cmd=Frames,Show=SHOW)

Return (STACK) ? StackHorizontal(o,Last) : Last


EDIT: The (1.0 - YC) thing is called Pearson's Distance [see WikiPedia]:- https://en.wikipedia.org/wiki/Pearson_correlation_coefficient#Pearson's_distance
We scale it by 255.0 to be more familiar [although max possible is 512.0 where correlation is -ve, -1.0, ie (1 - (-1)) * 255.0].
Pearson's distance

A distance metric for two variables X and Y known as Pearson's distance can be defined from their correlation coefficient as[37]

dx,y = 1 − ρx,y.

Considering that the Pearson correlation coefficient falls between [−1, +1], the Pearson distance lies in [0, 2]. The Pearson distance has been used in cluster analysis and data detection for communications and storage with unknown gain and offset[38]

[Atlantis]

The first script works I suppose it's for inspection. 8 BW stacked images. The text are not readable.
The second script doesn't work. Can't find RT_GetFullPathName function.

[poisondeathray]

The second script doesn't work. Can't find RT_GetFullPathName function.

You need RT_Stats. See bottom of StainlessS's posts, there are mediafire and sendspace links to a directory of his plugins

[StainlessS]

Thanx PDR.

Use this one [RT_Stats]:- https://www.mediafire.com/file/xa3t1wx234gyzfq/RT_Stats_25%252626_x86_x64_dll_v2.00Beta13_20201229.zip/file
ClipClop:- https://www.mediafire.com/file/0992cb3ge40qh11/ClipClop_x86_x64_dll_v1-26_20181222.zip/file

If you already have BWDif.dll in plugins, remove the LoadPlugin line.

[zorr]

https://i.postimg.cc/m2jdkS7j/how-it-started.png

Making progress with Defreq and Zopti. I'm still trying to remove that slight vertical ringing on the door.

[Atlantis]

What is doing the deinterlacing? Do you get blended fields at some of the scene changes or do you get clean cut?

[zorr]

What is doing the deinterlacing? Do you get blended fields at some of the scene changes or do you get clean cut?

I used QTGMC with default settings, mainly because it makes the moire pattern more visible. You can use bwdif too if you prefer it.

I just checked the clips you've shared, there seemed to be no problems with the scene changes.

[Atlantis]

Yes there are. Some scene changes are 2 fields from 2 different scenes. QTGMC does the de-interlacing correctly. You don't see those blended scenes in QTGMC result.

[zorr]

Yes there are. Some scene changes are 2 fields from 2 different scenes. QTGMC does the de-interlacing correctly. You don't see those blended scenes in QTGMC result.

Yes that's what I meant, there are no problems with the scene changes when using QTGMC. So if QTGMC + Defreq is used, that solves the scene change problem also.

[Atlantis]

Yes. That would be the best possible result.

[Atlantis]

Did not know that Defreq works on the progressive clips. If it can work on interlaced materials, the best would be to use it before deinterlacing on the original.

[zorr]

Did not know that Defreq works on the progressive clips. If it can work on interlaced materials, the best would be to use it before deinterlacing on the original.

It does work on both. In this case you have to do the moire removal before you do deinterlacing, otherwise QTGMC interprets the moving moire pattern as actual movement and the textures behind it move also. I noticed this after I had done a lot of testing where Defreq was AFTER deinterlacing. :mad: The current version does Defreq both before and after.

BTW the door is now fixed but I noticed another problem on a different frame.

[kedautinh12]

Can you share script QTGMC + Defreq and some image before and after you used it?? zorr

[StainlessS]

Here is frame 2768 [from my ClipClop script] using PDR BWDif script, on Postimge as PNG.
https://i.postimg.cc/B88QmxCs/2769.png (https://postimg.cc/B88QmxCs)

[2768 is source frame for Zorr image - EDIT: Not a mid frame scenechange]
Left Source [interlaced], right BWDif with Even/Odd Merge to singlerate.
EDIT: Looks good enough to me, and no moire. [PDR's BWDif + E/O merge works great, don't tknow if it requires QTGMC + DeFreq]

[zorr]

Can you share script QTGMC + Defreq and some image before and after you used it?? zorr

Here's the current version.

orig = FFVideoSource("Clip1.mkv")

fields = orig.SeparateFields()

# create blend version (for comparison, also for Zopti)
blend = FieldDeinterlace(orig)
blend = QTGMC(blend, InputType=2, TR2=2)

# pass 1 with Defreq, apply to separate fields
fx = 57 # optimize fx = _n_ | 0..100 | fx
fy = -27 # optimize fy = _n_ | -100..100 | fy
dx = 19 # optimize dx = _n_ | 5..50 | dx
dy = 22 # optimize dy = _n_ | 5..50 | dy
sharp = 2
defreq = Defreq(fields, fx=fx, fy=fy, dx=dx, dy=dy, sharp=sharp, show=0, info=false)

# boost result by amplifying whatever Defreq did
boost = 60/100.0 # optimize boost = _n_/100.0 | 0..100 | boost
diff = mt_makediff(defreq, fields, chroma="-127")
defreq = defreq.mt_adddiff(diff)
boosted = Overlay(fields, defreq, opacity=boost)

# deinterlace
deinterlaced = SelectOdd(QTGMC(boosted.weave()))
orig_deinterlaced = SelectOdd(QTGMC(fields.weave()))

# create mask
diff = mt_makediff(deinterlaced, orig_deinterlaced, chroma="-127")
diff = diff.mt_adddiff(diff)
mask = mt_lut(diff, expr="x 128 - abs 6 > 255 0 ?")
mask = mt_expand(mask)
mask = mt_expand(mask)
mask = mt_inpand(mask, chroma="-127")

# pass 2 with Defreq, apply to deinterlaced frames
fx = 48 # optimize fx = _n_ | 0..80 | fx
fy = -11 # optimize fy = _n_ | -80..0 | fy
dx = 10 # optimize dx = _n_ | 5..50 | dx
dy = 16 # optimize dy = _n_ | 5..50 | dy
sharp = 2
defreq2 = Defreq(deinterlaced, fx=fx, fy=fy, dx=dx, dy=dy, sharp=sharp, show=0, info=false)

# combine with pass 1 using mask
defreq_masked = Overlay(deinterlaced, defreq2, mask=mask)

# final result
return defreq_masked

# compare different phases
return Interleave(
\ orig_deinterlaced.Subtitle("QTGMC only"),
\ diff.Subtitle("detected pattern"),
\ mask.Subtitle("mask"),
\ deinterlaced.Subtitle("after pass 1"),
\ defreq2.Subtitle("after pass 2"),
\ defreq_masked.Subtitle("final masked result"),
\ blend.Subtitle("blend deinterlaced"))

I have tested it on Clip1 so far and it's pretty good, there's only one small detail that needs some attention. I noticed there are different patterns in the other clips and they are not all removed by this version. I will take a look at them once I'm satisfied with Clip1.

Comparison shots below, the same frame as before but the door (and some other parts) are better.

https://i.postimg.cc/8510Rbwm/01-combined-before.png
https://i.postimg.cc/1XX7T5kg/01-combined-after.png

Some details about the script: I noticed that while Defreq can detect the correct pattern to remove it always doesn't do a perfect job at removing it. It's probably because it only removes "peaks" and scales the amplitudes so that they match the background level instead of removing them altogether. Therefore I added a "boost" which amplifies whatever changes Defreq did to the frame. This boost parameter was also optimized by Zopti to find the best value.

The first pass does a pretty good job at removing the pattern but a second pass is needed to remove the rest (and there's still a little bit left if you look at the right places). The problem is that the second pass also introduces interference patterns elsewhere (the door problem). I fixed that by making a mask which allows the correction to be applied only to those locations where it is needed. The mask is just a quick hack and could probably be improved but it seems to do its job just fine.

You can see the different phases of the script and the mask if you comment out the second last line ("return defreq_masked").

The final problem area I'm focusing on is that the first pass also creates some interference patterns on some frames. Below you can see one such example. On the left is the output of the script and on the right is blend deinterlaced version which doesn't have the pattern.

https://i.postimg.cc/pLzG3S6B/02-combined-issue.png

So while you can get pretty impressive results with Defreq it's not a walk in the park. As vcmohan stated: "As a sharp discontinuity requires all frequencies of spectrum to be present, any filtering that affects the balance can create undesirable ringing." This is just the case here, the crown has a sharp edge and removing some frequencies made it ring.

I think this can be fixed with another mask for pass 1. It can be created with Defreq using yet another set of parameters. I will let Zopti figure out the best combination. :D

I also tested the cutx and cuty parameters of Defreq but didn't see use for them. They make the result blurrier but we don't want that. By the way the documentation states that the max value for cutx and cuty is 300 but actually it's 200.

EDIT: forgot to mention that the script returns 25fps so that it can be compared to the blend deinterlaced frames. You can take out SelectOdd() calls to return 50fps but then it's out of sync with the blend version.

[Frank62]

Looks great!
My idea (again, in this case to remove the residual pattern): Turn the picture before DeFreq so that the moire is exactly vertical. Would be worth a try.

[Atlantis]

Boy I did well by not touching DeFreq! :scared::D

[kedautinh12]

Here is frame 2768 [from my ClipClop script] using PDR BWDif script, on Postimge as PNG.
https://i.postimg.cc/B88QmxCs/2769.png (https://postimg.cc/B88QmxCs)

[2768 is source frame for Zorr image - EDIT: Not a mid frame scenechange]
Left Source [interlaced], right BWDif with Even/Odd Merge to singlerate.
EDIT: Looks good enough to me, and no moire. [PDR's BWDif + E/O merge works great, don't tknow if it requires QTGMC + DeFreq]

Can you share me final script??

[StainlessS]

Was from post #62, ie


EDIT:
And here a Clipclop script, one advantage is that you can remove or add frame numbers to the clipclop SingleRate.txt
file, so easily modded if problem frames found. ClipClop needs frames in this format [where 1 is FIXED, and 6699 is frame number
where mid frame scenechange lives].
1 6699

Clipclop script [USES SingleRate.txt generated by 2nd script]

# Requires ClipClop

LoadPlugin(".\BWDif.dll")
##########
SHOW = True
STACK = true

Frames = ".\SingleRate.txt"
Frames = Frames.RT_GetFullPathName

A=Avisource("clip3.mkv.avi") # Short clip
B=Avisource("clip.mkv.avi") # Longer one
C=Avisource("clip2.mkv.avi") # third one
A ++ B ++ C # All 3 clips [in supplied order]
Assumetff()
#########
o=Last
deint=bwdif(field=3, edeint=nnedi3(field=3))
deintblend=merge(deint.selecteven, deint.selectodd)
deinteven = deint.selecteven()

ClipClop(deintblend,deinteven,cmd=Frames,Show=SHOW)

Return (STACK) ? StackHorizontal(o,Last) : Last

[kedautinh12]

Thanks, i will try

[kedautinh12]

I don't seen any SingleRate.txt creat by script 2

[StainlessS]

SingleRate.txt is created via the 2nd script in post #62.

[kedautinh12]

I used script 2 but don't seen SingleRate.txt. I think your scripts use for twopass encode. So sad cause i use one pass encode

[StainlessS]

OK, here TwoPass


LoadPlugin(".\BWDif.dll")
##########

A=Avisource("clip3.mkv.avi") # Short clip
B=Avisource("clip.mkv.avi") # Longer one
C=Avisource("clip2.mkv.avi") # third one

A ++ B ++ C # All 3 clips [in supplied order]
assumetff()
o=last
############################
### Config ###
V_YCD_TH = 107.0 #
FINAL = False # SWITCH OFF ALL THE OTHER STUFF if TRUE
SUBS = True
SHOWMINMAX = True # Show Metric minimums where scenechange detected, AND metrics Maximums where NO Scenechange detected [tweak thresholds]
###
CLIPCLOPSHOW = True
STACK = True
SFrames = ".\SingleRate.txt"
############################
If (FINAL) {
SUBS = FALSE
SHOWMINMAX = FALSE
CLIPCLOPSHOW = False
STACK = FALSE
}

SFrames = SFrames.RT_GetFullPathName
RT_fileDelete(SFrames)

YCD_MIN = 512.0
YCD_MAX = 0.0
COUNT = 0 # COUNT valid only on forward ONLY scan
SSS="""
n = current_frame
YC = RT_LumaCorrelation(Last,Last,n=n,y=0,n2=n,y2=1,interlaced=True) # Correlation Between Even and Odd fields
YCD = (1.0-YC)*255.0 # Difference by Correlation [0.0->255, max poss 512.0=-1, -ve correlation]
SC = (YCD > V_YCD_TH)
COUNT = (SC) ? COUNT + 1 : COUNT
(SUBS)?RT_Subtitle("%d]\nYCD=%7.3f\nSceneChange=%s (COUNT=%d)",n,YCD,SC,COUNT) : NOP
if(SC) { RT_WriteFile(SFrames,"1 %d",n,Append=True) } # Write Frame file in Clipclop style.
if(SHOWMINMAX) {
if(SC) { YCD_MIN = (YCD < YCD_MIN) ? YCD : YCD_MIN }
else { YCD_MAX = (YCD > YCD_MAX) ? YCD : YCD_MAX }
RT_Subtitle(" SC: VCD_MIN = %7.3f\n!SC: VCD_MAX = %7.3f\n\a!SET: V_VCD_TH= %7.3f",
\ YCD_MIN,YCD_MAX,Round((YCD_MIN+YCD_MAX)/2.0),align=1)
}
Return Last
"""
Scriptclip(SSS)

RT_ForceProcess # force Pass 1

# Pass 2
deint=bwdif(field=3, edeint=nnedi3(field=3))
deintblend=merge(deint.selecteven, deint.selectodd)
deinteven = deint.selecteven()

ClipClop(deintblend,deinteven,cmd=SFrames,Show=CLIPCLOPSHOW)

(STACK) ? StackHorizontal(o,Last) : Last

Return last

[kedautinh12]

It's work, many thanks