Scythe42
10th July 2015, 12:30
The addition of QTGMC means no AviSynth MT (someday I'll figure out how to multi-thread QTGMC...), which is OK because from what I've read and experienced MPEG2 in Mkv can be inconsistent.True, when multithreading you can run into issues as access might not be linear anymore. Especially with a DirectShow approach that basically only deals with time and not frames, jumping around could cause issues/artifacts especially with a not frame exact setup. Also depends on how an MKV was created, some are not really seekable. All valid points.
In regards to MT QTGMC it should be pretty straight forward: (download MT plugin package from http://avisynth.nl/index.php/QTGMC)
#Setup MT
SetMTMode(3, 8) # 8 in my case for a quad core with 8 hardware threads.
SetMemoryMax(1000) # just a big number, have enough RAM
#VideoSource
# Load your video with what ever method you prefer or need
# Change MT mode
SetMTMode(2) # adjust mode later for a specific filter again, if needed
#Deinterlace
QTGMC(EdiThreads=4) # half your hardware threads
# yada yada yada
That works general for me with AviSynth 2.6 x86.
Depending on the hardware you might need to tweak stuff for the best performance you can get. In general you don't want the CPU maxed out all the time or stuff get's slower because threads are waiting.
LouieChuckyMerry
19th July 2015, 09:29
True, when multithreading you can run into issues as access might not be linear anymore. Especially with a DirectShow approach that basically only deals with time and not frames, jumping around could cause issues/artifacts especially with a not frame exact setup. Also depends on how an MKV was created, some are not really seekable. All valid points.
In regards to MT QTGMC it should be pretty straight forward: (download MT plugin package from http://avisynth.nl/index.php/QTGMC)
#Setup MT
SetMTMode(3, 8) # 8 in my case for a quad core with 8 hardware threads.
SetMemoryMax(1000) # just a big number, have enough RAM
#VideoSource
# Load your video with what ever method you prefer or need
# Change MT mode
SetMTMode(2) # adjust mode later for a specific filter again, if needed
#Deinterlace
QTGMC(EdiThreads=4) # half your hardware threads
# yada yada yada
That works general for me with AviSynth 2.6 x86.
Depending on the hardware you might need to tweak stuff for the best performance you can get. In general you don't want the CPU maxed out all the time or stuff get's slower because threads are waiting.
Thanks for the tutorial :) . I've actually taken the time to sort out AviSynth MT with the Simpsons script (but decided not to use it) and with my usual SMDegrain-FastLineDarkenMod-GradFun3 go to script, but I gave up on QTGMC after little effort because my typical AviSynth usage is pretty simple and SMDegrain has always treated me well. Someday I'll figure out AviSynt MT with QTGMC just because, and I think that your EdiThreads will be the reason as that's something I never tried before.
I've finally finished encoding Season 1 of The Simpsons with the CRF 13.0: with 320 Kbps combined main and commentary audios, the file sizes vary from 325-405 MB and the encoding times varied from 7-20 hours per episode :confused: . I'm not sure why some episodes took so much longer than others--the source? my laptop? the script? running 2 episode simultaneously? gremlins?--but the results are very nice indeed. I've started encoding Season 2 with CRF 12.0 and, with the same 320 Kbps audio combo, file sizes have varied from 301-451 MB and encoding times have varied from 7-10 hours after 17 episodes. Guess I'm trading wildly inconsistent encoding times for wildly inconsistent file sizes, ha ha. Ahhh, didn't you mention earlier that the Season 1 DVD's came from bad tape transfers? Maybe that's why. Anyway, I hope that as the sources improve (slightly, ha ha, very slightly) encoding time will become more consistent.
LouieChuckyMerry
27th July 2015, 09:34
I made the mistake of asking one too many questions and now my script has again been revamped. Scythe42's Deinterlacing-Field Matching-Decimating-Line Doubled Fields Fixer©® is unchanged as it's extremely functional, especially given the horrendous source quality, but after retesting and comparing I've decided to eliminate the QTGMC call. Someone helpfully pointed out to me that QTGMC is NNEDI3-based and, since there's already a necessary NEEDI3 line for the line-doubled fields, running QTGMC on top of that was making a mess of things. So I took a closer look, then actually checked from viewing distance from the screen (instead of typing distance), and decided that a bit of aliasing with more detail was preferable to the converse. The video just looks crisper to my eyes.
It also seems that, excepting a few bad bits here and there, the sources aren't nearly as shaky as I'd originally thought. So I've removed the Stab(Mirror=15) call and if necessary (when I finally ever get to watch some of the rerereencoded episodes, ha ha) I'll rererereencode troublesome ones with Stab(Mirror=15) reinserted in the script.
The biggest change is replacing the Gibbs noise block and SMDegrain line with KNLMeansCL. My eyes see no difference and there's quite a boost in encoding speed, too, which, in conjunction with the removal of QTGMC, makes for an about three (~3) times improvement in fps.
Everything else is basically the same save for some minor changes in syntax.
Finally, I'm still using CRF 13.0 for Season 1 and CRF 12.0 for Seasons 2-10, as the output video's bit rate is about the same with the new script. And don't forget the necessary x264 10-bit command line additions
--demuxer raw --input-depth 16 --input-res 640x480 --fps 23.976 --sar 1:1
Thus The Final Final (I Swear I'm Having Fun ;) ) Script is:
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\DGIndexNV\DGDecodeNV.dll")
DGSource("SourcePath")
### Deinterlace-Match Fields-Decimate ###
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
Function FieldMatch(Clip C) {
Global PP = C.DuplicateFrame(0)
Global CC = C
Global NN = C.DeleteFrame(0)
P2 = PP.SeparateFields()
C2 = CC.SeparateFields()
N2 = NN.SeparateFields()
Global PC = Interleave(P2.SelectEven(),C2.SelectOdd()).Weave()
Global CP = Interleave(C2.SelectEven(),P2.SelectOdd()).Weave()
Global CN = Interleave(C2.SelectEven(),N2.SelectOdd()).Weave()
Global NC = Interleave(N2.SelectEven(),C2.SelectOdd()).Weave()
Global Deint = QTGMC(CC).SelectEven()
Return ScriptClip(CC, \
"!CC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CC : " + \
"!NN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NN : " + \
"!CN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CN : " + \
"!NC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NC : " + \
"!PP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PP : " + \
"!CP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CP : " + \
"!PC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PC : Deint")
}
TFM(Order=-1,Mode=5,PP=2,Clip2=FieldMatch(),Slow=2,MChroma=False,Ubsco=False,CThresh=12,Chroma=True)
TDecimate(Mode=1)
### Fix Line-Doubled Fields ###
NNEDI3(Field=-2)
Merge(SelectEven(),SelectOdd())
### Stabilize, If Necessary ###
# Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Convert Source To Stacked ###
Dither_Convert_8_To_16()
### Overall Spatio-Temporal Denoise (Including Gibbs Noise) ###
KNLMeansCL(D=1,A=1,h=5.0,Lsb_InOut=True,Device_Type="GPU")
### Resize ###
LinearResize(640,480,Lsb_In=True,Lsb_Out=True)
### Darken-Thin Lines ###
F=DitherPost(Mode=-1)
S=F.FastLineDarkenMod(Strength=20,Prot=6).aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)
D=MT_MakeDiff(S,F).Dither_Convert_8_To_16()
Dither_Add16(Last,D,Dif=True,U=2,V=2)
### Deband ###
GradFun3(thR=0.55,SMode=2,Lsb_In=True,Lsb=True,StaticNoise=True)
### Preview Source OR Send 16-bit Output To x264 10-bit ###
# DitherPost()
Dither_Out()
Unfortunately KNLMeansCL won't presently run on my slower laptop because of a problem with the OpenCL compiler, and I normally use my faster laptop for HD sources, so I won't be able to attack until a new version of KNLMeansCL is released. Once I've encoded some episodes I'll report back with speeds, bit rates, etc. :) .
Edit: It seems I was wrong. After the initial excitement of Three Times Faster! encoding wore off I took a closer look. From viewing distance it's hard to see a difference, but from typing distance, and especially when comparing individual frames, the Gibbs noise block-SMDegrain method just looks better than the KNLMeansCL way. So, thinking about that hoped for future large screen TV, it's back to the original, minus QTGMC. Thus, The Final Final Oh So Final That It Hurts Script is:
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\DGIndexNV\DGDecodeNV.dll")
DGSource("SourcePath")
### Deinterlace-Match Fields-Decimate ###
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
Function FieldMatch(Clip C) {
Global PP = C.DuplicateFrame(0)
Global CC = C
Global NN = C.DeleteFrame(0)
P2 = PP.SeparateFields()
C2 = CC.SeparateFields()
N2 = NN.SeparateFields()
Global PC = Interleave(P2.SelectEven(),C2.SelectOdd()).Weave()
Global CP = Interleave(C2.SelectEven(),P2.SelectOdd()).Weave()
Global CN = Interleave(C2.SelectEven(),N2.SelectOdd()).Weave()
Global NC = Interleave(N2.SelectEven(),C2.SelectOdd()).Weave()
Global Deint = QTGMC(CC).SelectEven()
Return ScriptClip(CC, \
"!CC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CC : " + \
"!NN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NN : " + \
"!CN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CN : " + \
"!NC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NC : " + \
"!PP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PP : " + \
"!CP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CP : " + \
"!PC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PC : Deint")
}
TFM(Order=-1,Mode=5,PP=2,Clip2=FieldMatch(),Slow=2,MChroma=False,Ubsco=False,CThresh=12,Chroma=True)
TDecimate(Mode=1)
### Fix Line-Doubled Fields ###
NNEDI3(Field=-2)
Merge(SelectEven(),SelectOdd())
### Stabilize, If Necessary ###
# Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",ThY1=20,ThY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(DFTTest(),Mask,Luma=True)
### Overall Temporal Denoise ###
SMDegrain(TR=2,ThSAD=400,ContraSharp=True,RefineMotion=True,Plane=0,PreFilter=2,Chroma=False,Lsb=True,Lsb_Out=True)
### Resize ###
LinearResize(640,480,Lsb_In=True,Lsb_Out=True)
### Darken-Thin Lines ###
F=DitherPost(Mode=-1)
S=F.FastLineDarkenMod(Strength=20,Prot=6).aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)
D=MT_MakeDiff(S,F).Dither_Convert_8_To_16()
Dither_Add16(Last,D,Dif=True,U=2,V=2)
### Deband ###
GradFun3(ThR=0.55,SMode=2,StaticNoise=True,Lsb_In=True,Lsb=True)
### Preview Source OR Send 16-bit Output To x264 10-bit ###
# Trim()
# DitherPost()
Dither_Out()
This outputs video with about the same bit rate (+/- 5%) as with the old QTGMC line, but it's much more consistent when it comes to encoding time. I'm beginning Season 2 tonight and will update this post accordingly. Happy Saturday :) .
Edit: with a 224 Kbps main audio and a 96 Kbps commentary track combination, running two episodes simultaneously overnight:
Season 1: at CRF 13.0, episodes took from 7.5-10 hours to encode and averaged 380 MB in size, ranging from 332 MB to 424 MB.
Season 2: at CRF 12.0, episodes took from 7-8.5 hours to encode and averaged 349 MB in size, ranging from 282 MB to 436 MB.
Season 3: at CRF 12.0, episodes took from 6.5-7.5 hours to encode and averaged 330 MB in size, ranging from 246 MB to 346 MB.
Since it seems that the source quality is leveling out, I'm lowering the CRF to 11.0 starting with Season 4 so as to return the video bit rate to ~1850 Kbps. Also, I'll probably redo the episodes from Season 2 and Season 3 that were less than 340 MB (~1735 Kbps), just so I never have to think about them again ;) .
FinalEdit: Happy Saturday! I've finally finished Season 10 and, given what I've learned, if I were starting from the beginning then I'd simply use The Final Final Oh So Final That It Hurts Script above with CRF 12.0 for Season 1 and CRF 10.0 for Seasons 2-10 and call it a day. This would make for a reasonably consistent average video bit rate of ~2100 Kbps per episode, perhaps a bit higher than I'd originally planned but I reckon better a bit too much bit rate than a bit too little. Also, given the fluctuation in file size this would make the episodes with the lowest video bit rate closer to my original goal of ~1850 Kbps (some of the Season 2-11 episodes were resulting in video bit rates ~1500 Kbps at CRF 13.0).
vBulletin® v3.8.11, Copyright ©2000-2026, vBulletin Solutions Inc.