QTGMC Deinterlacing Script (v3.384)

[ravewulf]

Originally Posted by StainlessS

ravewulf, you should have done timing tests with/without YUY2 hacks in-situ, to tell if any benefit in removal, and then post
results, removal just for the sake of it is not a good idea.

In general, removing the SSE2Tools Planar YUY2 hacks to use the built in YV16 support is a good idea as well as creates cleaner code. But here are the numbers:

AviSynth 2.6 MT - Athlon II x4 640 OC'd to 3.6 GHz
500 frame 720x480 60i - 1000 frames of output
MT Mode not used, EdiThreads=1 to force single threaded performance for testing

3.33 2016-12-06 mod
YUY2 4.608 fps
YV16 5.591 fps

3.34
YV16 5.608 fps

The YV16 numbers are very close, but the old YUY2 code is slow. The main benefits are always using the YV16 path when possible and having less code to maintain (3.33 2016-12-06 mod is 91.4 KB, 3.34 is 77.1 KB). As I said in a previous post, YUY2 support could easily be added back to QTGMC by immediately converting to YV16 and converting back to YUY2 before returning at the end.

[ravewulf]

Originally Posted by real.finder

you did it in another way, you did support yv16 in hack way, and dropped YUY2, my suggested is to did YUY2 support in more clean and fast way (same as you did but for YUY2 not yv16)

and the yv16 support is work in pure yv16 and that mean now not all features of qtgmc work with it, until all plugins supports yv16, and when that happened you will not need to did another changes

QTGMC core

MaskTools2 Supports YV16, doesn't support YUY2
MVTools supports YV16, YUY2 requires SSE2Tools hack so YV16 is preferable
RgTools supports YV16, YUY2 requires SSE2Tools hack so YV16 is preferable
Nnedi3 supports both YV16 and YUY2

Optional
Support both YV16 and YUY2 - AddGrainC, Dfttest
Supports YV16, doesn't support YUY2 - yadifmod2, KNLMeansCL
Don't support YV16, but support YUY2 directly without SSETools hack - FFT3DFilter, NNEDI2, NNEDI, EEDI3, EEDI2, TDeint

Going with YV16 by default is simpler than YUY2 by default. Certainly a lot simpler than previous versions. If any of the older plugins do get updated with YV16 support, the conditional conversions for that specific plugin can easily be removed.

Also, as pointed out in my post directly above, using YV16 instead of YUY2 wherever possible is faster and uses less code.

[ravewulf]

Originally Posted by ndjamena

The version string I'm being given now has r2337 in it, so unless I'm misreading the code QTGMC isn't using ExtractU with the latest build of AVS+ anyway.

I'm using AviSynth 2.6 MT, so I'm not sure about AVS+. real.finder added ExtractU in the 3.33 2016-12-06 update, but maybe that part should be reverted?

I'm having trouble with MvTools2 2.7.7.22 (20161215) stalling at the end, but MvTools2 2.7.0.22d seems to be ok.

Edit:
I'll work on a version of QTGMC 3.34 that uses the 3.33 2016-04-11 version as a base instead of the 3.33 2016-12-06 version as it seems to be more stable.

[real.finder]

here is my mod with ravewulf idea http://pastebin.com/1rdYVVKq

note: not tested enough, test it with cautiously

[ravewulf]

Now based on real.finder's QTGMC 3.33 2016-04-11 mod and should be more stable.

v3.34 2016-12-17

• YV16 is used instead of YUY2 wherever possible
• YUY2 input is automatically converted to YV16 for processing then back to YUY2 at the end
• Removed most YUY2 workarounds except where still needed for plugins that don't support YV16
• Removed reliance on SSE2Tools. All plugins support either YV16 or YUY2 directly
• Minimum requirements updated for YV16 support
• RemoveGrain and Vertical Cleaner replaced by RgTools
• Yadif replaced by yadifmod2

AddGrainC, dfttest, MaskTools2, MvTools2, NNEDI3, KNLMeansCL, RgTools, and yadifmod2 support YV16

FFT3DFilter, NNEDI2, NNEDI, EEDI3, EEDI2, TDeint still rely on YUY2, but accept it directly without SSE2Tools.

# --- REQUIREMENTS ---
#
# Input colorspaces: YV12, YV16, YUY2
#
# Core plugins:
# MVTools2 (2.7.0.22c or above from pinterf https://github.com/pinterf/mvtools/releases)
# MaskTools v2 (2.0b1 or above)
# NNEDI3 (0.9.4.6 or above)
# RgTools
#
# Additional plugins:
# NNEDI2, NNEDI, EEDI3, EEDI2, TDeInt - if selected directly or via a source-match preset
# yadifmod2 - for Preset="Ultra Fast" or if selected directly
# FFT3DFilter - if selected for noise processing
# dfttest (1.9.4 or above) - if selected for noise processing
# For FFT3DFilter & ddftest you also need the FFTW3 library (FFTW.org). On Windows the file needed for both is libfftw3f-3.dll. However, for FFT3DFilter the file needs to be called FFTW3.dll, so you will need two copies and rename one. On Windows put the files in your System32 or SysWow64 folder
# AddGrainC (1.7.0 or above) - if NoiseDeint="Generate" selected for noise bypass

For now I recommend MVTools2 2.7.0.22d as the newer versions with 10-16 bit support have been freezing AVSMeter at the end of test runs.

Please let me know if there's something I missed or if you find a problem with my changes.

[ndjamena]

Originally Posted by ravewulf

I'm using AviSynth 2.6 MT, so I'm not sure about AVS+. real.finder added ExtractU in the 3.33 2016-12-06 update, but maybe that part should be reverted?

I'm having trouble with MvTools2 2.7.7.22 (20161215) stalling at the end, but MvTools2 2.7.0.22d seems to be ok.

Edit:
I'll work on a version of QTGMC 3.34 that uses the 3.33 2016-04-11 version as a base instead of the 3.33 2016-12-06 version as it seems to be more stable.

Actually I read it wrong. Sorry, having trouble concentrating.

ssispmt = Findstr(VersionString(), "AviSynth+") != 0 && Findstr(VersionString(), "r1576") == 0

Lets see, if in the Version String there's the text "AviSynth+" (not 0) and there's NOT the text "r1576" (0) then use ExtractU.

Which I'm thinking at the moment means any version of AviSynth+ other than "r1576" will attempt to use ExtractU, even though the filter was only added to the most recent build and there were many releases between the two.

Does that sound about right?

I'm thinking it would have to figure out some way of converting the revision number to an integer in order to be viable in the short term and avoid this particular bug, but what do I know.

[ravewulf]

Originally Posted by ndjamena

Actually I read it wrong. Sorry, having trouble concentrating.

ssispmt = Findstr(VersionString(), "AviSynth+") != 0 && Findstr(VersionString(), "r1576") == 0

Lets see, if in the Version String there's the text "AviSynth+" (not 0) and there's NOT the text "r1576" (0) then use ExtractU.

Which I'm thinking at the moment means any version of AviSynth+ other than "r1576" will attempt to use ExtractU, even though the filter was only added to the most recent build and there were many releases between the two.

Does that sound about right?

I'm thinking it would have to figure out some way of converting the revision number to an integer in order to be viable in the short term and avoid this particular bug, but what do I know.

Yeah, that sounds about right. Maybe something like this?

BlankClip()
str = VersionString()      #get version string
strPos = FindStr(str,"(r") #find start of (r####
str = MidStr(str,strPos+2) #get substring that starts with the number
vn = Value(str)            #convert to a number value
str = String(vn > 1576)    #check against a known version
Subtitle(str)

Unfortunately it would break if they change the format of the version string. It would be easier with regular expressions.

Edit:
Better idea - take a leaf out of my web dev book and avoid version detection

BlankClip(pixel_type="YV12")
try {
    c420 = Is420(last) #code which may fail if the functions don't exist
} catch (errStr) {
    c420 = IsYV12(last) #fallback code
}
Subtitle(String(c420))

So put all the new AVS+ functions in the try block and all the fallback functions in the catch block

[real.finder]

ok new edit in the same post here http://forum.doom9.org/showpost.php?...postcount=2041

don't forget SMDegrain if you use tv_range=true (Default for now) and if you use avs+mt (not the r1576 that has no mt, r1576 is work like any normal avs 2.6) use the most last updated here or newer

[pinterf]

Originally Posted by ravewulf

I'm having trouble with MvTools2 2.7.7.22 (20161215) stalling at the end, but MvTools2 2.7.0.22d seems to be ok.

Edit:
I'll work on a version of QTGMC 3.34 that uses the 3.33 2016-04-11 version as a base instead of the 3.33 2016-12-06 version as it seems to be more stable.

If you experienced the stalling with YUY2 input, it may have been fixed in 2.7.8.22

[thanhvanchi]

công ty chúng tôi chuyên về lắp https://internetvietnam.net/dang-ky-lap-dat-internet-tai-huyen-binh-chanh.html và https://forum.audacityteam.org/viewtopic.php?t=93857 và https://band.us/@dulichviet và tiếp thị kỹ thuật số cho khách hàng.
Ngoài ra đây là điểm đến lý tương để khách hàng nhu cầu về cho mọi doanh nghiệp và hộ gia đình.

[odyssey]

What's the best way to eliminate crashes when using MT?

I'm currently down to this, but it still crashes quite frequently, while encoding with x264 (source is PAL DVD):

SetMTMode(5, 5)
SetMemoryMax(700)
DGDecode_mpeg2source(...)
AssumeTFF()
ColorMatrix(hints=true, threads=0)
SetMTMode(2)
QTGMC(Preset="Slow", EdiThreads=1)
Distributor()

I get about 30fps which I think is pretty decent (on i7-6700K stock settings). I've got 32GB memory, so that shouldn't be a problem, but it could be that thing needs to be fine tuned better.

A few things I've noticed:

- Often when it crashes, the x264 file was encoded much bigger than it should (???)
- When I encoded a tv-series from a mostly progressive source, to remove bad deinterlacing artifacts, keeping the fps at 25, it did not crash at all.

[LigH]

Use Distributor() only when a specific tool requires it. The x264 encoder does usually not. Using it unnecessarily may multiply threads!

[hello_hello]

odyssey, try this:

DGDecode_mpeg2source(...)
AssumeTFF()
ColorMatrix(hints=true, threads=0)
QTGMC(Preset="Slow", EdiThreads=1)
Trim(0,4999) #or whatever frame range encodes half the video

Then create another script to encode the second half and run them simultaneously. Even if they both only run at 15fps, it's be 2x15fps and should be far more stable.

Add --stitchable to the x264 command line and append the encoded video with MKVToolnix (assume an MKV output is fine).

[ndjamena]

Parts=2
Part=1

Trim(floor(framecount/Parts)*(Part-1), (floor(framecount/Parts)*Part)-1)

You could set part number by either creating the avs file via batch, adding the number to the avs file name and reading the file name from the script or checking to see if an output file for a previous part exists and if it is start the next part.

Multithreading is working fine for me though.

[odyssey]

Originally Posted by LigH

Use Distributor() only when a specific tool requires it. The x264 encoder does usually not. Using it unnecessarily may multiply threads!

Didn't work

Originally Posted by hello_hello

odyssey, try this:

DGDecode_mpeg2source(...)
AssumeTFF()
ColorMatrix(hints=true, threads=0)
QTGMC(Preset="Slow", EdiThreads=1)
Trim(0,4999) #or whatever frame range encodes half the video

Then create another script to encode the second half and run them simultaneously. Even if they both only run at 15fps, it's be 2x15fps and should be far more stable.

Add --stitchable to the x264 command line and append the encoded video with MKVToolnix (assume an MKV output is fine).

Originally Posted by ndjamena

Parts=2
Part=1

Trim(floor(framecount/Parts)*(Part-1), (floor(framecount/Parts)*Part)-1)

You could set part number by either creating the avs file via batch, adding the number to the avs file name and reading the file name from the script or checking to see if an output file for a previous part exists and if it is start the next part.

Multithreading is working fine for me though.

Not a bad idea, and may even give better fps. One major drawback is however that it complicates scripting this

I'm encoding shows, so I could probably make it easier by just encoding multiple episodes at the same time.

Also I would probably join them using ffmpeg afterwards. Gotta play a bit with this Thanks for the suggestions so far!

[Groucho2004]

Originally Posted by odyssey

What's the best way to eliminate crashes when using MT?

I'm currently down to this, but it still crashes quite frequently, while encoding with x264 (source is PAL DVD):

SetMTMode(5, 5)
SetMemoryMax(700)
DGDecode_mpeg2source(...)
AssumeTFF()
ColorMatrix(hints=true, threads=0)
SetMTMode(2)
QTGMC(Preset="Slow", EdiThreads=1)
Distributor()

A few pointers/comments:

1. You may be using old/outdated plugins. In order for us to suggest new versions you should use AVSMeter to create a "snapshot" of your Avisynth install and associated plugins. Run "AVSMeter -avsinfo -log" from the command line and post the generated log file ("avsinfo.log").
2. "SetMemoryMax(700)" may be a bit low for 5 base threads.
3. Consider using Avisynth+ instead of SEt's MT. At this stage it's quite stable and tests have shown good results multi-threading QTGMC().

Lastly, how are you feeding the script to x264?

[odyssey]

Originally Posted by Groucho2004

A few pointers/comments:

1. You may be using old/outdated plugins. In order for us to suggest new versions you should use AVSMeter to create a "snapshot" of your Avisynth install and associated plugins. Run "AVSMeter -avsinfo -log" from the command line and post the generated log file ("avsinfo.log").
2. "SetMemoryMax(700)" may be a bit low for 5 base threads.
3. Consider using Avisynth+ instead of SEt's MT. At this stage it's quite stable and tests have shown good results multi-threading QTGMC().

Lastly, how are you feeding the script to x264?

I'm using the latest plugins recommended in the first thread, with only the optimized plugin package collection.

I started out with memory at about 3200 AFAIR, but then read that it may be too high.

I'll try the Avisynth+ Thanks for the suggestion.

x264_8bit_x86.exe --crf 16 --preset slower --level 4.1 --fps 50 --sar !sar! --output "!h264file!" "!avsfile!"

[ndjamena]

Hmmm. I told a lie... I AM having problems multi-threading but only when denoising is turned on (which is used by default in placebo mode). It doesn't crash but it does fill the frame with crap occasionally... I tracked it down to the dnWindow stage... maybe I'll try a different version of fft3dfilter...

Grr, I'm trying got figure out what's causing it but...

Grr:

http://forum.doom9.org/showpost.php?...postcount=2041
http://pastebin.com/raw/r4esUcVF

Whoever last edited that used the variable name "EdiMode" in SourceMatch rather than MatchEdi...

[Groucho2004]

Originally Posted by odyssey

I'm using the latest plugins recommended in the first thread, with only the optimized plugin package collection.

Yeah, these are outdated.

Posting the AVSMeter log would avoid a lot of questions, for example what OS you're running. You can also use AVSMeter to check the memory consumption of your script.

[ndjamena]

There's no equivalent to ChromaEdi inside SourceMatch, so I have no idea what the hell to do with that.

Were EdiMode and ChromaEdi supposed to have been passed to SourceMatch or should that bit of code be in the main function?

*sigh*

I'll go back to my modified version and go back to what I was doing, maybe I'll be able to "update to the latest version" for testing tomorrow.