Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 17th December 2016, 18:59   #201  |  Link
ravewulf
Registered User
 
Join Date: Dec 2010
Posts: 22
Quote:
Originally Posted by StainlessS View Post
ravewulf, you should have done timing tests with/without YUY2 hacks in-situ, to tell if any benefit in removal, and then post
results, removal just for the sake of it is not a good idea.
In general, removing the SSE2Tools Planar YUY2 hacks to use the built in YV16 support is a good idea as well as creates cleaner code. But here are the numbers:

AviSynth 2.6 MT - Athlon II x4 640 OC'd to 3.6 GHz
500 frame 720x480 60i - 1000 frames of output
MT Mode not used, EdiThreads=1 to force single threaded performance for testing

3.33 2016-12-06 mod
YUY2 4.608 fps
YV16 5.591 fps

3.34
YV16 5.608 fps

The YV16 numbers are very close, but the old YUY2 code is slow. The main benefits are always using the YV16 path when possible and having less code to maintain (3.33 2016-12-06 mod is 91.4 KB, 3.34 is 77.1 KB). As I said in a previous post, YUY2 support could easily be added back to QTGMC by immediately converting to YV16 and converting back to YUY2 before returning at the end.

Last edited by ravewulf; 17th December 2016 at 19:24.
ravewulf is offline   Reply With Quote
Old 17th December 2016, 19:04   #202  |  Link
ravewulf
Registered User
 
Join Date: Dec 2010
Posts: 22
Quote:
Originally Posted by real.finder View Post
you did it in another way, you did support yv16 in hack way, and dropped YUY2, my suggested is to did YUY2 support in more clean and fast way (same as you did but for YUY2 not yv16)

and the yv16 support is work in pure yv16 and that mean now not all features of qtgmc work with it, until all plugins supports yv16, and when that happened you will not need to did another changes
QTGMC core

MaskTools2 Supports YV16, doesn't support YUY2
MVTools supports YV16, YUY2 requires SSE2Tools hack so YV16 is preferable
RgTools supports YV16, YUY2 requires SSE2Tools hack so YV16 is preferable
Nnedi3 supports both YV16 and YUY2

Optional
Support both YV16 and YUY2 - AddGrainC, Dfttest
Supports YV16, doesn't support YUY2 - yadifmod2, KNLMeansCL
Don't support YV16, but support YUY2 directly without SSETools hack - FFT3DFilter, NNEDI2, NNEDI, EEDI3, EEDI2, TDeint

Going with YV16 by default is simpler than YUY2 by default. Certainly a lot simpler than previous versions. If any of the older plugins do get updated with YV16 support, the conditional conversions for that specific plugin can easily be removed.

Also, as pointed out in my post directly above, using YV16 instead of YUY2 wherever possible is faster and uses less code.
ravewulf is offline   Reply With Quote
Old 17th December 2016, 19:22   #203  |  Link
ravewulf
Registered User
 
Join Date: Dec 2010
Posts: 22
Quote:
Originally Posted by ndjamena View Post
The version string I'm being given now has r2337 in it, so unless I'm misreading the code QTGMC isn't using ExtractU with the latest build of AVS+ anyway.
I'm using AviSynth 2.6 MT, so I'm not sure about AVS+. real.finder added ExtractU in the 3.33 2016-12-06 update, but maybe that part should be reverted?

I'm having trouble with MvTools2 2.7.7.22 (20161215) stalling at the end, but MvTools2 2.7.0.22d seems to be ok.

Edit:
I'll work on a version of QTGMC 3.34 that uses the 3.33 2016-04-11 version as a base instead of the 3.33 2016-12-06 version as it seems to be more stable.

Last edited by ravewulf; 17th December 2016 at 19:59.
ravewulf is offline   Reply With Quote
Old 17th December 2016, 23:52   #204  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
here is my mod with ravewulf idea http://pastebin.com/1rdYVVKq

note: not tested enough, test it with cautiously
__________________
See My Avisynth Stuff

Last edited by real.finder; 18th December 2016 at 03:30.
real.finder is offline   Reply With Quote
Old 17th December 2016, 23:55   #205  |  Link
ravewulf
Registered User
 
Join Date: Dec 2010
Posts: 22
Qtgmc 3.34 2016-12-17

Now based on real.finder's QTGMC 3.33 2016-04-11 mod and should be more stable.

v3.34 2016-12-17
  • YV16 is used instead of YUY2 wherever possible
  • YUY2 input is automatically converted to YV16 for processing then back to YUY2 at the end
  • Removed most YUY2 workarounds except where still needed for plugins that don't support YV16
  • Removed reliance on SSE2Tools. All plugins support either YV16 or YUY2 directly
  • Minimum requirements updated for YV16 support
  • RemoveGrain and Vertical Cleaner replaced by RgTools
  • Yadif replaced by yadifmod2

AddGrainC, dfttest, MaskTools2, MvTools2, NNEDI3, KNLMeansCL, RgTools, and yadifmod2 support YV16

FFT3DFilter, NNEDI2, NNEDI, EEDI3, EEDI2, TDeint still rely on YUY2, but accept it directly without SSE2Tools.

Quote:
# --- REQUIREMENTS ---
#
# Input colorspaces: YV12, YV16, YUY2
#
# Core plugins:
# MVTools2 (2.7.0.22c or above from pinterf https://github.com/pinterf/mvtools/releases)
# MaskTools v2 (2.0b1 or above)
# NNEDI3 (0.9.4.6 or above)
# RgTools
#
# Additional plugins:
# NNEDI2, NNEDI, EEDI3, EEDI2, TDeInt - if selected directly or via a source-match preset
# yadifmod2 - for Preset="Ultra Fast" or if selected directly
# FFT3DFilter - if selected for noise processing
# dfttest (1.9.4 or above) - if selected for noise processing
# For FFT3DFilter & ddftest you also need the FFTW3 library (FFTW.org). On Windows the file needed for both is libfftw3f-3.dll. However, for FFT3DFilter the file needs to be called FFTW3.dll, so you will need two copies and rename one. On Windows put the files in your System32 or SysWow64 folder
# AddGrainC (1.7.0 or above) - if NoiseDeint="Generate" selected for noise bypass
For now I recommend MVTools2 2.7.0.22d as the newer versions with 10-16 bit support have been freezing AVSMeter at the end of test runs.

Please let me know if there's something I missed or if you find a problem with my changes.
Attached Files
File Type: zip QTGMC.zip (18.5 KB, 118 views)

Last edited by tebasuna51; 21st April 2017 at 12:43. Reason: code -> quote
ravewulf is offline   Reply With Quote
Old 18th December 2016, 02:31   #206  |  Link
ndjamena
Registered User
 
Join Date: Sep 2012
Posts: 366
Quote:
Originally Posted by ravewulf View Post
I'm using AviSynth 2.6 MT, so I'm not sure about AVS+. real.finder added ExtractU in the 3.33 2016-12-06 update, but maybe that part should be reverted?

I'm having trouble with MvTools2 2.7.7.22 (20161215) stalling at the end, but MvTools2 2.7.0.22d seems to be ok.

Edit:
I'll work on a version of QTGMC 3.34 that uses the 3.33 2016-04-11 version as a base instead of the 3.33 2016-12-06 version as it seems to be more stable.
Actually I read it wrong. Sorry, having trouble concentrating.

Code:
ssispmt = Findstr(VersionString(), "AviSynth+") != 0 && Findstr(VersionString(), "r1576") == 0
Lets see, if in the Version String there's the text "AviSynth+" (not 0) and there's NOT the text "r1576" (0) then use ExtractU.

Which I'm thinking at the moment means any version of AviSynth+ other than "r1576" will attempt to use ExtractU, even though the filter was only added to the most recent build and there were many releases between the two.

Does that sound about right?

I'm thinking it would have to figure out some way of converting the revision number to an integer in order to be viable in the short term and avoid this particular bug, but what do I know.
ndjamena is offline   Reply With Quote
Old 18th December 2016, 03:12   #207  |  Link
ravewulf
Registered User
 
Join Date: Dec 2010
Posts: 22
Quote:
Originally Posted by ndjamena View Post
Actually I read it wrong. Sorry, having trouble concentrating.

Code:
ssispmt = Findstr(VersionString(), "AviSynth+") != 0 && Findstr(VersionString(), "r1576") == 0
Lets see, if in the Version String there's the text "AviSynth+" (not 0) and there's NOT the text "r1576" (0) then use ExtractU.

Which I'm thinking at the moment means any version of AviSynth+ other than "r1576" will attempt to use ExtractU, even though the filter was only added to the most recent build and there were many releases between the two.

Does that sound about right?

I'm thinking it would have to figure out some way of converting the revision number to an integer in order to be viable in the short term and avoid this particular bug, but what do I know.
Yeah, that sounds about right. Maybe something like this?

Code:
BlankClip()
str = VersionString()      #get version string
strPos = FindStr(str,"(r") #find start of (r####
str = MidStr(str,strPos+2) #get substring that starts with the number
vn = Value(str)            #convert to a number value
str = String(vn > 1576)    #check against a known version
Subtitle(str)
Unfortunately it would break if they change the format of the version string. It would be easier with regular expressions.

Edit:
Better idea - take a leaf out of my web dev book and avoid version detection

Code:
BlankClip(pixel_type="YV12")
try {
    c420 = Is420(last) #code which may fail if the functions don't exist
} catch (errStr) {
    c420 = IsYV12(last) #fallback code
}
Subtitle(String(c420))
So put all the new AVS+ functions in the try block and all the fallback functions in the catch block

Last edited by ravewulf; 18th December 2016 at 03:36.
ravewulf is offline   Reply With Quote
Old 18th December 2016, 03:45   #208  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
ok new edit in the same post here http://forum.doom9.org/showpost.php?...postcount=2041

don't forget SMDegrain if you use tv_range=true (Default for now) and if you use avs+mt (not the r1576 that has no mt, r1576 is work like any normal avs 2.6) use the most last updated here or newer
__________________
See My Avisynth Stuff

Last edited by real.finder; 3rd April 2017 at 01:50.
real.finder is offline   Reply With Quote
Old 18th December 2016, 12:55   #209  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Quote:
Originally Posted by ravewulf View Post
I'm having trouble with MvTools2 2.7.7.22 (20161215) stalling at the end, but MvTools2 2.7.0.22d seems to be ok.

Edit:
I'll work on a version of QTGMC 3.34 that uses the 3.33 2016-04-11 version as a base instead of the 3.33 2016-12-06 version as it seems to be more stable.
If you experienced the stalling with YUY2 input, it may have been fixed in 2.7.8.22
pinterf is offline   Reply With Quote
Old 19th December 2016, 09:05   #210  |  Link
thanhvanchi
Registered User
 
Join Date: Dec 2016
Posts: 2
công ty chúng tôi chuyên về lắp https://internetvietnam.net/dang-ky-lap-dat-internet-tai-huyen-binh-chanh.htmlhttps://forum.audacityteam.org/viewtopic.php?t=93857https://band.us/@dulichviettiếp thị kỹ thuật số cho khách hàng.
Ngoài ra đây là điểm đến lư tương để khách hàng nhu cầu về cho mọi doanh nghiệp và hộ gia đ́nh.
__________________
G9!

Last edited by thanhvanchi; 16th February 2020 at 08:35.
thanhvanchi is offline   Reply With Quote
Old 21st December 2016, 10:02   #211  |  Link
odyssey
Registered User
 
Join Date: Dec 2003
Posts: 155
What's the best way to eliminate crashes when using MT?

I'm currently down to this, but it still crashes quite frequently, while encoding with x264 (source is PAL DVD):

SetMTMode(5, 5)
SetMemoryMax(700)
DGDecode_mpeg2source(...)
AssumeTFF()
ColorMatrix(hints=true, threads=0)
SetMTMode(2)
QTGMC(Preset="Slow", EdiThreads=1)
Distributor()

I get about 30fps which I think is pretty decent (on i7-6700K stock settings). I've got 32GB memory, so that shouldn't be a problem, but it could be that thing needs to be fine tuned better.

A few things I've noticed:

- Often when it crashes, the x264 file was encoded much bigger than it should (???)
- When I encoded a tv-series from a mostly progressive source, to remove bad deinterlacing artifacts, keeping the fps at 25, it did not crash at all.
odyssey is offline   Reply With Quote
Old 21st December 2016, 10:12   #212  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
Use Distributor() only when a specific tool requires it. The x264 encoder does usually not. Using it unnecessarily may multiply threads!
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 21st December 2016, 10:14   #213  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
odyssey, try this:

DGDecode_mpeg2source(...)
AssumeTFF()
ColorMatrix(hints=true, threads=0)
QTGMC(Preset="Slow", EdiThreads=1)
Trim(0,4999) #or whatever frame range encodes half the video

Then create another script to encode the second half and run them simultaneously. Even if they both only run at 15fps, it's be 2x15fps and should be far more stable.

Add --stitchable to the x264 command line and append the encoded video with MKVToolnix (assume an MKV output is fine).
hello_hello is offline   Reply With Quote
Old 21st December 2016, 11:02   #214  |  Link
ndjamena
Registered User
 
Join Date: Sep 2012
Posts: 366
Code:
Parts=2
Part=1

Trim(floor(framecount/Parts)*(Part-1), (floor(framecount/Parts)*Part)-1)
You could set part number by either creating the avs file via batch, adding the number to the avs file name and reading the file name from the script or checking to see if an output file for a previous part exists and if it is start the next part.

Multithreading is working fine for me though.
ndjamena is offline   Reply With Quote
Old 21st December 2016, 11:44   #215  |  Link
odyssey
Registered User
 
Join Date: Dec 2003
Posts: 155
Quote:
Originally Posted by LigH View Post
Use Distributor() only when a specific tool requires it. The x264 encoder does usually not. Using it unnecessarily may multiply threads!
Didn't work
Quote:
Originally Posted by hello_hello View Post
odyssey, try this:

DGDecode_mpeg2source(...)
AssumeTFF()
ColorMatrix(hints=true, threads=0)
QTGMC(Preset="Slow", EdiThreads=1)
Trim(0,4999) #or whatever frame range encodes half the video

Then create another script to encode the second half and run them simultaneously. Even if they both only run at 15fps, it's be 2x15fps and should be far more stable.

Add --stitchable to the x264 command line and append the encoded video with MKVToolnix (assume an MKV output is fine).
Quote:
Originally Posted by ndjamena View Post
Code:
Parts=2
Part=1

Trim(floor(framecount/Parts)*(Part-1), (floor(framecount/Parts)*Part)-1)
You could set part number by either creating the avs file via batch, adding the number to the avs file name and reading the file name from the script or checking to see if an output file for a previous part exists and if it is start the next part.

Multithreading is working fine for me though.

Not a bad idea, and may even give better fps. One major drawback is however that it complicates scripting this

I'm encoding shows, so I could probably make it easier by just encoding multiple episodes at the same time.

Also I would probably join them using ffmpeg afterwards. Gotta play a bit with this Thanks for the suggestions so far!
odyssey is offline   Reply With Quote
Old 21st December 2016, 12:13   #216  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by odyssey View Post
What's the best way to eliminate crashes when using MT?

I'm currently down to this, but it still crashes quite frequently, while encoding with x264 (source is PAL DVD):

SetMTMode(5, 5)
SetMemoryMax(700)
DGDecode_mpeg2source(...)
AssumeTFF()
ColorMatrix(hints=true, threads=0)
SetMTMode(2)
QTGMC(Preset="Slow", EdiThreads=1)
Distributor()
A few pointers/comments:
  1. You may be using old/outdated plugins. In order for us to suggest new versions you should use AVSMeter to create a "snapshot" of your Avisynth install and associated plugins. Run "AVSMeter -avsinfo -log" from the command line and post the generated log file ("avsinfo.log").
  2. "SetMemoryMax(700)" may be a bit low for 5 base threads.
  3. Consider using Avisynth+ instead of SEt's MT. At this stage it's quite stable and tests have shown good results multi-threading QTGMC().

Lastly, how are you feeding the script to x264?
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 21st December 2016, 12:53   #217  |  Link
odyssey
Registered User
 
Join Date: Dec 2003
Posts: 155
Quote:
Originally Posted by Groucho2004 View Post
A few pointers/comments:
  1. You may be using old/outdated plugins. In order for us to suggest new versions you should use AVSMeter to create a "snapshot" of your Avisynth install and associated plugins. Run "AVSMeter -avsinfo -log" from the command line and post the generated log file ("avsinfo.log").
  2. "SetMemoryMax(700)" may be a bit low for 5 base threads.
  3. Consider using Avisynth+ instead of SEt's MT. At this stage it's quite stable and tests have shown good results multi-threading QTGMC().

Lastly, how are you feeding the script to x264?
I'm using the latest plugins recommended in the first thread, with only the optimized plugin package collection.

I started out with memory at about 3200 AFAIR, but then read that it may be too high.

I'll try the Avisynth+ Thanks for the suggestion.

x264_8bit_x86.exe --crf 16 --preset slower --level 4.1 --fps 50 --sar !sar! --output "!h264file!" "!avsfile!"
odyssey is offline   Reply With Quote
Old 21st December 2016, 13:06   #218  |  Link
ndjamena
Registered User
 
Join Date: Sep 2012
Posts: 366
Hmmm. I told a lie... I AM having problems multi-threading but only when denoising is turned on (which is used by default in placebo mode). It doesn't crash but it does fill the frame with crap occasionally... I tracked it down to the dnWindow stage... maybe I'll try a different version of fft3dfilter...

Grr, I'm trying got figure out what's causing it but...

Grr:

http://forum.doom9.org/showpost.php?...postcount=2041
http://pastebin.com/raw/r4esUcVF

Whoever last edited that used the variable name "EdiMode" in SourceMatch rather than MatchEdi...
ndjamena is offline   Reply With Quote
Old 21st December 2016, 13:09   #219  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by odyssey View Post
I'm using the latest plugins recommended in the first thread, with only the optimized plugin package collection.
Yeah, these are outdated.

Posting the AVSMeter log would avoid a lot of questions, for example what OS you're running. You can also use AVSMeter to check the memory consumption of your script.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 21st December 2016, 13:29   #220  |  Link
ndjamena
Registered User
 
Join Date: Sep 2012
Posts: 366
There's no equivalent to ChromaEdi inside SourceMatch, so I have no idea what the hell to do with that.

Were EdiMode and ChromaEdi supposed to have been passed to SourceMatch or should that bit of code be in the main function?

*sigh*

I'll go back to my modified version and go back to what I was doing, maybe I'll be able to "update to the latest version" for testing tomorrow.
ndjamena is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 05:51.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.