Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 28th February 2016, 16:22   #1321  |  Link
THEAST
Registered User
 
Join Date: Apr 2009
Posts: 76
I used the MT build 1689 for a while alongside with QTGMC without any issues and encoded a lot of videos. Later I switched to latest MT build 1825 but didn't encode anything for a while. Recently I started encoding again and realized that all my of encodes run into a deadlock very early into the job. After going back and force between builds 1689, 1779 and 1825, I noticed that even though the exact same script runs fine under 1689, it deadlocks under 1779 and 1825, while all of the three versions give more or less the same speed (checked with AVSMeter). Anybody has any idea about this? My scripts look like this:

1689:
Code:
SetMemoryMax(8000)
SetFilterMTMode("", 2)
#SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("DGDecode.dll")
DGDecode_mpeg2source("video.d2v", info=3)
LoadPlugin("ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)
TRIM(x,x)
crop(x,x,x,x)
QTGMC( Preset="Very Fast", x,x,x,x,x,x ).SelectEven()
Prefetch(6)
1779 & 1825:
Code:
SetMemoryMax(8000)
#SetFilterMTMode("", 2)
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("DGDecode.dll")
DGDecode_mpeg2source("video.d2v", info=3)
LoadPlugin("ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)
TRIM(x,x)
crop(x,x,x,x)
QTGMC( Preset="Very Fast", x,x,x,x,x,x ).SelectEven()
Prefetch(6)
THEAST is offline  
Old 29th February 2016, 09:59   #1322  |  Link
thescrapyard
Registered User
 
thescrapyard's Avatar
 
Join Date: Feb 2008
Posts: 59
Reduce your memory, maybe your running out of memory. 8GB I think is a bit much, I use no more than 2GB (2048). Unless you have 32GB of memory. I've got 16GB and are now running with an i7 slightly overclocked

Try reducing the number of threads as I think each thread has its own memory requirements, maybe start at 2 threads and work up until it deadlocks again
thescrapyard is offline  
Old 29th February 2016, 10:21   #1323  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by THEAST View Post
Recently I started encoding again and realized that all my of encodes run into a deadlock very early into the job. After going back and force between builds 1689, 1779 and 1825, I noticed that even though the exact same script runs fine under 1689, it deadlocks under 1779 and 1825
Start reading here for a potential cause for your issue.
Groucho2004 is offline  
Old 29th February 2016, 11:19   #1324  |  Link
THEAST
Registered User
 
Join Date: Apr 2009
Posts: 76
Quote:
Originally Posted by thescrapyard View Post
Reduce your memory, maybe your running out of memory. 8GB I think is a bit much, I use no more than 2GB (2048). Unless you have 32GB of memory. I've got 16GB and are now running with an i7 slightly overclocked

Try reducing the number of threads as I think each thread has its own memory requirements, maybe start at 2 threads and work up until it deadlocks again
I have 16 GB of memory, I don't think Avisynth x86 can use more than 2 GB memory anyway, that number I am using is mostly for cosmetics. In my experience, on the latest builds, the code deadlocks even with two threads, it might just take longer. The only way to get it to work is to remove the "prefetch" line in which case the avisynth part of the work will run in single-threaded mode. Even in that case, the encode starts at something like 8 FPS but the speed gradually goes down until it becomes 1 FPS or so.

Quote:
Originally Posted by Groucho2004 View Post
Start reading here for a potential cause for your issue.
That doesn't seem to be a multi-threaded test, at least based on the reported CPU usage values. Still, it confirms my observations regarding the encoding speed going down with time using the latest builds. Do you guys recommend sticking with avisynth+ for MT or switching to normal Avisynth and SET's MT dll? I might try build r1765 later and see if the deadlock also happens there. I have a feeling this issue started when the syntax for setting default MT mode changed from "SetFilterMTMode("", 2)" to "SetFilterMTMode("DEFAULT_MT_MODE", 2)".
THEAST is offline  
Old 29th February 2016, 11:48   #1325  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by THEAST View Post
I have 16 GB of memory, I don't think Avisynth x86 can use more than 2 GB memory anyway, that number I am using is mostly for cosmetics.
Cosmetics? WTF? RTFM.

All credit to ultim and the rest of the AVS+ dev team but the latest builds, particularly with MT and QTGMC(), are not stable. If you really have to multi-thread QTGMC(), use SEt's AVS MT.

Since you're using the "very fast" QTGMC() preset, I don't think you even need Avisynth(+) MT. Some of the filters are already internally multi-threaded. Your bottleneck is most likely the encoding.

For reference, here are some benchmarks with SEt's AVS MT.

Last edited by Groucho2004; 29th February 2016 at 11:52.
Groucho2004 is offline  
Old 1st March 2016, 04:24   #1326  |  Link
THEAST
Registered User
 
Join Date: Apr 2009
Posts: 76
Cosmetics as in that amount is supposed to be some absolute maximum that avisynth+ is never going to reach; trying to follow the "better safe than sorry" approach, even though it might not make much sense here. My script seems to use 400-500 MB of memory with 6 threads, on build 1689.

Regarding QTGMC, I am not actually using the Very Fast preset out of the box. The preset just acts as a baseline and I have like 15-20 of QTGMC's options manually set (I removed them here for the sake of brevity), which I will modify based on the source or encoding speed constraints. The final result is something along the lines of slow or very slow.

I remember I did some comparison between Avisynth+ MT and SEt's MT build a while back and the former was quite a bit faster and hence, I stuck with Avisynth+. It was before Avisynth v2.6.0 final, though. Maybe I should take a look at SEt's MT build's performance again.
THEAST is offline  
Old 1st March 2016, 09:53   #1327  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by THEAST View Post
Cosmetics as in that amount is supposed to be some absolute maximum that avisynth+ is never going to reach; trying to follow the "better safe than sorry" approach
Had you read the documentation for SetMemoryMax you would know that your interpretation of the purpose of that function is completely wrong. You should hardly ever have to set it to more than about 1000.
Groucho2004 is offline  
Old 2nd March 2016, 08:07   #1328  |  Link
THEAST
Registered User
 
Join Date: Apr 2009
Posts: 76
Quote:
Originally Posted by Groucho2004 View Post
Had you read the documentation for SetMemoryMax you would know that your interpretation of the purpose of that function is completely wrong. You should hardly ever have to set it to more than about 1000.
Hmm, okay, I will change the value that I am using, thanks for the tip.
THEAST is offline  
Old 23rd March 2016, 13:19   #1329  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Hello,

Last week I was starting to play with avs+ source, specifically with the MT branch because I was interested in the background of the slowdown issue.
The issue was probably solved, but before I upload test dlls I would like to ask some questions.

Question#1

I'm working with VS2015 on 64 bit Windows 7.
I have cloned the MT branch, run cmake and built x86. After minor modifications I succeeded.
Then tried to build x64 but this configuration did not exist, so I created one from VS Configuration manager. Linking was not successful, because the machine type of obj files did not match the linking target, even if I set it explicitly in the project properties (under Linker, target machine type).
LNK1112: module machine type 'x64' conflicts with target machine type 'X86'
I think that the generated cmake configuration is overriding this setting.

The only workaround I found was the following:

Compile for x86:
download cmake x86
delete CMakeCache.txt from project root
run cmake-gui from c:\Program Files (x86)\cmake\bin
choose directories
configure
Specify the generator for this project: Visual Studio 14 2015
(appears /machine=x86 in xxxxx_LINKER_FLAGS)
generate
Start visual studio, only Win32 is available: build.

Compile for x64:
download cmake x64
delete CMakeCache.txt from project root
run cmake gui from c:\Program Files\cmake\bin
choose directories
configure
Specify the generator for this project: Visual Studio 14 2015 Win64
(appears /machine=x64 in xxxxx_LINKER_FLAGS)
generate
Start visual studio, only x64 is available: build.

My question is that how can I tell cmake to generate the solution for both Win32 and x64 targets?

Question#2
Forgive me, I am new to git and these open source things (it was a whole day for me to see only MT branch as a separete source ), what is the usual workflow if I'd like to participate in the project?
Fork the project to my git account, and make modification inside it? Clone it?
How is it done if I put a lot of debug mess in my source (and don't want to remove it on my side) but the original project does not need them? Can the owner of the main project merge only specific lines?

Question#3
When I build the dll's they have the very same version (build number, etc. as the latest 'official' release. Version.h is autogenerated by some magic cmake thing.
How can I specify this version manually? E.g. 1825-test instead of 1825?

Thank you and sorry for the lame questions
pinterf is offline  
Old 23rd March 2016, 18:23   #1330  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
Just a small remark: hg (Mercurial) is not completely equal to git.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline  
Old 25th March 2016, 03:07   #1331  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
Quote:
Originally Posted by pinterf View Post
Hello,

Last week I was starting to play with avs+ source, specifically with the MT branch because I was interested in the background of the slowdown issue.
The issue was probably solved, but before I upload test dlls I would like to ask some questions.
Awesome, I'll gladly help out with testing. Hopefully someone comes along and answer your other questions.
Reel.Deel is offline  
Old 25th March 2016, 04:32   #1332  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,419
Quote:
Originally Posted by pinterf View Post
My question is that how can I tell cmake to generate the solution for both Win32 and x64 targets?
There's no need to. AviSynth+'s build system is via CMake, not MSVC project files. CMake doesn't work this way (to my knowledge). It certainly would cause problems for the other generators - NMake and Ninja - that you can use with MSVC's compilers.

And you don't need 64-bit CMake to do this either, since the Win64-specific generator calls in the correct MSVC toolchain regardless of what CMake is.

Quote:
Question#2
Forgive me, I am new to git and these open source things (it was a whole day for me to see only MT branch as a separete source ), what is the usual workflow if I'd like to participate in the project?
Fork the project to my git account, and make modification inside it? Clone it?
How is it done if I put a lot of debug mess in my source (and don't want to remove it on my side) but the original project does not need them? Can the owner of the main project merge only specific lines?
This is the usual for git projects:
  • Clone project
  • Create new branch with descriptive name
  • Make modifications, commit changes (broken up topically, usually).
  • Submit pull request (Github) or patchset (on mailing list oriented projects) to upstream.
  • Upstream reviews patches, suggests changes, and either allows the user to fix those in the patchset or sometimes upstream does it after merging it.
If you want to 'put a lot of debug mess in my source (and don't want to remove it on my side)', I'd suggest keeping a branch specifically *for* that, and having a pull request-ready branch with the actual changes but without the debug stuff.

Also remember that AviSynth+ is extraordinarily unlikely to accept compiler-specific stuff, what with sporadic efforts to move beyond MSVC for compilation. See the coding guidelines for more general information on what should or should not be acceptable.

Quote:
Question#3
When I build the dll's they have the very same version (build number, etc. as the latest 'official' release. Version.h is autogenerated by some magic cmake thing.
How can I specify this version manually? E.g. 1825-test instead of 1825?
Don't do that. Make sure the CMake versioning generator is working correctly and allow it to do its job. If you want to keep your branched DLLs straight, I'd put the branch name in the filename of the archive you pack the DLLs in.

Last edited by qyot27; 25th March 2016 at 04:40.
qyot27 is offline  
Old 25th March 2016, 04:54   #1333  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,419
On a quasi-related note, I've been meaning to check the build instructions I posted before against VS Community 2015, but haven't gotten around to it. They still should work, just change the -G parameter (and optionally the -T parameter if XP support still matters to you) to the correct name.

Last edited by qyot27; 25th March 2016 at 05:02.
qyot27 is offline  
Old 25th March 2016, 16:28   #1334  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Quote:
Originally Posted by qyot27 View Post
This is the usual for git projects:
  • Clone project
  • Create new branch with descriptive name
  • Make modifications, commit changes (broken up topically, usually).
  • Submit pull request (Github) or patchset (on mailing list oriented projects) to upstream.
  • Upstream reviews patches, suggests changes, and either allows the user to fix those in the patchset or sometimes upstream does it after merging it.
If you want to 'put a lot of debug mess in my source (and don't want to remove it on my side)', I'd suggest keeping a branch specifically *for* that, and having a pull request-ready branch with the actual changes but without the debug stuff.
Thank you, suppose you mean "fork" in the first step?

Regarding the multi target make file, I have read your instructions, but I was hoping that CMake evolved in the last one or two years to do that.

Just recognized that VS2015 has git integration.
Now it points to the official MT repository, tell me that I cannot do any harm (commit, false merge) to that, wherever I click with my mouse to
pinterf is offline  
Old 25th March 2016, 16:34   #1335  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
No, "clone" means "get a copy from the remote repository to your local working directory" in git slang (CVS or SVN use "checkout", IIRC).
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline  
Old 29th March 2016, 11:55   #1336  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
After two weeks of learning curve, I have uploaded a new avs+ build, both 32 and 64 bit MT dlls.

This build is primarily for addressing the huge slowdown issue of recent MT builds.

Please give it a try.

AviSynth+ r1828-pfmod

I have no slowdowns (none or very minor), so longer clips could be tested: QTGMC fast with a 92 minutes DV-AVI source.

Some notes.

The artificial test case of colorbars is not quite good, although it allowed me to find out the main cause of the slowdown. Colorbars is using a static source framebuffer that is kept referenced throughout the whole qtgmc process, so its ref_count will never be released. During the qtgmc script the framebuffer-reusing logic had to iterate many hundred thousand (!) entries until it would find a free entry or see that no free buffer exists. After thousands of frames, one failed lookup needed almost 0.03 seconds (normally it should lookup 1000-10000x faster!), and it got linearly worse.

Todo: "hack" or parameterize colorbar to be able to generate new frames: copy the pre-generated pattern into a brand new frame instead of just linking to it.

Having no slowdown I realized the next problem: framebuffers sometimes are not released.

Example.
The first 17800 frames of an AVI source needs approx. 200 MBytes of framebuffers to deinterlace with Qtgmc, number of framebuffers is 300. Suddenly avs+ cannot find any buffers that has reference count 0 and starts allocating new framebuffers. Reaching the 20000 frames limit (that is 40000 frames after Qtgmc) we need already 500MB, so with SetMemoryMax(512) the script is soon exiting.

And later the memory consumption goes up in waves.

40525 frames (81050 QTGMC): 973 framebuffers, 613MB
49381 frames (99600 QTGMC): 1178 framebuffers, 745MB
56725 frames (113450 QTGMC): 1255 framebuffers, 816MB
61046 frames (122092 QTGMC): 1348 framebuffers, 876MB
65084 frames (130168 QTGMC): 1508 framebuffers, 980MB
68988 frames (137978 QTGMC): 1736 framebuffers, 1128MB
114512 frames (129024 QTGMC): 2121 framebuffers, 1376MB
etc...

Good news: my 1h32min 720x576 dv-avi video - that has finally 278000 50p frames - just succeeded with the SetMemoryMax(2048) setting. Needed approx. 1700MB memory.

But it can be done with much less memory.
Whick plugin or avs core logic is getting those frames stuck?
Have to figure out, how. Seems it is not related to MT.

Script (+having rgtools, mvtools2, masktools2.dll and nnedi3.dll in the plugins64 directory)
Code:
SetFilterMTMode("DEFAULT_MT_MODE",2)
SetFilterMTMode("AviSource",3)
SetMemoryMax(2048) 
#colorbars(width = 640, height = 480, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 29999)
Avisource("c:\tape13\Videos\Tape02_digitall_Hi8.avi").killaudio().assumefps(25,1)#.trim(0, 29999)
AssumeBFF()
QTGMC(Preset="Fast")
prefetch(3)
pinterf is offline  
Old 29th March 2016, 11:59   #1337  |  Link
ryrynz
Registered User
 
ryrynz's Avatar
 
Join Date: Mar 2009
Posts: 3,646
Gosh, might we finally have an MT build stable enough to use as a replacement to SEt's MT? Or perhaps sometime soon? Nice to see some progress finally!
Will see how it fares, thanks for the considerable contribution!
ryrynz is offline  
Old 29th March 2016, 12:15   #1338  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
Your efforts are very appreciated, pinterf.

__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline  
Old 29th March 2016, 12:16   #1339  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
One note, my mod was built with VS2015, v140_xp toolset.
I have no "virgin" machine, so it works out-of-box for me. It may require Visual C++ Redistributable for Visual Studio 2015 Update 1 for you.
pinterf is offline  
Old 29th March 2016, 14:55   #1340  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,664
Quote:
Originally Posted by pinterf View Post
After two weeks of learning curve, I have uploaded a new avs+ build, both 32 and 64 bit MT dlls.

This build is primarily for addressing the huge slowdown issue of recent MT builds.

Please give it a try.

AviSynth+ r1828-pfmod

I have no slowdowns (none or very minor), so longer clips could be tested: QTGMC fast with a 92 minutes DV-AVI source.
.....
Thanks pinterf! Any comments on what was the problem, or how did you fix it? Just curious if it had anything to do with this.

I ran the same test as I did when the issue was first discovered. So far it's looking good, r1576 is still faster but that has always been the case. Here's the results:

[General info]
Code:
Log file created with:                 AVSMeter 2.1.5 (x86)
Script file:                           M:\AviSynth+ Testing\Regression\AviSynth+ r1828 pfmod.avs
Avisynth version string:               AviSynth+ 0.1 (r1828, MT-pfmod, i386)
Avisynth file version:                 0.1.0.0
Avisynth Interface Version:            6
Avisynth MT support:                   Yes
Avisynth.dll linker/compiler version:  14.0
Avisynth.dll location:                 C:\Windows\SysWOW64\AviSynth.dll
Avisynth.dll time stamp:               2016-03-29, 07:50:15
PluginDir+    (HKLM, x86):             C:\Program Files (x86)\AviSynth+\plugins+
PluginDir2_5  (HKLM, x86):             C:\Program Files (x86)\AviSynth+\plugins

[Clip info]
Code:
Number of frames:                43596
Length (hh:mm:ss.ms):     00:12:07.327
Frame width:                       720
Frame height:                      480
Framerate:                      59.940 (60000/1001)
Colorspace:                       YV12

[Runtime info]
Code:
Frames processed:               43596 (0 - 43595)
FPS (min | max | average):      5.967 | 174.4 | 19.99
Memory usage (phys | virt):     166 | 175 MB
Thread count:                   38
CPU usage (average):            10%
Efficiency index:               1.999
Time (elapsed):                 00:36:21.112

[Script]
Code:
SetMemoryMax(1024)
LWLibavVideoSource("S:\ITAS_Chappelle\VIDEO_TS\VTS_02_1.demuxed.m2v")
AssumeTFF()
QTGMC(Preset="slow")

Last edited by Reel.Deel; 29th May 2016 at 12:44. Reason: link
Reel.Deel is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 23:23.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.