Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
28th February 2016, 16:22 | #1321 | Link |
Registered User
Join Date: Apr 2009
Posts: 76
|
I used the MT build 1689 for a while alongside with QTGMC without any issues and encoded a lot of videos. Later I switched to latest MT build 1825 but didn't encode anything for a while. Recently I started encoding again and realized that all my of encodes run into a deadlock very early into the job. After going back and force between builds 1689, 1779 and 1825, I noticed that even though the exact same script runs fine under 1689, it deadlocks under 1779 and 1825, while all of the three versions give more or less the same speed (checked with AVSMeter). Anybody has any idea about this? My scripts look like this:
1689: Code:
SetMemoryMax(8000) SetFilterMTMode("", 2) #SetFilterMTMode("DEFAULT_MT_MODE", 2) LoadPlugin("DGDecode.dll") DGDecode_mpeg2source("video.d2v", info=3) LoadPlugin("ColorMatrix.dll") ColorMatrix(hints=true, threads=0) TRIM(x,x) crop(x,x,x,x) QTGMC( Preset="Very Fast", x,x,x,x,x,x ).SelectEven() Prefetch(6) Code:
SetMemoryMax(8000) #SetFilterMTMode("", 2) SetFilterMTMode("DEFAULT_MT_MODE", 2) LoadPlugin("DGDecode.dll") DGDecode_mpeg2source("video.d2v", info=3) LoadPlugin("ColorMatrix.dll") ColorMatrix(hints=true, threads=0) TRIM(x,x) crop(x,x,x,x) QTGMC( Preset="Very Fast", x,x,x,x,x,x ).SelectEven() Prefetch(6) |
29th February 2016, 09:59 | #1322 | Link |
Registered User
Join Date: Feb 2008
Posts: 59
|
Reduce your memory, maybe your running out of memory. 8GB I think is a bit much, I use no more than 2GB (2048). Unless you have 32GB of memory. I've got 16GB and are now running with an i7 slightly overclocked
Try reducing the number of threads as I think each thread has its own memory requirements, maybe start at 2 threads and work up until it deadlocks again |
29th February 2016, 10:21 | #1323 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
|
|
29th February 2016, 11:19 | #1324 | Link | ||
Registered User
Join Date: Apr 2009
Posts: 76
|
Quote:
Quote:
|
||
29th February 2016, 11:48 | #1325 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
All credit to ultim and the rest of the AVS+ dev team but the latest builds, particularly with MT and QTGMC(), are not stable. If you really have to multi-thread QTGMC(), use SEt's AVS MT. Since you're using the "very fast" QTGMC() preset, I don't think you even need Avisynth(+) MT. Some of the filters are already internally multi-threaded. Your bottleneck is most likely the encoding. For reference, here are some benchmarks with SEt's AVS MT. Last edited by Groucho2004; 29th February 2016 at 11:52. |
|
1st March 2016, 04:24 | #1326 | Link |
Registered User
Join Date: Apr 2009
Posts: 76
|
Cosmetics as in that amount is supposed to be some absolute maximum that avisynth+ is never going to reach; trying to follow the "better safe than sorry" approach, even though it might not make much sense here. My script seems to use 400-500 MB of memory with 6 threads, on build 1689.
Regarding QTGMC, I am not actually using the Very Fast preset out of the box. The preset just acts as a baseline and I have like 15-20 of QTGMC's options manually set (I removed them here for the sake of brevity), which I will modify based on the source or encoding speed constraints. The final result is something along the lines of slow or very slow. I remember I did some comparison between Avisynth+ MT and SEt's MT build a while back and the former was quite a bit faster and hence, I stuck with Avisynth+. It was before Avisynth v2.6.0 final, though. Maybe I should take a look at SEt's MT build's performance again. |
1st March 2016, 09:53 | #1327 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
|
|
2nd March 2016, 08:07 | #1328 | Link | |
Registered User
Join Date: Apr 2009
Posts: 76
|
Quote:
|
|
23rd March 2016, 13:19 | #1329 | Link |
Registered User
Join Date: Jan 2014
Posts: 2,314
|
Hello,
Last week I was starting to play with avs+ source, specifically with the MT branch because I was interested in the background of the slowdown issue. The issue was probably solved, but before I upload test dlls I would like to ask some questions. Question#1 I'm working with VS2015 on 64 bit Windows 7. I have cloned the MT branch, run cmake and built x86. After minor modifications I succeeded. Then tried to build x64 but this configuration did not exist, so I created one from VS Configuration manager. Linking was not successful, because the machine type of obj files did not match the linking target, even if I set it explicitly in the project properties (under Linker, target machine type). LNK1112: module machine type 'x64' conflicts with target machine type 'X86' I think that the generated cmake configuration is overriding this setting. The only workaround I found was the following: Compile for x86: download cmake x86 delete CMakeCache.txt from project root run cmake-gui from c:\Program Files (x86)\cmake\bin choose directories configure Specify the generator for this project: Visual Studio 14 2015 (appears /machine=x86 in xxxxx_LINKER_FLAGS) generate Start visual studio, only Win32 is available: build. Compile for x64: download cmake x64 delete CMakeCache.txt from project root run cmake gui from c:\Program Files\cmake\bin choose directories configure Specify the generator for this project: Visual Studio 14 2015 Win64 (appears /machine=x64 in xxxxx_LINKER_FLAGS) generate Start visual studio, only x64 is available: build. My question is that how can I tell cmake to generate the solution for both Win32 and x64 targets? Question#2 Forgive me, I am new to git and these open source things (it was a whole day for me to see only MT branch as a separete source ), what is the usual workflow if I'd like to participate in the project? Fork the project to my git account, and make modification inside it? Clone it? How is it done if I put a lot of debug mess in my source (and don't want to remove it on my side) but the original project does not need them? Can the owner of the main project merge only specific lines? Question#3 When I build the dll's they have the very same version (build number, etc. as the latest 'official' release. Version.h is autogenerated by some magic cmake thing. How can I specify this version manually? E.g. 1825-test instead of 1825? Thank you and sorry for the lame questions |
25th March 2016, 04:32 | #1332 | Link | |||
...?
Join Date: Nov 2005
Location: Florida
Posts: 1,420
|
Quote:
And you don't need 64-bit CMake to do this either, since the Win64-specific generator calls in the correct MSVC toolchain regardless of what CMake is. Quote:
Also remember that AviSynth+ is extraordinarily unlikely to accept compiler-specific stuff, what with sporadic efforts to move beyond MSVC for compilation. See the coding guidelines for more general information on what should or should not be acceptable. Quote:
Last edited by qyot27; 25th March 2016 at 04:40. |
|||
25th March 2016, 04:54 | #1333 | Link |
...?
Join Date: Nov 2005
Location: Florida
Posts: 1,420
|
On a quasi-related note, I've been meaning to check the build instructions I posted before against VS Community 2015, but haven't gotten around to it. They still should work, just change the -G parameter (and optionally the -T parameter if XP support still matters to you) to the correct name.
Last edited by qyot27; 25th March 2016 at 05:02. |
25th March 2016, 16:28 | #1334 | Link | |
Registered User
Join Date: Jan 2014
Posts: 2,314
|
Quote:
Regarding the multi target make file, I have read your instructions, but I was hoping that CMake evolved in the last one or two years to do that. Just recognized that VS2015 has git integration. Now it points to the official MT repository, tell me that I cannot do any harm (commit, false merge) to that, wherever I click with my mouse to |
|
29th March 2016, 11:55 | #1336 | Link |
Registered User
Join Date: Jan 2014
Posts: 2,314
|
After two weeks of learning curve, I have uploaded a new avs+ build, both 32 and 64 bit MT dlls.
This build is primarily for addressing the huge slowdown issue of recent MT builds. Please give it a try. AviSynth+ r1828-pfmod I have no slowdowns (none or very minor), so longer clips could be tested: QTGMC fast with a 92 minutes DV-AVI source. Some notes. The artificial test case of colorbars is not quite good, although it allowed me to find out the main cause of the slowdown. Colorbars is using a static source framebuffer that is kept referenced throughout the whole qtgmc process, so its ref_count will never be released. During the qtgmc script the framebuffer-reusing logic had to iterate many hundred thousand (!) entries until it would find a free entry or see that no free buffer exists. After thousands of frames, one failed lookup needed almost 0.03 seconds (normally it should lookup 1000-10000x faster!), and it got linearly worse. Todo: "hack" or parameterize colorbar to be able to generate new frames: copy the pre-generated pattern into a brand new frame instead of just linking to it. Having no slowdown I realized the next problem: framebuffers sometimes are not released. Example. The first 17800 frames of an AVI source needs approx. 200 MBytes of framebuffers to deinterlace with Qtgmc, number of framebuffers is 300. Suddenly avs+ cannot find any buffers that has reference count 0 and starts allocating new framebuffers. Reaching the 20000 frames limit (that is 40000 frames after Qtgmc) we need already 500MB, so with SetMemoryMax(512) the script is soon exiting. And later the memory consumption goes up in waves. 40525 frames (81050 QTGMC): 973 framebuffers, 613MB 49381 frames (99600 QTGMC): 1178 framebuffers, 745MB 56725 frames (113450 QTGMC): 1255 framebuffers, 816MB 61046 frames (122092 QTGMC): 1348 framebuffers, 876MB 65084 frames (130168 QTGMC): 1508 framebuffers, 980MB 68988 frames (137978 QTGMC): 1736 framebuffers, 1128MB 114512 frames (129024 QTGMC): 2121 framebuffers, 1376MB etc... Good news: my 1h32min 720x576 dv-avi video - that has finally 278000 50p frames - just succeeded with the SetMemoryMax(2048) setting. Needed approx. 1700MB memory. But it can be done with much less memory. Whick plugin or avs core logic is getting those frames stuck? Have to figure out, how. Seems it is not related to MT. Script (+having rgtools, mvtools2, masktools2.dll and nnedi3.dll in the plugins64 directory) Code:
SetFilterMTMode("DEFAULT_MT_MODE",2) SetFilterMTMode("AviSource",3) SetMemoryMax(2048) #colorbars(width = 640, height = 480, pixel_type = "yv12").killaudio().assumefps(25, 1).trim(0, 29999) Avisource("c:\tape13\Videos\Tape02_digitall_Hi8.avi").killaudio().assumefps(25,1)#.trim(0, 29999) AssumeBFF() QTGMC(Preset="Fast") prefetch(3) |
29th March 2016, 11:59 | #1337 | Link |
Registered User
Join Date: Mar 2009
Posts: 3,650
|
Gosh, might we finally have an MT build stable enough to use as a replacement to SEt's MT? Or perhaps sometime soon? Nice to see some progress finally!
Will see how it fares, thanks for the considerable contribution! |
29th March 2016, 12:16 | #1339 | Link |
Registered User
Join Date: Jan 2014
Posts: 2,314
|
One note, my mod was built with VS2015, v140_xp toolset.
I have no "virgin" machine, so it works out-of-box for me. It may require Visual C++ Redistributable for Visual Studio 2015 Update 1 for you. |
29th March 2016, 14:55 | #1340 | Link | |
Registered User
Join Date: Mar 2012
Location: Texas
Posts: 1,666
|
Quote:
I ran the same test as I did when the issue was first discovered. So far it's looking good, r1576 is still faster but that has always been the case. Here's the results: [General info] Code:
Log file created with: AVSMeter 2.1.5 (x86) Script file: M:\AviSynth+ Testing\Regression\AviSynth+ r1828 pfmod.avs Avisynth version string: AviSynth+ 0.1 (r1828, MT-pfmod, i386) Avisynth file version: 0.1.0.0 Avisynth Interface Version: 6 Avisynth MT support: Yes Avisynth.dll linker/compiler version: 14.0 Avisynth.dll location: C:\Windows\SysWOW64\AviSynth.dll Avisynth.dll time stamp: 2016-03-29, 07:50:15 PluginDir+ (HKLM, x86): C:\Program Files (x86)\AviSynth+\plugins+ PluginDir2_5 (HKLM, x86): C:\Program Files (x86)\AviSynth+\plugins [Clip info] Code:
Number of frames: 43596 Length (hh:mm:ss.ms): 00:12:07.327 Frame width: 720 Frame height: 480 Framerate: 59.940 (60000/1001) Colorspace: YV12 [Runtime info] Code:
Frames processed: 43596 (0 - 43595) FPS (min | max | average): 5.967 | 174.4 | 19.99 Memory usage (phys | virt): 166 | 175 MB Thread count: 38 CPU usage (average): 10% Efficiency index: 1.999 Time (elapsed): 00:36:21.112 [Script] Code:
SetMemoryMax(1024) LWLibavVideoSource("S:\ITAS_Chappelle\VIDEO_TS\VTS_02_1.demuxed.m2v") AssumeTFF() QTGMC(Preset="slow") Last edited by Reel.Deel; 29th May 2016 at 12:44. Reason: link |
|
|
|