Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 8th April 2016, 01:20   #1381  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,098
7z files on mediafire isn't a reasonable way to contribute to open source projects in 2016. Please consider putting your stuff on github so other people can easily backport your changes to the official repo or to their own forks if they want.

e: oh wait your source isn't even in the 7z. Not that anyone cares, but you should really put it up somewhere or you're technically violating the GPL.

Last edited by TheFluff; 8th April 2016 at 01:23.
TheFluff is offline  
Old 8th April 2016, 02:57   #1382  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,277
Quote:
Originally Posted by pinterf View Post
Thanks pinterf! Will definitely test it out.


Quote:
Originally Posted by TheFluff View Post
7z files on mediafire isn't a reasonable way to contribute to open source projects in 2016. Please consider putting your stuff on github so other people can easily backport your changes to the official repo or to their own forks if they want.

e: oh wait your source isn't even in the 7z. Not that anyone cares, but you should really put it up somewhere or you're technically violating the GPL.
It's already on GitHub: https://github.com/pinterf/AviSynthPlus/tree/MT-pfmod
Reel.Deel is offline  
Old 8th April 2016, 03:00   #1383  |  Link
raffriff42
Retried Guesser
 
raffriff42's Avatar
 
Join Date: Jun 2012
Posts: 1,373
It's fixed my little complaint, so I can come back to AVS+. Thanks pinterf!
raffriff42 is offline  
Old 8th April 2016, 03:44   #1384  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,098
Quote:
Originally Posted by Reel.Deel View Post
Oh okay, sorry. I didn't see it in the fork list for some reason.
TheFluff is offline  
Old 8th April 2016, 08:51   #1385  |  Link
chainik_svp
Registered User
 
Join Date: Mar 2012
Location: Saint-Petersburg
Posts: 239
ffdshow fix: https://github.com/pinterf/AviSynthPlus/pull/1

sadly memory leak when the script environment is re-inited by the ffdshow 64-bit still not fixed... (while it can be a ffdshow's issue)
__________________
SVPflow motion interpolation

Last edited by chainik_svp; 8th April 2016 at 09:05.
chainik_svp is offline  
Old 8th April 2016, 09:52   #1386  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,042
Quote:
Originally Posted by chainik_svp View Post
sadly memory leak when the script environment is re-inited by the ffdshow 64-bit still not fixed... (while it can be a ffdshow's issue)
I'd say it's definitely a ffdshow issue. ffdshow initialises IScriptEnvironment in several places and releases it with "delete env;" and "delete clip;". It's recipe for disaster. With AVS 2.6 one can use "DeleteScriptEnvironment()" which properly releases the pointers and memory.

Are you sure the same does not happen with 32 bit ffdshow?
Groucho2004 is offline  
Old 8th April 2016, 10:10   #1387  |  Link
chainik_svp
Registered User
 
Join Date: Mar 2012
Location: Saint-Petersburg
Posts: 239
> Are you sure the same does not happen with 32 bit ffdshow?

Absolutely. It's like hundreds of MBs on each re-init.
We've already talked about this back in March 2015 as I can remember...
__________________
SVPflow motion interpolation
chainik_svp is offline  
Old 8th April 2016, 10:20   #1388  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,042
Quote:
Originally Posted by chainik_svp View Post
We've already talked about this back in March 2015 as I can remember...
True, just found it.
Groucho2004 is offline  
Old 10th April 2016, 14:08   #1389  |  Link
jones1913
random user
 
Join Date: May 2014
Location: #Neuland
Posts: 108
Quote:
Originally Posted by pinterf View Post
Thanks, this has also fixed the crashes I had with QTGMC and AVS+.
__________________
BeHappy Audio Transcoder > <Doom9 forum> <Gleitz forum> <GitHub>
MP4.tool GUI for MP4Box & L-SMASH muxer > https://www.mediafire.com/folder/3i6y6cbkyhblm/MP4.tool
jones1913 is offline  
Old 11th April 2016, 00:31   #1390  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,964
Now I have a special 32 bit debug build that only differs from the working version that there is a 1/10000 or 1/1000 sec wait cycle in the caching code in the "not found but get frame from child then push it in cache".
This delay results in zero pointer frames appearing in the cache. My tests run fine with mt mode 2 but with this version Qtgmc fast is giving C0000005 exceptions at random places, if thread count is big enough. E.g. prefetch(8).
If I set all masktools2 filters to mt mode 1 (serialize) then it works again.
But its not because of masktools. Having its filters serialized only modifies the timing conditions.
There must be a race condition, the lru cacheing code seems to be perfect, I suspect the prefetcher and the worker threads and the caches are not perfercly synchronized.
Then it turned out that I can make it to freeze with simple 3 lines. When i will back at my PC I will show you.

Is there anybody here who was deeply involved in these mt and prefetch queue and cache core things? Not an easy read to reverse engineer it
pinterf is offline  
Old 11th April 2016, 09:59   #1391  |  Link
chainik_svp
Registered User
 
Join Date: Mar 2012
Location: Saint-Petersburg
Posts: 239
I can try!
__________________
SVPflow motion interpolation
chainik_svp is offline  
Old 11th April 2016, 11:48   #1392  |  Link
Chikuzen
typo lover
 
Chikuzen's Avatar
 
Join Date: May 2009
Posts: 595
Quote:
Originally Posted by pinterf View Post
Is there anybody here who was deeply involved in these mt and prefetch queue and cache core things? Not an easy read to reverse engineer it
I think the man is tp7.
I recommend you to just send a mail to him.
__________________
my repositories
Chikuzen is offline  
Old 11th April 2016, 12:19   #1393  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
tp7 hardly remembers anything about the threading codebase, pretty much all of which was written by ultim.
Have fun.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth
TurboPascal7 is offline  
Old 11th April 2016, 13:13   #1394  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,964
Thanks tp7, anyway.
There are clear drawbacks of a one man show. At least the codebase is pretty nice.
pinterf is offline  
Old 11th April 2016, 19:09   #1395  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,964
Meanwhile I think the cache corruption is solved.

Problem:
- a debug build with an extra command placed in the caching code
- random crashes 0xC0000005, null frame pointers, etc..., somewhere in the big black hole of QTGMC
- could not debug in IDE, since the 8 threads and 1/10000 sec timing differences made the problem disappear. Debugging was possible only by analyzing long debug outputs.

After some days I managed to shrink down the original QTGMC.avsi to this very-very complex script:
Code:
Blankclip(width = 320, height = 200, pixel_type = "yv12").KillAudio() 
# no freeze without KillAudio. KillAudio is giving us a new caching level
SeparateFields()
Prefetch(8)
Nice, isn't is?

And this is the code fragment that deals with case, when the Nth frame is not found in cache. (Sorry, just to feel the pain )
Code:
  PVideoFrame result;
  LruCache<size_t, PVideoFrame>::handle cache_handle;

  switch(_pimpl->VideoCache->lookup(n, &cache_handle, true))
  {
  case LRU_LOOKUP_NOT_FOUND:
    {
      try
      {
        cache_handle.first->value = _pimpl->child->GetFrame(n, env);
  #ifdef X86_32
        _mm_empty();
  #endif
        _pimpl->VideoCache->commit_value(&cache_handle);
      }
      catch (...)
      {
        _pimpl->VideoCache->rollback(&cache_handle);
        throw;
      }
#ifdef _DEBUG 
      // !!! some process during the next 1/10000 seconds is overwriting the 
      // content of this cache handle (frame) with NULL!
      // 1/10000 sec delay, but a simple _RPT debug line is enough, albeit the 
      // corruption occurs more rarely
      std::chrono::time_point<std::chrono::high_resolution_clock> t_start2, t_end2;
      std::chrono::duration<double> elapsed_seconds;
      t_start2 = std::chrono::high_resolution_clock::now(); 
      do {
        t_end2 = std::chrono::high_resolution_clock::now();
        elapsed_seconds = t_end2 - t_start2;
      } while (elapsed_seconds.count() < 1.0 / 10000.0);
      // end of delay
      assert(NULL != cache_handle.first->value); // and now it's NULL!!!
#endif
      result = cache_handle.first->value;
      // its content may change after commit when the last lock is released 
      // (cache is being restructured by other threads, new frames?)
      break;
[...]
Solution: fill 'result' _before_ the commit. Easy eh? Six letters and plus one line and a deleted line at the bottom. It took me three or four days, again. But now you can ask me about the caching.

Although it happened only to the debug version, theoretically it would occur anytime. I don't know, at different processor speeds, thread count, load, etc...

And as a new adventure I had to plunge into masktools2 a bit when I thought it would be the suspect, since putting its filters into serialized MT mode, the corruption did not occur in QTGMC. It was false alarm, it works fine in mode 2 (MT_MULTI_INSTANCE).

Last edited by pinterf; 11th April 2016 at 19:10. Reason: was too wide
pinterf is offline  
Old 11th April 2016, 20:38   #1396  |  Link
ryrynz
Registered User
 
ryrynz's Avatar
 
Join Date: Mar 2009
Posts: 3,464
Quote:
Originally Posted by pinterf View Post
Meanwhile I think the cache corruption is solved.
Great work, I think this entire thread now loves you. All hail the new Ultim.
ryrynz is offline  
Old 11th April 2016, 23:12   #1397  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,459
Let's be happy about it:

__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline  
Old 12th April 2016, 05:33   #1398  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,964
Haha
Then all readers here are obeyed to run with me on that 54K 2800D+
Just for plain solidarity
pinterf is offline  
Old 12th April 2016, 05:43   #1399  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,964
(Those numbers are not the newest avs+ versioning scheme but the data of the trail running race next weekend on which I trained almost nothing because of these avs+ riddles.
pinterf is offline  
Old 18th April 2016, 20:03   #1400  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,964
New AVS+ build: r1847

x64/x86, including XP versions

r1847 MT-pfmod (2016.04.18)
- fixed broken ffdshow integration (chainikdn)
- fixed theoretical and debug case cache corruption

Download from
https://github.com/pinterf/AviSynthP...r1847-MT-pfmod
or
http://www.mediafire.com/download/r7...r1847-pfmod.7z
pinterf is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:23.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.