Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 21st October 2017, 21:35   #3681  |  Link
Groucho2004
►◄
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Location: A wretched hive of scum and villainy
Posts: 4,397
Quote:
Originally Posted by Myrsloik View Post
What's the proper way to detect if a plugin is compiled for the avs 2.6 or avs+ api? By whether or not it calls SetFilterMTMode() on init? Is there some other way? I want to make sure 2.6 plugins don't get exposed to avs+ only formats.
I'm not aware of any way to extract that information. For a compiled plugin DLL you can query the type and the version (C 2.0, C 2.5, CPP 2.0, CPP 2.5, CPP 2.6) by checking the DLL exports but that's about it.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 21st October 2017, 22:07   #3682  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,071
Quote:
Originally Posted by Myrsloik View Post
What's the proper way to detect if a plugin is compiled for the avs 2.6 or avs+ api? By whether or not it calls SetFilterMTMode() on init? Is there some other way? I want to make sure 2.6 plugins don't get exposed to avs+ only formats.
Is that actually strictly necessary in practice? Most plugins tend to check for the colorspaces they can handle in the constructor, and I think the colorspace checking functions in 2.6 won't misdetect new stuff as something old, although 2.5 plugins will pretty much think everything is YV12 AFAIK.
TheFluff is offline   Reply With Quote
Old 22nd October 2017, 00:03   #3683  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Ikea Chair
Posts: 2,000
Quote:
Originally Posted by TheFluff View Post
Is that actually strictly necessary in practice? Most plugins tend to check for the colorspaces they can handle in the constructor, and I think the colorspace checking functions in 2.6 won't misdetect new stuff as something old, although 2.5 plugins will pretty much think everything is YV12 AFAIK.
I don't trust plugin code to handle it properly. Never trust plugins.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 24th October 2017, 07:50   #3684  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,240
Quote:
Originally Posted by Myrsloik View Post
What's the proper way to detect if a plugin is compiled for the avs 2.6 or avs+ api? By whether or not it calls SetFilterMTMode() on init? Is there some other way? I want to make sure 2.6 plugins don't get exposed to avs+ only formats.
Avs+ filters can only provide a hint about their MT modes when avisynth core requests for it, by returning the proper mt enum on the CACHE_GET_MTMODE poll. So they do not actively set multithreading modes.
Code:
  int __stdcall SetCacheHints(int cachehints, int frame_range) override {
    return cachehints == CACHE_GET_MTMODE ? MT_NICE_FILTER : 0;
  }
Quote:
Originally Posted by Myrsloik View Post
And speaking of formats... are only formats with a pre-combined constant allowed or can I have some 16bit yuv with insanely high subsampling?
Only those pre-combined constants are allowed. NewVideoFrame explicitly checks against them (Btw., that was the reason when I was starting to mod 8-bit only avs+, because NewVideoFrame did not allow me to create a video frame with arbitrary set bit-mask flags. Nor allows arbitrary flags today but the list of the preset video format was hugely expanded wiht 10+bit and planar-with-alpha constants)

Quote:
Originally Posted by Myrsloik View Post
Btw, do any filters have meaningful planar YUVA or RGBA support or can I skip implementing it properly for now?
Masktools, Rgtools accepts planar formats with alpha. But you asked meaningful, I suppose you mean when a filter is using the clip's alpha channel for mask operation for example. I'm not aware such filter.

Quote:
Originally Posted by Myrsloik View Post
Oh, and is the threadpool and jobcompletion stuff actually documented somewhere?
No. Nor have I studied that part of avs+ in deep. I don't know whether ultim treated them as a finalized chapter in avs+ development, but since he left the scene, we'll never know.
pinterf is offline   Reply With Quote
Old 24th October 2017, 09:20   #3685  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 1,751
Quote:
Originally Posted by Myrsloik View Post
Oh, and is the threadpool and jobcompletion stuff actually documented somewhere?
Probably the most important documentation you can find is here.
jpsdr is offline   Reply With Quote
Old 24th October 2017, 09:23   #3686  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Ikea Chair
Posts: 2,000
Quote:
Originally Posted by jpsdr View Post
Probably the most important documentation you can find is here.
Interesting, with that description I could actually implement it if an interesting plugin using it is found.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 24th October 2017, 11:04   #3687  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 1,751
Personnaly i don't because i don't want my plugins be "avs+ only", and unfortunately i don't think any plugin is using it actualy.
jpsdr is offline   Reply With Quote
Old 25th October 2017, 21:37   #3688  |  Link
sausuke
Registered User
 
Join Date: Dec 2011
Posts: 23
can I ask why the thread count of avisynth+ is always at 37? I don't use MT in the script and on avs meter why my CPU usage is only 2%? I used avisynth MT too and the CPU usage is only 8% max>

script:
Quote:
AVISource("E:\2Encoded Files\klsdjfjklfdsakjlfdsa.avi", audio=false).AssumeFPS(60000,1001)
ConvertToYV12(matrix="PC.709")


I'm frameserving from Sony Vegas 15. It uses all the cores 99% with no speed benefit (with MT when using MEGUI). I even used the ultrafast preset on x264 same 49fps. Is this a bug?

PS: when I used Prefetch(4) the thread count goes to 41

Last edited by sausuke; 25th October 2017 at 21:39.
sausuke is offline   Reply With Quote
Old 26th October 2017, 10:31   #3689  |  Link
tuanden0
Registered User
 
Join Date: Oct 2016
Posts: 97
@sausuke:
I think they not yet optimize for your CPU. Almost people using AMD Ryzen TR have same issue with you.
tuanden0 is offline   Reply With Quote
Old 26th October 2017, 11:06   #3690  |  Link
sausuke
Registered User
 
Join Date: Dec 2011
Posts: 23
Quote:
Originally Posted by tuanden0 View Post
@sausuke:
I think they not yet optimize for your CPU. Almost people using AMD Ryzen TR have same issue with you.
yeah same with Avisynth MT, when I used high threads on SetMTMode the speed is faster but sometimes the video jitters randomly on the first seconds. Some encodes are not. Still testing...
sausuke is offline   Reply With Quote
Old 26th October 2017, 11:27   #3691  |  Link
Groucho2004
►◄
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Location: A wretched hive of scum and villainy
Posts: 4,397
Quote:
Originally Posted by sausuke View Post
can I ask why the thread count of avisynth+ is always at 37?
It's not always 37 threads, just in your specific scenario.


Quote:
Originally Posted by sausuke View Post
I don't use MT in the script and on avs meter why my CPU usage is only 2%? I used avisynth MT too and the CPU usage is only 8% max>
I think that Vegas simply doesn't serve frames to Avisynth fast enough and therefore creating a bottleneck. That also explains the low CPU usage.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 26th October 2017, 12:19   #3692  |  Link
sausuke
Registered User
 
Join Date: Dec 2011
Posts: 23
Quote:
Originally Posted by Groucho2004 View Post
It's not always 37 threads, just in your specific scenario.


I think that Vegas simply doesn't serve frames to Avisynth fast enough and therefore creating a bottleneck. That also explains the low CPU usage.
the video preview on sony vegas is so fast though even the video realtime in frameserver (100%+) when using avsmeter (100-150fps) using other scripts and avisynth.

the thing is all the avisynth I used (MT,avs+MT) I need to push the Queue button on MEGUI many times (1x to 3x) to get a fast FPS, when I pushed it and the video is slow they have the same cpu usage too, so I abort again and push then pray that it will be fast (40fps=slow 75fps=fast)

here's the screenshot using this script with avsmeter:

script:

Quote:
SetMemoryMax(1536)
SetMTMode(5,4)
AVISource("E:\2Encoded Files\kljsdfkjlfdskjlfds.avi", audio=false).AssumeFPS(60000,1001)
SetMTMode(2,4)
ConvertToYV12(matrix="PC.709")
return last



same scenario on avsmeter, in that screenshot that is the fastest, I first drag the script and it only have 50fps so I close avsmeter and drag again then that's the fastest (I always look on Vegas Preview too and the frameserver percentage). Dunno if it's the architecture of the threadripper though first time experiencing this.

EDIT:

here's what I'm saying

Slow: 30+ FPS



Fast: 70+ FPS (too lazy to wait) xD



That's the two scenario when I want to encode either get the 30fps one or the 70fps one. When using the normal avisynth always 40fps

PS: Look also the jobs, 11 to 13 hence I aborted in Job 12 'cause I get the same slow fps and get the fast FPS in Job 13

Last edited by sausuke; 26th October 2017 at 12:42.
sausuke is offline   Reply With Quote
Old 26th October 2017, 12:44   #3693  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 267
What is the pixel format for the ouput , 8bits or 32bits floating ?
Also if not already done disable the preview rendering in vegas before frameserving, this make it run like a turtle when frameserving.

I don't have this CPU ( 3770k@4Ghz here) but I frameserve from vegas14 my output file directly to a 4GB Ramdisk , vegas temp folder is also set on the Ramdrive. On AVSmeter I usually get around 10-12% CPU usage for 1080p60fps footage and 20-25% CPU usage for 720p30FPS footage with average 250frames output according to avsmeter.

When i have the possibility, when it's not too big i put my source video on the Ramdisk too and work from there on vegas and also output to the ramdrive the frameseved and avisynth created files, wish i had more RAM to make bigger ramdisk tho.

Edit: took me time to answer with window opened, just seen the previous post after i sent mine.

Last edited by Yanak; 26th October 2017 at 22:35.
Yanak is offline   Reply With Quote
Old 26th October 2017, 12:59   #3694  |  Link
sausuke
Registered User
 
Join Date: Dec 2011
Posts: 23
Quote:
Originally Posted by Yanak View Post
What is the pixel format for the ouput , 8bits or 32bits floating ?
Also if not already done disable the preview rendering in vegas before frameserving, this make it run like a turtle when frameserving.

I don't have this CPU ( 3770k@4Ghz here) but I frameserve from vegas14 my output file directly to a 4BG Ramdisk , vegas temp folder is also set on the Ramdrive. On AVSmeter I usually get around 10-12% CPU usage for 1080p60fps footage and 20-25% CPU usage for 720p30FPS footage with average 250frames output according to avsmeter.

When i have the possibility, when it's not too big i put my source video on the Ramdisk too and work from there on vegas and also output to the ramdrive the frameseved and avisynth created files, wish i had more RAM to make bigger ramdisk tho.

Edit: took me time to answer with window opened, just seen the previous post after i sent mine.
8bits only, afaik the encoding suffers when I do 32bit floating in sony vegas. (same when I'm using my i7-5960x) so I always use 8bits.

Gonna test that preview rendering thank you

EDIT:

Same even the preview rendering is disabled. It goes slow or fast fps when encoding.

Last edited by sausuke; 26th October 2017 at 14:42.
sausuke is offline   Reply With Quote
Old 26th October 2017, 18:24   #3695  |  Link
Yanak
Registered User
 
Join Date: Oct 2011
Posts: 267
I don't know what is the bottleneck for you, preview screw up my speed on my system :/

Made a quick test and using aviynth x64 that is not compatible with the frame-server output format from debugmodeserver but under MP_Pipeline and a win32
process inside it i got this for the cpu usage:


Was a short 1080p video, stored on the ramdisk and debugmodeframserver was outputting it at 211% speed into avisynth.

Like Groucho said i think the problem is not avisynth itself, or not totally at least, maybe the HDD speed vs me using a Ramdisk mostly, or some vegas performances parameters maybe, hard to say more on this.

Edit : forgot i had megui installed on my machine, using only Staxrip since a while, but made a test in native x86 with it :


Output from frameserver was 230 to 238% ...

Last edited by Yanak; 22nd January 2018 at 22:47.
Yanak is offline   Reply With Quote
Old 26th October 2017, 21:06   #3696  |  Link
sausuke
Registered User
 
Join Date: Dec 2011
Posts: 23
hmm I've tried a normal avi and the normal speed is 400fps (not frame serving) but I've tried my nvenc codec recordings and it will not have preview on megui so I can't test (megui crashing). Maybe the nvenc codec is the bottleneck problem? I can't try the avi on sony vegas 'cause it can't accept now which is weird.

EDIT:

oh my it seems the nvenc codec (in MP4) is the bottleneck, I run a test using avi file and it's so fast in sony vegas (frameserver/avisynth+without mt), the problem is now I will use AVI on my recordings on bandicam though (have problems in sounds)

will test the other option though

EDIT:

Finally solved it, it seems the source is the culprit all along (should have read the bandicam faq) I've been using H264 on the settings all these years and not X264 in FourCC code options. Thank you guys for the comments, Groucho2004 for avsmeter (helps alot), Yanak (got the idea of using other source because of your screenshots) xD

EDIT:

Can I ask if avsmeter preset is the ultrafast of x264 which why is it fast? thank you

Last edited by sausuke; 27th October 2017 at 03:08.
sausuke is offline   Reply With Quote
Old 27th October 2017, 08:39   #3697  |  Link
Groucho2004
►◄
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Location: A wretched hive of scum and villainy
Posts: 4,397
Quote:
Originally Posted by sausuke View Post
Can I ask if avsmeter preset is the ultrafast of x264 which why is it fast? thank you
AVSMeter simply reads the frames from your script, there's no encoding involved. It measures how fast Avisynth can serve frames to your encoder (x264).
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 27th October 2017, 11:21   #3698  |  Link
sausuke
Registered User
 
Join Date: Dec 2011
Posts: 23
Quote:
Originally Posted by Groucho2004 View Post
AVSMeter simply reads the frames from your script, there's no encoding involved. It measures how fast Avisynth can serve frames to your encoder (x264).
now I understand, hmm the problem appears again but not as bad, the script on avs meter is fast now



now it's either the Sony vegas or my computer, dunno if my it's my ram or my overclock, the speed is random when encoding (more on fast now) I always check avsmeter it's always on that speed so I already have fast frameserver.

what did I do:

Increased Sony vegas 15 threads to 48 (already tried only 1 and the encoding slows down)

I've already done the ramdisk and the slow fps does happen too so I think I don't have bottleneck on my HDD, I checked the taskmanager too

Gonna reset all the parameters on BIOS if it will help

EDIT:

Alright I finally know what the cause, 100% fix. This is all about the ram. I thought my 1 of my 4 ram is busted (can't boot with XMP on) so I've removed it then my computer runs in triple channel (2666Mhz XMP On). Yesterday I'm googling about threadripper performance problem and it said that low speed and not quad channel ram causes performance decrease up to 50% and infinity fabric relies on faster ram speed. So I put again my 1 8gb ram (for quadchannel again), overclock all my 32gb to 3000Mhz and it boots. I test it on all my .veg files with MEGUI and wallahh: always fast encoding, no more abort and pray to be faster.

It seems the ram speed needed to communicate to the processor, I have feeling about this though cause why sometimes it only uses the other half of the die



Hope this will help who gonna buy Ryzen or Threadrippers, buy high speed rams or overclock it (if you want to save money with previous ram). Didn't know this can affect the CPU performance that match in AMD (first time using AMD) haven't research much about this I always thought it's the same with intel, all about CPU not affecting by ram specification

Last edited by sausuke; 29th October 2017 at 21:17.
sausuke is offline   Reply With Quote
Old 31st October 2017, 15:26   #3699  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,071
Continuing the discussion from here, let's talk about those nasty memory offsets again.

In avisynth.h from 2.6:
Code:
size_t (VideoFrameBuffer::*GetDataSize)() const;
Code:
size_t (VideoFrame::*GetOffset)(int plane) const;
Code:
class VideoFrameBuffer {
  BYTE* const data;
  const size_t data_size;
  (...)
  protected:
    VideoFrameBuffer(size_t size);
  (...)
  public:
    size_t GetDataSize() const AVS_BakedCode( return AVS_LinkCall(GetDataSize)() )
Code:
class VideoFrame {
  (...)
  const size_t offset;
  const int pitch, row_size, height;
  const size_t offsetU, offsetV; // U&V offsets are from top of picture.
  (...)
  VideoFrame(VideoFrameBuffer* _vfb, size_t _offset, int _pitch, int _row_size, int _height);
  VideoFrame(VideoFrameBuffer* _vfb, size_t _offset, int _pitch, int _row_size, int _height,
             size_t _offsetU, size_t _offsetV, int _pitchUV, int _row_sizeUV, int _heightUV);

  void* operator new(size_t size);

  // generally you shouldn't use these three
  (...)
  size_t GetOffset(int plane=0) const AVS_BakedCode( return AVS_LinkCall(GetOffset)(plane) )
In Avs+ all these size_t's are (32-bit signed) int instead. In Avisynth it doesn't really matter in practice because a frame is always allocated as one big chunk of memory, and nobody really has any use for more than 2^31 bytes of vfb. If you ever wanted to stop doing this though and use one individual pointer per plane like VS does, this won't fly because you can't reliably stuff a 64-bit pointer in a 32-bit integer, and it's generally bad coding practice to not use size_t (or ptrdiff_t) for these things.

Way back in the day ultim claimed that he didn't want to follow IanB's lead and switch to size_t because it'd break a number of completely irrelevant plugins (that would have been trivial to recompile). I didn't realize it at the time, but I'm pretty sure that in practice, he was wrong even on the technical aspect. If you look at these functions above, the only one that conceivably might end up getting called by plugins in reality is VideoFrame::GetOffset(), and I'm 99% sure that its return value just gets passed in a register and zero-extended, so as long as you keep passing the same old less-than-2^31 offsets, everything will keep working just fine (but most plugins just use GetRead/WritePtr). New VideoFrames and VFB's are only constructed by env->NewVideoFrame, so changing the signature of the VideoFrame constructor shouldn't be an issue.

Last edited by TheFluff; 31st October 2017 at 15:36.
TheFluff is offline   Reply With Quote
Old 31st October 2017, 18:32   #3700  |  Link
enctac
Registered User
 
Join Date: May 2017
Posts: 8
r2508 RGB->Y8 conversion problem

r2508 RGB->Y8 [incorrect]
r2508 RGB->YV24->Y8 [correct]
2.6MT RGB->Y8 [correct]

OS: Win10 CU
CPU:i7-4702MQ(Haswell)

Code:
function LumaBlock(int n){
    return BlankClip(width=60,height=400,pixel_type="RGB32",color=n*(65536+256+1)).Subtitle(String(n),align=2).KillAudio()
}

StackHorizontal( LumaBlock(0), LumaBlock(8), LumaBlock(16), LumaBlock(32), LumaBlock(64), LumaBlock(96), LumaBlock(128), \
    LumaBlock(160), LumaBlock(192), LumaBlock(224), LumaBlock(235), LumaBlock(245), LumaBlock(255) )

# Avisynth+ r2508 needs ConvertToYV24()
# ConvertToYV24()

ConvertToY8()
Subtitle(VersionString,align=9)

# Show Histogram
ConvertToYV24()
Histogram("levels")

Subtitle("r2508 RGB32------->Y8",align=7)
#Subtitle("r2508 RGB32->YV24->Y8",align=7)
#Subtitle("2.6MT RGB32------->Y8",align=7)

ConvertToRGB()
Attached Images
 

Last edited by enctac; 31st October 2017 at 19:17.
enctac is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 01:53.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.