Log in

View Full Version : Avisynth+


Pages : 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 [51] 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112

qyot27
27th October 2016, 23:23
Any thoughts on 16-bit packed RGBA (the other question)?

AFAIK, all RGB formats exit AviSynth+ the same way it stores RGB: bottom-first. The cases in FFmpeg's AviSynth demuxer (https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/avisynth.c#L667) demonstrate this, as the existing 2.6-era check for flipping BGR/BGRA worked as-is with BGR48/BGRA64. Even the planar formats need to be flipped by the receiving application (to use FFmpeg as an example again, the check for Planar RGB in the AviSynth demuxer uses the exact same method to flip the video, but has to be separated from the packed RGB check simply because it has to be hidden from 2.6 for compatibility reasons).

Whether AviSynth(+) flips top-first RGB data when input so that it's bottom-first internally, I don't know.

qyot27
27th October 2016, 23:34
This may be strange, but why we don't has hex colors (aka Web colors (https://en.wikipedia.org/wiki/Web_colors))?

I think it will be useful in Motion Analyze (http://forum.doom9.org/showpost.php?p=1783788&postcount=80)
colors_rgb.avsi?

Although, I'm not sure whether there was some kind of regression which makes AviSynth+ not autoload colors_rgb.avsi even though it's in an autoload directory (might be because it sets global values?). You have to manually Import() it to get avsplus to recognize the color constants, but specifying them manually (color=$VALUE) does work, script loaded or not.

It also seems to work regardless of bit depth. Painting a frame as color_crimson - or its corresponding $VALUE in colors_rgb.avsi - in BlankClip outputs the same shade of red whether you're on YUV420P10, GBRP12, YUV444P16, etc. Unless, of course, that's just a deficiency of my computer monitor.

real.finder
27th October 2016, 23:42
I've no idea how you would want the hex colors arranged (if more than 8 bits per channel),
but maybe some of the scripting here could be persuaded to provide whatever you require.

http://forum.doom9.org/showthread.php?p=1711065&highlight=colors_rgb.avsi#post1711065

https://dl.dropboxusercontent.com/s/akrph1tq1tszmsi/hexyuv.gif

EDIT: Colors are from Avisynth provided colors_rgb.avsi in plugins folder.


The duplicate color in colors_rgb_avsi should have been fixed in v2.6, but not in above GIF.

colors_rgb.avsi?

Although, I'm not sure whether there was some kind of regression which makes AviSynth+ not autoload colors_rgb.avsi even though it's in an autoload directory (might be because it sets global values?). You have to manually Import() it to get avsplus to recognize the color constants, but specifying them manually (color=$VALUE) does work, script loaded or not.

It also seems to work regardless of bit depth. Painting a frame as color_crimson - or its corresponding $VALUE in colors_rgb.avsi - in BlankClip outputs the same shade of red whether you're on YUV420P10, GBRP12, YUV444P16, etc. Unless, of course, that's just a deficiency of my computer monitor.

thank you both, but I mean as new color format that has one plane with 16 and 24 or more bits, to use it in Motion Analyze for instance

Ignus2
27th October 2016, 23:47
AFAIK, all RGB formats exit AviSynth+ the same way it stores RGB: bottom-first. The cases in FFmpeg's AviSynth demuxer (https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/avisynth.c#L667) demonstrate this, as the existing 2.6-era check for flipping BGR/BGRA worked as-is with BGR48/BGRA64. Even the planar formats need to be flipped by the receiving application (to use FFmpeg as an example again, the check for Planar RGB in the AviSynth demuxer uses the exact same method to flip the video, but has to be separated from the packed RGB check simply because it has to be hidden from 2.6 for compatibility reasons).

Whether AviSynth(+) flips top-first RGB data when input so that it's bottom-first internally, I don't know.

In avs+ it does for AviSource but it doesn't for DirectShowSource currently (doesn't flip if the height is negative), so for DSSource that's a bug I know of, but I'll fix that.

The question precisely is:
In avi/vfw/dshow for fccs BGR0 (BGR[48]) and BRA@ (BRA[64])

should we assume that they are bottom-first for positive height and top-first for negative height like DIB formats (BI_RGB 24/32 bits) are, OR
should we assume that they are always top-first regardless of the sign of the height?


There really is no precedent in this case, so I believe it will be as we define, and I'd like to have opinions on this.
My take would be point 2. above.

BTW, it doesn't matter how avisynth stores it internally to decide the above question (as it's about how to expect data in avi/vfw/dshow for the above fccs, not about how avisynth stores it). The flipping you referenced in ffmpeg has to deal with flipping as it's directly accessing avisynth's internal representation (and yes, it will be correct for CS_BGR48 and CS_BGR64 too).

Greets,
I.

pinterf
28th October 2016, 08:16
colors_rgb.avsi?

Although, I'm not sure whether there was some kind of regression which makes AviSynth+ not autoload colors_rgb.avsi even though it's in an autoload directory (might be because it sets global values?). You have to manually Import() it to get avsplus to recognize the color constants, but specifying them manually (color=$VALUE) does work, script loaded or not.

It also seems to work regardless of bit depth. Painting a frame as color_crimson - or its corresponding $VALUE in colors_rgb.avsi - in BlankClip outputs the same shade of red whether you're on YUV420P10, GBRP12, YUV444P16, etc. Unless, of course, that's just a deficiency of my computer monitor.
8 bit colors constants are automatically scaled for high bit-depth.
When I was porting that part for high bit-depth, there came the first surprise: could't use 64 bit values.

So there is no int64 in avisynth. I then wanted to introduce "double" floating point type, when I fixed double usage for Virtualdub filters, and encountered the same barrier.

Later ultim told me why this is normal: AVSValue struct has no place for 64 bit integer in 32 bit Avisynth. Under 64 bit Avisynth, the pointers are 8 bit, so they are the biggest placeholders, and we would use long long or double. But under 32 bit Avisynth, the pointers are 32 bit, so there is no place for double or int64.

I had to put a nice comment in order not to spend hours again with the impossible mission.

AVSValue struct:
short type; // 'a'rray, 'c'lip, 'b'ool, 'i'nt, 'f'loat, 's'tring, 'v'oid, or RFU: 'l'ong ('d'ouble)
short array_size;
union {
IClip* clip;
bool boolean;
int integer;
float floating_pt;
const char* string;
const AVSValue* array;
#ifdef X86_64
// if ever, only x64 will support. It breaks struct size on 32 bit
__int64 longlong; // 8 bytes
double double_pt; // 8 bytes
#endif


The second surpise I experienced came two days ago. I wanted to specify exact 16 bit color values for BlankClip, but I could't of course, and decided to introduce a new parameter.
I decided to use an array, because I'm fed up with the zillions of parameter names. I wanted to use a simple parameter usage like colors = [65535, 65535, 65535]. Failed.

Searched topics, but I found no evidence of arrays in native Avisynth scripting language. Internally there exists an array type for AVSValue.
So my question: is it possible to use arrays somehow, without any external hack?

pinterf
28th October 2016, 08:20
AFAIK, all RGB formats exit AviSynth+ the same way it stores RGB: bottom-first. The cases in FFmpeg's AviSynth demuxer (https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/avisynth.c#L667) demonstrate this, as the existing 2.6-era check for flipping BGR/BGRA worked as-is with BGR48/BGRA64. Even the planar formats need to be flipped by the receiving application (to use FFmpeg as an example again, the check for Planar RGB in the AviSynth demuxer uses the exact same method to flip the video, but has to be separated from the packed RGB check simply because it has to be hidden from 2.6 for compatibility reasons).

Whether AviSynth(+) flips top-first RGB data when input so that it's bottom-first internally, I don't know.

Planar RGB is not flipped, packed RGB formats are flipped internally (and causing headaches sometimes :) )

jpsdr
28th October 2016, 13:11
A little question...
Where (or when) resizer_h_ssse3_as_avx_generic_int16_float is used ? Maybe i've missed it but don't see it in horizontal GetResampler function.

pinterf
28th October 2016, 13:27
A little question...
Where (or when) resizer_h_ssse3_as_avx_generic_int16_float is used ? Maybe i've missed it but don't see it in horizontal GetResampler function.
resample.cpp.
Use my repo (https://github.com/pinterf/AviSynthPlus/tree/MT), that is the current one, including avisynth.h, etc..
AVX/AVX2/SSE2 functions can be compiled with avx or avx2 flag per file, while the whole project can support only SSE2.
In such modules we can use SSE2 intrinsics using AVX/AVX2 features (less used registers, optimized intrinsics) that may run faster.
Even if we are not using mm256 intrinsics. (And don't forget _mm256_zeroupper() at the end to avoid AVX-SSE2 transition penalties)

jpsdr
28th October 2016, 13:34
Ok, i'll check, thanks.

pinterf
28th October 2016, 13:37
Ok, i'll check, thanks.
Oh, I can see, it's RFU.
Tried to compile the existing SSSE3 code under avx flag, but did not use it finally, there was no measurable speed gain for me. Maybe I give it another test for it.

LigH
28th October 2016, 15:00
Just to mention a related issue here, even though it may be the responsibility of the source filters after all:

AviSynth+ may support high bit depths. But reading source videos with high bit depth may not be simple in every case. Source filters may have to be aware of AviSynth+ being able to support HBD. AviSynth+ may have to request HBD output of source filters if desired. Native source plugins based on libavcodec may be able to support HBD but indexing intra-frame only HBD source formats (like Apple ProRes) could require a lot of preparation time and either disk space or RAM, depending on where the index is kept. Source filters relying on Windows provided APIs (VfW, DirectShow) may access such files faster but will depend on the ability of these APIs to provide HBD decoding results.

Unfolding these dependencies may require some deeper insight in the AviSynth(+) kernel, and probably also cooperation with authors of plugins like DirectShowSource, FFMS2, L-SMASH Works, or DSS2Mod (which can use both DS and LAV API).

This thread may or may not be the optimal location to discuss this matter further; your decision...

leoenc
28th October 2016, 16:43
It would be very helpful if DirectShowSource could support 10-bit from LAV filters.
That would enable skipping the indexing, which is taking really long with big ProRes files.

See discussion here: http://forum.doom9.net/showthread.php?p=1784106#post1784106

jpsdr
28th October 2016, 16:44
Euh... ... ... No... I don't konw what RFU mean, and i'm not able to figure out, even if i have an idea of the meaning.

pinterf
28th October 2016, 17:12
Reserved for future use. I left there for experimenting

Stephen R. Savage
29th October 2016, 01:46
Is there an avs2avi, avs2pipemod, etc. for AVS+? How can it be piped or dumped to RAW file?

Reel.Deel
29th October 2016, 02:01
Is there an avs2avi, avs2pipemod, etc. for AVS+? How can it be piped or dumped to RAW file?

See here: https://github.com/chikuzen/avs2pipemod/releases

qyot27
29th October 2016, 02:54
Is there an avs2avi, avs2pipemod, etc. for AVS+? How can it be piped or dumped to RAW file?
FFmpeg can be used for this as well. The new pix_fmts are supported by FFmpeg-git and 3.2. The only limiting factor being whether FFmpeg actually supports dumping X format to RAW.

A mostly equivalent patch for Libav is on their mailing list. Not as many of the new pix_fmts are supported as with FFmpeg, due to Libav just not having those formats in swscale/avutil yet. Of course, neither of them support *all* of the new formats (neither project has GRAY10/12/14 or Float formats).

The most recent x264 builds also support some of the new formats on the YUV side, which for some users and usecases would obviate the need for piping.

Stephen R. Savage
29th October 2016, 03:54
Here's some info about PINTERF resizer:


Resize 4:4:4 from 1280x720 to 1920x1080
Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30 GHz
16.0 GB DDR3-2400

AVS+ r2294 (pinterf) ST
Taps BYTE WORD HALF FLOAT
0 183.117 82.142 N/A 108.707
1 153.657 69.759 N/A 97.38
2 126.769 61.851 N/A 86.828
3 110.12 55.154 N/A 76.412
4 97.542 49.995 N/A 68.442

VapourSynth R35 ST
Taps BYTE WORD HALF FLOAT
0 157.44 200.08 256.02 190.77
1 153.82 196.2 220.86 176.05
2 122.25 148.96 183.66 152.78
3 105.39 126.4 149.01 132.08
4 91.5 106.28 125.43 114.43

AVS+ r2294 (pinterf) MT
Taps BYTE WORD HALF FLOAT
0 795.545 364.365 N/A 345.453
1 668.338 303.398 N/A 273.654
2 569.314 267.255 N/A 266.011
3 489.536 233.877 N/A 262.123
4 429 210.537 N/A 251.525

VapourSynth R35 MT
Taps BYTE WORD HALF FLOAT
0 679.12 802.75 881 451.18
1 675.3 782.14 818.08 437.35
2 535.58 662.38 744.48 432.46
3 446.2 539.06 651.33 418.06
4 379.52 453.13 563.34 403.91


Looks like the new code paths could use some work. Maybe it's because of the lack of softWIRE?

pinterf
29th October 2016, 04:26
Here's some info about PINTERF resizer:


Resize 4:4:4 from 1280x720 to 1920x1080
Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30 GHz
16.0 GB DDR3-2400

AVS+ r2294 (pinterf) ST
TapsBYTEWORDHALFFLOAT
0183.11782.142N/A108.707
1153.65769.759N/A97.38
2126.76961.851N/A86.828
3110.1255.154N/A76.412
497.54249.995N/A68.442

VapourSynth R35 ST
TapsBYTEWORDHALFFLOAT
0157.44200.08256.02190.77
1153.82196.2220.86176.05
2122.25148.96183.66152.78
3105.39126.4149.01132.08
491.5106.28125.43114.43

AVS+ r2294 (pinterf) MT
TapsBYTEWORDHALFFLOAT
0795.545364.365N/A345.453
1668.338303.398N/A273.654
2569.314267.255N/A266.011
3489.536233.877N/A262.123
4429210.537N/A251.525

VapourSynth R35 MT
TapsBYTEWORDHALFFLOAT
0679.12802.75881451.18
1675.3782.14818.08437.35
2535.58662.38744.48432.46
3446.2539.06651.33418.06
4379.52453.13563.34403.91


Looks like the new code paths could use some work. Maybe it's because of the lack of softWIRE?
Thank you for the benchmarks! Resizers do have intrinsics, but for 10-16 bits they use float internally, it was the fastest solution at that time replacing pure C. As jpsdr also revealed, I left yet-not-live parts in the code for I was experimenting.

Sure, phase#2 will be about making things faster and smarter and also avx2 friendly. Just give hints what function needs polishing (o.k. many of them, a lot of 10+ bit code paths don't have optimizazions). I'd better put behind rarely used filters in the queue.

Stephen R. Savage
29th October 2016, 07:57
Thank you for the benchmarks! Resizers do have intrinsics, but for 10-16 bits they use float internally, it was the fastest solution at that time replacing pure C. As jpsdr also revealed, I left yet-not-live parts in the code for I was experimenting.

Sure, phase#2 will be about making things faster and smarter and also avx2 friendly. Just give hints what function needs polishing (o.k. many of them, a lot of 10+ bit code paths don't have optimizazions). I'd better put behind rarely used filters in the queue.

I took a look at resample.cpp. The biggest change you could make that would improve performance is to combine FilteredResizeH and FilteredResizeV into a FilteredResize2D wrapper. Instead of having the individual kernels process the entire plane, make them generate scanline bundles and work from a temporary buffer. This pipeline operation is one of the big tricks in z.lib used by VapourSynth. Adding AVX integer paths is not as beneficial as it might seem, because of high overhead in unpacking across the vector lanes.

...You could also just link in z.lib into your project instead of writing it all again.

BTW: It's very confusing how you have both MT and MT-pinterf branch (and how MT is the real branch).

vcmohan
29th October 2016, 08:39
Some help needed. I tried on my laptop windows 10 home edition to place in the system32 and systemwow64 folders the avisynth dll from the avisynthplus latest build. Both folders I got error messages from 7zip by which I am trying to extract. I tried deleting the older version manually, ( it disappeared from view alright) but 7zip still finds it and asks me whether I should replace. After yes, it again gives error. Now I do not find the older avisynth.dll in the system32 folder even if I search for it, but 7zip always finds the old one. What should I do?
In the zip file there are devil.dll in folders named system. Are these to be placed in plugins folders?

LigH
29th October 2016, 09:18
Your confusion sounds like those of 64 bit system files being shadowed from 32-bit applications.

If you use a 32-bit 7-zip, it can only access the SysWOW64 folder, which is camouflaged as system32 for all 32-bit applications. Are you sure you use a 64-bit 7-zip?

There are file managers which do not depend as much on the result of Windows system calls, but prefer showing the disk content as it is stored. Also they support privilege escalation if required to overwrite UAC protected files. Being old enough to have used the Norton Commander long ago, I enjoy the Far manager v3.0 in 64 bit in these days. It supports many archive formats internally, like a sub-directory, including 7-zip. And in addition, the 7-zip "extra" pack even provides an always up-to-date Far plugin.

jpsdr
29th October 2016, 10:11
I took a look at resample.cpp. The biggest change you could make that would improve performance is to combine FilteredResizeH and FilteredResizeV into a FilteredResize2D wrapper.

When i take a look at the 1rst (or original) version of the code, there was only the resizev (if i remember properly). It took me a while to understand the "why", until i realise that it was just a trick, to handle the same way interleaved standard (RGB24, YUYV for exemple) than you process planar mode, without the need to change/convert them.
Because if horizontaly the pixel may not be "related" according the format, they are always verticaly.
But i agree that for planar modes probably improvements are possible.
Don't forget that a lot of planar mode has been added since the original, so the choice of vertical only at the time may not have been a so bad idea.
The actual situation is a progressive evolution from the original situation, not a rewrite from scratch.

Reel.Deel
29th October 2016, 14:30
...You could also just link in z.lib into your project instead of writing it all again.
.

+1, it'll be nice if we had the zimg (http://forum.doom9.org/showthread.php?t=171334) for AviSynth+. I know you've mentioned this before but you have a weird tendency of deleting your post :devil:.

bilditup1
30th October 2016, 11:42
Now that there's a new build out - does anybody know if DGIndex is working again?

Groucho2004
30th October 2016, 12:13
Now that there's a new build out - does anybody know if DGIndex is working again?
I assume that you mean dgdecode, dgindex doesn't use Avisynth. Why don't you just try it?

Reel.Deel
30th October 2016, 14:42
Now that there's a new build out - does anybody know if DGIndex is working again?

Ultim went on break right after you reported (http://forum.doom9.org/showthread.php?p=1779373#post1779373) this issue so I don't think he fix it. However, pinterf did submit a similar issue (https://github.com/AviSynth/AviSynthPlus/issues/103) with a hotfix. Like Groucho2004 suggested, try it out, see if it works and report back :).

Groucho2004
30th October 2016, 18:27
Min. installer (w/o docs/MS runtimes) for the latest build (r2290) (https://www.dropbox.com/s/4oesnhc4nb3nops/AviSynth%2B%20r2290.7z?dl=0)
Latest All-In-One runtimes (if needed) (http://repacks.net/forum/viewtopic.php?f=6&t=125)

sl1pkn07
30th October 2016, 19:11
Groucho2004. the installer install the plugins dlls in root of C:/

https://sl1pkn07.wtf/paste/view/0bc0fb36

Groucho2004
30th October 2016, 19:21
Groucho2004. the installer install the plugins dlls in root of C:/
I can't reproduce that. I tested on XP32 and XP64. All good.

sl1pkn07
30th October 2016, 19:26
win7 here

EDIT: false alarm. uninstall the old (by uninstaller) and reinstall the new, solve the problem

greetings

Reel.Deel
30th October 2016, 20:03
Min. installer (w/o docs/MS runtimes) for the latest build (r2290) (https://www.dropbox.com/s/4oesnhc4nb3nops/AviSynth%2B%20r2290.7z?dl=0)
Latest All-In-One runtimes (if needed) (http://repacks.net/forum/viewtopic.php?f=6&t=125)


Hate to be that guy but latest release is r2294 (https://github.com/pinterf/AviSynthPlus/releases).

Edit: ↓ ↓ ↓ Thanks Groucho.

Groucho2004
30th October 2016, 21:09
Hate to be that guy but latest release is r2294 (https://github.com/pinterf/AviSynthPlus/releases).
Sorry, here is r2294:

Min. installer (w/o docs/MS runtimes) for the latest build (r2294) (https://www.dropbox.com/s/f0a55fixmvagroa/AviSynth%2B%20r2294.7z?dl=0)
Latest All-In-One runtimes (if needed) (http://repacks.net/forum/viewtopic.php?f=6&t=125)

bilditup1
31st October 2016, 00:53
Ultim went on break right after you reported (http://forum.doom9.org/showthread.php?p=1779373#post1779373) this issue so I don't think he fix it. However, pinterf did submit a similar issue (https://github.com/AviSynth/AviSynthPlus/issues/103) with a hotfix. Like Groucho2004 suggested, try it out, see if it works and report back :).
Yeah I noticed he'd gone on hiatus but before that in his followup to me ultim said he would take care of it 'by the weekend' so I was hoping that that happened, and was basically just waiting for someone to make a new build as I don't want to try building myself, heh.
I assume that you mean dgdecode, dgindex doesn't use Avisynth. Why don't you just try it?
Yessiree, DGDecode. And you guys are right, I should have just tried it myself. So: I'm happy to report that DGDecode now works in MT, both your build and neuron2's :)

TDecimate() in MT is still out of commission though - results in "TDecimate: major internal error. Please report this to tritical ASAP!" or "TDecimate: internal error during prebuffering" error messages. I just tried going back to r2172 and seeing what happens, and it's weird: last time I tried to use TDecimate() with MT (http://forum.doom9.org/showthread.php?p=1779374#post1779374), it threw basically the same errors as it does now. But this time using r2172, it ends up throwing the "Only a single prefetcher is allowed per script" error even if TDecimate() is commented out and only TFM() is called (using DGDecodeIM as source filter, of course). Not sure what accounts for this, I thought maybe I switched TIVTC builds or something (between tritical's and groucho2004's) but that wasn't it. So perhaps this is still progress, of sorts, as at least TFM() runs without the 'single prefetcher' error in r2294. In any case, yeah, TDecimate() is still out for MT as of now.

pinterf
1st November 2016, 21:19
I was so annoyed on missing array type variables and function parameters in Avisynth, that I had to experiment a bit.

I don't know if it breaks anything but I had to try that.
Treat it as my fun part of the work. If you say that nobody will use it, I leave it out. If it breaks plugin compatibility, I will also undo it. Still have to work on it.

This is already working:

defining Array expressions
(not nice, [] would be o.k. but I'm glad I could get it work this way
Array nesting
Access by index 0..ArraySize-1
Access by Name
use arrays in function parameters (like my colors parameter in BlankClip)


clip=clip.ConvertBits(16)

black_yuv_16 = Array( 0,32768,32768)
white_yuv_16 = Array(65535,32768,32768)
aSelectColors = Array(Array("black", black_yuv_16), Array("white", white_yuv_16))

test_array2 = Array(99, 1.0, "this is a string") # mixed types
test_array = test_array2
n = ArraySize(test_array) # 3
clip = clip.SubTitle("Array size = " + String(n) +\
" [0]=" + String(ArrayGet(test_array,0)) + \
" [1]=" + String(ArrayGet(test_array,1)) + \
" [2]=" + ArrayGet(test_array,2))

black=blankClip(clip, length=1, colors = ArrayGet(aSelectColors,"black")) # value lookup by name
white=blankClip(clip, length=1, colors = aSelectColors.ArrayGet("white")) # value lookup by name, another syntax
#white=blankClip(clip, length=1, colors = Array(65535,32768,32768)) # or direct array



Opinions?

StainlessS
1st November 2016, 22:01
I sure hope that you can get the arrays working without problems, Avisynth+/Standard are in dire need of built-in arrays. Good Luck.

Edit: string memory garbage collection really needs some attention too. Would still though have to comply with current SaveString (or whatever its called), only new plugs could use some alternative function to alloc mem and give [EDIT: total] ownership to avisynth.
Mobile.

pinterf
2nd November 2016, 00:51
Does Avisynth have problems with strings?

StainlessS
2nd November 2016, 01:11
String memory is only released when avisynth closes down. Strings allocated in functions (local) survive until 'the end', I was personally shocked when I discovered how greedy avisynth is on string memory.
Mobile.

Edit: scriptclip can be terribly wastefull.

Gavino
2nd November 2016, 01:17
use arrays in function parameters (like my colors parameter in BlankClip)
What about arrays as parameters to user-written functions?
This would require introduction of a new type-identifier (eg 'array') to rank alongside 'clip', 'int', 'float', etc in the function definition.

Overall, your idea seems like a good one, but careful thought (which I haven't yet had time for) is required to ensure that it doesn't break anything or introduce language inconsistencies of some kind.

StainlessS
2nd November 2016, 01:49
Ooooow, pinterf and the big G, both considering type Array, you guys got me excited http://www.cosgan.de/images/smilie/musik/c060.gif

qyot27
2nd November 2016, 02:18
Does Avisynth have problems with strings?
What kind of problem(s) exactly? tp7 mentioned the probable need to add something better in the core (https://github.com/AviSynth/AviSynthPlus/issues/28) to deal with string formatting and concatenation a couple years ago, but that may not be the same thing you're referring to.

StainlessS
2nd November 2016, 03:52
I was referring to memory usage, ie not freeing of temporary strings, strings used in eg any single instance of a Scriptclip frame
are not released until Avisynth closure, despite them never being used again. We should be able to automatically release local strings that are
created in any script function and are not returned as the function result. I dont know if memory used for other non string local variables
additionally are not released until closure.
EDIT: Plugin writers can rely upon strings returned via SaveString() to be still available later on, so to be able to release strings
allocated via plugins, there must be another function akin to SaveString() that hands over total ownership to Avisynth and such
memory cannot be later accessed by the plugin. The original SaveString(), copies the string into Avisnth allocated memory.
How to return Strings (I'm guessin that you dont need this, perhaps some do):- http://forum.doom9.org/showthread.php?p=1633936&highlight=SaveString#post1633936


Creating an expanded system environment string + file is not I think such a big problem, (does not need to be implemented in core,
although GetSystemEnv() could be a good idea).


AVSValue __cdecl RT_GetSystemEnv(AVSValue args, void* user_data, IScriptEnvironment* env) {
// char * myName="RT_GetSystemEnv: ";
const char *s=args[0].AsString();
char * sysenv = getenv(s);
if(sysenv==NULL)
sysenv="";
return env->SaveString(sysenv);
}


fn=RT_GetSystemEnv("HOMEPATH") + "\MyFile.xyz" although that aint exactly bullet proof.


this is string formatting as implemented in RT_Stats RT_string


AVSValue __cdecl RT_String(AVSValue args, void* user_data, IScriptEnvironment* env) {
char *myName="RT_String: ";
const char *s = args[0].AsString(); // text
int arrsz = args[1].ArraySize();
int esc = args[2].AsInt(1);

if(esc < 0 || esc > 2)
env->ThrowError("%sEsc range 0 -> 2 only",myName);

enum {
CHICKEN=64
};

// what size buffer we need ?
int i,mem=strlen(s) + 1 + CHICKEN;
for(i=0;i<arrsz;++i) {
if(args[1][i].IsString()) {
const char *st=args[1][i].AsString();
mem += strlen(st) + 1 + CHICKEN;
} else {
mem += 8 + CHICKEN; // no particular reason why so big, just chicken factor.
}
}

char *pbuf = new char[(mem+1)*2];
if(pbuf==NULL)
env->ThrowError("%sCannot allocate memory",myName);

char *ptem=pbuf+(mem+1); // temp buffer

const unsigned char * r= (const unsigned char *)s;
char *p=pbuf;
int c,ix=0;
int t=0;

// Parse text and insert variables
while(c=*r) {
if(c=='%') {
++r;
if(*r=='\0') {
*p++ ='%';
} else if(*r=='%') {
*p++=*r++; // replace escaped double % with single
} else {
if(ix>=arrsz) {
delete [] pbuf;
env->ThrowError("%sExpecting data arg (%d)",myName,ix+1);
}
char *tp=ptem;
*tp++='%';
if(*r=='-' || *r=='+' || *r=='0' || *r==' ' || *r=='#') // flags
*tp++=*r++;
if(*r=='*') { // int holds length
t=args[1][ix].IsBool() ?1: \
args[1][ix].IsString() ?2: \
args[1][ix].IsInt() ?3: \
args[1][ix].IsFloat() ?4: \
0;
if(t!=3) {
delete [] pbuf;
env->ThrowError("%sUnsupported data type, Expecting Width as Int (%d)",myName,ix+1);
}
tp+=sprintf(tp,"%d",args[1][ix].IsInt());
++r; // skip '*'
++ix; // next data
} else {
while(*r>='0' && *r<='9') {
*tp++ = *r++;
}
}
if(*r=='.') {
*tp++ = *r++; // precision prefix
if(*r=='*') { // int holds length
t=args[1][ix].IsBool() ?1: \
args[1][ix].IsString() ?2: \
args[1][ix].IsInt() ?3: \
args[1][ix].IsFloat() ?4: \
0;
if(t!=3) {
delete [] pbuf;
env->ThrowError("%sUnsupported data type, Expecting Precision as Int (%d)",myName,ix+1);
}
tp+=sprintf(tp,"%d",args[1][ix].AsInt());
++r; // skip '*'
++ix; // next data
} else {
while(*r>='0' && *r<='9') {
*tp++ = *r++;
}
}
}
t=args[1][ix].IsBool() ?1: \
args[1][ix].IsString() ?2: \
args[1][ix].IsInt() ?3: \
args[1][ix].IsFloat() ?4: \
0;
// type
if( (*r=='c' ) || (*r=='C' ) || // char as int
(*r=='d' || *r=='i') || // int
(*r=='o' || *r=='u' || *r=='x' || *r=='X')) { // unsigned int
if(t!=3) {
delete [] pbuf;
env->ThrowError("%sType='%c', Expecting Int data (%d)",myName,*r,ix+1);
}
*tp++=*r++;
*tp='\0';
p+=sprintf(p,ptem,args[1][ix].AsInt());
++ix; // next data
} else if(*r=='e' || *r=='E' || *r=='f' || *r=='g' || *r=='G') { // double
if(t!=4&&t!=3) {
delete [] pbuf;
env->ThrowError("%sType='%c', Expecting Float (%d)",myName,*r,ix+1);
}
*tp++=*r++;
*tp='\0';
p+=sprintf(p,ptem,args[1][ix].AsFloat());
++ix; // next data
} else if((*r=='s')||(*r=='S')) { // string
if(t!=2&&t!=1) {
delete [] pbuf;
env->ThrowError("%sType='s', Expecting String (%d)",myName,ix+1);
}
*tp++=*r++;
*tp='\0';
if(t==1) { // Bool
p+=sprintf(p,ptem,args[1][ix].AsBool()?"True":"False");
} else { // String
p+=sprintf(p,ptem,args[1][ix].AsString());
}
++ix; // next data
} else {
delete [] pbuf;
env->ThrowError("%sUnknown format type '%c' (%d)",myName,*r,ix+1);
}
}
} else if(c == '\\' && esc == 1) {
++r;
c=*r;
// abfnrtv
switch (c) {
case '\0' : *p++='\\'; break; // copy single backslash at end of string
case '\\' : *p++=*r++; break; // replace double backslash with single backslash
case 'n' : ++r; *p++='\n'; break;
case 'r' : ++r; *p++='\r'; break;
case 't' : ++r; *p++='\t'; break;
case 'v' : ++r; *p++='\v'; break;
case 'f' : ++r; *p++='\f'; break;
case 'b' : ++r; *p++='\b'; break;
case 'a' : ++r; *p++='\a'; break;
default : *p++='\\'; *p++=*r++; break; // anything else we copy backslash and whatever follows
}
} else {
*p++=*r++;
}
}
*p=0; // nul term

if(ix<arrsz) {
delete [] pbuf;
env->ThrowError("%sUnexpected data arg (%d)",myName,ix+1);
}

// dprintf("RT_String: MEM Estimate=%d : Actual=%d\n",mem+1,strlen(pbuf));

if(esc==2) {
p=pbuf;
r=(const unsigned char*)p;
while(c=*r) {
if(c=='\\') {
++r;
c=*r;
// abfnrtv
switch (c) {
case '\0' : *p++='\\'; break; // copy single backslash at end of string
case '\\' : *p++=*r++; break; // replace double backslash with single backslash
case 'n' : ++r; *p++='\n'; break;
case 'r' : ++r; *p++='\r'; break;
case 't' : ++r; *p++='\t'; break;
case 'v' : ++r; *p++='\v'; break;
case 'f' : ++r; *p++='\f'; break;
case 'b' : ++r; *p++='\b'; break;
case 'a' : ++r; *p++='\a'; break;
default : *p++='\\'; *p++=*r++; break; // anything else we copy backslash and whatever follows
}
} else {
*p++=*r++;
}
}
*p=0; // nul term
}
AVSValue ret = env->SaveString(pbuf,p-pbuf);
delete [] pbuf;
return ret;



Much better than the basic Avisynth String() function.

pinterf
2nd November 2016, 12:23
What about arrays as parameters to user-written functions?
This would require introduction of a new type-identifier (eg 'array') to rank alongside 'clip', 'int', 'float', etc in the function definition.

Overall, your idea seems like a good one, but careful thought (which I haven't yet had time for) is required to ensure that it doesn't break anything or introduce language inconsistencies of some kind.
Yes, there were heavy side-effects.

A couple of hours I went on a bad direction.

First I was thinking about using an existing feature: unnamed function parameters already accepted array-like parameter values, by the '+' specifier, e.g. in StackHorizontal, Spline or Select.

{ "StackHorizontal", BUILTIN_FUNC_PREFIX, "cc+", StackHorizontal::Create },
{ "Spline", BUILTIN_FUNC_PREFIX, "[x]ff+[cubic]b", Spline },
{ "Select", BUILTIN_FUNC_PREFIX, "i.+", Select },


But it is only for one level.
And the broader support in AVSValue mechanism (constructor, etc.) failed. When I wanted to return such an array value, e.g. store it in a variable, the content of the array disappeared outside the internal method, because after instancing a function or method the array member AVSValues were freed up.

Another problem was that the existing array parameter type has special handling internally, the array content is "flattened" at some point.

So I had to introduce a "real" script array type, now I call it "A" in order not to mix it with the existing 'a' internal array specifier.
It is a vector<AVSValue>

Because of the working mechanism of '+' parameter specifier, which is putting together a one-level array AVSValue (type=='a') for the functions, I introduced a # (since it resembles the + sign) specifier that results a new script-array (type=='A') instead of that.

In my sample, BlankClip can have a real array, that can be used for a named parameter. The definition is like this (A= the new script array type)
{ "BlankClip", BUILTIN_FUNC_PREFIX, "[]c*[length]i[width]i[height]i[pixel_type]s[fps]f[fps_denominator]i[audio_rate]i[channels]i[sample_type]s[color]i[color_yuv]i[clip]c[colors]A", Create_BlankClip },


and the other array specific functions are defined as
{ "Array", BUILTIN_FUNC_PREFIX, ".#", ArrayCreate }, // # instead of +: creates script array
{ "IsArray", BUILTIN_FUNC_PREFIX, ".", IsArray2 },
{ "ArrayGet", BUILTIN_FUNC_PREFIX, "Ai", ArrayGet },
{ "ArrayGet", BUILTIN_FUNC_PREFIX, "As", ArrayGet },
{ "ArraySize", BUILTIN_FUNC_PREFIX, "A", ArraySize },


It works as-is, parameter type is nicely checked against type 'A'.
Script function can check the script-array type with IsArray2 at the moment.
At least I had a little insight to the parser, too. Anyhow, this part was nicely written, I hope I can easily change to [] for definition and indexing arrays.

A little off-topic, but I relaxed with this problem after my weekend 85K trail running race :) Fun after fun.

BTW did you know that there is built-in for and while?

Gavino
2nd November 2016, 13:20
{ "Array", BUILTIN_FUNC_PREFIX, ".#", ArrayCreate }, // # instead of +: creates script array
Perhaps # should be allowed to also match zero or more values (like the existing * rather than +), so that you can create empty arrays. Or maybe you want another new symbol in addition to #, to be the equivalent of * for arrays.

Actually, on further thought, I'm not sure you really need # anyway.
Can't the built-in Array() function be defined as ".*", and simply copy its arguments into an new type 'A' AVSValue?

BTW did you know that there is built-in for and while?
Yes (in Avisynth+).
In the early stages of development, I gave ultim permission to incorporate my GScript code into Avisynth+.

may24
3rd November 2016, 08:52
Hi all,

I recently swapped avisynth for avisynth+

Yesterday I modified one of my avs Scripts. Just added some new pics - basically three more ImageSource(...)

But now I got: "Could not map shared memory into local space" in VirtualDub.

Any ideas ?

EDIT: I fall back to std. Avisynth 2.6.1 ... no problems anymore ...

pinterf
3rd November 2016, 12:49
Hi all,

I recently swapped avisynth for avisynth+

Yesterday I modified one of my avs Scripts. Just added some new pics - basically three more ImageSource(...)

But now I got: "Could not map shared memory into local space" in VirtualDub.

Any ideas ?

EDIT: I fall back to std. Avisynth 2.6.1 ... no problems anymore ...

Anyway, needed the
- avs script
- whether the script works alone w/o vdub (to rule out avs+ install problems, such as missing VS2015 update 3 redistributables)
- type and size of the images
- virtualdub version
- avisynth+ version

LigH
3rd November 2016, 13:07
- whether the script works alone w/o vdub (to rule out avs+ install problems, such as missing VS2015 update 3 redistributables)

:confused: AviSynth scripts (no matter which fork of AviSynth) will always need a host application. The minimum is probably AVSMeter (which also may print more useful error messages). In case of AviSynth+ as engine, may24 can (and should!) test both the 32 bit and 64 bit version of AVSMeter as host.

If I understood may24 correctly, the script worked well with a few images less loaded, but he crossed some threshold by adding some more sources?

pinterf
3rd November 2016, 13:16
:confused: AviSynth scripts (no matter which fork of AviSynth) will always need a host application. The minimum is probably AVSMeter (which also may print more useful error messages). In case of AviSynth+ as engine, may24 can (and should!) test both the 32 bit and 64 bit version of AVSMeter as host.

If I understood may24 correctly, the script worked well with a few images less loaded, but he crossed some threshold by adding some more sources?
AVSMeter or x264 was in my mind (other than virtualdub).
I found that error message (http://read.pudn.com/downloads9/sourcecode/multimedia/34422/VirtualDub/source/FilterSystem.cpp__.htm) and all I could see that a MapViewOfFile after a successful CreateFileMapping has failed with a specific buffer size.

jpsdr
3rd November 2016, 14:24
@pinterf :
Where is _MM_PACKUS_EPI32 defined ?
When i tried to build, the compiler complained that it can't find it.
I've not been able to find it either, at leat in the resample source files.

pinterf
3rd November 2016, 16:09
@pinterf :
Where is _MM_PACKUS_EPI32 defined ?
When i tried to build, the compiler complained that it can't find it.
I've not been able to find it either, at leat in the resample source files.
internal.h, it was used at several places