Avisynth+

[GMJCZP]

I have a problem with ImageWriter, when using the following script, for example:

ConverttoRGB24().ImageWriter("", 5,5, "bmp").ConvertToYV12()

I get an error with the DevIL library. I am using AVS + 2728. I never had problems with the old AVS.

[LigH]

Which is "an error"? Maybe it tells about the reason.

[pinterf]

Originally Posted by StainlessS

Yep.
With my test clip (854x480@30, 3775 Frames), memory usage max's out at about 1.73GB

I checked the 32 bit vdub2 process with SysInternals vmmap. It seems that we are near 2GB with this script and parameters. As the internal caches are getting filled (until the set limit) Windows suddenly cannot allocate memory. Even when I set the max cache size to the minimum of 64MB, it could not request new memory buffer from the system after a while. Either avisynth or virtualdub - whichever is unlucky - will feel deep unhappyness and crash. So I think the problem is not avisynth related.

[StainlessS]

So I think the problem is not avisynth related.

Thank you for looking in to this P, much appreciated.

[GMJCZP]

Originally Posted by LigH

Which is "an error"? Maybe it tells about the reason.

I am using a borrowed PC because I have not had internet for several months, so I do not remember exactly how the message says, the issue is that I can not save the frame and it is something related to DevIL. I do not know if anyone can try to recreate the error using the script.

[pinterf]

Perhaps it wants to write to c: root which is not allowed?

[StainlessS]

Originally Posted by pinterf

Perhaps it wants to write to c: root which is not allowed?

Yeh[EDIT: good catch] , possibly. Maybe try,

Colorbars
ConverttoRGB24().ImageWriter(".\", 5,5, "bmp").ConvertToYV12()

".\" is current directory, and at least it works as expected here.

EDIT: With just "" here on W7x64Avs32, I got nothing, no output and no error message, perhaps reason that you could not
remember the error message

EDIT: "." alone did NOT work [EDIT: Same result as ""].

[StainlessS]

Originally Posted by StainlessS

EDIT: "." alone did NOT work [EDIT: Same result as ""].

Actually, I was in sub directory of D:\ drive ("D:\NewFolder"), and "." actually wrote to the root of D:\.

EDIT:
So far as I remember,
"D:" should be current directory on D drive,
"D:\" should be root directory on D drive,
"." should be current directory, current drive (also think ".\" should be same).

EDIT: Damn, forum keeps swallowing '\' characters.

EDIT: Also, ".." should be parent directory to current directory, think "..\" should be same.

EDIT:
Where below script in "D:\NewFolder\NewFolder"
This writes to root directory of D: (I think should be parent directory to current directory ie D:\NewFolder\)

Colorbars ConverttoRGB24().ImageWriter("..", 5,5, "bmp").ConvertToYV12()

This writes to D:\NewFolder (I think should be parent directory to current directory ie D:\NewFolder\)

Colorbars ConverttoRGB24().ImageWriter("..\", 5,5, "bmp").ConvertToYV12()

Something seems to be messed up somewhere, dont know if ImageWriter or Windows 7.

EDIT: Also "" should be interpreted same as "." and ".\", ie current directory.
Excepting for case where eg "D:\" specifies root directory of drive D, a trailing '\' just explicitly specifies that it is a directory
rather than a file, but "." and ".." already imply a directory where trailing "\" should be superfluous unless a node is appended after
the slash (ie another directory or file name node).

[manolito]

In one of my video conversion scripts I need to remove the station logo, for this I have been using LogoAway 4.01 by Krzysztof Wojdon for a long time. After switching to AVS+ I got random crashes using this filter, and luckily I believe I solved the issue, but it took me a long time and a lot of detective work...

My hardware and software setup first:

Core i5 3rd generation with 8GB RAM
Win7 64-bit
Avisynth+ r2728-MT 32-bit
SetMTMode.avsi from this link: http://publishwith.me/ep/pad/view/ro.rDkwcdWn4k9/latest
StaxRip v. 1.1.9.0 (last stable 32-bit version)
X264 r2935 32-bit

I use "Prefetch(4)" with this CPU (2 physical cores plus Hyperthreading). This normally runs stable. But after getting these random crashes I first made tests to find out which filter was to blame, and it was LogoAway.

It does not happen with all sources, and sometimes the crash occurs shortly after the conversion start, sometimes it happens in the middle and sometimes at the end. I get this Windows popup that X264.exe has stopped working and must be shut down.

I am quite sure that X2364 is not to blame, I tried 3 different builds of the same release without any difference. Reducing the AVS+ threads from 4 to 2 still gave me the crashes, but this time without the Windows popup. X264 just stopped working, CPU load went down to 0, no error message.

Removing the "SetMTMode.avsi" also made no difference (except slowing down encoding speed by 0.3 fps). What did fix it of course was to disable MT altogether, and forcing MT_SERIALIZED for LogoAway also worked, but speed was not any faster than without MT.

The final solution came just by trial and error, and I really do not understand why it works. If I specify MT_MULTI_INSTANCE explicitly for LogoAway either in the conversion script or in the "SetMTMode.avsi" then the crashes miraculously disappear. This is reproduceable each and every time.

How is this possible? The docs say that MT_MULTI_INSTANCE is the default MT mode which always gets used as long as no different MT mode is explicitly specified. So it should not make a difference if I specify MT_MULTI_INSTANCE in the script or not, right? But obviously it does...

Or does AVS+ treat VDub plugins differently from other filters?


Any thoughts?

Cheers
manolito

[pinterf]

Originally Posted by manolito

How is this possible? The docs say that MT_MULTI_INSTANCE is the default MT mode which always gets used as long as no different MT mode is explicitly specified. So it should not make a difference if I specify MT_MULTI_INSTANCE in the script or not, right? But obviously it does...

Or does AVS+ treat VDub plugins differently from other filters?

Just looking at the code I see that no special MT mode is set for vdub filters, so I expect them to behave as MT_MULTI_INSTANCE. I'll check it.

[pinterf]

I could not reproduce, for me it works like MT_MULTI_INSTANCE. I'd need the source clip or a short fragment with that it still crashes. And a (minimized) script itself.
Maybe it's not the filter itself which fails, just when it appears in a specific sequence.
Once I had an issue when I was debugging TIVTC for weeks because it crashed x264, then it turned out that an mvtools2 function did not clear the processors's mmx state and conflicted with x264 and it was just a coincidence that tivtc seemed to be a culprit.

[manolito]

Thanks pinterf for looking into it...

The source file I used for my tests was a huge 2 hour HD captured TV file. Please give me some time to find a smaller source which reliably crashes...

Otherwise I did not use any elaborate filters in my script. I should probably mention that for FineSharp I use old and proven versions of RemoveGrain (original by Kassandro from 2005) and MaskTools v2.0a48. Upgrading to later versions is not an option because I need to use the same scripts under WinXP with a non-SSE2 CPU.

I will report back soon...

Cheers
manolito

[wonkey_monkey]

A question about expr(), if someone can answer - when the AVX/SSE optimisations are enabled, it processes four (or more) pixels at once - is that correct? Like if you were adding the values of two clips, it would load several pixels into two SSE/AVX registers and add them together in one operation?

[GMJCZP]

Thanks for the replies pinterf and TinMan, and sorry for responding late, but I still do not have internet.

On the AVS wiki page, when you talk about ImageWriter, give this example:

# Write frame 5 to "C: \ 000005.PNG"
ImageWriter ("", 5, 5, "png")

When putting the quotes "" I always worked with the old AVS, capturing the frame and saving it in the same place in the script, no matter where in the Hard Disk it is. When placing

".\"

It works for AVS +, so my question is why the difference between AVS and AVS+?

Note: the error message varies, which does not always appear, says:

ImageWriter: error 'Could not open file' in DevIL library writing file "000005.bmp"
DevIL version 0.

Until I got an error message with strange characters, the problem is the incompatibility of "" in ImageWriter with AVS+ , but not with AVS, why?

[pinterf]

Originally Posted by davidhorman

A question about expr(), if someone can answer - when the AVX/SSE optimisations are enabled, it processes four (or more) pixels at once - is that correct? Like if you were adding the values of two clips, it would load several pixels into two SSE/AVX registers and add them together in one operation?

Expr handles 2x4 pixels (XMM registers) or 2x8 pixels (AVX2) at a time. Code is generated for two sets of XMM/YMM registers unless optSingleMode=true - I introduced this parameter for curiosity, if it is any faster to use only a single register set - one lane. (Looking at the generated code of a slightly more complex expression it turned out that the registers were heavily swapped to and from temporal memory variables because the code required too many physical XMM/YMM registers but we had only a limited number of them).

Note that I added AVX2 - and not AVX - code generation to Expr, though the calculations are using 32bit floats inside. Many helper asm instructions I'm using are available only from AVX2. On the other hand, old processors with AVX-only support suffer from not having real 256 bit internally, e.g. memory access would still happen in 2x128bit mode which is sometimes slower than using 2x128bit memory load manually which can use different ports in parallel. At least for my old AVX-only i7 that was the case.

See optSSE2, optAVX2 and optSingleMode in http://avisynth.nl/index.php/Expr

EDIT:
In Avisynth+ frame alignment is 64 bytes since I added AVX2 codegen to Expr to allow the two-lane (2x256 bit YMM register) operation for 32 bit float-type inputs - and as a side effect allows painless AVX512 usage for present and future filters. Why 64 bytes? For float-type pixel input in order to use 2xYMM registers we have to read and write 512bits = 64 bytes at a time, which requires 64 byte line padding.

Later when one-lane mode (optSingleMode=true option) was put in the codegen core, it became possible to have all calculations in two-lane (2x256bit = 2x8 float pixels) mode, except the last chunk of a line which could be done either for 8 pixels (one 256 bit YMM register, like in optSingleMode=true) or 2x8 pixels (2x256bit YMM registers), depending on the source clip width. Using such an adaptive treating of the last pixels it would no longer require 64 byte alignment and line padding, because the old 32 bytes padding would have been enough, anyway I left the 64 bytes frame aligment in general thinking of the plenty future filters using AVX512.

[pinterf]

Originally Posted by GMJCZP

On the AVS wiki page, when you talk about ImageWriter, give this example:

# Write frame 5 to "C: \ 000005.PNG"
ImageWriter ("", 5, 5, "png")

... so my question is why the difference between AVS and AVS+?

Until I got an error message with strange characters, the problem is the incompatibility of "" in ImageWriter with AVS+ , but not with AVS, why?

Yep, avs+ worked like wiki says, but that part of wiki is wrong.

"c:\"

is the default value of "file" parameter if it is not provided. When file parameter is given and it's empty string it should get the current directory.

Probably when it was rewritten for avs+ from the classic avs source, they missed that special case.

EDIT: fixed the example on wiki

And yes, it should work (and now the git dev version works) like classic avisynth does.

[manolito]

Originally Posted by manolito

I will report back soon...

Finding a shorter source which still revealed the issue was unsuccessful. On two long source files I did get the crashes during the first 10 minutes into the clip, but cutting out the first 15 minutes did not bring any results. With the cut-out sources the conversions went without problems. So the source files themselves are not to blame.

Specifying MT_MULTI_INSTANCE explicitly for the LogoAway filter again seemed to make the crashes disappear first, but on a very long source file with a length of more than 2 and a half hours the crashes did return towards the end of the clip.

My conclusion is that the LogoAway filter needs MT_SERIALZED to work reliably. Sorry for the false alarm...


Cheers
manolito

[pinterf]

O.K. then. By seeing the source code it could be established whether it behaves well in MT environment or not but I suppose we don't have it.

[StainlessS]

but I suppose we don't have it

I found this, dont know if he has the original source (from the VirtualDub2 thread).

by b2kguga,

I´m giving a try building a new version of LogoAway. So far, the rebuild is doing well. I´m doing it in assembly and debugging it also with my own assembler. One of the things i´m having difficulties to understand is that inside my ConfigDialog it have a call to the toggle function of Vdub.

https://forum.doom9.org/showthread.p...01#post1777801

EDIT: b2kguga last activity April 2017, so maybe a bum steer.

[GMJCZP]

Originally Posted by pinterf

Yep, avs+ worked like wiki says, but that part of wiki is wrong.

"c:\"

is the default value of "file" parameter if it is not provided. When file parameter is given and it's empty string it should get the current directory.

Probably when it was rewritten for avs+ from the classic avs source, they missed that special case.

EDIT: fixed the example on wiki

And yes, it should work (and now the git dev version works) like classic avisynth does.

Thanks pinterf.
What you say means that the next version of avs+ will admit "" as it does the classic avs?