Avisynth+

[Reel.Deel]

qyot, Reel, you two are kings! I checked the sources of nnedi3 after Reel found it was a possible cause, and realized what the problem is. The merge commit that qyot bisected reinforced my analysis. In short, nnedi3 needs to be recompiled at a minimum, and better, a small fix be made to it to avoid the problem in the future.

Here is what happened:
1) In commit 2ee19354 of classic Avs (JEEB's repo), GetVarDef() was added to the IScriptEnv interface, and this was merged to Avs+ in the commit found by qyot.
2) The addition of GetVarDef is in general OK, because it maintains ABI for IScriptEnv, and while it breaks ABI for IScriptEnv2 (because it is a derived interface), it was merged anyway because IScriptEnv2 is not (yet) supposed to be used by external plugins. There is a big fat warning block above the interface definition that warns plugin developers, that the interface is not stable and should only be used for testing.
3) Now comes the fun part. nnedi3 ignored this warning, and in its newest version is relying on the unstable interface, which just got changed, and hence the plugin got broken.
4) War, famine, and epidemics, not to mention the wasted hours for the community in debugging this. Warnings are there for a reason. Once that warning gets remove, only then can you rely publicly on the interface.

This also explains why I couldn't reproduce this: I have an older version of nnedi3 that does not use IScriptEnv2.

It seems nnedi3 is using the unstable API to register its MT mode. The proper way to implement this functionality is instead of using IScriptEnv2, respond to the CACHE_GET_MTMODE query in the plugin's SetCacheHints() with the proper constant. Here (https://github.com/AviSynth/AviSynthPlus/blob/MT/avs_core/filters/field.cpp#L618) is an example of doing it right. This is future proof and is compatible with both classic Avs and Avs+.

There's actually a few recent plugins using the same method as nnedi3. If I'm not mistaken a few from Chikuzen, and MysteryX's AviSynthShader.

It was asked about not too long ago: http://forum.doom9.org/showthread.php?p=1768175#post1768175

See this also: http://forum.doom9.org/showthread.php?p=1768838#post1768838

[ultim]

There's actually a few recent plugins using the same method as nnedi3. If I'm not mistaken a few from Chikuzen, and MysteryX's AviSynthShader.

It was asked about not too long ago: http://forum.doom9.org/showthread.php?p=1768175#post1768175

See this also: http://forum.doom9.org/showthread.php?p=1768838#post1768838

Ah, okay, so nnedi3 actually had a proper reason for using IScriptEnv2. Either because of bug #37 (https://github.com/AviSynth/AviSynthPlus/issues/37), or because I actually disabled the mechanism for CACHE_GET_MTMODE, also in a sad attempt to work around #37. Darn.

I see how this leaves plugins like nnedi3 in a crossfire... I'll try to come up with something for this problem.

[ultim]

Is there a way to use IScriptEnv2 so that it doesn't crash if the interface changes?

Unfortunately, no there is not. Or else the warning wouldn't be there. Or at least not if the IScriptEnv is extended using inheritance, but any other method would be really ugly, also for plugin developers.

Basically what I need is a proper fix for #37 so that I can re-enable CACHE_GET_MTMODE.

[Reel.Deel]

Ah, okay, so nnedi3 actually had a proper reason for using IScriptEnv2. Either because of bug #37 (https://github.com/AviSynth/AviSynthPlus/issues/37), or because I actually disabled the mechanism for CACHE_GET_MTMODE, also in a sad attempt to work around #37. Darn.

I see how this leaves plugins like nnedi3 in a crossfire... I'll try to come up with something for this problem.

Correct, but nnedi3() works fine, it's nnedi3_rpow2() that's affected by bug #37. I think eedi3_rpow2() will have the same problem but I haven't checked.

[easyfab]

Hi,

I'm a simple user ( not a dev ) and I update to r2003 and it crashed for me with 32bit version, AvsPmod doesn't open for example. After research it comes with rawsource26.dll ( https://github.com/chikuzen/RawSource_2.6x/releases ). Remove this from plugins, it works again ? No problem with 64bit version.
Is this a problem with rawsource26 or avisynth+ ?

[Reel.Deel]

Hi,

I'm a simple user ( not a dev ) and I update to r2003 and it crashed for me with 32bit version, AvsPmod doesn't open for example. After research it comes with rawsource26.dll ( https://github.com/chikuzen/RawSource_2.6x/releases ). Remove this from plugins, it works again ? No problem with 64bit version.
Is this a problem with rawsource26 or avisynth+ ?

RawSource26 self-registers the MT mode the same way that latest nnedi3 does. See post #1955 (http://forum.doom9.org/showthread.php?p=1772809#post1772809).

[easyfab]

Ok Thanks so it's a known problem.

[Reel.Deel]

Ok Thanks so it's a known problem.

Just recently :)

[qyot27]

It wasn't nnedi3 that caused it for me, I have an older version of it and hadn't updated. It was a mismatch between the plugins built with an early 16-bit test build from pinterf's branch, and the core of r2003/r2005, since I was using Windows' directory-first dll override tactic to make ffmpeg see the new build instead of the one in system32. Once I put the new plugins in the autoload directory, the problem went away.

So something changed in the plugins too. Probably DirectShowSource, since Reel.Deel mentioned that earlier.


And this wouldn't be the first time *VarDef has caused an issue. I couldn't figure out how to integrate it into avsplus properly back during RC1 when it was first brought into 2.6, so it may be piggybacking on that too.

[hector40]

Hi,

I'm a simple user ( not a dev ) and I update to r2003 and it crashed for me with 32bit version, AvsPmod doesn't open for example. After research it comes with rawsource26.dll ( https://github.com/chikuzen/RawSource_2.6x/releases ). Remove this from plugins, it works again ? No problem with 64bit version.
Is this a problem with rawsource26 or avisynth+ ?

I have the same problem with r2003 32bit, the plugins that crashing are, RawSource26-20160528, TCannyMod-1.2.0, yadifmod2-0.0.4.
The r2003 64bit, don't have the same problem.

[ultim]

It wasn't nnedi3 that caused it for me, I have an older version of it and hadn't updated. It was a mismatch between the plugins built with an early 16-bit test build from pinterf's branch, and the core of r2003/r2005, since I was using Windows' directory-first dll override tactic to make ffmpeg see the new build instead of the one in system32. Once I put the new plugins in the autoload directory, the problem went away.

So something changed in the plugins too. Probably DirectShowSource, since Reel.Deel mentioned that earlier.


And this wouldn't be the first time *VarDef has caused an issue. I couldn't figure out how to integrate it into avsplus properly back during RC1 when it was first brought into 2.6, so it may be piggybacking on that too.

DirectShowSource uses iscriptenv2 too, so it has the same problem as nnedi3. except dss is automatically rebuilt with every build of avs+, so as soon you updated dss, the problem went away. it will go away for nnedi3 too if you recompile it for the newest headers. With one word: same problem.

[pinterf]

LOL... It took months from my first post but... Hey... At last...
Tormento, just one more thing (C) Columbo, do your Windows use comma or dot for decimal separator?

[tormento]

Tormento, just one more thing (C) Columbo, do your Windows use comma or dot for decimal separator?

Comma, italian regional setting.

[pinterf]

O.K. then. Same for me.

After approximately 150 iterations (delete-run-view) I managed to shrink down that nice, easy-to-read SmDegrain to a simple return Dither_Luma_Rebuild line. And the visual experience was staggering. Big white blocks were moving to and fro on my screen. No wonder that the MSuper/MAnalyze did bad conclusion with this input.

Why?

Inside Dither_Luma_Rebuild the lut expression is assembled dinamically.

(Values come from the real test case)
S0=1.0
c=0.0625

k = (s0 - 1) * c
t = "x 4096 - 56064 / 0 1 clip"
e = String(k)+" "+String(1+c)+" "+String((1+c)*c)+" "+t+" "+String(c)
\ +" + / - * "+t+" 1 "+String(k)+" - * + 65536 *"
... and e is used here:
Dither_lut16 (yexpr=e,expr="x 32768 - 32768 * 28672 / 32768 +",y=3, u=uv, v=uv)

With these values variable "e" should look like something like this (I left "t" as is):
"0 1.0625 0.06640625 "+t+" 0.0625 "+" + / - * "+t+" 1 0 - * + 65536 *"


But we are using commas for decimal separator.
String(float_type) results in commas in this case, but the lut functions do not like it. Surely they rely strictly on dots. This becomes a mess for the function:
"0 1,0625 0,06640625 "+t+" 0,0625 "+" + / - * "+t+" 1 0 - * + 65536 *"


The fix is probably to override windows settings and use dot only for formatting floats.

The annoying thing that the debug build was O.K. as it did not take into account windows settings and used dots for formatting floats.

[ultim]

The fix is probably to override windows settings and use dot only for formatting floats.
+1

The annoying thing that the debug build was O.K. as it did not take into account windows settings and used dots for formatting floats.
Sounds a lot like Connect #2185894 (https://connect.microsoft.com/VisualStudio/feedback/details/2185894), with the exception that MS claims it has been already fixed in Update 2.

[ultim]

sry guys, i know i promised, but no builds today. time flew and now i need to sleep. instead there was plenty of diagnosing and exchanging with others. i'm also tempted to only publish next build either when there is a proper fix for #37, or if not that, at least a recommendation for plugin authors about how to handle it. these are next on my agenda.

[ajp_anton]

I'm throwing another vote on it being called "ConvertToStacked" and "ConvertFromStacked", without the "Hbd" in the name.

The Hbd there is unnecessary, because what else would you convert it to/from? There is no advantage, but it has the disadvantage of it being a relatively unknown abbreviation (no matter how "official" and "real" it might be). Whoever said it looks just like a random pile of letters, I agree.
The "Stacked" is already well established in the community, at least with everyone who actually uses it. It's also not an abbreviation, and it's an easy word to pronounce and thus remember.

Another minor problem with HBR is that we don't know how many bits it actually is. Not really a problem now when only 16 bits is introduced (right?), but more formats might be added in the future, in which case we will need "ConvertTo8bit", "ConvertTo12bit" and so on. "HBR" has no place there, it's better to call them by how many bits they actually are.
Stacked is and will ever be only 16 bits, because it's a dead end and will never evolve past that.

[Chikuzen]

Hi.

I updated some my plugins.

RawSource26 (https://github.com/chikuzen/RawSource_2.6x/releases)
PlanarTools (https://github.com/chikuzen/PlanarTools/releases)
yadifmod2 (https://github.com/chikuzen/yadifmod2/releases)
TMM2 (https://github.com/chikuzen/TMM2/releases)
CombMask (https://github.com/chikuzen/CombMask/releases)
TCannyMod (https://github.com/chikuzen/TCannyMod/releases)
VapourSource (https://github.com/chikuzen/VapourSource/releases)

Recompile is easy.
But the announcement is annoy for me...

[MysteryX]

HBD is basically a technical prerequisite of HDR. In addition to HBD, HDR allows to process color and brightness values outside of your limited display range. This usually means 1) using HBD to represent values outside of the valid displayable range, and 2) at the end some kind of tone mapping algorithm to bring it back down to valid pixel values. With HBD Avisynth+ can now do the first of part of HDR, for the second some kind of plugin to perform the tone mapping would be required.
Even without the tone mapping plugin, HDR is still useful to avoid clipping values in the middle of calculations and conversions. It can still be added now. It would be wise to plan this at the same time as HBD. Tone mapping can be added later.

[tormento]

The annoying thing that the debug build was O.K. as it did not take into account windows settings and used dots for formatting floats.

If debug is usable, can you upload and make me test it?

[ultim]

I'm throwing another vote on it being called "ConvertToStacked" and "ConvertFromStacked", without the "Hbd" in the name.

The Hbd there is unnecessary, because what else would you convert it to/from? There is no advantage, but it has the disadvantage of it being a relatively unknown abbreviation (no matter how "official" and "real" it might be). Whoever said it looks just like a random pile of letters, I agree.
The "Stacked" is already well established in the community, at least with everyone who actually uses it. It's also not an abbreviation, and it's an easy word to pronounce and thus remember.

Another minor problem with HBR is that we don't know how many bits it actually is. Not really a problem now when only 16 bits is introduced (right?), but more formats might be added in the future, in which case we will need "ConvertTo8bit", "ConvertTo12bit" and so on. "HBR" has no place there, it's better to call them by how many bits they actually are.
Stacked is and will ever be only 16 bits, because it's a dead end and will never evolve past that.

Since ppl on IRC were also whining about the names, they'll be renamed to ConvertToStacked and ConvertFromStacked. Another suggestion I've received and I like it, is to move these functions into a plugin. This plugin will be distributed along with the core for some time.

[ultim]

Even without the tone mapping plugin, HDR is still useful to avoid clipping values in the middle of calculations and conversions. It can still be added now. It would be wise to plan this at the same time as HBD. Tone mapping can be added later.

No need for special considerations. float type support automatically handles this case. All we need to ensure is to not clamp float to 0..1 at every operation.

[ultim]

EDIT: There is a newer build here (http://forum.doom9.org/showthread.php?p=1773721#post1773721).

Okay, new build r2022 (avs-plus.net/builds).
Please also update the included plugins (if you have an older version installed) to avoid problems. Also, use the updated plugins (http://forum.doom9.org/showpost.php?p=1772902&postcount=1975) from chikuzen for the same reason.

The changes compared to r2003 include:
- Fix for corrupted output of HBD resizers.
- ConvertHbd*() functions renamed to ConvertToStacked() and ConvertFromStacked().
- DirectShowSource() compatibilty fix for VapourSynth.
- Restored XP support.
- (pinterf) New HBD functions added: ConvertToYUV420/422/444(). These convert between different plane formats while keeping the current bit depth. Bit depth conversions will be added in an upcoming build.
- (pinterf) Floating point formatting bug fix, hopefully resolving issues like this (http://forum.doom9.org/showpost.php?p=1771397&postcount=1789).
- (chikuzen) Added ASM-optimized Turn functions for HBD.

Furthermore, the download link now includes new AvsPmod versions, including one also for x64.

As always, please test and give us feedback.

[ultim]

Achievement Unlocked: Page 100.

[LigH]

Well, that escalated quickly...

http://cosgan.de/images/smilie/musik/e050.gif

[pinterf]

2022. So we are in the future from now on :)

[Groucho2004]

r2022 running smoothly on XP32 so far.

[MysteryX]

- (pinterf) New HBD functions added: ConvertToYUV420/422/444(). These convert between different plane formats while keeping the current bit depth. Bit depth conversions will be added in an upcoming build.
This will create confusion with ConvertToYV12, ConvertToYV16 and ConvertToYV24.

Why not keep the same standard functions and add a parameter? They can maintain compatibility with existing scripts while providing the extra feature.

[qyot27]

This will create confusion with ConvertToYV12, ConvertToYV16 and ConvertToYV24.

Why not keep the same standard functions and add a parameter? They can maintain compatibility with existing scripts while providing the extra feature.
The old names didn't go anywhere.

[MysteryX]

Here's an idea for a new feature. Someone wrote AvsFilterNet (https://avsfilternet.codeplex.com/) to allow writing plugins in .NET. One issue is that the .NET plugin must be loaded by AvsFilterNet which is not practical for the user. A .NET library can't be loaded directly by AviSynth.

Could AviSynth+ natively provide an interface for .NET plugins?

This could come useful for utility plugins dealing with multi-threading, multi-processes or performing other complex tasks without requiring raw assembly binary computations.

MP_Pipeline, for example, would have much simpler code by being written in C#. Creating a LoadImage plugin also could be done with just a few lines of code.

Then someone can write a plugin that sends the workload across 10 cloud servers and compares all the data against a database with all communications highly encrypted :P If that'd be useful for someone.

[pinterf]

Ideally we shouldn't have distinct names for each format conversion, just one function with parameters. But it started when only Y8, YV12 and YUY existed besides rgb.

[Chikuzen]

There are already ConvertAudioTo8bit/16bit/Float, and no one complained to those name.
I think ConvertVideoTo8bit/16bit/Float are simple enough.

[tormento]

- (pinterf) Floating point formatting bug fix, hopefully resolving issues like this (http://forum.doom9.org/showpost.php?p=1771397&postcount=1789).
Sorry, not :(

[pinterf]

Sorry, not :(
Are you sure you use the fresh 2022 version? What does avsmeter say (avsmeter -avsinfo) ?
Downloaded ultim's version, and also mine, no artifact. Changed back to a previous r1858, and my intermediate r1908-pfmod, and the artifact came back.

[TurboPascal7]

Here's an idea for a new feature. Someone wrote AvsFilterNet (https://avsfilternet.codeplex.com/) to allow writing plugins in .NET. One issue is that the .NET plugin must be loaded by AvsFilterNet which is not practical for the user. A .NET library can't be loaded directly by AviSynth.

Could AviSynth+ natively provide an interface for .NET plugins?

First of all, .NET really wouldn't simplify stuff too much. Plugins rarely contain anything but the most trivial code and C++ is perfectly fine for that these days.

Second, I see no reason why something like AvsFilterNet couldn't just autoload other plugins on its own. What's the point of moving something like this into the core?

[pinterf]

There are already ConvertAudioTo8bit/16bit/Float, and no one complained to those name.
I think ConvertVideoTo8bit/16bit/Float are simple enough.
Yes, those are tasks for today. 8 bit comes without dithering options first. Float will have a scale parameter defaulting to 0-1.0.

[pinterf]

Another suggestion I've received and I like it, is to move these functions into a plugin. This plugin will be distributed along with the core for some time.

I was thinking the same. And there can be other converting functions (real.finder mentioned lnterleaved16 format), that can ease the transition until filters handle the new colorspaces natively.

[jpsdr]

Euh... Where do you get the new avisynth.h header ? I don't see any link, and it's not put in the package of the new release. Do i have to search it on the github repo ? It would be easier to put it with the release.

Edit : Found it (i think...) on searching on github.

[tormento]

Are you sure you use the fresh 2022 version? What does avsmeter say (avsmeter -avsinfo) ?
Downloaded ultim's version, and also mine, no artifact. Changed back to a previous r1858, and my intermediate r1908-pfmod, and the artifact came back.

Sorry. I had so many mkv outputs that I watched the wrong one.

No artifacts here.

Without going too much OT, can you please paste here the latest modified version of SMDegrain you are using and the script too?

Thanks.

[pinterf]

Sorry. I had so many mkv outputs that I watched the wrong one.

No artifacts here.

Without going too much OT, can you please paste here the latest modified version of SMDegrain you are using and the script too?

Thanks.
Nice. Avisynth Plus Customer Service is happy.

I tested with this script
SetFilterMTMode("DEFAULT_MT_MODE", 2)
SetFilterMTMode("lsmashvideosource", 3)
lsmashvideosource("13HoursCUT.mp4", format="YUV420P8")
Crop(0, 140, 0, -140)
SMDegrain (tr=4,PreFilter=4,thSAD=400,contrasharp=false,refinemotion=false,plane=4,chroma=true,lsb=true,mode=6)
Prefetch(8)

and for the comparison I used the script from here (http://forum.doom9.org/showthread.php?p=1771735#post1771735)

[tormento]

Nice. Avisynth Plus Customer Service is happy.
:thanks:

and for the comparison I used the script from here (http://forum.doom9.org/showthread.php?p=1771735#post1771735)

What about this (http://pastebin.com/JtWFN124)?

[MysteryX]

Euh... Where do you get the new avisynth.h header ? I don't see any link, and it's not put in the package of the new release. Do i have to search it on the github repo ? It would be easier to put it with the release.

Edit : Found it (i think...) on searching on github.
Good question. I'm searching for it.

Here (https://github.com/AviSynth/AviSynthPlus/blob/MT/avs_core/include/avisynth.h)

[tormento]

@real.finder

Can you please explain me the following changes?


add Prefilter auto 16 (lsb) support
add TV_range bool
less lsb if not use lsb things and get ready for yv16 and yv24

Thanks ;)

[Groucho2004]

Some odd behaviour (XP64, 4GB RAM) using a script that should cause Avisynth throw an out of memory error:

Script:
setmemorymax(16000)
n = 12
colorbars(width = 1920 * n, height = 1080 * n, pixel_type = "rgb32").killaudio().assumefps(25, 1)
turnleft().pointresize(width() - 1920, height() - 1080)
turnright().pointresize(width() + 1920, height() + 1080)


Exception with r2022:
Exception 0xC0000005 [STATUS_ACCESS_VIOLATION]
Module: D:\WINNT\system32\vcruntime140.dll
Address: 0x000000000207CA44

Exception with r1576:
GetFrameBuffer: Returned a VFB with a 0 data pointer!
size=1194393631, max=3196588032, used=3583180893
I think we have run out of memory folks!


This happens for both, x86 and x64.

r1576 throws what you would expect. I tried the same with the last of pinterf's builds - No exception thrown, Avisynth simply never returns from the first GetFrame() call.

[pinterf]

Some odd behaviour (XP64, 4GB RAM) using a script that should cause Avisynth throw an out of memory error:

Script:
setmemorymax(16000)
n = 12
colorbars(width = 1920 * n, height = 1080 * n, pixel_type = "rgb32").killaudio().assumefps(25, 1)
turnleft().pointresize(width() - 1920, height() - 1080)
turnright().pointresize(width() + 1920, height() + 1080)



This happens for both, x86 and x64.

r1576 throws what you would expect. I tried the same with the last of pinterf's builds - No exception thrown, Avisynth simply never returns from the first GetFrame() call.
The cause is possibly the resizer temporary buffer allocation failure that is not caught. I think "simply does not return" means: with 8G RAM, the first frame was OK, but even a turnleft took 1200 seconds.
More info later.

[Groucho2004]

The cause is possibly the resizer temporary buffer allocation failure that is not catched I think "simply does not return" means: with 8G RAM, the first frame was OK, but even a turnleft took 1200 seconds.
More info later.
Thanks for looking into this.
I should have mentioned that I used AVSMeter to test this which reports the module in which an exception occurs. Then again, you probably knew that. ;)

[pinterf]

Thanks for looking into this.
I should have mentioned that I used AVSMeter to test this which reports the module in which an exception occurs. Then again, you probably knew that. ;)
The basic frame is 1.2Gbytes (23040x12960 rgb32)
When the horizontal resizer works, it creates a *21120 x 23040 sized (1.9GByte) videoframe, plus two temporary buffers, one 1.1Gbyte and a 1.9GByte one.

No wonder that with 4/8GB physical RAM is not enough (and this is the real limit, not the declared 160000MB you set in SetMemoryMax, it is overridden). 1.1GB for the Colorbars source frame, another 3-4x frame size during the resize process.

The 1200sec/frame was because of the OS memory swapping.

*This may be suboptimal to work with a new_horizontal*original_horizontal sized temporary frame, maybe it is needed for the horizontal resizer, because it works as turn+resize+turnback. Have to check it.

[Groucho2004]

The basic frame is 1.2Gbytes (23040x12960 rgb32)
When the horizontal resizer works, it creates a *21120 x 23040 sized (1.9GByte) videoframe, plus two temporary buffers, one 1.1Gbyte and a 1.9GByte one.

No wonder that with 4/8GB physical RAM is not enough (and this is the real limit, not the declared 160000MB you set in SetMemoryMax, it is overridden). 1.1GB for the Colorbars source frame, another 3-4x frame size during the resize process.
I'm aware of how much memory these frame sizes need (more or less).
I'm not sure if you realize that the point of that script is simply to force Avisynth to run out of memory and see how it behaves.

So, the only question is why the new version behaves differently.

[sl1pkn07]

any installer for dummies?

greetings

[ryrynz]

any installer for dummies?


Groucho has an installer but it's for 1858. Perhaps he can update it with the latest build and link it.

If you're needing a hand installing it then you'll likely wish to read a guide on how to use Avisynth as well.