AviSynth+ thread Vol.2

[stax76]

I created an issue:

https://github.com/mysteryx93/AviSynthShader/issues/21

[ChaosKing]

@stax76 btw there is a vapoursynth plugin that supports "GLSL shaders in mpv syntax" in case you want to integrate it in staxrip https://github.com/Lypheo/vs-placebo

[stax76]

@stax76 btw there is a vapoursynth plugin that supports "GLSL shaders in mpv syntax" in case you want to integrate it in staxrip https://github.com/Lypheo/vs-placebo

I've bookmarked it but currently there are way to many requests for staxrip and there is also mpv.net and other plans.

[markiemarcus]

I've bookmarked it but currently there are way to many requests for staxrip and there is also mpv.net and other plans.

The updated TMM2 works fine by the way.

Appreciate all of your efforts. Staxrip is my go-to!

[MysteryX]

Wow there's been a lot going on lately!

Glad to see lots of improvements and active development to Avisynth! With the MT improvements, how does AVS' multi-threading now compares to VapourSynth?

@stax76 btw there is a vapoursynth plugin that supports "GLSL shaders in mpv syntax" in case you want to integrate it in staxrip https://github.com/Lypheo/vs-placebo
That GLSL interpreter is written in C++? Perhaps it wouldn't be a bad idea to merge it with AvisynthShader -- and have it working on both Avisynth and VapourSynth.

I've bookmarked it but currently there are way to many requests for staxrip and there is also mpv.net and other plans.
Stax76, I didn't realize MPV.NET was from you! I was planning to use that at some point; perhaps add my own mods to it.

Got the notice about updating AvisynthShader. Should be quick enough to fix.

[vcmohan]

What purpose does the variable "t" have? Can you elaborate?
Sorry I missed out on this. In some cases even with one cpu several threads run. So here t is the number of threads per cpu. I was keeping this as 2. Now of course I deleted this entire construct.
@pinterf. You refered me to a code where SetCacheHints is used. I find that GET_CACHE_MTMODE is one of the items in long list of CachePolicy. Does it mean that this code may be called a number of times? Or with some other item in the list? The code checks whether cachehints==CACHE_GET_MTMODE and if negative, returns a zero. Thinking about I now got doubts. Request clarification.

[LigH]

(@Myrsloik - 20 bit)

Special exception: 20 bit LPCM stereo audio tracks in DVD Video use an interleaved storage format, combining the least significant "nibbles" (half-bytes) of both channels to a fifth byte. This is not compliant to PCM formats supported by most operating systems...

Just as funny side note.

[FranceBB]

Out of curiosity, is high bit depth availability going to be implemented in Histogram("color2") ?
'cause currently we have Luma that can be high bit depth, but chroma limited to 8bit planar...

[markiemarcus]

(@Myrsloik - 20 bit)

Special exception: 20 bit LPCM stereo audio tracks in DVD Video use an interleaved storage format, combining the least significant "nibbles" (half-bytes) of both channels to a fifth byte. This is not compliant to PCM formats supported by most operating systems...

Just as funny side note.

Thanks for that, had absolutely no idea this was ever used in DVD-Video. I have only ever come across this format once before, about 15 years ago so I'd forgotten all about it. It was used as a storage format for an in-house instrument sample library that, although dated by today's standards, was way ahead of its time. It briefly caused some confusion because it was to be ported to a much more modern interface.

[pinterf]

Out of curiosity, is high bit depth availability going to be implemented in Histogram("color2") ?
'cause currently we have Luma that can be high bit depth, but chroma limited to 8bit planar...
Yes, sooner or later, I have started already but have put it aside. I guess I'd need some more hours but I cannot schedule it.

[qyot27]

AviSynth+ 3.6.1 has been released. (https://github.com/AviSynth/AviSynthPlus/releases)

Fix: proper handling of autoload directories precedence:
PluginDir+ in Software/Avisynth in HKEY_CURRENT_USER (highest priority)
PluginDir+ in Software/Avisynth in HKEY_LOCAL_MACHINE
PluginDir2_5 in Software/Avisynth in HKEY_CURRENT_USER
PluginDir2_5 in Software/Avisynth in HKEY_LOCAL_MACHINE (lowest priority)
Plugin (dll file name) found in a lower priority folder will not load if a similarly named plugin
already existed earlier.
fix: GeneralConvolution: incorrect parse of negative integer coefficient (added +1)
Regression since r2772.
Fix: GeneralConvolution: possible crash when chroma=true for 420 and 422 formats
Fix: ScriptClip + Runtime function object (which are new in 3.6) under heavy multithreading
New: Histogram("levels") to allow greyscale
Fix 3.6 regressions
when explicit "return last" was needed when followed by legacy function definition.
Windows XP is supported again (thread local storage workaround)
Stabilize CPP 2.5 plugins
allow forced named arrays usage again from plugins (MP_PipeLine)
Frame property related constants to match existing enum style in avisynth.h.
Plus they are not colliding now with VapourSynth's definitions.

Effectively, this is the same as the test8 build, just formalized so it shows up in the tags and release tarballs.

As part of the various builds, there is also a test build for ARM64 this time around. I'm not too sure of how widely Windows 10 on ARM devices are available, but it was easy enough to build (it does lack several of the default plugins because those are strictly for x86(-64) processors). You'll probably need to make sure you have the vcredist for ARM64.

[manolito]

Thanks very much for the new stable build. Just running a longer conversion under the x86 version of this build, everything looks fine... :D

Two questions:

I noticed that the files from the Test8 package are much smaller than the files in the 3.6.1 final build. I could only see that DevIL.dll is packed with UPX, all other files are not. Is pinterf using a different packer for his files (if so, which one), or is there another reason for the difference in size?

Why are there still versions for WinXP and for Win7 and above? Pinterf published his test versions only for "WinXP and above", and when I asked him if there was any disadvantage in using the XP version under a newer Windows he answered that nothing was disabled in the XP version. All CPU features would be used, and I could confirm that I did not get any speed penalty when using the x86 XP version under Win7-64.
So is there any good reason to maintain two different compiles when the XP compatible build has no drawbacks at all when used under a newer Windows version?

Thanks again
manolito

[pinterf]

Windows XP builds are with thread safe init compiler flag switched off. It's not healthy. Once I had problems with masktools2 because of that, a perfectly fine c++ initialization structure failed when multithreading. I had debugged it for at least three days then made a workaround. So far I did not encounter such problem with Avisynth+ or anything else but since the code is always changing let's not make problems for regular users.

[qyot27]

I noticed that the files from the Test8 package are much smaller than the files in the 3.6.1 final build. I could only see that DevIL.dll is packed with UPX, all other files are not. Is pinterf using a different packer for his files (if so, which one), or is there another reason for the difference in size?
Most likely it's the difference between MSVC's Release and RelWithDebInfo configurations.

[Groucho2004]

I noticed that the files from the Test8 package are much smaller than the files in the 3.6.1 final build. I could only see that DevIL.dll is packed with UPX, all other files are not.There are no UPX packed files in the "files_only" package. I have not tried the installer. There used to be upx compression ages ago but it would surprise me if these compression commands made it into the AVS+ setup scripts. Anyhow, UPX compression is even more outdated than XP. :D

Oh yeah, thanks qyot27 and pinterf for the new release! I'll update the Universal Installer later.

[FranceBB]

Yes, sooner or later, I have started already but have put it aside. I guess I'd need some more hours but I cannot schedule it.

Ok, got it. :)
Thank you for the new release, by the way.
It required a lot of testing from the community, 8 test build and probably several hours on your end, but I'm glad that everything is finally working again. :)

[StainlessS]

Oh yeah, thanks qyot27 and pinterf for the new release!

+1 on that, great work guys, thanx very much
https://www.cosgan.de/images/midi/froehlich/p017.gif

[manolito]

There are no UPX packed files in the "files_only" package.

I was talking about the pinterf test 8 build. And in this build the x86 DevIL.dll is packed with UPX. And yes, being the retro person I am, I really like UPX... :devil:

[manolito]

Most likely it's the difference between MSVC's Release and RelWithDebInfo configurations.

I am totally ignorant about different compilers, but if I come across two releases with identical functionality, and the files in one of these builds are half the size of the other build then I know which build I will use...

After a couple of more test conversions I did get a few crashes under the 3.6.1 x86 standard build in MT mode where the pinterf test8 x86 build behaved nicely (under Win7-64). Probably needs more testing, but for now I have some real world conversions to do, so I am going back to the pinterf test8 version...

[Groucho2004]

I was talking about the pinterf test 8 build. And in this build the x86 DevIL.dll is packed with UPX.Right, I should have read your post thoroughly, I apologise.

And yes, being the retro person I am, I really like UPX... :devil:That makes no sense at all. UPX was helpful when hard drives had average sizes of 250-500 MB 30 years ago.

UPX just adds another layer of complexity, not to mention that virus engines love to report false positives for UPX compressed binaries.

Since you're so hell-bent on being retro you're probably still contributing to global warming by using inefficient incandescent light bulbs instead of LED bulbs that consume less 8-10 times less energy, right?

To each his own I always say but you have to get out of your retro cave eventually (or not, whatever).

[StainlessS]

I loved my 500MHz AMD Athlon with IronSides chipset, was faster than 640MHz Intel, but have dropped it long since,
I now only have a single P4 with hyperthreading, P4's without HT now binned. I'm a bit retro but you gotta come out
of your Bat-ty Cave at some point, otherwise the Joker's on you.

[qyot27]

There are no UPX packed files in the "files_only" package. I have not tried the installer. There used to be upx compression ages ago but it would surprise me if these compression commands made it into the AVS+ setup scripts. Anyhow, UPX compression is even more outdated than XP. :D
Despite best practices for git stating otherwise, both x64 and x86 DevIL.dll themselves are files in the source tree. The x86 .dll is UPX-packed, but the x64 one isn't.

This usually means I have to remember to unpack it before creating the installers. Sometimes I forget, so if you compare back to previous installers for 3.4.0, 3.5.0, 3.5.1, and 3.6.0, it might have been unpacked in a couple of them, but not in others.

Now, my actual preference would be that we really fix ImageSeq for cross-platform and use the opportunity to remove DevIL.dll from the source tree, because we switch to using the version installed to the system. On Windows this could manifest itself as DevIL.dll only getting shipped in the installers/.7z, or it no longer being necessary at all (because it's been linked in statically).

[real.finder]

seems dither=-1 in convertbits need fix/update

https://forum.doom9.org/showthread.php?p=1916319

[pinterf]

seems dither=-1 in convertbits need fix/update

https://forum.doom9.org/showthread.php?p=1916319
ConvertBits is doing bit shift for YUV color spaces when converting up or down in either way. What fix is needed for that?

[real.finder]

ConvertBits is doing bit shift for YUV color spaces when converting up or down in either way. What fix is needed for that?

what about when using it with fulls=true?

https://forum.doom9.org/showthread.php?p=1916338#post1916338

it still not ok

[pinterf]

Define "not OK". Full range conversion is done as:
destination_pixel = round(source_pixel * range_target_max / range_source_max)
in C: truncate(source_pixel * range_target_max / range_source_max + 0.5)

Could zou compare with z_ConvertFormat behaviour?

[real.finder]

Define "not OK". Full range conversion is done as:
destination_pixel = round(source_pixel * range_target_max / range_source_max)
in C: truncate(source_pixel * range_target_max / range_source_max + 0.5)

Could zou compare with z_ConvertFormat behaviour?

using this code

ColorBars
converttoyv12
AddGrainC
Interleave(neo_vd.luma_histogram(),convertbits(16).neo_vd().convertbits(8,fulls=true).luma_histogram())

and seems fine if you replace convertbits(8,fulls=true) with z_ConvertFormat(pixel_type="YV12",dither_type="none")

[pinterf]

fulls=fulld=true is the rule for RGB. You cannot use it for YUV.
It's not questions that downshift must be used, the question is that with or without rounding (how we define it). Logically "with rounding" is the way to go, but it may affect more like only this conversion.
Then there are places where simplified view at 8 bits is used (like Histogram), all these places have to be re-thought.

[MeteorRain]

Right. Like I said, you need rounding down from 16-bit to 8-bit. Not "fulls", not "fulld", but rounding.
One of the option is to dither down, so, use tools like f3kdb or dither tools. 8-bit operations are inaccurate, so rounding error is expected.

You get the high accuracy from high bit depth, you are supposed to take advantage of it, do more processing under high bit depth, and at the last step, dither it back to target bit depth.
Raise it to 16-bit, do some fine operation and lower it down, this whole process doesn't make much sense -- just do it in 8-bit then.

[pinterf]

I think ConvertBits was among the first things when I started playing with high bit depth support many years ago. Strange that no one complained on it so far, it's so obvious. I suppose z_xxxx is using float data type inside (colorspace and bit depth conversions can be chained) which is rounded in a usual way when goint back to the integer.

[real.finder]

Right. Like I said, you need rounding down from 16-bit to 8-bit. Not "fulls", not "fulld", but rounding.
One of the option is to dither down, so, use tools like f3kdb or dither tools. 8-bit operations are inaccurate, so rounding error is expected.

You get the high accuracy from high bit depth, you are supposed to take advantage of it, do more processing under high bit depth, and at the last step, dither it back to target bit depth.
Raise it to 16-bit, do some fine operation and lower it down, this whole process doesn't make much sense -- just do it in 8-bit then.

the point of using full is to not use bit shift method (for testing against other methods)

and I know about high accuracy and high bit depth, and no point from 8-bit -> 16-bit -> 8-bit but it was for test the output of VD in 16bit in first place then we note the problem in convertbits with dither=-1

[Myrsloik]

Does avs+ work on big endian cpus now?

[mp3dom]

asd posted some updated plugins with the v8 interface on the AviSynth+ x64 plugins page on the wiki: (http://avisynth.nl/index.php/AviSynth%2B_x64_plugins)


(Added DCTFilter v8 interface)


I don't know if this is the right place to made a "request", but the original Tom Barry's DCTfilter had "offx" and "offy" parameters for offsets which apparently have been removed in later versions. These were useful to run this script (https://forum.doom9.org/showpost.php?p=1191009&postcount=6) that mimic a lowpass filter. Is it possible to add these offset parameters back into current DCTFilter or maybe there're other ways to get the exact same "lowpass" results?
Thanks

[MeteorRain]

I was refactoring convert_audio class.

The reference code ToFloat() and FromFloat() are simply bitshifting, while 8 to 16 and 16 to 8 are doing full range conversion.

That is,
0x00 => 0x8000
0x80 => 0x0080
0xFF => 0x7FFF

Now I'm trying to implement conversions between 16, 24 and 32 bits.

16=>24
0x0000 => 0x000080
0x0010 => 0x001080
0x0100 => 0x010081
0x1000 => 0x100090
0x7FFF => 0x7FFFFF
0x8000 => 0x800000
0x8010 => 0x801000
0xFFFF => 0xFFFF7F

Does that look right to you? Or should I simply bit shift?

[tebasuna51]

@MeteorRain
Simple bit shift.

The other option seems a fix inverted dither (the random dither must be aplied from high to low precission)
For what add a fix noise to pure silence (0x0000 => 0x000080)?

There are soft (eac3to) than read the low bits to know the exact precission.
For instance a lossless TrueHD track are always decode like 24 bits int, but eac3to can convert it to 16 bits if all the low 8 bits are 0, and can be recompressed to Flac 16 bits without lose precission.

There are also audio with 20 bits precission than need 24 bits with the last 4 bits to 0.

[MeteorRain]

That makes sense. Any idea why 8 to 16 code added the extra noise?
0x80 (unsigned byte center) => 0x0080

[pinterf]

AviSynth+ 3.6.2-test1 (https://drive.google.com/file/d/1_IwlcyDiv-0gnV4vPttLWV4urEaDAdMH/view?usp=sharing)
20200624 3.6.2-dev
------------------
- Fix: ConvertBits (YUV): proper rounding when bit depth is reduced and origin is 10-16 bits
(added rounder before bit-shift)
- New: Histogram("color2") to support 10+ bits.
Allow bits=x (x=8,9,10,11,12) parameter for this kind of histogram as well.

[tebasuna51]

Any idea why 8 to 16 code added the extra noise?
0x80 (unsigned byte center) => 0x0080
Nope.

But I never see audio 8 bits. Don't worry with that unused format.

[feisty2]

Nope.

But I never see audio 8 bits. Don't worry with that unused format.

audio from last century video games?

[pinterf]

(I was programming 6 bit audio on the good old PC speaker :) )

[pinterf]

Does avs+ work on big endian cpus now?
I would be surprised if it would work seamlessly. I'm totally uneducated on big endian CPUs other that once I was programming POS terminals with Motorola 68000.

[Myrsloik]

I would be surprised if it would work seamlessly. I'm totally uneducated on big endian CPUs other that once I was programming POS terminals with Motorola 68000.

If you didn't test it then the answer is definitely no with all that old code lying around...

[pinterf]

If you didn't test it then the answer is definitely no with all that old code lying around...
Yep, for example the tokenizer and the parser would have definitely failed a year ago until we used the _i string literal construct
https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/core/parser/tokenizer.cpp#L196
https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/core/parser/scriptparser.h#L98

[real.finder]

AviSynth+ 3.6.2-test1 (https://drive.google.com/file/d/1_IwlcyDiv-0gnV4vPttLWV4urEaDAdMH/view?usp=sharing)
20200624 3.6.2-dev
------------------
- Fix: ConvertBits (YUV): proper rounding when bit depth is reduced and origin is 10-16 bits
(added rounder before bit-shift)
- New: Histogram("color2") to support 10+ bits.
Allow bits=x (x=8,9,10,11,12) parameter for this kind of histogram as well.

that fix that convertbits(8) case, thanks

btw, is it possible to support dither for float to int?

[MeteorRain]

Nope.

But I never see audio 8 bits. Don't worry with that unused format.

Alright, in that case I'm gonna leave 8-16 conversion standalone and untouched, and rewrite everything else. @pinterf @qyot how's that work for you guys?

By the way I always mistype pinterf as printf...

[Sharc]

Nope.

But I never see audio 8 bits. Don't worry with that unused format.
8 bit PCM audio from VHS transfers, for example.

[Boulder]

8 bit PCM audio from VHS transfers, for example.

Is there some capture device that uses only 8-bit audio? Personally I see no reason to use anything below 16 bits.

[pinterf]

Alright, in that case I'm gonna leave 8-16 conversion standalone and untouched, and rewrite everything else. @pinterf @qyot how's that work for you guys?

I don't mind, audio part was always out of my scope. I even left the inline asm there.

By the way I always mistype pinterf as printf...
s-printf (senior :) )

[MeteorRain]

I don't mind, audio part was always out of my scope. I even left the inline asm there.

Yea that's exactly why I want to improve it a bit.

By the way, what's the development process now? Shall I raise a PR directly to main repo, or should I raise it to your copy first? Anyone available for a code review? I notice that you are also in the process of porting to ARM, maybe that parts should be tested by someone before the merge?

[pinterf]

qyot27 is our merge-master, my own repo is a bit abandoned. Let's wait for qyot27 what he recommends.