Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 8th July 2016, 23:24   #2061  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,258
Quote:
Originally Posted by tormento View Post
Any modernization to SMDegrain with newer 16 bit filters would be glad
A big bottleneck is mvtools2. It is easier to backport it from VS than make it work with other colorspaces than 8 bit YV12/YUY.
pinterf is offline   Reply With Quote
Old 8th July 2016, 23:27   #2062  |  Link
vivan
/人 ◕ ‿‿ ◕ 人\
 
Join Date: May 2011
Location: Russia
Posts: 649
Quote:
Originally Posted by colours View Post
The only standard I know of defining what to do with full-range YCbCr is JFIF
H.264 defines full range, in Annex E. HEVC probably does it too.
vivan is offline   Reply With Quote
Old 8th July 2016, 23:27   #2063  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 1,039
Quote:
Originally Posted by pinterf View Post
A big bottleneck is mvtools2. It is easier to backport it from VS than make it work with other colorspaces than 8 bit YV12/YUY.
So, how does SMDegrain in LSB works?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 8th July 2016, 23:36   #2064  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,258
Mdegrain has lsb output option AFAIK
pinterf is offline   Reply With Quote
Old 9th July 2016, 00:17   #2065  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by pinterf View Post
And now one week holiday mode ON.
Probably I will read the forum but my internet will be limited.
Well deserved, thank you pinterf! By the time you come back, I should have the fixes for the MT mode ready. Though this weekend is for family, and the next week there will be a lot of traveling for work... The code changes aren't that earth-shaking as far as line counts go, but there are a lot of edge cases and I have probably rewritten everything about 2-3 times already. These are causing me a lot of headache.
__________________
AviSynth+
ultim is offline   Reply With Quote
Old 9th July 2016, 03:38   #2066  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,173
I'm porting AvsFilterNet to work with AviSynth+ and Visual Studio 2015. The project was never finished; the hard work was done but a few things were missing or done wrong.

Ultim, I was wondering what are the MT issues you're talking about. Now I'm seeing weird things (testing with the latest AVS build).

When running in MT mode 3, the calls are coming from many different threads. When running in MT mode 2, I would expect the GetFrame calls to come from the same threads as the calls to the constructors, but that's not the case. C++ will either "work" or give undefined behaviors, but C# is a lot more thread-aware and won't let it slip by when working with non-thread-safe objects. I'm surprised that C++ code works anyway. I tested MT mode 3 in pure C++ code too and the calls were indeed coming from many different threads.

Also, if the CreateFilter function returns a "FlipVertical" which is MT mode 1, then the plugin also runs in MT mode 1.

Now that I'm testing with AviSynthShader, I'm also seeing something else weird. If I run even without MT mode, my class gets instantiated 6 times? Not sure who is responsible for this.

Better wait until you bring out your next version for testing.
MysteryX is offline   Reply With Quote
Old 9th July 2016, 07:27   #2067  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
Quote:
When running in MT mode 3, the calls are coming from many different threads. When running in MT mode 2, I would expect the GetFrame calls to come from the same threads as the calls to the constructors, but that's not the case.
At the same time?
It's perfectly fine to get calls from different threads at different points of time.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth
TurboPascal7 is offline   Reply With Quote
Old 9th July 2016, 08:33   #2068  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,146
Quote:
Originally Posted by colours View Post
In case you're bad at arithmetic, 127.5 is equally close to both 127 and 128.
I said "rounded to ...", floor(127.5)=127 and round(127.5)=128, and fine, "rounded to one of its closest colors [0, 128, 128] at uint8_t", I like your suggestions and they are healthy but, you always overreact about bits and pieces of rigorousness problems in the description, it's not a scientific dissertation so it's okay long as you get what I was saying.

Quote:
Edit: Since vivan mentioned H.264, I went and checked that, and it defines a nominal chroma range of 0.5 to 2^n+0.5 in Annex E. (I assume HEVC says the same thing.)
think that should be 2^n-0.5 not +0.5 (255 = 2^8 - 1 not 2^8), who's bad at arithmetic now? , anyways, the midpoint goes 128 and that's good, I'll take that.

Quote:
But really, my original point was that these all map RGB values to YCbCr values linearly; the exact endpoints to use for the range is something we can decide after we get rid of the strange idea of mapping 0 to 0, 128 to 32768, and 255 to 65535.
0-0 128-32768 255-65535, that's almost linear, my point was little disturbance to the linearity to set the midpoint straight was the kind of tradeoff I would take.
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 9th July 2016, 08:48   #2069  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,146
Quote:
AVS (and by extension, AVS+) uses a different convention for the Convert* filters, where the nominal luma range is 0 to 255 and the nominal chroma range is 1 to 255.
I think that's just fairly wrong, midpoint goes (255-1)/2=127 by that definition and 32767 at uint16_t, and 127*257=32639 and that's pretty far from 32767..
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 9th July 2016, 09:22   #2070  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by MysteryX View Post
I'm porting AvsFilterNet to work with AviSynth+ and Visual Studio 2015. The project was never finished; the hard work was done but a few things were missing or done wrong.

Ultim, I was wondering what are the MT issues you're talking about. Now I'm seeing weird things (testing with the latest AVS build).

When running in MT mode 3, the calls are coming from many different threads. When running in MT mode 2, I would expect the GetFrame calls to come from the same threads as the calls to the constructors, but that's not the case. C++ will either "work" or give undefined behaviors, but C# is a lot more thread-aware and won't let it slip by when working with non-thread-safe objects. I'm surprised that C++ code works anyway. I tested MT mode 3 in pure C++ code too and the calls were indeed coming from many different threads.

Also, if the CreateFilter function returns a "FlipVertical" which is MT mode 1, then the plugin also runs in MT mode 1.

Now that I'm testing with AviSynthShader, I'm also seeing something else weird. If I run even without MT mode, my class gets instantiated 6 times? Not sure who is responsible for this.

Better wait until you bring out your next version for testing.
That you get the calls from different threads in MT mode 3 is normal. 3 only guarantees that the calls are never made at the same time, but none of the MT modes limits the calls to the same thread.

The main MT issue that I'm talking about is exactly the problem you described with FlipVertical. Of course it does not only affect FlipVertical, but any filter whose constructor returns another filter than itself which is of lesser protection. This is the infamous issue #37 in the bugtracker. When this gets solved, it will also allow plugin developers to specify MT modes again without relying on the non-final IScriptEnv2. I am also trying to improve the efficiency of MT mode 3, planning improvements to the threadpool(s), and add automated diagnostics. More details when each of these gets done.
__________________
AviSynth+
ultim is offline   Reply With Quote
Old 9th July 2016, 09:47   #2071  |  Link
colours
Registered User
 
colours's Avatar
 
Join Date: Mar 2014
Posts: 312
Quote:
Originally Posted by feisty2 View Post
think that should be 2^n-0.5 not +0.5 (255 = 2^8 - 1 not 2^8), who's bad at arithmetic now?
You and me alike, I guess. (Fixed.)

Quote:
Originally Posted by feisty2 View Post
I think that's just fairly wrong, midpoint goes (255-1)/2=127 by that definition and 32767 at uint16_t, and 127*257=32639 and that's pretty far from 32767..
The midpoint is still 128 for the AVS Convert* convention, and the scaling factor would be 32767/127 ≈ 258.01 if we were to assume a nominal chroma range of 1 to 65535 for 16-bpc YCbCr.

The nonlinear mapping you're suggesting is bad for the reason that it does not mesh with any of the full-range conventions in use, and, being non-linear, necessarily requires a bit more computation than a linear mapping. Changing bit depth should be an essentially free operation, not one that requires branching on every other pixel.

The mappings involved for the different chroma full-range conventions are thus:

Code:
# naive full-range ([0,255] to [0,65535])
x *= 257

# Dither_convert_* ([0.5,255.5] to [0.5⋅256,255.5⋅256])
x *= 256

# H.264-like ([0.5,255.5] to [0.5,65535.5])
# note: needs to be clamped to [0,65535]
x = 257 * x - 128

# AVS Convert*-like ([1,255] to [1,65535])
# note: needs to be clamped to [0,65535]
x = (32767 * x - 32640) / 127
__________________
Say no to AviSynth 2.5.8 and DirectShowSource!

Last edited by colours; 9th July 2016 at 10:59.
colours is offline   Reply With Quote
Old 9th July 2016, 10:17   #2072  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,146
dammit, just got myself another reason to stick with floating point by.. yeah, the H264 method seems pretty cool to me tho..
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 9th July 2016, 19:15   #2073  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,173
Quote:
Originally Posted by ultim View Post
The main MT issue that I'm talking about is exactly the problem you described with FlipVertical. Of course it does not only affect FlipVertical, but any filter whose constructor returns another filter than itself which is of lesser protection.
I've heard several people being very skeptical of MT and who are staying away from it. But really this issue is not that bad. For the most part, MT is working perfectly fine. As for this issue, it is not a context-dependent issue resulting in "undefined" behaviors. It will give a defined behavior for a plugin returning another plugin with a different MT mode.

One way to "patch" the issue is to return a dummy plugin at the end with the desired MT mode. It should work; however it only patches the issue without resolving it.
MysteryX is offline   Reply With Quote
Old 9th July 2016, 21:53   #2074  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 733
Quote:
Originally Posted by colours View Post

The mappings involved for the different chroma full-range conventions are thus:

Code:
...

# AVS Convert*-like ([1,255] to [1,65535])
# note: needs to be clamped to [0,65535]
x = (32767 * x - 32640) / 127
Looks like this is the same convention as used by graphics hardware to unpack signed normalized integers
Probably the right one?
__________________
VirtualDub2
shekh is offline   Reply With Quote
Old 10th July 2016, 00:47   #2075  |  Link
ajp_anton
Registered User
 
ajp_anton's Avatar
 
Join Date: Aug 2006
Location: Stockholm/Helsinki
Posts: 783
Quote:
Originally Posted by pinterf View Post
Unfortunately UtoY and VToY is reserved already, and works with 16/32 bit. They do not truncate to a Y8/Y16/Y32 clip.

So UtoY8 and VToY8 is silently working like UtoY16/32 and VtoY16/32. The result clip is greyscale one.
I did not like the names like UtoLumaOnly, etc... so it remained with the Y8 ending, until someone finds a proper name for it and everybody agrees with it
What about "ShowU" and "ShowV"? Similar to "ShowRed" etc for RGB. They could then all agree on what pixel_type to accept. Though IMO they should all just return a luma plane only.
ajp_anton is offline   Reply With Quote
Old 10th July 2016, 04:29   #2076  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 1,297
Quote:
Originally Posted by ajp_anton View Post
What about "ShowU" and "ShowV"? Similar to "ShowRed" etc for RGB. They could then all agree on what pixel_type to accept. Though IMO they should all just return a luma plane only.
I agree with you, but still see no problem with UtoYx and VToYx

If pinterf make UtoY8 and VToY8 silently working like UtoY16/32 and VtoY16/32 then he should make the same thing for ConvertToY8 so the old scripts work in 16/32 bit if the plugins support that
__________________
My Avisynth Stuff

Last edited by real.finder; 10th July 2016 at 05:28.
real.finder is offline   Reply With Quote
Old 10th July 2016, 07:49   #2077  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 1,258
He did ConvertToY8 works for 16/32. But this is illogical.
pinterf is offline   Reply With Quote
Old 10th July 2016, 11:31   #2078  |  Link
ajp_anton
Registered User
 
ajp_anton's Avatar
 
Join Date: Aug 2006
Location: Stockholm/Helsinki
Posts: 783
What about...

Include only "sane" function names in Avisynth+, and include an .avsi file with all legacy aliases needed for compatibility. That way the actual function name should already explain what it does ("does converttoy8 support 16-bit, and will it silently convert to 8-bit?"), and the aliases can be examined manually to see what they do, and can be modified if the user likes.
ajp_anton is offline   Reply With Quote
Old 11th July 2016, 21:43   #2079  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
EDIT: There is newer build available here.

In case you guys want to make sure pinterf has some work when he gets back, be sure to give a spin to the high bit-depth filters in r2043 and report any issues

Compared to the previous test build, we have:
- A lot more internal filters now that support 16/32-bit processing.
- A correctness fix to the float resampler.
- ConvertTo/FromStacked() moved to their own DLL, and added the ConvertTo/FromDoubleWidth() functions to support some yet-another-hbd-format-that-shouldn't-exist-but-is-apparently-in-use-by-LSMASHWorks-or-avs2yuv. Please rely on this plugin only when and only as long as necessary because it is provided only to support the transition to the proper high bit-depth formats. They will be killed off in due time.
- Resamplers and some other filters should now properly throw an out-of-memory error instead of sometimes crashing when low on memory.
- 8->16bit scaling is in this build done as described by ITU standards and Microsoft. This is what you want most of the time, unless you are saving to some rare exotic codec or a series of RGB images. This is not to say Avs+ will not support other methods, but this is what is done atm.
__________________
AviSynth+

Last edited by ultim; 20th July 2016 at 23:36.
ultim is offline   Reply With Quote
Old 12th July 2016, 04:47   #2080  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,173
Quick questions.
- How do I know a clip has UINT16 or FLOAT formats?
- What about Dithering?
MysteryX is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:03.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.