Port of NNEDI under new v2.6 AVS API [Archive] - Page 10

View Full Version : Port of NNEDI under new v2.6 AVS API

Pages : 1 2 3 4 5 6 7 8 9 [10] 11 12 13

burfadel

22nd August 2017, 16:38

Apologies for the thread mixup :).

The only thing required for the script I listed that doesn't work with nnedi3() before it (but works if prefetch(4) is place in between) is the latest MVTools:
https://github.com/pinterf/mvtools/releases

jpsdr

22nd August 2017, 18:14

I've been able to reproduce with the following script (script working but freeze on exit) :

nnedi3()
super = MSuper (hpad=16, vpad=16)
bvec2 = MAnalyse (super, isb = true, delta = 2)
bvec1 = MAnalyse (super, isb = true, delta = 1)
fvec1 = MAnalyse (super, isb = false, delta = 1)
fvec2 = MAnalyse (super, isb = false, delta = 2)
MDegrain2 (super, bvec1, fvec1, bvec2, fvec2)

I'll investigate tomorrow. I hope i'll find something, because it's even more odd than the resample issue...

jpsdr

22nd August 2017, 19:26

When you have eliminated the impossible, whatever remains, however improbable, must be the truth.
I've made a little test, i have a little idea, but i was saying "no, impossible"...
When you add MDegrain2, i don't know why, but when closing the file, the destructor of nnedi3 is not called anymore !!!!
WTF is happening ??????

jpsdr

23rd August 2017, 08:01

I've tested with all my filters, it's not specific to nnedi3, i don't know how it's possible, but MDegrain2 clearly mess-up things, and the destructor of the filters is not called anymore when closing the script.
There is nothing i can do about it. I can try to find a workaround for avoiding the freeze, but there is nothing i can do for the resources allocated by the filters and not free properply because when the dll is unloaded, the destructor of the filters have not been called.

DJATOM

23rd August 2017, 08:40

jpsdr
If you think that's MVTools bug, you should open new issue on pinterf's repo :)

pinterf

23rd August 2017, 13:54

I wonder whether it happens with the old 2.6.0.5 mvtools2 version, which was the base of my current mod.

jpsdr

23rd August 2017, 15:52

No it doesn't, i see the destructor called properly with 2.6.0.5.

jpsdr

27th August 2017, 12:17

New version, see first post, should work as partial workaround the the MDgrain2 issue.

pinterf

28th August 2017, 10:24

Now I'm really back from holiday, I hope things will be solved soon. The problems first occured in mvtools2 2.7.1.22, I'll check those modifications, I already have a suspect.

jpsdr

28th August 2017, 11:49

When fixed, i'll take a look at the commit, because i'm realy curious to know how you can make possible to break the destructor call... ;)

TheFluff

28th August 2017, 16:38

I dunno what might be up in this case but C++ certainly has no shortage of ways to shoot yourself in the foot. One easy way to create a similar problem is by deleting an instance of a derived class through a pointer typed for the base class without declaring the base class destructor as virtual (https://stackoverflow.com/questions/461203/when-to-use-virtual-destructors).

jpsdr

28th August 2017, 16:48

Well, all my destructors are virtual, and all filters are derivated from GenericVideoFilter. But if it was this, i think the issue would be all the time, it seems more tricky.

Dreamland

20th October 2017, 08:23

Help me!! I have just installed win 10 fall creator update 64 bit (fresh installation) and
and i have this error
https://t14.pixhost.org/thumbs/59/54587993_2017-10-20_090756.png (https://pixhost.org/show/59/54587993_2017-10-20_090756.png)
usually installation..usually plugins ,usually dependence installed, but now, with FCU i have this issue !

why?

NNEDI3 V 0.9.4.47 (same error with older versions)

I'm getting crazy! :(

thx for helps :)

jpsdr

20th October 2017, 09:59

Dreamland

20th October 2017, 10:13

I don't have Windows 10, and have no idea what "Win 10 fall creator" or "FCU" are...

The only basic "stupid" questions i can see for now are :
- What DLL version are you using ? Be sure to not use a 32bits DLL for 64Bits and reverse.
- Have you installed the Visual Studio redistribuable required ? And the Intel redistribuable if you're using the Intel version ?

If everything for these two are correct, i personnaly have no further idea.

DLL=NNEDI3_v0_9_4_47/x86/Release_Intel_W7_Core2_AVX2 for avisinth 2.6

VBCRedist_AIO_x86_x64.exe
15/10/2017
http://repacks.net/forum/viewtopic.php?t=125

intel redistribuable: icl_redist_intel64_2016.4.246.msi

https://software.intel.com/en-us/articles/redistributables-for-intel-parallel-studio-xe-2016-composer-edition-for-windows

thx a lot

jpsdr

20th October 2017, 10:20

The readme said :

Release_Intel versions are compiled with Intel Parallel Studio 2017 (icl 17.0.4.210)

So maybe you need icl_redist_intel64_2017.4.210.msi instead. Useless of course if you're not using an Intel release version.
As for the "All in One package", i don't know, i don't use this kind of thing, so, if it provides the same redistribuable than provided in the readme link, it should be fine.

Euh... Just noticed... You're using an x86 DLL, and install the x64 Intel, and with x64 things...? I don't understand this mess. Either you need to use the x64 DLL, either you need to install icl_redist_ia32_2017.4.210.msi

Groucho2004

20th October 2017, 10:35

FYI - error 0X3E6 = ERROR_NOACCESS
Possible cause:
https://support.microsoft.com/en-us/help/196069/prb-loadlibrary-api-fails-with-998-error-noaccess-error

Dreamland

20th October 2017, 10:39

no works icl_redist_intel64_2017.4.210.msi

jpsdr

20th October 2017, 11:07

Apparently you use an x86 DLL, so as i said, you may probably need icl_redist_ia32_2017.4.210.msi instead.

Dreamland

20th October 2017, 13:15

icl_redist_ia32_2017.4.210.msi

same error..
before, with win 10 creator update, (the same plugins and nnedi3 ) worked 100%
i don't know where is the issue

thx a lot for you help

Groucho2004

20th October 2017, 13:30

icl_redist_ia32_2017.4.210.msi

same error..
before, with win 10 creator update, (the same plugins and nnedi3 ) worked 100%
i don't know where is the issue
A couple of things you could try:
1. Use the original 0.9.4 version of nnedi3, download from here (https://www.dropbox.com/s/zinhar4jru6u1p7/nnedi3.7z?dl=1).
2. Run "AVSMeter -avsinfo" and check if there are any errors.

Dreamland

20th October 2017, 13:43

ok ..give me time ...thanks

Dreamland

20th October 2017, 13:59

now works
https://forum.doom9.org/showthread.php?p=1788850#post1788850
http://www.mediafire.com/file/5rp8jtv78otxxku/icl+-+xp.rar
this was my problem : My old DLL was corrupted (i think) :sly:...overwrite with new DLL and now works

jpsdr

20th October 2017, 17:17

Why do you need Windows XP things when you are under Windows 10 ???

Dreamland

20th October 2017, 20:25

I know it's so strange..but putting and overwriting in system32/ syswow64 this DLL for XP, i have no error in my NNEDI3

https://t14.pixhost.org/thumbs/67/54634829_appunti01.jpg (https://pixhost.org/show/67/54634829_appunti01.jpg)

(excuse me for my bad english)

jpsdr

24th November 2017, 12:50

New version, see first post.

monohouse

28th January 2018, 12:02

is there no build of NNEDI3 for AMD ? I am using latest version and my script on core 2 duo give 0.03 fps/core but on my thuban 1090T I have only 0.01 fps/core

your version has SSE2 but the thuban has newer instructions that nnedi3 doesn't appear to use (SSE3, SSE4A, 3DNow!, 3DNow+!) and there is no AVX in thuban :x

jpsdr

28th January 2018, 14:09

No, sorry, and i don't think there would great step with the instructions sets you list.

monohouse

28th January 2018, 16:06

so 0.01 in thuban is normal when core 2 duo have 0.03 ? I thought they supposed to be same performance per core

Groucho2004

28th January 2018, 16:18

so 0.01 in thuban is normal when core 2 duo have 0.03 ? I thought they supposed to be same performance per coreHow are you measuring the speed? Can you post the script you used for your test?

monohouse

28th January 2018, 17:10

MCTD(radius=2, twopass=false, refine=true, useTTmpSm=true, limit=1, limit2=1, stabilize=true, maxr=1, TTstr=1, chroma=true, sharp=false, adapt=false, strength=1000, sigma=0, pMode=0,fixband=false,pp=false,search=3, searchparam=4, pel=4, pelsearch=4, bwbh=512, owoh=256, blksize=4, overlap=2, edgeclean=false, ECrad=1, ECthr=32, ECmode="Removegrain(2,0)",protect=false,AA=false,deblock=false,post=0,interlaced=true,GPU=true,precision=2,bt=4,enhance=false)
daa3(nthreads=1)
LSFmod(defaults="slow",strength=20,Smode=3,Smethod=3,Lmode=0,overshoot=0,preblur="ON",secure=false,edgemode=0,soft=0,soothe=false,ss_x=1.00,ss_y=1.00)

Groucho2004

28th January 2018, 18:12

You did not answer my first question. Also, do you realize how many plugins are used in your script? How do you single out nnedi3 as the bottleneck?

monohouse

29th January 2018, 01:26

x264 is writing the encoding speed, I tested with nthreads=6 on the thuban using all 6 cores and got 0.04 fps, nthreads parameter of the daa3 is only giving nnedi3 the number of threads :)

I tested:
with linux in wine32 version 3-rc6
linux wine64-rc6
windows 7 WOW64 (virtualbox)
windows7 x64 (virtualbox)

they all give the same speed :x

avs2yuv.exe -raw "A:\test.avs" -o - | x264.x86.exe --input-res 720x480 --fps 30000/1001 --qp 0 --level 6.2 --ref 16 --bframes 16 --b-adapt 2 --8x8dct --mixed-refs --direct auto --slow-firstpass --no-fast-pskip --me tesa --merange 24 --partitions all --rc-lookahead 60 --subme 11 --trellis 2 - -o "W:\test.264"

jpsdr

29th January 2018, 10:06

Seriously...? You have a lot of plugin and running x264 and think that the difference of fps comes only from nnedi3...??!! And more, you seems to use wine/virtualbox kind of things, which also doesn't help !
How can you be so sure ?
I don't say it's not possible, but personnaly, i also think of x264, which is more optimised for Intel CPU than for AMD.

Andouille

29th January 2018, 20:25

avs2yuv.exe -raw "A:\test.avs" -o - | x264.x86.exe --input-res 720x480 --fps 30000/1001 --qp 0 --level 6.2 --ref 16 --bframes 16 --b-adapt 2 --8x8dct --mixed-refs --direct auto --slow-firstpass --no-fast-pskip --me tesa --merange 24 --partitions all --rc-lookahead 60 --subme 11 --trellis 2 - -o "W:\test.264"

Are you aware that most your "placebo" and ultramegagigaslow settings are not even used when lossless encoding ?

And Groucho also pointed out that a LOT of plugins are used for your filtering.

monohouse

30th January 2018, 10:58

I am sure it is nnedi3 because when nnedi3 plugin threads increase to 6 the speedup is linear from 0.01 to 0.04, and the reason that it is 0.04 (and not 0.06) is because nnedi3 with threads=6 is not consistently using all 6 cores to 100%. you don't wanne help ? that's fine, at least have the decency to admit it, instead of trying to "drop the bag on something else" claiming that I am wrong or something else is wrong in the way that I do it.

WTF is the problem with wine/virtualbox ? do you even know these systems that you claim them to have effect on the performance ?!

what do all the other plugins have to do with anything when there is almost linear scaling in performance when nnedi3 threads increase ?!

you think you are the only one who can use question marks with inclamations ?! well guess what - so can I

if you would have simply said "no, we will not help you" you would have attained a little respect, now the community will look at you like fools trying to hide the true and the facts and for being snakes trying to make me look bad as if I don't know what I am doing. now that you got pwned - you ruined your reputation, instead of being honest and say things for what they are. but now that problem is your problem not my - it out of my hands now, I wasn't looking to hurt you - I was looking to get help.

real.finder

30th January 2018, 12:27

I am sure it is nnedi3 because when nnedi3 plugin threads increase to 6 the speedup is linear from 0.01 to 0.04

as they told you, you can make sure by that

use this to test

ColorBars(pixel_type="yv12")
nnedi3

using avsmeter not x264

edit: and try to not use Release_Intel_XXXXXX dlls

monohouse

30th January 2018, 13:16

jpsdr

30th January 2018, 15:24

I like this kind of results more.
If you want to compare the speed of the same code path, in the Haswell test, add opt=2 (force not more than SSE2) or opt=3 (force not more than SSE4.1), i don't know between both, what your thuban can do.
If you want to be 200% sure, put the same on the thuban test, this way, you will be sure that both are using the same code path, and maybe you can compare speed more accurate.

I don't know the default size for colorbar, but using pscrn=0 to bench is in a way a good thing. Why...?
Because colorbar is probably around 95% of flat area, meaning 95% of the process will be the simple bilinear upscale with default pscrn=2, not the neural network. Meaning, very few computation versus a lot of memory access, so, the increase speed and CPU usage will be not linear with threads increase (and be very poor) with this specific test case.
Puting pscrn=0 will force the neural network computation on all pixels, and you've also changes the parameters to put the maximum neural network size. This way, you have a lot of computation and less memory access. This way, you can more accurately check the increase speed and CPU usage with threads increase.

Warning ! : Doing this kind of test, i've noticed that if you're doing this most stressfull configuration, and if you have AVX2, you'll skyrock the CPU usage and the T° increase of the CPU, almost the same way the most stressfull test of Prime95 on AVX2 is overheating your CPU.

monohouse

30th January 2018, 16:41

yhe, thank to intel for put shit for grease in the CPU

TheFluff

30th January 2018, 18:13

if you would have simply said "no, we will not help you" you would have attained a little respect, now the community will look at you like fools trying to hide the true and the facts and for being snakes trying to make me look bad as if I don't know what I am doing. now that you got pwned - you ruined your reputation, instead of being honest and say things for what they are. but now that problem is your problem not my - it out of my hands now, I wasn't looking to hurt you - I was looking to get help.

First, and most importantly: lol

it looks like under synthetic load the haswell is 3 times more fast, under real load it's more than 5 times more fast, neither are correct: the thuban is supposed to be only half as fast as a haswell

when adjusted for clock rate difference between 4200 and 3000, the haswell under real load is still 4.07 times faster than the thuban

notice that on haswell, between default and proper load the difference is a factor of 1906.8, but on the thuban it is 3196.9 (1300 times slower than it should be)

NNEDI3 on colorbars with the prescreener on is almost certainly memory bound, so that one is completely irrelevant unless you really want to benchmark your memory speed. You'll most likely get essentially the same fps from bicubicresize.
(edit: sources tell me the prescreener is actually not that fast so the above might not actually be completely true)

Moving on, by what metric exactly is Thuban supposed to be half as fast as Haswell? cpubenchmark.net numbers, or what? Those are pretty much meaningless in this context. NNEDI3 is full of 128-bit SSE operations which (IIRC) took two uops on K10 (since it was implemented as two 64-bit ops) but only one on Intel CPU's starting with the... man I don't even know, Core 2? Well before Haswell at any rate. K10 could also only use two floating point execution units - FADD and FMUL - for a lot of the SSE stuff, with a third (FMISC) also being usable for loads and stores. It's hard to compare directly with Haswell's more complex fused architecture which has eight execution units with different capabilities, but in general you should expect Haswell to have at least twice the SSE throughput per clock cycle compared to the K10.

To verify I wasn't talking completely out of my butt, I went to take a quick look at Agner Fog's instruction reference, and sure enough, on Haswell you can get two XMM multiplies (addps/mulps) per clock cycle compared to one on K10.

tl;dr: don't expect much of an almost 10 year old CPU

edit: Addendum: I misremembered, the "128-bit uops implemented as two 64-bit uops" thing was only on K8, not K10. Nevertheless, the K10 has other issues, particularly with scheduling, and the general statement that Haswell should have at least twice the SSE throughput per clock cycle stands. In fact it's probably reasonable to expect better than that; it should be realistically possible to get an average throughput of four instructions per clock cycle.

monohouse

31st January 2018, 04:42

thank, that is most informative, I was not using such detailed analyse of the architectural differences, I was expecting half as fast based on the timeframe of the cpu (nehaylem timeframe), and yes, something like cpubenchmark.net.

so SSE is crap on thuban and combined with scheduling problems it get worse :x
that explain why core 2 duo is so mutch better
what you say it make sense on haswell and sits well with the results: 4 times faster: twice faster from SSE architecture and twice more for the rest (scheduling)
what do I do now :x
I should buy a xeon or something...

does anyone of you guys have a ryzen AM4, could anyone test if the performance of nnedi per core is good ?

feisty2

31st January 2018, 05:02

what do I do now :x
I should buy a xeon or something...

if u do plan to spend some money, get a GPU, 1080Ti or something, most image/video processing algorithms are like a zillion times faster on gpu, primitive algorithm like nnedi3 could even run faster than real time on GPUs far from nearly as fast as 1080Ti, and a lot of filters are moving on to GPU implementations, like
knlmeanscl (https://forum.doom9.org/showthread.php?t=171379)
nnedi3cl (https://forum.doom9.org/showthread.php?t=174902)
eedi3cl (https://forum.doom9.org/showthread.php?t=174845)
and more

monohouse

31st January 2018, 05:09

I whant faster than real time :) but there is some problems...
I tested knl-CL, the problem with him is that he make stains in the video :x
I don't think I can do vaporsynth because MCTD and daa3 not available on vaporsynth :x and I think neither is LSFmod (but that I could give up and find something else)
I also tested nnedi3ocl from 2013 - his output is very bad with -2 (which daa3 is using) (I tested him on both nvidea and radeon) :x

leeeeeel: i7-6700k at 4.0 GHz vs GTX 1060: 1060 2x more fast

if only the original didee daa3 was available on vaporsynth, I could do the nnedi3/deinterlacing on vaporsynth separate from avisynth and then combine both - mybe even on the same command line if there is the tools

daa3 looks like this:
nn = c.nnedi3(field=-2,nsize=size,nns=neurons,qual=2,pscrn=0,fapprox=0,threads=nthreads)
dbl = mt_average(selecteven(nn),selectodd(nn),U=3,V=3)

I wonder if it's possible to have the nnedi3 process done in vaporsynth and the clip output to avisynth if the selecteven(nn)/selectodd(nn) functions will work from an external clip...

Aktan

31st January 2018, 05:49

Hello there! I'm just curious if this could be explained. I am on a Ryzen 1700X overclocked to 3.8 Ghz. I am running Avisynth+ Revision 2574-MT 64-bit. I use the following line in the AVS:

AVISource("deint.avi")
nnedi3_rpow2(rfactor=4, cshift="lanczosresize", fwidth=2880, fheight=1920, ep0=2, nsize=0, nns=4, qual=2, pscrn=4, opt=0)

It should be noted that the original resolution is 720x480 and that it's just a clip I capture from my capture card of a VHS tape. I also know that Ryzen support of AVX2 is emulated since there is no 256bit register in Ryzen. I've read that it will be half the speed of Intel, but I didn't think it be this bad. Anyway, testing out opt 0 - 7 gave the following results:

opt FPS
0 0.56
1 5.89
2 14.6
3 14.48
4 14.7
5 0.57
6 0.57
7 0.56

Can Ryzen AVX2 be that bad? I also noticed opt 0, 5-7 uses all 16 logical cores while opt 1-4 barely hits 50% while being so much faster. I am using Release_Intel_W7_Core2_AVX2 version of the DLL.

Edit: FMA4 is not officially support, but varies people said it still works on Ryzen with bugs, I tested it for fun.

monohouse

31st January 2018, 05:57

I suspect that you need this: https://github.com/jimenezrick/patch-AuthenticAMD
it a program that patch intel-compiled binary that use intel cpu detection to detect an AMD cpu

jpsdr

31st January 2018, 10:24

@Aktan
It seems at "0" it detects AVX2 (same has 5), but unfortunately results are not good, but ten time slower than "only C"... Ouuah !!!
At one moment, you'll see why there is a price differences.
After, on Intel CPU, there is not so much differences between AVX and AVX2, so, the speed gain can't worth the price difference for a lot of people.
Set manualy opt to 4 (AVX).
Issue is that if you stick to CPU possibility, you're screwed with AVX2, and on some program may have bad surprises.
Probably some program like x264 are a little clever, and detect Intel/AMD, and don't go over AVX on AMD, because they know AVX2 is slow as hell.
What should be done, is to detect if CPU is AMD or Intel, and don't allow AVX2 on AMD on automatic detection, but i don't know how to detect AMD with the parameters avaible on avs...

Aktan

31st January 2018, 14:52

I suspect that you need this: https://github.com/jimenezrick/patch-AuthenticAMD
it a program that patch intel-compiled binary that use intel cpu detection to detect an AMD cpu

Thanks for the link, though it doesn't seem easy to compile, lol.

@Aktan
It seems at "0" it detects AVX2 (same has 5), but unfortunately results are not good, but ten time slower than "only C"... Ouuah !!!
At one moment, you'll see why there is a price differences.
After, on Intel CPU, there is not so much differences between AVX and AVX2, so, the speed gain can't worth the price difference for a lot of people.
Set manualy opt to 4 (AVX).
Issue is that if you stick to CPU possibility, you're screwed with AVX2, and on some program may have bad surprises.
Probably some program like x264 are a little clever, and detect Intel/AMD, and don't go over AVX on AMD, because they know AVX2 is slow as hell.
What should be done, is to detect if CPU is AMD or Intel, and don't allow AVX2 on AMD on automatic detection, but i don't know how to detect AMD with the parameters avaible on avs...

I'm still surprise it is that much slower. So you don't think it is something with the compiler, instead it really is that slow on AMD? I should mention doing a longer test, on AVX it does hit near 100% sometimes and the FPS does go down to 8 FPS sometimes. The beginning frames were basically a blank blue screen (no signal screen) and AVX2 is still that slow.

TheFluff

31st January 2018, 15:20

AVX2 shouldn't be that slow on Ryzen, I don't think. There's probably something weird going on.

x264 has a "slow SSE2" flag for Athlon 64/K8, but no "slow AVX2" equivalent.

Aktan

31st January 2018, 15:26

I was doing some research on benchmarks for Ryzen and AVX2, and it seems in general while not as fast as Intel, AVX2 should still be faster than AVX. For example I found this:

http://www.numberworld.org/y-cruncher/internals/arch-optimizations.html

I do admit they manually tune though. I think it is still related to Intel compiler. I wonder how hard is it to compile the DLL on VS. I'll try it out later today.