FFT3DFilter 2.6 [Archive] - Page 3

MeteorRain

27th March 2020, 21:29

WinXP has been EOL for long, so I'm not going to keep a copy myself to install it. I have no problem if someone would point out a fix for any potential issue and I can just merge or patch it.

On the other hand, the old filter still works despite some small issues which I pointed out and can be fixed quickly. If XP users would like to use it, they can simply use the original one.

Again, I won't deliberately remove XP support to make it unusable. Its unsupported in the sense of I'm not going to test it during my development.

Also you may notice that the AVX that I'm planning to add is not supported in XP anyway.

StainlessS

27th March 2020, 21:59

Again, I won't deliberately remove XP support to make it unusable. Its unsupported in the sense of I'm not going to test it during my development.
Understand you position entirely, thanks very much, [and thanks for not being like that XP hatin' Myrsloik, hangin's too good for that sort :) ]

FranceBB

27th March 2020, 22:00

WinXP has been EOL for long
Well, updates were kept up and running for Windows XP Embedded and all the other version like POSReady etc 'till April 2019 and were further extended by Microsoft 'till June 2019 when they dropped support for everyone but companies that were willing to pay 15'000 bucks every year for the Premium Extended Support. I really wished they offered an option to pay for them for private users as well, 'cause I would have definitely bought the Extended Support past July 2019 if it had an affordable price like $50 a year...

I'm not going to keep a copy myself to install it
Fair enough.

I have no problem if someone would point out a fix for any potential issue and I can just merge or patch it.
Oh, that's great. Thank you. :D

I won't deliberately remove XP support to make it unusable. Its unsupported in the sense of I'm not going to test it during my development.
Also you may notice that the AVX that I'm planning to add is not supported in XP anyway.
Ok, that's fine, I guess. Beside, I'm pretty sure there will be people that will test it on their own XP machines for you once you'll release it and they'll eventually point out errors, so it's fine. :)
AVX won't work of course, but I don't think you'll remove SSE code either, so I think XP users are gonna be fine.
The idea would be to have the filter to check for the CPU instructions and use the available assembly optimizations or eventually make them user-available with an opt string parameter; that would work as well.
The only think I'm asking, though, in all this, since it has to be compiled, is to compile it two times (just like Ferenc does): one with whatever you want (either visual studio or Intel Parallel Studio o Clang, whatever) for modern OS and the other one with Visual Studio 2019 targeting v141_xp. If it's not too effort, please just make sure that it does compile successfully. If it does, then it's probably going to run and it's gonna be fine for us.

Sorry for bothering you with this, but I really hope you can understand. :)

Thank you in advance,
Frank.

Side note:

Understand you position entirely, thanks very much, [and thanks for not being like that XP hatin' Myrsloik, hangin's too good for that sort :) ]

Yeah... Myrsloik... I remember when I asked for ffms2 support and other people did too and he then updated the initial post with

TRIGGER WARNING

The posts contained in this thread may be very upsetting for users of old operating systems, old cpus and stupid people

And I was like "Oh, c'mon..." :(

videoh

27th March 2020, 22:26

You're a hero for me, Frank, but I agree with Myrsloik. For what it's worth, LOL. We should ask Danila. ;)

Stereodude

27th March 2020, 22:35

...We should ask Danila. ;)
No we need to get him and Katie into the same thread and watch the positive feedback loop spiral out of control. :devil:

MeteorRain

27th March 2020, 23:04

FranceBB

28th March 2020, 00:38

FranceBB, I'm not particularly sure if anyone would want to continue to use WinXP for this particular work. For embedded environment I can understand that, but for daily use I can't. Win7 however, is much more reasonable to use as a daily drive. But still even Win7 may lack a few modern syscalls (APIs) and could potentially make things harder.

Yes, I know, which is why I have Fedora (Linux) on my computer and a Windows 10 partition for work-related stuff as well as physical workstations with Win10 at work, however... I do keep a few VMs on my PC: one of them is with my original installation of Windows98SE 'cause it's been my very first operating system, the one I learned to format and re-install and I kinda got attached to it. I never use it, but, once a year, when it's its "birthday", I fire it up, I look at it, I open a few old programs to check that they're still working, I think about the good old days and then I shut it down for the rest of the year 'till the next year comes.
For XP however, I do have a partition with it and I've been part of a group called "Windows XP Forever" for quite some time (and I still am). We're the ones working at the "One Core API (https://github.com/Skulltrail192/One-Core-Api)" together with Samuel (SkullTrail), Dibya, Peter, Henry, Svyatpro, and so on. One Core API (https://github.com/Skulltrail192/One-Core-Api) is basically a set of APIs which have been backported from newer OS to Windows XP. We do it on our spare time and just for fun as we all love our beloved OS, but of course we all have Win10 as well.
If anyone is curious: GitHub Development Link (https://github.com/Skulltrail192/One-Core-Api) - GitHub Binaries Link (https://github.com/Skulltrail192/One-Core-API-Binaries) - Website (outdated) (http://shorthornproject.com/)

If you were wondering what does FranceBB do in his spare time (beside encoding), well... this is pretty much it xD

I think that I am, in some way, for some aspects, a person who's fascinated about the new technologies but also, at the same time, the younger version of Manolito hahahahahaha

No we need to get him and Katie into the same thread and watch the positive feedback loop spiral out of control. :devil:

lol... Out of curiosity, but... do you have the same impression that we're all dudes except for Katie? Like... is Katie Boundary the only woman in the whole Doom9? xD
Like... does encoding only attract dudes? xD

SSE code won't get removed of course. CPU flag based SIMD route selection is already there.

Perfect. :D

My concern is I'm not sure if the xp targeted compilers would complain anything. I may give it a try very soon.

Well, I guess we'll see soon, then. :)

Stereodude

28th March 2020, 00:55

FranceBB, I'm not particularly sure if anyone would want to continue to use WinXP for this particular work.
They're just very patient and aren't tempted by the idea of running hardware that's 100x faster.

lol... Out of curiosity, but... do you have the same impression that we're all dudes except for Katie? Like... is Katie Boundary the only woman in the whole Doom9? xD
Like... does encoding only attract dudes? xD
Yes, I expect this place has an extremely heavy bias toward individuals with a Y chromosome. I'm not sure that Katie is actually a girl, but that's gotta be the least significant thing about Katie's unique status on the forum.

Maybe we need to start an initiative?

#learntoencode / #girlswhoencode

videoh

28th March 2020, 03:35

the younger version of Manolito Not something to boast about! :p

So it's like collecting antiques. You'd never drive drywall screws manually, but the old tools sure are cool.

Stereodude

28th March 2020, 03:45

Not something to boast about! :p

So it's like collecting antiques. You'd never drive drywall screws manually, but the old tools sure are cool.
Drywall?!? They're still using lath and plaster! :sly:

videoh

28th March 2020, 04:05

Good point!

feisty2

28th March 2020, 04:16

the younger version of Manolito hahahahahaha

whos manolito?

Atlantis

28th March 2020, 04:43

Please, where is this Neo_FFT3D to try? I use bt=5 a lot and if there is speed improvements, it will be great.

MeteorRain

28th March 2020, 07:02

https://down.7086.in/AviSynthPlus%20Filters/neo-fft3d-alpha~20200328.zip

As I haven't finished all the work, this is considered only alpha / early dev version. Use at your risk.

neo_fft3d(..., y=3, u=3, v=3)

pinterf

28th March 2020, 07:22

XP does not support AVX so it is not even reported as valid processor flag in GetCPUFlags.

tormento

28th March 2020, 09:37

As I haven't finished all the work, this is considered only alpha / early dev version. Use at your risk.
Any difference in syntax and/or added switches?

MeteorRain

28th March 2020, 10:48

Yes, as I said,

neo_fft3d(..., y=3, u=3, v=3)

y u v being the plane to process, 3=process, 2=copy, 1=undefined content. Alpha channel, if there's any, will be copied regardless.
The old "plane=" and "multiplane=" options are removed.
All other options remain the same.

bt = 3 / 4 / 5 should give slightly different result due to rounding errors.
For bt=3, the constant 0.33333333333 will be replaced by 1/3 in current or future version.
For bt=4 / 5, an addition of a,b,c,d,e will be performed as (a+b)+(c+d)+e instead of (((a+b)+c)+d)+e.
In all cases, rounding error should be minimal to observe.

sl1pkn07

28th March 2020, 13:59

whos manolito?

https://forum.doom9.org/member.php?u=41597

StainlessS

28th March 2020, 14:35

whos manolito?
Bit of a cowboy really.
He got a sister Victoria who is married to a gringo big John Cannon,
and they live together on a fair size ranch with Victoria's stepson {Billy} Blue Boy.

Family Photo (from the right, clockwise):
https://i.postimg.cc/7ff1ZtJg/Mani-Famility-Photo.jpg (https://postimg.cc/7ff1ZtJg)

Manizorgsky, Pictures at An Exhibition:- https://www.google.com/search?q=high+chaparral&tbm=isch&source=iu&ictx=1&fir=qUEJu_9RsyTtjM%253A%252CBAAzhHNPWFEdmM%252C_&vet=1&usg=AI4_-kSAL9PCfwZ7nqTaUahXB3Ytp66KiQ&sa=X&ved=2ahUKEwiKrvLV_L3oAhULUhUIHbMcA00Q_h0wInoECAkQDQ#imgrc=qUEJu_9RsyTtjM:

MeteorRain

30th March 2020, 11:40

@Atlantis
Good news for you, Apply3D5 (the main function used by bt=5) is a few times faster than the pure C code.
I've measured Apply3D5 9.3 seconds of a 45 seconds test, down from 90 seconds of a 130 seconds test.

I also added multi-threading to it, so under MT prefetch 4 it's running event faster -- though up to 2x the speed due to locks.

hello_hello

30th March 2020, 14:11

They're just very patient and aren't tempted by the idea of running hardware that's 100x faster.

I doubt many people refrain from upgrading their computer because it won't run XP, or because there's no drivers for newer hardware, but I suspect there's many who don't see much point spending money to upgrade Windows on an old PC.

Do most household only have a single PC? There's four used regularly here, one of which is an old Pentium 3.

I've procrastinated replacing my PC because I can't bring myself to install Windows Spyware 10, and switching to Linux would require more effort and a learning curve, although as Linux Avisynth appears to be a thing now it's probably time, but this PC will keep running XP because I use some old Windows-only software, including a program that was abandoned after the upgrade for Windows 2000. There's a fair chance I'll still use this PC to encode video occasionally too.

Old PCs might be slow, but you can still encode faster using them alongside a new computer running the latest hardware than you can if you don't.

Stereodude

30th March 2020, 15:36

I doubt many people refrain from upgrading their computer because it won't run XP, or because there's no drivers for newer hardware, but I suspect there's many who don't see much point spending money to upgrade Windows on an old PC.

Do most household only have a single PC? There's four used regularly here, one of which is an old Pentium 3.

I've procrastinated replacing my PC because I can't bring myself to install Windows Spyware 10, and switching to Linux would require more effort and a learning curve, although as Linux Avisynth appears to be a thing now it's probably time, but this PC will keep running XP because I use some old Windows-only software, including a program that was abandoned after the upgrade for Windows 2000. There's a fair chance I'll still use this PC to encode video occasionally too.

Old PCs might be slow, but you can still encode faster using them alongside a new computer running the latest hardware than you can if you don't.
I have 5 PCs that are used "regularly" excluding servers in my house. Two are laptops. One has a 4th gen i5 the other has an 8th gen i7. Desktop 1 has a 2nd gen Xeon and desktop 2 is a Ryzen 9 3950X. My HTPC has a 4th gen i3.

4 of them run Windows 10, the HTPC runs Windows 8.1. Every P3, P4, & Core2 duo/quad system have long been put out to pasture.

With x265 the Ryzen 9 system is ~4x faster than the 2nd E5 Xeon. Everything else is much slower* (technically the 8th gen i7 laptop [Dell XPS 13] is about as fast as the 2nd E5 Xeon in x265 thanks to AVX2 and other core enhancements but it sits in the high 90C's encoding thermally throttling the whole time with the internal fans cranked and I don't think that's good for the system for hours on end). Sure, I could get slightly more throughput using all the systems or even including old retired systems collecting dust, but it doesn't make sense.

1) It's a pain to split encodes if you want to use a qpfile file and have full control of the encoding process. The more pieces the bigger the hassle. Do I want to be messing with getting yet another system encoding it's tiny part of the job for another 5% of encoding throughput? Nope...

2) Even if not splitting single encodes, power efficiency is a thing. Electricity isn't super expensive here and I'm not exactly a treehugger, but doubling my power usage to get another 5% of total encoding throughput by using an old system hardly seems prudent.

hello_hello

30th March 2020, 16:09

So you're the person using QPFiles. I figured someone must... but I didn't know who.... :)

I'm OCD about a lot of things, but my enthusiasm for driving myself insane hasn't extended to QPFiles. I did consider it.... as there's software for converting QPFiles from chapters etc, but if you remove the studio promos from the beginning of a movie, which I generally do, they become too much trouble for me. And of course it makes splitting scripts harder, which I do more than I'd like to. Usually to fix a problem I missed, such as cropping or because a word in the hard-coded subs wasn't spelled correctly....

Edit: thinking about it, I could always encode the studio promos and cut them out later if I was using a QPFile created from chapters, although I'd need to split the script or use a QPFile to make sure there's a keyframe in the splitting position.

Actually, I might need to rethink my entire position on QPFiles, now I'm talking about them. In order to edit the crap from the beginning or end of movies and because I'm lazy, I usually encode them separately after splitting the script to ensure there's a keyframe where one needs to be, then I append the split output, add the other streams and tell MKVToolNix to split the output at the same point, so it splits off the audio and subtitles etc too and I don't have to think about it. Instead of splitting the script to ensure there's a keyframe at the split point, I assume I could be using a QPFile. I'll give it a spin to see if it's quicker than copying the script and adding Trim() to the end. Although I would have to remember to add the QPFile to the command line, which I'm sure I'll regularly forget to do for at least the first 100 encodes :(.

Sharc

30th March 2020, 17:26

Interlaced source.

script 1:
AVIsource("source").ConverttoYV12(interlaced=true).AssumeTFF()
separatefields()
e=selecteven().fft3Dfilter(interlaced=false)
o=selectodd().fft3Dfilter(interlaced=false)
interleave(e,o)
weave()

script 2:
AVIsource("source").ConverttoYV12(interlaced=true).AssumeTFF()
fft3Dfilter(interlaced=true)

Shouldn't the 2 scripts produce identical interlaced outputs?
When separating the fields only 1 field is the same, the other field is different for the 2 scripts :confused:

Stereodude

30th March 2020, 18:05

So you're the person using QPFiles. I figured someone must... but I didn't know who.... :)
Well I used to re-encode for blu-ray which AFAIK required a keyframe where the chapter fell. Even with MKV it works nicer if there's a keyframe at the chapters on some devices.

I made a fairly simple spreadsheet to sort it all out when splitting. It suggests what frames to split the files (length / number of pieces). I open it in VD2 and use the closest scene change to the suggested number, enter those new frame numbers in a different row of the spreadsheet and then based on the original frame numbers of keyframes for chapter stops will tell me what frame number they need to be in each segment with only a little manual manipulation. It's then a copy paste fiesta to put the appropriate trim command into 8 avs files based on the spreadsheet and copy and paste to make 8 .chp files. It's not hard to do, but i'd estimate it still takes 20-30 minutes from start to finish.

MeteorRain

30th March 2020, 18:58

I wrote scripts to split AVS into multiple segments by converting chapter files, and then send them to different servers to encode, and later join the GOP files together.
Usually I only have to eac3to the chapters and prepare a universal avsi file to be imported, and all other works are for loop with scripts.

function p(string c, int fstart, int fend)
{
MP_Pipeline("""
lwlibavvideosource(""" + chr(34) + c + ".m2ts" + chr(34) + """, threads=1)

### prefetch: 32, 16
### lock threads to cores 4
### ###

QTGMC(Preset="fast")

### prefetch: 32, 16
### lock threads to cores 6
### ###

left = crop(0,0,976,0).MCTD().crop(0,0,960,0)
crop(944,0,0,0)

### export clip: left
### prefetch: 32, 16
### lock threads to cores 8
### ###

right = MCTD().crop(16,0,0,0)
stackhorizontal(left, right)

### prefetch: 32, 16
### lock threads to cores 10
### ###
""")

trim(fstart,fend)

f3kdb(output_depth=10,y=48,cb=48,cr=48,grainy=0,grainc=0)
}

import("mm.avsi")
p("01", 141600, 157957)

Hope this gives you some idea.

MeteorRain

30th March 2020, 23:11

I have uploaded the very first formal version here at GitHub (https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/releases).

141XP is included. msvc-x86 and -x64 are for daily drive. SSE and AVX routine is included for all core functions.

Let me know how it works for ya.

ChaosKing

30th March 2020, 23:47

I guess VS suport is still WIP? The function name is DelogoHD for VS :D

MeteorRain

31st March 2020, 00:24

Oops, that was a copy paste mistake. Please re-download and try.
FYI, I don't have VapourSynth environment here so I only made sure it compiles. If it's convenient for you to test on VS please let me know any issue.

ChaosKing

31st March 2020, 00:59

hmm, it seems like the filter does nothing. I even tried very high sigma values, but the image is unchanged: clip.neo_fft3d.FFT3D(sigma=90)

MeteorRain

31st March 2020, 01:32

I shall find a way to make planes default to [0,1,2]. Also reading some unfreed memory blocks, needs investigating.

Try v3.

ChaosKing

31st March 2020, 08:37

v3 clang x64 gets "stuck" in vsedit if more then 1 frame is requested simultaneously. If I hit the play button or benchmark (tested it also with blankclip just to be sure it's not the source)
msvc looks a bit better, it gets only stuck sometimes :D

Tested with R48 / R49 x64, win10, ryzen 2600

oh and sigma works now in v3

Edit:
core.std.SetMaxCPU("none") did not help

MeteorRain

31st March 2020, 09:02

SetMaxCPU changes the instruction set to VapourSynth, so I don't think that matters.

I tried clang x64 and msvc x64 with VirtualDub2, both look pretty normal. Does it only break on VSEdit? Mind pasting the full script? Are you on YUV420P8 or some more "advanced" pixel format?

ChaosKing

31st March 2020, 09:20

Same thing happens in vdub

import vapoursynth as vs
core = vs.core

clip = core.std.BlankClip(format=vs.YUV420P8)
clip = clip.neo_fft3d.FFT3D()
clip.set_output()

See here what it looks like in vdub: https://i.imgur.com/WBAYLy1.gifv

MeteorRain

31st March 2020, 10:07

Reproduced.

When I copy paste to create my VPY script it had set threads to 1 already, which is why I didn't reproduce it.

Multi-threading upstream request needs some special routine and I've fixed it accordingly (hopefully!).

Try v4.

ChaosKing

31st March 2020, 18:26

FranceBB

31st March 2020, 23:27

This is weird...
I tried the version from msvc-xp_x86 and Dependency Walker says that everything is fine (https://i.imgur.com/qCVtudI.png), however when I try to use the plugin, it prompts me to an "Access Violation".
Yet, when I run AVSMeter on my plugin folder, it doesn't complain at all, it just shows me neo-fft3d in my plugin+ folder.

FFVideoSource("test.mkv")
neo_FFT3D()

access violation

FFVideoSource("test.mkv")
neo_FFT3D(sigma=2.0, bt=3, y=3, u=3, v=3)

access violation

ColorBars(width=640, height=480, pixel_type="YV12")
neo_FFT3D(sigma=2.0, bt=3, y=3, u=3, v=3)

access violation

https://i.imgur.com/VyOq4VV.png

However... it works fine on Windows 10 (same PC, same hardware, same Avisynth installation, same fftw3.3.8, same AVS Script, same version of AVSPmod)...

The former version made by Ferenc Pinter, however, works like a charm on XP.
But I don't understand why... it should work, every dependency is included, the compiler compiled fine and didn't complain about anything and the CPU - although it supports 'till AVX2 - should be able to use the SSE/SSE2/SSSE3/SSE4.1/SSE4.2 on XP...

Is there a way to force SSE instead of AVX on Windows 10? Just to make sure that nothing is broken in the code and that it has really something (weird and deeply obscure) to deal with XP?

MeteorRain

31st March 2020, 23:55

If you can compile it youself, here's the compiling script: https://gist.github.com/msg7086/bb237a503d866d74c1cc8c2c49f90c0e

To remove AVX routine, comment this (https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/blob/master/src/functions.h#L184) out.

pinterf

1st April 2020, 07:06

It's probably not an AVX problem, for XP you have to build with v141_xp platform toolset (Visual Studio 2017 - Windows XP (v141_xp)) _and_ manually adding
/Zc:threadSafeInit-
to the additional options for C/C++ compiler

MeteorRain

1st April 2020, 08:09

Thanks, please try NeoFFT3D_r1v5-windows-xp.zip

FranceBB

1st April 2020, 11:15

If you can compile it youself, here's the compiling script: https://gist.github.com/msg7086/bb237a503d866d74c1cc8c2c49f90c0e

Ok, sure.

It's probably not an AVX problem, for XP you have to build with v141_xp platform toolset (Visual Studio 2017 - Windows XP (v141_xp)) _and_ manually adding
/Zc:threadSafeInit-
to the additional options for C/C++ compiler

Setting threadsafeinit in the compiler did the trick!
It's working flawlessly now on XP! :)

Thanks, please try NeoFFT3D_r1v5-windows-xp.zip

Your new build works too!

Thank you both! :D

Groucho2004

2nd April 2020, 00:04

@FranceBB
Time to update AVSMeter. 2.8.5 is 1.5 years old.

FranceBB

2nd April 2020, 00:57

@FranceBB
Time to update AVSMeter. 2.8.5 is 1.5 years old.

Oops.
How can I have missed that...?
Updated right now! :)
By the way, thank you for AVSMeter, I found it very useful on several occasions, especially when I was walking by some of my colleagues who had any kind of fucked up installation with double/triple plugins, random names etc.
It also helped me to keep my installation and plugin folder perfectly clean with no redundant things. ;)

MeteorRain

7th April 2020, 12:17

r2 -- marked as pre-release for now -- has been uploaded to GitHub.

I spend a few hours to completely rewrite the dual interface wrappers and it should be easier to work with now. I'm planning to move some of my filters into the new platform if I have time, including f3kdb (before fixing some raised issue).

Let me know how it works -- and if it is faster than it was. I experimentally used C++ Parallel STL in a few hot code paths and should have improved speed a bit.

ChaosKing

7th April 2020, 13:31

v4 is looking good
A quick speed test (x64) with a ntsc DVD, default values:
- neo fft clang ~ 71.8 fps
- neo fft msvc ~73.9 fps
- neo fft msvc-xp ~74.1 fps
- "old" fft ~ 67.4 fps

run multiple times 1500 frames in vsedit
With r2 -- marked as pre-release
clang 109.2 fps
msvc 113.3 fps

Everything is tested on a Ryzen 2600
std.BlankClip(format=vs.YUV420P8, width=1920, height=1080, length=500).neo_fft3d.FFT3D()

x64 r2-pre:
clang:Output 500 frames in 25.57 seconds (19.55 fps)
msvc: Output 500 frames in 24.22 seconds (20.64 fps)

x64 r1
msvc: Output 500 frames in 39.87 seconds (12.54 fps)

x64 original fft3dfilter
Output 500 frames in 42.38 seconds (11.80 fps)

Nearly x2 faster :eek:

And some warning to go :devil:
Core freed but 1866240000 bytes still allocated in framebuffers

In Avisynth r1 complains about the pixel_type for BlankClip() with a correct error msg, but r2 just throws an error: There is no function named neo_fft3d(). It works if I use a valid pixel type BlankClip(pixel_type = "YUV420P8")

EDIT: No wait, I mixed up the dlls: r2 always throws this error: There is no function named neo_fft3d(). I tried all x64+x86 versions