4K / UHD and Avisnyth

[Hotte]

Whenever I feed Avisynth with 4K footage (3.840 x 2.160) I am running into troubles with strange crashes when using a bit more complex filters like MVTools.

Often it starts to convert and produces avis output but then would not be able to pass the famous 101-frame barrier.

Before diving into details let me just know if general issues are to be expected with good 'ol and beloved Avisynth when it comes to 4K due to its architecture, memory limitations or whatever. Or if it makes sense to have a closer look if there is another issue.

Thanks

[Groucho2004]

It's quite possible that you run out of memory with 4K clips. The obvious solution is to use 64 bit Avisynth+.

However, in order to gather more info:
1. Post a script that gives you trouble
2. Run "AVSMeter avsinfo -log" and post the log file

the famous 101-frame barrier
Never heard of it.

[FranceBB]

If you wanna stick with 32bit Avisynth, you are definitely gonna use MPPipeline.

https://forum.doom9.org/showthread.php?t=163281

Using MPPipeline, you can "split" different blocks of code into a different process.
Each process can then allocate up to 2 GB of RAM.
This pretty much solves the memory issue on 32bit Avisynth.
The only downside is that it doesn't support audio.

This is an example:



MP_Pipeline("""

FFVideoSource("file1.mxf")

### platform: win32
### ###

tdeint(mode=2, order=-1, field=-1, mthreshL=6, mthreshC=6, map=0, type=2, debug=false, mtnmode=1, sharp=true, cthresh=6, blockx=16, blocky=16, chroma=true, MI=64, tryWeave=true, link=1, denoise=true, slow=2, opt=4)

### platform: win32
### ###

Converttoyv12(interlaced=false, chromaresample="Spline64", matrix="Rec709")

### platform: win32
### ###

tweak(sat=1.53 , dither=true)

### platform: win32
### ###

dfttest(sigma=64, tbsize=1, lsb_in=false, lsb=false, Y=true, U=true, V=true, opt=3, dither=0)

### platform: win32
### ###

Spline64ResizeMT(3840, 2160, threads=4, logicalCores=true, MaxPhysCore=true, SetAffinity=true)

### platform: win32
### ###

f3kdb(range=15, Y=45, Cb=30, Cr=30, grainY=0, grainC=0, sample_mode=2, blur_first=true, dynamic_grain=false, opt=3, mt=true, keep_tv_range=true, input_depth=8, output_depth=8)

### platform: win32
### ###

TextSub("sample.ass")

### platform: win32
### ###
""")


This is an example with a 1920x1080p input that is indexed, deinterlaced, converted to 4:2:0, tweaked, denoised, upscaled, debanded and hardsubbed.
Each step creates a different process.

1) FFVideo -> Process0, can allocate up to 2 GB of RAM.
2) tdeint -> Process1, can allocate up to 2 GB of RAM.
3) Converttoyv12 -> Process2, can allocate up to 2 GB of RAM.
4) tweak -> Process3, can allocate up to 2 GB of RAM.
5) dfttest -> Process4, can allocate up to 2 GB of RAM.
6) Spline64Resize -> Process5, can allocate up to 2 GB of RAM.
7) f3kdb -> Process6, can allocate up to 2 GB of RAM.
8) TextSub -> Process7, can allocate up to 2 GB of RAM.

If you used normal Avisynth, you only had 2 GB of RAM available to use, but with MPPipeline you have 16 GB of RAM to use in Avisynth.
Please note that you won't need so many processes in real-life usage, even if you encode in 4K and this is just an example to show you how to use MPP.

I hope it solves your problem, as it solved mine years ago.

Cheers.

[Hotte]

Thanks Groucho. I went through AVSMeter withe the script below. It used to get stuck in VirtualDub at frame Nr. 51 and never returned. With AVSMeter it seem to have no problem.

I am currently running a more complex script (0.2 fps) but it seems, that it also runs fine with AVSMeter while it crashes in Virtualdub.

I suspect that the problem lies within either VDub or the "Video for Windows" interface I am using in VDub to direct AVI-Output to my Canopus HQX Intermediate Codec (which is also Input). This Encoder runs fine with any footage. But maybe the vfw-interface....

Any ideas ?

I will send the log in 2nd message due to length restrictions ins the forum.

[Hotte]

How do you find the memory usage or the other parameters in the log - is there anything threatening ?

[QUOTE]Log file created with: AVSMeter 2.8.0 (x86)
Script file: Z:\Temp\4KTEst.avs

[OS/Hardware info]
Operating system: Windows 10 (x64) (Build 15063)
CPU brand string: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
CPU features: MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX, AVX2, FMA3, MOVBE, POPCNT, AES, F16C


[Avisynth info]
VersionString: AviSynth 2.60, build:Mar 31 2015 [16:38:54]
VersionNumber: 2.60
File / Product version: 2.6.0.6 / 2.6.0.6
Interface Version: 6
Multi-threading support: No
Avisynth.dll location: C:\WINDOWS\SysWOW64\avisynth.dll
Avisynth.dll time stamp: 2015-03-31, 06:40:58 (UTC)
PluginDir2_5 (HKLM, x86): C:\Program Files (x86)\AviSynth\plugins


[Clip info]
Number of frames: 144
Length (hh:mm:ss.ms): 00:00:05.760
Frame width: 3840
Frame height: 2160
Framerate: 25.000 (25/1)
Colorspace: i420
Audio channels: 2
Audio bits/sample: 32 (Float)
Audio sample rate: 48000
Audio samples: 276480


[Runtime info]
Frames processed: 144 (0 - 143)
FPS (min | max | average): 1.654 | 14.31 | 6.961
Memory usage (phys | virt): 827 | 844 MiB
Thread count: 20
CPU usage (average): 46%

Time (elapsed): 00:00:20.688


[Script]
PluginPath = "C:\Portables\AvisynthPlugins"
LoadPlugin(PluginPath+"svpflow1.dll")
LoadPlugin(PluginPath+"svpflow2.dll")
LoadPlugin(PluginPath+"TimeLapseDF.dll")
LoadPlugin(PluginPath+"FFT3DFilter.dll")
LoadPlugin(PluginPath+"variableblur.dll")
LoadPlugin(PluginPath+"dither.dll")
LoadPlugin(PluginPath+"RgTools.dll")
LoadPlugin(PluginPath+"undot.dll")
LoadPlugin(PluginPath+"nnedi3.dll")
LoadPlugin(PluginPath+"masktools.dll")
LoadPlugin(PluginPath+"masktools2.dll")
LoadPlugin(PluginPath+"deen.dll")
LoadPlugin(PluginPath+"DGDecode.dll")
LoadPlugin(PluginPath+"MVTools2.dll")
LoadPlugin(PluginPath+"LSMASHSource.dll")
LoadPlugin(PluginPath+"ExactDedup.dll")
LoadPlugin(PluginPath+"TWriteAVI.dll")
LoadPlugin(PluginPath+"RT_Stats26.dll")


v = LSMASHVideoSource("F:\Video\Pool\4K Panasonic G81\2017-07-02\P1100325.MP4")
a = LSMASHAudioSource("F:\Video\Pool\4K Panasonic G81\2017-07-02\P1100325.MP4")
clp = AudioDub(v,a)
clp = Trim(clp,81,224)


# Filter: MD1
clp = ConvertToYV12(clp)
super = MSuper(clp, pel=2, sharp=1)
bv1 = MAnalyse(super, isb = true, delta = 1, blksize=16, overlap=8)
fv1 = MAnalyse(super, isb = false, delta = 1, blksize=16, overlap=8)
clp = MDegrain1(clp,super,bv1,fv1,thSAD=400)
return(clp)

[Hotte]

[Performance data]
Frame Frames/sec Time/frame(ms) CPU(%) Threads PhysMEM(MiB) VirtMEM(MiB)
1 1.654 604.642014 57 13 470 486
2 2.432 411.102693 60 13 494 510
3 5.460 183.138922 53 13 627 644
4 7.095 140.935267 44 13 688 705
5 6.962 143.644720 42 13 762 779
6 6.427 155.600642 64 13 825 842
7 7.240 138.121706 43 13 825 842
8 7.085 141.145179 35 13 825 842
9 6.827 146.485589 38 13 825 842
10 7.377 135.551459 56 13 825 842
11 7.470 133.874051 48 13 825 842
12 6.926 144.392725 42 13 825 842
13 7.404 135.063115 50 13 825 842
14 7.419 134.786878 42 13 825 842
15 6.944 144.007748 51 13 824 841
16 7.430 134.597927 33 13 824 841
17 7.366 135.752360 34 13 824 841
18 6.733 148.521146 48 13 827 844
19 7.373 135.638745 45 13 827 844
20 7.455 134.142248 55 13 827 844
21 6.938 144.130384 44 13 827 844
22 7.507 133.212356 46 13 827 844
23 7.499 133.355717 45 13 827 844
24 7.060 141.634509 48 13 827 844
25 7.638 130.927374 42 13 827 844
26 7.623 131.184105 47 13 827 844
27 6.991 143.043479 41 13 826 843
28 7.441 134.396783 45 13 826 843
29 7.366 135.754310 56 13 826 843
30 6.883 145.276048 38 13 827 844
31 7.351 136.038105 46 20 827 844
32 7.108 140.682921 43 20 827 844
33 7.033 142.184539 45 20 827 844
34 7.672 130.342230 58 20 827 844
35 7.601 131.566394 49 20 827 844
36 7.056 141.724472 55 20 827 844
37 7.555 132.355125 36 20 827 844
38 7.490 133.516138 45 20 827 844
39 6.994 142.970097 44 20 826 843
40 7.429 134.600605 45 20 826 843
41 7.419 134.797361 45 20 826 843
42 6.648 150.427011 53 20 827 844
43 7.420 134.770784 35 20 827 844
44 7.447 134.287798 46 20 827 844
45 6.731 148.563326 44 20 827 844
46 7.384 135.431751 48 20 827 844
47 7.353 135.993732 34 20 827 844
48 6.899 144.949582 47 20 827 844
49 7.291 137.160853 35 20 827 844
50 7.299 137.003104 42 20 827 844
51 6.743 148.301963 47 20 826 843
52 7.030 142.238910 52 20 826 843
53 7.283 137.300068 43 20 826 843
54 6.719 148.833956 48 20 827 844
55 7.373 135.631920 55 20 827 844
56 7.324 136.531822 41 20 827 844
57 7.011 142.638755 55 20 827 844
58 7.339 136.254606 45 20 827 844
59 7.390 135.319109 45 20 827 844
60 6.783 147.434492 45 20 827 844
61 7.299 137.010417 55 20 827 844
62 7.434 134.509905 46 20 827 844
63 7.038 142.094819 40 20 826 843
64 7.414 134.879280 51 20 826 843
65 7.312 136.755641 53 20 826 843
66 7.341 136.223885 50 20 827 844
67 7.180 139.275899 33 20 827 844
68 7.494 133.432029 46 20 827 844
69 6.739 148.397543 44 20 827 844
70 7.598 131.615159 46 20 827 844
71 7.516 133.047050 45 20 827 844
72 6.845 146.096704 44 20 827 844
73 7.458 134.086413 34 20 827 844
74 7.427 134.641083 41 20 827 844
75 7.262 137.706981 48 20 826 843
76 7.408 134.989479 47 20 826 843
77 7.383 135.450278 45 20 826 843
78 6.782 147.445946 44 20 827 844
79 7.301 136.974335 45 20 827 844
80 7.295 137.078685 46 20 827 844
81 6.672 149.871122 37 20 827 844
82 7.295 137.087216 54 20 827 844
83 7.314 136.720773 43 20 827 844
84 6.890 145.141711 53 20 827 844
85 7.281 137.350533 45 20 827 844
86 7.296 137.055527 43 20 827 844
87 6.606 151.376892 42 20 826 843
88 7.275 137.460494 44 20 826 843
89 7.272 137.517542 42 20 826 843
90 6.783 147.433269 47 20 827 844
91 7.409 134.966318 36 20 827 844
92 7.369 135.701165 40 20 827 844
93 6.822 146.592376 37 20 827 844
94 7.290 137.171816 45 20 827 844
95 7.358 135.898406 49 20 827 844
96 7.309 136.825611 50 20 827 844
97 7.364 135.797709 34 20 827 844
98 7.339 136.260219 45 20 827 844
99 6.964 143.594490 37 20 826 843
100 7.281 137.349310 52 20 826 843
101 7.329 136.445514 53 20 826 843
102 6.842 146.154251 48 20 827 844
103 7.449 134.245135 44 20 827 844
104 7.403 135.073110 41 20 827 844
105 6.532 153.087214 47 20 827 844
106 7.377 135.552195 45 20 827 844
107 7.339 136.266797 43 20 827 844
108 6.685 149.593914 47 20 827 844
109 7.376 135.574383 43 20 827 844
110 7.381 135.476126 44 20 827 844
111 6.874 145.469629 52 20 826 843
112 7.289 137.191082 44 20 826 843
113 7.260 137.738437 54 20 826 843
114 6.724 148.716444 37 20 827 844
115 7.380 135.492949 46 20 827 844
116 7.395 135.221099 56 20 827 844
117 6.944 144.015551 44 20 827 844
118 7.466 133.946466 46 20 827 844
119 7.501 133.316458 45 20 827 844
120 7.134 140.170437 54 20 827 844
121 7.582 131.887742 44 20 827 844
122 7.641 130.879828 43 20 827 844
123 7.105 140.748504 47 20 826 843
124 7.470 133.877221 43 20 826 843
125 7.277 137.416116 47 20 826 843
126 7.001 142.846731 46 20 827 844
127 7.498 133.375707 49 20 827 844
128 7.449 134.238304 46 20 827 844
129 7.071 141.419954 49 20 827 844
130 7.553 132.394129 44 20 827 844
131 7.595 131.672699 49 20 827 844
132 6.896 145.019555 52 20 827 844
133 7.523 132.920273 55 20 827 844
134 7.520 132.978302 46 20 827 844
135 6.930 144.306900 40 20 826 843
136 7.333 136.368714 53 20 826 843
137 7.340 136.231690 42 20 826 843
138 6.780 147.496417 48 20 827 844
139 7.412 134.919508 38 20 827 844
140 7.388 135.348610 43 20 827 844
141 6.794 147.190439 45 20 827 844
142 7.126 140.334028 50 20 827 844
143 7.537 132.681098 45 20 827 844
144 14.308 69.891633 53 20 827 844

[Groucho2004]

Memory usage is moderate, I don't think it's the problem.
First you should update to VirtualDub2 (https://forum.doom9.org/showthread.php?t=172021). Then try a different encoder (x264, for example) and check if this crashes too. It's possible that the encoder uses huge amounts of memory at this resolution and therefore crashes within a 32 bit process. You have to consider that both Avisynth and VDub operate within the same 32 bit process which can "only" allocate 4Gb of memory (max).

If there is a 64 bit version of the Canopus encoder I would definitely go for 64 bit Avisynth/VDub.

[Hotte]

Thanks, FranceBB. Very interesting idea!
I will keep it in mind.

First I will try to go for VDub2 an the other hints that Groucho gave me.

I might end up having to use 64bit Avisynth, but I heard, that not all the filters I need are avilable for Avi64. I need to check that out.

So if that is the case I will try MPPipeline.

Thank you so far.

[Hotte]

Groucho,
I just tried with VirtualDub2 at first stage with the script above. Results
- Stops prcoessing at FrameNr. 51 as above. Memory usage according to Task Manager is only 1,022 GB
- Trying output X.264 8-bit or 10-bit as output: Error Cannot start video-compression Error-code -100 - any ideas ?

Interesting: Starting another, more complex script and moving to another frame within VDub preview shows a memory error in the status bar of Vdub2.
Also in some error messages the red error symbol would not be visible. This points pretty much to memory issues.
However, I do not understand how memory issues can arise when taskmanager shows only 1,2 GByte of memory usage for VDub2...thought we´ve got up to 3GB or more

I will now try to go for 64-bit Avisynth and VDub2. Hopefully all my filters needed are available for 64-bit. Seems like it is hard way to go....

[Groucho2004]

I just tried with VirtualDub2 at first stage with the script above. Results
- Stops prcoessing at FrameNr. 51 as above. Memory usage according to Task Manager is only 1,022 GB
- Trying output X.264 8-bit or 10-bit as output: Error Cannot start video-compression Error-code -100 - any ideas ?
No idea. Maybe you should post this in the VDub2 thread.

[Hotte]

Groucho, I have started to migrate to Avisynth+ and VDub2.

Managed to run the script above successfully!!!

It was definitely a memory problem: Memory usage in taskmanager was around 1 GB in the 64-bit environment until it came to frame 51, where conversion paused while memory usage went up to around 3 GB before it continued to convert until the end of the file. Thank you so much!

Now I have started to migrate filter by filter, but I get stuck with "variableblur": Cannot load.

- The 64-bit variableblur.dll is in ...avisynth+/plugins64+ folder
- libfftw3-3.dll and libfftw3f-3.dll and libfftw3-3.dll 64-Bit versions are in C:\Windows\SysWOW64 directory
- Path-Variable points to ...avisynth+/plugins64+ and C:\Windows\SysWOW64
- Microsoft Visual C++ 2010 SP1 Redistributable Package (x86) updated. There was no explicit 64-Bit installation, I just did Standard vom Microsoft-Support-Page

Still it say "Cannot load file ...avisynth+/plugins64+/variableblur.dll

Any idea what I could do ?

[StainlessS]

SysWOW64 is for 32 bit plugs, (WOW64 == [32 bit] Windows on Windows 64 bit).
System32 is for 64 bit (yeh, I know, M$ is very mystical).

Runtimes, see here:- EDIT: LINK REMOVED

EDIT: OOps, above is Avisynth Universal Installer, I dont know why but I thought it was
Groucho2004 link to All In One Runtimes Installer.
Shall post link when I find it.

EDIT: Here, latest All-In-One package [2018/July/11] runtimes repack from RePacks.net:- https://repacks.net/viewtopic.php?f=6&t=247
Seems our very own Burfadel played some part in creation of the pack. [Guessin' that there are not too many Burfadels running around loose]

EDIT: XP users see first page before install of runtime pack.
EDIT: The Download link on first page brings up a text file on PasteBin where there will be about 5 file host links
to download from including MediaFire (The one I chose).

EDIT: Hotte, VS runtimes for eg 2017 do not include functionality for VS 2008 runtimes, you need them ALL, the All-In-One package is just the job (x86 & x64).

[Groucho2004]

I use this redist AIO:
https://repacks.net/viewtopic.php?f=6&t=237
Includes an automatic uninstaller of outdated runtimes that also works on XP32/64.

In order to check if all files are in the right place and if all dependencies are installed:
"AVSMeter avsinfo".

[StainlessS]

God bless you oh great and mighty venerable Groucho, methinks that I will install the most recent,
and then go a second time with the slightly older XP friendlier version you posted.
that should in 1st pass install most recent stuff, and in second remove any older un-necessary, methinks.

Thanks so very very much for being Groucho2004 :)

[sneaker_ger]

You have to consider that both Avisynth and VDub operate within the same 32 bit process which can "only" allocate 4Gb of memory (max).
The official/vanilla VirtualDub build isn't even Large Address Aware, so it's 2 GB. Guess it's different for VirtualDub2 builds?

[Groucho2004]

The official/vanilla VirtualDub build isn't even Large Address Aware, so it's 2 GB. Guess it's different for VirtualDub2 builds?Just checked with dumpbin, VDub2 is not LAA.

[sneaker_ger]

Ah, he switched Vdub2 64 bit. My bad.

[Hotte]

Thank you very much Stainless & Groucho.
I did evrything you suggested:

- https://repacks.net/viewtopic.php?f=6&t=247 (StainlessS)
- https://repacks.net/viewtopic.php?f=6&t=237 (Groucho)
- Reloaded FFTW3 32, 64 bit into SysWOW and into System32
- Ran AVSMeter (see log below)

Whatever I tried - it does not help. It refuses to load variableblur.dll. That really drives me crazy. Can you run 64-bit variableblur.dll on your systems ?

Any ideas left ?

EDIT: Just to add, my plugins-folder is: C:\Program Files (x86)\AviSynth+\plugins64+
AVSMeter tells a different story - strange.

AVSMeter 2.8.0 (x86) - Copyright (c) 2012-2018, Groucho2004

VersionString: AviSynth 2.60, build:Mar 31 2015 [16:38:54]
VersionNumber: 2.60
File / Product version: 2.6.0.6 / 2.6.0.6
Interface Version: 6
Multi-threading support: No
Avisynth.dll location: C:\WINDOWS\SysWOW64\avisynth.dll
Avisynth.dll time stamp: 2015-03-31, 06:40:58 (UTC)
PluginDir2_5 (HKLM, x86): C:\Program Files (x86)\AviSynth\plugins


[CPP 2.6 Plugins (32 Bit)]
C:\Program Files (x86)\AviSynth\plugins\DirectShowSource.dll [2.6.0.3]
C:\Program Files (x86)\AviSynth\plugins\TCPDeliver.dll [2.6.0.7]

[Scripts (AVSI)]
C:\Program Files (x86)\AviSynth\plugins\colors_rgb.avsi [2015-03-30]

[Groucho2004]

Thank you very much Stainless & Groucho.
I did evrything you suggested:

- https://repacks.net/viewtopic.php?f=6&t=247 (StainlessS)
- https://repacks.net/viewtopic.php?f=6&t=237 (Groucho)
- Reloaded FFTW3 32, 64 bit into SysWOW and into System32
- Ran AVSMeter (see log below)

Whatever I tried - it does not help. It refuses to load variableblur.dll. That really drives me crazy. Can you run 64-bit variableblur.dll on your systems ?

1. Update to AVSMeter 2.8.1

2. You have to use AVSMeter64 in order to check a 64 bit workflow

[Hotte]

Stupid me! AVSMeter64.exe of course.

AVSMeter281 is a great tool - it shows the issue very clearly:

Plugin errors/warnings]
Error loading "C:\Program Files (x86)\AviSynth+\plugins64+\variableblur.dll"
Cannot load 32 bit DLL with 64 bit Avisynth

...Although it is listed here:
http://avisynth.nl/index.php/AviSynth%2B_x64_plugins
... it is 32bit.

Could not find an x64 Version in another place...sad, so sad

[Groucho2004]

Cannot load 32 bit DLL with 64 bit Avisynth

...Although it is listed here:
http://avisynth.nl/index.php/AviSynth%2B_x64_plugins
... it is 32bit. You're right, I also checked. :(

[Groucho2004]

Found it:
http://www.mediafire.com/file/0z0hl43za6bwlb4/VariableBlur05_x64.7z

[Hotte]

Works now with this download.

You guys are really fantastic!

The full story is: You need 64-bit libfftw3f-3.dll in C:\Windows\System32

:thanks: :thanks:

[StainlessS]

Yep, everything has to be either 32, OR, 64 bit. (and never the twain shall meet).