Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old Yesterday, 18:28   #2841  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,915
I'm back with some examples.
After the last discussion about SetMemoryMax() I did some more tests and this time I used a real script from a real use case scenario (although I trimmed the first 1000 frames only as I didn't want to let it filter 40 minutes worth of documentary for nothing).


Source: DNX HQX UHD 4:2:2 12bit 728 Mbit/s BT709 SDR

Video
ID : 2
Format : VC-3
Commercial name : DNxHR HQX
Format version : Version 3
Format profile : RI@HQX
Format settings, wrapping mode : Frame
Codec ID : 0D01030102110100-0401020271250000
Duration : 40 min 53 s
Bit rate mode : Constant
Bit rate : 728 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Active Format Description : Full frame 16:9 image
Frame rate : 25.000 FPS
Color space : YUV
Chroma subsampling : 4:2:2
Bit depth : 12 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 3.512
Stream size : 208 GiB (100%)
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709



AVS Script:


Code:
video=LWLibavVideoSource("PARTE 1.mxf")
FL=WAVSource("Parte 1 Mix OnAir.L.wav")
FR=WAVSource("Parte 1 Mix OnAir.R.wav")
CC=WAVSource("Parte 1 Mix OnAir.C.wav")
LFE=WAVSource("Parte 1 Mix OnAir.LFE.wav")
LS=WAVSource("Parte 1 Mix OnAir.Ls.wav")
RS=WAVSource("Parte 1 Mix OnAir.Rs.wav")

Dolby=MergeChannels(FL, FR, CC, LFE, LS, RS)

Stereo=WAVSource("Parte 1 MIX Lt - Rt.wav")

Mute=BlankClip(length=61335, fps=25, audio_rate=48000, channels=2)

audio=MergeChannels(Stereo, Mute, Dolby, Mute, Mute, Mute)

AudioDub(video, audio)


ConvertBits(16)

z_ConvertFormat(pixel_type="RGBP16", colorspace_op="709:709:709:limited=>rgb:709:709:full", resample_filter_uv="spline64", dither_type="error_diffusion", use_props=0)

Cube("C:\Program Files (x86)\AviSynth+\LUTs\5a_BT709_HLG_UPCONVERT_DISPLAY_mode.cube", fullrange=1, interp=1)

z_ConvertFormat(pixel_type="YUV422P10", colorspace_op="rgb:std-b67:2020:full=>2020:std-b67:2020:limited", resample_filter_uv="spline64", dither_type="error_diffusion", use_props=0)

Limiter(min_luma=64, max_luma=940, min_chroma=64, max_chroma=960)

trim(0, 1000)


AVS Meter Test:


Code:
AvsMeter64.exe "Test.avs"

pause


Hardware:

CPU: Intel Xeon Gold 6238R 2.20GHz x2 (56c/112th)
RAM: 32 x 4 DDR4 1460 MHz (128 GB)
Motherboard: HPE ProLiant XL420 Gen10
GPU: HP Matrox Matrox G200eh3 32MB
Storage 1: 447GB Hitachi
Storage2: 4471GB Hitachi
NIC: HP 562SFP+ 10 Gigabit




Test 1 - no SetMemoryMax() specified (i.e default 4GB)

Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames:                     1001
Length (hh:mm:ss.ms):         00:00:40.040
Frame width:                          3840
Frame height:                         2160
Framerate:                          25.000 (25/1)
Colorspace:                      YUV422P10
Audio channels:                         16
Audio bits/sample:                      24
Audio sample rate:                   48000
Audio samples:                     1921920

Frames processed:                   1001 (0 - 1000)
FPS (min | max | average):          3.484 | 4.231 | 4.084
Process memory usage (max):         808 MiB
Thread count:                       73
CPU usage (average):                2.3%

Time (elapsed):                     00:04:05.089


Test 2 - SetMemoryMax(25000) specified (i.e 25GB)

Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames:                     1001
Length (hh:mm:ss.ms):         00:00:40.040
Frame width:                          3840
Frame height:                         2160
Framerate:                          25.000 (25/1)
Colorspace:                      YUV422P10
Audio channels:                         16
Audio bits/sample:                      24
Audio sample rate:                   48000
Audio samples:                     1921920

Frames processed:                   1001 (0 - 1000)
FPS (min | max | average):          3.574 | 4.406 | 4.201
Process memory usage (max):         841 MiB
Thread count:                       73
CPU usage (average):                2.3%

Time (elapsed):                     00:03:58.282


Test 3 - SetMemoryMax(90000) specified (i.e 90GB)

Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames:                     1001
Length (hh:mm:ss.ms):         00:00:40.040
Frame width:                          3840
Frame height:                         2160
Framerate:                          25.000 (25/1)
Colorspace:                      YUV422P10
Audio channels:                         16
Audio bits/sample:                      24
Audio sample rate:                   48000
Audio samples:                     1921920

Frames processed:                   1001 (0 - 1000)
FPS (min | max | average):          3.580 | 4.405 | 4.209
Process memory usage (max):         841 MiB
Thread count:                       73
CPU usage (average):                2.3%

Time (elapsed):                     00:03:57.812

As we can see from the tests above, even when working on UHD 12bit contents with 16bit precision and doing very heavyweight operations like converting to RGB full range, applying a LUT with tetrahedral interpolation, going back to YUV limited tv range and dithering down to 10bit, the RAM usage never exceeds 4GB, in fact it doesn't matter how much RAM I tell Avisynth to allocate with SetMemoryMax(), it never really uses it anyway and the performances stay the same. This is - of course - without using Prefetch() and I never use it in any of my encodes anyway, so I guess I'm safe to say that I don't really need to specify SetMemoryMax() and I'm pretty much fine with the default value.


Here's another test with a different kind of conversion, this time instead of going from BT709 SDR to BT2020 HLG with highlights expansions to 420 nits using the BBC LUT, we're using HDR Tools by Jean Philippe Scotto di Rinaldi to go to BT2020 SDR 100 nits:

AVS Script:

Code:
video=LWLibavVideoSource("PARTE 1.mxf")
FL=WAVSource("Parte 1 Mix OnAir.L.wav")
FR=WAVSource("Parte 1 Mix OnAir.R.wav")
CC=WAVSource("Parte 1 Mix OnAir.C.wav")
LFE=WAVSource("Parte 1 Mix OnAir.LFE.wav")
LS=WAVSource("Parte 1 Mix OnAir.Ls.wav")
RS=WAVSource("Parte 1 Mix OnAir.Rs.wav")

Dolby=MergeChannels(FL, FR, CC, LFE, LS, RS)

Stereo=WAVSource("Parte 1 MIX Lt - Rt.wav")

Mute=BlankClip(length=61335, fps=25, audio_rate=48000, channels=2)

audio=MergeChannels(Stereo, Mute, Dolby, Mute, Mute, Mute)

AudioDub(video, audio)


ConvertBits(16)

#BT709 SDR to BT2020 SDR

ConvertYUVtoXYZ()
ConvertXYZtoYUV(Color=1, pColor=2)

ConverttoYUV422(matrix="Rec.2020", interlaced=false)

ConvertBits(bits=10, dither=1)

Limiter(min_luma=64, max_luma=940, min_chroma=64, max_chroma=960)

trim(0, 1000)
Test 1 - no SetMemoryMax() specified (i.e default 4GB)

Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames:                     1001
Length (hh:mm:ss.ms):         00:00:40.040
Frame width:                          3840
Frame height:                         2160
Framerate:                          25.000 (25/1)
Colorspace:                      YUV422P10
Audio channels:                         16
Audio bits/sample:                      24
Audio sample rate:                   48000
Audio samples:                     1921920

Frames processed:                   1001 (0 - 1000)
FPS (min | max | average):          4.594 | 8.925 | 5.896
Process memory usage (max):         1377 MiB
Thread count:                       129
CPU usage (average):                16.5%

Time (elapsed):                     00:02:49.782

Test 2 - SetMemoryMax(25000) specified (i.e 25GB)

Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames:                     1001
Length (hh:mm:ss.ms):         00:00:40.040
Frame width:                          3840
Frame height:                         2160
Framerate:                          25.000 (25/1)
Colorspace:                      YUV422P10
Audio channels:                         16
Audio bits/sample:                      24
Audio sample rate:                   48000
Audio samples:                     1921920

Frames processed:                   1001 (0 - 1000)
FPS (min | max | average):          4.550 | 8.933 | 5.884
Process memory usage (max):         1377 MiB
Thread count:                       129
CPU usage (average):                16.3%

Time (elapsed):                     00:02:50.134

Test 3 - SetMemoryMax(90000) specified (i.e 90GB)

Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames:                     1001
Length (hh:mm:ss.ms):         00:00:40.040
Frame width:                          3840
Frame height:                         2160
Framerate:                          25.000 (25/1)
Colorspace:                      YUV422P10
Audio channels:                         16
Audio bits/sample:                      24
Audio sample rate:                   48000
Audio samples:                     1921920

Frames processed:                   1001 (0 - 1000)
FPS (min | max | average):          4.655 | 8.845 | 5.877
Process memory usage (max):         1377 MiB
Thread count:                       129
CPU usage (average):                16.0%

Time (elapsed):                     00:02:50.337
For those wondering why I'm "happy" with this, please consider that this server is part of a wider farm and would typically run several encodes at the same time, so the CPU is gonna be pegged at 100% anyway. Besides, those tests were made only considering Avisynth, but when x264 is encoding the files in XAVC Intra Class 300 then the usage consumption is of course much higher.



For reference, today the farm encoded 284 files:

FranceBB is online now   Reply With Quote
Old Today, 01:49   #2842  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,832
Quote:
Originally Posted by FranceBB View Post
This is - of course - without using Prefetch() and I never use it in any of my encodes anyway, so I guess I'm safe to say that I don't really need to specify SetMemoryMax() and I'm pretty much fine with the default value.
If you don't add Prefetch, Avisynth+ runs in single threaded mode. Adding Prefetch can make a huge difference to encoding speed, depending on the filtering in the script.

There's a file here that can be saved as an avsi script and auto-loaded by Avisynth+. It tells Avisynth+ the type of multi-threading to use for most of the common plugins. Many plugins register their multi-threading mode with Avisynth+ automatically these days, but they still require you to add Prefetch to the script to activate multi-threading.

https://publishwith.me/ep/pad/view/r...cdWn4k9/latest

More info
http://avisynth.nl/index.php/AviSynt...lling_MT_modes

I assume for plugins with "internal" multi-threading it's a different thing to enabling Avisynth's own multi-threading. Ideally for some plugins, such as the ones in the JPSDR plugins pack, you should use the plugin arguments to specify how many threads Avisynth+ is using when you've added Prefetch to the script. And sometimes it's better to disable a plugin's multi-threading. Apparently it's not a good idea to use avstp.dll for multi-threading when Avisynth's multi-threading is enabled. The script I linked to above also tells avstp.dll to run in single threaded mode, assuming it's loaded.

Last edited by hello_hello; Today at 02:25.
hello_hello is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 08:42.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.