Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
24th April 2024, 18:28 | #2841 | Link |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 3,032
|
I'm back with some examples.
After the last discussion about SetMemoryMax() I did some more tests and this time I used a real script from a real use case scenario (although I trimmed the first 1000 frames only as I didn't want to let it filter 40 minutes worth of documentary for nothing). Source: DNX HQX UHD 4:2:2 12bit 728 Mbit/s BT709 SDR Video ID : 2 Format : VC-3 Commercial name : DNxHR HQX Format version : Version 3 Format profile : RI@HQX Format settings, wrapping mode : Frame Codec ID : 0D01030102110100-0401020271250000 Duration : 40 min 53 s Bit rate mode : Constant Bit rate : 728 Mb/s Width : 3 840 pixels Height : 2 160 pixels Display aspect ratio : 16:9 Active Format Description : Full frame 16:9 image Frame rate : 25.000 FPS Color space : YUV Chroma subsampling : 4:2:2 Bit depth : 12 bits Scan type : Progressive Bits/(Pixel*Frame) : 3.512 Stream size : 208 GiB (100%) Color range : Limited Color primaries : BT.709 Transfer characteristics : BT.709 Matrix coefficients : BT.709 AVS Script: Code:
video=LWLibavVideoSource("PARTE 1.mxf") FL=WAVSource("Parte 1 Mix OnAir.L.wav") FR=WAVSource("Parte 1 Mix OnAir.R.wav") CC=WAVSource("Parte 1 Mix OnAir.C.wav") LFE=WAVSource("Parte 1 Mix OnAir.LFE.wav") LS=WAVSource("Parte 1 Mix OnAir.Ls.wav") RS=WAVSource("Parte 1 Mix OnAir.Rs.wav") Dolby=MergeChannels(FL, FR, CC, LFE, LS, RS) Stereo=WAVSource("Parte 1 MIX Lt - Rt.wav") Mute=BlankClip(length=61335, fps=25, audio_rate=48000, channels=2) audio=MergeChannels(Stereo, Mute, Dolby, Mute, Mute, Mute) AudioDub(video, audio) ConvertBits(16) z_ConvertFormat(pixel_type="RGBP16", colorspace_op="709:709:709:limited=>rgb:709:709:full", resample_filter_uv="spline64", dither_type="error_diffusion", use_props=0) Cube("C:\Program Files (x86)\AviSynth+\LUTs\5a_BT709_HLG_UPCONVERT_DISPLAY_mode.cube", fullrange=1, interp=1) z_ConvertFormat(pixel_type="YUV422P10", colorspace_op="rgb:std-b67:2020:full=>2020:std-b67:2020:limited", resample_filter_uv="spline64", dither_type="error_diffusion", use_props=0) Limiter(min_luma=64, max_luma=940, min_chroma=64, max_chroma=960) trim(0, 1000) AVS Meter Test: Code:
AvsMeter64.exe "Test.avs" pause Hardware: CPU: Intel Xeon Gold 6238R 2.20GHz x2 (56c/112th) RAM: 32 x 4 DDR4 1460 MHz (128 GB) Motherboard: HPE ProLiant XL420 Gen10 GPU: HP Matrox Matrox G200eh3 32MB Storage 1: 447GB Hitachi Storage2: 4471GB Hitachi NIC: HP 562SFP+ 10 Gigabit Test 1 - no SetMemoryMax() specified (i.e default 4GB) Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021 AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0) Number of frames: 1001 Length (hh:mm:ss.ms): 00:00:40.040 Frame width: 3840 Frame height: 2160 Framerate: 25.000 (25/1) Colorspace: YUV422P10 Audio channels: 16 Audio bits/sample: 24 Audio sample rate: 48000 Audio samples: 1921920 Frames processed: 1001 (0 - 1000) FPS (min | max | average): 3.484 | 4.231 | 4.084 Process memory usage (max): 808 MiB Thread count: 73 CPU usage (average): 2.3% Time (elapsed): 00:04:05.089 Test 2 - SetMemoryMax(25000) specified (i.e 25GB) Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021 AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0) Number of frames: 1001 Length (hh:mm:ss.ms): 00:00:40.040 Frame width: 3840 Frame height: 2160 Framerate: 25.000 (25/1) Colorspace: YUV422P10 Audio channels: 16 Audio bits/sample: 24 Audio sample rate: 48000 Audio samples: 1921920 Frames processed: 1001 (0 - 1000) FPS (min | max | average): 3.574 | 4.406 | 4.201 Process memory usage (max): 841 MiB Thread count: 73 CPU usage (average): 2.3% Time (elapsed): 00:03:58.282 Test 3 - SetMemoryMax(90000) specified (i.e 90GB) Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021 AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0) Number of frames: 1001 Length (hh:mm:ss.ms): 00:00:40.040 Frame width: 3840 Frame height: 2160 Framerate: 25.000 (25/1) Colorspace: YUV422P10 Audio channels: 16 Audio bits/sample: 24 Audio sample rate: 48000 Audio samples: 1921920 Frames processed: 1001 (0 - 1000) FPS (min | max | average): 3.580 | 4.405 | 4.209 Process memory usage (max): 841 MiB Thread count: 73 CPU usage (average): 2.3% Time (elapsed): 00:03:57.812 As we can see from the tests above, even when working on UHD 12bit contents with 16bit precision and doing very heavyweight operations like converting to RGB full range, applying a LUT with tetrahedral interpolation, going back to YUV limited tv range and dithering down to 10bit, the RAM usage never exceeds 4GB, in fact it doesn't matter how much RAM I tell Avisynth to allocate with SetMemoryMax(), it never really uses it anyway and the performances stay the same. This is - of course - without using Prefetch() and I never use it in any of my encodes anyway, so I guess I'm safe to say that I don't really need to specify SetMemoryMax() and I'm pretty much fine with the default value. Here's another test with a different kind of conversion, this time instead of going from BT709 SDR to BT2020 HLG with highlights expansions to 420 nits using the BBC LUT, we're using HDR Tools by Jean Philippe Scotto di Rinaldi to go to BT2020 SDR 100 nits: AVS Script: Code:
video=LWLibavVideoSource("PARTE 1.mxf") FL=WAVSource("Parte 1 Mix OnAir.L.wav") FR=WAVSource("Parte 1 Mix OnAir.R.wav") CC=WAVSource("Parte 1 Mix OnAir.C.wav") LFE=WAVSource("Parte 1 Mix OnAir.LFE.wav") LS=WAVSource("Parte 1 Mix OnAir.Ls.wav") RS=WAVSource("Parte 1 Mix OnAir.Rs.wav") Dolby=MergeChannels(FL, FR, CC, LFE, LS, RS) Stereo=WAVSource("Parte 1 MIX Lt - Rt.wav") Mute=BlankClip(length=61335, fps=25, audio_rate=48000, channels=2) audio=MergeChannels(Stereo, Mute, Dolby, Mute, Mute, Mute) AudioDub(video, audio) ConvertBits(16) #BT709 SDR to BT2020 SDR ConvertYUVtoXYZ() ConvertXYZtoYUV(Color=1, pColor=2) ConverttoYUV422(matrix="Rec.2020", interlaced=false) ConvertBits(bits=10, dither=1) Limiter(min_luma=64, max_luma=940, min_chroma=64, max_chroma=960) trim(0, 1000) Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021 AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0) Number of frames: 1001 Length (hh:mm:ss.ms): 00:00:40.040 Frame width: 3840 Frame height: 2160 Framerate: 25.000 (25/1) Colorspace: YUV422P10 Audio channels: 16 Audio bits/sample: 24 Audio sample rate: 48000 Audio samples: 1921920 Frames processed: 1001 (0 - 1000) FPS (min | max | average): 4.594 | 8.925 | 5.896 Process memory usage (max): 1377 MiB Thread count: 129 CPU usage (average): 16.5% Time (elapsed): 00:02:49.782 Test 2 - SetMemoryMax(25000) specified (i.e 25GB) Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021 AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0) Number of frames: 1001 Length (hh:mm:ss.ms): 00:00:40.040 Frame width: 3840 Frame height: 2160 Framerate: 25.000 (25/1) Colorspace: YUV422P10 Audio channels: 16 Audio bits/sample: 24 Audio sample rate: 48000 Audio samples: 1921920 Frames processed: 1001 (0 - 1000) FPS (min | max | average): 4.550 | 8.933 | 5.884 Process memory usage (max): 1377 MiB Thread count: 129 CPU usage (average): 16.3% Time (elapsed): 00:02:50.134 Test 3 - SetMemoryMax(90000) specified (i.e 90GB) Code:
AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021 AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0) Number of frames: 1001 Length (hh:mm:ss.ms): 00:00:40.040 Frame width: 3840 Frame height: 2160 Framerate: 25.000 (25/1) Colorspace: YUV422P10 Audio channels: 16 Audio bits/sample: 24 Audio sample rate: 48000 Audio samples: 1921920 Frames processed: 1001 (0 - 1000) FPS (min | max | average): 4.655 | 8.845 | 5.877 Process memory usage (max): 1377 MiB Thread count: 129 CPU usage (average): 16.0% Time (elapsed): 00:02:50.337 For reference, today the farm encoded 284 files: |
25th April 2024, 01:49 | #2842 | Link | |
Registered User
Join Date: Mar 2011
Posts: 4,898
|
Quote:
There's a file here that can be saved as an avsi script and auto-loaded by Avisynth+. It tells Avisynth+ the type of multi-threading to use for most of the common plugins. Many plugins register their multi-threading mode with Avisynth+ automatically these days, but they still require you to add Prefetch to the script to activate multi-threading. https://publishwith.me/ep/pad/view/r...cdWn4k9/latest More info http://avisynth.nl/index.php/AviSynt...lling_MT_modes I assume for plugins with "internal" multi-threading it's a different thing to enabling Avisynth's own multi-threading. Ideally for some plugins, such as the ones in the JPSDR plugins pack, you should use the plugin arguments to specify how many threads Avisynth+ is using when you've added Prefetch to the script. And sometimes it's better to disable a plugin's multi-threading. Apparently it's not a good idea to use avstp.dll for multi-threading when Avisynth's multi-threading is enabled. The script I linked to above also tells avstp.dll to run in single threaded mode, assuming it's loaded. Last edited by hello_hello; 25th April 2024 at 02:25. |
|
25th June 2024, 10:13 | #2843 | Link | |
͡҉҉ ̵̡̢̛̗̘̙̜̝̞̟̠͇̊̋̌̍̎̏̿̿
Join Date: Feb 2009
Location: No support in PM
Posts: 713
|
Quote:
Code:
++ loop_cnt; if (loop_cnt >= max_loop) { // This could indicate that the queue is: // - corrupted // - or in heavy contention assert (false); return nullptr; } Anyway, I love the irony of the fix: enclosing a “lock-free” code in a std::lock_guard BTW this could be achieved more simply using a trivial std::list or std::deque for storage with guarded access instead of the complicated lock-free procedure. But I’m glad you fixed it!
__________________
dither 1.28.1 for AviSynth | avstp 1.0.4 for AviSynth development | fmtconv r30 for Vapoursynth & Avs+ | trimx264opt segmented encoding Last edited by cretindesalpes; 25th June 2024 at 10:30. |
|
4th July 2024, 13:31 | #2844 | Link |
Broadcast Encoder
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 3,032
|
Currently, when a Convertto function is called, it uses the matrix from the frame properties.
This is ok, however it only really works when the file is right and the indexer populates the frame properties correctly, which is not always the case. Here's a sample file: https://we.tl/t-aJB9L84pmw (link valid for 7 days) (keep in mind that the file ain't important, it's just an example to make a point). When it's being indexed by FFVideoSource() it shows the following frame properties: Clearly Matrix(3) doesn't mean anything, but you can bet everything you want that ConverttoRGB24() is gonna try to use that and fail, in fact it fails with "Unknown matrix". Of course one could always nuke frame properties with propClearAll() before using ConverttoRGB24() and call it a day, however that's not exactly ideal. What I'm proposing here is to still allow the Convertto functions to use the matrix populated by the frame properties as it is today, but, if it's populated automatically and not passed by the user explicitly, to fallback to a safe default instead of throwing an error if what is passed through is garbage. |
8th July 2024, 04:01 | #2845 | Link | |
Registered User
Join Date: Dec 2019
Posts: 80
|
Quote:
I just tried it again, and I'm still having the issue. It's not happening with every script, but some. Source video Source audio Script: Code:
v = lwlibavvideoSource("f:\virtualdub\tape transfers\opening to swing time 1996 vhs - edit.mkv", fpsnum=30000,fpsden=1001) a = lwlibavaudiosource("d:\recordings\opening to swing time 1996 vhs - edit.flac") audiodub(v,a) #delayaudio(-.150) assumetff().converttoyuv422(matrix="rec601", interlaced=true).convertbits(10) #Crop(8, 4, -24, -6_ Levels(50, 1,920, 0, 1020, coring=false,dither=true).tweak(bright=0, cont=1.00, hue=-0, sat=1.00, coring=false, dither=true).convertbits(8) #turnRight().Histogram().TurnLeft() normalize(0.8912) Trim(415, 5286) prefetch(8) |
|
8th July 2024, 11:00 | #2846 | Link |
Moderator
Join Date: Feb 2005
Location: Spain
Posts: 7,076
|
Yes, there are a problem and I can't understand where.
Played your script in VirtualDub2 sound noise and distorted audio. Without the normalize() play fine, but also without the normalize play noise in mpc_hc. Tested also the FFAudioSource and BSAudioSource decoders, and only the audio without video, normalize, etc., always noise and distort. Maybe something related with AviSynth flac decoders? Decoded with ffmpeg or flac seems fine.
__________________
BeHappy, AviSynth audio transcoder. |
8th July 2024, 13:14 | #2847 | Link |
German doom9/Gleitz SuMo
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 7,076
|
There is FLAC with 24 bit resolution. That might be unusual for an Avisynth audio decoder and sound wrong when it does not report the correct sample layout.
Can you check technical details of your audio source, e.g. with MediaInfo or FLAC tools? |
8th July 2024, 21:23 | #2848 | Link | ||
Registered User
Join Date: Dec 2019
Posts: 80
|
Quote:
Quote:
Code:
General Complete name : D:\recordings\opening to swing time 1996 vhs - edit.flac Format : FLAC Format/Info : Free Lossless Audio Codec File size : 15.9 MiB Duration : 3 min 0 s Overall bit rate mode : Variable Overall bit rate : 740 kb/s Audio Format : FLAC Format/Info : Free Lossless Audio Codec Duration : 3 min 0 s Bit rate mode : Variable Bit rate : 740 kb/s Channel(s) : 2 channels Channel layout : L R Sampling rate : 48.0 kHz Bit depth : 16 bits Compression mode : Lossless Stream size : 15.9 MiB (100%) Writing library : libFLAC 1.3.2 (2017-01-01) MD5 of the unencoded content : 3C5F271BC902D1430FDC4F3D062F4A21 Last edited by ENunn; 8th July 2024 at 22:22. |
||
9th July 2024, 07:48 | #2853 | Link |
Big Bit Savings Now !
Join Date: Feb 2007
Location: close to the wall
Posts: 1,671
|
Confirmed on AviSynth r4066.
LWLibavAudioSource, normalize(0.8912) on 64 bit Avisynth 4066 -> too much gain, negative values only, positive sample values clipped at 0 BestAudioSource: the same -> too much gain, negative values only, positive sample values clipped at 0 FFAudioSource: the same -> too much gain, negative values only, positive sample values clipped at 0 normalize(0.001) the same -> too much gain, negative values only, positive sample values clipped at 0 normalize(0.001, show=true) tells amplify DB: -58.9998 while still clipping heavily. LWLibavAudioSource, normalize(0.8912) on 32 bit Avisynth 4066 -> fine All these sourcefilters decode that .flac file as 2.0 16bit int, SoundForgePro 11.0 see it as 2.0 16bit too. In SoundForgePro 11.0 I see a ± disbalance, not just a DC offset towards negative values. Negative peaks coming from the narrator's mike are much stronger. May be that is it what dips normalize64bit in the ocean ? BTW, I had a overused Shure SM58 here which delivered the same waveform. As a sound engineer I had to retire this one from duty. Misadjusted coil vs gap, so a diaphragm dip gave less efficient voltage than a lift. Nice to see on a scope or from the waveform. AviSynth 4073 Clang is a no-starter for me. Gotta look out for AviSynth r4076... Ah, AviSynth+ r4073 Intel LVM 64bit works ! And normalize fault is gone. Phew.
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain) "Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..." Last edited by Emulgator; 9th July 2024 at 08:51. |
9th July 2024, 10:48 | #2854 | Link | |
Moderator
Join Date: Feb 2005
Location: Spain
Posts: 7,076
|
Quote:
Code:
AviSynth+ 3.7.3 (r4073, 3.7, x86_64) IntelLLVM BestSource.dll [n/a, 2024-04-25] ffms2.dll [2390.0.0.0, 2024-03-06] LSMASHSource.dll [1194.0.0.0, 2024-04-08] Code:
#BSAudioSource("opening to swing time 1996 vhs - edit.flac") #FFAudioSource("opening to swing time 1996 vhs - edit.flac") LWLibavAudioSource("opening to swing time 1996 vhs - edit.flac", drc_scale=0) AudioDubEx(BlankClip(length=Int(1000*AudioLengthF(last)/Audiorate(last)), width=720, height=720, fps=25), last) Info() Normalize(0.95) Using AviSynth test 14 r4066 the output is like the attached image. But, with r4073, still is wrong played in mpc-hc
__________________
BeHappy, AviSynth audio transcoder. Last edited by tebasuna51; 9th July 2024 at 11:09. Reason: add info |
|
9th July 2024, 11:21 | #2855 | Link |
Big Bit Savings Now !
Join Date: Feb 2007
Location: close to the wall
Posts: 1,671
|
tebasuna51, did you try to play a .avs script in MPC-HC served from r4073 ?
Or did you play a transcoded .wav as the title bar suggests, maybe from SoundOut() AviSynth r4073?
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain) "Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..." |
10th July 2024, 10:35 | #2857 | Link |
Big Bit Savings Now !
Join Date: Feb 2007
Location: close to the wall
Posts: 1,671
|
So this would mean that r4073 still has the fault inherited.
AviSynth64+ r4073 Just throwing the script on MPC-HC64 2.3.0 here: plays fine. LAVSplitter -> SaneAR Render MPC-BEx64 1.7.2 using MPC Audio renderer: missing audio source filter, plays video only. Code:
AVI/WAV File Source::Avisynth audio #1 Media Type 0: -------------------------- Audio: 0xfffe 48000Hz 2.0 chn 1536 kbit/s AM_MEDIA_TYPE: majortype: MEDIATYPE_Audio {73647561-0000-0010-8000-00AA00389B71} subtype: MEDIASUBTYPE_0xfffe {0000FFFE-0000-0010-8000-00AA00389B71} formattype: FORMAT_WaveFormatEx {05589F81-C356-11CE-BF01-00AA0055595A} bFixedSizeSamples: 1 bTemporalCompression: 0 lSampleSize: 4 cbFormat: 40 WAVEFORMATEX: wFormatTag: 0xfffe nChannels: 2 nSamplesPerSec: 48000 nAvgBytesPerSec: 192000 nBlockAlign: 4 wBitsPerSample: 16 cbSize: 22 (extra bytes) WAVEFORMATEXTENSIBLE: wValidBitsPerSample: 16 dwChannelMask: 0x00000003 SubFormat: {00000001-0000-0010-8000-00AA00389B71}
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain) "Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..." Last edited by Emulgator; 10th July 2024 at 11:03. |
10th July 2024, 11:27 | #2858 | Link |
Registered User
Join Date: Jul 2018
Posts: 494
|
AviSynth64+ r4073 (both clang and Intel LLVM) + MPC-BE x64 1.7.2 (MPC Audio Renderer, LAV Filters 0.79.2) no issues with the following (also no issues with mpv 0.38.0 x86_64):
Code:
v = lwlibavvideoSource("opening to swing time 1996 vhs - edit.mkv", fpsnum=30000,fpsden=1001) a = lwlibavaudiosource("opening to swing time 1996 vhs - edit.flac") audiodub(v,a) #delayaudio(-.150) assumetff().converttoyuv422(matrix="rec601", interlaced=true).convertbits(10) #Crop(8, 4, -24, -6_ Levels(50, 1,920, 0, 1020, coring=false,dither=true).tweak(bright=0, cont=1.00, hue=-0, sat=1.00, coring=false, dither=true).convertbits(8) #turnRight().Histogram().TurnLeft() normalize(0.8912) Trim(415, 5286) |
11th July 2024, 10:04 | #2859 | Link |
Moderator
Join Date: Feb 2005
Location: Spain
Posts: 7,076
|
Test with some player options:
Code:
r4073 r4066 ------------------ ------------------ Player without Normalize without Normalize --------------------- ------- --------- ------- --------- mpc-hc 2.3.2 DirectS Ok Noise Noise Noise mpc-hc 2.3.2 + SaneAR Ok Ok Ok Noise VirtualDub2 2.1.1.607 Ok Ok Ok Noise mpc-be 1.7.1 MPC-Aud Ok Ok Ok Ok mpc-be 1.7.1 DirectS Ok Ok Ok Noise vlc 3.0.21 Ok Ok Ok Noise Maybe we need a new test15 and/or a new release 3.7.4, the 3.7.3 is 1 year old.
__________________
BeHappy, AviSynth audio transcoder. Last edited by tebasuna51; 11th July 2024 at 10:12. |
11th July 2024, 20:02 | #2860 | Link |
...?
Join Date: Nov 2005
Location: Florida
Posts: 1,445
|
Between r4066 and r4073 these were the changes:
Code:
Add cache for string heap to avoid duplicates. related to #389 pinterf committed Apr 17, 2024 Add basic ArraySort pinterf committed Apr 17, 2024 Update documentation and changelog (break/continue) pinterf committed Apr 10, 2024 SetLogParams defaults to follow doc: stderr and LOG_INFO (mentioned in #391) pinterf committed Apr 9, 2024 Add "continue" for for/while loops Implement #392 pinterf committed Apr 9, 2024 Fix: "break" to return up-to-date "last" in the expression chain pinterf committed Apr 9, 2024 fix when building with absolute paths jopejoe1 authored and qyot27 committed Feb 3, 2024 What everyone is actually comparing here - and a couple of the posts acknowledge this around the periphery without directly calling it out - are different compilers. Ubuntu uses GCC 13, which is unaffected. ClangCL and Intel are reported to be [mostly] unaffected. 3.7.3 was reported to be unaffected; it was built with MSVC 2019. It's pointing at there being a problem with MSVC 2022, at least as of the time of the last test build having been built. In the time since, maybe the issue has been resolved with an update to the compiler, or it would produce a build of r4073 that's still affected by the same problem, even when all the other compilers produce working builds. |
Thread Tools | Search this Thread |
Display Modes | |
|
|