faster CPU = faster conversion?

[spoRv]

I'm using a Threadripper 1950x with 64GB RAM 3200mhz; it's quite fast, but lately I've started to make projects involving 10bit UHD HDR, and conversion time is obviously increased a lot from 8bit HD SDR.

Now I'm thinking about the new AMD 7950x (still don't know if 32 or 64GB); according to CPUbenchmark, it's more or less twice faster than mine, both on single and multi thread:

https://www.cpubenchmark.net/compare/3058vs5031/AMD-Ryzen-Threadripper-1950X-vs-AMD-Ryzen-9-7950X

I'm curious: this will apply also to avisynth conversions? I mean, if for example with the 1950x it takes 24hrs, would it take around 12hrs using the 7950x?

Or there is something else to take in account, as RAM speed and type, CPU family (Threadripper vs Ryzen) etc?

Thanks!

[guest]

I'm using a Threadripper 1950x with 64GB RAM 3200mhz; it's quite fast, but lately I've started to make projects involving 10bit UHD HDR, and conversion time is obviously increased a lot from 8bit HD SDR.

Now I'm thinking about the new AMD 7950x (still don't know if 32 or 64GB); according to CPUbenchmark, it's more or less twice faster than mine, both on single and multi thread:

https://www.cpubenchmark.net/compare/3058vs5031/AMD-Ryzen-Threadripper-1950X-vs-AMD-Ryzen-9-7950X

I'm curious: this will apply also to avisynth conversions? I mean, if for example with the 1950x it takes 24hrs, would it take around 12hrs using the 7950x?

Or there is something else to take in account, as RAM speed and type, CPU family (Threadripper vs Ryzen) etc?

Thanks!

I would be very interested in what you encode, and with what app, that takes 24 hours :eek: :scared:

There are so many variables to this, what app's you're using, what filters & scripts you're using, the list goes on.

ATM, about the best you can do (unless you've got very deep pockets), would be the new Ryzen 7950X3D, which may or may not be faster than its "little" brother, the 7950X.

As the lucky owner of both a 7950X & an i9 13900KF, the choice is up to you, as they are so close, it's pretty hard to pick.

Having said all that, the Intel option CAN be a LOT cheaper, as you can go for a lesser chipset motherboard, and DDR4 RAM, I doubt that it would affect encoding performance, as overclocking does generate too much more heat, and they already run @ 100°C, so you need GOOD cooling for both the AMD & Intel's.

As for RAM, I find that 32Gb is plenty, but again, if you've got deep pockets, then go for 64Gb.

The one possible advantage is that the Ryzen's support AVX512.

Do your research, and choose well......

[Zarxrax]

I would look at what you are encoding and see if its utilizing multiple threads well, or is it largely using a few cores.
Avisynth is mostly CPU limited, and while it has multi-threading, it doesn't really utilize it to the fullest. There are some plugins that can make use of GPU as well.
I think upgrading the CPU would definitely give you a big boost, but would it cut speeds in half? I dunno.

Before upgrading the CPU, I would recommend looking at other ways to boost your processing speed if its important to you. First, have you tried vapoursynth? It might be able to make better use of multiple cores and improve the processing speed. For any plugins you are using, are there other plugins that do a similar task, and have you tried them out? Are there any GPU accelerated plugins that you can use? Is it possible to rearrange the order of the commands in your script? For example, if you are upscaling your video and then applying a heavy plugin, what if you use that plugin before upscaling? Or if you are doing conversions between colorspace or bitdepths, have you tried using those commands at different points in your script? Some plugins might be much faster when working in certain colorspaces.
Also, if you see that your cpu is only using half the threads, Could you run 2 videos at the same time?

[spoRv]

About deep pockets: actually they are not deep at all! :D I'm pretty satisfied of my actual rig, so if I "need" to change it, I'll try to sell it first, to recover some money that I hope to cover most of the expenses of CPU MB RAM.

Was thinking about the 7950x because it seems the CPU with the best price/speed ratio; I read a bit about the new 7950X3D and the main feature seems to be lower consumption; still, with its price being 25% or more than 7950x, would bring me to think about the latter instead; for the MB I was thinking about a cheap X670, and RAM something around 4000mhz as the ones with higher speed cost much more offering not too drastic speed increase.

About my encodes: I'm talking about restoration projects - the last one I'm doing right now involves a (blown up) UHD WEB regrading, taking DVD as color reference.

During the years I think (hope?) I already improved my skills to the point that I guess it's hard to "squeeze" more speed from my scripts; I started arranging the tasks in the best possible way, then I stepped to Avisynth+, then lately to Avisynth+ 64bit; the last step was the best IMHO to improve the speed and avoid RAM problems - that were not present using mere 8bit HD SDR sources.

Another is to use MP_pipeline; I usually split my script into more parts (like part 1 take care of frames 00000 to 15000, part to 15001 to 30000 and so on); setting up 4GB each, I reach almost always 100% CPU and RAM is almost full (even if I don't know why each script "eats" about 6/7GB RAM); this increase speed usually to 400/500% using 8 parts.

[DTL]

Or there is something else to take in account, as RAM speed and type, CPU family (Threadripper vs Ryzen) etc?


It looks the era of high-performance computing desktops is ended.
Today desktops are mostly about load windows (11) and open Word and web-browser not very slow.

When someone need good performance workstation it is about HP Z8 Fury G5 that top loaded looks like 2 TB ECC RAM (still DDR not HBM), 2 kW power supply, good W-Xeon like W-3375 38 cores 76 threads and several (up to 4) full-size data compute accelerator boards of professional class. Top single chip workstation Xeon may be w9-3495X 56 cores/112 threads, 8 channels of DDR5. Though the pricing of complete small undertable box is about $XXXXX. It is still not supercomputer in a separate cooled machine room.


As the lucky owner of both a 7950X & an i9 13900KF, the choice is up to you, as they are so close, it's pretty hard to pick.


Can I ask to test that release of vsTTempSmooth https://github.com/Asd-g/AviSynth-vsTTempSmooth/releases/tag/1.2.3 between AMD 7950X and i9-13900KF ?

Test script is

LoadPLugin("vsTTempSmooth.dll")
LoadPlugin("AddGrainC.dll")

BlankClip(15, 1920, 1080, color=color_gray)
#BlankClip(15, 3840, 2160, color=color_gray)

ConvertToYV12()
AddGrain(10)
Loop(10000000)

vsTTempSmooth(maxr=2) # using auto-detected default SIMD - AVX2 for intel (?) and AVX512 for AMD (?)

Prefetch(16) # equal for both AMD and intel ?


For 1080p and 2160p frame size.

The Xeon Gold 6134 (8 cores / 16 threads) with maxr=7 gives about 90fps at 2160p and 378fps at 1080p. And with maxr=2 - 300 and 1200 fps (about proportional to frame sample count). It looks 6 channels of DDR4-2666 RAM and 24 MB of L3 cache not very greatly helps.

The results of much cheaper i5-11600 is even not 2 times lower.

[guest]

It looks the era of high-performance computing desktops is ended.
Today desktops are mostly about load windows (11) and open Word and web-browser not very slow.

What do you mean, "ended"...it's the best it's ever been !!!

When someone need good performance workstation it is about HP Z8 Fury G5 that top loaded looks like 2 TB ECC RAM (still DDR not HBM), 2 kW power supply, good W-Xeon like W-3375 38 cores 76 threads and several (up to 4) full-size data compute accelerator boards of professional class. Top single chip workstation Xeon may be w9-3495X 56 cores/112 threads, 8 channels of DDR5. Though the pricing of complete small undertable box is about $XXXXX. It is still not supercomputer in a separate cooled machine room.


Yeah, we can all dream of a system similar to this, but who can afford a system that costs as much as a medium size car, or small house or apartment, AND uses as much energy to power that car or house !!!

Can I ask to test that release of vsTTempSmooth https://github.com/Asd-g/AviSynth-vsTTempSmooth/releases/tag/1.2.3 between AMD 7950X and i9-13900KF ?

Test script is

LoadPLugin("vsTTempSmooth.dll")
LoadPlugin("AddGrainC.dll")

BlankClip(15, 1920, 1080, color=color_gray)
#BlankClip(15, 3840, 2160, color=color_gray)

ConvertToYV12()
AddGrain(10)
Loop(10000000)

vsTTempSmooth(maxr=2) # using auto-detected default SIMD - AVX2 for intel (?) and AVX512 for AMD (?)

Prefetch(16) # equal for both AMD and intel ?


For 1080p and 2160p frame size.


You can always ask, but that script would not work with my go-to encoding app, and I doubt that I would have a use for it, and its dependencies.

I'm trying to be polite, I can't simply just load that, and test it for you :(

What do you use ??

The Xeon Gold 6134 (8 cores / 16 threads) with maxr=7 gives about 90fps at 2160p and 378fps at 1080p. And with maxr=2 - 300 and 1200 fps (about proportional to frame sample count). It looks 6 channels of DDR4-2666 RAM and 24 MB of L3 cache not very greatly helps.

The results of much cheaper i5-11600 is even not 2 times lower.

[guest]

I'm pretty satisfied of my actual rig...

Well, if you're happy with it, why ask...it's just that you need deep pockets these days to get something that will do the job fast (without going way overboard on house priced systems)

About my encodes: I'm talking about restoration projects - the last one I'm doing right now involves a (blown up) UHD WEB regrading, taking DVD as color reference.

Another is to use MP_pipeline; I usually split my script into more parts (like part 1 take care of frames 00000 to 15000, part to 15001 to 30000 and so on); setting up 4GB each, I reach almost always 100% CPU and RAM is almost full (even if I don't know why each script "eats" about 6/7GB RAM); this increase speed usually to 400/500% using 8 parts.

I would still like to know what you use for your "restoration projects" ?? I'm just curious about the script splitting, etc, why would you do that ?? to speed things up ??

I've recently started upscaling some older footage, with Topaz AI, and specific avisynth scripts, etc, etc, and getting reasonably good results.

[spoRv]

Even if I'm pretty satisfied, IF I can sell my actual hardware (at least CPU+cooler+MB) and get a 2x faster setup shelling a relatively small amount of money, why not?
Said so, I'd never spend $$$$ for EPYC or new Threadripper CPUs... you see, the 7950x seems the best price/speed ratio around right now.

Script splitting: as I wrote before.
Let's say I have a 120000 frame source to process.
I first write my script as I want.
Then I create X scripts that are the same as main script, with only added a trim line.
If with one unsplit script I have for example 0.2fps, with X=8 split scripts I can get 0.8fps (and never 0.2fps * X = 1.6fps)

Got it?

Topaz AI: I like it, but often I got too plastic-y result; if there are two or more sources, I much prefer to average: https://forum.fanres.com/thread-4733.html

[guest]

Even if I'm pretty satisfied, IF I can sell my actual hardware (at least CPU+cooler+MB) and get a 2x faster setup shelling a relatively small amount of money, why not?
Said so, I'd never spend $$$$ for EPYC or new Threadripper CPUs... you see, the 7950x seems the best price/speed ratio around right now.

It's NOT, a basic i9-13900KF, or 13900F, with a B660 MB & DDR4 RAM is a cheaper option !!!

Script splitting: as I wrote before.
Let's say I have a 120000 frame source to process.
I first write my script as I want.
Then I create X scripts that are the same as main script, with only added a trim line.
If with one unsplit script I have for example 0.2fps, with X=8 split scripts I can get 0.8fps (and never 0.2fps * X = 1.6fps)

Got it?

No, I don't "got it" !!!

Topaz AI: I like it, but often I got too plastic-y result; if there are two or more sources, I much prefer to average: https://forum.fanres.com/thread-4733.html

So how do I "get" this MergeHdUp ???

[FranceBB]

No, I don't "got it" !!!

On not fully parallelized filters if you trim the file in multiple parts and encode them at the same time only to append them at the end you can boost your speed by making use of the whole CPU. For instance, a 0.8fps job split in 2 would run at 0.8 and 0.8, but it would have half the frames to encode per process, so it would be as if you had just one encode running at 1.6fps.
You can do the same trick as many times as you want until you saturate the CPU, but since it's never that precise it won't be exactly twice the speed for twice the resource consumption but a bit less, so 0.8fps single would be like 1.4fps double (while mathematically it should be 1.6).

I know cause I'm aware of other people who do the same (although I don't).

[guest]

On not fully parallelized filters if you trim the file in multiple parts and encode them at the same time only to append them at the end you can boost your speed by making use of the whole CPU. For instance, a 0.8fps job split in 2 would run at 0.8 and 0.8, but it would have half the frames to encode per process, so it would be as if you had just one encode running at 1.6fps.
You can do the same trick as many times as you want until you saturate the CPU, but since it's never that precise it won't be exactly twice the speed for twice the resource consumption but a bit less, so 0.8fps single would be like 1.4fps double (while mathematically it should be 1.6).

I know cause I'm aware of other people who do the same (although I don't).

Thanks for the explanation :)

So what apps would one use, to do the above procedure, that's what I'm curious about.

And correct me if I'm wrong, but isn't that basically the same thing that RipBot264 does with it's Distributed Encoding, and Staxrip with its "chunks" encoding option ??

Sounds like a lot of mucking around, when there's app out there, that do it already ???

[spoRv]

I put the .avs split scripts into VirtualDub, and save encoded video as lossless; then I join them into a single lossless one and finally encode it.

Do RipBot264 or Staxrip the same? If so, do they encode in AV1?

[guest]

I put the .avs split scripts into VirtualDub, and save encoded video as lossless; then I join them into a single lossless one and finally encode it.

Do RipBot264 or Staxrip the same? If so, do they encode in AV1?

You could possibly load the scripts (but you probably wouldn't need to split them) into both, however, RipBot does not currently support AV1 :(, and I think Staxrip does, but as I don't use Staxrip, I'm not 100% sure.

EDIT:- Staxrip does indeed support AV1 (AOMEnc, Rav1c & SVT-AV1)

Have you heard of FastFlix ??

[DTL]

"What do you mean, "ended"."

The last 'enthusiasts edition' desktops intels were about 9 or 10 series. It was sort of 'small Xeons for home' with up to 4 memory channels and with 8..10+ cores and AVX512. After these series we have only degradation - no more 4 memory channels and no more AVX512 and a few of 'performance' cores in single chip. Intel-11 already have a limited memory channels to 2 even at best desktop editions and small core numbers.

In 2017 we still see SkyLake-X for HEDT with up to 18 cores and all AVX512 and 4 memory channels. Now in 2023 we have only 2 memory channels and cutoff AVX512 at low-end desktops intels even in i9-13900*.

"that script would not work with my go-to encoding app,"

I mean run it with simple avsmeter64.exe. At least several seconds to see stable average fps. It uses only AVS and 1 more external plugin AddGrainC (I hope it is typically installed at most of users - http://avisynth.nl/index.php/AddGrainC .

[guest]

"What do you mean, "ended"."

The last 'enthusiasts edition' desktops intels were about 9 or 10 series. It was sort of 'small Xeons for home' with up to 4 memory channels and with 8..10+ cores and AVX512. After these series we have only degradation - no more 4 memory channels and no more AVX512 and a few of 'performance' cores in single chip. Intel-11 already have a limited memory channels to 2 even at best desktop editions and small core numbers.

In 2017 we still see SkyLake-X for HEDT with up to 18 cores and all AVX512 and 4 memory channels. Now in 2023 we have only 2 memory channels and cutoff AVX512 at low-end desktops intels even in i9-13900*.

"that script would not work with my go-to encoding app,"

I mean run it with simple avsmeter64.exe. At least several seconds to see stable average fps. It uses only AVS and 1 more external plugin AddGrainC (I hope it is typically installed at most of users - http://avisynth.nl/index.php/AddGrainC .

Hi DTL, aah, there's nothing like living in the past, the "good 'ole days" are long behind us.

OK, I just had a quick look in my "plugins" folder, and indeed I have both, in question.

I've been using AMD for a LONG time, so I missed all the HEDT Intel's, it's only VERY recently I went back to Intel, mainly to compare it with the 7950X, which as least has AVX512...

So this is how I'm used to scripts (for RipBot), and I generally only use AVSMeter within it the app.

#Custom
SetCacheMode(1)
SetMemoryMax(16384)
LoadPlugin("%AVISYNTHPLUGINS%\masktools\masktools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
LoadPlugin("%AVISYNTHPLUGINS%\PD_TOOLS\MedianBlur2\MedianBlur2.dll")
Import("%AVISYNTHPLUGINS%\PD_TOOLS\EXTOOLS\ExTools.avs")
Import("%AVISYNTHPLUGINS%\PD_TOOLS\LSF-PLUS\LSFplus.avs")
Import("%AVISYNTHPLUGINS%\PD_TOOLS\RESIZERS-PACK\Resizers Pack.avs")
Import("%AVISYNTHPLUGINS%\PD_TOOLS\SHARPENERS-PACK\Sharpeners Pack.avs")
Import("%AVISYNTHPLUGINS%\PD_TOOLS\SMDEGRAIN\SMDegrain cpu.avs")
Import("%AVISYNTHPLUGINS%\PD_TOOLS\Zs_RF_Shared\Zs_RF_Shared.avs")
pre=ex_Median(video,mode="IQMV")
video=SMDegrain(video,tr=2,thSAD=200,thSADC=100,contrasharp=true,prefilter=pre,str=1.2,refinemotion=true)

So if you can suggest how to mod this to suit the script you would like tested, then I will run some tests for you...deal ??

[DTL]

" I just had a quick look in my "plugins" folder, and indeed I have both,"

For testing it is required to update vsTTempSmooth to latest release of https://github.com/Asd-g/AviSynth-vsTTempSmooth/releases/tag/1.2.3 . The AddGrainC should be enough any old version (it used only to fill some cached frames and do nothing after Loop()).

Having all plugins installed so simply save

SetCacheMode(1)
SetMemoryMax(16384)
LoadPlugin("%AVISYNTHPLUGINS%\AddGrainC\AddGrainC.dll")
LoadPlugin("%AVISYNTHPLUGINS%\vsTTempSmooth\vsTTempSmooth.dll") # I hope it is the right path where you have plugins installed ? I never use global plugins dir and always simply put all .dlls and script in same dir.

BlankClip(15, 1920, 1080, color=color_gray) # for 1080p test
#BlankClip(15, 3840, 2160, color=color_gray) # for 2160p test

ConvertToYV12()
AddGrain(10)
Loop(10000000)

vsTTempSmooth(maxr=2) # using auto-detected default SIMD - AVX2 for intel (?) and AVX512 for AMD (?)

Prefetch(32) # equal for both AMD and intel ?


into script file and run it with AVSMeter (x64 build) for about 10 seconds to have stable 'avg' fps (last number in fps line).

"to compare it with the 7950X, which as least has AVX512..."

7950x is 2 memory channels only and sort of AVX512-ready (can execute without invalid instruction crash but performance may be close to AVX2 because of only 256bit dispatch ports ?). It is expected 'full-width AVX512' is some next generations of Zen from AMD for desktops.

" there's nothing like living in the past, the "good 'ole days" are long behind us."

It looks we will never more see at poor degraded desktops something like i9-7980XE from 2017 - simple i9 but 18 cores 36 threads, 2.6/4.2 GHz and 4xDDR4 memory channels with all cores AVX512 and 2xFMA512 per core. It may be a candidate to try if found cheap enough secondhand.

Nowdays any serious HEDT is Workstation-class Xeon like W-series.

[guest]

" I just had a quick look in my "plugins" folder, and indeed I have both,"

For testing it is required to update vsTTempSmooth to latest release of https://github.com/Asd-g/AviSynth-vsTTempSmooth/releases/tag/1.2.3 . The AddGrainC should be enough any old version (it used only to fill some cached frames and do nothing after Loop()).

Having all plugins installed so simply save

SetCacheMode(1)
SetMemoryMax(16384)
LoadPlugin("%AVISYNTHPLUGINS%\AddGrainC\AddGrainC.dll")
LoadPlugin("%AVISYNTHPLUGINS%\vsTTempSmooth\vsTTempSmooth.dll") # I hope it is the right path where you have plugins installed ? I never use global plugins dir and always simply put all .dlls and script in same dir.

BlankClip(15, 1920, 1080, color=color_gray) # for 1080p test
#BlankClip(15, 3840, 2160, color=color_gray) # for 2160p test

ConvertToYV12()
AddGrain(10)
Loop(10000000)

vsTTempSmooth(maxr=2) # using auto-detected default SIMD - AVX2 for intel (?) and AVX512 for AMD (?)

Prefetch(32) # equal for both AMD and intel ?


into script file and run it with AVSMeter (x64 build) for about 10 seconds to have stable 'avg' fps (last number in fps line).

"to compare it with the 7950X, which as least has AVX512..."

7950x is 2 memory channels only and sort of AVX512-ready (can execute without invalid instruction crash but performance may be close to AVX2 because of only 256bit dispatch ports ?). It is expected 'full-width AVX512' is some next generations of Zen from AMD for desktops.

" there's nothing like living in the past, the "good 'ole days" are long behind us."

It looks we will never more see at poor degraded desktops something like i9-7980XE from 2017 - simple i9 but 18 cores 36 threads, 2.6/4.2 GHz and 4xDDR4 memory channels with all cores AVX512 and 2xFMA512 per core. It may be a candidate to try if found cheap enough secondhand.

Nowdays any serious HEDT is Workstation-class Xeon like W-series.

OK, DTL, I have given up for today, so I will have a fiddle with tomorrow.

Cheers.

[DTL]

About deep pockets: actually they are not deep at all! :D I'm pretty satisfied of my actual rig, so if I "need" to change it, I'll try to sell it first, to recover some money that I hope to cover most of the expenses of CPU MB RAM.



For cheap solutions it may be better to look into secondhand professional computing platforms like old renderfarms parts on sales from upgrades. At e-bay for example. The not very old generation Xeon (or may be AMD) too platforms may give more aggregative performance for distrubuted processing by pieces.

For example I have some old workstation-class motherboard (it have PCIe x16 for external video card and not very loaded with integrated networking) with 5000-series intel chipset and 2xE5420 Xeons (SSE-only) and 16 GB ECC DDR2 RAM in 4 channels (can be cheaply upgraded to 64 GB with 8x8 GB modules). It come secondhand at about $100 or less in begining of 2022 with free power supply about 300..400W (though very noisy and require separate room to install for overnight computing). So it is 4+4 cores system and 2x12MB L2 cache of increased number of ways (poor desktop CPUs also typically have poorer cache design with lower number of ways so easier to hit cache set overload condition) and good 4-channels memory controller. If put several such systems in a rig it may be cheaper solution in compare with single chip platform of several next generations newer.
Also as real SDRAM performance is about no-progress in decades from 199x (about 50..100ns latency from SDR SDRAM in 199x to DDR5 in 202x) the more real separate SDRAM channels you have the better may be real multithreading performance. Several cheap secondhand boards with 4 real SDRAM channels in each may outperform some new 2-channel only system.

[guest]

So after the mention of the old i9-7980XE, I had a look on eBay, and they are still readily available, and they still command "good" money, then you'd have to source a socket 2066 motherboard, which would still be fairly costly, so in the end, you'd probably spend similar or more than a 13th Gen system, and even tho they may have some good tech onboard, they are so far behind in just raw horse power (according to most benchmarks).

Sad but true..:eek:

And I too have several old Supermicro Server systems, with X5680's & 90's, with 32Gb DDR3 ECC Ram, and they used to power along nicely, but now that I have quite the collection of Ryzen's and the 13900, they're just a waste of energy :(:(

[DTL]

Though i9-7980XE is about the same as currently used TR 1950 - both 4channels of DDR4 and comparable number of cores. So i9-7980 may be a bit faster in some applications (where AVX512 is more or less good used) - https://static.techspot.com/articles-info/1493/bench/1HandBrake.png and somewhere slower.

Real decreasing of processing time twice is simply adding second TR 1950 system and spreading task to 2 parts. As time going on - the TR1950 will be more and more cheaper because at home systems people typically like to have one a bit more performance system and throw away for a low price second hand old system after upgrade.

" if for example with the 1950x it takes 24hrs, would it take around 12hrs using the 7950x?"

Very unlikely. 7950x uses 2channels of DDR5 and TR1950x 4channels of DDR4 (if TR is really 'unganged' pack of 2 separate 8cores chips on a single board it even better serve in multithreading). The effective memory bandwidth is about the same. Yes - the 2channels motherboard and CPU socket is cheaper in manufacturing so marketing can make more money from user if use DDR5 2channels instead of DDR4 4channels. The 2x more onboard L3cache is unlikely very helpful with AVS as it like to use very large caches around each filter in a chain and total working set easily run to GBs and 10ths of GBs and can not fit in any MBs cache.
The AVX512 support in 7950 also only the beginning and not run at 'full' speed but at the AVX2 rate only. If even yout get 2x performance boost at some few plugins in a chain it will not convert to 2x total performance boost in processing.

So to make TR1950x twice faster at real workload it is still more effective to simply add second TR1950x system in parallel and load half of frame processing to second system in a pool. It will have 8channels of memory in 'true/physical unganged' mode of 2 completely independent streams of DDR4 x4.

[guest]

Though i9-7980XE is about the same as currently used TR 1950 - both 4channels of DDR4 and comparable number of cores. So i9-7980 may be a bit faster in some applications (where AVX512 is more or less good used) - https://static.techspot.com/articles-info/1493/bench/1HandBrake.png and somewhere slower.

Real decreasing of processing time twice is simply adding second TR 1950 system and spreading task to 2 parts. As time going on - the TR1950 will be more and more cheaper because at home systems people typically like to have one a bit more performance system and throw away for a low price second hand old system after upgrade.

That's the one massive advantage of using RipBot, with the Distributed Encoding, you can have up to 16 other PC's encoding the one job, and depending on how much filtering you need to do, I can get a full 4K movie encoded in less time than is would be to watch the movie :), and that's using all 4 Ryzens, the 13900KF, and sometimes a dual E5-2697v2 server/workstation :)

[guest]

OK, DTL, I have given up for today, so I will have a fiddle with tomorrow.
Cheers.

Hi DTL,

Well, talk about fiddle...

I ended up using this script (showing only the 1080 one):-

#Custom
SetCacheMode(1)
SetMemoryMax(16384)
LoadPlugin("%AVISYNTHPLUGINS%\PD_TOOLS\AddGrainC\AddGrainC.dll")
LoadPlugin("%AVISYNTHPLUGINS%\PD_TOOLS\vsTTempSmooth\vsTTempSmooth.dll")
ConvertToYV12(video)
AddGrain(10)
video=BlankClip(video, width=1920, height=1080, color=color_gray)
video=vsTTempSmooth(video,maxr=2)

I updated both AddGrainC & vsTTempSmooth, and I also had to change the test files from 10 bit to 8 bit :scared:

I would just like to add that it would NOT be possible to run this using "standard" RipBot264 !!!!

Anyway, here are the screenshots of the results, interesting to say the least.

Green borders is the 7950X, Red is the 13900KF...

https://i.imgur.com/6Ou2vH5.jpg

Now, I'd like to see your test results....

[DTL]

Your script not completely correct - you need to add noise to input of vsTTempSmooth. Using clear frame from BlankClip not load gathering engine as with real noised frames and in run mostly in 'broadcast' mode of single index gathered. It run somehow faster in compare with real life noised load and random indexes gathering for weight table.
Also using Trim+Loop after adding noise allow to use cache of AVS as uncompressed frameserver to vsTTempSmooth without calculating AddNoise for each frame - it allow to better test RAW vsTTempSmooth performance.


SetCacheMode(1)
SetMemoryMax(16384)
LoadPlugin("vsTTempSmooth.dll")
LoadPlugin("AddGrainC.dll")

video=BlankClip(width=1920, height=1080, color=color_gray, pixel_type="YV12")
video=AddGrain(video,10)
video=Trim(video, 0, 15)
video=Loop(video, 100000000) #cache frames from AddGrain to skip time for AddGrain activity from testing
video=vsTTempSmooth(video,maxr=2)



My current results with maxr=2 from Xeon Gold 6134 (16 threads prefetch - full hyperthreading):
1080p
AVX2 (opt=2) - 1010 fps
AVX512 (default) - 1200 fps

2160p
AVX2 (opt=2) - 255 fps
AVX512 (default) - 300 fps

Enabling hyperthreading at 8 cores Xeon adds only about 5..15% (difference between Prefetch(16) and Prefetch(8)) depending on maxr param and others.

From i5-9600K (AVX2 only, 6 threads):
1080p - 480 fps
2160p - 118 fps

Using AVX512 adds about only about 20% above AVX2 in full gathering mode. If use only frame-number based weighting - the benefit may be about 50% (no random memory gathering - more easy task for memory subsystem).
To enable frame-number only weighting mode I use params set

ythresh=10, ymdiff=20, uthresh= 10, vthresh=10, umdiff=20, vmdiff=20

for vsTTempSmooth().

Also I not understand why so small CPU usage reported. May you not add Prefetch(number of cores or HT threads) at the script ?

[guest]

Your script not completely correct....

Also I not understand why so small CPU usage reported. May you not add Prefetch(number of cores or HT threads) at the script ?

Gee's I dunno, you do a guy a favour, and he picks it to pieces. :(

Like I said, RipBot can't use certain scripts, and the way they are written AFAIK.

I don't know what you use to run your scripts, but I only "know" RipBot, it's not widely used, but it certainly does what I need it to do, well the PD builds, that is.

I had to fiddle around with it until AVSMeter (within RipBot) actually ran the script.

Also, Prefetch is set elsewhere in the script, not here, and it's set @ 12, which seems to be the best for 16 core CPU's, and Avisynth, in RipBot.

It's not something I would use, so that's it.

[DTL]

Well - thank you for results. May be AVSMeter may report not correct CPU usage when run not from command line. The fps numbers looks like many threads are used and CPU usage by script should be about 100%.. It is still visible some better performance of AMD 7xxx over intel 13 series.

[guest]

Well - thank you for results. May be AVSMeter may report not correct CPU usage when run not from command line. The fps numbers looks like many threads are used and CPU usage by script should be about 100%.. It is still visible some better performance of AMD 7xxx over intel 13 series.

You are welcome, sorry it didn't quite go to your plan.

Most of the more experienced guys on Doom9 know how to test & benchmark all these complex scripts & commands, etc, etc

All I want to learn is to run filters & scripts that I can actually use.

I agree that the 7950X does have a slight advantage when actually encoding videos, they are so close, and for anyone that wants a good encoding system (without going overboard), an Intel i9-13900 can be a cheaper option than the 7950X.

[MysteryX]

Just something to keep in mind, Avisynth is not always super effective at using many cores. For heavy scripts and HD content, you can get 16GB saturated with just 8 instances. Running 32 concurrent threats... can be quite a lot. Depends whether your filters need to be recreated for each threat, the type of video you process, and how much RAM you got available.

[FranceBB]

run the script with AVSMeter (x64 build) for about 10 seconds to have stable 'avg' fps (last number in fps line).

Test 1:
Type: Personal PC (Desktop)
CPU: i7 5930K 6c/12th AVX2 3.50GHz
RAM: 32GB DDR4 (4x8 GB)
GPU: NVIDIA GTX 980Ti
OS: Windows 10 Enterprise x64

AVS Script:


SetCacheMode(1)
SetMemoryMax(32000)
BlankClip(15, 1920, 1080, color=color_gray)
ConvertToYV12()
AddGrain(10)
Loop(10000000)
vsTTempSmooth(maxr=2)
Prefetch(12)




FPS (min | max | average): 97.29 | 211864 | 514.6
Process memory usage (max): 515 MiB
Thread count: 25
CPU usage (average): 93.2%


Test 2:
Type: Server
CPU: Intel Xeon Gold 6238R x2 56c/112th AVX512 2.20GHz
RAM: 128GB DDR4 (4x32 GB)
GPU: NVIDIA Quadro M5000
OS: Windows Server 2019 x64


AVS Script:


SetCacheMode(1)
SetMemoryMax(128000)
BlankClip(15, 1920, 1080, color=color_gray)
ConvertToYV12()
AddGrain(10)
Loop(10000000)
vsTTempSmooth(maxr=2)
Prefetch(112)




FPS (min | max | average): 140.1 | 6557 | 2948
Process memory usage (max): 2853 MiB
Thread count: 169
CPU usage (average): 82.5%


unfortunately Avisynth isn't numa node aware and it was therefore using only one CPU while the other was sitting there doing absolutely nothing.

https://i.imgur.com/ud9jG4f.png


This means that having so many threads in prefetch probably harmed performances, therefore the test has been repeated with a lower thread count pretending to be a single socket configuration (i.e Prefetch(56)):


FPS (min | max | average): 174.8 | 22862 | 3174
Process memory usage (max): 1652 MiB
Thread count: 113
CPU usage (average): 90.8%


and indeed it helped and achieved faster speeds.


Test 3:
Type: Workstation
CPU: Intel Xeon E-2640 v4 x2 20c/40th AVX2 2.40 GHz
RAM: 64GB DDR3 (8x8 GB)
GPU: NVIDIA Quadro P4000
OS: Windows 10 Enterprise x64


Same as before, being a dual socket configuration, Prefetch had to be set to 20 instead of 40 as Avisynth isn't numa node aware:

AVS Script:


SetCacheMode(1)
SetMemoryMax(64000)
BlankClip(15, 1920, 1080, color=color_gray)
ConvertToYV12()
AddGrain(10)
Loop(10000000)
vsTTempSmooth(maxr=2)
Prefetch(20)




FPS (min | max | average): 132.1 | 119904 | 974.1
Process memory usage (max): 758 MiB
Thread count: 61
CPU usage (average): 49.1%



This last one is gonna be a very interesting test 'cause I happen to have an old server with the very same CPU running in single socket configuration (although with less RAM :( ):


Test 4:
Type: Server
CPU: Intel Xeon E-2640 v4 10c/20th AVX2 2.40 GHz
RAM: 32GB DDR3 (4x8 GB)
GPU: NVIDIA Quadro P4000
OS: Windows Server 2016 x64

AVS Script:


SetCacheMode(1)
SetMemoryMax(32000)
BlankClip(15, 1920, 1080, color=color_gray)
ConvertToYV12()
AddGrain(10)
Loop(10000000)
vsTTempSmooth(maxr=2)
Prefetch(20)




FPS (min | max | average): 74.33 | 105215 | 799.6
Process memory usage (max): 765 MiB
Thread count: 41
CPU usage (average): 92.6%





So, in a nutshell:

514.6fps - (i7 5930K 6c/12th AVX2 3.50GHz)
799.6fps - (Intel Xeon E-2640 v4 10c/20th AVX2 2.40 GHz)**
974.1fps - (Intel Xeon E-2640 v4 x2 20c/40th AVX2 2.40 GHz)**
3174fps - (Intel Xeon Gold 6238R x2 56c/112th AVX512 2.20GHz)

**the speed bump above comes from the fact that the dual socket server has twice the RAM of the single socket one, NOT from the fact that it's dual socket as Avisynth isn't numa nodes aware and will only use 1 socket**

Note: AVSMeter is "confused" by dual socket configs in server and non server versions of Windows. On Server editions of Windows, it calculates the % of the single socket, so if 1 socket is 100% and the other is at 1% it will say 100% while it should be 51%. On non server versions of Windows like Windows 10 Enterprise, instead, it *sees* the % as the sum of the individual sockets, so cpu 0 and cpu 1, therefore if 1 socket is at 100% and the other is at 1% it says 51%.

[DTL]

"3174fps - (Intel Xeon Gold 6238R x2 56c/112th "

It looks this chip is hardly limited with very low RAM performance relative to number of cores. The math calculation in vsTTempSmooth is very simple and most of time it looks only move data from global process working set in system RAM to CPU caches. For such massive multicore chips it is better to have task with many small workunits about AVX512 registerfile size of 2 kBytes or may be L1D cache size of about 32 kBytes or a bit more and long computing inside this dataset. So performance of massive multicore chips is limited by number of RAM channels and DDR-generation and Xeon Gold 6238R with 6channels DDR4 easily enough wins. Unfortunately Windows Task Manager show busy/load state of core even if it really stall doing nothing and waiting for data from memory subsystem.
Though I thought 56Core Xeon should have >6 memory channels like 8. Or Gold is too cheap and only Platinum have 8 and more memory channels ? Like Xeon Platinum 9221 - https://en.wikichip.org/wiki/intel/xeon_platinum/9221 - supports up to twelve channels of DDR4-2933 memory. For 32 core only.

The really fast data compute accelerators already go away from this very old and slow DDR-RAM and use HBM RAM. Typical NVIDIA DCA like Tesla A100 have about 2 TB/s RAM speed @80 GB card and 12channels DDR4 at expensive Xeon only about 250 GB/s (0.25 TB/s). So poor even for sub $10000 price. Intel CPUs still about many computing with relatively small data workunits. HBM RAM expected in 'general purpose' CPUs setups in 202x years. So may be too early to upgrade to one more expensive Xeon with still very slow RAM.

For real more calculating load of core it may be used some 'gold pack' of AVS typical linear processing operations as 'AVS performance test script': 1. denoise with mvtools, 2. spatial correction with generalconvolution not very small 5x5 matrix, 3. downsize (rip for release creation)
So it will be not only test of speed of memory subsystem.
Something like:

LoadPlugin("mvtools2.dll")
LoadPlugin("AddGrainC.dll")

BlankClip(30, 1920, 1080, color=color_gray) # big number of frames in AVS source cache may make benefit of large L2/L3 cache CPUs ?
#BlankClip(30, 3840, 2160, color=color_gray)

ConvertToYV12()
ConvertBits(16) #? or in 8bit ? not very nice for 202x with HDR and 10bit enduser h.265
AddGrain(30) # or 50 or more ?
Loop(10000000)
#source preparation end here

tr=7
super=MSuper(pel=2) #medium quality
multi_vec=MAnalyse(super, multi=true, delta=tr, search=3, overlap=4, trymany=true) #exhaustive search, full 4x overlap, all predictors refining, lots of hard-to-SIMD algorithmic load
MDegrainN(super, multi_vec, tr, thSAD=500)

GeneralConvolution(0, "
-1 -2 -2 -2 -1
-2 -2 -3 -2 -2
-2 -3 80 -3 -2
-2 -2 -3 -2 -2
-1 -2 -2 -2 -1", auto=true, luma=true, chroma=false)

LanczosResize(1280, 720, taps=2) # need resize with support=2 or more to have equal load to SinPow/UD2 resizers

ConvertBits(10, dither=1)