RipBot264 v1.18.3 - Simple and easy to use GUI -> IPOD . PSP . CONSOLES . BLURAY

[Ryushin]

I've created a SMDegrain Lite package. It can be found here:
https://cloud.chrisdos.com/s/yiXwKbkMsQoNo6X

This is new creation using the original sources.

This does not contain any derivative works from Pauly Dunne / TDS / Retired@55.

Installation instructions are included.

I spun up a fresh Windows 10 VM and downloaded Ripbot264 and had it do it's updates and installed this package. Everything seems to work fine. I've included rlev11's and my scripts so you can try out both and see what works for you.

Please let me know if there are any problems. Enjoy.

[guest]

I've created a SMDegrain Lite package. It can be found here:
https://cloud.chrisdos.com/s/yiXwKbkMsQoNo6X

This is new creation using the original sources.

This does not contain any derivative works from Pauly Dunne / TDS / Retired@55.

Installation instructions are included.

I spun up a fresh Windows 10 VM and downloaded Ripbot264 and had it do it's updates and installed this package. Everything seems to work fine. I've included rlev11's and my scripts so you can try out both and see what works for you.

Please let me know if there are any problems. Enjoy.

Well, that didn't take you very long :(

Of course I have downloaded this to see what you've done, and of course it's a "copy" of my work, but at least you renamed everything, so thanks for that, at least.

I also had a look at the scripts, and they are pretty old, and there are some performance settings missing (every little bit helps).

Have you "consulted" with rlev11 on this as well ??

Also, thanks for the shout out in the .txt.

So see how easy that was, and yet Atak can't be bothered, but then why should he, when others are doing the ground work !!!

If only RB had proper DV support, and more codec & container options, like AV1, mp4, etc, when using x265.

[Ryushin]

Well, that didn't take you very long :(

Of course I have downloaded this to see what you've done, and of course it's a "copy" of my work, but at least you renamed everything, so thanks for that, at least.

Have you "consulted" with rlev11 on this as well ??


I'm afraid it's not a "copy" of your work. I specifically made sure of that. I sourced all the files from their original sources and I left them named the way they came from the source. I created new install and uninstall scripts using Atak's scripts as a derivative work and adding to it other pieces such as the Admin escalation. Perhaps the only thing that was copied was the order of the files in the custom avs script itself, but I'm not even sure of that as I went though so much learning when I creating my scripts learning SMDegrain. Either way I gave you credit for getting me into SMDegrain and for showing me the way.

Others have asked for rlev11's scripts and since he uses just the same few basic scripts as I do, I thought it best to include his as well.

So this is a lite version for sure. SMDegrain is a swiss army knife and the more advanced features need more packages. This one weighs in 7.8 MiB compressed and half that if I did not include Avisynth+. So this is very small and it's a good start.

Atak, if this works out, I'd like to petition for you to include this in your build, even though it uses custom scripts. Only change to your build is adding the Avisynth+ update to 3.7.3. It's small enough that it would only be a simple add on.

I hope the Lite version gets others using SMDegrain as this makes things simple.

[guest]

I'm afraid it's not a "copy" of your work. I specifically made sure of that. I sourced all the files from their original sources and I left them named the way they came from the source. I created new install and uninstall scripts using Atak's scripts as a derivative work and adding to it other pieces such as the Admin escalation. Perhaps the only thing that was copied was the order of the files in the custom avs script itself, but I'm not even sure of that as I went though so much learning when I creating my scripts learning SMDegrain. Either way I gave you credit for getting me into SMDegrain and for showing me the way.

Others have asked for rlev11's scripts and since he uses just the same few basic scripts as I do, I thought it best to include his as well.

So this is a lite version for sure. SMDegrain is a swiss army knife and the more advanced features need more packages. This one weighs in 7.8 MiB compressed and half that if I did not include Avisynth+. So this is very small and it's a good start.

Atak, if this works out, I'd like to petition for you to include this in your build, even though it uses custom scripts. Only change to your build is adding the Avisynth+ update to 2.7.3. It's small enough that it would only be a simple add on.

I hope the Lite version gets others using SMDegrain as this makes things simple.

OK, it's not a copy, copy, but I'm sure you know what I meant, you had the basics to work off.

I'm confused about the AVS build you've included, isn't 3.7.3 the latest official release ??

And again, you're asking Atak to add this to RB....if he did, I'm sure he would integrate the SMDegrain scripts.

I know that there must be 100's of users using my old stuff, by the huge amount of downloads I use to see.

[rlev11]

OK, it's not a copy, copy, but I'm sure you know what I meant, you had the basics to work off.

I'm confused about the AVS build you've included, isn't 3.7.3 the latest official release ??

And again, you're asking Atak to add this to RB....if he did, I'm sure he would integrate the SMDegrain scripts.

I know that there must be 100's of users using my old stuff, by the huge amount of downloads I use to see.

3.7.3 is what is included, it's a typo

[guest]

3.7.3 is what is included, it's a typo

I didn't check for myself, but typo's, when it comes to build versions, are a bit off putting.

Here's the VERY latest builds:-

https://gitlab.com/uvz/AviSynthPlus-Builds/

[stryker412]

Is there a way to edit something in the files to get the old selection option back for audio on batch files? I'd prefer to have the full range of dropdown options like we had in the past.

[guest]

Is there a way to edit something in the files to get the old selection option back for audio on batch files? I'd prefer to have the full range of dropdown options like we had in the past.

I doubt there are any options, but I agree, some of the recent changes are too complex, and unnecessary.

I think the only way you could get back to functions you like is to go back & use an earlier version of RB.

You could still use the current client & server, with the updates, and just use an older RipBot264.exe.

[rlev11]

Is there a way to edit something in the files to get the old selection option back for audio on batch files? I'd prefer to have the full range of dropdown options like we had in the past.

Looked around in the ini files and did not see anything that would do this, so it is probably coded in the exe file.

You could just make additional profiles and drop them in the preferences folder to do what you want. So say you want to convert everything in the batch to flac. Open up the default profile, so in my case [AUTO] 7.1 or 5.1 or 2.0.ini. and edit it how you want changing all the extensions to encode as flac, then save it off as flac.ini. Once in batch mode then it should be available as a choice in the drop down to do the conversions.

I know it is a round-a-bout way of doing it, but probably just adding a few extra profiles will do what you want.

I have an extra profile I use to convert eac3 with atmos to regular ac3 basically stripping off the atmos. if it is eac2 without atmos, I use the default profile to do an xcopy stream, but if it has atmos, I choose the profile to convert it.

[stryker412]

I doubt there are any options, but I agree, some of the recent changes are too complex, and unnecessary.

I think the only way you could get back to functions you like is to go back & use an earlier version of RB.

You could still use the current client & server, with the updates, and just use an older RipBot264.exe.

Any idea which version introduced that?

[guest]

Any idea which version introduced that?

No I don't, but it sounded like you did ??

You could try 1.26.0 from page 1...

or check here:-

https://www.videohelp.com/software/RipBot264

You'll notice that it states no more updates...

[rlev11]

Any idea which version introduced that?

Pretty sure the new audio stuff was added in 1.27.2

[stryker412]

Thanks, I rolled back for now. Perhaps Atak can post files for each revision from 1.26.0 onwards? We can them archive them in case they're needed in the future.

[guest]

Thanks, I rolled back for now. Perhaps Atak can post files for each revision from 1.26.0 onwards? We can them archive them in case they're needed in the future.

Fat chance of that ever happening, he can't even keep it up to date with the current versions, let alone what you just said.

[rlev11]

Thanks, I rolled back for now. Perhaps Atak can post files for each revision from 1.26.0 onwards? We can them archive them in case they're needed in the future.

Remembered I had made backup copies of all my folders since the bunch of updates that took place earlier this year.

This has all from 1.27.1 and 1.26.2 RipBot264.exe's along with the associated encoding server and encoding client exe's. Was all I could find on my systems.

https://drive.google.com/file/d/1VKNOJzJcltX-uF3t_FNS-YlD7LeYei3m/view?usp=sharing

[Ryushin]

So it is looking like this effort is having some positive results for me now. End result is now we no longer have to cripple the thread count of the high core cpu's down to 12-14 when doing 4k encoding regardless of using the built in mdegrain or in my case using a custom smdegrain script.

The bottleneck for 4k encoding is the avisynth max memory setting. For both mdegrain and my custom smdegrain scripts having SetMemoryMax(8192) restores performance on the high core cpu's in my results I am seeing. I tried 16384 and saw very little difference (1-3%) so no need to go above 8192 unless you need to for other reasons.

So now I will set prefetch threads to number of physical cores regardless on all servers, add in SetCacheMode() and SetMemoryMax(8192) to all my custom degraining scripts, and enjoy a little better performance over my previous method of crippling my R9's to 12 prefetch threads. Back to using my full 13 server distributed encoding farm, On my same GOT test I have been using, using SetMemoryMax(8192) was about 7% faster than prefetch=12 in total fps. This should also speed up 1080p and below a bit since I don't have to remember to switch the high core encoding servers to full prefetch (16 or 20 depending on the cpu)

Not sure how this may translate to other users workflows though, guess time will tell... but thanks for suggestions and input on this issue that has been bugging me for years now

I know I'm resurrecting this from August. I saved this link and it was on my todo list. I purchased a EPYC Turin 32 core processor for my server and I'll be installing the hardware in a couple of weeks and I'll need to adjust my settings again.

So you have found that there is no need to set prefetch threads any longer (RB should set it to the number of cores detected). Just add the following two options for both 4K and HD and you're good to go?
SetCacheMode()
SetMemoryMax(8192)

Any difference if SMT threading is enabled? Do you leave SMT enabled in your UEFI CPU settings?

[rlev11]

I know I'm resurrecting this from August. I saved this link and it was on my todo list. I purchased a EPYC Turin 32 core processor for my server and I'll be installing the hardware in a couple of weeks and I'll need to adjust my settings again.

So you have found that there is no need to set prefetch threads any longer (RB should set it to the number of cores detected). Just add the following two options for both 4K and HD and you're good to go?
SetCacheMode()
SetMemoryMax(8192)

Any difference if SMT threading is enabled? Do you leave SMT enabled in your UEFI CPU settings?

Going to be interesting to see how this works out. So yes, I no longer have to cripple the prefectch threads down to 12 on my high core machines when doing 4k. We do not either have to actually add in the 2 memory settings to the scripts manually. The last update from Atak he adds those settings into the command automatically.

Ripbot should automatically setup the prefetch and x265 threads settings automatically, however I start my servers with a .cmd script and just hardcode each one accordingly. I would just check the automatic detection and if it is not correct, just set it manually either in a script, or the client can be setup in the ripbot.ini file to hard code the threads correctly. I think the automatic detection is to set prefetch to physical count, and x264/x265 to logical count

Not sure how you plan to use this, but no telling how if you set 32 prefetch threads and 64 x265 threads if 8 gig memory for the cache is going to be enough. Might have to play and bump that up, but probably stay low enough if you have lower memory servers in the mix.

By SMT, I assume you are just referring to just plain hyperthreading being enabled. If so, then yest I have that enabled in the bios.

[Ryushin]

Going to be interesting to see how this works out. So yes, I no longer have to cripple the prefectch threads down to 12 on my high core machines when doing 4k. We do not either have to actually add in the 2 memory settings to the scripts manually. The last update from Atak he adds those settings into the command automatically.

Ripbot should automatically setup the prefetch and x265 threads settings automatically, however I start my servers with a .cmd script and just hardcode each one accordingly. I would just check the automatic detection and if it is not correct, just set it manually either in a script, or the client can be setup in the ripbot.ini file to hard code the threads correctly. I think the automatic detection is to set prefetch to physical count, and x264/x265 to logical count

Not sure how you plan to use this, but no telling how if you set 32 prefetch threads and 64 x265 threads if 8 gig memory for the cache is going to be enough. Might have to play and bump that up, but probably stay low enough if you have lower memory servers in the mix.

By SMT, I assume you are just referring to just plain hyperthreading being enabled. If so, then yest I have that enabled in the bios.

I'd like to just let RB handle it. If the default settings are working now then that would be perfect. Do we need to run more than one encoding server if we are wanting 100% CPU usage?

[rlev11]

I'd like to just let RB handle it. If the default settings are working now then that would be perfect. Do we need to run more than one encoding server if we are wanting 100% CPU usage?

I only ever run 1 encoding server per PC. Regardless of what shows in the encoding server usage graph, in Windows task manager I am showing over 90% usage for the Ripbot processes. My thing is also especially running distributed encoding is if you are essentially running 2 servers at half speed (or just a tick more), are we really saving anything on the total time. You could end up with the last 2 chunks both running at half speed on the same machine, while a faster machine sits idle.

Now like I said, a 32 physical core machine may be a totally different animal than my experience is with setting up how best to utilize it to it's full potential.

My thinking is that you may end up in the end the best setup may be to essentially cut that in half, and run dual encoding servers on the new Epyc as dual 16 core 32 thread machines. This would be easy to setup reverting back to using an affinity mask to setup each machine as a 16core 32 thread server for ripbot. That would also probably eliminate the need to play with the prefetch memory cache. I did some tests doing that on the 7950x back a bit running essentially as a dual 7700x, and in some limited runs, 4k was a touch faster with dual servers, and 1080p was a few fps slower, but close to similar.

[guest]

The last update from Atak he adds those settings into the command automatically.

Ripbot should automatically setup the prefetch and x265 threads settings automatically,

I haven't used RB for several months, and there have been some updates in that time, so after your comment on the auto settings, I tried it on my little 2 core laptop, and sure enough, this is what was auto generated :-

#MT
SetCacheMode(CACHE_FAST_START)
SetMemoryMax(8192)
#PREFETCH_LIMIT=12

Yet to be determined if this actually does work, and change with different PC's....

I DID mention when SMDegrain_Lite was introduced that the scripts were lacking some "performance" calls, but now this might "do the trick".

There is also a very important setting that needs to be changed when using any SMDegrain script, and I'm not sure if this has been mentioned.

Also, @Ryushin, does the EPYC have "numa" ?? if so that might be able to be used with DE to some advantage.

[Ryushin]

I only ever run 1 encoding server per PC. Regardless of what shows in the encoding server usage graph, in Windows task manager I am showing over 90% usage for the Ripbot processes. My thing is also especially running distributed encoding is if you are essentially running 2 servers at half speed (or just a tick more), are we really saving anything on the total time. You could end up with the last 2 chunks both running at half speed on the same machine, while a faster machine sits idle.

Now like I said, a 32 physical core machine may be a totally different animal than my experience is with setting up how best to utilize it to it's full potential.

My thinking is that you may end up in the end the best setup may be to essentially cut that in half, and run dual encoding servers on the new Epyc as dual 16 core 32 thread machines. This would be easy to setup reverting back to using an affinity mask to setup each machine as a 16core 32 thread server for ripbot. That would also probably eliminate the need to play with the prefetch memory cache. I did some tests doing that on the 7950x back a bit running essentially as a dual 7700x, and in some limited runs, 4k was a touch faster with dual servers, and 1080p was a few fps slower, but close to similar.

This is good to know. Since I pass through the cores from Linux to the RB virtual machine, I can mix and match. Since my server does other work other then just encoding, I'm thinking of just passing 24 cores (48 threads) to the VM and reserving 8 cores for server work. This still will give the VM 48 CPUs. If I can get it to run 90% then that will be ideal. I'll see what happens.



There is also a very important setting that needs to be changed when using any SMDegrain script, and I'm not sure if this has been mentioned.

Also, @Ryushin, does the EPYC have "numa" ?? if so that might be able to be used with DE to some advantage.

I've added instructions in the SMDegrain_Lite package to disable "Limit to the following threads".

EPYC absolutely has NUMA.
Motherboard: https://www.supermicro.com/en/products/motherboard/h13ssl-nt
Processor: https://www.amd.com/en/products/processors/server/epyc/9005-series/amd-epyc-9355p.html
Memory: 6 x Micron MTC20F2085S1RC64BD2 Memory 32GB DDR5 6400MHz RDIMM

My testing with NUMA with RB showed no real performance gains compared to using threads. Since I'm using a virtual machine and my server is doing other workloads, using NUMA would not give me the flexibility to fully utilize the resources of the server. So threaded is what I'll need to use.

[Atak_Snajpera]

Would be a problem for you guys using Distributed Encoding mode if I replaced PC name to IP address in UNC paths?

\\192.168.0.1\Ripbot264temp instead of \\MY-PC\Ripbot264temp

Is there any disadvantage of using IP address in path? What are your thoughts?

ChatGPT4 gives this answer

When deciding whether to use an IP address or a local computer name in a UNC (Universal Naming Convention) path, consider the following:

Local Computer Name:
Using the computer name (e.g., \\ComputerName\SharedFolder) is generally preferred because it is more readable and easier to remember.
It allows for easier management, especially in environments where IP addresses may change (e.g., DHCP environments).

IP Address:
Using an IP address (e.g., \\192.168.1.10\SharedFolder) can be useful in situations where the computer name cannot be resolved (e.g., DNS issues).
It may be necessary in some network configurations or when dealing with legacy systems.

In most cases, using the local computer name is recommended for its simplicity and ease of use. However, if you encounter issues with name resolution, using the IP address can be a good alternative.

[rlev11]

Would be a problem for you guys using Distributed Encoding mode if I replaced PC name to IP address in UNC paths?

\\192.168.0.1\Ripbot264temp instead of \\MY-PC\Ripbot264temp

Is there any disadvantage of using IP address in path? What are your thoughts?

ChatGPT4 gives this answer

I don't see any disadvantage in making that change.
In fact I used to have a machine I wanted to add to the mix, but I kept an active vpn connection on that one, and I had issues with the name resolution back to the client. \\ip\share would have connected back just fine. I don't remember if I found a work around or not.

I say go for it!!

[Emulgator]

Go for direct IP addressing +1, as soon as I adressed storage servers manually here: even for control, not bulk data the whole rig became snappier.

[guest]

@ Ryushin,

So why an EPYC, and not Threadripper or Xeon ?

@ rlev11,

Yes, that last chunk is the worst thing with DE, and inevitably the last is is always the biggest, and 99% of the time ends up being on the slowest PC in the "pool" :(
And what exacerbates the situation more, is if the slowest PC is way slower than the rest :(
In a perfect world all the PC's need to be the same.
It would be nice if the last chunk was the smallest.

@ Atak_Snajpera,

Actually shocked that you're asking the "community" about a possible change to DE, you generally make changes, release them without much or any explanation or instruction, and the user has to figure it out for themselves.
If this change is implemented, and provides better stability & connectivity, then by all means.

[guest]

I just updated RB on my 7950X, and I noticed that this setting doesn't change depending on the PC used, or the video that's loaded...

SetCacheMode(CACHE_FAST_START)
SetMemoryMax(8192)
#PREFETCH_LIMIT=12

So if this call is in the custom script, which settings are used ??

SetCacheMode()
SetMemoryMax(20480)

So now this seems to be uneditable, could it be made to be ??

The Prefetch_Limit setting can be changed, so why not the SetMemoryMax, on the same page.

[Boulder]

Is there any disadvantage of using IP address in path? What are your thoughts?


IP addresses are always the best option. For example, I cannot access my media player over the local network using anything else than the static IP address. It took me one evening to figure it out :devil:

[Ryushin]

Would be a problem for you guys using Distributed Encoding mode if I replaced PC name to IP address in UNC paths?

\\192.168.0.1\Ripbot264temp instead of \\MY-PC\Ripbot264temp

Is there any disadvantage of using IP address in path? What are your thoughts?


IP Addresses are strongly preferred. Though it would be nice to have something like a hosts file that you can associate a name with the IP address.

[rlev11]

IP Addresses are strongly preferred. Though it would be nice to have something like a hosts file that you can associate a name with the IP address.

I would think that the only visible change would be in the encoding server window, the ffmpeg command line would show call backs to \\client ip instead of \\client name

[stryker412]

Remembered I had made backup copies of all my folders since the bunch of updates that took place earlier this year.

This has all from 1.27.1 and 1.26.2 RipBot264.exe's along with the associated encoding server and encoding client exe's. Was all I could find on my systems.

https://drive.google.com/file/d/1VKNOJzJcltX-uF3t_FNS-YlD7LeYei3m/view?usp=sharing

Awesome, thank you!

[rlev11]

Would be a problem for you guys using Distributed Encoding mode if I replaced PC name to IP address in UNC paths?

\\192.168.0.1\Ripbot264temp instead of \\MY-PC\Ripbot264temp

Is there any disadvantage of using IP address in path? What are your thoughts?

ChatGPT4 gives this answer

No issues so far with the new encodignclient.exe which uses ip address instead of hostname for the callback from the encoding servers

[Ryushin]

I set up a new job to run and it failed. I got an error about info.txt missing. Went into the job and found info1.txt. Renamed it to info.txt and the job ran fine form that point on. Could be a new bug from the recent changes.

[rlev11]

I set up a new job to run and it failed. I got an error about info.txt missing. Went into the job and found info1.txt. Renamed it to info.txt and the job ran fine form that point on. Could be a new bug from the recent changes.

Anytime I have had that info.txt error (and i have talked about this in the past) was if I did a crop and using smdegrain and the aspect ratio was just a little off of a standard. Usually adjusting the horizontal pixels just a bit clears it up. I have found that once you do the crop, if you hit preview script, and the mediaplayer window at least opens without an error, the encode will go through

I only did a few quick encodes so far with the updated encodingclient with no issues. will be able to fire up the whole farm in a couple days to do a more thorough test.

Pretty sure the only file that changed was encodingclient.exe. Looked like everything else remained at the same version number.

I would try the same video again and just use mdegrain and see if it at least starts encoding as a test.

[stryker412]

I know the topic came up earlier. I now have a 5700X3D CPU. When running Ripbot my CPU temps are around 70-73c. Is there anything I can do to bring temps down a bit without sacrificing too much speed on encodes?

[rlev11]

I know the topic came up earlier. I now have a 5700X3D CPU. When running Ripbot my CPU temps are around 70-73c. Is there anything I can do to bring temps down a bit without sacrificing too much speed on encodes?


See my post from 27 November about adding in the boost setting to the control panel power settings
https://forum.doom9.org/showthread.php?p=2010572#post2010572

[Ryushin]

Anytime I have had that info.txt error (and i have talked about this in the past) was if I did a crop and using smdegrain and the aspect ratio was just a little off of a standard. Usually adjusting the horizontal pixels just a bit clears it up. I have found that once you do the crop, if you hit preview script, and the mediaplayer window at least opens without an error, the encode will go through

I only did a few quick encodes so far with the updated encodingclient with no issues. will be able to fire up the whole farm in a couple days to do a more thorough test.

Pretty sure the only file that changed was encodingclient.exe. Looked like everything else remained at the same version number.

I would try the same video again and just use mdegrain and see if it at least starts encoding as a test.

Odd then. Never saw RB make a info<jobnumber>.txt file before, it was always info.txt. Didn't do any cropping this time. Mediaplayer played it fine as well as I was debugging the problem. So that is what led me to believe it was a code change.

[stryker412]

See my post from 27 November about adding in the boost setting to the control panel power settings
https://forum.doom9.org/showthread.php?p=2010572#post2010572

Thanks, my temps dropped by 20 degrees when disabling it. I'll have to enable/reenable each time as I do use this machine to game as well.

[rlev11]

Thanks, my temps dropped by 20 degrees when disabling it. I'll have to enable/reenable each time as I do use this machine to game as well.

So what I did on all my machines was make the balanced power plan setup for boost disabled, then make the power plan for high performance setup the way I want it for regular use (I'll set minimum processor speed to whatever it is on the balanced plan either 0 or 5%).

then I made two .bat files on my desktop, balanced.bat and highperf.bat. for balanced.bat I used this command
powercfg /s 381b4222-f694-41f0-9685-ff5bb260df2e

For Highperf I used this command:
powercfg /s 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c

These worked on all my machines, so the strings should be the same. A cmd of powercfg /l will list all your power plans along with the string to use if there is a difference.

Now I can switch on the fly at anytime just by double clicking the power plan i need at the time.

[Ryushin]

I only ever run 1 encoding server per PC. Regardless of what shows in the encoding server usage graph, in Windows task manager I am showing over 90% usage for the Ripbot processes. My thing is also especially running distributed encoding is if you are essentially running 2 servers at half speed (or just a tick more), are we really saving anything on the total time. You could end up with the last 2 chunks both running at half speed on the same machine, while a faster machine sits idle.

Now like I said, a 32 physical core machine may be a totally different animal than my experience is with setting up how best to utilize it to it's full potential.

My thinking is that you may end up in the end the best setup may be to essentially cut that in half, and run dual encoding servers on the new Epyc as dual 16 core 32 thread machines. This would be easy to setup reverting back to using an affinity mask to setup each machine as a 16core 32 thread server for ripbot. That would also probably eliminate the need to play with the prefetch memory cache. I did some tests doing that on the 7950x back a bit running essentially as a dual 7700x, and in some limited runs, 4k was a touch faster with dual servers, and 1080p was a few fps slower, but close to similar.

I've been doing some testing and it is indeed possible to just have a single encoding server which is nice with high CPU core processors. I gave 24 cores and 48 threads to my VM. I found that having 24 avisynth threads and 48 x265 threads I would have some threads being starved for data. Changing avisynth threads to 32 kept the 48 x265 threads fully loaded and I was seeing about 90-95% processor usage.

So everything is happy now. Full speed ahead.

[rlev11]

I've been doing some testing and it is indeed possible to just have a single encoding server which is nice with high CPU core processors. I gave 24 cores and 48 threads to my VM. I found that having 24 avisynth threads and 48 x265 threads I would have some threads being starved for data. Changing avisynth threads to 32 kept the 48 x265 threads fully loaded and I was seeing about 90-95% processor usage.

So everything is happy now. Full speed ahead.

My only real question was if the cache value that is set now by default to 8096 to fix the 16 cores 4k encoding, was if that would be enough when you threw a bunch more cores into the mix. I am assuming you did not or you would have mentioned that??

[Ryushin]

My only real question was if the cache value that is set now by default to 8096 to fix the 16 cores 4k encoding, was if that would be enough when you threw a bunch more cores into the mix. I am assuming you did not or you would have mentioned that??

Actually, I was looking at just the new EPYC CPU. I'm not doing any 4K content yet, just HD with my SMDegrain Medium script.

Looking at the 16 core CPU, the same problem was happening as it had 16 avisynth threads and 32 h265 threads. Raising the avisynth threads to 20 seems to keep the threads filled and not starving. I'll have to monitor the 16 core server a bit more as I have Topaz Video AI also running on it, but that mostly uses the GPU.

When I get these jobs done in a week or so, I'll take my 4K Blade Runner clip that I used for testing last time and run some benchmarks with it and see what the results are

The default 8096 sure seems to have fixed the problem of having to run multiple encoding servers to fully utilize the CPU, so I like that. Having to run only a single enconding server is nice.

Not sure if I fully understand your question though. I haven't tested any 4K yet so I don't know if I can fully answer it yet.

[Atak_Snajpera]

Actually, I was looking at just the new EPYC CPU. I'm not doing any 4K content yet, just HD with my SMDegrain Medium script.

Looking at the 16 core CPU, the same problem was happening as it had 16 avisynth threads and 32 h265 threads. Raising the avisynth threads to 20 seems to keep the threads filled and not starving. I'll have to monitor the 16 core server a bit more as I have Topaz Video AI also running on it, but that mostly uses the GPU.

When I get these jobs done in a week or so, I'll take my 4K Blade Runner clip that I used for testing last time and run some benchmarks with it and see what the results are

The default 8096 sure seems to have fixed the problem of having to run multiple encoding servers to fully utilize the CPU, so I like that. Having to run only a single enconding server is nice.

Not sure if I fully understand your question though. I haven't tested any 4K yet so I don't know if I can fully answer it yet.

if you had 32 core cpu and 4k video would you also have to raise memory from 8192 to 16384 to avoid poor cpu utilization like in the past with 4096?

[rlev11]

if you had 32 core cpu and 4k video would you also have to raise memory from 8192 to 16384 to avoid poor cpu utilization like in the past with 4096?

Exactly what I am getting at, the 8192 may not be enough once we start hitting "Ludicrous Core :D" counts when doing 4k. The only issue I see if we have to bump that up for one high core machine, that cache memory value also appears to get sent to all the servers in the farm, so they all have to have enough memory to support the higher cache value. That was why it was good to find the lowest value before we got diminished returns and came up with the 8192.

Perhaps when doing an update in the future, it may be beneficial to add that cache memory value to the main ripbot settings. Default it at 8192, but allow us to change that. A tool tip for what it does and why to change it would also be helpful

[Atak_Snajpera]

or maybe it should scale automatically with number of cores? That's why I'm asking.

[Ryushin]

Well, I spoke too soon. It looks like if I'm just doing pure x265 with no filters I'll only get about 40% processor usage. Running two encoding servers with 24 cores each gets it to 85% CPU. I tried both 8192 and 16384. I'm only doing 2K right, now, 2560x1440. Same thing was happening with just x265 with the 16 core processor and I had to move back to using two encoding servers.

I should be able to test my Blade Runner 4K clip with and without SMDegrain in a couple of days.

[rlev11]

or maybe it should scale automatically with number of cores? That's why I'm asking.

Would it be possible to set the cache individually on the encoding server side for each server, or does the same cache setting need to go out to each distributed server?

My only concern would be say you have a 32 core as your client. Just to pick a number, say 16 gig is optimum for the cache on that. If that 16 gig cache number is sent to a lower powered server, say an 8 core with only 16 gig total memory, whats going to happen to the 8 core machine that gets the memory pegged just for the encoding cache set at 16 gig?? A machine having to go back to using a swap file even with ssd's would not be a good thing IMO.

Scaling automatically (and we still need to wait and see if we need to scale the cache up with higher than 16 core machines while doing full frame 4k) would be ideal, but my thinking is that the cores and especially total individual memory of an entire distributed encoding farm would need to be taken into consideration and might be difficult to accomplish.

[slalom]

I understood that the cache was set automatically, according to the number of cpus. What is the problem you have?

[rlev11]

I understood that the cache was set automatically, according to the number of cpus. What is the problem you have?

I believe, and i may be incorrect, that the update Atak did where he adds in automatically the "SetMemoryMax(8192)" to the avisynth command which fixes the performance issue doing 4k on 16 core machines is set on everything now, it is not based on any cpu count calculation.

We are discussing now that Ryushin is running an Epyc system and creating a much larger core count computer if that 8192 for the cache is going to be enough with all the added cores and avisynth-prefetch-threads. Before figuring out what was going on and the cache update, anything above 12 for the avisynth-prefetch-threads just killed 4k performance on 16 core computers. It may or not be depending on some future testing. If it needs to be bumped up, just throwing out ideas on how the best way to handle that will be in a distributed environment.

[slalom]

Before figuring out what was going on and the cache update, anything above 12 for the avisynth-prefetch-threads just killed 4k performance on 16 core computers. It may or not be depending on some future testing. If it needs to be bumped up, just throwing out ideas on how the best way to handle that will be in a distributed environment.
So in his case, the program maxed the memory usage of the system?

[rlev11]

So in his case, the program maxed the memory usage of the system?

Not really maxing the memory, it may max out the avisynth cache when doing 4k encoding.

Setting the cache to 8192 works for avisynth-prefetch-threads of 16 when doing 4k so we don't have to "cripple" the 16 core ryzens either using a lower prefetch thread of 12 or using an affinity mask to only use 12 cores for ripbot. It is yet to be determined if that will need to be a higher number with avisynth-prefetch-threads of 24 or 32 on a higher core cpu.