RipBot264 v1.18.3 - Simple and easy to use GUI -> IPOD . PSP . CONSOLES . BLURAY

[supersnakeyez]

Yes it is :- (as of 08-06-2024)

I would rely on your signature showing version available but no longer see it there. Thx for confirming though

[Guest]

I would rely on your signature showing version available but no longer see it there. Thx for confirming though

Have no idea why that disappeared, but thank you for letting me know.

However, updates are few & far between. :(

I might have been hacked, as most of my info was changed.

[supersnakeyez]

Have no idea why that disappeared, but thank you for letting me know.

However, updates are few & far between. :(

I might have been hacked, as most of my info was changed.

damn that sucks if that happened to you. hope all is good tho

[mikehbkwm]

Have a weird situation. I've been encoding DVD's (MPEG2) or Blu-ray (h.264) and upscaling them to HEVC 4K. Sometimes there's no issues when I playback in Plex, but other times when I go to play back the movie Plex states: Playback Stopped, Not enough disk space to convert this item. Utilizing Apple TV's as the client and never had this issue before.

Another example: Return of the Jedi (2006 Limited Edition DVD - GOUT) - upscaled to 4K HEVC with subtitles, and plays back in Plex no problems.
Star Wars (2006 Limited Edition DVD - GOUT) - upscaled to 4K HEVC with subtitles and get the playback stopped message.

I'm truly baffled as it doesn't make sense why one encode would work and the next one does not.

Any thoughts on this?

RipBot264 v1.27.3 (08-06-24)

[Guest]

Have a weird situation. I've been encoding DVD's (MPEG2) or Blu-ray (h.264) and upscaling them to HEVC 4K. Sometimes there's no issues when I playback in Plex, but other times when I go to play back the movie Plex states: Playback Stopped, Not enough disk space to convert this item. Utilizing Apple TV's as the client and never had this issue before.

Another example: Return of the Jedi (2006 Limited Edition DVD - GOUT) - upscaled to 4K HEVC with subtitles, and plays back in Plex no problems.
Star Wars (2006 Limited Edition DVD - GOUT) - upscaled to 4K HEVC with subtitles and get the playback stopped message.

I'm truly baffled as it doesn't make sense why one encode would work and the next one does not.

Any thoughts on this?

RipBot264 v1.27.3 (08-06-24)

Not sure why you're upscaling DVD's to 4K ?!?!?!, but each to their own, I do some stuff that others don't understand !!!!

Could it be PLEX itself ???, it must be trying to decode stuff, and it's clearly running out of room, is there a setting for this "Not enough disk space" ??

https://www.youtube.com/watch?v=ZUfyK4ShsUk

I don't use PLEX, so good luck.

[Guest]

I would be very interested to know how many Threadripper owners are using RipBot successfully, with Distributed Encoding mainly.

Can it be "loaded up" to encode @ close to 100% CPU usage.

Anyone that own's a Ryzen 7950X (and 5950X's have a similar issue) will know what I'm on about, need to have specific encoder commands to get the best out of them, surely a CPU with more than 16 cores will suffer the same "dropping off a cliff" with especially 4K encodes.

I would suggest that the upcoming 9950X's will also be the same.

All and any comments will be appreciated :)

[rlev11]

So a couple years ago I ran tests with doing some 4k encoding on my 5950x and 7950x to see what the best way the "cripple" them was. As a review, a 7950x running full ran at 4.5fps, using aviprefetch set to 12 was 8.27fps, using affinity mask to run essentially as a 7900x was 8.41fps. Actually the best way was to using affinity mask to split the 7950x in half running essentially as 2 separate 8 core machines which had a combined fps as 9.73. I don;t remember if this was with mdegrain or smdegrain, but i don't think the percentages changed any.

So my GUESS would be for a threadripper with 24 or 32 actual cores, the best way to fully load them up would be to run them broken up into either 8 or 12 cpu separate servers when doing 4k encodes using affinity mask and use distributed encoding.

My testing also indicated when doing 1080p and below, that separating out the 16 core Ryzens into dual 8 cores had a very significant HIT in performance (about 30% lower). So I would think one would want to run a large core threadripper with all cores active, and probably would see a performance gain running multiple servers in this configuration

[Atak_Snajpera]

can't wait to see the same tests with latest Zen5 cpus. I wonder if larger L3 cache in TR would help in this case?

[rlev11]

can't wait to see the same tests with latest Zen5 cpus.

As soon as they are released and drop from the initial above retail price I will find out.

I also found that encoding 4k with the 7950x with avx-512 in the encoding profile helps a little bit if the cpu is running all 16 avi-synth prefetch threads, but still no where near the performance with the avi-synth prefetch threads set to 12. It's just not AMD, my 14700 I had to adjust the prefetch threads down to 12 as well even with the P and E cores adding up to 20.

Like we have surmised, there is some buffer that just gets overloaded with all the additional bits in a 4k stream.

[Atak_Snajpera]

Do you see the same problem even with wide aspect 4k content. For example 3840x1600 (2.40:1)

[rlev11]

Do you see the same problem even with wide aspect 4k content. For example 3840x1600 (2.40:1)

yes , but it is not as bad. Issue is worse at a full 3840x2160 and does get better as the aspect ratio gets "thinner", but not to the full speed of a crippled 16 core cpu. I always do an autocrop, so i don't encode black space.. I don't remember if the level of degraining plays any part in it as well.

I haven't really done any real bench marking lately as those numbers were from Jan 2023. Now I'm just noticing if I forget to do my 4k encoding server script that the numbers look down, then a quick swap of the server and things are back to normal.

I'll see if over the next week or so I can run some tests again on different aspect ratio clips with different parameters of both mdegrain and smdegrain scripts. One thing I really haven't tested is if not adding any degraining, just crop and go, if that makes any difference. Since it is avisynth prefetch threads, it might not.

It would be nice to figure out eventually exactly what is causing the issue, even if it can't easily be fixed.

[Guest]

yes , but it is not as bad. Issue is worse at a full 3840x2160 and does get better as the aspect ratio gets "thinner", but not to the full speed of a crippled 16 core cpu. I always do an autocrop, so i don't encode black space.. I don't remember if the level of degraining plays any part in it as well.

I haven't really done any real bench marking lately as those numbers were from Jan 2023. Now I'm just noticing if I forget to do my 4k encoding server script that the numbers look down, then a quick swap of the server and things are back to normal.

That is why I have setup 2 RipBot's, one for 4K and its settings & scripts, and another for 1080p and under, each uses a separate temp & encodes folder works well, and you don't have to "remember" to swap thing between different resolutions. And even tho I haven't tested this, you could possibly run 2 instances of RB....

I'll see if over the next week or so I can run some tests again on different aspect ratio clips with different parameters of both mdegrain and smdegrain scripts. One thing I really haven't tested is if not adding any degraining, just crop and go, if that makes any difference. Since it is avisynth prefetch threads, it might not.

It would be nice to figure out eventually exactly what is causing the issue, even if it can't easily be fixed.

I just don't understand the purpose of cropping !!!

I did a couple of basic tests just the other day, and did a crop & no crop sample 4K run, and when played on my TV, it looks exactly the same !!!!

And my understanding is, don't Threadrippers have some form of NUMA, where the Ryzen's don't, per se.

If that's the case, then they can be "split" into 2 separate encoders.

But as I said to rlev11 in a PM the other day, for the price of even a budget Threadripper 7960X, you could almost have 3 basic 7950X's, and that would way more productive for ppl using DE

[rlev11]

So got a chance to run some test in the most controlled manner i could think of. Did a test with a GOT episode at 3840x2160, ST Strange New Worlds at 3840x1600, and The Wire at 1920x1080 just to see how a crippled 7950x reacts to 1080p. Each 7950x fps score was on Chunk 5 and a comparison 7900x score from Chunk6. Dual server 7950x scores were from chunks 5 and 6 combined. Resultant spreadsheet is linked below.

My take on what I saw this time. AVX512 on the 7900 and 7950 makes very little difference, and in most cases actually hinders performance a bit.
Anything more than 12 avisynth-prefetch threads just kills 3840x2160, but does get much less a difference at 3840x1600. I would imagine the percentages difference for say 1:1.85 ,1:2.0, and 1:2.35 would fall in line somewhere in between.
SMdegrain is just so much faster than mdegrain3 especially in anything 4k.
Running a 7950 as a pseudo 7900 using affinity mask is NOT the way to go.
Though the actual numbers will be different, I would imagine that the best way to cripple a 7950x would also be the best way to cripple a 5950x. Intels would probably be best using the lower avisynth prefetch threads of 12. No idea how affinity mask would even work with the whole E and P cores
Was actually surprised by the numbers of turning a 7950x into a pseudo dual server 7700x (column G) in Ripbot.


https://docs.google.com/spreadsheets/d/1xhiNHSmiG73ZzqeeJW3eY5AJr0AhEvLD/edit?usp=sharing&ouid=106754144954314069404&rtpof=true&sd=true

[Guest]

So got a chance to run some test in the most controlled manner i could think of. Did a test with a GOT episode at 3840x2160, ST Strange New Worlds at 3840x1600, and The Wire at 1920x1080 just to see how a crippled 7950x reacts to 1080p. Each 7950x fps score was on Chunk 5 and a comparison 7900x score from Chunk6. Dual server 7950x scores were from chunks 5 and 6 combined. Resultant spreadsheet is linked below.

My take on what I saw this time. AVX512 on the 7900 and 7950 makes very little difference, and in most cases actually hinders performance a bit.
Anything more than 12 avisynth-prefetch threads just kills 3840x2160, but does get much less a difference at 3840x1600. I would imagine the percentages difference for say 1:1.85 ,1:2.0, and 1:2.35 would fall in line somewhere in between.
SMdegrain is just so much faster than mdegrain3 especially in anything 4k.
Running a 7950 as a pseudo 7900 using affinity mask is NOT the way to go.
Though the actual numbers will be different, I would imagine that the best way to cripple a 7950x would also be the best way to cripple a 5950x. Intels would probably be best using the lower avisynth prefetch threads of 12. No idea how affinity mask would even work with the whole E and P cores
Was actually surprised by the numbers of turning a 7950x into a pseudo dual server 7700x (column G) in Ripbot.


https://docs.google.com/spreadsheets/d/1xhiNHSmiG73ZzqeeJW3eY5AJr0AhEvLD/edit?usp=sharing&ouid=106754144954314069404&rtpof=true&sd=true

Interesting results, and a bit of time spent doing that :)

So all these tests were using DE on a single PC ??

Are the MDegrain & SMDegrain settings as close to each other as you could get ??? I'm thinking MD3 is a lot "stronger than SMD tr=3....and did you change the mutli thread settings for MD on page 1 ??

Also, I am currently using prefetch threads of 14, on all doing 4K encodes, even my 13900KF.

===============================

On another topic, I have watched several YT reviews of the new 9600X's & 9700X's, not looking too good, let's hope the "big" one's are on point, otherwise it could be a waste of money, money you could spend on a Threadripper, for example....or a couple more 7950X's.

[rlev11]

Interesting results, and a bit of time spent doing that :)

So all these tests were using DE on a single PC ??

Are the MDegrain & SMDegrain settings as close to each other as you could get ??? I'm thinking MD3 is a lot "stronger than SMD tr=3....and did you change the mutli thread settings for MD on page 1 ??

Also, I am currently using prefetch threads of 14, on all doing 4K encodes, even my 13900KF.

===============================

On another topic, I have watched several YT reviews of the new 9600X's & 9700X's, not looking too good, let's hope the "big" one's are on point, otherwise it could be a waste of money, money you could spend on a Threadripper, for example....or a couple more 7950X's.

All test runs on same computer which is my main 7950x using distributed encoding. My 7900x was in the mix for all the runs as well.

The mdegrain3 was just choosing it in the menu with strength 200, no other custom settings. This is what I used to use before switching to smdegrain for light degraning on mostly clean source material. The smdegrain custom script is what I use now on similar material and the results are comparable to my eyes.

Interesting on the 14 prefetch threads, Pretty sure back a while ago I determined that 12 was the magic number, but I will try 14 and see if that works a little better now.

Main point was to answer Atak's question a few threads above about 4k 2.40 aspect ration and how it compared. I knew what the answer was from all my encoding I have done over the years, but wanted to confirm with some hard numbers. I don't know enough about avisynth multi threading to really dig into where the bottleneck is, but maybe someone else can figure it out. It's not really a big deal to switch the encoding server on the 16 core ryzens when doing 4k using a .bat file, in fact my 3 5950's and 1 3950 that I have in my encoding farm have a startup script that loads 12 prefetch threads and I just leave them like that for everything. I only switch my main client 7950 when I do a 4k run.

[Guest]

All test runs on same computer which is my main 7950x using distributed encoding. My 7900x was in the mix for all the runs as well.

The mdegrain3 was just choosing it in the menu with strength 200, no other custom settings. This is what I used to use before switching to smdegrain for light degraning on mostly clean source material. The smdegrain custom script is what I use now on similar material and the results are comparable to my eyes.

Interesting on the 14 prefetch threads, Pretty sure back a while ago I determined that 12 was the magic number, but I will try 14 and see if that works a little better now.

Main point was to answer Atak's question a few threads above about 4k 2.40 aspect ration and how it compared. I knew what the answer was from all my encoding I have done over the years, but wanted to confirm with some hard numbers. I don't know enough about avisynth multi threading to really dig into where the bottleneck is, but maybe someone else can figure it out. It's not really a big deal to switch the encoding server on the 16 core ryzens when doing 4k using a .bat file, in fact my 3 5950's and 1 3950 that I have in my encoding farm have a startup script that loads 12 prefetch threads and I just leave them like that for everything. I only switch my main client 7950 when I do a 4k run.

So in reference to this post:-

https://forum.doom9.org/showthread.php?p=2005202#post2005202

Why crop ??

[rlev11]

So in reference to this post:-

https://forum.doom9.org/showthread.php?p=2005202#post2005202

Why crop ??

Pretty simple, Why encode the bits in the black bars. Just re-ran the strange new worlds run (Block D-17 on the spreadsheet). Doing an autocrop reducing the 3840x2160 original file to 3840x1600, at 1600 I had speeds of 18.11fps for the 7950x and 15.5 for the 7900x. The same chunks without an autocrop at 2160 they dropped to 14.18 and 12.57. That's about a 20% drop in performance...

[Guest]

Pretty simple, Why encode the bits in the black bars. Just re-ran the strange new worlds run (Block D-17 on the spreadsheet). Doing an autocrop reducing the 3840x2160 original file to 3840x1600, at 1600 I had speeds of 18.11fps for the 7950x and 15.5 for the 7900x. The same chunks without an autocrop at 2160 they dropped to 14.18 and 12.57. That's about a 20% drop in performance...

OK, so the way I understand that is, it removes the black bars for the encoding process, but looks exactly the same on a TV ???

[rlev11]

Correct, It will only encode the bits that are needed, regardless of what aspect ratio it is, even a 4:3 encode will not include the left and right bars. TVs don't care, it will always just center whatever is sent to it

[Guest]

Correct, It will only encode the bits that are needed, regardless of what aspect ratio it is, even a 4:3 encode will not include the left and right bars. TVs don't care, it will always just center whatever is sent to it

Thanks very much for that, like I said I very rarely crop, and if I do, I rely on the autocrop function...

So I wiil defintely start using that, and see what happens.

[rlev11]

The only issue I have found with an autocrop, and this only affects it when using SMdegrain (never an issue with just Mdegrain) is that occasionally the autocrop will come up with an aspect ratio that is just a bit off of a standard aspect ratio and when it goes to actually start the encode there will be an error saying something like info.txt could not be found. In this case usually just adjusting the aspect ratio by removing a few bits on top and bottom (or left and right) will clear that up. Over the last couple of years I have had maybe a half dozen encodes that whatever I did SMdegrain just would not work. In those few cases, i just ran them using regular old Mdegrain and called it a day.

In fact Strange New worlds is like this. When I do an autocrop, it detects 278 lines on the top and bottom and shows an aspect ratio of 2.394:1. This will fail the encode. I add 2 lines to the top and bottom, so it is 280 on both using the up arrows, and it will run just fine with SMdegrain. I know in the past it looks like if it comes up as 2.391 or 2.4 it is always good, but 2.394 might fail "sometimes". Just one of the quirks of using SMdegrain.... It happens too infrequently to be bothered by it

[mikehbkwm]

I updated server 2 with Ripbot to the latest firmware. The same thing happened once again with Plex but I've encoded 5ish movies since and it hasn't done that again. Plex is showing Direct Play on the server end no matter what, technically nothing should be transcoding and I'm not out of HDD space. Only thing I can think of, just a random issue with Ripbot...I don't know.

[rlev11]

I updated server 2 with Ripbot to the latest firmware. The same thing happened once again with Plex but I've encoded 5ish movies since and it hasn't done that again. Plex is showing Direct Play on the server end no matter what, technically nothing should be transcoding and I'm not out of HDD space. Only thing I can think of, just a random issue with Ripbot...I don't know.

So first off, what are the sizes of the videos you are playing? Also what are you using to do the conversion to 4k, something like Topaz?? If you are just up converting in ripbot, I doubt you are really accomplishing anything. Using a program like Topaz AI to clean the video before running it through ripbot to do the HEVC conversion is a different thing altogether and can really clean up DVD sources. Although with Topaz, just from a time standpoint, you would be better off upscaling to 1080p, convert to HEVC in ripbot, then let you TV do the upconversion from 1080p to the panel native 4k

Lots of results on that space error on google for plex. Most results center around setting your transcoding directory in the settings menu. Are all the ones failing have subtitles? There was/is a bug in plex that caused this error due to having to transcode the subtitles for proper viewing

[Juha]

Hi, how will Ripbot work with 3D Blu-rays? Can (or will) 3D content be converted into 2D?

[Guest]

Hi, how will Ripbot work with 3D Blu-rays? Can (or will) 3D content be converted into 2D?

If I recall correctly, side by side & over/under 3D movies will retain the 3D effect, BUT the MVC types WILL lose the 3D.

However, you can demux those with tsmuxer to extract the mvp track, and then run the "normal" track thru RB if you need to do some filtering (but not much), then mux everything back together again with tsmuxer to create the 3D in .m2ts.

And have to be encoded with x264 !!!

[LigH]

MVC (Multi-View Coding) requires support for two parallel video streams. Right now x265 is being tested to support it, but that broke ffmpeg integration for now, and there are crashes in different circumstances.

[Guest]

can't wait to see the same tests with latest Zen5 cpus. I wonder if larger L3 cache in TR would help in this case?

Well, the Zen 5's are out, and pretty much EVERY reviewer has said the same thing....disappointed, not worth the money to upgrade, etc, etc.

Also a complicated procedure to make the 9950X behave properly.

So, I guess we'll have to wait another couple of years to the next generation, 10950X, or whatever.

OR, "risk" a 14th Gen Intel, but maybe a recent microcode may "fix" that issue.

OR, blow the budget completely, and go for a TR. (a 7960X would be good enough)

[Atak_Snajpera]

I think it is safe to say that this threading issues is most likely caused by avisynth code. Something wrong with caching? Too small memory allocation?

[Guest]

I think it is safe to say that this threading issues is most likely caused by avisynth code. Something wrong with caching? Too small memory allocation?

What is this reply in response to ??

What about Vapoursynth ??

[Atak_Snajpera]

What is this reply in response to ??

What about Vapoursynth ??

Regarding number of prefetch threads which has to be reduced for 4k content to 12.

[Guest]

Regarding number of prefetch threads which has to be reduced for 4k content to 12.

I just thought of a test that I don't think I've ever done before, and that would be to run a 5 minute sample 4K using DE with one server, and a 5 minute chunk setting, with those Avisynth server settings, and then doing the same sample on single mode where those server settings are probably not in play.

Will they take the same time to complete ???

[rlev11]

I just thought of a test that I don't think I've ever done before, and that would be to run a 5 minute sample 4K using DE with one server, and a 5 minute chunk setting, with those Avisynth server settings, and then doing the same sample on single mode where those server settings are probably not in play.

Will they take the same time to complete ???


With my 7950x
So it looks like single mode uses the setting "use multiple processing threads" in the main settings window. Running GOT with Mdegrain3-200 in single mode with that set to it's default of 0 which I would take as All. After a couple minutes that settled into about 2.5 fps. If I change that setting from 0 to 12, bumped that up to 6fps after a couple minutes.

The fact that it scales up where it should be when no avisynth filters are being thrown at it, I agree it is definitely something coming from Avisynth code with some buffer being overloaded especially with full frame 16x9 4k. If you are just doing a x265 encode everything runs as expected speed wise in comparison

[Guest]

With my 7950x
So it looks like single mode uses the setting "use multiple processing threads" in the main settings window. Running GOT with Mdegrain3-200 in single mode with that set to it's default of 0 which I would take as All. After a couple minutes that settled into about 2.5 fps. If I change that setting from 0 to 12, bumped that up to 6fps after a couple minutes.

The fact that it scales up where it should be when no avisynth filters are being thrown at it, I agree it is definitely something coming from Avisynth code with some buffer being overloaded especially with full frame 16x9 4k. If you are just doing a x265 encode everything runs as expected speed wise in comparison

Interesting, I think I will do some other tests, (when I can), but if it's an Avisynth thing, this needs to be addressed on the appropriate thread of the forum.

It would be interesting to test a basic file with StaxRip, using similar filters, with Avisynth & Vapoursynth.

Unfortunately, AFAIK, Atak hasn't got access to a high core count system, so he can't check the problems we have :(

[Boulder]

In any multithreading related issues, I'd start the analysis by restricting the amount of prefetched frames. With 4K sources, the frame sizes are getting so big that the default 2*threads can easily cause Avisynth cache trashing if you have not raised the size of the cache. The Avisynth default cache amount has not been changed in a long time, and it's very low.

I have these on top of my mtmodes.avsi to make sure there's enough cache memory for all situations (I have 64GB of RAM). Also mode 1 works better on my use cases.
SetMemoryMax(32000)
SetCacheMode(1)

[Guest]

In any multithreading related issues, I'd start the analysis by restricting the amount of prefetched frames. With 4K sources, the frame sizes are getting so big that the default 2*threads can easily cause Avisynth cache trashing if you have not raised the size of the cache. The Avisynth default cache amount has not been changed in a long time, and it's very low.

I have these on top of my mtmodes.avsi to make sure there's enough cache memory for all situations (I have 64GB of RAM). Also mode 1 works better on my use cases.
SetMemoryMax(32000)
SetCacheMode(1)

Thankyou Boulder,

I have had these settings in my scripts, but now that I also have 64Gb, I might up it to what you've suggested :)

SetCacheMode()
SetMemoryMax(20480)

Wouldn't (32768) be more accurate ???

[Boulder]

Wouldn't (32768) be more accurate ???

It really doesn't matter, the highest amount of memory consumed has been around 23-24 gigabytes. Avisynth+ only uses the memory it needs for caching (based on what and how the plugins themselves request cache). It's good to take a look at Task Manager and see the memory consumption of the corresponding processes.

For Prefetch, I've used Prefetch(threads=32, frames=12) on my 5950X. Usually there's not much difference between frames 4-12.

[Guest]

It really doesn't matter, the highest amount of memory consumed has been around 23-24 gigabytes. Avisynth+ only uses the memory it needs for caching (based on what and how the plugins themselves request cache). It's good to take a look at Task Manager and see the memory consumption of the corresponding processes.

For Prefetch, I've used Prefetch(threads=32, frames=12) on my 5950X. Usually there's not much difference between frames 4-12.

In RipBot prefetch is set elsewhere in the app, and is added to the final script.

But there are options to customise certain settings, but when using the Distributed Encoding function you can customise each encoding server like so:-

Server1CommandLine=/port 1000 /priority normal /restart-if-no-progress /avisynth-prefetch-threads 12 /x264-threads 32 /x265-threads 32

Changing the "avisynth-prefetch-threads 12", has the most impact on encoding speed, on 16 core CPU's using x265, 12 - 14 seems to be the sweet spot, Ryzen 5950X's, and especially the 7950X are very particular with this setting.

With x264 & x265 threads, not sure if those numbers make any significant difference.

[Boulder]

What you would need to do is somehow restrict the amount of prefetched frames. With threads 12, Avisynth is prefetching 24 frames which is probably far from optimal and will consume a lot of memory.

[Guest]

What you would need to do is somehow restrict the amount of prefetched frames. With threads 12, Avisynth is prefetching 24 frames which is probably far from optimal and will consume a lot of memory.

I'm pretty sure I've tried the frames "call", but from what I remember, I couldn't detect any difference, and tbh, not sure where it would need to be added to work with RipBot :(

This is the only reference in the avs script to prefetch:-

#MT
#PREFETCH_LIMIT=12

And this is determined by a "check box" on the settings page, the only other place I can think it could be put is in a custom script.

[rlev11]

Just not sure where to set frames into the ripbot script or the proper syntax to use in the job script. Don't see a "/avisynth-prefetch-frames" setting in the help for the encoding server command line like threads. If it actually the frames and not threads that's killing it, once we figure out where and how to set the frames, we could set threads to 16 (or total physical cores), and just play with the threads to see where things go south and see if that same number works across different platforms. It may just end up that 1 proper frames count may be optimized for BOTH 4k and 1080p which would be perfect, or at the minimum very little difference between them. Set the frames once and be done with it.

Ideal test would be to set threads to logical cpu's, then find happy frame count doing a full frame 4k. The run 1080p and start increasing the frames and see if there is any appreciable performance gain.

A buffer size increase might work as well, but could cause issues using distributed if you have servers with varying memory sizes. I would think the buffer size would get sent out to all the servers and if you have one with say 16gig instead of a setting optimized for 32 or 64, things could go crazy real fast.


Just a quick initial test, On my 1080p test with Mdegrain3-200, I set avisynth prefetch threads to 32 on my 7950x, so 64 prefetch frames, and I saw about a 9% speed increase just on that alone.

[Guest]

Just not sure where to set frames into the ripbot script or the proper syntax to use in the job script. Don't see a "/avisynth-prefetch-frames" setting in the help for the encoding server command line like threads. If it actually the frames and not threads that's killing it, once we figure out where and how to set the frames, we could set threads to 16 (or total physical cores), and just play with the threads to see where things go south and see if that same number works across different platforms. It may just end up that 1 proper frames count may be optimized for BOTH 4k and 1080p which would be perfect, or at the minimum very little difference between them. Set the frames once and be done with it.

Ideal test would be to set threads to logical cpu's, then find happy frame count doing a full frame 4k. The run 1080p and start increasing the frames and see if there is any appreciable performance gain.

That command line may not be where the threads setting needs to go, maybe it's a script thing ?!?!?

If it IS a command line option, Atak may need to change something to allow such a setting....

A buffer size increase might work as well, but could cause issues using distributed if you have servers with varying memory sizes. I would think the buffer size would get sent out to all the servers and if you have one with say 16gig instead of a setting optimized for 32 or 64, things could go crazy real fast.

Setting it to what Boulder suggested could max out any PC that has equal to, or less than 32Gb, but he did mention he hasn't seen it much higher than 23 - 24Gb, so I guess that needs to be observed, and if you have some with only 16Gb, you might have to add some.


Just a quick initial test, On my 1080p test with Mdegrain3-200, I set avisynth prefetch threads to 32 on my 7950x, so 64 prefetch frames, and I saw about a 9% speed increase just on that alone.

So I'm curious why you're using MDegrain for you tests ??

I hope you keeping an eye on this setting on the Settings page:-

"Limit to following filters only"....MDegrain needs that checked, SMDegrain doesn't !!!!!

I think you should be concentrating your tests using SMDegrain, tho.

[LigH]

Atak_Snajpera, Pauly Dunne, whoever keeps developing RipBot264:

Before the patches to support MVC were committed, specifying the input file name for x265 was optional.

Now that x265 may be able to read multiple input sources, it may also be mandatory now to specify them with a preceding parameter "--input".

[rlev11]

So I'm curious why you're using MDegrain for you tests ??

I hope you keeping an eye on this setting on the Settings page:-

"Limit to following filters only"....MDegrain needs that checked, SMDegrain doesn't !!!!!

I think you should be concentrating your tests using SMDegrain, tho.

Just using mdegrain as it is what is built into Ripbot proper and also has the most profound affect of the issue with the prefetch threads/frames.

If we can figure out what the proper settings to use on high core cpu's so the built in Mdegrain works properly, should fix everything else using whatever avisynth scripts are running

[Guest]

Just using mdegrain as it is what is built into Ripbot proper and also has the most profound affect of the issue with the prefetch threads/frames.

If we can figure out what the proper settings to use on high core cpu's so the built in Mdegrain works properly, should fix everything else using whatever avisynth scripts are running

Fair enough....I did some quick tests with a simple SMD script, altering the commandline, unchecking those boxes, adding a extra command to a script, and I got a lot of mixed results, some good, some bad, but I sort of lost track, so I will have to go over things again, and try and jot it down as I go.

Now there seems to be some major change to x265 :(

Also, more & more info about 9950X's, too.

[Atak_Snajpera]

If I had 16c cpu I would do this:
1) create 4k job in standard mode (no de)
2) manually open job1.avs in notepad and then add those at the very beginning

SetCacheMode(CACHE_OPTIMAL_SIZE)

or / and

SetMemoryMax(32768)

3) And then play with

Prefetch(16,16) #(threads,frames)

4) encode manually via job1_EncodeVideoPass1.CMD file

or just check how fast job1.avs is being processed in AVSMeter.

[Guest]

If I had 16c cpu I would do this:
1) create 4k job in standard mode (no de)
2) manually open job1.avs in notepad and then add those at the very beginning

SetCacheMode(CACHE_OPTIMAL_SIZE)

or / and

SetMemoryMax(32768)

3) And then play with

Prefetch(16,16) #(threads,frames)

4) encode manually via job1_EncodeVideoPass1.CMD file

or just check how fast job1.avs is being processed in AviSynth Meter.

But what if this was added to a custom script ??

How would you add the Prefetch(16,16) to a script to work with RB ??

Your above suggestion, does that make DE redundant ??

======================

Have your read anything about the changes to x265 ??

[rlev11]

If I had 16c cpu I would do this:
1) create 4k job in standard mode (no de)
2) manually open job1.avs in notepad and then add those at the very beginning

SetCacheMode(CACHE_OPTIMAL_SIZE)

or / and

SetMemoryMax(32768)

3) And then play with

Prefetch(16,16) #(threads,frames)

4) encode manually via job1_EncodeVideoPass1.CMD file

or just check how fast job1.avs is being processed in AVSMeter.

just setting
SetCacheMode(CACHE_OPTIMAL_SIZE)
SetMemoryMax(16384)

running GOT mdegrain3-200 resulted in a 300% increase from little over 2fps to 6.4 fps which is right around setting prefetch treads to 12 in previous tests

tried 32 and 64 gig in that memory setting and saw very little difference. In my task manager adding in the memory setting ffmpeg.exe memory usage jumped from around 7 gig to 10-11 gig regardless of what I put in for memory cache size

so this is what is in job1.avs
#Prefetch
video=Prefetch(video,16)

so how would I change this syntax for the Prefetch(16,16) #(threads,frames) setting

We are making progress....

[rlev11]

And let me just add, step 1 now is just trying to figure out exactly where the bottleneck is occurring. Just figuring this out is success in my mind. Is it prefetch threads, frames, memory allocation, or some hybrid combination of them all?? Once we figure this out, then the next step is going to see how those translate to other high core processors. Right now I am testing with my 7950x. Remains to be seen if the same settings will work on say a 5950x,3950x, or a high core Intel, although they probably will.

After that will come how these results can best be integrated into Ripbot that are both easy to set, and a novice user will understand. There is a point where too many settings can overload and frustrate users. We also have distributed encoding to add into the mix and how that will play with any settings we deem as wanting to set.

[Boulder]

It's clear that the Avisynth default cache (512MB or something like that..) is not enough for your use case.

[rlev11]

It's clear that the Avisynth default cache (512MB or something like that..) is not enough for your use case.


Looks like AviSynth+ default is 4GB according to the wiki. 1080p does not come close to that, I saw about 2 gig for ffmpeg during a 1080 run,

Thanks....