Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
7th June 2022, 03:14 | #19503 | Link |
Grumpy Old Man.
Join Date: Jul 2019
Location: Out There....
Posts: 692
|
Time for another update...
Hi,
There are several significant updates in this build (read the changelog) Also, I have completely re-organised the Script & Custom Scripts folders, so it's not so cluttered in the Custom Script list. https://forum.doom9.org/showthread.p...72#post1970572 And as always, backup your existing Tools folder before using this, just to be sure. Good luck.
__________________
Not poorly done, just doin' it my way !!! Live every day like it's your last, because one day, it will be !! (M$B) Last edited by Pauly Dunne; 20th June 2022 at 04:27. |
9th June 2022, 04:38 | #19505 | Link |
Formally known as .......
Join Date: Sep 2021
Location: Down Under.
Posts: 995
|
I'm curious why my Ryzen 3950X & 5950X (16 core) are encoding quite a bit slower than the 5900X (12 core), even an old dual Xeon X5690 is faster.
This a 4K encode, but on other encodes they "catch up". https://i.imgur.com/xRW77Rb.jpg |
9th June 2022, 13:55 | #19506 | Link | |
Registered User
Join Date: Mar 2011
Posts: 432
|
Quote:
In my post a few weeks ago, when I got my new to me dual CPUs with 14 cores and 28 threads I had awful performance when using MDegrain at the time. Once I dug into it I found I had to limit the number threads. https://forum.doom9.org/showthread.p...47#post1968247 In my case, for that system, 14 threads was optimal for my general encoding even though I had 48 CPU/Threads. My suggestion is to use a grainy 4K 15-20 minute sample as your test and tune each encoding server individually with degraining enabled. Enable the "Limit to the following filters only" option. Change the chunk size to 2-5 minutes (for testing), enable only one Server process. Then tune that machine to the number of threads that get the highest fps possible. Watch the Task Manager during this time and if you are getting less than 80% utilization and if so, after tuning for one Server process, increase it to two and see what your overall FPS is. Once each computer has been tuned for the number of threads, then run your encodes again. Your set up should just fly when each machine is tuned. |
|
9th June 2022, 14:50 | #19507 | Link | |
Formally known as .......
Join Date: Sep 2021
Location: Down Under.
Posts: 995
|
Quote:
Then I did add SMDegrain to the list, and checked the "Limit to the following filters only", and changed the Threads to 24, didn't make a noticable difference, if anything, it was a little slower. But now that you have provided a fairly detailed process of "tuning", I will definitely take some time to get it set up. Having said that, each encode can be quite different to another, even each chunk can change, due to it's "content's". Well done, thanks. Edit:- Well after spending a couple of hours testing this process, with & without SMDegrain in the "list", trying many different threads, and I thought I'd found the sweet spot @ 22 threads. And I found the difference between having SMDegrain in the "list", and "Limit to the following filters only" checked or not, made very little difference. So at this point, I'll just have to put up with the 16 Core Ryzens being slower than the 12 Core 5900X (at least on this particular encode), I'm using a pretty high tr setting. video=SMDegrain(video,tr=10,thSAD=1000,thSADC=500,thSCD1=400,thSCD2=130,contrasharp=true,prefilter=2,str=1.2,refinemotion=true) Last edited by TDS; 10th June 2022 at 05:39. |
|
10th June 2022, 04:29 | #19508 | Link | |
Registered User
Join Date: Feb 2022
Posts: 2
|
Quote:
https://youtu.be/tftERYq94GE Last edited by wamar21; 11th June 2022 at 12:03. Reason: hyperlink |
|
10th June 2022, 05:30 | #19509 | Link | |
Formally known as .......
Join Date: Sep 2021
Location: Down Under.
Posts: 995
|
Quote:
Welcome to the Forum, and I'm sure a LOT of people will appreciate that you took the time to do this. Even tho you're using a VM, the basic steps are the same. Cheers Last edited by TDS; 10th June 2022 at 08:09. |
|
10th June 2022, 20:12 | #19510 | Link |
Registered User
Join Date: Feb 2022
Posts: 2
|
Thanks, I also noticed a few ambiguities I wanna clear up anyway. Hopefully this one stays useful for as long as possible. Ill tweak it a bit.
I wanted it to be somewhere between "good, clear, useful tutorial" and "annoying 2000s homage/shitpost" but I may have rode that line a bit too close. UPDATE: Removed the "brb helping w/ groceries" bit for clarity's sake, RIP cultural artifact. Last edited by wamar21; 11th June 2022 at 12:06. |
11th June 2022, 13:22 | #19511 | Link | |
Registered User
Join Date: Mar 2011
Posts: 432
|
Quote:
|
|
11th June 2022, 14:48 | #19512 | Link | |
Formally known as .......
Join Date: Sep 2021
Location: Down Under.
Posts: 995
|
Quote:
It seems that it's the particular filter I am using, a VERY strong high "tr" #. When I go back to using "normal" filter's, ALL the Ryzen's are pretty much equal. Fortunately, I don't think I have too many "extra" noisy 4K's to filter, and it's not much of an issues with lesser resolutions. I might give MDegrain2 a go when I have some spare time, but for now I just want to get some encoding done, while the weather is nice & sunny, which it hasn't been for months & months. As I don't "waste" electricity doing encodes on overcast days, or over night !!! It's a fine weather, Solar Energy only, process !! EDIT:- SSDD....so I tried something a little weird today, I actually limited the Cores available to the CPU in BIOS, so the 16 core CPU, would now run with 12 cores !!! Surprisingly enough, it made a significant difference...FASTER !!!, but only on that very heavy filter script. Another thing I tried was the x265 commands...I added --pools 24 that didn't seem to have much of an affect. Last edited by TDS; 12th June 2022 at 06:09. |
|
12th June 2022, 11:25 | #19513 | Link | |
Registered User
Join Date: Mar 2011
Posts: 432
|
Quote:
I think if you spend a couple of hours or so tuning your threads for each of your Servers it will pay big dividends in encoding time vs watts used. Atak: You might want to create a tool that uses MDegrain2/SMDegrain that will find the sweet spot for thread use per Computer. As CPU core count seems to be increasing more and more people are going to hit this issue. |
|
12th June 2022, 14:31 | #19514 | Link | |
Formally known as .......
Join Date: Sep 2021
Location: Down Under.
Posts: 995
|
Quote:
Even when it's encoding very slowly, it's still pushing the CPU's to pretty much 100%, can't ask anymore than that. Correct me if I'm wrong, BUT all the encoding servers use the RipBot settings of the client machine, so that individual server settings "do not apply" !!! As Atak mentioned once, in reality the server PC's ONLY need the Encoding Server, not the whole "app", you'd only need to manually start the Encoding Server, and DE will do the rest, so there's no settings required. But it IS strange that in some situations, a 12 core system, does a faster job than a 16 core system.... Last edited by TDS; 12th June 2022 at 14:40. |
|
12th June 2022, 15:27 | #19515 | Link |
RipBot264 author
Join Date: May 2006
Location: Poland
Posts: 7,815
|
You can manually define number of prefetch threads for each server like this
Code:
EncodingServer.exe /avisynth-prefetch-threads 8
__________________
Windows 7 Image Updater - SkyLake\KabyLake\CoffeLake\Ryzen Threadripper Last edited by Atak_Snajpera; 12th June 2022 at 15:30. |
13th June 2022, 02:00 | #19516 | Link | |
Formally known as .......
Join Date: Sep 2021
Location: Down Under.
Posts: 995
|
Quote:
You know, that option has been there all along, staring us in the face, but not really knowing what it does. So, under "Settings", "Distributed Encoding" tab, click on the little "Spanner", then "Command line help"..... EDIT:- I just tried this on my 3950X 16 Core (which is currently running as an encoding server), and adding that command, only using 24 instead of 8, hasn't changed anything, that I can detect. In the Encoding Server "window", it still shows :- x265 [info]: Thread pool created using 32 threads So again, ALL the server machines take their "instructions" and "settings" from the main Client PC, regardless how the server PC is setup ! EDIT #2:- OK, changed the command line on the Client PC, and even tho it states that Avisynth is only using 24 threads, the x265 info remains @ what ever the CPU provides. The ONLY way it seems, to change the amount of threads that x265 uses is in the command line for x265, by adding --pools [integer] Last edited by TDS; 13th June 2022 at 03:05. |
|
13th June 2022, 16:43 | #19518 | Link |
Registered User
Join Date: Mar 2011
Posts: 432
|
RipBot 2.0
Hi Atak,
If your schedule allows, it would be nice to get a RipBot 2.0. You might want to change the name to just RipBot or maybe something like RipBotEncoder. SMDegrain has been a game changer in quality, though not without its steeper learning curve. I have transitioned over to SMDegrain exclusively though using Pauly Dunnes Tools. Though I create my own scripts to use the CPU only instead of the GPU. SMDegrain is powerful and flexible enough to be the primary degrainer for RB. The options for Blur plus Contrasharp on very grainy sources produces output that defies what I would think would be possible. Dolby Vision is also easier to deal with and is scriptable: https://forum.doom9.org/showthread.php?t=183479 There are some options that have to be passed if cropping is going to be used in the final output. It would be nice to see a UI overhaul of the AVI Synth settings. If the window could be made quite a bit larger to see the 2.40:1 bottom borders for 4K sources like you can with HD sources. The larger window will also give you more space for the options. For the SMDegrain settings. A pull down menu with a few presets for most people, something like this: Light: SMDegrain(video,tr=2,thSAD=200,thSADC=100,refinemotion=true) Medium: SMDegrain(video,tr=4,thSAD=400,thSADC=200,refinemotion=true) Heavy: SMDegrain(video,tr=8,thSAD=800,thSADC=400,refinemotion=true) Then add a Blur+Sharp radio button next to SMDegrain filter that adds the following depending how heavy they wanted it: Light: contrasharp=true,prefilter=0 Medium: contrasharp=true,prefilter=1 Heavy: contrasharp=true,prefilter=2 Then having an Advanced option on the bottom of the pull down that opens up another new window with more switches and options. Such as having the tr, thSAD (I think thSADC should always be half of thSAD), refinemotion, contrasharp, prefilter, str (for darker sources), blksize, etc. A little "?" mark next to each option would be nice with a mouse hover pop up that provides details on that option. Changes for the Distributed Encoding model would be good too. The Primary server would dole out work to the clients. The RipBot install on the client would have something like a "DE Client" startup option and then RB will just start up minimized to the system tray with no Server processes started. The Primary RB server would then communicate with the RB Client to start up how many server processes along with the options for those server processes. The DE window can also have a per Client Thread Calibration button that you can click that will open up a window with two grainy 1 minutes sources, one for HD and one for 4K that the user can choose select one they want, then pop up a warning that this could take a long time, then the Primary server will send the Job to the RB Client, start up a single Server Process along with the number of threads to test for SMDegrain Heavy or MDegrain2-400, over and over until the optimum number of threads is found. Though some intelligence can be added, like start with half threads, then the lower quarter threads, upper quarter of threads, then fine tune it between the two quarters that had the best speed. Atak, I know I've asked this before, but it would be nice to get RB up on Git along with the source and a issue tracker. |
14th June 2022, 04:06 | #19519 | Link | |
Formally known as .......
Join Date: Sep 2021
Location: Down Under.
Posts: 995
|
Quote:
It's ONLY happening with the VERY very strong filter I am using, on ALL the other filters, they are all pretty much equal encoding speed. So there must be something going on when you bump up the "tr", "thSAD" & "thSADC" levels/numbers. Up to "tr=8" is fine... But one thing I need to repeat, is, ALL the server PC's, regardless of any tweaks & tuning, take ALL their settings & instructions from the client PC. So if you went to all the trouble of tuning every server, when running them as a client, is all over ridden, when they are used as a server, because the servers ONLY require the Encoding Server to do the DE process. Now I'm pretty sure that running RipBot on a VM, wouldn't change the fundamentals, and I don't use VM's for ANYTHING !! But there's obviously a significant difference between your hardware & setup, to what I have. I have tried everything that has been suggested, and have not seen any significant change, so I think we need to put this topic to rest, it's just going around in circles. |
|
14th June 2022, 04:48 | #19520 | Link | |
Grumpy Old Man.
Join Date: Jul 2019
Location: Out There....
Posts: 692
|
Quote:
I knew when I started to implement SMDegrain into RipBot, that is was superior, and that was before I realised that there were options to use pre-filters as well. It wasn't until I discovered Dogway's Doom9 page, that I realised all this. https://forum.doom9.org/showthread.php?t=182881 I certainly can't take any credit for what SMDegrain can do !!! But I have to say that there are a lot more pre-filters, than just 0,1 & 2, sure there are GPU based pre-filters that other users could benefit from. There also many other filters available in my builds, too, not just focusing on SMDegrain. And finally, since I first post my RipBot build(s), there has been 100's of downloads, the latest build posted only a week ago, has been downloaded over 30 times. There has only been a few say thanks, but for Ryushin to say these words, is pretty amazing.
__________________
Not poorly done, just doin' it my way !!! Live every day like it's your last, because one day, it will be !! (M$B) |
|
Tags |
264, 265, appletv, avchd, bluray, gui, iphone, ipod, ps3, psp, ripbot264, x264 2-pass, x264 gui, x264_64, x265, xbox360 |
Thread Tools | Search this Thread |
Display Modes | |
|
|