multiprocess system .. x264

[Hiritsuki]

Hi, I have using E5-4650L*4 to run x264
but it's seem not compatible for CPU multi process loading....

[LoRd_MuldeR]

So, do you have any specific question or just wanted to post a random guess? ;)

[Hiritsuki]

How do i run x264 faster with compatible multiprocess?

[LoRd_MuldeR]

x264 creates multiple threads by default, so usually you don't need to do anything at all.

But since you are not telling use what you are doing right now (x264 command-line, type of input, etc. pp.) nor what exactly you are observing, it's pretty much impossible to give any useful advice...

[Hiritsuki]

I using x264 with MeGUI, AVS is using MT.
usual I encode video is with out frame filter else source.

[LoRd_MuldeR]

If you expect some help, post your complete x264 command-line as well as your AVS script. Most of our users are not clairvoyants ;)

Anyway, the first obvious thing to check, before even worrying about x264, is whether your AVS script is the bottleneck. You can use something like AVSMeter (http://forum.doom9.org/showthread.php?t=165528) to benchmark your Avisynth script.

In case your Avisynth script alone (e.g. in AVSMeter) runs at a very low frame-rate while using only a single CPU core, it is obvious that this script would bottleneck x264...

[Hiritsuki]

my avs script is usual like this

LoadPlugin("C:\Program Files (x86)\MeGUI\tools\dgindex\DGDecode.dll")
Load_Stdcall_Plugin("C:\Program Files (x86)\MeGUI\tools\yadif\yadif.dll")
DGDecode_mpeg2source("G:\BSD\2013-10\リトルバスターズ！～Refrain～\リトルバスターズ！～Refrain～　第５話-20131104.d2v").sorathread().Yadif(mode=1,order=1).sorathread().SelectEven
__film = last
__t0 = __film.trim(691, 6204)
__t1 = __film.trim(8004, 26164)
__t2 = __film.trim(27964, 46843)
__t3 = __film.trim(48643, 49241)
__t0 ++ __t1 ++ __t2 ++ __t3

[LoRd_MuldeR]

Except for sorathread(), which I don't know at all, your script doesn't seem to do any fancy stuff. Still I urge you to run your concrete script through AVSMeter and post the results, just to be sure.

[Guest]

Here's what sorathread() is about:

http://forum.doom9.org/showthread.php?t=164073

[LoRd_MuldeR]

I see. So maybe the OP should reflect on whether it is really required/reasonable for a very fast filter like Yadif()?

[Hiritsuki]

How to use the avsmeter?

[Hiritsuki]

G:\BSD\2013-10\リトルバスターズ！～Refrain～>AVSMeter リトルバスターズ！～Refrain～　第５話-20131104.avs

AVSMeter 1.7.1 [AVS2.6] by Groucho2004

Avisynth 2.6 (Alpha 4 and above) is required.

G:\BSD\2013-10\リトルバスターズ！～Refrain～>

it only show that

[LoRd_MuldeR]

In the most simple way one can think of:
AVSMeter.exe "c:\Path to your script\foobar.avs"

[EDIT]

You obviously tried using the 2.6 version of AVSMeter with Avisynth 2.5. If you are using Avisynth 2.5, try the 2.5 version of AVSMeter!

(there are two versions included in the download)

[Hiritsuki]

ok wait a moment

[Hiritsuki]

finish

G:\BSD\2013-10\リトルバスターズ！～Refrain～>AVSMeter リトルバスターズ！～Refrain～　第５話-20131104.avs

AVSMeter 1.7.1 [AVS2.5] by Groucho2004
AviSynth 2.60, build:May 25 2011 [19:58:41]

Number of frames: 43154
Length (hhh:mm:ss.ms): 000:23:59.905
Frame width: 1920
Frame height: 1080
Framerate: 29.970 (30000/1001)
Interlaced: No
Colorspace: YV12

Frames processed: 43154 (0 - 43153)
FPS (min | max | average): 1.46 | 108.09 | 71.98
CPU usage (average): 5%
Thread count: 7
Physical Memory usage (peak): 550 MB
Virtual Memory usage (peak): 575 MB
Time (elapsed): 000:09:59.566

[LoRd_MuldeR]

Okay, 5% CPU usage is very low. But that's perfectly fine for a fast script. And 71.98 FPS (in average) should be more than fast enough to feed x264 ;)

Now try something simple like:
x264.exe -o "C:\Some Path\Output.mkv" "G:\BSD\2013-10\リトルバスターズ！～Refrain～>AVSMeter リトルバスターズ！～Refrain～　第５話-20131104.avs"

Also make sure you use an up-to-date (http://komisar.gin.by/) x264!

[Groucho2004]

I have using E5-4650L*4
Are you saying that you have a server board with 4 of those? What board?

[poisondeathray]

mobo specs are in his signature link
Supermicro X9QRi-F

bottleneck is likely mpeg2source with HD footage , even with sorathread. I would test another source filter

[Hiritsuki]

that is i say not compatible with multi process.

http://i.minus.com/iyBJkfonYw6Va.png


Are you saying that you have a server board with 4 of those? What board?

Supermicro X9QRI-F

[Groucho2004]

mobo specs are in his signature link
Supermicro X9QRi-F
I think he just updated that link to his current config, it showed something different before.
Anyway, holy crap, that's some serious hardware. Theoretically, x264 would use 96 threads. Not surprising that the CPUs are not fully saturated.

[LoRd_MuldeR]

Well, according to his results he got ~72 FPS from his input script, which is more than real-time. Unless he's going for the "ultrafast" presets, this should be enough to keep x264 busy.

But if he indeed has a machine with four octo-core processors (32 physical cores, 64 logical cores in total!), it might make sense to simply run several encodes at the same time...

[LoRd_MuldeR]

that is i say not compatible with multi process.

Uhm, what? :eek:

You are getting 50+ FPS encoding though-put with "medium" (standard) preset on 1080p footage.

And about 50% overall CPU load on a 64(!) core machine from a single encoder instance. That is anything but "not compatible with multi process" :D

I'd say this is excellent use of a multi-core processor. And you could even be running two of these encodes in parallel! Or try a slower preset...

(BTW: Please don't include ultra large screenshots directly in your post, just use a link or a thumbnail in this case)

[Groucho2004]

that is i say not compatible with multi process.
Have a look at this thread (http://forum.doom9.org/showthread.php?t=168875).

[Hiritsuki]

Sorry about my screenshot... but it clear to view what i say.

i had try lowest preset placebo and using 6~70% CPU

why x264 can't using full cpu?

[Groucho2004]

why x264 can't using full cpu?
Did you read the thread to which I pointed you?

[Hiritsuki]

i read that thread... but i set any --threads x264 also use not full

[Groucho2004]

i read that thread
OK, then you should realize that efficiency over ~30 threads with 1080p material will decrease.
As already suggested here before - run multiple encodes in parallel to efficiently use your 64 logical cores.

[Hiritsuki]

ok ... so .... that is final workaround with multiple encodes ...

[Groucho2004]

ok ... so .... that is final workaround with multiple encodes ...
Yes, that's how it is.
However, putting things into perspective, x264 is very efficient when it comes to multi-threading. If you were using 2160p material it would certainly make even your awesome hardware sweat. :)

[Atak_Snajpera]

You can always try Distributed encoding mode in RipBot264 with 8 local servers.
example -> http://www.youtube.com/watch?v=b-BICnj62iE&feature=youtu.be (http://www.youtube.com/watch?v=b-BICnj62iE&feature=youtu.be)

[Hiritsuki]

You can always try Distributed encoding mode in RipBot264 with 8 local servers.
example -> http://www.youtube.com/watch?v=b-BICnj62iE&feature=youtu.be (http://www.youtube.com/watch?v=b-BICnj62iE&feature=youtu.be)

is this can run in one server with 4 physical cpu? :eek:

[Asmodian]

Yes. It breaks the video into "chunks" that each get encoded in parallel. This is effectively running multiple encodes in parallel but each encode is a part of the same video. These separate parts are spliced together after all the encodes finish to give the complete video.

[Hiritsuki]

Yes. It breaks the video into "chunks" that each get encoded in parallel. This is effectively running multiple encodes in parallel but each encode is a part of the same video. These separate parts are spliced together after all the encodes finish to give the complete video.

is this will compatible with x264 2pass? :confused:

[Hiritsuki]

// Supported input formats AVI / FLV / MP4 / MKV / DVD / HD-DVD / Blu-ray

...umm ...little trouble

[Hiritsuki]

i had view this video..
but i don't know how to run distributed :stupid:
can you tell me how it run?

http://i.minus.com/jbnEDoHhwT92Xr.png (http://minus.com/lbnEDoHhwT92Xr)

and try to load mp4 file

DirectShowSource: couldn't open file D:\DOWN\videoplayback.mp4:
このファイルのメディアの種類は認識されません。
(D:\Temp\RipBot264temp\job1\getinfo.avs, line 2)

[turbojet]

You probably have default decoder set to microsofts. If you are using Windows 7 and maybe 8 the decode needs to be changed with Win7DSFilterTweaker. Another tip for more speed is decode using lavfilter's nvcuvid or if your xeon's have gpu's quicksync should be even faster.

OT: I'm surprised ripbot264 is still using directshowsource over standalone dss2.

[Hiritsuki]

oh... I find the problem is.. I have not Haali ....sorry

[Hiritsuki]

http://i.minus.com/jUzuJMmPTH7oc.png (http://minus.com/lUzuJMmPTH7oc)

--profile high10 --preset placebo --ref 8 --bframe 8

[Hiritsuki]

ripbot264 most servers is 8
distributed well but can't use avs for source...

[Rumbah]

Well, you could use trim to divide your avs video in 8 parts and encode them in parallel (don't forget to set threads = XX as the encoding quality lowers with many threads).

[Hiritsuki]

um... but the trim ... have some problem of 2pass quilty...
I know ripbot264 also using trim.

[Rumbah]

x264 uses the same rate control for crf and 2 pass so if you don't have to meet a special file size or have vbv restrictions you can just use 1 pass crf without quality loss.
Then you won't have a penalty for separating the video in several parts.