Only 60 - 80% CPU usage on X264 ?

[S3Ri0US]

Hi there... i have installed a new CPU on my System.. but now X264 only use 60 - 80% of the Core Speed..

Source: M2TS - 1080p - AVC

Commandline: --profile high --level 4.1 --preset fast --pass 2 --aud --bitrate 12920 --threads auto --thread-input --b-pyramid 2 --keyint 24 --min-keyint 1 --b-adapt 1 --me umh --ref 3 --vbv-maxrate 38000 --vbv-bufsize 30000 --direct auto --mixed-refs --subme 7 --trellis 1 --8x8dct

CPU: Core i7 980EX

I also have tryed to use 14 - 16 and 18 Threads but noting happens.

can this be a problem that the FrameServer (DGAVC or AviSynth) are to slow ?

Thanks for Help

[osgZach]

try leaving off the --threads flag?

Also.. how are you measuring the usage? Task Manager is not always that realiable, I believe.. But I don't know what methods people use to really monitor that stuff, other than looking at the impact of the encoding time.

I specify --thread-input when I encode, but not the --threads flag and MediaInfo reports "threads=3" under encode settings.. So unless that is for something else I don't understand, I would say just leave the flag undeclared and x264 should automatically handle it.

[Boolsheet]

You can check the throughput of your AviSynth script with VirtualDub (http://virtualdub.org).

Load the avs in VirtualDub.
Check if the input and output color depth is the same as your script is generating (in "Video" -> "Color Depth") and set the compression to uncompressed (in "Video" -> "Compression", should already be the default).
If you now start "File" -> "Run video analysis pass", VirtualDub reads the input as fast as it can. The displayed framerate is your throughput. Additional workload, like encoding with x264, makes it only slower.

[woah!]

that cpu has HT which shows as 12 real cores in your task manager, but it isnt. your lucky if you get 25% per HT thread = 6 real cores + 1.5HT fake cores = 7.5 core true power. 12/7.5 = 64%ish you say you get from 60-80%.

[JoeH]

I have at times seen that when x264 doesn't use all the CPU it is because the computer simply can't decode the video as fast as x264 can render it. If that is the case a fast GPU plus DGIndexNV could be the solution.

[Audionut]

Try --preset slow and check!!

[osgZach]

I have at times seen that when x264 doesn't use all the CPU it is because the computer simply can't decode the video as fast as x264 can render it. If that is the case a fast GPU plus DGIndexNV could be the solution.

That makes no sense.. If his CPU was wasting time trying to play catchup with the decoding, he would be seeing full CPU usage. Or even then, I would have to think that the very act of decoding and encoding at the same time would slow things down enough so that the CPU was encoding frames as fast as it could decode them. In which case, again, you would likely be seeing full CPU utilization.

However, turning off hyper-threading might actually be a solution in this case.. It tends to mess up programs that are properly multi-threaded from what I hear. I'm sure an x264 dev would be better qualified to say for sure though. But I've seen this said around here, so who knows...

[mp3dom]

Both DVAVCDec and Avisynth aren't multithreaded so they probably are the bottleneck. Since you're using the 'fast' preset, most probably x264 is encoding at a superior speed than the decoding speed of DVAVCdec+AviSynth. If you want 100% Cpu utilization you can:
- Slowdown the x264 preset to veryslow/placebo
- Use DGIndexNV (GPU decoding) or FFDShowMT
- Use an intermediate lossless avi that support multithread (for example Ut Video Codec)

[S3Ri0US]

if i use

--preset slow i only have 30fps and 16% CPU in the first pass

--preset fast 65fps and 60% CPU

Both with DGAVCIndexNV on GTX 285

[7ekno]

As explained above by woah! (http://forum.doom9.org/showthread.php?p=1388937#post1388937), it's a HyperThreading "artifact" ... while Hyperthreading shows to the OS as 2 distinct cores, there is significant overhead involved in swapping between threads ... In reality with most general multi-threaded apps, 1 hyperthreaded core = 1.5x the throughput (but windows expects it to be 2x) ... so by default, any HyperThreaded CPU running at full CPU speed (and throughput) will only ever be reported by the OS as 1.5/2 (or a max of 75%!!!) ...

Of course the efficiency of hyperthreading is application specific (and the above 1.5x throughput is a "generic" estimation), so it does not surprise me you get reported maxes ranging from 60-80% ;)

7

[JoeH]

It does surprise me... I have a Core i7 920 OC'ed to 3.8GHz with Hyperthreading turned on, and the only time I don't see 100% CPU utilization on all 8 "cores" on the 2nd pass is when the decoder can't keep up with x264... and in these cases if I set x264 to a slower preset it always goes right up to 100%.

[Blue_MiSfit]

You should able to get 100% utilization.

7enko's expanation of HyperThreading is simply incorrect.

While it's true that HT will never double the real performance of a given set of physical cores, they will not affect how CPU usage is calculated (other than increasing the number of logical cores, obviously).

If you can't get up to 100% CPU usage, you are almost certainly bottlenecked by decoding your source. The "fastest" decoders for H.264 in my experience (on a Q6600 CPU):

CoreAVC 2.0 / DiAVC >> Divx H.264 >> ffmpeg-mt >> DGDecNV / CUDA / DXVA >> libavcodec

Keep in mind, while CUDA decoding (via CoreAVC or Neuron2's DGDecNV) uses almost no CPU time, it has a relatively low MAXIMUM decoding speed, and in some cases this can bottleneck a workflow. AFAIK, this performance does not increase by using a faster GPU, since the same ASIC is doing the decoding (not the GPU shaders).

So, give us more information please :)

1) What's the source format, and how are you decoding it into avisynth
2) Are you running any filers? Post the full script
3) What builds of x264, AviSynth, and any filters are you using?

As a quick check, run your source using --preset placebo, and you'd damned better get 100% cpu usage :). Obviously the goal is to get 100% CPU usage and still run faster than --preset placebo!

Finally, you can always just run two encodes in parallel. But this can sometimes be a headache. :)

~MISfit

[7ekno]

they will not affect how CPU usage is calculated

Depends on OS, XP/Vista calculates CPU utilisation differently from 7 ... you will typically see higher CPU utilisations with HT on 7 than XP/Vista ... but try --preset placebo with a faster source (MPEG2 / AVI) and see if you can max CPUs ...

7ek

[S3Ri0US]

now i have try a simple check with Std Xvid file and Dshow (ffdshow)

Format : MPEG-4 Visual
Format profile : Advanced Simple@L5
Format settings, BVOP : Yes
Format settings, QPel : No
Format settings, GMC : No warppoints
Format settings, Matrix : Default (H.263)
Codec ID : XVID
Codec ID/Hint : XviD
Duration : 1h 37mn
Bit rate : 860 Kbps
Width : 640 pixels
Height : 336 pixels
Display aspect ratio : 1.905
Frame rate : 25.000 fps
Resolution : 8 bits
Colorimetry : 4:2:0
Scan type : Progressive
Bits/(Pixel*Frame) : 0.160
Stream size : 598 MiB (86%)
Writing library : XviD 1.2.1 (UTC 2008-12-04)

AVS:

DirectShowSource("D:\test.avi", fps=25.000, audio=false)

------------------------------------------------------------
Commandlines:

--pass 1 --bitrate 1075 --threads auto --level 4.1 --me dia --ref 1 --subme 1 --no-dct-decimate --partitions none --stats "D:\x264.stats" --output NUL

--pass 2 --level 4.1 --bitrate 1075 --threads auto --me umh --ref 4 --sar 1:1 --no-fast-pskip --stats "D:\x264.stats" --output "D:\video.mkv"

Pass 1: 13% CPU - 420 FPS
Pass 2: 76 - 82% - 172 FPS
------------------------------------------------------------

Commandlines:

--profile high --level 4.1 --preset fast --pass 1 --bitrate 1075 --threads auto --me dia --ref 1 --subme 1 --no-dct-decimate --partitions none --stats "D:\x264.stats" --output NUL

--profile high --level 4.1 --preset fast --pass 2 --bitrate 1075 --threads auto --me umh --ref 4 --sar 1:1 --no-fast-pskip --stats "D:\x264.stats" --output "D:\video.mkv"

Pass 1: 20 - 25% CPU - 411 FPS
Pass 2: 50 - 70% - 180 FPS

------------------------------------------------------------

Commandlines:

--profile high --level 4.1 --preset placebo --pass 1 --bitrate 1075 --threads auto --me dia --ref 1 --subme 1 --no-dct-decimate --partitions none --stats "D:\x264.stats" --output NUL

--profile high --level 4.1 --preset placebo --pass 2 --bitrate 1075 --threads auto --me umh --ref 4 --sar 1:1 --no-fast-pskip --stats "D:\x264.stats" --output "D:\video.mkv"

Pass 1: 13 - 15% CPU - 70 FPS
Pass 2: 84 - 92% - 38 FPS

---------------------

The CPU Load will be better with placebo preset but the encoding time takes to long. I think i will use the fast preset because i cant wait so long time.. :-)

[osgZach]

Wait... so you're trying to isolate a bottleneck within your x264 encoding process.. and then for a test you up and switch to another codec?

:confused:

[Blue_MiSfit]

With incredibly fast settings like that, you are indeed bottlenecking the MPEG-4 ASP decoder :)

I wouldn't complain about 420fps being "slow" :)

Also, your settings are silly.

--no-dct-decimate shouldn't be used basically ever, unless you are using insanely high bitrates and are trying to preserve fine grain structure. When you specify --pass 1, x264 automatically takes any settings you provide, and scales them down to "fast" settings.

Therefore, here's a command line you could try (based on your "slow" command) that should simplify things:


x264.exe --profile high --level 41 --preset veryslow --tune film --pass 1 --bitrate 1075 --stats "D:\x264.stats" --output NUL input.avs
x264.exe --profile high --level 41 --preset veryslow --tune film --pass 2 --bitrate 1075 --stats "D:\x264.stats" --output "D:\Video.mkv" input.avs


~MiSfit

[Cman21]

hello, i just got an i7 950 and i am only getting about 17% usage in one instance but i have ran 5 at once to get it up to ~90%. i would like to know what i can do to get my encodes using MeGui and x264 to get my usage at least 50% per instance. i do have CoreAVC installed, with a nvidia 460, but i am pritty sure it is NOT being used for the decoding and since it was mentioned as one of the possible fixes for a bottleneck in decoding i will mention it. below is the media info of the source, then my avs script for MeGui, and then the MeGui x264 profile settings. now i have tried setting threads to 0(auto) 4, 8, and 16 but it doesn't change a thing.

General
Complete name : D:\Encoding Work\test_video.264
Format : AVC
Format/Info : Advanced Video Codec
File size : 3.67 GiB

Video
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.1
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 23.976 fps
Resolution : 8 bits
Colorimetry : 4:2:0
Scan type : Progressive
Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.709-5, BT.1361, IEC 61966-2-4 709, SMPTE RP177



LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\MaskTools.dll")
global MeGUI_darx = 16
global MeGUI_dary = 9
LoadPlugin("C:\Program Files (x86)\megui\tools\dgavcindex\DGAVCDecode.dll")
AVCSource("D:\Encoding Work\Test.dga")

super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

crop( 2, 2, -2, -2)
LanczosResize(1920,1080) # Lanczos (Sharp)

program --preset slow --tune animation --crf 19.0 --chroma-qp-offset 2
--level 4.1 --bframes 3 --ref 4 --slices 4 --aud --nal-hrd vbr --b-pyramid strict
--keyint 24 --min-keyint 2 --vbv-bufsize 30000 --vbv-maxrate 40000 --weightp 0
--colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --output "output" "input"

if there is any other info you need just let me know.

[Groucho2004]

i do have CoreAVC installed, with a nvidia 460, but i am pritty sure it is NOT being used

Of course it's not being used, you're using the outdated and not recommended DGAVCDecode as decoder. Possible options in your case would be FFMS2 or DGDecodeNV.

Apart from that, with this heavy filtering, you should use a multithreaded AVISynth.

[Cman21]

ok and how? i didnt do anything for that stuff other that what i have been told by MeGui, call me a noob if you want i dont know the technicals.

EDIT:
OK i used the file indexer, DGAVDIndex, on the .264 file and i cant use DGIndexNV as it is grayed out in my window. i could try FFMSIndex now but i still dont know what to do about a multithreaded avisynth. also what is the difference between DGIndexNV and FFMSIndex and which would be better in this instance?

[Groucho2004]

The problem you have is with the GUI, not the encoder. I suggest you post in the MEGUI forum.

[Cman21]

ok, that may be true but i can still force MeGui to do what i want and what you are suggesting i just need a little more info on how to use these things.

[Groucho2004]

Sorry, can't help you with these GUIs. As I said, your best bet is to post in the appropriate forum.

[Cman21]

ok then help me with x264 as using your recommended FFMS didnt help at all. and point in the right direction on getting a multithreaded AVISynth. or how to do this without a gui, thought of that?

[Groucho2004]

ok then help me with x264

How? What exactly do you want to know?

and point in the right direction on getting a multithreaded AVISynth.

http://forum.doom9.org/showthread.php?t=152800

or how to do this without a gui, thought of that?

1. Create an Avisynth script
2. Create a batch file with the x264 command lines
3. Encode your video
4. Depending on your target format, use TSMuxer or mkvmerge to multiplex the video and audio stream(s).

[Cman21]

ok and what is this batch file you speak of? how do i go about making one? and i'll be using mkvmerge to put everything together at the very end.

[Groucho2004]

ok and what is this batch file you speak of?

Here is an example:

cls
@echo off
echo.
SET ENCODER=E:\Apps\VideoTools\x264\x264.exe
SET SOURCE=F:\Ratatouille\Ratatouille.avs
SET STATS=F:\Ratatouille\Ratatouille.stats
SET TARGET=F:\Ratatouille\Ratatouille.264
SET BITRATE=8000


SET PASS1="%SOURCE%" --bitrate %BITRATE% --profile high --level 4.0 --tune film --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --b-pyramid none --nal-hrd vbr --pic-struct --weightp 1 --aud --qpmin 0 --vbv-bufsize 15000 --vbv-maxrate 15000 --keyint 48 --bframes 3 --ipratio 1.1 --pbratio 1.1 --direct auto --threads auto --thread-input --stats "%STATS%" --pass 1 --slow-firstpass --ref 2 --me hex --subme 2 --trellis 0 --output NUL

SET PASS2="%SOURCE%" --bitrate %BITRATE% --profile high --level 4.0 --tune film --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --b-pyramid none --nal-hrd vbr --pic-struct --weightp 1 --aud --qpmin 0 --vbv-bufsize 15000 --vbv-maxrate 15000 --keyint 48 --bframes 3 --ipratio 1.1 --pbratio 1.1 --direct auto --threads auto --thread-input --stats "%STATS%" --pass 2 --subme 8 --ref 6 --me umh --trellis 1 --output "%TARGET%"

"%ENCODER%" %PASS1%
echo.
"%ENCODER%" %PASS2%

[Cman21]

ok, i think i get that and for CQ and not 2 pass bitrate i assume i remove the "BITRATE" and the remove pass 2 and set pass 1 to the settings i originally posted and the file extension is a .bat?

also incorporating that multithreaded x64 version of avisynth is a no go as i have no idea what filters do what i want to do to remake that original filter since they renamed everything and even changed parameters in them....

like:

cls
@echo off
echo.
SET ENCODER=C:\Program Files (x86)\MeGui\tools\x264\x264.exe
SET SOURCE=D:\Encoding Work\test.avs
SET TARGET=D:\Encoding Work\test_1080p.264


SET PASS1="%SOURCE%" --preset slow --tune animation --crf 19.0 --chroma-qp-offset 2 --level 4.1 --threads auto --thread-input --bframes 3 --ref 4 --slices 4 --aud --nal-hrd vbr --b-pyramid strict --keyint 24 --min-keyint 2 --vbv-bufsize 30000 --vbv-maxrate 40000 --weightp 0 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --output "%TARGET%"

"%ENCODER%" %PASS1%

[Groucho2004]

also incorporating that multithreaded x64 version of avisynth is a no go as i have no idea what filters do what i want to do to remake that original filter since they renamed everything and even changed parameters in them....


Either you acquire that knowledge about Avisynth and filters (Doom9 is a very good starting point) or you keep using a gui.
Your choice.


like:

cls
@echo off
echo.
SET ENCODER=C:\Program Files (x86)\MeGui\tools\x264\x264.exe
SET SOURCE=D:\Encoding Work\test.avs
SET TARGET=D:\Encoding Work\test_1080p.264


SET PASS1="%SOURCE%" --preset slow --tune animation --crf 19.0 --chroma-qp-offset 2 --level 4.1 --bframes 3 --ref 4 --slices 4 --aud --nal-hrd vbr --b-pyramid strict --keyint 24 --min-keyint 2 --vbv-bufsize 30000 --vbv-maxrate 40000 --weightp 0 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --output "%TARGET%"

"%ENCODER%" %PASS1%

Pretty much, yes.

[Cman21]

cool, is that string all valid x264 arguments, i dont think they are right? i used media info on the other one i encoded earlier and it has this listed under Encoding settings, is that what i use?

cabac=1 --ref=4 --deblock=1:1:1 --analyse=0x3:0x113 --me=umh --subme=8 --psy=1 --psy_rd=0.40:0.00 --mixed_ref=1
--me_range=16 --chroma_me=1 --trellis=1 --8x8dct=1 --cqm=0 --deadzone=21,11 --fast_pskip=1 --chroma_qp_offset=0
--threads=12 --sliced_threads=0 --slices=4 --nr=0 --decimate=1 --interlaced=0 --constrained_intra=0 --bframes=3
--b_pyramid=1 --b_adapt=2 --b_bias=0 --direct=3 --weightb=1 --weightp=0 --keyint=24 --keyint_min=2 --scenecut=40
--intra_refresh=0 --rc_lookahead=24 --rc=crf --mbtree=1 --crf=19.0 --qcomp=0.60 --qpmin=10 --qpmax=51 --qpstep=4
--vbv_maxrate=40000 --vbv_bufsize=30000 --crf_max=0.0 --ip_ratio=1.40 --aq=1:0.60 --nal_hrd=vbr

well i have no problem with learning i just dont like relearning the same thing with different names and functionality by reading 500 lines of a readme with i dont even know if it is worth the effort in the return it might give.

[nm]

also incorporating that multithreaded x64 version of avisynth is a no go as i have no idea what filters do what i want to do to remake that original filter since they renamed everything and even changed parameters in them....

Eh? You only need to add SetMTMode calls. Nothing is changed in the MVTools2 inteface, AFAIK.

cool, is that string all valid x264 arguments, i dont think they are right?

Sure they are. See x264 --fullhelp.

well i have no problem with learning i just dont like relearning the same thing with different names and functionality by reading 500 lines of a readme with i dont even know if it is worth the effort in the return it might give.

It's a reference, you don't need to read it like a book. If you encounter a parameter that you don't know, look it up. If you want to do something specific, see if there's a parameter for that. Usually people only need --preset x, --tune y, --crf/--bitrate and settings for some specific hardware decoder or Blu-ray.

[Cman21]

EDIT: fuk me i grabbed MaskTools instead -_- ...... lets look at that again. also to use this do all i need to change in the aviscript is the LoadPlugin to the new version?

[Groucho2004]

The encoding options depend on the target playback device and the time you're willing to invest. If you are not limited by Bluray restrictions you would usually just pick a preset, CRF value and possibly a tuning option.

[Cman21]

oh that is nice so i only need --preset slow --tune animation --crf 19.0 --level 4.1 --output "%TARGET% in that command line? also megui doesnt like this new mvtools so do i need virtualdub x64 to test my avs script right?

EDIT: wait if im using the mvtools 64bit filter does the whole script have to built around 64bit filters? and how do i get the source in the avisyntch script? as i cant use:

LoadPlugin("C:\Program Files (x86)\megui\tools\dgavcindex\DGAVCDecode.dll")
AVCSource("D:\Encoding Work\Test.dga")

right?

EDIT 2: ok i got it to load in virtualdub now i will test this batch file and see what happens

EDIT 3: ok running this .bat

cls
@echo off
echo.
SET ENCODER=C:\Program Files (x86)\MeGui\tools\x264\x264.exe
SET SOURCE=D:\Encoding Work\test.avs
SET TARGET=D:\Encoding Work\test_1080p.264


SET PASS1="%SOURCE%" --preset slow --tune animation --crf 19.0--level 4.1 --output "%TARGET%"

"%ENCODER%" %PASS1%

with this AVIScript

# Set DAR in encoder to 16 : 9. The following line is for automatic signalling
global MeGUI_darx = 16
global MeGUI_dary = 9
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\DirectshowSource.dll")
DirectShowSource("D:\Encoding Work\test.mkv")

LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins64\mvtools2.dll")
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

crop( 2, 2, -2, -2)
LanczosResize(1920,1080) # Lanczos (Sharp)

gives me a CPU usage of , ya..., ~20%

[TheRyuu]

LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\MaskTools.dll")
global MeGUI_darx = 16
global MeGUI_dary = 9
LoadPlugin("C:\Program Files (x86)\megui\tools\dgavcindex\DGAVCDecode.dll")
AVCSource("D:\Encoding Work\Test.dga")

super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

crop( 2, 2, -2, -2)
LanczosResize(1920,1080) # Lanczos (Sharp)

program --preset slow --tune animation --crf 19.0 --chroma-qp-offset 2
--level 4.1 --bframes 3 --ref 4 --slices 4 --aud --nal-hrd vbr --b-pyramid strict
--keyint 24 --min-keyint 2 --vbv-bufsize 30000 --vbv-maxrate 40000 --weightp 0
--colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --output "output" "input"

if there is any other info you need just let me know.

You're clearly limited by your avisynth script (which is single threaded). Encode to a lossless format first such as ut video, then pass that to x264. This limits that amount of time you have to filter your source to 1.

As for the other person having issues reaching 100% usage, you're limited by either avisynth or the decoding speed of whatever you're decoding (dgavcdecode is slow). Try using something faster like a ffmpeg-mt build of ffms2 (http://ffmpegsource.googlecode.com/files/ffms2-mt-r375.7z).

Whoever's claiming that hypertheading is causing this wrong 'display issue' is incorrect. If you cannot reach 100% cpu usage with our without HT you're limited in some capacity by something in your encoding chain. HT will give about a 25% performance gain on x264 so do please keep it on.

So if you're limited by something in your encoding chain you can try encoding your filtered source to lossless so you only have to waste cpu time doing this once (then encode to x264 as many times as you want), or if you're limited by decoding speed you can try using a decoder which is faster/threaded.

[Cman21]

You're clearly limited by your avisynth script (which is single threaded). Encode to a lossless format first such as ut video, then pass that to x264. This limits that amount of time you have to filter your source to 1.

ok, can you give me an example of what you mean by this, i dont quite understand?

[Blue_MiSfit]

Cman21:
You need to multithread your Avisynth script. Your decoder is painfully slow, and mdegrain2 is also very slow without MT.

Note my changes below:


LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\MaskTools.dll")
global MeGUI_darx = 16
global MeGUI_dary = 9

#DGAVCIndex is totally deprecated. Don't use it.
#LoadPlugin("C:\Program Files (x86)\megui\tools\dgavcindex\DGAVCDecode.dll")
#AVCSource("D:\Encoding Work\Test.dga")

ffvideosource("d:\encoding work\foo.avi")
#or uncomment below to use dss2
#dss2 ("d:\encoding work\foo.avi")


MT("""
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
""" 8,16)

#This looks like trouble
#crop( 2, 2, -2, -2)

LanczosResize(1920,1080) # Lanczos (Sharp)


Or, using SetMTMode (try both)

SetMTMode(5,0)
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\MaskTools.dll")
global MeGUI_darx = 16
global MeGUI_dary = 9

#DGAVCIndex is totally deprecated. Don't use it.
#LoadPlugin("C:\Program Files (x86)\megui\tools\dgavcindex\DGAVCDecode.dll")
#AVCSource("D:\Encoding Work\Test.dga")

ffvideosource("d:\encoding work\foo.avi")
#or uncomment below to use dss2
#dss2 ("d:\encoding work\foo.avi")

SetMTMode(2)

super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

#This looks like trouble
#crop( 2, 2, -2, -2)

LanczosResize(1920,1080) # Lanczos (Sharp)

Distributor()


You may run into stability issues with SetMTMode, but it's much faster, in my experience.

If you're doing 2 pass encoding (which you shouldn't do unless you must hit a specific file size), it may be worth it to render out a lossless file like TheRyuu suggests. I'd use x264, and do it just like this:

x264.exe foo.avs --qp 0 --preset faster --tune fastdecode --keyint 24 --output foo_lossless.mp4


Then take the resulting MP4 file and transcode with x264. Look for an x264 build that uses a MT version of ffms2. Most aren't.

x264.exe foo_lossless.mp4 --index foo_lossless.idx --pass 1 bitrate xxx --vbv-maxrate xxx --vbv-bufsize xxx --tune film --output nul
x264.exe foo_lossless.mp4 --index foo_lossless.idx --pass 2 bitrate xxx --vbv-maxrate xxx --vbv-bufsize xxx --tune film --output foo_2pass.mp4


Alternatively, load your MP4 into avisynth with dss2 or something, and use ffdshow or coreavc to decode.

Derek

[Cman21]

yes, thank you! i just tested a script that did nothing but resize and it runs @ ~100% i'll give those implementations a try tomorrow and see how they turn out. i really hope i can get this working correctly, thanks for all your help.

[Didée]

[* This was a notice about a small syntax error in the MT() script *]

[Blue_MiSfit]

OOPS!!! Thanks, Didee :)

Fixed

[TheRyuu]

*snip*

You should probably use mkv for lossless h264 since it's just an intermediate step and ffms2 will probably handle it better that way (if x264 is using that as it's input).

[Cman21]

sry for being slow to reply to you but i have been quite busy and unable to return to this hobby. well anyway, i get a script error message saying it expected to see a , or ) on that last line of the MT, the """ 8,16) and so i tried the second one and it runs but is still using ~20%.... what is the differences between these implementations and where could i read up on how to properly use them?

also i just noticed that my x264 has mostly been running off the first core around 90% with the 3 others bouncing around 5-10%, there are 8 total with the hyperthreading.

i cant believe i didnt notice this earlier! this could be a very good hint to my problem, but i dont know.

[Blue_MiSfit]

The first core is AviSynth :)

Post your whole script, please.

Derek

[Cman21]

well, now that i finally get a moment to myself again im here to tell you that you missed a comma on your MT implementation. line """ 8,16) needed to be """,8,16) and that worked, and im currently testing CPU usage.

Original:
MT("""
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
""" 8,16)

Corrected:
MT("""
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
""",8,16)

EDIT: that did it im now at ~80% consistently, thank you so very much for the time and effort you put into helping my figure out this issue.

Oh and Groucho2004 though it may have sounded like at some point i hated you, as initially you where no help and only making me frustrated, but im glad you stayed in there and now i truly thankful for you teaching me how to make these .bat files to directly encode at x64 and not x32 that megui is set to.

[LoRd_MuldeR]

Are you only checking CPU usage or also looking at the actual throughput (frames per second) ???

Increased CPU usage doesn't necessarily mean that you script is processing any faster, so you should keep an eye on that as well...

[Cman21]

yes, my FPS went from .8 to 2.8 so my actual throughput has increased as well.

[Didée]

Increased CPU usage doesn't necessarily mean that you script is processing any faster,
And where do you think would the increased CPU usage have gone in/to?

If he had 17% CPU before (with Avisynth single-threaded) and now he has 80% CPU with Avisynth multi-threaded, it is pretty safe to assume that the fps have increased, too. :D

Edit -- ah, indeed. ;)

[Groucho2004]

Oh and Groucho2004 though it may have sounded like at some point i hated you

Not at all. :)

truly thankful for you teaching me how to make these .bat files to directly encode at x64 and not x32 that megui is set to.

I did?

[Cman21]

yes i never heard about using x264 before other than through a gui and when i looked into my x264.exe folder i saw that there was also a x64 version which help improve speeds even more.

[ajp_anton]

And where do you think would the increased CPU usage have gone in/to?

If he had 17% CPU before (with Avisynth single-threaded) and now he has 80% CPU with Avisynth multi-threaded, it is pretty safe to assume that the fps have increased, too. :D

Edit -- ah, indeed. ;)I've had some weird Avisynth behavior where CPU usage doubles while speed goes down to 1/10.

[ginhead]

Quick word of thanks to Blue_MiSfit, TheRyuu and the rest of the contributors to this thread.

Using the ffmpeg-mt build of ffms2 posted in this thread took me from 30-40% cpu utilization and ~24fps (both passes) to +90-100% cpu utilization and 45/38fps. this is for a blu-ray to portable re-encode on a new i7 laptop.