Slow Encoding on a i7 980x

[fabje]

Hello,

I have an i7 980x and if I compare my speeds with other reviews etc I believe my speeds are way to slow.
First of all the specs of my pc:
i7 980x @ 3.33Ghz with Turbo on and HT off.
3GB Ram
1x 60GB OZC Vertex SSD
2x 500GB WD AAKS in RAID0
OS is Windows 7 32bit.
Windows 7 is installed on the SSD but the files that needs to be encode and x264.exe it self are on the RAID0.

I get in pass 1 and pass 2 around 23fps, I also noticed when I look at my task manager that x264.exe only uses around 25% of the CPU power so 75% of the cpu isn't doing nothing.

Somebody here has an idea how I can fix this so x264 is using at least 80% of the cpu or something like that.
I also noticed that xvid is also only using about 25% of the cpu, but it encodes here with about 200fps in pass1 and 175fps or something in pass2.

Anyone here that can help me?

[nurbs]

How about posting the x264 command line and avs script you used?

Most likely it's a combination of lots of b-frames with --b-adapt 2 that slows down your first pass. Most likely you use insanely slow settings, which is why the second pass is much slower than xvid. Also turning hyperthreading off is a bad idea.

[fabje]

How about posting the x264 command line and avs script you used?

Most likely it's a combination of lots of b-frames with --b-adapt 2 that slows down your first pass. Most likely you use insanely slow settings, which is why the second pass is much slower than xvid. Also turning hyperthreading off is a bad idea.
Oeps I forget to post that. I'm using Staxrip for encoding.

I turned hyperthreading off for a reason, when I want to encode an xvid with Staxrip it crashes when pass 2 begins, when I turned hyperthreading of it didn't crash anymore.

Here is pass1:
"D:\x264.exe" --pass 1 --bitrate 3936 --stats "D:\test.stats" --keyint 240 --min-keyint 24 --bframes 3 --direct auto --scenecut 0 --threads 6 --thread-input --subme 1 --partitions none --me dia --output NUL "D:\test.avs"

Pass 2:
"D:\x264.exe" --pass 2 --bitrate 3936 --stats "D:\test.stats" --keyint 240 --min-keyint 24 --ref 3 --mixed-refs --bframes 3 --weightb --direct auto --subme 6 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --scenecut 0 --threads 6 --thread-input --output "D:\test.avs"


I also just did this: http://www.techarp.com/showarticle.aspx?artno=520&pgno=0
And this are the results:
x264 HD BENCHMARK 3.0 RESULTS

Please do not compare it with older versions of the benchmark!
Please copy/paste everything below the line to to report your data

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Results for x264.exe r1342
--------------------------
encoded 1442 frames, 94.48 fps, 3898.70 kb/s
encoded 1442 frames, 94.59 fps, 3898.70 kb/s
encoded 1442 frames, 94.56 fps, 3898.70 kb/s
encoded 1442 frames, 94.78 fps, 3898.70 kb/s
encoded 1442 frames, 40.24 fps, 3967.11 kb/s
encoded 1442 frames, 39.92 fps, 3968.78 kb/s
encoded 1442 frames, 40.24 fps, 3969.91 kb/s
encoded 1442 frames, 40.07 fps, 3961.35 kb/s


System Details
--------------
Name Intel Core i7
Codename Gulftown
Specification Intel(R) Core(TM) i7 CPU 980x @ 3.33GHz
Core Stepping
Technology 32 nm
manufacturer Gigabyte Technology Co., Ltd.
vendor Gigabyte Technology Co., Ltd.
manufacturer Gigabyte Technology Co., Ltd.
Stock frequency 3200 MHz
Core Speed 3750.0 MHz

Northbridge Intel X58 rev. 12
Southbridge Intel 82801JR (ICH10R) rev. 00

Memory Type
Memory Size 3072 MBytes

Windows Version Microsoft Windows 7 (6.1) Ultimate Edition (Build 7600)

Number of processors 1
Number of threads 6
Number of threads 6 (max 32)
L2 cache 6 x 256 KBytes, 8-way set associative, 64-byte line size
Instructions sets MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, EM64T
Package (platform ID) Socket 1366 LGA (0x1)

Temperature 0 47°C (116°F) [0x2F] (TMPIN0)
Temperature 1 41°C (105°F) [0x29] (TMPIN1)
Temperature 2 62°C (143°F) [0x3E] (TMPIN2)
Temperature 0 47°C (116°F) [0x36] (Core #0)
Temperature 1 41°C (105°F) [0x3C] (Core #1)
Temperature 2 44°C (111°F) [0x39] (Core #2)
Temperature 8 45°C (112°F) [0x38] (Core #8)
Temperature 9 44°C (111°F) [0x39] (Core #9)
Temperature 10 50°C (121°F) [0x33] (Core #10)

[nurbs]

Is Staxrip still being developed? That command line uses settings that would be work with a more than one year old x264 version. Today half of the stuff is redundant. Especially the first pass settings are not needed. Any particular reason why you use --scenecut 0?

The settings themselves are actually pretty fast. Probably input bottleneck. Post your avs script.

Hyperthreading can bring up to 25% encoding speed boost with x264.

[fabje]

Yes Staxrip is still being developed. No I don't have a particular reason why I use --scenecut 0.

This are the settings inside Staxrip (all settings that are not mentioned are off):

Common:
Mode: Two Pass
B-Frams: 3
Reference Frames: 4
Loop Filer: Strength: -2 : Threshold: -1

B-Frames:
Adaptive = On
Weighted Prediction = On
Direct Mode: Audo
Bias: 0

Frame Options:
Scene Cut: 40
Cabac = On
Gop Size:
Min: 25 Max: 250

Analysis:
Mixed References: On
Meximum motion vector search ragnge: 16
Motion Estimation Method: Multi Hex
Subpixel Motion Estimation: 6
Trellis RD Quantization: Final MB

Partitions:
I4x4 = On
P8x8 = On
B8x8 = On
I8x8 = On
8x8 DCT = On

Rate Control:
AQ Mode: Move bits between frames
AQ Strength: 1.0
VBV Buffer Size: 0
VBV Max Bitrate: 0
VBV Initial Buffer: 0,9
QP Minimum: 10
QP Compression 0,6
IP Ratio: 1,4
PB Ratio: 1,3

Input/Output:
Thread Input: 6
Level: Unrestricted

Command Line:
Remove from preceding passes:
--ref --mixed-refs --weightb --subme --trellis --partitions --me --8x8dct --no-fast-pskip

Add to preceding passes:
--subme 1 --partitions none --me dia

This is the avs script that Staxrip generates:
AVCSource("D:\test.dga")
LoadCplugin("C:\Program Files\AviSynth 2.5\plugins\yadif.dll")
Yadif()
Crop(0,0,-0,-0)
Lanczos4Resize(1280,720)
Trim(43,37915)

[mp3dom]

No surprise that x264 is going slow.
Basically you're deinterlacing a (supposed) AVC 1080i to 720p on the fly.
DGAVCDec is *slow* in decoding frames, also the deinterlacing process took time and both DGAVCDec and AviSynth are single-threaded. So basically your script is the bottleneck.

[nurbs]

I presume your source is 1080i. Yadif is pretty fast, but deinterlacing HD still takes time, so maybe that's the reason. Your staxrip version is probably completely outdated. The settings in post #5 also don't match the settings in post #3. If you don't know what --scenecut 0 does then get rid of it because I don't see why you'd want to turn off scenecut detection.

[fabje]

I check the source with mediainfo:
General
ID : 1410
Complete name : D:\test.ts
Format : MPEG-TS
File size : 2.86 GiB
Duration : 32mn 38s
Overall bit rate : 12.5 Mbps

Video
ID : 69 (0x45)
Menu ID : 1 (0x1)
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Main@L4.0
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Duration : 32mn 37s
Bit rate : 11.5 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 fps
Resolution : 8 bits
Colorimetry : 4:2:0
Scan type : Interlaced
Scan order : Top Field First
Bits/(Pixel*Frame) : 0.221
Stream size : 2.61 GiB (91%)
Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.709-5, BT.1361, IEC 61966-2-4 709, SMPTE RP177

Audio
ID : 68 (0x44)
Menu ID : 1 (0x1)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Duration : 32mn 39s
Bit rate mode : Constant
Bit rate : 192 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Video delay : -2s 1ms
Stream size : 44.8 MiB (2%)
Language : Dutch

[fabje]

I did a short test what the result is without a Deinterlace filter, this is the result:

AVCSource("D:\test.dga")
Crop(0,0,-0,-0)
Lanczos4Resize(1280,720)
Trim(13880,21380)

Pass 1:
"D:\Staxrip\Settings\Applications\x264\x264.exe" --pass 1 --bitrate 3944 --stats "D:\test.stats" --bframes 3 --direct auto --filter -2:-1 --threads 6 --thread-input --subme 1 --partitions none --me dia --output NUL "D:\test.avs"

encoded 7501 frames, 25.76 fps, 3824.10 kb/s

Pass 2:
"D:\Staxrip\Settings\Applications\x264\x264.exe" --pass 2 --bitrate 3944 --stats "D:\test.stats" --ref 4 --mixed-refs --bframes 3 --weightb --direct auto --filter -2:-1 --subme 6 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --trellis 1 --me umh --threads 6 --thread-input --output "D:\test_EncoderOutput.264" "D:\test.avs"

encoded 7501 frames, 23.43 fps, 3933.40 kb/s

Well it is maybe 1 fps faster.

[nm]

So DGAVCDec remains as the main bottleneck. Use a multithreaded software decoder (ffmpeg-mt, CoreAVC, DiAVC, DivX7, ...) or get DGNVTools and an Nvidia GPU.

[fabje]

So DGAVCDec remains as the main bottleneck. Use a multithreaded software decoder (ffmpeg-mt, CoreAVC, DiAVC, DivX7, ...) or get DGNVTools and an Nvidia GPU.
I have a nVidia GPU with CUDA support.
Do you mean with DGNVTools this click (http://neuron2.net/dgdecnv/dgdecnv.html) ?

[nm]

Do you mean with DGNVTools this click (http://neuron2.net/dgdecnv/dgdecnv.html) ?

Correct.

[Blue_MiSfit]

Yes, that will be worlds faster than using DGAVCDec. It also lets you deinterlace in hardware, for free (no CPU required really). The quality is pretty good IMO - certainly not as good as TGMC or other top tier avisynth deinterlacers, but certainly as good as YADIF in my limited tests, if not a little better ;)

Derek

[Guest]

So DGAVCDec remains as the main bottleneck. Use a multithreaded software decoder (ffmpeg-mt, CoreAVC, DiAVC, DivX7, ...) or get DGNVTools and an Nvidia GPU. I can't believe libavcodec would be a bottleneck on an i7-980X. In fact, I think it would decode faster than CUVID. Where is the bottleneck according to you?

[nm]

I can't believe libavcodec would be a bottleneck on an i7-980X.
According to fabje's description, it seems to me that libavcodec is only using one thread for decoding. Perhaps the source only has one slice or it can't be deblocked properly when using slice-parallel decoding (in which case you'd need to set the "fast" flag in libavcodec and sacrifice accuracy)? How many threads does DGAVCDec ask for?

Edit: Also note that he's resizing to 720p and encoding with relatively light x264 settings. I'd estimate that decoding the 1080i source consumes more CPU time overall.

[Guest]

DGAVCDec uses only one decoding thread. I suppose you could be right.

It would interesting to compare to DGDecNV.

[fabje]

DGAVCDec uses only one decoding thread. I suppose you could be right.

It would interesting to compare to DGDecNV.
Well I hit the Donate button for DGDecNV but until now I didn't get a mail *hint*hint* :p

[Guest]

As my webpage says I do that stuff only twice a day, and I am not at my admin machine right now, so please be patient for a few hours. Sorry for the inconvenience and thank you for your donation.

[Rumbah]

It's not that difficult to run into a DGAVCDec bottleneck, at least with my C2Q 9650.

But you mustn't forget your resize filter that is single threaded. When converting for my PSP that's actually often the bottleneck (at least with laczos4 or spline36 and such).

In addition to that it seems you use an old version of Staxrip? 1.1.6.0 is the latest version. And doesn't the i7 have hyperthreading? Then you should use more than 6 threads. EDIT: Just saw that you have disabled HT.

[Motenai Yoda]

if u can use an avisynth-mt build u shoud be able to use more thread to take advantage of your cpu.
also u can test

AVCSource("D:\test.dga",deblock=false)
#Crop(0,0,-0,-0)#this 0,0,0,0 crop is useless
Lanczos4Resize(1280,720)
Trim(13880,21380)

[Blue_MiSfit]

When trying for very high speeds, using bicubic or maybe lanczos resizing is a good idea in my book.

Derek

[fabje]

if u can use an avisynth-mt build u shoud be able to use more thread to take advantage of your cpu.
also u can test

AVCSource("D:\test.dga",deblock=false)
#Crop(0,0,-0,-0)#this 0,0,0,0 crop is useless
Lanczos4Resize(1280,720)
Trim(13880,21380)
Do you mean this with Avisynth-mt (http://forum.doom9.org/showthread.php?t=94996&page=56)?

And why did you removed this line:
LoadCplugin("C:\Program Files\AviSynth 2.5\plugins\yadif.dll")
Yadif()

?

[Motenai Yoda]

I've copied from an your post...
now I'm using the set's 32bit build http://forum.doom9.org/showthread.php?t=148117
the code for mt should be


LoadCplugin("C:\Program Files\AviSynth 2.5\plugins\yadif.dll")
setmtmode(5,2)#mode5 for source work, 2 thread
AVCSource("D:\test.dga",deblock=false)
setmtmode(2)#change to mode 2 for filters
#Crop(0,0,-0,-0)#this 0,0,0,0 crop is useless
Yadif()
Lanczos4Resize(1280,720)
Trim(13880,21380)
ditributor()#I don't know exactly what hell it do, I think is a kind of threads "realigneator"

[fabje]

I've copied from an your post...
now I'm using the set's 32bit build http://forum.doom9.org/showthread.php?t=148117
the code for mt should be


LoadCplugin("C:\Program Files\AviSynth 2.5\plugins\yadif.dll")
setmtmode(5,2)#mode5 for source work, 2 thread
AVCSource("D:\test.dga",deblock=false)
setmtmode(2)#change to mode 2 for filters
#Crop(0,0,-0,-0)#this 0,0,0,0 crop is useless
Yadif()
Lanczos4Resize(1280,720)
Trim(13880,21380)
ditributor()#I don't know exactly what hell it do, I think is a kind of threads "realigneator"
Thanks, but why 2 thread in this line:
setmtmode(5,2)#mode5 for source work, 2 thread
Why not 6 because I have 6 cores?

[Motenai Yoda]

because:
1-more threads means more memory usage, 32bit applications should be limited to 2gb
2- more threads means more cpu overhead too, that affect the speed
3- x264 needs threads too, if u fill cpu with the avs script then x264 must wait.

usually for fast scripts 1 or 2 threads are enough.

ps I've a 920 cpu, with ht enable it run significantly faster.

[fabje]

because:
1-more threads means more memory usage, 32bit applications should be limited to 2gb
2- more threads means more cpu overhead too, that affect the speed
3- x264 needs threads too, if u fill cpu with the avs script then x264 must wait.

usually for fast scripts 1 or 2 threads are enough.

ps I've a 920 cpu, with ht enable it run significantly faster.
Ok thanks, I just did a test on 5min video.

Old: Pass1: encoded 7501 frames, 22.68 fps, 3072.28 kb/s | Pass2: encoded 7501 frames, 21.80 fps, 3025.01 kb/s
New: Pass1: encoded 7501 frames, 41.13 fps, 3072.22 kb/s | Pass2: encoded 7501 frames, 29.19 fps, 3025.01 kb/s

Old Duration: 00:11:48
New Duration: 00:07:51

And I have disabled HT for a reason what you can read in a older post ;)

[nm]

Note that you should never set deblock=false in an actual encode because skipping deblocking corrupts the video. It can be used for testing if DGAVCDec/libavcodec is a bottleneck and that's how it looks.

[fabje]

Note that you should never set deblock=false in an actual encode because skipping deblocking corrupts the video. It can be used for testing if DGAVCDec/libavcodec is a bottleneck and that's how it looks.
Thanks for the reminder :)
At this moment I'm testing a bigger video and I already notice that it only requires around 50% of my cpu power. Oke I'm still using DGAVCIndex. First I need to find out how I can get DGDecNV to work with Staxrip.

Someone else that have more tips?
Maybe an other setting in setmtmode?
At this moment x264.exe is using around 750MB of RAM.

[fabje]

Hmm I just wanted to encode an other video and I got this error:
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

Problem signature:
Problem Event Name: APPCRASH
Application Name: x264.exe
Applicable version: 0.0.0.0
Timestamp apply: 4c07de20
Fault Module Name: libavcodec.dll
Fault Module Version: 0.0.0.0
Fault Module Timestamp: 489a5621
Exception Code: 40000015
Exception Offset: 001678f7
OS Version: 6.1.7600.2.0.0.256.1
Locale ID: 1043
Additional Information 1: 0f0b
Additional Information 2: 0f0b0ea6f8f9837a3691b342da2fb08b
Additional Information 3: 0cb4
Additional Information 4: 0cb4614bd0b1b8fd43b0f51e77593a83

Read online privacy statement:
http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0413

If the online privacy policy is not available, please read our offline privacy notice:
C:\Windows\system32\en-US\erofflps.txt

Someone that has an idea?

[Guest]

Possibly corrupt source material. What is it and where did you get it?

[fabje]

Possibly corrupt source material. What is it and where did you get it?
It is a IPTV record of the match between Uruguay and France.
Is there a way to check if the source is fine? Like ProjectX does with SD recording.

I don't get it, it crashes at frame 74490 when it is encoding, when I directly go to that frame it crashes.
But when i first go to frame 74480 and then go frame-by-frame to 74490 it doesn't crash and the picture is fine.
I also watched the memory usage and that is below 1GB.

-----------------------------------------------------------
I just checked the file with MPEG2Repair and this is the result.
MPEG2Repair: D:\uruguay.vs.france.ts

Sequence Frame 18857(0-X) / Time 0:05:42 :
AudioError: Corrupted AC3 frame of 10278238 payload bytes at file offset 581850098
AudioWarning: Timestamp gap of 279.588659 sec. ending at file offset 581850098

Sequence Frame 34035(0-X) / Time 0:10:26 :
AudioError: Corrupted AC3 frame of 10290014 payload bytes at file offset 1051230450
AudioWarning: Timestamp gap of 279.583092 sec. ending at file offset 1051230450

Sequence Frame 49271(0-X) / Time 0:15:04 :
AudioError: Corrupted AC3 frame of 10290014 payload bytes at file offset 1520822490
AudioWarning: Timestamp gap of 279.586414 sec. ending at file offset 1520822490

Sequence Frame 64326(0-X) / Time 0:19:43 :
AudioError: Corrupted AC3 frame of 10290014 payload bytes at file offset 1990748606
AudioWarning: Timestamp gap of 279.587992 sec. ending at file offset 1990748606

Sequence Frame 182261(0-X) / Time 0:56:12 :
AudioError: Corrupted AC3 frame of 80598622 payload bytes at file offset 5671375006
AudioWarning: Timestamp gap of 2190.313280 sec. ending at file offset 5671375006

Sequence Frame 196992(0-X) / Time 1:00:51 :
AudioError: Corrupted AC3 frame of 10290014 payload bytes at file offset 6140068218
AudioWarning: Timestamp gap of 279.585081 sec. ending at file offset 6140068218

Sequence Frame 212039(0-X) / Time 1:05:34 :
AudioError: Corrupted AC3 frame of 10290014 payload bytes at file offset 6609705190
AudioWarning: Timestamp gap of 279.590414 sec. ending at file offset 6609705190

Sequence Frame 226863(0-X) / Time 1:10:13 :
AudioError: Corrupted AC3 frame of 10290014 payload bytes at file offset 7079633750
AudioWarning: Timestamp gap of 279.582870 sec. ending at file offset 7079633750

Sequence Frame 241903(0-X) / Time 1:14:51 :
AudioError: Corrupted AC3 frame of 10285598 payload bytes at file offset 7549305878
AudioWarning: Timestamp gap of 279.585414 sec. ending at file offset 7549305878

Sequence Frame 256967(0-X) / Time 1:19:30 :
AudioError: Corrupted AC3 frame of 10290014 payload bytes at file offset 8019029706
AudioWarning: Timestamp gap of 279.585370 sec. ending at file offset 8019029706

Sequence Frame 374793(0-X) / Time 1:56:04 :
AudioError: Corrupted AC3 frame of 80605246 payload bytes at file offset 11699636366
AudioWarning: Timestamp gap of 2190.311602 sec. ending at file offset 11699636366

Sequence Frame 389914(0-X) / Time 2:00:42 :
AudioError: Corrupted AC3 frame of 10290014 payload bytes at file offset 12169214870
AudioWarning: Timestamp gap of 279.584648 sec. ending at file offset 12169214870

Sequence Frame 399469(0-X) / Time 2:03:34 :
Info: End of H.264 sequence

Sequence Summary:

File Size Processed: 11.61 GB, Play Time: 00h:00m:00s
1040283.85 FPS (Average), 231601.00 Mbps (Average).
AC3 Audio: Unknown, 48.0 kHz, 0 kbps.
Dialog Normalization: -31.0 dB, Surround Mix Level: -6.0 dB
12 of 12 audio frames found with errors.
7176.484835 seconds of audio timestamp gaps.

End of Log

-----------------------------------------------------------

I just tested an encode of frame 74400 until 74500, when it is encoding I get an error it is this one.

Problem signature:
Problem Event Name: APPCRASH
Application Name: conhost.exe
Version of application: 6.1.7600.16385
Timestamp apply: 4a5bc271
Fault Module Name: ntdll.dll
Fault Module Version: 6.1.7600.16385
Fault Module Timestamp: 4a5bdadb
Exception Code: c0000005
Exception Offset: 0001dffc
OS Version: 6.1.7600.2.0.0.256.1
Locale ID: 1043
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789

Read online privacy statement:
http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0413

If the online privacy policy is not available, please read our offline privacy notice:
C:\Windows\system32\en-US\erofflps.txt

When I close this error, x264.exe keeps encoding until it is done and the result is fine.

[fabje]

I just demuxed the recording with TSRemuxer and I get two errors:
B-pyramid level 1 detected. Shift DTS to 2 frames
B-pyramid level 2 detected. Shift DTS to 3 frames

What does this mean?