Current Patches, Where to get them, How they affect speed/output

[Egh]

Because testing showed that "threads = 3/2 * cores" works best. And that finding doesn't seem to be random at all ;)

The intuitive solution "threads = cores" would only give optimal performance if no thread ever becomes idle (needs to wait for another thread).

still random. nearly 25% cpu% loss (total % for both cores) just with 2 threads instead of 3. With the same .bat file of course. So you effectively imply in this scenario either one thread is idle 50% of time of both threads idle 25% of time. Suboptimal?

[LoRd_MuldeR]

still random. nearly 25% cpu% loss (total % for both cores) just with 2 threads instead of 3. With the same .bat file of course. So you effectively imply in this scenario either one thread is idle 50% of time of both threads idle 25% of time. Suboptimal?

I don't imply anything. I just say that in a "real life" application you cannot assume that threads are working 100% independently of each other. Threads need to synchronize now and then! And that means that any thread unavoidably will have some idle time while waiting for another thread. Also you must keep in mind that the "look ahead" part of x264 is not multi-threaded. Lookahead needs to run each time a new encoder thread is spawned. Now imagine you were running exactly four threads on a quadcore CPU. When the first thread/frame completes, only three encoder threads are left running in parallel. Now lookahead will need to run again before a new thread is spawned. With lookahead and the three encoder thread still all four cores are busy. But if the second thread/frame completes, while lookahead has not completed yet, only two threads are left running in parallel - one CPU core becomes idle. This easily explains why running more encoder threads than cores makes sense. Last but not least the multi-threading in x264 obviously is pretty optimal: When you configure the number of threads properly (cores * 3/2), then x264 will fully utilize a quadcore CPU and give ~4x speed of a single core.

[skystrife]

x264 r1136 x64 (unpatched) (http://www.mediafire.com/?irjnnnnwtmk) - Alternate Download (http://skystrife.com/x264/revision1136/x264.exe)
gcc 4.3.4 fprofiled build.

-------------------------

x264.1136M.x86.exe (http://www.mediafire.com/?znmzz21lnm5) - Alternate Download (http://skystrife.com/x264/x264.1136M.x86.exe) / x264.1136M.x64.exe (http://www.mediafire.com/?nknmwgtgw2t) - Alternate Download (http://skystrife.com/x264/x264.1136M.x64.exe)
gcc 3.4.5 fprofiled build with -march=pentium2. / gcc 4.3.4 fprofiled build.

Patches used:

x264_hrd_pulldown.11_interlace.diff
x264_win_zone_parse_fix_05.diff

[techouse]

x264_x86_r1136_techouse (http://techouse.project357.com/builds/x264_x86_r1136_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1136_techouse.txt)
GCC 4.3.3, fprofiled, -march=core2

x264_x64_r1136_techouse (http://techouse.project357.com/builds/x264_x64_r1136_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1136_techouse.txt)
GCC 4.3.4 20090328 (prerelease) (x64.generic.Komisar), fprofiled, -march=core2

Patches used:

x264_hrd_pulldown.11_interlace.diff
x264_win_zone_parse_fix_05.diff

[burfadel]

According to the GIT, the last patch was added 32 hours ago (faster CABAC RDO), and yet there are no 1137 builds...

www.x264.nl doesn't seem to be autoupdating it, at least quickly?...

[kemuri-_9]

According to the GIT, the last patch was added 32 hours ago (faster CABAC RDO), and yet there are no 1137 builds...

www.x264.nl doesn't seem to be autoupdating it, at least quickly?...

the latest patch wasn't released until 5 hours ago.
the devs often keep the patches locally for testing and only push to the global repository after testing is complete.
however the times are from the local repository, not from the global.

if you want the actual times it hits the global, you can check the x264-devel mailing list for subjects that start with 'commit: '
like r1137 (http://mailman.videolan.org/pipermail/x264-devel/2009-April/005792.html)

as for x264.nl it could either be on hold or there was a problem somewhere.

[skystrife]

x264 r1137 x64 (unpatched) (http://www.mediafire.com/?z3bnxnm1n14) - Alternate Download (http://skystrife.com/x264/revision1137/x264.exe)
gcc 4.3.4 fprofiled build.

-------------------------

x264.1137M.x86.exe (http://www.mediafire.com/?nmdgndqjmzn) - Alternate Download (http://skystrife.com/x264/x264.1137M.x86.exe) / x264.1137M.x64.exe (http://www.mediafire.com/?0gymdznnwtz) - Alternate Download (http://skystrife.com/x264/x264.1137M.x64.exe)
gcc 3.4.5 fprofiled build with -march=pentium2. / gcc 4.3.4 fprofiled build.

Patches used:

x264_hrd_pulldown.11_interlace.diff
x264_win_zone_parse_fix_05.diff

[burfadel]

the latest patch wasn't released until 5 hours ago.
the devs often keep the patches locally for testing and only push to the global repository after testing is complete.
however the times are from the local repository, not from the global.

if you want the actual times it hits the global, you can check the x264-devel mailing list for subjects that start with 'commit: '
like r1137 (http://mailman.videolan.org/pipermail/x264-devel/2009-April/005792.html)

as for x264.nl it could either be on hold or there was a problem somewhere.


Ah ok! that makes sense :)

[3ngel]

I'm getting this on Phenom with latest

x264.1137M.x86.exe --pass 1 --bitrate 1270 --stats ".stats" --progress --keyint 250 --bframes 16 --qpmin 10 --qpmax 51 --aq-mode 0 --psy-rd 0.8:0 --no-psnr --no-ssim --no-fast-pskip --mixed-refs --b-adapt 0 --trellis 2 --ref 9 --no-deblock --subme 7 --direct auto --me umh --merange 32 --nf --weightb --b-pyramid --partitions all --8x8dct --threads auto --thread-input --no-dct-decimate --level 41 --output NUL "t.avs"

Original image size 844*480

Second Pass has identical parameters

Same result with --no-asm too

http://img10.imageshack.us/img10/4880/glitch.png

EDIT:
About the same glitches (not exactly the same) i obtain with previous versions too back to about 1113

These types of glitches are present randomly on all the video

Tested with CoreAVC, MPC Decoder and with MPC, graphedit etc...

[komisar]

3ngel, what/whose build you use? Can you repeat this "bug" with other builds?

[3ngel]

http://skystrife.com/x264/x264.1137M.x86.exe

[kemuri-_9]

post the sample you're encoding so i can do testing on my phenom computers when i get off work.

[akupenguin]

@3ngel
Your artifacts are definitely produced by a median-predicted lossless codec, such as ffvhuff or lagarith. Either a decoder bug or a corrupt bitstream.
mpeg-like codecs do not have failure modes that look like that.

[3ngel]

Mmm... i see.
I've tested the script in VDub before feeding to encoder, and it's a regular not corrupted lagarith. Moreover playing the .avs (and the lags .avi within) all is regular.
Now i'm doing a new encoding.

[Dark Shikari]

I've seen that before as well--in a broken lagarith decode.

[3ngel]

But if it can be a lagarith issue, why i don't see the same in vdub?
Both VDub and avs use the same decoding routine (and i see both fine).

[LoRd_MuldeR]

But if it can be a lagarith issue, why i don't see the same in vdub?
Both VDub and avs use the same decoding routine (and i see both fine).

Try converting your video to HuffYUV or FFv1 in VirtualDub. Then try to feed that into x264...

[3ngel]

Yes, doing that right now (doing uncompressed).

[techouse]

x264_x86_r1137_techouse (http://techouse.project357.com/builds/x264_x86_r1137_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1137_techouse.txt)
GCC 4.3.3, fprofiled, -march=core2

x264_x64_r1137_techouse (http://techouse.project357.com/builds/x264_x64_r1137_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1137_techouse.txt)
GCC 4.3.4 20090408 (prerelease) (x64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_hrd_pulldown.11_interlace.diff
x264_win_zone_parse_fix_05.diff

[Egh]

Also you must keep in mind that the "look ahead" part of x264 is not multi-threaded. Lookahead needs to run each time a new encoder thread is spawned. Your explanation is very good, however can you describe in more detail how lookahead works? For instance, how much time or cpu% is spent for lookahead thread. As well, how do the things work in non-multithreaded mode, on single core?

[LoRd_MuldeR]

Your explanation is very good, however can you describe in more detail how lookahead works? For instance, how much time or cpu% is spent for lookahead thread.

Read this please:
http://git.videolan.org/gitweb.cgi?p=x264.git;a=blob_plain;f=doc/threads.txt;h=3777b516117c6a7dc8e4943dcee0386449dc405c;hb=1fda88277f6b2eda27a0f741d58b31532ad0664d

"New threading method: frame-based" is what x264 uses nowadays. For further information you may want to ask a x264 developer :p

As well, how do the things work in non-multithreaded mode, on single core?

I guess it works the same way. Just that only one encoder thread is created at a time.

[3ngel]

Yes, doing that right now (doing uncompressed).

UPDATE
So apparently, it is a lagarith issue, but indeed is a VDub Issue wich corrupt the video doing a "DirectStream Encoding" from an Integral LAGS Video.
So, source lags is intact, you do a "DirectStream" (like to join two lags avis), and the results is a corrupted lags.
So it's ok on x264.

I take the occasion to ask:

Is it possible to add a switch to set the "priority"?
Now everytime i have to go to taskmanager in order to put it in "Below normal" priority (because in normal priority every application is slowed down).

Thank you very much

[kemuri-_9]

Is it possible to add a switch to set the "priority"?
Now everytime i have to go to taskmanager in order to put it in "Below normal" priority (because in normal priority every application is slowed down).

use
start /belownormal x264 [opts]

[komisar]

3ngel, or use patch k.61.x264_thread_priority.02.diff (http://komisar.gin.by/x.patch/k.61.x264_thread_priority.02.diff) with --threads-boost <integer> Tune priority for encoding threads [0]
-2: LOWEST
-1: BELOW_NORMAL
0: NORMAL
1: ABOVE_NORMAL
2: HIGHEST
--thread-input-boost <integer> Tune priority for input thread (Values as in 'threads-boost') [0]

[3ngel]

@kemuri-_9
Thanks for the suggestion

@komisar
Thanks, is this patch integrated in the link downloadable in the official thread, or alternative builds?

[komisar]

3ngel, I make this patch for my kMod and kVAQmod builds. Also present variant for thread-pool patch by BugMaster.

[Egh]

"New threading method: frame-based" is what x264 uses nowadays. For further information you may want to ask a x264 developer :p

I guess it works the same way. Just that only one encoder thread is created at a time.

Basically that means threads parameter defines only the number of encoding threads, and there can be one additional input thread and the main thread of application as well which does B-adapt and ratecontrol (not sure if that is what you called "look ahead" thread).

Still not sure yet about timing issues, how much time is taken by lookahead and what kind of scenarios are possible regarding any of the threads committing ealier than the rest. As well does really mean that the threads parameter now specifies how many frames are encoded at once? If I specify it as 10, would it try to encode 10 at once?

[Dark Shikari]

As well does really mean that the threads parameter now specifies how many frames are encoded at once? If I specify it as 10, would it try to encode 10 at once?Yes, it would.

[Manao]

3ngel : lagarith creates those broken frames randomly at decoding. You could encode the same file three times and end up with or without broken frames at different places in the encoded files.

[skystrife]

x264 r1139 x64 (unpatched) (http://www.mediafire.com/?yzt1gyyi5ow) - Alternate Download (http://skystrife.com/x264/revision1139/x264.exe)
gcc 4.3.4 fprofiled build.

-------------------------

x264.1139M.x86.exe (http://www.mediafire.com/?oxzioj2vmym) - Alternate Download (http://skystrife.com/x264/x264.1139M.x86.exe) / x264.1139M.x64.exe (http://www.mediafire.com/?idymnzmkk5y) - Alternate Download (http://skystrife.com/x264/x264.1139M.x64.exe)
gcc 3.4.5 fprofiled build with -march=pentium2. / gcc 4.3.4 fprofiled build.

Patches used:

x264_hrd_pulldown.11_interlace.diff
x264_win_zone_parse_fix_05.diff

[TheRyuu]

hi

pm me if you want a build with threaded slicetype

[kemuri-_9]

x264-r1139 (http://sempai-net.com/ss-trainee/x264-r1139.rar)

Experimental build (gcc 4.3.3, fprofiled) with v14 of the threaded slicetype patch.
(x264-r1134-threaded-slicetype-v14.diff)
This is known to seg fault using too many bframes (>=10) or too high a lookahead (see correlation between bframes and lookahead value below)

I wouldn't use this for any 'serious' encoding. ;)

To use 'threaded slicetype':
--lookahead 30 (I guess good starting point, I really have no idea what this value should be but have seen this in the past)

The higher the lookahead you use, the less bframes it will take to seg fault it. :p

I wouldn't go higher than 8 or 30 for bframes or lookahead values respectively.

You shouldn't have posted a build with the patch as it's known to be broken....
it could distribute across the internet without the 'broken under conditions x,y,z' disclaimer and complaints will only build up.
you should wait until it makes the repository....

but... you also forgot to mention that it only seg faults on the above conditions for b-adapt 2... b-adapt 1 should be fine.

[LoRd_MuldeR]

You shouldn't have posted a build with the patch as it's known to be broken....
it could distribute across the internet without the 'broken under conditions x,y,z' disclaimer and complaints will only build up.
you should wait until it makes the repository....

but... you also forgot to mention that it only seg faults on the above conditions for b-adapt 2... b-adapt 1 should be fine.

May I ask what the status of this patch is? Is threaded look-ahead a work in progress is or this more a rejected/suspended patch?

[kemuri-_9]

May I ask what the status of this patch is? Is threaded look-ahead a work in progress is or this more a rejected/suspended patch?

it's a work in progress....
it gets worked on when Mike (DaKaz on irc) has time, which doesn't come around the most often.
the latest version of the patch was v14; that may help with signifying how much progress has gone into it.

from what i understand, it's been mostly having issues on windows, whereas linux has less (if any) problems....

[techouse]

x264_x86_r1139_techouse (http://techouse.project357.com/builds/x264_x86_r1139_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1139_techouse.txt)
GCC 4.3.3, fprofiled, -march=core2

x264_x64_r1139_techouse (http://techouse.project357.com/builds/x264_x64_r1139_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1139_techouse.txt)
GCC 4.3.4 20090408 (prerelease) (x64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_hrd_pulldown.11_interlace.diff
x264_win_zone_parse_fix_05.diff

[techouse]

x264_x64_r1140_unpatched (http://techouse.project357.com/builds/revision1140/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1140/x264.md5)
GCC 4.3.4 20090408 (prerelease) (x64.core2.Komisar), unpatched, generic, fprofiled

________________________________________________________________________________

x264_x86_r1140_techouse (http://techouse.project357.com/builds/x264_x86_r1140_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1140_techouse.txt)
GCC 4.3.3, fprofiled, -march=core2

x264_x64_r1140_techouse (http://techouse.project357.com/builds/x264_x64_r1140_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1140_techouse.txt)
GCC 4.3.4 20090408 (prerelease) (x64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_hrd_pulldown.11_interlace.diff
x264_win_zone_parse_fix_05.diff

[JEEB]

Since I've been building x264 for some time now, posting here as well.

x264 r1143 x86
download (http://jeeb.fiveforty.jp/x264/1143/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1143/relnotes.txt)

built on Apr 19 2009, gcc: 4.3.3
fprofiled, -march=i686


x264 r1143 x64
download (http://jeeb.fiveforty.jp/x264/1143_x64/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1143_x64/relnotes.txt)

built on Apr 19 2009, gcc: 4.3.4 20090220 (prerelease) (x64.generic.Komisar)
fprofiled, -march=core2


Both patched with:

x264_win_zone_parse_fix_05.diff
x264_hrd_pulldown.11_interlace.diff

[jefrey]

Hi there, i have a strange probleme, since rls 1139, megui or x264.exe didnt show the quant and compressions infos, when the job is done. in megui there is no info tree and cli shows only ready, push any button to continue:(

what could it be?

[nurbs]

Works fine for me with megui and x264 1139.

[jefrey]

very strange! i didnt change anything :(

[techouse]

x264_x64_r1145_unpatched (http://techouse.project357.com/builds/revision1145/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1145/x264.md5)
GCC 4.3.4 20090408 (prerelease) (x64.core2.Komisar), unpatched, generic, fprofiled

________________________________________________________________________________

x264_x86_r1145_techouse (http://techouse.project357.com/builds/x264_x86_r1145_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1145_techouse.txt)
GCC 4.3.3, fprofiled, -march=core2

x264_x64_r1145_techouse (http://techouse.project357.com/builds/x264_x64_r1145_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1145_techouse.txt)
GCC 4.3.4 20090408 (prerelease) (x64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_hrd_pulldown.11_interlace.diff
x264_win_zone_parse_fix_05.diff

[Sharktooth]

request: skystrife's updated build or
x264 x86 static built with x264_hrd_pulldown.11_interlace.diff and x264_win_zone_parse_fix_05.diff patches, fprofiled, mp4 output, pthreads, GCC 3.4.x, -march=pentium2

[imk]

ICC builds of r1145M (http://imk.cx/pc/x264/x264-r1145M-imk.7z)

Build info can be found here (http://imk.cx/pc/x264/win_build_info.txt) and it's also included in the .7z

[alexins]

request: skystrife's updated build or
x264 x86 static built with x264_hrd_pulldown.11_interlace.diff and x264_win_zone_parse_fix_05.diff patches, fprofiled, mp4 output, pthreads, GCC 3.4.x, -march=pentium2

x264 Video Codec rev. 1145 x86 -march=pentium2, gcc 3.4.5, fprofiled, ... (http://www.xvidvideo.ru/component/option,com_docman/task,doc_download/gid,1633/)

[Sharktooth]

thanx

[turbojet]

Is there a patch that if --level is used and exceeded it would enforce the maximum?

For example: 1920x1080p 23.976 fps input, "x264 --level 4.0 --ref 8 --b-pyramid" would be changed to "x264 --level 4.0 --ref 4 --b-pyramid --vbv-maxrate 25000 --vbv-buffer 25000" instead of x264 warning it would say something like 'L4.0 --ref 4 forced for 1920x1080 resolution and using 25 mbps maximum'

But it shouldn't touch if it's below the maximum such as "x264 --level 4.0 --ref 1 --vbv-maxrate 10000 --vbv-buffer 25000" nothing would be changed.

[JEEB]

x264 r1148 32bit
download (http://jeeb.fiveforty.jp/x264/1148/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1148/relnotes.txt)

built on May 10 2009, gcc: 4.3.3
fprofiled, -march=i686


x264 r1148 64bit
download (http://jeeb.fiveforty.jp/x264/1148_x64/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1148_x64/relnotes.txt)

built on May 10 2009, gcc: 4.3.4 20090220 (prerelease) (x64.generic.Komisar)
fprofiled, -march=core2


Both patched with:

x264_win_zone_parse_fix_05.diff
x264_hrd_pulldown.11_interlace.diff

[skystrife]

x264 r1148 x64 (unpatched) (http://skystrife.com/x264/revision1148/x264.exe)
gcc 4.3.4 fprofiled build.

-------------------------

x264.1148M.x86.exe (http://skystrife.com/x264/x264.1148M.x86.exe) / x264.1148M.x64.exe (http://skystrife.com/x264/x264.1148M.x64.exe)
gcc 3.4.5 fprofiled build with -march=pentium2. / gcc 4.3.4 fprofiled build.

Patches used:

x264_hrd_pulldown.11_interlace.diff
x264_win_zone_parse_fix_05.diff

[techouse]

x264_x64_r1148_unpatched (http://techouse.project357.com/builds/revision1145/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1148/x264.md5)
GCC 4.3.4 20090408 (prerelease) (x64.core2.Komisar), unpatched, generic, fprofiled

________________________________________________________________________________

x264_x86_r1148_techouse (http://techouse.project357.com/builds/x264_x86_r1148_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1148_techouse.txt)
GCC 4.3.3, fprofiled, -march=core2

x264_x64_r1148_techouse (http://techouse.project357.com/builds/x264_x64_r1148_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1148_techouse.txt)
GCC 4.3.4 20090408 (prerelease) (x64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_hrd_pulldown.11_interlace.diff
x264_win_zone_parse_fix_05.diff

[komisar]

And my 1148 builds... :)
CLI/VFW, fprofiled, mp4 output, pthread, avis input, gcc 4.3.4 20090408 (prerelease) (xXX.YY.Komisar)
http://komisar.gin.by/
Builds abbreviation:
clear (no patches)
kGIT01_x264_custom_strtok_r.r1089.diff (by BugMaster)
x264_hrd_pulldown.11_interlace.diff
kModkGIT plus:
01_x264_debug_defines.r1089.diff (by BugMaster)
01_x264_fix_float_point_exception.r1089.diff (by BugMaster)
01_x264_fix_stats_file_work.r1089.diff (by BugMaster)
01_x264_multithreading_bug_check.r1089.diff (by BugMaster)
02_bm_x264_error_memoryleaks.04.r1089.diff (by BugMaster)
03_bm_x264_thread_pool.02.r1089.diff (by BugMaster)
k.62.x264_thread_priority_with_pool.02.diff (by Komisar)
k.76.k_Cosmetic.03.diff (by Komisar)
k.66.k_Profile.01.diff (by Komisar)
k.56.x264_log_file.03k.diff (by Komisar)
k.60.x264_restore_console_title.diff (by Komisar)
kAVAQkMod plus:
x264_AutoVAQ.02.diff (new version of VAQ modification by BugMaster)