Current Patches, Where to get them, How they affect speed/output [Archive] - Page 58

View Full Version : Current Patches, Where to get them, How they affect speed/output

Pages : 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 [58] 59 60 61 62 63 64 65 66 67 68 69

Emulgator

28th December 2009, 15:38

Lyris: h264visa. 30 days trial. (Thanks Lord Mulder !)

rack04

28th December 2009, 16:50

x264_x86_r1376M (http://www.mediafire.com/?nmznyjzngey)

Toolchain:
mingwrt-3.16
w32api-3.13
binutils-2.20
gcc-core-4.4.2
zlib-1.2.3
yasm-0.8.0
Patch:
x264_hrd_pd_interlace.16_r1369_fix.diff (http://komisar.gin.by/x.patch/last.used/x264_hrd_pd_interlace.16_r1369.diff)
Build:
./configure --extra-cflags="-march=core2"
make fprofiled VIDS="/c/x264/crew_4cif.y4m /c/x264/parkrun.1280x720.yuv"

aegisofrime

29th December 2009, 02:01

LoRd_MuldeR

29th December 2009, 02:06

I tried building one myself with those flags, but it just won't reach the performance of yours of other builds. Perhaps I'm missing quite a few links in the toolchain.

You did have ASM (YAMS) and PThreads enabled in your build, did you?

aegisofrime

29th December 2009, 02:27

You did have ASM (YAMS) and PThreads enabled in your build, did you?

You query that using ./configure right? If so, yes.

Pardon me for the noob questions, as I'm still a noob compiler. I merely following the guide over at Doom10. But seeing as I'm using a Phenom II I thought of compiling a -march=amdfam10 version for myself, to see if there's any performance increase.

LoRd_MuldeR

29th December 2009, 02:50

You query that using ./configure right? If so, yes.

Nope. I ask if you have both, YASM and PThreads, installed in your build environment.

If those are missing, your build will have ASM and/or multi-threading disabled, no matter how ./configure is called :p

And it would explain why your build is noticeably slower than others!

Things like the "--march" switch and the GCC version you use only have a very minor influence on the encoding speed.

However ICL builds are known to be bit faster than GCC builds, at least on Intel CPU's.

Dark Shikari

29th December 2009, 02:54

Nope. I ask if you have both, YASM and PThreads, installed in your build environment.

If those are missing, your build will have ASM and/or multi-threading disabled, no matter how ./configure is called :pNo, x264 will outright refuse to build if you don't have yasm installed.

aegisofrime

29th December 2009, 03:45

rack04

29th December 2009, 04:53

rack04, would you mind doing a build with ./configure --extra-cflags="-march=amdfam10"?

I tried building one myself with those flags, but it just won't reach the performance of yours of other builds. Perhaps I'm missing quite a few links in the toolchain.

x264_x86_r1376M (http://www.mediafire.com/?tet041nlyim)

Toolchain:
mingwrt-3.16
w32api-3.13
binutils-2.20
gcc-core-4.4.2
zlib-1.2.3
yasm-0.8.0
Patch:
x264_hrd_pd_interlace.16_r1369_fix.diff (http://komisar.gin.by/x.patch/last.used/x264_hrd_pd_interlace.16_r1369.diff)
Build:
./configure --extra-cflags="-march=amdfam10"
make fprofiled VIDS="/c/x264/parkrun.1280x720.yuv"

aegisofrime

29th December 2009, 05:55

Woot! Thanks Rack04! *downloads*

LoRd_MuldeR

29th December 2009, 13:58

I followed the guide here, http://doom10.org/index.php?topic=26.0, which didn't mention anything about Pthreads. Can I assume that it's not installed? Anyway, time to Google I guess!

Look what "./configure" says ;)

If you have ptheads installed in your build environment, it should look like this:

MuldeR@MULDER-PC /x264/x264-src
$ ./configure
Platform: X86
System: MINGW
asm: yes
avs input: avs
mp4 output: no
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no

You can run 'make' or 'make fprofiled' now.

Latest MinGW/GCC with pthreads included can be found here, for example:
http://www.xvidvideo.ru/2009-10-22-10-49-14/cat_view/28-cross-mingwgcc-x86x64/160-cross-mingwgcc-45-x86x64.html

(I guess you'd want that with "static" pthreads, so your x264.exe won't depends on the pthreads DLL)

qyot27

29th December 2009, 14:15

EDIT: Was poking around in my MinGW\bin folder and found that it already has the pthreads dlls. :/
The dlls aren't what you need, and that's the wrong directory. Go into /lib and find all the files with names starting with libpthread. Delete all of them except libpthreadGC2-static.a. Rename it to libpthreadGC2.a, and voila. You may need to adjust where it resides, though - depending on your MinGW environment, you could have several /lib directories (for instance, C:\msys\lib, C:\msys\local\lib, and C:\msys\mingw\lib; I install everything to root so I only have to deal with C:\msys\lib and C:\msys\local\lib, but other guides and packages have things set up differently).

The dlls in /bin are there for the .exe files that are dynamically linked to them, rather than statically linked. Without the dlls being there, those apps that need them wouldn't run. But they don't affect how things get compiled elsewise.

roozhou

29th December 2009, 17:10

My workaround:
1) Download pthreads-win32 source code. I got it from here (http://sources-redhat.mirrors.airband.net/pthreads-win32).
2) Create "pthreads" folder under x264 and copy all pthread source code files to it.
3) Add pthreads/pthreads.c to x264's Makefile.

This also applies to GPAC.

kemuri-_9

29th December 2009, 21:51

roozhou

30th December 2009, 03:11

that is the ugliest thing i have ever heard of.
Ugliest? It makes compilation much easier and allows better optimization from compilers. Look at mplayer's code. It has a lot of "internal" libraries including libmpeg2, liba52, libdvdread, libfaad2, etc.

Why should everyone search and download dependency libraries? Sometimes the official sites only provide source code and you have to configure && make && make install yourself. And you may have to download more tools to configure and compile these package. I hate that.
i do hope you don't do that for when lavf/ffms support is added.
Why? This will still work when lavf/ffms support is added.

aegisofrime

1st January 2010, 14:08

Look what "./configure" says ;)

If you have ptheads installed in your build environment, it should look like this:

MuldeR@MULDER-PC /x264/x264-src
$ ./configure
Platform: X86
System: MINGW
asm: yes
avs input: avs
mp4 output: no
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no

You can run 'make' or 'make fprofiled' now.

Latest MinGW/GCC with pthreads included can be found here, for example:
http://www.xvidvideo.ru/2009-10-22-10-49-14/cat_view/28-cross-mingwgcc-x86x64/160-cross-mingwgcc-45-x86x64.html

(I guess you'd want that with "static" pthreads, so your x264.exe won't depends on the pthreads DLL)

Yes, my screen looks like that, except that I have mp4 support with 'yes'. That shouldn't affect my speeds, or lack thereoff though, I guess.

LoRd_MuldeR

1st January 2010, 14:32

Yes, my screen looks like that

Then you are set up correctly :)

except that I have mp4 support with 'yes'. That shouldn't affect my speeds, or lack thereoff though, I guess.

It only means you have GPAC installed (which I did not have) and thus your build will have MP4 output enabled.

However to squish out the optimal speed with GCC, you should use ./configure --extra-cflags="-march=<cpu>" and make fprofiled VIDS=foobar.avs

Brazil2

2nd January 2010, 15:12

Bug or feature ?

I encode using --psy-rd=1:0.15 but when it's done MediaInfo reports psy_rd=1.0:0.2

So what is wrong ? Is x264 rounding the input value or is MediaInfo showing a rounded value ?
I'm quite sure that the correct value of 0.15 was shown with older builds of x264 like six months ago, maybe more.
if( p->analyse.b_psy )
s += sprintf( s, " psy_rd=%.1f:%.1f", p->analyse.f_psy_rd, p->analyse.f_psy_trellis );

:p
x264 rounds the values that get written to the file to one decimal, but the value used during encoding is not changed. AFAIK it's always been that way.
Probably because "psy_rd" was added to param2string() before the "film" tune started using Psy RDO 0.15 ^^

Lord Mulder, in this video (http://mulder.brhack.net/temp/100_5084.mkv) you have posted on the Avidemux' forums (http://avidemux.org/admForum/viewtopic.php?pid=40731#p40731), MediaInfo reports a value of 0.15 for Psy Trellis:
Writing library : x264 core 80 r1376M 3feaec2
Encoding settings : cabac=1 / ref=8 / deblock=1:-1:-1 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / psy_rd=1.00:0.15 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-3 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / mbaff=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 / wpredp=2 / keyint=750 / keyint_min=25 / scenecut=40 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=20.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Same goes with Avinaptic:
[ About H.264 encoding ]

User data: x264
User data: core 80 r1376M 3feaec2
User data: H.264/MPEG-4 AVC codec
...
User data: psy=1
User data: psy_rd=1.00:0.15

I've also found a video that I've encoded myself in October from a DVB capture with an older version of Avidemux using your libx264 build for Core2 which also reports 0.15:
Writing library : x264 core 75 r1259M dd026f2
Encoding settings : cabac=1 / ref=4 / deblock=1:-1:-1 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / psy_rd=1.00:0.15 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-3 / threads=3 / nr=0 / decimate=1 / mbaff=0 / bframes=3 / b_pyramid=0 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 / keyint=250 / keyint_min=25 / scenecut=40 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=22.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00

So I'm not sure about what is happening there. Maybe you have removed the rounding part of the code from your builds (which I personally find a good idea) ?
Anyway I would like to see the real value used for the encoding, I still don't understand the reason why this specific value is rounded.

LoRd_MuldeR

2nd January 2010, 16:04

Brazil2, I think you are reading those information from a stream that was encoded with my build of libx264 for Avidemux ;)

My builds include the following patch:
http://pastie.org/763983

(And no, it does not "remove" the rounding, because that would make the length of the string unpredictable. I simply increased the number of digits to 2)

Brazil2

2nd January 2010, 17:57

Brazil2, I think you are reading those information from a stream that was encoded with my build of libx264 for Avidemux ;)
My video was encoded with Avidemux and one of your 'special' builds as I said.
But your own video (http://mulder.brhack.net/temp/100_5084.mkv) was not encoded with Avidemux and reports 0.15 for Psy Trellis.

LoRd_MuldeR

2nd January 2010, 17:59

But your own video (http://mulder.brhack.net/temp/100_5084.mkv) was not encoded with Avidemux and reports 0.15 for Psy Trellis.

It was. Who said it wasn't encoded with Avidemux?

I only muxed the H.264 stream and the audio with AVI-Mux GUI, as Avidemux' MKV muxing capabilities aren't that reliable yet ;)

Brazil2

2nd January 2010, 18:15

It was. Who said it wasn't encoded with Avidemux?
I'm saying it because since r5341 the encoding informations are not available anymore in H264 streams encoded with Avidemux as already reported here:
http://forum.doom9.org/showthread.php?p=1350354#post1350354
http://forum.doom9.org/showthread.php?p=1351866#post1351866

LoRd_MuldeR

2nd January 2010, 18:26

Brazil2

2nd January 2010, 18:38

They are! But apparently not in MP4 output (see the quote in this (http://forum.doom9.org/showthread.php?p=1350419#post1350419) post for details). In AVI output they are present for sure ;)
Same goes for both MP4 and MKV output, see this post (http://forum.doom9.org/showthread.php?p=1351866#post1351866) about this issue.

Also I think this Avidemux-specific discussion is getting off-topic here...
My point is not Avidemux, my point is the real Psy Trellis value which was used for the encoding and which is sometimes correctly reported and sometimes not, and I'm just trying to understand why. And now I know that with your builds the real value is shown :)
And I still don't understand why it's not the default in x264 :p

LoRd_MuldeR

2nd January 2010, 18:54

And I still don't understand why it's not the default in x264 :p

Because the x264 developers obviously have decided that printing out the Psy RDO/Trellis with a precision of one digit is sufficient.

The only reason why I increased the precision in my builds is because I wanted to checked whether my attempts to tweak the x264 defaults were successful ;)

And I will remove those tweaks as soon as Avidemux adds a GUI option to configure the Psy-Trellis at runtime...

Brazil2

3rd January 2010, 21:24

Because the x264 developers obviously have decided that printing out the Psy RDO/Trellis with a precision of one digit is sufficient.
The question remains: why ?
Why is a 2 digits input allowed but only 1 digit reported ? There must be a reason, we don't do things without a reason, do we ?

The only reason why I increased the precision in my builds is because I wanted to checked whether my attempts to tweak the x264 defaults were successful ;)
And I will remove those tweaks as soon as Avidemux adds a GUI option to configure the Psy-Trellis at runtime...
Please don't remove this so usefull tweak!

LoRd_MuldeR

3rd January 2010, 21:36

Why is a 2 digits input allowed but only 1 digit reported ?

It's a float, so it can have more than 2 digits. Actually the smallest value could be ∓1.401 * 10^(-45) ;)

So at some point you must cut off when printing out a float value...

Please don't remove this so usefull tweak!

If Avidemux allows to set Psy-Trellis at runtime, then there's no need (and thus no reason) to tweak x264's default Psy-Trellis value.

rack04

11th January 2010, 15:52

Test version for AutoVAQ v4 patch.

x264_x86_r1376M (http://www.filedump.net/dumped/x264x86r1376m1263221344.zip)

Toolchain:
gcc-core-4.4.2 (x86.core2.Komisar)
Patch:
x264_hrd_pd_interlace.16_r1369_fix.diff (http://komisar.gin.by/x.patch/last.used/x264_hrd_pd_interlace.16_r1369.diff)
x264_AQ_experiments_v4.diff (http://stashbox.org/751926/x264_AQ_experiments_v4.diff)
Build:
./configure --extra-cflags="-march=core2"
make fprofiled VIDS="/c/x264/crew_4cif.y4m /c/x264/parkrun.1280x720.yuv"

aegisofrime

14th January 2010, 16:53

A new version is out! Could I request a "amdfam10" build for the 64-bit version for us AMD users? Thanks!

kemuri-_9

15th January 2010, 00:02

since Jeeb is afk for a bit, I've made a generic x86_64 build:

Platform: X86_64
System: MINGW
asm: yes
avs input: yes
lavf input: yes
ffms input: yes
mp4 output: yes
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no

ffmpeg r21217
ffms2 r257

x264_r1391_x64 (http://kemuri9.net/dev/x264/x264_r1391_x64.exe)

bob0r

15th January 2010, 01:06

Can you make it as:
http://kemuri9.net/dev/x264/revisionXXXX/x264.exe
http://kemuri9.net/dev/x264/revisionXXXX/x264.md5
?

Make x264.md5 as:
md5sum x264.exe | awk '{print $1}' >x264.md5

Then i can mirror it on x264.nl

Also can you x264.exe --version on that file so i know what GCC is used?

kemuri-_9

15th January 2010, 02:34

aegisofrime

15th January 2010, 02:56

where as my previous x86_64 build was with gcc 4.4.2, I did one now with ICL...
due to ffmpeg not supporting ICL, requires a bit of windows DLL magic to have working ffms/lavf support for ICL x86_64.
x264_r1391_icl_x64.zip (http://kemuri9.net/dev/x264/x264_r1391_icl_x64.zip)

Thank you for that. Any chance of an AMD optimized 64-bit build as well?

imk

15th January 2010, 06:08

Updated build of mine, as well...

r1391 built with Intel C++ Compiler v11.1.054

win_build_info.txt (http://imk.cx/pc/x264/win_build_info.txt)
x264-r1391M-imk-win.7z (http://imk.cx/pc/x264/x264-r1391M-imk-win.7z)

Edit:
I updated the build with LAVF/FFMS2 support, thanks to kemuri9.
I also added x264_hrd_pd_interlace.16_r1391.diff.

Selur

15th January 2010, 14:39

generic x86_32 build with hrd would be nice :)

Cu Selur

rack04

15th January 2010, 14:59

generic x86_32 build with hrd would be nice :)

Cu Selur

hrd needs an update to patch the latest git.

Selur

15th January 2010, 16:46

thanks for the info :) (fragged my x264/ffmpeg/mencoder-build machine a month or so ago and so I'm not up-to-date which patches work and don'T work atm. ;))

VFR maniac

16th January 2010, 00:45

x264_rev1391M_release2.rar (http://www.megaupload.com/?d=5P8FHFF0)

gcc: 4.4.2-dw2
pthreads: 2.9.0.0 (autostatic)
ffmpeg: rev21259 (--enable-pthreads)
libswscale: rev30341
ffms2: rev260
gpac: 2010-01-15 UTC
patch:
x264_hrd_pd_interlace.20_r1391.diff (http://pastebin.com/m76f87038) <--- I updated nal-hrd patch. However, there are no guarantee, and VFR + NAL HRD is not implemented in this version.
x264_tcfile_io_v4_r1391.diff (http://pastebin.com/m2b81af43)
x264_thread_pool_v2.5.diff (http://stashbox.org/740452/x264_thread_pool_v2.5.diff)
x264_fgo_r1369.diff (http://www.esnips.com/doc/9f6713d7-8733-4d9b-9ec1-dc952dca893f/x264_fgo_r1369)
x264_AQ_experiments_v4.diff (http://komisar.gin.by/test/x264_AQ_experiments_v4.diff)
$ ./configure --extra-cflags="-I/usr/local/include -march=core2 -fno-strict-aliasing" --extra-ldflags="-L/usr/local/lib"
Platform: X86
System: MINGW
asm: yes
avs input: yes
lavf input: yes
ffms input: yes
mp4 output: yes
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no

My compilation guide is here (http://forum.doom9.org/showpost.php?p=1363644&postcount=86).

julius666

16th January 2010, 17:13

x264_fgo_r1369.diff (http://www.esnips.com/doc/9f6713d7-8733-4d9b-9ec1-dc952dca893f/x264_fgo_r1369)

Whoa, is this the old Film Grain Optimization? :eek:
How does it work? Does it work with Psy-RD now?

rack04

16th January 2010, 21:32

x264_x64_r1391M (http://www.mediafire.com/?0qoygbyytad)

x264_x86_r1391M (http://www.mediafire.com/?mmym4zeuqkl)

Toolchain:
gcc-core-4.4.2 (x86_64.core2.Komisar) (http://komisar.gin.by/mingw/index.html)
Patch:
patch -p1 < /c/x264/nal_hrd_patch_v1.0.diff
x64 Build:
./configure --extra-cflags="-march=core2" --host="x86_64-pc-mingw32" --cross-prefix="x86_64-pc-mingw32-"
Platform: X86_64
System: MINGW
asm: yes
avs input: yes
lavf input: no
ffms input: no
mp4 output: yes
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no
x86 Build:
./configure --extra-cflags="-march=core2"
Platform: X86
System: MINGW
asm: yes
avs input: yes
lavf input: no
ffms input: no
mp4 output: yes
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no
fprofiled:
make fprofiled VIDS="/c/x264/parkrun.1280x720.yuv"

schweinsz

19th January 2010, 10:04

a problem about encoding with x264.
C:\Downloads>x264_lavfffms2.exe --fps 24 --level 41 --bitrate 6000 --pass 2 --quiet --keyint 100 -
-b-pyramid strict --ref 5 --direct auto --me umh --chroma-qp-offset 2 --partitions all --merange 12
8 --cqm jvt --profile high -o video1_6mbps.264 tractor.yuv 1920x1080 1>dflsa.log
x264 [error]: x264_encoder_open failed
the tractor.yuv is 1.9G big.

[ReX]

19th January 2010, 10:18

where as my previous x86_64 build was with gcc 4.4.2, I did one now with ICL...
due to ffmpeg not supporting ICL, requires a bit of windows DLL magic to have working ffms/lavf support for ICL x86_64.
x264_r1391_icl_x64.zip (http://kemuri9.net/dev/x264/x264_r1391_icl_x64.zip)

I need to set any environment variables to make your Intel patch work?
I always get: "No working C compiler found." :confused:

komisar

19th January 2010, 11:08

[ReX], i'm trying to adapt icl compiler in to pseudo mingw environment to work as gcc or as msvc. in short - you must set almost all env-variables from windows environment in to mingw (in mingw notation). e.g CPATH=${ICPP_COMPILER11}/ipp/em64t/include:${ICPP_COMPILER11}/mkl/include
sufficient to set: ICPP_COMPILER11;VSINSTALLDIR;WindowsSdkDir;INTEL_LICENSE_FILE;CPATH;FPATH;IPPROOT;LIBPATH;LIBRARY_PATH;INCLUDE;LIB;PATH variables.

imk

19th January 2010, 13:00

komisar

19th January 2010, 13:13

imk, sure... but main goal is trying to porting/replace gcc options to icl. maybe some projects will builds in this pseudo-gcc... also this allow to build/link staic dependence of x264 (maybe other too)

kemuri-_9

19th January 2010, 16:44

kemuri let me in on a trick that works with cygwin at least. Start the ICL IA-32/Intel64 command prompt first, and then launch cygwin from that prompt. It will automatically pass over all environment settings. This might work with mingw, too.

yes, i do this for msys, which is where i picked up the trick.

imk, sure... but main goal is trying to porting/replace gcc options to icl. maybe some projects will builds in this pseudo-gcc... also this allow to build/link staic dependence of x264 (maybe other too)

the patch allows for building x264 with icl/icc as both static and shared.
as x264 already heavily requires a posix (or at least posix-like, e.g. msys) environment for compilation,
i continued with that as allowing icl compilation outside of a posix environment would likely involve msvc solutions and those are a true waste of space imo.

komisar

19th January 2010, 17:38

kemuri-_9, maybe you have trick to build ffmpeg/ffms2 library with icl in the msys environment? :)

kemuri-_9

19th January 2010, 23:41

kemuri-_9, maybe you have trick to build ffmpeg/ffms2 library with icl in the msys environment? :)

lol.
ffmpeg does not support compilation with ICL, the last time i tried (just a couple weeks ago), it resulted in large numbers of compilation failures revolving around the inline assembly.
this is something I'd rather not go diving into when not even x264 officially supports ICL.

ffms2 officially supports compilation with MSVC, and thus ICL as well.
this is relying on the fact that msvc can read the mingw libraries (i.e. it can read mingw compiled ffmpeg).
this works fine for x86; but it is near, if not completely, impossible to have the same occur for x86_64 as there's a few places where large incompatibilities occur.

the above is the reason why i went the DLL route to have a ICL x86_64 build with ffms/lavf support.

burfadel

20th January 2010, 02:52

kemuri-_9

20th January 2010, 03:02

Anybody seen this patch? dated 14 Jan 2010:
http://akuvian.org/src/x264/x264_plane_copy.diff

It would be interesting to see what difference it would make speed wise, combined with aq4 patch of course :)

there are a number of patches that are under benchmark testing, i can't remember what the status on this specific one is though.