Current Patches, Where to get them, How they affect speed/output

[Dark Shikari]

nah.. doesnt corrupt.. purposefully omits... i wrote a --user-data option that will add it back in... will have a diff out tonight.. user data sei message is supposed to be safe as long as buffering sei message is first in the AU. i'll leave it an defaulted off until i get a few all clear reports.I will not endorse any patch that allows disabling of the user data.

[Trahald]

:(... ok.. the next patch will add it back unconditionally.. i would just say to anyone that uses it for the first time make sure your application (standalone, whatever) doesnt have an issue with it.. it shouldnt.. but you never know.

[bob0r]

yeah, i know that too. but most of the OSS is based on black box RE or RE...
so i cant see that one as a valid point.

1: I saw the patch was still kinda in developement, to dot some i's. Is it done for now?
2: I had horrible isp problems, as you can read on x264.nl
I had 5kb/s upload with timeouts every 200+kb :)
These problems all seem solved.
So if the patch is ready for now, i guess ill update.

But my first thoughts were with you, telling you you should only mind your health and not megui updates before that.
But as now, you quit IRC already :)

[Trahald]

hrd patch no longer omits user data when nal-hrd is used. writes user data in a nal w/buffering period msg. also small change to bs_align_10 code sugstd by pv

[bob0r]

hrd patch no longer omits user data when nal-hrd is used. writes user data in a nal w/buffering period msg. also small change to bs_align_10 code sugstd by pv

Cool, any more plans for the patch any time soon? I'ill compile a new build anyway tomorrow to test!

Edit:
Any plans to get rid of the compiling errors as shown in my build posts?

encoder/set.c: In function `x264_sps_init':
encoder/set.c:217: warning: unused variable `cpbBrVclFactor'
etc..

[skystrife]

x264.914.modified.02.exe (http://www.mediafire.com/?myobctkxd9x) - Alternate Download (http://skystrife.com/x264/x264.914.modified.02.exe)
(Source) (http://www.mediafire.com/?ktihlrmugmd) - Alternate Download (http://skystrife.com/x264/x264.914.modified.02.source.7z)

Patches used:

x264_psyRDO_0.5.diff
x264_hrd_pulldown.08_interlace.diff
x264.progress.indication.01.diff

All patches are included with the source.

gcc 3.4.5 fprofiled build.

[skystrife]

x264.915.modified.exe (http://www.mediafire.com/?1glgjl3czcw) - Alternate Download (http://skystrife.com/x264/x264.915.modified.exe)
(Source) (http://www.mediafire.com/?xhg9hctjn1h) - Alternate Download (http://skystrife.com/x264/x264.915.modified.source.7z)

Patches used:

x264_psyRDO_0.5.diff
x264_hrd_pulldown.09_interlace.diff (http://skystrife.com/x264/x264_hrd_pulldown.09_interlace.diff) - Slight change to get it to patch with r915, /encoder/set.h changes with the validate levels line.
x264.progress.indication.01.diff

All patches are included with the source.

gcc 3.4.5 fprofiled build.

[Sharktooth]

@skystrife: while waiting for an "official" x264.nl ... [removed - no longer necessary]

[Sharktooth]

ok, do it :) ill delete my previous post then

[bob0r]

x264.915.modified.01.exe (http://files.x264.nl/x264.915.modified.01.exe)
libx264-60.915.modified.01.dll (http://files.x264.nl/libx264-60.915.modified.01.dll)

x264-psyrd-0.5.diff
x264_hrd_pulldown.09_interlace.diff
x264.progress.indication.01.diff

[Sharktooth]

done.

[jefrey]

can anyone explain why
boboirs build is ~1-3fps slower than skystrifes build?
used trellis and psy:
boboirs build 915 : 6-7fps
skystrife915 : 7-8 fps

[skystrife]

I would probably say that's within the margin of error, but it may have something to do with the build environment and cflags.

[TheRyuu]

I would probably say that's within the margin of error, but it may have something to do with the build environment and cflags.

AFAIK the only cflag of any particular use is march=pentium2 to enable mmx/cmov support. Anything else won't change anything and even that is trivial (I guess). (aside from the defaults I mean)

Maybe the fprofile is different as well although that shouldn't matter either.

[Sharktooth]

how did you compile pthreads and gpac? what cflags you used for them?
IIRC if you optimize those 2 with MMX too you could get a nice speed boost in certain situations.

[skystrife]

x264.918.modified.exe (http://www.mediafire.com/?l9swnudyglc) - Alternate Download (http://skystrife.com/x264/x264.918.modified.exe)
(Source) (http://www.mediafire.com/?yxglwgsjgwb) - Alternate Download (http://skystrife.com/x264/x264.918.modified.source.7z)

Patches used:

x264_psyRDO_0.5.diff
x264_hrd_pulldown.09_interlace.diff
x264.progress.indication.01.diff

All patches are included with the source.

gcc 3.4.5 fprofiled build.



I don't recall having any extra cflags for pthreads or gpac, so it might be the fprofile like dragon suggested? Who knows.

[akupenguin]

how did you compile pthreads and gpac? IIRC if you optimize those 2 with MMX too you could get a nice speed boost in certain situations.
... No.
If you spend any measurable amount of time in a muxer, you're doing something wrong.
And while there are SSE3 optimizations possible for threading, they're privileged instructions. i.e. the OS can use them but no user-level library can.

[Sharktooth]

optimizing gpac is not necessary, however, i remember (i could be also easily wrong!) when i optimized pthreads there was a around 1-2% increase in x264 encoding speed.
that was so long ago though and i may be wrong as well...
the speed difference may also come from the gcc version used and profiling.
ill upload the skystrife build into megui autoupdate to see if MeGUI users notice any speed difference.

[skystrife]

x264.920.modified.exe (http://www.mediafire.com/?dwxxwcvhe2d) - Alternate Download (http://skystrife.com/x264/x264.920.modified.exe)
(Source) (http://www.mediafire.com/?iwgngyggtit) - Alternate Download (http://skystrife.com/x264/x264.920.modified.source.7z)

Patches used:

x264_psyRDO_0.5.diff
x264_hrd_pulldown.09_interlace.diff
x264.progress.indication.01.diff

All patches are included with the source.

gcc 3.4.5 fprofiled build.

[Sharktooth]

request: modified build with usual patches + new bframes decision patch (if available) for speed testing...

[Dark Shikari]

request: modified build with usual patches + new bframes decision patch (if available) for speed testing...Here's the patch (http://pastebin.com/d495ce2e). Please do not use it any ordinary patched builds--this is just for your testing.

[Sharktooth]

as i said... for testing so i can see the speed difference and test the future megui changes needed to support it.
i will NOT include the build in megui auto-update until it's committed to git ... if this is what you mean...

[bob0r]

FOR TESTING PURPOSES ONLY

x264.920.new.bframes.decision.01.exe (http://files.x264.nl/x264.920.new.bframes.decision.01.exe)

[Sharktooth]

thanx

[bob0r]

I would probably say that's within the margin of error, but it may have something to do with the build environment and cflags.

Can you share your fprofile sample + script used to make your .exe?

Also post how you compile x264.exe
(i use configure and make, or make fprofiled, no extra flags)

Also please test my files: x264 920 gcc 3.4.5 versus 3.4.6, please test! (http://forum.doom9.org/showthread.php?p=1163184)

Edit:
Also what yasm version do you use? Even though pengvado explained to me, a different yasm version cannot make a speed difference.
It can solve bugs, its worth testing. i use 0.6.2.

[Sharktooth]

what's your configure commandline?
try with this:

--enable-avis-input --enable-mp4-output --enable-pthread -march=pentium2 -mmmx -O3 -finline-functions -funroll-loops -ffast-math -fomit-frame-pointer

make fprofiled VIDS="../video.yuv"

some options may be reduntant and can be removed (like -mmmx...)
gcc 3.4 is faster than 4.x. at least on my pc. 3.4.5 and 3.4.6 should produce almost the same speeds. i would use 3.4.6 though.

[kemuri-_9]

practically all of those options are redundant, the configure script automatically checks for gpac and pthread existence when it runs:
pthreads - checks for libpthread(GC2).a in lib dir(s) and pthread.h in include dir(s)
gpac - checks for libgpac_static.a in lib dir(s) and gpac/isomedia.h in include dir(s)
avis input - automaticly enabled when building on MINGW/CYGWIN system.

configure automatically adds -O4 and -ffast-math flags;
-funroll-loops is assumed with fprofiling
-finline-functions is assumed at -O3
-fomit-frame-pointer is assumed at -O and higher
-mmmx should be implied with -march=pentium2 as gcc creates code optimized for the pentium2 which has mmx capabilities.

so the only one out of all of those that actually does something is -march=pentium2
oh and don't forget if you give it a yuv to profile with, it needs the resolution in the name, but since you're more than likely building on mingw for the windows version, you can give it an avisynth script granted you have avisynth installed.

[Dark Shikari]

-fomit-frame-pointer is assumed at -O and higherOnly on 64-bit machines, I think?

But yes, -march=i686 -O3 -fomit-frame-pointer should be fine... and the latter two are already on by default!

[kemuri-_9]

whatever
-O also turns on -fomit-frame-pointer on machines where doing so does not interfere with debugging.
implies on i686-pc-mingw... but yes, it's already in the configure regardless...

oh yes, according to the gcc manual again, march=i686 does not include mmx instructions, so should use march=pentium2 which does have them.

[Adub]

what exactly is the advantage of fprofiling? Is it something that I can do on my x64 gentoo build?

[LoRd_MuldeR]

what exactly is the advantage of fprofiling? Is it something that I can do on my x64 gentoo build?

The code is analyzed while actually running. This allows the compiler to do optimization that cannot be done by just "looking" at the sources.
In software engineering, performance analysis, more commonly profiling, is the investigation of a program's behavior using information gathered as the program runs (i.e. it is a form of dynamic program analysis, as opposed to static code analysis). The usual goal of performance analysis is to determine which parts of a program to optimize for speed or memory usage.

Should be possible to run "make fprofiled" with every build that is made using GCC ...

[kemuri-_9]

you can fprofile on any system by giving it a/several video file(s) the x264 compiled on that system can recognize in the
make fprofiled VIDS=infile1 infile2 ..."
on linux that would be more than likely a .y4m which can created from wine'ing avs2yuv which was created by akupengiun (iirc) for linux based encoding...
that is if you don't already have something that can write .y4m files.

[LoRd_MuldeR]

Or you can grab one of the samples from here:
http://meru.cecs.missouri.edu/free_download/videos/

Simply rename "foreman_cif" to "foreman_cif.352x288.yuv" and that's it ;)

make fprofiled VIDS="/c/samples/foreman_cif.352x288.yuv"

[Selur]

just wondering would it be a good idea to use also high resolution files for fprofile?

[skystrife]

Can you share your fprofile sample + script used to make your .exe?

Also post how you compile x264.exe
(i use configure and make, or make fprofiled, no extra flags)

Also please test my files: x264 920 gcc 3.4.5 versus 3.4.6, please test! (http://forum.doom9.org/showthread.php?p=1163184)

Edit:
Also what yasm version do you use? Even though pengvado explained to me, a different yasm version cannot make a speed difference.
It can solve bugs, its worth testing. i use 0.6.2.

I use a 100 frame 1080p sample from Elephants Dream (image sequence, yay avisynth input), 200 frames 640x480 from Dark Shikari's Lossless Touhou sample (lossless h264, an encoder torture clip; it can be found on here, I can do a search if you want), and a 200 frame 1080p image sequence of a huge crowd running (and now that I think about it, I don't even remember where I got that... I could probably dig around and find it though).

./configure --extra-cflags="-march=pentium2"
make fprofiled VIDS='elephants_dream.avs LosslessTouhou.avs crowd.avs'

YASM version is 0.7.1.

I can upload the fprofile samples somewhere if it's needed, it's just a lot of data.

[Dark Shikari]

I use a 100 frame 1080p sample from Elephants Dream (image sequence, yay avisynth input), 200 frames 640x480 from Dark Shikari's Lossless Touhou sample (lossless h264, an encoder torture clip; it can be found on here, I can do a search if you want), and a 200 frame 1080p image sequence of a huge crowd running (and now that I think about it, I don't even remember where I got that... I could probably dig around and find it though).

./configure --extra-cflags="-march=pentium2"
make fprofiled VIDS='elephants_dream.avs LosslessTouhou.avs crowd.avs'

YASM version is 0.7.1.

I can upload the fprofile samples somewhere if it's needed, it's just a lot of data.The Crowdrun sequence is actually a 2160p sample I posted a while back. LosslessTouhou.mkv can be found here (http://mirror05.x264.nl/Dark/LosslessTouhou.mkv).

[akupenguin]

-O also turns on -fomit-frame-pointer on machines where doing so does not interfere with debugging.
-fomit-frame-pointer does interfere with debugging on x86.
It is theoretically possible for a compiler to record its planned stack offset at every instruction that modifies the stack, and then a debugger can find the stack frames without having a reserved frame pointer. But I don't know of any object format that actually allows that, and so frame pointer is useful for debugging on all architectures I know of.

[kemuri-_9]

The Crowdrun sequence is actually a 2160p sample I posted a while back..

did that originate from
ftp://vqeg.its.bldrdoc.gov/HDTV/SVT_MultiFormat/2160p50_CgrLevels_Master_SVTdec05_/1_CrowdRun_2160p50_CgrLevels_MASTER_SVTdec05_/
?

[Dark Shikari]

did that originate from
ftp://vqeg.its.bldrdoc.gov/HDTV/SVT_MultiFormat/2160p50_CgrLevels_Master_SVTdec05_/1_CrowdRun_2160p50_CgrLevels_MASTER_SVTdec05_/
?Yup, it did.

[LoRd_MuldeR]

ehm, what to do with those .sgi files? :o

[Dark Shikari]

ehm, what to do with those .sgi files? :oimagemagick -> convert to png -> imagesource

[akupenguin]

imagemagick -> convert to yuv -> cat

[LoRd_MuldeR]

imagemagick -> convert to png -> imagesource
imagemagick -> convert to yuv -> cat

:D

I'll try Imagemagick then :thanks:

[Dethis]

imagemagick -> convert to yuv -> cat

Why not sgi2yuv ??

The companion pdf
ftp://vqeg.its.bldrdoc.gov/HDTV/SVT_MultiFormat/SVT_MultiFormat_v10.pdf

suggests just this
http://www.ldv.ei.tum.de/Members/tobias/videotools/sgi2yuv.zip/view?set_language=en

[akupenguin]

Because sgi2yuv is a specialized program. If I had a separate program for each pair of formats I wanted to convert between, I'd need O(N^2) programs total.

[Dethis]

Sorry for not been clear.

Not being familiar whith any of them, i wonder if there are any quality differencies between sgi2yuv/ImageMajic sgi to yuv convertions

[Lux Delux]

Hi, first thanks for the patches :)

Few days ago I switched to Vista x64 and noticed a severe FPS drop in 2nd pass (during first pass it's 1fps slower than in XP). Actually it starts rather normal going from 1.5 fps to somewhere around 2.5 and at about 0.6% of the pass it drops to 0.3 fps and stays around that :eek:

Am encoding an .avs to mkv with Udegrain (with fft3dgpu), latest Megui with last skystrife patched build http://forum.doom9.org/showthread.php?p=1163061#post1163061

Am just wondering if someone has an idea for this strange behaviour and if it's connected to the build or perhaps fft3dgpu not operating properly under x64 (though then I missed the thread completely.)

EDIT: along with the FPS the CPU usage drops basically to idle or none. Thus I assume there's some bottleneck (am in a hurry now gonna try without avs scripts later to see if that's the prob)

Searched the forum but couldn't find anything. Cheers :)

[lexor]

Do you switch user accounts while it's working? Maybe you have it so it goes to user selection screen once it resumes from screen saver? If you do that, it will mess with fft3dgpu. Officially that has been fixed since 0.6. Unofficially if you still get it (like me) you are screwed. Just make sure you don't switch users (not necessarily load other users, just going to the user selection screen is enough to break it).

[Adub]

Thanks very much for the info. I'll wait till i get back to Uni, and then download those samples and get fprofiling!!

[Ranguvar]

I had the same problem as Lux Deluxe... Thanks, lexor!