x264 development

[kemuri-_9]

seems it was what i was beginning to suspect:
the zone parser patch is apparently breaking on vista (but works fine on my XP)
guess i'll have to look at it some more.

Edit:
i think i found the problem with the original version.
here is a build with the updated version, give it a try please.
x264_mod2_debug.exe (http://kemuri9.net/dev/x264/x264_mod2_debug.exe)

if this doesn't work, i'll just have to crack at it some more, debugging ftw...

[Emp3r0r]

that fixed it! thanks

[kemuri-_9]

< pengvado> macro, or call x264_log_default

like this?
x264.muxers.respect.log.level.diff (http://kemuri9.net/dev/x264/patches/x264.muxers.respect.log.level.diff)
(same url, updated file)

[Japhsoncross]

lossless encoding is broken with rev999+. Rev987 is OK.

[Adub]

Nope, it isn't. They changed their lossless format to a better one, meaning much more compression, it's just that there is only one decoder that supports it so far. There will hopefully be a patch to ffmpeg/mplayer soon.

[nm]

lossless encoding is broken with rev999+. Rev987 is OK.
It's not broken but replaced with a different kind of lossless profile that is not yet supported by most decoders. Try latest CoreAVC.

commit b35a044b95c7eab2c91f55a3ac4100ca26a29d92 r994
Author: Jason Garrett-Glaser <darkshikari@gmail.com>
Date: Sat Sep 27 16:37:27 2008 -0700

Replace High 4:4:4 profile lossless with High 4:4:4 Predictive.
This improves lossless compression by about 4-25% depending on source.
The benefit is generally higher for intra-only compression.
Also add support for 8x8dct and i8x8 blocks in lossless mode; this improves compression very slightly.
In some rare cases 8x8dct can hurt compression in lossless mode, but its usually helpful, albeit marginally.
Note that 8x8dct is only available with CABAC as it is never useful with CAVLC.
High 4:4:4 Predictive replaced the previous profile in a 2007 revision to the H.264 standard.
The only known compliant decoder for this profile is the latest version of CoreAVC.
As I write this, JM does not actually correctly decode this profile.
Hopefully this lack of support will soon change with this commit, as x264 will be (to my knowledge) the first compliant encoder.

[Selur]

Note that 8x8dct is only available with CABAC as it is never useful with CAVLC.
is this generally true or just with lossless encoding ?

[Dark Shikari]

is this generally true or just with lossless encoding ?Only true with lossless.

[Selur]

Okay, thanks. :)

[Japhsoncross]

Sorry, it's my bad :P

[kemuri-_9]

Hey Dark Shikari, I'm getting some crashes on x264 when running SSEMisalign w/ threads >1
--threads 1 and --thread-input still work fine, but --threads >1 with SSEMisalign crashes pretty quickly after startup.


(gdb) file x264_debug
Reading symbols from C:\MinGW\projects\x264/x264_debug.exe...done.
(gdb) run --threads 2 -B 1000 ../x264_build1.y4m -o test.mp4
[New thread 4516.0x19b0]
[New thread 4516.0x3d1c]
[New thread 4516.0x1acc]
[New thread 4516.0x231c]
[New thread 4516.0x1024]

Program received signal SIGSEGV, Segmentation fault.

bt full
#0 0x0048cb44 in x264_hpel_filter_c_sse2_misalign ()
No symbol table info available.
#1 0x004738db in x264_hpel_filter_sse2_misalign (dsth=0x2971a40 "",
dstv=0x29cf240 'ë' <repeats 200 times>..., dstc=0x2a2ca40 "",
src=0x2914248 'ë' <repeats 200 times>..., stride=704, width=656,
height=16) at common/x86/mc-c.c:232
buf = <incomplete type>
realign = 44223704
src = (uint8_t *) 0x2914240 'ë' <repeats 200 times>...
width = 664
height = 15
#2 0x0044d52e in x264_frame_filter (h=<incomplete type>,
frame=<incomplete type>, mb_y=0, b_end=0) at common/mc.c:380
b_interlaced = 0
stride = 704
width = 640
start = -8
height = 8
offs = -5640
x = -5640
y = 0
mb_y = -664
...


any ideas? Oh and let me know if you need some more info

[Dark Shikari]

... don't tell me you're serious.

The misalign mask has to be set once per thread? :rolleyes:

Try this in encoder.c (find the function, add the two lines):

static int x264_slices_write( x264_t *h )
{
int i_frame_size;
if(h->param.cpu&X264_CPU_SSE_MISALIGN)
x264_cpu_mask_misalign_sse();

[kemuri-_9]

that did the trick, working fine now. :)

[Mad_Martha]

Hi , have had a continuing issue with later versions of x264 (since about v750 , ish , onwards in fact)

Did a search here to see if it was already mentioned but came up blank , so I ask here...

Apologies if this has been covered elsewhere.

I'm using the AutoMKV front end , while updating (i.e. replacing) the x264 cli encoder , currently using v1062.

For some time now I've had hellishly slow encoding.
That said , yes , I'm using VERY slow settings BUT from what I'm seeing it is looking like x264 is NOT firing up all my CPU's functions (like SSE3 for example) - can't help but feel something isn't right...

I include below 3 things - a pic of the encoder working , a pic from CPU-Z describing my CPU and the input script being generated by AutoMKV.

http://homepage.ntlworld.com/mad.martha/x264encoding.png


http://homepage.ntlworld.com/mad.martha/cpu.png

Command Line 1' Pass X264: F:\BACKUPS\AutoMKV\AutoMKV0981\exe\encoder\x264.exe --pass 1 --bitrate 1274 --stats "D:\X264 Tryout Area\0981 - Standard - v1062\temp\.stats" --progress --keyint 250 --bframes 8 --qpmin 10 --qpmax 51 --no-psnr --no-fast-pskip --mixed-refs --trellis 2 --ref 12 --filter -2,-1 --subme 8 --direct auto --vbv-bufsize 14000 --vbv-maxrate 25000 --me tesa --pre-scenecut --no-ssim --level 4.1 --merange 32 --weightb --b-adapt 2 --b-pyramid --partitions p8x8,b8x8,i4x4,i8x8,p4x4 --8x8dct --threads auto --thread-input --aq-mode 1 --aq-strength 1 --psy-rd 1.0:1.0 --sar 15787623:11119680 --output NUL "D:\X264 Tryout Area\0981 - Standard - v1062\temp\movie.avs"


Any ideas anyone ?

Thanks ,

[MM]

[Dark Shikari]

Hi , have had a continuing issue with later versions of x264 (since about v750 , ish , onwards in fact)

Did a search here to see if it was already mentioned but came up blank , so I ask here...

Apologies if this has been covered elsewhere.

I'm using the AutoMKV front end , while updating (i.e. replacing) the x264 cli encoder , currently using v1062.

For some time now I've had hellishly slow encoding.
That said , yes , I'm using VERY slow settings BUT from what I'm seeing it is looking like x264 is NOT firing up all my CPU's functions (like SSE3 for example) - can't help but feel something isn't right...SSE3 provides no useful functionality for x264 except on the Pentium 4D. We're not going to be dishonest and display capabilities that aren't being used.

[Mad_Martha]

Thanks for the quick reply - interesting that SSE3 don't help much..

OK , follow on question : Given the rather "intense" settings I'm using - any suggestions on speeding up processing , while maintaining quality ?
(while also remembering I can't afford a qx9775)
I encode at very low bitrates and would rather use as many features (that aid in compression) as possible..

[MM]

[LoRd_MuldeR]

First of all "--aq-mode" never existed in official builds and "--aq-strength 1" is useless, as it's the default value. Also "--bframes 8" is overkill, more than "--bframes 3" doesn't help much and is only significant slower with "--b-adapt 2". I'd also say that "--ref 12" is overkill, except for anime maybe. The same applies to "--merage 32", as more than "--merage 16" doesn't help much. Last but not least you should leave "--partitions" out and use the defaults instead. The default is "all" partitions, except for 4x4 (as 4x4 isn't helpful, except for very low resolutions maybe). Oh, and Psy-Trellis 1.0 may be a bit too strong...

[Dark Shikari]

First of all "--aq-mode" never existed in official buildswhat.

[LoRd_MuldeR]

First of all "--aq-mode" never existed in official buildswhat.

Whoops, seems I thought of "--aq-sensitivity" :p

But "--aq-mode 1" is useless anyways, as it's just the default value. And I forgot that "--me tesa" is overkill too, unless you really don't care about speed ;)

[Mad_Martha]

And I forgot that "--me tesa" is overkill too, unless you really don't care about speed ;)

Yeah , my main concern is quality/bit rate rather than speed - I push x264's compression to it's limits (and sometimes a bit beyond the knife edge).

Unfortunately (for me) there are quite a few people out there that *literally* go over my encodes frame by frame looking for what they percieve as "errors". :(

...and oh boy do they moan about it...

The "redundant" settings were in because I am currently playing with things like AQ and the PSY-trellis settings.
(they were folded back for the example).

Thanks for the info tho - gives me summat to play with over the next few days.

[MM]

[D734]

Yeah , my main concern is quality/bit rate rather than speed - I push x264's compression to it's limits (and sometimes a bit beyond the knife edge).

Unfortunately (for me) there are quite a few people out there that *literally* go over my encodes frame by frame looking for what they percieve as "errors". :(

...and oh boy do they moan about it...

The "redundant" settings were in because I am currently playing with things like AQ and the PSY-trellis settings.
(they were folded back for the example).

Thanks for the info tho - gives me summat to play with over the next few days.

[MM]
indeed like the quality over speed mentality :)
one more thing to tweak would be subme 9 instead of 8 (barely slows things down)
something a little more specific about the ref frame is that i've heard any more then 5-6 really wont help unless anime/cartoon content.
some people trying to do max quality use 3pass, but according to the x264 developers 3pass does not increase the quality.

i've personally encoded and compared 720x576 encodes at 1mbit and the command line below worked a lot better then the old automkv "insane_quality" you've used before.

"c:\x264.exe" --pass 1 --bitrate #### --stats "c:\x264.stats" --level 4.1 --ref 5 --bframes 3 --subme 9 --me tesa --trellis 2 --b-adapt 2 --mixed-refs --no-fast-pskip --b-pyramid --weightb --direct auto --filter -2,-1 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --threads auto --thread-input --sar 1:1 --cqmfile "C:\Prestige CQM.cfg" --progress --no-dct-decimate --no-psnr --no-ssim --output NUL "x:\some.avs"

"c:\x264.exe" --pass 2 --bitrate #### --stats "c:\x264.stats" --level 4.1 --ref 5 --bframes 3 --subme 9 --me tesa --trellis 2 --b-adapt 2 --mixed-refs --no-fast-pskip --b-pyramid --weightb --direct auto --filter -2,-1 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --threads auto --thread-input --sar 1:1 --cqmfile "C:\Prestige CQM.cfg" --progress --no-dct-decimate --no-psnr --no-ssim --output x:\file.mkv "x:\some.avs"

not sure if it can be improved or not as far as quality (cqm matrice is obviously up to the encoder to use or not, i've just heard good stuff about prestige cqm and seen non scene hd encoders use it)
overall the speed was around 33% longer then automkv 2_pass_insane_quality profile (with equal 1 and 2 pass settings) because the automkv profile used ref 16 (vs 5or6) which doubled its encoding time.

[Snowknight26]

Non flat cqm and no --psy-rd. /facepalm

[Dark Shikari]

Non flat cqm and no --psy-rd. /facepalmBecause it isn't like psy-rd is on by default, is it? :sly:

[Snowknight26]

My mistake, switch my 'and' with 'or.' ;)

[check]

Unfortunately (for me) there are quite a few people out there that *literally* go over my encodes frame by frame looking for what they percieve as "errors". :(
Hint: --crf :)

[tal.aloni]

Hey Guys,
Amazing work on x264, Dark Shikari especially,
There is a feature I'm really missing, and I'm willing to get my hands dirty (been involved with ffdshow too)

good x264 encode is taking about a week on my Phenom 8450, and since all my computer fans are auto-adjusted to temperature, I can't have anybody over for a movie during the encoding week... if I could just hit 'spacebar' and pause the encoding for two hours (memory usage can stay as is), it would be terrific.

I would appreciate if you could guide me through how this can be implemented the best way. (if at all)

Thanks!
Tal Aloni

Edit:
I'm guessing it should be done right before updating the status line (x264.c),
and if 'spacebar' has been pressed, just print "[Paused] [58.5%] 128170/218980 frames. (press spacebar to continue)", and enter a sleep loop, untill 'spacebar' has been pressed again, is this ok? (fps statistics should be resetted somehow)

[nurbs]

I think process explorer can do that.

[J_Darnley]

If you are using a frontend then you will need to use nurbs suggestion, but if you are using the x264 binary from a command prompt then just use the pause key (and spacebar to resume).

[tal.aloni]

I think process explorer can do that.

Great, I wasn't aware of that, Thanks!

if you are using the x264 binary from a command prompt then just use the pause key (and spacebar to resume).
I do, I wasn't aware of that either! Thanks!

[kemuri-_9]

I'm guessing it should be done right before updating the status line (x264.c),
and if 'spacebar' has been pressed, just print "[Paused] [58.5%] 128170/218980 frames. (press spacebar to continue)", and enter a sleep loop, untill 'spacebar' has been pressed again, is this ok? (fps statistics should be resetted somehow)

there's not much particular interest for this, especially since the main devs don't use windows!

i believe there was a patch floating around to somewhat handle this...

but overall, if you pause it, your fps encoding rate will become junk,
as x264 currently doesn't keep track of when it's "paused" to subtract from the total time the process has been alive.

[Dark Shikari]

Hey Guys,
Amazing work on x264, Dark Shikari especially,
There is a feature I'm really missing, and I'm willing to get my hands dirty (been involved with ffdshow too)

good x264 encode is taking about a week on my Phenom 8450, and since all my computer fans are auto-adjusted to temperature, I can't have anybody over for a movie during the encoding week... if I could just hit 'spacebar' and pause the encoding for two hours (memory usage can stay as is), it would be terrific.It's the operating system's job to pause a process, not x264's.

It's trivial to pause both on Windows and Linux (Ctrl-S, iirc).the main devs don't use windows!¯\(º o)/¯

[ajp_anton]

but overall, if you pause it, your fps encoding rate will become junk,
as x264 currently doesn't keep track of when it's "paused" to subtract from the total time the process has been alive.I often have this "problem" when I use the CPU for other things with the encode running in the background.
What about changing "--progress" into "--progress n" where "n" is the number of frames used to average the fps/eta, defaulting to 0 which means everything encoded so far?

[J_Darnley]

If I recall correctly there are, or were, a couple of patches that do this last time people started banging the pause/suspend/resume drum. Even I wrote one, so perhaps I should clean it up and post it.

[professor_desty_nova]

I just noticed that H.264 (2007) Corrigendum 1 (01/09) was released last week in http://www.itu.int/rec/T-REC-H.264. Does it have any implication on x264?

[akupenguin]

Does it have any implication on x264?
Dunno. It was published as a whole new document, and PDFs aren't very convenient for taking diffs. I don't plan to reread it just to see if anything interesting was added, so if there are any new features people want implemented, they had better speak up.

[Dark Shikari]

It has SVC in it, which officially makes the spec longer than MPEG-4 Part 2.

[Raptus]

[...]I don't do releases or "big iterations". I just do features.[...]
[...]if there are any new features people want implemented, they had better speak up.
Since there are no plans or roadmaps for x264, by which criteria do you devs choose what features to implement? What people ask for the most and/or what seems reasonable and/or doable?

[Dark Shikari]

Since there are no plans or roadmaps for x264, by which criteria do you devs choose what features to implement?I have an idea. It sounds like a good idea. So I try it. My development is idea-bottlenecked, generally, not by coding time or anything.

Also, money is a good motivator.

[Chengbin]

I have an idea. It sounds like a good idea. So I try it. My development is idea-bottlenecked, generally, not by coding time or anything.

Also, money is a good motivator.

There is only so much you can improve a 1MB program. After almost 1100 revisions the room for improvement certainly is running dry.

I heard x264 focus on main and high profile. Some features in simple profile has not yet been added. Maybe you can waste some time doing that.

I would donate, but my parents won't let me for obvious reasons. So the least I can say is thank you, Dark Shikari, and other people who contributed to x264.

[akupenguin]

There is only so much you can improve a 1MB program
You say that as if the size is a constant, unaffected by our improvements.

[Dark Shikari]

You say that as if the size is a constant, unaffected by our improvements.Indeed, we make it consistently smaller over time ;)

[Chengbin]

What I mean is that 1MB is a small program. Not a gazillion codes. It has gone through over 1000 revisions. I think Windows XP only had about 2000 fixes. (I know I'm comparing apples to oranges)

Or maybe I'm naive and a 1MB program can be extremely complicated as well.

[Sagittaire]

Or maybe I'm naive and a 1MB program can be extremely complicated as well.

Well you are naive here because size of program and complexity of developpement are not same thing. There are many possible improvement for x264:
- HRD SEI support
- Complete H264 interlacing support
- FGM support (really complexe)

And not exclusive part of H264 with HVS improvement: In this domain there are virtualy infinite possibility with AQ, Custom Matrix, temporal stability ... etc etc

[cogman]

What I mean is that 1MB is a small program. Not a gazillion codes. It has gone through over 1000 revisions. I think Windows XP only had about 2000 fixes. (I know I'm comparing apples to oranges)

Or maybe I'm naive and a 1MB program can be extremely complicated as well.

Sometimes we just don't know how big 1MB really is. I know our ideas are dwarfed by the GB we deal with today, however take note.

1 MByte = 1024 kByte ~= 1024000 bytes ~= 8192000 bits. There are two possibilities for each of those bits, 1, or 0. so that is 2^81920000 possible combinations. Now, I realize that not all of those combinations actually mean something, however even if 1/1,000,000,000th of those combination are valid H.264 encoders, we can see that we have a lot of possibilities available to us.

Given that x264 is starting to use asm on an increasingly large scale, I have little doubt that we will keep seeing quality / speed improvements for a good while (until some other standard MPEG 8 perhaps comes and knocks it off its pedistal, even then, my bet is that a fair amount of x264 code will still be valid for a new MPEG standard).

Lots of wiggle room when you think about it. Not to mention the fact that it isn't beyond reason for x264 to hit 1.5 mb *gasp* dare I speak it!

[Guest]

I would donate, but my parents won't let me for obvious reasons. It's not obvious to me. What is the reason?

[Chengbin]

It's not obvious to me. What is the reason?

Mom, can I use your credit card to donate to this guy who wrote this really nice software for free?

I have to have insanely rich parents to do that.

[Shinigami-Sama]

the room for improvement certainly is running dry.

theres still C-code in it theres still room for improvment
not to mention there always ALWAYS microfixes, 1cycle here, 3 cycles there, couple dozen of those on things called a few times a second add up ;)



edit
Mom, can I use your credit card to donate to this guy who wrote this really nice software for free?

I have to have insanely rich parents to do that.

lern 2 paypal?

[Guest]

I have to have insanely rich parents to do that. Your parents have to be "insanely rich" to let you make a ten buck donation?!

[Dark Shikari]

You know, when I was a kid, and I wanted to buy something or donate money or whatever, I would go get the cash out of my own pocket and do it myself.

But apparently that's out of fashion these days.

[Chengbin]

Your parents have to be "insanely rich" to let you make a ten buck donation?!

Wouldn't you be embarassing yourself if you donate $10?

You know, when I was a kid, and I wanted to buy something or donate money or whatever, I would go get the cash out of my own pocket and do it myself.

But apparently that's out of fashion these days.

I would, but my parents hold all my money, and I'm not exactly free to use that money on anything I like. Especially when my dad lost a job, not exactly in a position to ask. Maybe if Dark Shikari is still around in 5 years doing this, I'll donate, since I'm in university by then, and I'll be free to use money on whatever I want. But then I probably not have time to waste on encoding and learning how to use x265.