x264 development

[LoRd_MuldeR]

I see :)

[plonk420]

how hard would it be to make nehalem pass 1 use 4 cores (if hyperthreading is on)? ... or whatever is most optimal (like maybe "cores" 0, 2, 4, and 6)

[akupenguin]

taskset -c 0,2,4,6 x264 ...
(and x264's cpu detection takes that into account, so it'll default to 6 threads.)

[plonk420]

i'm assuming if you use 6 threads on pass 1, you *should* use 6 threads on future passes?

[Dark Shikari]

i'm assuming if you use 6 threads on pass 1, you *should* use 6 threads on future passes?Not particularly.

[prOnorama]

Hardly an "issue", but there's a small typo in the x264 --longhelp/--fullhelp commands:

--b-pyramid <string> Keep some B-frames as references [none]
- none: Disabled
- strict: Strictly heirarchical pyramid
- normal: Non-strict (not Blu-ray compatible)

heirarchical = hierarchical

[Dark Shikari]

Hardly an "issue", but there's a small typo in the x264 --longhelp/--fullhelp commands:



heirarchical = hierarchicalFixed locally.

[Groucho2004]

how hard would it be to make nehalem pass 1 use 4 cores (if hyperthreading is on)? ... or whatever is most optimal (like maybe "cores" 0, 2, 4, and 6)

x264 already chooses the most optimal number of threads because it is the most bestest h.264 encoder.
:):)

[Revgen]

So what's the plan on open GOP's? I saw a patch awhile ago, but haven't seen anything since.

Did the author go MIA?

[kemuri-_9]

So what's the plan on open GOP's? I saw a patch awhile ago, but haven't seen anything since.

Did the author go MIA?

no, from what i understand Trahald (wewk on IRC) is still working on it, but his MMCO patch was a higher priority so focus went to get it committed first.

the open gop patch also is going to need an overhaul with the latest commits afaik.

[easyfab]

Can I have a idea or better a list or link of the functions that are not implemented yet in x264 ( such as weightp ) and that are in H.264 specifications or compatible with H.264 speficications.

[seemees]

Dark Shikari
In a source code (for example analyse.c) we count many warnings and messages on compile with mingw environment.
Why do not clear code from this stuck? Example: array[t]+!!i; why using some weird constructions? What does it mean !!i?
And what can happen if we replace

const float x264_log2_lut[128] = {
0.00000, 0.01123, 0.02237, 0.03342, 0.04439, 0.05528, 0.06609, 0.07682,
0.08746, 0.09803, 0.10852, 0.11894, 0.12928, 0.13955, 0.14975, 0.15987,
.....
/* Avoid an int/float conversion. */
const float x264_log2_lz_lut[32] = {
31,30,29,28,27,26,25,24,23,22,21,20,19,18,17,16,15,14,13,12,11,10,9,8,7,6,5,4,3,2,1,0
};

with

const float x264_log2_lut[128] = {
0.00000f, 0.01123f, 0.02237f, 0.03342f, 0.04439f, 0.05528f, 0.06609f, 0.07682f,
0.08746f, 0.09803f, 0.10852f, 0.11894f, 0.12928f, 0.13955f, 0.14975f, 0.15987f,
.....
/* Avoid an int/float conversion. */
const float x264_log2_lz_lut[32] = {
31f,30f,29f,28f,27f,26f,25f,24f,23f,22f,21f,20f,19f,18f,17f,16f,15f,14f,13f,12f,11f,10f,9f,8f,7f,6f,5f,4f,3f,2f,1f,0f
};
I think that 0.3 is a double, and 0.3f is a float. Isn't it?
Please help me to understand politics of coding h264.
Why *.c but not *.cpp? :-)

Here is the example of warning messages:

gcc -O4 -ffast-math -Wall -I. -march=i686 -mfpmath=sse -msse -DHAVE_MMX -DARCH_X86 -DSYS_MINGW -DPTW32_STATIC_LIB -DHAVE_PTHREAD -s -fomit-frame-pointer -c -o encoder/macroblock.o encoder/macroblock.c
In file included from ./common/common.h:698,
from encoder/macroblock.c:25:
./common/macroblock.h: In function 'x264_mb_transform_8x8_allowed':
./common/macroblock.h:465: warning: dereferencing type-punned pointer will break strict-aliasing rules
encoder/macroblock.c: In function 'x264_macroblock_encode':
encoder/macroblock.c:893: warning: dereferencing type-punned pointer will break strict-aliasing rules
encoder/macroblock.c: In function 'x264_macroblock_probe_skip':
encoder/macroblock.c:918: warning: 'mvp[1]' may be used uninitialized in this function
encoder/macroblock.c:918: warning: 'mvp[0]' may be used uninitialized in this function



With best regards, seemees

[Dark Shikari]

Why *.c but not *.cpp? :-)To keep out people who don't understand basic coding constructs, like !!i ;)

[seemees]

Dark Shikari
Thank you. :-)

I understand.
!!0 = 1;
!!2 = 1;
!!3 = 1;
Does it +!!i means +1? :-)
Or it means +0? :-)
Or am I wrong?

But can we see clear and warning free code in the future? :-)
With best regards, seemees

[Guest]

If you think !!0 = 1 then you should take up knitting instead of coding, and move your queries to the Newbies forum.

[seemees]

Here is the full message list (from analyse.c and below):

encoder/analyse.c: In function 'x264_mb_analyse_inter_b8x16':
encoder/analyse.c:1999: warning: dereferencing pointer 'mvc.443' does break strict-aliasing rules
encoder/analyse.c:1999: note: initialized from here
encoder/analyse.c:2000: warning: dereferencing pointer '({anonymous})' does break strict-aliasing rules
encoder/analyse.c:2000: note: initialized from here
encoder/analyse.c:1999: warning: dereferencing pointer 'mvc.443' does break strict-aliasing rules
encoder/analyse.c:1999: note: initialized from here
encoder/analyse.c:2000: warning: dereferencing pointer '({anonymous})' does break strict-aliasing rules
encoder/analyse.c:2000: note: initialized from here
encoder/analyse.c: In function 'x264_mb_analyse_inter_b16x8':
encoder/analyse.c:1930: warning: dereferencing pointer 'mvc.436' does break strict-aliasing rules
encoder/analyse.c:1930: note: initialized from here
encoder/analyse.c:1931: warning: dereferencing pointer '({anonymous})' does break strict-aliasing rules
encoder/analyse.c:1931: note: initialized from here
encoder/analyse.c:1930: warning: dereferencing pointer 'mvc.436' does break strict-aliasing rules
encoder/analyse.c:1930: note: initialized from here
encoder/analyse.c:1931: warning: dereferencing pointer '({anonymous})' does break strict-aliasing rules
encoder/analyse.c:1931: note: initialized from here
encoder/analyse.c: In function 'x264_intra_rd_refine':
encoder/analyse.c:1104: warning: 'i_nnz[1]' may be used uninitialized in this function
encoder/analyse.c:1104: warning: 'i_nnz[0]' may be used uninitialized in this function
encoder/analyse.c:1103: warning: 'pels_v[6]' may be used uninitialized in this function
encoder/analyse.c:1103: warning: 'pels_v[5]' may be used uninitialized in this function
encoder/analyse.c:1103: warning: 'pels_v[4]' may be used uninitialized in this function
encoder/analyse.c:1103: warning: 'pels_v[3]' may be used uninitialized in this function
encoder/analyse.c:1103: warning: 'pels_v[2]' may be used uninitialized in this function
encoder/analyse.c:1103: warning: 'pels_v[1]' may be used uninitialized in this function
encoder/analyse.c:1103: warning: 'pels_v[0]' may be used uninitialized in this function
encoder/analyse.c: In function 'x264_macroblock_analyse':
encoder/analyse.c:1407: warning: dereferencing pointer 'mvc.380' does break strict-aliasing rules
encoder/analyse.c:1407: note: initialized from here
encoder/analyse.c:1408: warning: dereferencing pointer '({anonymous})' does break strict-aliasing rules
encoder/analyse.c:1408: note: initialized from here
encoder/analyse.c:1409: warning: dereferencing pointer '({anonymous})' does break strict-aliasing rules
encoder/analyse.c:1409: note: initialized from here
encoder/analyse.c:1456: warning: dereferencing pointer 'mvc.388' does break strict-aliasing rules
encoder/analyse.c:1456: note: initialized from here
encoder/analyse.c:1457: warning: dereferencing pointer '({anonymous})' does break strict-aliasing rules
encoder/analyse.c:1457: note: initialized from here
encoder/analyse.c:1458: warning: dereferencing pointer '({anonymous})' does break strict-aliasing rules
encoder/analyse.c:1458: note: initialized from here
In file included from ./common/common.h:698,
from encoder/me.c:25:
./common/macroblock.h: In function 'x264_mb_transform_8x8_allowed':
./common/macroblock.h:465: warning: dereferencing type-punned pointer will break strict-aliasing rules
In file included from ./common/common.h:698,
from encoder/ratecontrol.c:31:
./common/macroblock.h: In function 'x264_mb_transform_8x8_allowed':
./common/macroblock.h:465: warning: dereferencing type-punned pointer will break strict-aliasing rules
In file included from ./common/common.h:698,
from encoder/set.c:26:
./common/macroblock.h: In function 'x264_mb_transform_8x8_allowed':
./common/macroblock.h:465: warning: dereferencing type-punned pointer will break strict-aliasing rules
In file included from ./common/common.h:698,
from encoder/macroblock.c:25:
./common/macroblock.h: In function 'x264_mb_transform_8x8_allowed':
./common/macroblock.h:465: warning: dereferencing type-punned pointer will break strict-aliasing rules
encoder/macroblock.c: In function 'x264_macroblock_encode':
encoder/macroblock.c:893: warning: dereferencing type-punned pointer will break strict-aliasing rules
encoder/macroblock.c: In function 'x264_macroblock_probe_skip':
encoder/macroblock.c:918: warning: 'mvp[1]' may be used uninitialized in this function
encoder/macroblock.c:918: warning: 'mvp[0]' may be used uninitialized in this function
encoder/macroblock.c: In function 'x264_macroblock_encode':
./common/macroblock.h:413: warning: dereferencing pointer 'x' does break strict-aliasing rules
./common/macroblock.h:407: note: initialized from here
In file included from ./common/common.h:698,
from encoder/cabac.c:25:
./common/macroblock.h: In function 'x264_mb_transform_8x8_allowed':
./common/macroblock.h:465: warning: dereferencing type-punned pointer will break strict-aliasing rules
In file included from ./common/common.h:698,
from encoder/cavlc.c:25:
./common/macroblock.h: In function 'x264_mb_transform_8x8_allowed':
./common/macroblock.h:465: warning: dereferencing type-punned pointer will break strict-aliasing rules
In file included from ./common/common.h:698,
from encoder/encoder.c:27:
./common/macroblock.h: In function 'x264_mb_transform_8x8_allowed':
./common/macroblock.h:465: warning: dereferencing type-punned pointer will break strict-aliasing rules
encoder/encoder.c: In function 'x264_slice_write':
encoder/encoder.c:1303: warning: 'bs_bak.p_start' may be used uninitialized in this function
encoder/encoder.c:1303: warning: 'bs_bak.p' may be used uninitialized in this function
encoder/encoder.c:1303: warning: 'bs_bak.p_end' may be used uninitialized in this function
encoder/encoder.c:1303: warning: 'bs_bak.cur_bits' may be used uninitialized in this function
encoder/encoder.c:1303: warning: 'bs_bak.i_left' may be used uninitialized in this function
encoder/encoder.c:1303: warning: 'bs_bak.i_bits_encoded' may be used uninitialized in this function
In file included from ./common/common.h:698,
from encoder/lookahead.c:37:
./common/macroblock.h: In function 'x264_mb_transform_8x8_allowed':
./common/macroblock.h:465: warning: dereferencing type-punned pointer will break strict-aliasing rules
In file included from ./common/common.h:698,
from common/x86/mc-c.c:29:
./common/macroblock.h: In function 'x264_mb_transform_8x8_allowed':
./common/macroblock.h:465: warning: dereferencing type-punned pointer will break strict-aliasing rules
In file included from ./common/common.h:698,
from common/x86/predict-c.c:24:
./common/macroblock.h: In function 'x264_mb_transform_8x8_allowed':
./common/macroblock.h:465: warning: dereferencing type-punned pointer will break strict-aliasing rules


How can fix it? Please help. :-)
With best regards, seemees

[Dark Shikari]

Use a gcc that doesn't suck (i.e. not 4.4.x).

(We do intend to fix those warnings eventually, but the fact that the C specification is rather braindead with respect to aliasing rules doesn't help.)

[seemees]

Dark Shikari
Thank you.
What gcc version we can use to construct not only 32bit but 64bit x264.exe?
I see in SSE4.1 SAD instruction with 8 sad at one core cycle. Do you plan to use it to remake sad procedure in SSE4.1?
2.2.9 Improved Sums of Absolute Differences (SAD) for 4-Byte
Blocks
SSE4.1 adds an instruction (MPSADBW) that performs eight 4-byte wide SAD operations
per instruction. Compared to PSADBW, MPSADBW operates on smaller chunks
(4-byte instead of 8-byte chunks); this makes the instruction better suited to video
coding standards such as VC.1 and H.264. MPSADBW performs four times the
number of absolute difference operations than that of PSADBW (per instruction). This
can improve performance for dense motion searches.
MPSADBW uses a 4-byte wide field from a source operand; the offset of the 4-byte
field within the 128-bit source operand is specified by two immediate control bits.
MPSADBW produces eight 16-bit SAD results. Each 16-bit SAD result is formed from
overlapping pairs of 4 bytes in the destination with the 4-byte field from the source
operand. MPSADBW uses eleven consecutive bytes in the destination operand, its
offset is specified by a control bit in the immediate byte (i.e. the offset can be from
byte 0 or from byte 4). Figure 2-1 illustrates the operation of MPSADBW. MPSADBW
can simplify coding of dense motion estimation by providing source and destination
offset control, higher throughput of SAD operations, and the smaller chunk size.

With best regards, seemees

[Dark Shikari]

MPSADBW is completely useless. It's not actually much faster than regular SAD (all the benchmarks vs regular SAD were not done with cacheline-split-avoiding regular SAD), and it's only useful in an exhaustive search. And Sequential Elimination is over 5 times faster for an exhaustive search than MPSADBW is.

[seemees]

Dark Shikari
Understand.
Thanks again.
With best regards, seemees

[jakor]

BTW are there actually useful instructions for video encoding in SSSE3, SSE4.1, SSE 4.2 whatsoever ?
I happenned to use one instruction of SSE3 once though - it was a kind of useful packed multiply, but in overall it seemes that intel engineers ran out of ideas since SSE2 which operations can be useful for coding people....

[Dark Shikari]

BTW are there actually useful instructions for video encoding in SSSE3, SSE4.1, SSE 4.2 whatsoever ?
I happenned to use one instruction of SSE3 once though - it was a kind of useful packed multiply, but in overall it seemes that intel engineers ran out of ideas since SSE2 which operations can be useful for coding people....
SSSE3: pabsw, psignw, palignr, pmaddubsw
SSE4.1: pblendw, and the new multiply instructions are useful in various cases related to LPCs (but not in x264). the insert/extract ones are sometimes marginally useful.
SSE4.2: useless crap

[akupenguin]

How can you forget pshufb?

[_Woland_]

It seems to be a bug with --b-pyramid "strict". For example:
http://s44.radikal.ru/i105/0910/32/fc34bd152189t.jpg (http://radikal.ru/F/s44.radikal.ru/i105/0910/32/fc34bd152189.png.html)
Here is sample - http://narod.ru/disk/14401053000/Futurama_S01E01_Trim.mkv.html - problems starts from ~1:20.
I hope it will be useful. =)

[Dark Shikari]

Bug confirmed; JM agrees. Forwarded to Trahald.

There's also corruption around ~1:47.

[jpsdr]

Hello.
I don't know where to ask, so i'm asking this here. I've look at the options given in the full help, and unless i've missed
something, there is no such option i'm asking.
Is it possible to have in the future an option to add chapters (final purporse is of course for blu-ray authoring) on
the encoded stream (like it's possible on CineVision).
What could be done is something like :
--chapters file.txt
Chapters could be in format like : HH:MM:SS:CC (hours,minutes,seconds, 100/th seconds) => 01:20:56:89 => chapter
at 1h 20min 56.89s (Or anything like this).
In the files, the chapters will simply be like this :
00:20:56:05
00:35:10:68
01:00:05:54
etc...
If this already exist or is possible on another way, i'm willing to know it.

[detmek]

That is a job for muxer and all muxers (mkvmerge, tsmuxer, mp4box) have that option.

[shon3i]

No x264 have --qpfile option to force frame types. To make qpfile which is text file with framenubers and frametypes i recomend you to use ChapterGen or Chapter Grabber tools. Btw. Cinevision use HH:MM:SS:ff format

[jpsdr]

@detmek : If you want to have the chapters where you want, you have to do it during the encoding process, otherwise, you may not be able to put the chapter at the exact frame you want, you can't put chapter on any type of frame.
@shon3i (or anyone else who know) : I've look at the QP file option... For chapters, what kind of frametype it should be ?
I first think I, but is it the best ? Is there a better choice ? I think QP to -1 is the best to leave ?

[J_Darnley]

Adding the actual chapter metadata is the job the the muxer. If you want to seek instantly to a chapter point, it will have to be an IDR frame (I in the qpfile) at that time. Yes, if you don't want to force the quantiser, set it to -1 and let x264 decide as it normally would.

[jpsdr]

Well, in Scenarist at least, you can't put chapter on any kind of frame, so, you've to make sure during the encoding
process, that frames where you'll put chapter are compatible. Maybe there is not these restrictions on muxer
for pc files (mkvtoolnix or other).
Anyway, thanks for these informations.

[shon3i]

I first think I, but is it the best ? Is there a better choice ? I think QP to -1 is the best to leave ? That's why i recomend to use some of apps to generate qpfile from some chapter source, if you reencode.

[jpsdr]

@shon3i : The purpose is to purely encode, not re-encode, so, I put the chapters on a lagarith YV12 avi file which will be encoded.
The chapters on my file in the first place didn't exist, so chapter source didn't exist. Nevetheless, i'll take a look a the tools you're
talking, to see what they can do.
Last question : frame number in qpfile, is frame begin to 0 (like in VDub) or 1 ?
If i decide to put a chapter on frame number 999 given by VDub (wich so is in fact the 1000th frame). The frame number parameter
of the qpfile, should it be 999 or 1000 ?

[Dark Shikari]

@shon3i : The purpose is to purely encode, not re-encode, so, I put the chapters on a lagarith YV12 avi file which will be encoded.
The chapters on my file in the first place didn't exist, so chapter source didn't exist. Nevetheless, i'll take a look a the tools you're
talking, to see what they can do.
Last question : frame number in qpfile, is frame begin to 0 (like in VDub) or 1 ?
If i decide to put a chapter on frame number 999 given by VDub (wich so is in fact the 1000th frame). The frame number parameter
of the qpfile, should it be 999 or 1000 ?Frames in x264 begin at 0.

[jpsdr]

Thanks DS.

@shion3 : As i was afraid, the tools you're talking about are totaly useless for me, as they extract chapters from files wich already have them, and that's absolutely not my case, as i create them. So, notepad method !

But with all of this, i think i've got now all the informations i need, and thanks to those who have answered to me,
and i'll close this wich may be off topic of this thread finaly.

[bob0r]

I couldn't help but notice x264 r1301 is a lot smaller, at 961KB, than revisions before, which were 1100KB.

I accidentally deleted a wrong directory.
x264 did not fprofile, but did however compile.
This is fixed with revision 1309.
New fprofile line:
make fprofiled VIDS="../1280x720p50_parkrun_ter.yuv";

Thanks!

[thewebchat]

x264 r1310 HEAD
http://img294.imageshack.us/img294/4742/noweightp.png

x264 r1301 WEIGHTP
http://img294.imageshack.us/img294/560/weightp.png

2pass encode. ABR 2500 kb/s. Good work, guys. :devil:

[Chengbin]

Weight-p??????? It is not even committed yet.

[thewebchat]

Doesn't mean you can't use it.

[Chengbin]

Doesn't mean you can't use it.

Can't figure it out.

But I must say, that's very impressive!

[poisondeathray]

How were you able to use it?

What about other frames? Did the other ones that were initially improved with mb-tree get worse?

[thewebchat]

Compiled the GIT GSoC repository (well, JEEB did). For some reason, there doesn't seem to be a way to turn weightp off, so the reference was generated with x264 HEAD. I did not test against MBTree from HEAD, but MBTree still produces crappy fades with weightp on. Without MBTree, quality is all-around better (since this is a pure efficiency advance), as bits that would have been wasted on fades can be put on keyframes and other scenes. Test clip was an artificial composite of 3 dark fadeouts and 1 low-motion bright scene.

[burfadel]

The AQ mod patch works well with fades, maybe a combination of the AQ mod patch (with the setting 1.0:1.0 as standard) combined with weightp will give well balanced results?...

[Puncakes]

Compiled the GIT GSoC repository (well, JEEB did).

Where'd you get that build? Obviously the site is... uninformative, and I don't see any posts here about it...

[thewebchat]

http://repo.or.cz/w/x264/x264-p-frames.git?a=shortlog;h=refs/heads/gsoc

There seems to have been a few useful updates since I posted that screenshot, one of them being labeled "mbtree stuffs," so it seems that this is getting somewhere. I believe JEEB told me that compilation fails on Win64 though.

[Puncakes]

Thanks! Now to get the slightest clue on how to compile it. Google, here I come.

[Wishbringer]

@Puncakes:

Get Komisars MinGW http://komisar.gin.by/mingw/index.html / and Tools

Get MSys Phoenix http://www.cadforte.com/msys.html

Get YASM http://www.tortall.net/projects/yasm/wiki/Download
and you've done the first step.

[wyti]

You can compile it under win64 but you have to use --disable-asm

error log if someone is interested :
common/x86/mc-a.asm:395: error: cannot override register size
common/x86/mc-a.asm:396: error: cannot override register size
common/x86/mc-a.asm:397: error: cannot override register size
common/x86/mc-a.asm:398: error: cannot override register size
common/x86/mc-a.asm:399: error: cannot override register size
common/x86/mc-a.asm:400: error: cannot override register size
common/x86/mc-a.asm:402: error: cannot override register size
common/x86/mc-a.asm:403: error: cannot override register size
common/x86/mc-a.asm:404: error: cannot override register size
common/x86/mc-a.asm:405: error: cannot override register size
make[1]: *** [common/x86/mc-a.o] Error 1

so it isn't interesting to have an x64 build without asm :p

but here is the build for win32 fprofiled build if interested : http://www.mediafire.com/download.php?dtdxzk4d0nl (note : mp4 and mkv oupture not activated, it can only produce elementary streams (you will have to mux) but avisynth input enabled)

[easyfab]

to complete Wishbringer post : How to compile x264

http://forum.doom9.org/showthread.php?t=148615

[moviefan]

Concerning custom builds including a weightp patch: Where can I get the latest revision of the patch and where can I find the current development status of it to know if there are major issues or not?