Current Patches, Where to get them, How they affect speed/output

[skystrife]

x264.912.modified.exe (http://www.mediafire.com/?mc9even3yhx)
(Source) (http://www.mediafire.com/?9mzs4dwdndm)

qpfile_relax has been comitted to git, no longer included.

Patches used:

x264_psyRDO_0.5.diff
x264_hrd_pulldown.05_interlace.diff
x264.progress.indication.01.diff

All patches are included with the source.

gcc 3.4.5 fprofiled build.

[Zwitterion]

I can't mux raw streams created with x264_hrd_pulldown.05_interlace.diff to matroska. Yamb+MP4box works fine though. Mkvmerge displays the following error:
---------------------------
File identification failed
---------------------------
File identification failed for '4129_05.264'. Return code: 2

err end-of-file

Error: File 4129_05.264 has unknown type. Please have a look at the supported file types
('mkvmerge --list-types') and contact the author Moritz Bunkus <moritz@bunkus.org>
if your file type is supported but not recognized properly.


---------------------------
OK
---------------------------

h264 file created with x264_hrd_pulldown.04_interlace.diff (http://www.mediafire.com/?1xdb4mgqlxm)
h264 file created with x264_hrd_pulldown.05_interlace.diff (http://www.mediafire.com/?kcpnymouox1)

command line:
--crf 26 --level 4.1 --keyint 24 --min-keyint 1 --ref 3 --mixed-refs --bframes 3 --b-rdo --bime --weightb --direct auto --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 30000 --vbv-maxrate 38000 --qcomp 0.5 --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input" --mvrange 511 --aud --sar 1:1 --nal-hrd

skystrife's x264.912.modified.exe for x264_hrd_pulldown.05_interlace.diff
bob0r's x264.901.modified.01.exe for x264_hrd_pulldown.04_interlace.diff

[Sagittaire]

I can't mux raw streams created with x264_hrd_pulldown.05_interlace.diff to matroska.


Well HRD flag are here for HDDVD/BD hardware/software compatibility. It's completely useless for matroska mux. Anyway it's true: there are difference between 04 and 05 build.

[Trahald]

Thanks for the updated patch!
I have one question, though: When activating --nal-hrd no userdata which describes the encoder settings is being written (at least with x264_hrd_pulldown.04_interlace.diff.txt, haven't tried 05):

I mean like this:
x264
core 60 r901M 58d7d06
H.264/MPEG-4 AVC codec
Copyleft 2003-2008
http://www.videolan.org/x264.html
cabac=1
ref=3
...
...
...

Can this be fixed or is it unavoidable? :thanks:

yeah.. i will after im sure the other issue is fixed. (i havent looked at zwitterions streams yet tho)

[nm]

Well HRD flag are here for HDDVD/BD hardware/software compatibility. It's completely useless for matroska mux.
The interlace flags are useful though, when encoding as interlaced. Otherwise many decoders/players won't deinterlace automatically.

[lexor]

The interlace flags are useful though, when encoding as interlaced. Otherwise many decoders/players won't deinterlace automatically.

Isn't that what interlace patch is for? The one everyone keeps mistaking for hdr replacement.

[nm]

Isn't that what interlace patch is for? The one everyone keeps mistaking for hdr replacement.
Perhaps, but the interlace signaling was originally built on top of the HRD patch, and the versions discussed here contain these modifications, as the name (x264_hrd_pulldown.05_interlace.diff) suggests. I'm just saying that the combined patch isn't as useless for Matroska output as Sagittaire said.

[lexor]

Perhaps, but the interlace signaling was originally built on top of the HRD patch, and the versions discussed here contain these modifications, as the name (x264_hrd_pulldown.05_interlace.diff) suggests. I'm just saying that the combined patch isn't as useless for Matroska output as Sagittaire said.
Actually the origin of that name is that orginal hrd patch didn't work for interlaced, it was expanded (by a different person, I believe) and became x264_hrd_pulldown.interlace.diff. The point is that you don't need hrd for general interlaced footage, you need it for blu-ray/hddvd only.

As x264 has built in interlaced support, I think that should be expanded to tag interlaced stuff correctly, it seems like an omission that should be fixed in git not a patch. HRD patch really isn't there to fix the interlaced stream.

[nm]

Actually the origin of that name is that orginal hrd patch didn't work for interlaced, it was expanded (by a different person, I believe) and became x264_hrd_pulldown.interlace.diff.Yes, that's what I was trying to say. It's just that currently there aren't many builds available that contain the separated interlace signaling patch instead of the HRD version. Not that I care about Windows binaries myself, but Zwitterion was using them above.

The point is that you don't need hrd for general interlaced footage, you need it for blu-ray/hddvd only.
Agreed.

As x264 has built in interlaced support, I think that should be expanded to tag interlaced stuff correctly, it seems like an omission that should be fixed in git not a patch.
Agreed, although there is probably a good reason why the patch is not yet in git.

[bob0r]

The reasons are simple:
1: the bluray/hddvd/other devices specs are not public
2: the universe must explode before pengvado will waste any time regarding interlacing (unless paid sufficient).

I say drop interlacing completely, with enough x264 mkv movies out there, the first company that provides full support may count themselfs rich, very rich.

We should not adapt to the companies, they should adapt to us. And by simply NOT encoding with these useless patching or stupid levels, we will win.
But i guess laziness will overcome all.

[Trahald]

i have to talk to dark shikari -- the ue code on the large bitrate values are foobaring depending on the bit alignment at at that point of the sps (thats why when i testing it seemed ok) im sure due to the size of bitrate and cpb ue values. my brain hurts and i have alot going on.. i'll look at it again.. but for now .. dont patch any binaries until i release .6.

[Dark Shikari]

i have to talk to dark shikari -- the ue code on the large bitrate values are foobaring depending on the bit alignment at at that point of the sps (thats why when i testing it seemed ok) im sure due to the size of bitrate and cpb ue values. my brain hurts and i have alot going on.. i'll look at it again.. but for now .. dont patch any binaries until i release .6.Isn't there a UE_big function just for this purpose?

[CruNcher]

The Real Question is why should Consumer support HD-DVD or Blu-Ray @ all we don't have to , we are independent of this we don't need to restrict ourselves (but i have to admit i also try to stay Blu-Ray compatible most of the time), but i question me myselve for what, wich user is really gonna ever use a Blu-Ray Player that restricts the Quality of his content, hacking firmware (giving us freedom) unrestrict us from them that's what we should do not play their Game. If DivX inc. really means it true they are gonna give us this freedom finaly in the next incarnation of their Ecosystem :)

[Trahald]

ok... i rewrote the scale routine to make smaller values for now until i can figure out why it hates big values. what that means is the precision of the values will be off to the low side.. ie a bitrate of 30000000 hay be stored as 29999995 .. that shouldnt be an issue (high values would be in some cases, low values should be fine).. i'll have a patch file avail in a few mins

[Trahald]

Isn't there a UE_big function just for this purpose?
yeah... thats the routine i use.. and it works sometimes.. but other times depending on the offset it doesnt (ie depending on some values of i_left).. at least thats what a few ours of scratching my head have come up with.. my brain is fried from lack of sleep so i could be wrong

[akupenguin]

As x264 has built in interlaced support, I think that should be expanded to tag interlaced stuff correctly, it seems like an omission that should be fixed in git not a patch.
Be wary of using an argument like that, for I might take it as an argument for dropping interlaced support instead.

[Trahald]

heres a new patch.. again.. use only for testing... i ran a couple quick encodes with different settings and output was correct. (keeping in mind the workaround may use slightly lower values for bitrate and cpb size.)
*edit* attachment removed

[Dark Shikari]

yeah... thats the routine i use.. and it works sometimes.. but other times depending on the offset it doesnt (ie depending on some values of i_left).. at least thats what a few ours of scratching my head have come up with.. my brain is fried from lack of sleep so i could be wrongWhat's the size of the exp-golomb code that it breaks on?

[Trahald]

What's the size of the exp-golomb code that it breaks on?

anything 0x10000 or bigger...

[Dark Shikari]

anything 0x10000 or bigger...static inline void bs_write_ue_big( bs_t *s, unsigned int val )
{
int size = 0;
int tmp = ++val;
if( tmp >= 0x10000 )
{
bs_write32( s, 0 );
tmp >>= 16;
}
if( tmp >= 0x100 )
{
size = 16;
tmp >>= 8;
}
size += x264_ue_size_tab[tmp];
bs_write( s, size, val );
}So you're saying that first if-block fails... that strikes me as odd. Talk to pengvado about this, he wrote that section.

[skystrife]

x264.913.modified.exe (http://www.mediafire.com/?dnc9bhgm4ty)
(Source) (http://www.mediafire.com/?dhbmglwzjfm)

Patches used:

x264_psyRDO_0.5.diff
x264_hrd_pulldown.06_interlace.diff
x264.progress.indication.01.diff

All patches are included with the source.

gcc 3.4.5 fprofiled build.

[lexor]

Be wary of using an argument like that, for I might take it as an argument for dropping interlaced support instead.

Eh? I want it gone, waste of code lines and effort! I thought Avail, not you, are the guys who put interlacing in there, they should've been made to do a better job before it was allowed in.

[Dark Shikari]

Eh? I want it gone, waste of code lines and effort! I thought Avail, not you, are the guys who put interlacing in there, they should've been made to do a better job before it was allowed in.Akupenguin worked at Avail for over two years--and he was the one who wrote the interlacing support ;)

[LoRd_MuldeR]

Well, in an ideal video world interlacing would have gone long ago and no interlacing support would be required.
But in the reality of the video world interlacing is there and it's going to stay for a while. HDTV and AVCHD use it - if we want or not.
Therefore I think ignoring that fact and dropping interlacing support would be a bad idea, now that x264 is one of the most AVC encoders available.
If nobody is willing to improve interlacing support at the moment, which I understand very well, why not keep it as-is and that's it?

If the interlacing code was blocking future improvements in x264, then I would vote for dropping it :)


// General question //

Why not stack the fields of interlaced video, encode the frames progressive and re-weave them at playback?

This would require a "stack fields" filter before/in the encoder, an "unstack fields" filter in/after the decoder and one new flag in the bitstream.
All of the rest could be done 100% identical to progressive encoding. No special interlaced handling would be needed at all. Nothing.
Sounds to me like this could be implemented very easy. So why is it not done in real video formats like H.264 that way ???

[akupenguin]

Why not stack the fields of interlaced video, encode the frames progressive and re-weave them at playback?

This would require a "stack fields" filter in the encoder, an "unstack fields" filter in the decoder and one new flag in the bitstream.
All of the rest could be done 100% identical to progressive encoding. No special interlaced handling would be needed at.
That would indeed make things simple. But it reduces compression, even compared to non-adaptive field-based encoding, because you can't MC from the opposite parity (at least not without coding an excessively long mv), and it screws extrapolation of mvs over the packing boundary (which would otherwise be a frame edge).

[LoRd_MuldeR]

I see that MC is less efficient. Then maybe separating the fields (instead of stacking them) would be the better way MC-wise?
You would encode the double number of (progressive) frames, but each frame is only half of the vertical size. Overhead should be small.
At playback you simply re-weave two consecutive frames after decoding to get an interlaced frame again...

[akupenguin]

Yep. And that's pretty much what non-adaptive PAFF is, except that PAFF somehow made that conceptually simple process very complex. (e.g. by requiring special cases in the reference list generation, and special cases in MCing opposite parity, and special cases in deblocking, and ...)

[Trahald]

ok.. this change to the UE code works better int size = 0;
int tmp = ++val;
if( tmp >= 0x10000 )
{
bs_write( s, 16, 0 ) // CHANGED:[bs_write32( s, 0 );] -- (32 bits of 0's were too many)
size = 16; // ADDED: size wasnt being pushed up causing a truncated 'val' to be written
tmp >>= 16;
}
if( tmp >= 0x100 )
{
size += 16; //CHANGED: to += in case of really long values.
tmp >>= 8;
}
size += x264_ue_size_tab[tmp];
bs_write( s, size, val ); i'll put it in .7 patch in a bit

[akupenguin]

Nothing ever needs to write a value >= 0x1000000 ?

[Trahald]

i made the edit as you were typing i guess... :D

[Trahald]

i'll test that last edit.. as just thought that will pass > 32 bit values into the bitstream writer... i'll see if can handle it...

[akupenguin]

Nope, still tries to write up to 47 bits at once, and the point of the original is that we can't do more than 31.
Try this, which should work up to 0x7ffffffe.

--- a/common/bs.h
+++ b/common/bs.h
@@ -169,16 +169,17 @@ static inline void bs_write_ue_big( bs_t *s, unsigned int val )
int tmp = ++val;
if( tmp >= 0x10000 )
{
- bs_write32( s, 0 );
+ size = 32;
tmp >>= 16;
}
if( tmp >= 0x100 )
{
- size = 16;
+ size += 16;
tmp >>= 8;
}
size += x264_ue_size_tab[tmp];
- bs_write( s, size, val );
+ bs_write( s, size>>1, 0 );
+ bs_write( s, (size>>1)+1, val );
}

/* Only works on values under 255. */

[Dark Shikari]

i'll test that last edit.. as just thought that will pass > 32 bit values into the bitstream writer... i'll see if can handle it...No, it can't. The bitstream writer cannot handle any size larger than 31 bits, I believe.

[Trahald]

Nope, still tries to write up to 47 bits at once, and the point of the original is that we can't do more than 31.
Try this, which should work up to 0x7ffffffe.

--- a/common/bs.h
+++ b/common/bs.h
@@ -169,16 +169,17 @@ static inline void bs_write_ue_big( bs_t *s, unsigned int val )
int tmp = ++val;
if( tmp >= 0x10000 )
{
- bs_write32( s, 0 );
+ size = 32;
tmp >>= 16;
}
if( tmp >= 0x100 )
{
- size = 16;
+ size += 16;
tmp >>= 8;
}
size += x264_ue_size_tab[tmp];
- bs_write( s, size, val );
+ bs_write( s, size>>1, 0 );
+ bs_write( s, (size>>1)+1, val );
}

/* Only works on values under 255. */

works perfect! test large values and semi large (as w/large bitrates) all came out good on the other end.

[Sharktooth]

so, coukd you please link the new patch?

[Trahald]

i was kinda hoping he would commit the ue code changes to the main. all that has to happen is the akus patch be used then .05 after (not .06 which has the workaround) on top of it... i'll post a combined patch in a few mins tho

[Dark Shikari]

i was kinda hoping he would commit the ue code changes to the main. all that has to happen is the akus patch be used then .05 after (not .06 which has the workaround) on top of it... i'll post a combined patch in a few mins thohe already did (http://git.videolan.org/?p=x264.git;a=commit;h=fee2f2ce3cda8e39cc4c123afa726933b6b9b877)

[Trahald]

he already did (http://git.videolan.org/?p=x264.git;a=commit;h=fee2f2ce3cda8e39cc4c123afa726933b6b9b877)

ahh.. excellent... attached is .07 which is really .05 but renamed to prevent confusion.

[Sharktooth]

nice stuff. thanks.
however i still cant understand why the hrd_pulldown is not in the official repo since it's absolutely needed to crete BD compliant bitstreams.
ok, we all know those additional info are not necessary and can be added anytime if needed, however x264 is in a strong position against commercial codecs and it's just a shame it cant produce BD compliant streams by default.

[lexor]

nice stuff. thanks.
however i still cant understand why the hrd_pulldown is not in the official repo since it's absolutely needed to crete BD compliant bitstreams.
ok, we all know those additional info are not necessary and can be added anytime if needed, however x264 is in a strong position against commercial codecs and it's just a shame it cant produce BD compliant streams by default.

last time that was asked, aku said that because the patch is based on stream analysis + guesswork and not backed up by any solid spec, he doesn't want it the main code base.

[Sharktooth]

yeah, i know that too. but most of the OSS is based on black box RE or RE...
so i cant see that one as a valid point.

[skystrife]

x264.914.modified.exe (http://www.mediafire.com/?egggwkonzyy) - Alternate Download (http://skystrife.com/x264/x264.914.modified.exe)
(Source) (http://www.mediafire.com/?bfzbddsbkiz) - Alternate Download (http://skystrife.com/x264/x264.914.modified.source.7z)

Patches used:

x264_psyRDO_0.5.diff
x264_hrd_pulldown.07_interlace.diff
x264.progress.indication.01.diff

All patches are included with the source.

gcc 3.4.5 fprofiled build.

[Sharktooth]

bobor? what about a new patched/modified build? since i still have an internet connection im sure i can upload at least the x264 update to megui auto-update server.
psy RD0 0.5 rox, and the new hrd-pulldown 0.7 is working perfectly.
i have to learn how to crosscompile x264 from linux to win32.

[Sharktooth]

@skystrife: medifire says: NOTICE: No servers are currently available with the requested data on them. Please retry your request in a moment.
could you plaese upload to another service?

[skystrife]

@skystrife: medifire says: NOTICE: No servers are currently available with the requested data on them. Please retry your request in a moment.
could you plaese upload to another service?

That's odd. Yes, I'll put another link up.

EDIT: It's up. =)

[TheRyuu]

x264.914.modified.exe (http://www.mediafire.com/?twzmpzqjsmz)
(alt DL link (http://www.zshare.net/download/156128790c0f3f89/))

GCC 3.4.5 fprofiled build with static-pthreads.
Platform: X86
System: MINGW
asm: yes
avis input: yes
mp4 output: yes
pthread: yes
Everything else (useless for a cli build) is no.

Patches:
psyRDO 0.5
hrd pulldown 0.7 interlace
progress indication

[desta]

I've noticed with these latest builds, that if I include --nal-hrd in the commandline, when I later check the encode in something like avinaptic, I no longer see the list of encoding options I used. If I omit the --nal-hrd option, then everything shows up as normal.

[Dark Shikari]

I've noticed with these latest builds, that if I include --nal-hrd in the commandline, when I later check the encode in something like avinaptic, I no longer see the list of encoding options I used. If I omit the --nal-hrd option, then everything shows up as normal.Sounds like NAL-HRD managed to corrupt the SEI or something...

[desta]

Ah yeah, sorry, just looked back through the topic to see it's already been mentioned. :rolleyes:

[Trahald]

nah.. doesnt corrupt.. purposefully omits... i wrote a --user-data option that will add it back in... will have a diff out tonight.. user data sei message is supposed to be safe as long as buffering sei message is first in the AU. i'll leave it an defaulted off until i get a few all clear reports.