x264 and high profile: lossless avc encoding is here

[bond]

pengvado/akupenguin seems to be unstopable! :D

after he brought us 8x8dct and i8x8 (http://forum.doom9.org/showthread.php?t=95441) of avc high profile in revision 251, he now also added support for lossless encoding starting with revision 264 of x264, thx a lot! :)

for enabling it you simply have to set the quant (--qp/-q) to 0
of course you can also still enable all the other goodies, like loop, cabac, weighted biprediction, b-frames aso without loosing the lossless mode but still enhancing the compression strength

pengvado also added support for lossless decoding to libavcodec/ffmpeg (edit: nero's decoder should not support it)

enjoy! :)

[leowai]

Sounds great! Lossless encoding for x264 rev264 (note the two magic numbers). :)

Wondering how is the compression for the lossless encoding compared to default setting of --qp 26.


regards,
leowai

[Manao]

Actually, loop will always be disabled in lossless, and as akupenguin stated on irc, i8x8 tends to deserve the compression ratio.

[bond]

Actually, loop will always be disabled in lossless, and as akupenguin stated on irc, i8x8 tends to deserve the compression ratio.
interesting, i already wondered how loop fits lossless

[ernstblaauw]

Wow! I'm interested how it compares to hufyuv, for example

[JoeBG]

I never used a lossless codec for encoding, so I don´t understand everything from th background of it. I hope someone can help with the following questions:
- when I use -q, then I dont´t use "bitrate" ?
- is it one or two pass encoding?

Thanks in advance

[Doom9]

- when I use -q, then I dont´t use "bitrate" ?correct, and completely unrelated to lossless vs lossy.. it's been like that forever (remember the CQ first passes in MeGUI?)
- is it one or two pass encoding?CQ modes are one pass so I think that'll apply here two (in Ateme HP lossless was apparently possible in two pass mode but if I read correctly, the ability to perform the second pass will be eliminated now). You can't really improve upon lossless by a second pass, reach the target size more accurately (cq has no target size), be more constant bitrate (wmv9 allows 2 pass cbr)..

[bond]

in lame speak (my language :D ):

lossy compression works that way that the size of every frame gets lossly reduced (leaving for example details away) by a specific value, the quant
how strong a frame gets "smaller" is defined by the size of the quant: the higher the quant, the smaller the framesize and therefore the stronger the compression
now the idea is that a quant of 0 means no lossy compression of the frame -> lossless encoding

what frame gets compressed by what quant gets decided by the ratecontrol (eg via a 2pass), but as all frames get a quant of 0 you dont need that decision and therefore no 2pass

[Tommy Carrot]

I've tried lossless mode in the vfw codec, and here are my observations.

If it's really lossless, then it's significantly more efficient than the lossless mode of ateme's beta (624,878k vs. 668,720k). Although it's still behind snow and msu, but beats ffv1 on the one sample i tried (of course the order might be different on other samples).

Apparently 8x8 transform is not used in lossless mode, i've got identical files with disabled and enabled 8x8 transform.

And the strangest: ateme's lossless mode can be decoded correctly by a week old mplayer build, which doesn't contain the lossless fixes, while x264 only works with fresh cvs build.

[JoeBG]

@ doom9 and bond

many thanks, everything understood :)

[berrinam]

I suppose this could be rather useful for doing two-pass encodes with large amounts of filtering, i.e. filter on the first pass, write a stats file and a lossless encode, then on the second pass, use the lossless encode as the source ... make sense?

[lexor]

I suppose this could be rather useful for doing two-pass encodes with large amounts of filtering, i.e. filter on the first pass, write a stats file and a lossless encode, then on the second pass, use the lossless encode as the source ... make sense?
Do you use lossy on 1st pass then? Because if you don't then using Lossless with filters (avisynth for say), then encoding it again for 2nd pass in lossless without any additional filters should yield no difference from 1st pass.

[bill_baroud]

i think the idea behind is that if you use the lossless first pass in a smart way, you won't have to re-do all the H264 compression sheme (since you just didn't do the quantization). Then you would have just to do the quant (rate control) + CABAC in the 2nd pass. But perhaps i'm way off what he's thinking and what i'm talking about is totally impossible.

[stephanV]

I suppose this could be rather useful for doing two-pass encodes with large amounts of filtering, i.e. filter on the first pass, write a stats file and a lossless encode, then on the second pass, use the lossless encode as the source ... make sense?

Theoretically that should be possible... but the stats file created would probably not be a good basis for a 2nd pass. The difference in behaviour of the (or any) codec between q0 and say > qp18 can be quite large, so estimations made during the first pass can be way off.

Usually the rule is, if you do a 1st pass based on a qp-value, you should use a qp-value that matches your target bit rate as closely as possible. With qp0 this is way off of course.

[baer999]

Has anybody experience in the new feature ? My question is how big is the file when encoding lossless (normal DVD movie ~ 100 min).

thx

[SeeMoreDigital]

Has anybody experience in the new feature ? My question is how big is the file when encoding lossless (normal DVD movie ~ 100 min).

thxI asked this question a few days ago... And got this reply (http://forum.doom9.org/showthread.php?p=663599#post663599) from StephanV.


Cheers

[baer999]

Thanks, but if it would be 297 MB per minute every movie had at least a size of 30-40 GB or what does 297MB/min means ?

[SeeMoreDigital]

It means a 100min movie would be: -

297MB per min x 100 min = 29,700MB or 29GB


That said, earlier today I attempted to generate my own "1 minute" AVC lossless encodes and they both came in at around 81.36MB per min... However, they were borked :eek:

http://img297.echo.cx/img297/2857/avclossless4ew.jpg


Cheers

[baer999]

Oh ok I thought it would be a DVD movie with max 9 GB ... but you can use lossless also @ a DVD movie or not ?

[Tommy Carrot]

That said, earlier today I attempted to generate my own "1 minute" AVC lossless encodes and they both came in at around 81.36MB per min... However, they were borked :eek:
Cheers
See my previous post for the answer. Libavcodec got a small fix to be able to decode lossless h.264 videos, and ffdshow and mplayer hasn't been upgraded yet, so they don't contain that fix. But strangely enough, ateme's lossless mode doesn't require that fix, even older libavcodec can decode it correctly. :confused:

@Baer: DVDs are already compressed, don't expect to be able to compress it further losslessly.

[baer999]

Yes of course but I thought, that you can encode to x264 without lose any quality. Thats what I understood by lossless... sorry

[Tommy Carrot]

No problem. :)

Meanwhile i made a little comparison test between a few lossless codecs:

│ │ encoding
codec │ filesize │ time (m:s)
─────────┼────────────┼────────────
Huffyuv │ 1,380,676k │ 2:16
ffvhuff │ 891,784k │ 2:43
Lagarith │ 819,462k │ 3:41
Ateme │ 682,562k │ 14:04
ffv1(ac) │ 680,100k │ 6:29
x264 │ 665,470k │ 19:38
snow │ 653,783k │ 22:39
msu │ 613,828k │ 26:42

The lossless capability of h.264 is not as hopeless as i first thought, x264 is already quite close to the best lossless codecs, and considering that it just recently got lossless mode, i reckon there are still many ways left to tweak it for even better.

Edit: disabled b-frames in ateme codec, this way i gained more than 30 MB in filesize. I think i can say that b-frames are generally not good for lossless encoding. :D

[SeeMoreDigital]

... But strangely enough, ateme's lossless mode doesn't require that fix, even older libavcodec can decode it correctly. :confused: I tried my lossless encodes with the newest version of NeroDigital's DSdec filter in ShowTime and MediaPlayer Classic.... So I'm confused too!

EDIT: I would not mind being able to buy a camcorder that recorded in lossless AVC :)


Cheers

[bond]

See my previous post for the answer. Libavcodec got a small fix to be able to decode lossless h.264 videos, and ffdshow and mplayer hasn't been upgraded yet, so they don't contain that fix. But strangely enough, ateme's lossless mode doesn't require that fix, even older libavcodec can decode it correctly. :confused:
i cant reproduce this with a lossless clip bobololo posted some days ago

but i can reproduce this with lossless streams encoded with the reference

i noticed that the stream which plays in mplayer (and nero too, which doesnt handle lossless) has the Qpprime Y Zero Transform Bypass flag set to 0, which means its not lossless (still the filesize aso indicates its lossless)

try extracting the avc stream from .mp4 to raw (eg with mp4box or mp4creator) and than run mpeg4ips h264_parse tool over it and check what the above mentioned flag says

[berrinam]

i think the idea behind is that if you use the lossless first pass in a smart way, you won't have to re-do all the H264 compression sheme (since you just didn't do the quantization). Then you would have just to do the quant (rate control) + CABAC in the 2nd pass. But perhaps i'm way off what he's thinking and what i'm talking about is totally impossible.
This is partly what I was thinking about, but I think that could be way too hard to code. What I was really talking about was this:

Traditionally, if you wanted to do an encode with VERY time-consuming filters (eg restore24), for the best over-all encode time, you would:

1. Filter, and output to Huffyuv or something else lossless.
2. Read in the Huffyuv file and do a first pass.
3. Read in the Huffyuv file and do a second pass.

You would do it this way, because, if you were doing an encode which took (say) 3 hours, the filtering could well take 10 hours, so it is best to only filter once. What I propose for filtering like this now, is:

1. Filter, output to lossless x264, while writing first-pass stats file.
2. Use lossless x264 file as filtered source, do second pass from first-pass stats file.

So this saves the extra time of encoding to something lossless, and then decoding twice.

[akupenguin]

i noticed that the stream which plays in mplayer (and nero too, which doesnt handle lossless) has the Qpprime Y Zero Transform Bypass flag set to 0, which means its not lossless (still the filesize aso indicates its lossless)
Lossy qp0 is similar size to lossless qp0.

1. Filter, output to lossless x264, while writing first-pass stats file.
2. Use lossless x264 file as filtered source, do second pass from first-pass stats file.
1. Filter, write to a fast lossless codec (ffvhuff) and also pass the same frames to x264 normal 1st pass.
2. Read the ffvhuff and encode x264 2nd pass.

[berrinam]

1. Filter, write to a fast lossless codec (ffvhuff) and also pass the same frames to x264 normal 1st pass.

How do you pass the same frames to two codecs?

[berrinam]

Meanwhile i made a little comparison test between a few lossless codecs

Do you have any speed information?

[akupenguin]

How do you pass the same frames to two codecs?
avs2yuv (http://students.washington.edu/lorenm/src/avisynth/avs2yuv/) and sclavc (http://students.washington.edu/lorenm/src/avisynth/sclavc/) both do it.

[berrinam]

Thanks, I didn't know about that.

[unmei]

As my lossless looks the same way in FFDShow as SMDs, and OSD shows "mean frame quantizer: not available", please someone clear this for me:
Is it enough to set CQ 0, or do i need to set B-frames to zero or B-frame reduction to zero as well? (does x264 "realize" when i want lossless and turn them off internally or do i have to do it manually? - in my understanding if i have B-frames with Q > 0, that file is not really lossless).

[bond]

@ unmei

of course you can also still enable all the other goodies, like loop, cabac, weighted biprediction, b-frames aso without loosing the lossless mode but still enhancing the compression strength

Libavcodec got a small fix to be able to decode lossless h.264 videos, and ffdshow and mplayer hasn't been upgraded yet, so they don't contain that fix.
:o

[akupenguin]

As my lossless looks the same way in FFDShow as SMDs, and OSD shows "mean frame quantizer: not available", please someone clear this for me:
Is it enough to set CQ 0, or do i need to set B-frames to zero or B-frame reduction to zero as well? (does x264 "realize" when i want lossless and turn them off internally or do i have to do it manually? - in my understanding if i have B-frames with Q > 0, that file is not really lossless).
As a special case, cqp=0 will also set B-frame qp=0, regardless of the value of B-frame reduction.

[ReinerSchweinlin]

Please help me out:
I encoded some testmaterial in vd using the new 265build of x264 in losless-mode. When I try to play it back or load it in vd, the video is all grey. I used the latest build on celtric-druids page (june 11,2005). Some people in this thread seemed to get this working. Am I missing something?

EDIT: When openig in Virtualdub, it shows some INTEL ITU H.264 FOURCC... Could this have to do something with the faulty playback? How did i get this CODEC, I don´t remember installing it. Can I delete it some how?

[bond]

hell, is it really so hard to read???

so once again, so everyone understands it: there is no ffdshow compile out till now which handles playing avc lossless. so with the current compiles you will not be able to decode avc lossless correctly

the next one who reports that issue will get banned! :p

[ReinerSchweinlin]

Sorry ;) It seemes I misunderstood this post

As my lossless looks the same way in FFDShow as SMDs, and OSD shows "mean frame quantizer: not available", please someone clear this for me:
Is it enough to set CQ 0, or do i need to set B-frames to zero or B-frame reduction to zero as well? (does x264 "realize" when i want lossless and turn them off internally or do i have to do it manually? - in my understanding if i have B-frames with Q > 0, that file is not really lossless).

...and didn´t read the following post... :rolleyes:
Shame on me..

[Tommy Carrot]

Do you have any speed information?Added.

[berrinam]

Added.
Thanks.

I see x264 fits in nicely with the speed hierarchy, as opposed to Ateme, which is way too slow for the comparative quality.

[Tommy Carrot]

I made an another comparison test, this time on cartoon content:

huffyuv : 1,725,572k
ffvhuff : 1,324,222k
lagarith : 1,258,654k
ffv1(ac) : 1,101,190k
snow : 464,032k
x264 : 442,630k
msu : 397,542k

Apparently for the task of losslessly compressing cartoons, the motion-compensation capable codecs have great advantage over the keyframe-based codecs (which is logical, but i didn't think that the difference is going to be this much).

[SeeMoreDigital]

Hi Tommy,

For those who can't work it out... How many MegaBytes per minute or persecond would this be?


Cheers

[Tommy Carrot]

Hi Tommy,

For those who can't work it out... How many MegaBytes per minute or persecond would this be?


Cheers
Well, the whole point of this test was to compare the efficiency of the lossless codecs to each other, so i don't think the bitrates are important (not to mention that the bitrate depends on the source very strongly).

But if you insist:
msu: 7803 kbps (~975 KB/S)
ffv1: 21624 kbps (~2.6 MB/s)

I did not keep the other encodings, but you can calculate the bitrates for them. The source was 640x352 with 23.976 fps.

[bill_baroud]

:o
i can't encode a 1080p video with x264 because 1080 isn't a mod16 resolution .... wtf did they choose 1080 for HD ???
with FFV1 i'm at 268mb for 10sec, i was hoping to go under 256mb (to fit on my usb dongle) with x264, to have a HD sample to play with :(

[Manao]

bill_baround : it's x264's fault. The AVC/h264 standard allows for mod8 resolutions.

[bill_baroud]

oh ok, i found that rather strange. I just added a "Addborders(0,4,0,4)" to have a mod16 resolution, but now i got this : http://moodub.free.fr/x264/x264ls_bork0000.jpeg

That's too bad because it was 236mb, so that would have fit on my usb dongle. So i'll stick with ffv1, split the file in two, use my mp3 player too, and will test Ateme HP on HD content this evening ;)


i liked the speed on the crappy Celeron 2ghz here : 0.2 fps at max :D

[akupenguin]

oh ok, i found that rather strange. I just added a "Addborders(0,4,0,4)" to have a mod16 resolution, but now i got this : http://moodub.free.fr/x264/x264ls_bork0000.jpeg Fixed.

[bill_baroud]

wow, many thanks :)

i think i'm in love with this type of reply ;)

[TEB]

does x264 support 4:2:2 coding in lossless mode? if not, when will it?
teb

[Sharktooth]

..... no. it will support it if someone will do it .....

[bond]

Fixed.
does this mean x264 now supports mod8?

[Sharktooth]

no, it means the coded buffer size is now automatically adapted so the HD lossless encoding does no longer b0rk... :)

[Tommy Carrot]

Aku, do you have any idea why the keyframe compression of x264 is so inefficient in lossless mode? They are about 1.5 times bigger than the keyframes of snow or ffv1, they are on par or even a bit bigger than the same frames compressed with huffyuv, which uses much less powerful toolset. I'm just wondering if the keyframe-compression (and i suppose the intra-block coding) described in the standard is so inefficient or something is borked in x264.

[Shinigami-Sama]

I made an another comparison test, this time on cartoon content:

huffyuv : 1,725,572k
ffvhuff : 1,324,222k
lagarith : 1,258,654k
ffv1(ac) : 1,101,190k
snow : 464,032k
x264 : 442,630k
msu : 397,542k

Apparently for the task of losslessly compressing cartoons, the motion-compensation capable codecs have great advantage over the keyframe-based codecs (which is logical, but i didn't think that the difference is going to be this much).

if that is a single episode of animated stuff there, I;d use this for it then, only 1.5x bigger thna I shoot for :D

[Manao]

Tommy Carrot : because h264 was never meant to be lossless in the first place ? The efficiency of h264 in lossless in comparison to ffv1 is only due to temporal prediction. If there's a lot of movement, you can see that ffv1 becomes as efficient ( if not more ) than x264.

Just to remind you : lossless mode works exactly the same in h264 that other mode, except that we skip the transformation / quantization / inverse transformation part. That was intended for simplicity, but clearly not for efficiency.

[Tommy Carrot]

Manao, i know this, i probably shouldn't expect wonders from h.264, it's just that there are some indications that something is not right with x264. I made keyframe-only encodings with x264 and ateme, and ateme produced ~20% smaller filesizes. Now i know that ateme is not really lossless, but afaik the reason for that is a bug, which shouldn't really influence the filesize. Also Snow, which is similar in concept to h.264, has about the same keyframe-compression efficiency as ffv1.

Shinigami-Sama: those encodes are 10000 frames long, a full episode would be about 1.5-2 GB losslessly. :D

[Manao]

Now i know that ateme is not really losslessWhen that happens, a quick visual check will show in an obvious way the lossyness. But it isn't systematic. Now, i find surprising that x264's iframes are less efficient than ateme's. Does that happen even with -m 6 ? Did you enabled i4x4 ?

For Snow, it's wavelet based, so it doesn't work as h.264

[Tommy Carrot]

When that happens, a quick visual check will show in an obvious way the lossyness. But it isn't systematic. Now, i find surprising that x264's iframes are less efficient than ateme's. Does that happen even with -m 6 ?Yep, and in my experience RDO has minimal impact on the filesize in lossless mode, usually around 1-2%.
Did you enabled i4x4 ?Of course. :)
For Snow, it's wavelet based, so it doesn't work as h.264Yep, but it uses spatial transform as well, and despite that it's still able to reach the same efficiency as ffv1. I know that wavelets are more suited for lossless coding, but there shouldn't be so big difference because of the spatial transform imo.

[Tommy Carrot]

I think i found the cause of the bloated keyframes. It doesn't matter if i enable or disable 4x4 (or 8x8) intra search, nothing will change, x264 is only using 16x16 intra search, the other search options are ignored.

[CREXbzh]

I think i found the cause of the bloated keyframes. It doesn't matter if i enable or disable 4x4 (or 8x8) intra search, nothing will change, x264 is only using 16x16 intra search, the other search options are ignored.

I'm not 100% sure, but I think aku disabled 4x4 and 8x8 as their were rarely used in lossless more, hence were not worth the additionnal computation time. What's more, less blocs types might decrease some sort of overhead in the bitstream, but I'm not sure.

[Manao]

Tommy Carrot, CREXbzh : 16x16 intra were useless, not 4x4 intra. And i'm pretty sure that at least the first revision to have lossless in x264 was using something like 98 % of intra 4x4.

[Shinigami-Sama]

Shinigami-Sama: those encodes are 10000 frames long, a full episode would be about 1.5-2 GB losslessly. :D
damit

[lithoc]

I think loseless mode can't help hd-dvd or blu-ray in term of transfer rate. Loseless easily hog the maximum rate in hd-dvd/bluray.

But it definitely helps in master copy/HDD archiving.
:D

[SeeMoreDigital]

I think loseless mode can't help hd-dvd or blu-ray in term of transfer rate. Loseless easily hog the maximum rate in hd-dvd/bluray.

But it definitely helps in master copy/HDD archiving.
:DIt could also be a great as a professional high-def video camera format!


Cheers

[Shinigami-Sama]

SMD really now..
are you going to as a studio, pay a few thousand dollors for a realy realy cpu intenice camra that you won't realy get full use out of in the next 10years?

[akupenguin]

Do you still see bloat after r321?