x264 Lossless question

[stpdrgstr]

Hi, good day.

I was wondering if is safe to use x264 lossless, and is comparable to Lagarith.

I mean, if I reencode this x264 lossless to x264 lossy, it's the same as to reencode a Lagarith to a x264 lossy?

And, should I be using CoreAVC or libavcodec to decode this lossless through DSS2()?

These question had been retaining my sleep these days, so I would be very gratefull if they could be answered :)

Thanks.

[Dark Shikari]

x264 lossless has pretty much strictly superior compression to Lagarith, even in intra-only mode IIRC.

Unlike Lagarith, it only supports the YV12 colorspace.

To decode it correctly, you need a libavcodec from within the past ~2-3 weeks (as that's when support was added) or CoreAVC 1.8.5.

[Atak_Snajpera]

How to encode to lossless?

[J_Darnley]

Atak_Snajpera, huh? --qp 0 is that what you mean?

stpdrgstr, lossless is lossless, the input to "x264 lossy" in both cases should be the same.

[Atak_Snajpera]

Huh? --qp 0 is that what you mean?
yep! I meant switch. Thanks

[stpdrgstr]

Thank you very much. Now I can keep encoding at ease :)

[LoRd_MuldeR]

Note that for frame-accurate access to H.264 streams (lossless or not) you should use DGAVCIndex/AVCSource() instead of DirectShow.

I'm not sure whether AVCSource() already has an updated libavcodec with support for x264 lossless mode though...

[Dark Shikari]

Note that for frame-accurate access to H.264 streams (lossless or not) you should use DGAVCIndex/AVCSource() instead of DirectShow.Frame-accurate access should never be an issue with lossless regardless of the decoder being used... since you shouldn't be using B-frames with lossless, so there is no frame delay.

[LoRd_MuldeR]

you shouldn't be using B-frames with lossless

Didn't know that. Why is it like that? And does x264 automatically set "--bframe 0" when "--qp 0" is used?

(Could check the sources, but I'm too lazy right now ^^)

[Dark Shikari]

Didn't know that. Why is it like that?It always reduces compression.And does x264 automatically set "--bframe 0" when "--qp 0" is used?No, because x264 isn't designed to protect you from your own stupidity.

A warning might be useful though.

[LoRd_MuldeR]

It always reduces compression.

With "always" you mean "B-Frames always reduce compression in lossless mode". Not in general. Right?

No, because x264 isn't designed to protect you from your own stupidity.

:D

[Dark Shikari]

With "always" you mean "B-Frames always reduce compression in lossless mode". Not in general. Right?Yes, hence the "lossless" in the thread title.

[kemuri-_9]

i thought the h.264 spec declared that predictive lossless profile was to only consist of IDR frames?

Edit:
nope, confirmed against the spec... it does allow I,P,B.

what's the point of allowing B frames in the first place if it reduces the compression?

[LoRd_MuldeR]

I created streams with x264's mode "predictive lossless" mode using P-Frames and even B-Frames (didn't know b-frames are bad for lossless) and CoreAVC as well as recent ffdshow do handle it properly.

[Dark Shikari]

what's the point of allowing B frames in the first place if it reduces the compression?What is the point of prohibiting them if it requires an extra line of text in the spec? :devil:

(Furthermore, one theoretically could improve lossless compression by doing the "every frame is a B-frame" trick and then the "duplicate frames in both reference lists", because that would allow for 8thpel MC, which would help lossless.)

[LoRd_MuldeR]

What is the point of prohibiting them if it requires an extra line of text in the spec? :devil:

Reduce the number of cases a decoder needs to be prepared for ...

[Dark Shikari]

Reduce the number of cases a decoder needs to be prepared for ...What, by making things even more complicated by not making High 4:4:4 Predictive profile a strict superset of High Profile?

[vmrsss]

so what's a sensible command line for lossless?

[Dark Shikari]

so what's a sensible command line for lossless?Ultra-fast: --subme 1 --no-cabac --partitions none --me dia

Normal: --subme 5 --partitions all

Slow: --subme 6 --8x8dct --partitions all --ref 2

Slower: --subme 8 --8x8dct --partitions all --ref 4 --me umh --mixed-refs

Slowest: --subme 8 --8x8dct --partitions all --ref 16 --me esa --mixed-refs

Note that no-cabac loses about 10-15% compression but massively speeds up playback. Other things that hurt compression the most are subme=0 (vs other values) and partitions=none.

[vucloutr]

no --subme 9 for Slowest ?
typo or is there actually something to it ?

[Dark Shikari]

no --subme 9 for Slowest ?
typo or is there actually something to it ?No B-frames, so it isn't as if there's any difference between 8 and 9...

[Daiz]

I've made some of my own tests regarding various lossless video formats, and found out that not only does lossless H.264 compress the best, it also provides the fastest playback speeds.

In the test I made a while back, I encoded the same 2000-frame clip from a DVD source to various lossless formats and then made a 2-pass encode with x264 from the various lossless files using simple AviSynth inputs. Processor used was Intel C2Q Q6600 @ 2.4 GHz.

These were the x264 encoding settings used for the lossless -> lossy encode:
x264.exe --pass 1 --stats ".stats" --progress -B 768 -b 16 -r 1 --me dia --progress --subme 1 --no-dct-decimate --partitions none --threads 6
x264.exe --pass 2 --stats ".stats" --progress -B 768 -b 16 -r 6 --mixed-refs --b-pyramid --weightb --me umh --subme 7 --merange 24 --bime --b-rdo -A all --8x8dct --threads 6 --trellis 2 --aq-strength 1.0

And these were the results:
Lossless H.264 (encoded with --no-cabac) - average 47.587 fps - DSS2() with CoreAVC
1st pass: 143.07 fps, 790.41 kb/s
2nd pass: 28.54 fps, 778.01 kb/s

Lossless H.264 (encoded with --no-cabac) - average 47.014 fps - DSS2() with CCCP 2008-01-24's ffdshow
1st pass: 136.90 fps, 790.41 kb/s
2nd pass: 28.38 fps, 778.01 kb/s

Lossless H.264 (encoded with --no-cabac) - average 45.901 fps - FFMPEGSource()
1st pass: 133.39 fps, 790.41 kb/s
2nd pass: 27.72 fps, 778.01 kb/s

Lossless H.264 (encoded with --no-cabac) - average 45.250 fps - DirectShowSource() with CoreAVC
1st pass: 128.94 fps, 790.41 kb/s
2nd pass: 27.44 fps, 778.01 kb/s

FFVHuffy - average 42.722 fps - AVISource()
1st pass: 109.81 fps, 789.82 kb/s
2nd pass: 26.52 fps, 777.78 kb/s

Lossless H.264 - average 40.713 fps - DirectShowSource() with CoreAVC
1st pass: 92.26 fps, 793.68 kb/s
2nd pass: 26.12 fps, 778.26 kb/s

HuffYUV - average 27.449 fps - AVISource()
1st pass: 35.19 fps, 792.69 kb/s
2nd pass: 22.50 fps, 779.06 kb/s

Lagarith - average 15.130 fps - AVISource()
1st pass: 12.61 fps, 792.69 kb/s
2nd pass: 18.91 fps, 779.06 kb/s

I've been using x264 for my lossless needs ever since :p

Edit: Fixed the averages.

[Dark Shikari]

Note the proper way to average "20fps" and "100fps" is to do 2/(((1/20)+(1/100))), not just average the FPS (since the computer spends more time doing the 20fps encoding than the 100fps, so the average has to be weighted).

[Selur]

Just to be on the save side I ask: (otherwise I know there would be a thread in a not to distant future where Dark Shikari writes something like: "How people come to such assumtions ..."

Are there any other options aside form b-frame related once and "-nr" that should be (need to be) disabled when aiming for a lossless encode?

Cu Selur

[Dark Shikari]

Just to be on the save side I ask: (otherwise I know there would be a thread in a not to distant future where Dark Shikari writes something like: "How people come to such assumtions ..."

Are there any other options aside form b-frame related once and "-nr" that should be (need to be) disabled when aiming for a lossless encode?

Cu SelurNoise reduction is forced off during lossless encoding, for obvious reasons.

[LoRd_MuldeR]

Lossless H.264 (encoded with --no-cabac) - average 47.014 fps - DSS2() with CCCP 2008-01-24's ffdshow
1st pass: 136.90 fps, 790.41 kb/s
2nd pass: 28.38 fps, 778.01 kb/s

If you used such an old version of ffdshow to decode your lossless H.264 source, that source obviously was encoded was a very old version of x264 :eek:

That's because x264 uses the "predictive lossless" profile for quite some time now. And decoding support for that profile was added to libavcdodec/ffdshow only recently!

So I recommend to repeat your test with up-to-date x264 (http://x264.nl/) and up-to-date ffdshow-tryout (http://ffdshow-tryout.sourceforge.net/download.php) ;)

[Daiz]

Yeah, I made the test over 7 months ago. I doubt the situation has changed so that some other lossless format would beat lossless CAVLC H.264 though, which is the reason why I haven't bothered to redo the test yet.

[Selur]

Noise reduction is forced off during lossless encoding, for obvious reasons.
What about custom quant matrices and zone options, will these be ignored automatically or too forced off ? (I assume both can mess up a lossless encode) :)

Cu Selur

[Dark Shikari]

What about custom quant matrices and zone options, will these be ignored automatically or too forced off ? (I assume both can mess up a lossless encode) :)

Cu SelurZone options can be used, I don't see why not (though bitrate weighting won't do anything). CQMs are of course forced off.

[Selur]

Thanks, good to know, just wanted to make sure since I previously believed that b-frame could also be used with lossless and or would be automatically disabled.
(Thinks like "No, because x264 isn't designed to protect you from your own stupidity." really can cause some scare about assumptions one has about x264. ;))

Cu Selur

[Dark Shikari]

Thanks, good to know, just wanted to make sure since I previously believed that b-frame could also be used with lossless and or would be automatically disabled.
(Thinks like "No, because x264 isn't designed to protect you from your own stupidity." really can cause some scare about assumptions one has about x264. ;))

Cu SelurThough it doesn't protect you from your own stupidity, x264 doesn't lie to you either (allow the use of options that would make lossless not lossless).

[Selur]

... x264 doesn't lie to you either (allow the use of options that would make lossless not lossless).
okay, reread the statements form above, so enabling b-frames would only hurt compression but not break losslessness. :)

Cu Selur

[LoRd_MuldeR]

Yeah, I made the test over 7 months ago. I doubt the situation has changed so that some other lossless format would beat lossless CAVLC H.264 though, which is the reason why I haven't bothered to redo the test yet.

The situation has changed in a way that x264 now uses the "predictive lossless" profile, which significantly improves the compression in lossless mode.

I got no idea how that effects decoding speed though...

[Dark Shikari]

The situation has changed in a way that x264 now uses the "predictive lossless" profile, which significantly improves the compression in lossless mode.

I got no idea how that effects decoding speed though...Better compression == faster decoding speed, at least with CABAC.

[Sagekilla]

using prediction -may- be faster than not at all. To look at it one way, you'll end up with a smaller bitstream --> less to decode is always faster. Not sure if it'll offset the speed lost from doing MC etc.

[Daiz]

The situation has changed in a way that x264 now uses the "predictive lossless" profile, which significantly improves the compression in lossless mode.

I got no idea how that effects decoding speed though...

I did say that the situation haven't changed in the sense that Lossless H.264 is still the fastest choice for lossless out there, which is why I haven't bothered to redo the test (as I doubt other lossless formats would have suddenly become a lot faster than lossless H.264, especially since lossless H.264 encoded with x264 is even faster than before).

[LoRd_MuldeR]

I did say that the situation haven't changed in the sense that Lossless H.264 is still the fastest choice for lossless out there, which is why I haven't bothered to redo the test (as I doubt other lossless formats would have suddenly become a lot faster than lossless H.264, especially since lossless H.264 encoded with x264 is even faster than before).

I understood that part. However x264 lossless should compress even better and decode even faster now ;)

So you may consider updating your numbers...

[Aktan]

I did some lossless encodes with B-Frames for the heck of it and I've noticed that seemly the B-Frames aren't lossless. Maybe I'm doing something wrong.

Here is my source avs:


# Set DAR in encoder to 9113 : 5000. The following line is for automatic signalling
global MeGUI_darx = 9113
global MeGUI_dary = 5000
DGDecode_mpeg2source("C:\Temp DVD\VTS_01_PGC_01_1.d2v",info=3)
ColorMatrix(hints=true)
#deinterlace
#crop
#resize
#denoise


Here are my x264 settings according to MeGUI:


program --qp 0 --keyint 999999 --min-keyint 0 --ref 16 --mixed-refs --no-fast-pskip --bframes 16 --b-adapt 2 --b-pyramid --weightb --direct auto --nf --subme 9 --trellis 2 --partitions all --8x8dct --me tesa --threads 2 --thread-input --aud --progress --no-psnr --no-ssim --output "output" "input"


here is my Compare script avs:


loadplugin("avss.dll")
Compare(DGDecode_mpeg2source("C:\Temp DVD\VTS_01_PGC_01_1.d2v",info=3).ColorMatrix(hints=true),dss2("VTS_01_3.mp4"))


The version of x264 I'm using according to MeGUI is: 1069 - Skystrife's patched build

I don't think it is my Compare script because if I turn off B-Frames, the compare script says it's lossless. With B-Frames on, seemly, though I have not confirmed, that the frames from the B-Frames are the lossy one.

Any ideas?

[Dark Shikari]

Are we sure that this isn't a bug in the decoder?

Use --dump-yuv to dump x264's internal representation of the video and compare that with the decoded video. Also, try using CoreAVC instead of FFDshow (or vice versa).

Also, try one by one removing B-frame options (weightb, pyramid, direct auto, etc).

[Aktan]

I was using ffdshow tryouts revision 2583 built on Jan 4th 2009.

I'll try your suggesions Dark, thanks.

[Aktan]

After turning off some options, I found that just turning off B-Frame mode (--direct none) has fixed the problem.

I'm not sure how to use --dump-yuv as the x264.exe doesn't even list it as an option. Can you give an example Dark?

I have not tried decoding with CoreAVC yet.

I will try with no --direct parameter (Spatial) and --direct temporal.

[J_Darnley]

Sure it does: --dump-yuv <string> Save reconstructed frames

[Aktan]

oops, it was in longhelp.. I am still unsure how to use it, I tried something like this and got an error:

x264.exe --dump-yuv "C:\Temp DVD\Test.264" "C:\Temp DVD\VTS_01.avs"
x264 [error]: No output file. Run x264 --help for a list of options.

Edit:

Nevermind I think I know what I did wrong... which makes sense now

[J_Darnley]

It is no subsitute for normal output. Use x264 as normal and add the dump-yuv option. Use NUL for the output file if you don't want one.

[Sagekilla]

x264.exe "C:\Temp DVD\VTS_01.avs" --output "NUL" --dump-yuv "C:\Temp DVD\yuv_dump.yuv"

I'm not sure what extension is the proper one to use for a raw yv12 file but that's trivial.

[Dark Shikari]

If direct none fixes it, then that means direct is broken (either encoder or decoder side).

Try setting to "spatial" just to make sure it isn't temporal that's broken.

[Aktan]

Okay, a few more tests:

direct = 1 (Spatial) (aka no --direct parameter specified at all) has no problems

direct = 2 (Temporal) (aka --direct temporal) has no problems

Old results:

direct = 0 (None) (aka --direct none) has no problems

direct = 3 (Auto) (aka --direct auto) has problems

I tried using CoreAVC decoder but it is worse off. My clip that was not encoded with B-Frames outputted with CoreAVC is not lossless in certain areas while outputting with FFDShow was. I'm not sure if it is due to a setting in CoreAVC, but I didn't think so since parts of the clip was outputted the same as the source.

The version of CoreAVC is 1.8.5 (Trial Edition)

[Dark Shikari]

In the case where it "has problems", does --dump-yuv's stream have problems, or is it exactly correct?

[Dark Shikari]

You're right, direct auto is bugged, the code as written assumes non-lossless.

Solution coming soon.

Edit: solution applied.

[Aktan]

Cool thanks Dark!

Well I did the YUV Dump anyway and got a big file (which is obvious) but I have no idea how to load it into AviSynth, sorry for the OT, but any ideas?