Output file size - MeGui vs. FFmpeg

[teddg]

I've noticed that MeGUI creates smaller x264 output than ffmpeg at the same crf setting. Anyone else notice this?

I ran some tests starting with MeGui's psp profile, crf 26, and translated to ffmpeg, which gave an output twice as large. So I started adding default settings to the ffmpeg command line that shouldn't have made any difference, since they were x264 defaults, but they did make a difference. I even added the "-qmin 10 -qmax 51", which according to the MeGui's mousover hints are disregarded in crf mode. After running through all the options I eventually got ffmpeg's output to ~150% the size of the MeGui output.

The only time ffmpeg's output was the same as MeGui was when I had the typo "-qmin 10 -qmin 51", then they both came out to 22Mb. (tested with updated MeGui no avs filters, and FFMpeg 15394 and 16043).

[Dark Shikari]

I ran some tests starting with MeGui's psp profile, crf 26, and translated to ffmpeg, which gave an output twice as large. So I started adding default settings to the ffmpeg command line that shouldn't have made any difference, since they were x264 defaults, but they did make a difference.Welcome to the world of ffmpeg. ffmpeg uses its own defaults, designed for its mpeg-1/2/4 encoders--defaults which are complete nonsense for x264 and at times in the past have actually caused crashes.

Use a preset with ffmpeg (http://rob.opendot.cl/index.php/useful-stuff/ffmpeg-x264-encoding-guide/), or better yet, don't use it at all.

[teddg]

Welcome to the world of ffmpeg. ffmpeg uses its own defaults, designed for its mpeg-1/2/4 encoders--defaults which are complete nonsense for x264 and at times in the past have actually caused crashes.

Use a preset with ffmpeg (http://rob.opendot.cl/index.php/useful-stuff/ffmpeg-x264-encoding-guide/), or better yet, don't use it at all.

I use MeGui for quality backups, but ffmpeg has its uses too; easy automation, encoding miscellaneous stuff for my ipod, even image dumps and mux/demuxing...

I'm still looking for ffmpeg settings that match x264 output file size at the same crf.

[Dark Shikari]

I'm still looking for ffmpeg settings that match x264 output file size at the same crf.I just gave you a link showing you how to use the presets. Do you need me to open your shell and circle the "libx264-default.ffpreset" file for you? :confused:

[teddg]

I just gave you a link showing you how to use the presets. Do you need me to open your shell and circle the "libx264-default.ffpreset" file for you? :confused:

Dude, I've read those pages. I've used presets.

My question was about why MeGui's x264 output files are way smaller than ffmpeg output files at the same crf and similar settings.

[Dark Shikari]

Dude, I've read those pages. I've used presets.

My question was about why MeGui's x264 output files are way smaller that ffmpeg output files at the same crf and similar settings.Are you using exactly identical settings--with all the x264 defaults properly specified in ffmpeg--and the exact same version of x264 in ffmpeg as in MeGUI?

[teddg]

Are you using exactly identical settings--with all the x264 defaults properly specified in ffmpeg--and the exact same version of x264 in ffmpeg as in MeGUI?


That was exactly what I tried to do, though I don't know how well I did.

Starting with MeGui's psp profile at crf 26, I translated it into ffmpeg cl. The result was wildly different output file size. Then I went through all the settings listed at Swains site and here too (http://ffmpeg.x264.googlepages.com/mapping). Adding a few at a time with default settings. I even tried some commands that are supposedly disregarded in crf mode. In twenty runs the ffmpeg outputs were all about 1.5x (or more) the size of the MeGui outputs, except the typo where they matched.

[Dark Shikari]

That was exactly what I tried to do, though I don't know how well I did.Did you make sure to compile ffmpeg with the same version of x264? If so, did you check the linkages to make sure you didn't accidentally link it to an older libx264 on your system?

[teddg]

Did you make sure to compile ffmpeg with the same version of x264? If so, did you check the linkages to make sure you didn't accidentally link it to an older libx264 on your system?

No, I downloaded a couple of recent binaries for this test. And asked here if other people have noticed the same thing.

[frank]

AFAIK they changed the scaling for CRF since sept. 2008 in x.264.
MeGUI updates it and so no old script workes anymore! Resulting file sizes are larger.
That's why I don't use the new x.264 updates.
CRF 18-22 were the good old values for good quality.

To change the scaling was no good idea.
What would happen if CCE changed it's q param?

[Sharktooth]

...tell the whole story...
scaling didnt directly change, there are some new options (some enabled by default) that increase the bitrate by a factor of "x" to increase quality by a factor of "y", where y > x (by much).
the result is you have to use a higher CRF to hit more or less the same size as before (however, by using CRF you should NEVER care of the filesize... otherwhise use a bitrate mode RC)... and you will gain a major quality increase.

[teddg]

AFAIK they changed the scaling for CRF since sept. 2008 in x.264.

I believe both the ffmpeg versions I used had the updated crf scale.

[Dark Shikari]

AFAIK they changed the scaling for CRF since sept. 2008 in x.264.No, there was no such change.

What happened was ffmpeg was using a broken rceq around that time with x264, resulting in vastly different bitrate scaling as compared to the real x264 binary. I got this fixed, and so the scaling was corrected.

[ScootyPuffJr]

I also noticed that files sizes seem to be getting progressively bigger with the newer x264 builds, but I am not using FFMPEG. I am using the compiled builds from x264.nl.

I originally got my command-line from MeGUI and have just been doing it from scratch ever since, updating the exe, but not the command options. I am in no way claiming that my options are correct, just that the file size seems to change significantly with the newer versions, and the only variable that changes for me is the x264.exe version. Everything else is exactly the same.

x264 options: --level 1.3 --no-cabac --subme 6 --analyse p8x8,b8x8,i4x4,p4x4 --qpmin 16 --vbv-maxrate 768 --me umh --merange 12 --threads 1 --thread-input --progress --no-psnr --no-ssim --crf 21

For File A (42 minue 640x352)AVS Script:
ColorMatrix(hints=true)
LanczosResize(656,370)
Crop(8,8,-8,-10)

x264 build:
[2007-01] version 620: 252MB
[2007-09] version 680C: 255MB
[2008-06] version 883: 274MB
[2008-07] version 920: 263MB
[2008-12] version 1057: 303MB


For File B (42 minute 640x480)AVS Script:
ColorMatrix(hints=true)
AssumeTFF()
Telecide(guide=1,post=2,vthresh=26)
Decimate(cycle=5)
LanczosResize(656,492)
Crop(8,6,-8,-6)

x264 build:
[2007-09] version 680C: 501MB
[2008-12] version 1057: 625MB


That extra size really starts to add up.

Am I doing something wrong in my command-line to make them artificially bigger, or do the new builds just make bigger files because of increased quality or something? As far as quality goes, I really can't tell the difference between the files encoded with the old builds versus the newer builds. They all look great!

Thanks,
-spj

[Sharktooth]

crf 21 and maxrate 768? (well the buffersize is not specified too...)
also why qpmin 16?
however, rise the CRF value until you get more or less the same filesize and compare the visual quality.
if it looks better or even the same, there's nothing going wrong...

[ScootyPuffJr]

Honestly I can't remember why the different paramaters. I spent a lot of time figuring them out last year, but didn't document why I chose them, so I need to go back and figure it out again. I'll be glad to apply any suggested changes you have to the options :)

However, my question is: If I use exactly the same command-line options shouldn't I get roughly the same file size from each version?

Your comment about raising the CRF until I get the same file size would lead me to believe that you have changed (increased) the CRF quality, so that a 20 in build 830 is now considered a 22 in build 1050???

I've always expected the coding changes to increase quality / decrease file size given the same encoding options, so I was surprised to see the file sizes go up. I'd like to keep using the latest version possible, but I also don't want to have to do test files each time I upgrade to make sure my quality/file size remains constant.

Does that make sense?

[Sharktooth]

about filesize: https://forum.doom9.org/showthread.php?p=1233549#post1233549

about the options: it all depends on what you need. all i can say is --vbv-maxrate will be ignored if you dont specify also --vbv-bufsize but with your options, it was good it was ignored since a maxrate of 768 is easily not enough for CRF 21 at 656x370... it depends on the source compressibility though.
also, using 1 pass and force VBV restrictions is not a smart move. it may lead to unwanted results. if you need VBV restrictions use 2 passes.

about avisynth script: always crop BEFORE resizing... so you end up with the desired resolution ;)

[Sagekilla]

There was a revision a while back that changed how crf worked. It ended up increasing the quality for a whole lot more than the bitrate it took, but not all videos were affected. Some stayed at more or less the same size, others jumped up by 20 - 30%. Other than that, there hasn't been many changes that caused a noticeable leap in file size

[ScootyPuffJr]

The link you posted was back to this thread ???

My goal is to encode a standard file that plays back on all Apple Devices (iPod, iPhone and AppleTV). I've always been happy with the results I've gotten with single pass and the apple products don't seem to need the restrictions, so should I just dump the restrictions and go with: x264 options: --level 1.3 --no-cabac --subme 6 --analyse p8x8,b8x8,i4x4,p4x4 --qpmin 16 --me umh --merange 12 --threads 1 --thread-input --progress --no-psnr --no-ssim --crf 21 ... ? I guess I need to relook at --qpmin no matter what.

I tried very hard to figure out how to crop before resizing, but was never comfortable with my ability to maintain the proper aspect ratio. I spent hours trying to figure out a way to systematically do it and keep the proper aspect ratio, while at the same time trimming all the overscan area (I find it distracting when watching). When I can do it easily I do it first, but sometimes it just seemed easier to do it afterwards to maintain the proper aspect ratio. I saw a lot of comments about doing it first, but never really any good explanations as to why, so I guess now is the time to ask :).

If I resize the original source to 16 pixels wider than I want (maintaining the proper aspect ratio) and then trim down to the size I want (still using the proper ratio on the trim, or as close to it as I can get to maintain the 16 pixel block sizes for encoding) why is that worse than trimming first, then resizing? If I take care to maintain the aspect ratio don't I still end up with the same resolution either way?

Thanks Sagekilla, that makes sense. Maybe I just encode more of the older junk that gets bigger :).

[Selur]

little side node: for automation you could also use ffmpeg for decodering and then pipe to x264 for encoding.

[J_Darnley]

The addition of AQ (Mar 08) and psyRD (Sept 08) and the fixing of AQ (Sept 08) are the major factors in the size differences since 2007. If you want to compare sizes with a rev. from '07 then you need to turn off AQ and psyRD.

Sagekilla does point out a significant shift in crf is the fixing of AQ. I felt that the increase in size of my videos was not worth it so I lowered my chosen crf value by 1

There was a revision a while back that changed how crf worked. It ended up increasing the quality for a whole lot more than the bitrate it took, but not all videos were affected. Some stayed at more or less the same size, others jumped up by 20 - 30%. Other than that, there hasn't been many changes that caused a noticeable leap in file size

Actually, that particular revision changed how AQ works.
commit 173b40c276ee187de31ef630fd207a62de1b2cc7
Author: Jason Garrett-Glaser <darkshikari@gmail.com>
Date: Sat Sep 13 14:03:12 2008 -0700

Move adaptive quantization to before ratecontrol, eliminate qcomp bias
This change improves VBV accuracy and improves bit distribution in CRF and 2pass.
Instead of being applied after ratecontrol, AQ becomes part of the complexity measure that ratecontrol uses.
This allows for modularity for changes to AQ; a new AQ algorithm can be introduced simply by introducing a new aq_mode and a corresponding if in adaptive_quant_frame.
This also allows quantizer field smoothing, since quantizers are calculated beofrehand rather during encoding.
Since there is no more reason for it, aq_mode 1 is removed. The new mode 1 is in a sense a merger of the old modes 1 and 2.
WARNING: This change redefines CRF when using AQ, so output bitrate for a given CRF may be significantly different from before this change!

P.S. Dark Shikari, there is a spelling error in that text: s/beofrehand/beforehand/ Do you feel it is too late to change history?

[vmrsss]

Hello everybody.

Interesting thread. I found that the recent introductions of AQ and PSY and their successive refinements made a tremendous difference, and raised the perceived quality of encodings quite a bit. It is a fact that the bitrate has increased quite substantially, yet I generally find the increase worthwhile, and only now and then find myself fighting a bitrate higher than I wished.

Typically, I encode SD PAL DVB and my standard command line is:


x264 stream.y4m --pass 1 --threads auto --crf 21 --direct-8x8 -1 --8x8dct --b-adapt 2 --bframes 3
--weightb --mixed-refs --b-pyramid --me hex --subme 7 --ref 6 --direct auto --trellis 1 --level 31
--progress --stats x264OUT.log --output x264OUT.264


(depending on how I feel, I add --deblock -2:-1 --psy-rd 0.7:0.0, not so infrequently). This is my compromise between speed and quality (eg I regretfully renounce to --me umh and higher --merange and to --subme 9 to gain some 6-8fps, not too mention --trellis 2).

Typically, with recent x264s, this gives me bitrates ranging from 1150kb to 1500kb for encodings of size 720x576 or 704x576 (SAR 64:45), which I then watch back on a HD TV with essentially no perceived loss of quality from the originals. Very pleased. When I want smaller files (eg for contents like documentaries where the video stream is not particularly exciting), I scale down to 704x400 or 640x368 (and essentially square SAR) and use --crf 24 again with pleasing results.

I accept that people who mainly watch their encodes on smaller screens, computers or TVs, may find the recent increase in bitrates not worth the quality gain but, after some tests, my experience remains that again I prefer to run at --crf 24 as indicated above (and possibly add a bit of --nr, but very rarely because I don't really like the blurred effect), rather than disable AQ and PSY.

In any case, in answer to some of the previous postings, I have observed no difference between ffmpeg+libx264 and x264 (although I preferentially use the latter) once you have made sure you're using the same parameters.

As I said, occasionally I bump into a nasty piece of video which would not compress: at the moment I am struggling with some mpeg2video which also at --crf 22 (where at 704x576 I can start to notice a degradation from the the original) yields a bitrate of 2200kb and a size reduction of about 50% wrt the original (climbing to almost 2900kb and about 35% with --crf 21), which somehow I feel I should be able to improve upon.

Has anybody got any suggestions about what to attempt in cases like that? Or is that as good a result as I can hope for?

Thanks

[Sagekilla]

Easy solution? Bump up to --trellis 2, --me umh, and --subme 9. That can give you a bit more compression. Otherwise, try doing (light) denoising in avisynth to see if it can help any.

[vmrsss]

Easy solution? Bump up to --trellis 2, --me umh, and --subme 9. That can give you a bit more compression.


In fact, --trellis 2 and --subme 9 raise my bitrate at a fixed crf --which I can't really explain--, while umh produces very small savings.


Otherwise, try doing (light) denoising in avisynth to see if it can help any.

Ah!, avisynth: that is really a tool I long for. But I am on Mac, and unfortunately, it is not available. I've tried to run avisynth through wine (actually Darwine), which would work, but then I don't know how to compile x264 with AVS support on my architecture. Does anybody know?

[Sagekilla]

The thing is, you have to look at the quality level and bitrate at the same time, not just bitrate. If turning on --trellis 2 and --subme 9 increased bitrate by 10%, but improved quality by 15% (just some numbers) then it would probably be a good idea to raise crf a bit so that you get back to the original bitrate but at better quality (or lower bitrate, for the same quality).

You always have to look at quality vs bitrate when deciding whether turning on an option really reduced bitrate for the same quality (or increased for an even larger increase in quality) ;)

[nm]

Ah!, avisynth: that is really a tool I long for. But I am on Mac, and unfortunately, it is not available. I've tried to run avisynth through wine (actually Darwine), which would work, but then I don't know how to compile x264 with AVS support on my architecture. Does anybody know?
You could try Avs2YUV (http://akuvian.org/src/avisynth/avs2yuv/) to pipe to a native x264 (compiled without AviSynth support).

[ScootyPuffJr]

Thanks for all the information and suggestions guys. I see it's now time for me to go back and refresh myself with all of the options and do another round of testing. Thankfully there are now profiles for every target I'm looking for, so it will make things a lot easier.

... someone PLEASE convince me I don't need to go back and recode everything I've done over the last 18 months with my junk command line options :) ...

[roozhou]

You could try Avs2YUV (http://akuvian.org/src/avisynth/avs2yuv/) to pipe to a native x264 (compiled without AviSynth support).

You can also use mplayer and piping y4m to x264. That is a much easier and faster way.

[nm]

You can also use mplayer and piping y4m to x264. That is a much easier and faster way.
vmrss was specifically asking how to use AviSynth. One might be able to use a win32, AviSynth-enabled build of MPlayer to replace Avs2YUV, but that's hardly easier.

And for MPlayer's filtering, why not just use x264 through MEncoder instead of piping.

[Selur]

And for MPlayer's filtering, why not just use x264 through MEncoder instead of piping.
x264 is easier to build and so easier to replace and it's for more often updated.

Cu Selur

[Esurnir]

Thanks for all the information and suggestions guys. I see it's now time for me to go back and refresh myself with all of the options and do another round of testing. Thankfully there are now profiles for every target I'm looking for, so it will make things a lot easier.

... someone PLEASE convince me I don't need to go back and recode everything I've done over the last 18 months with my junk command line options :) ...

If it looks good why bother ? The only persons your going to impress with higher settings are those loading the movie in media info (wow me tesa and subme 9, some people got no life...) or people that paid hundreds of bucks for elecard streameye that will marvel at your ingenuity.

If it look good leave it that way, unless it was too big a file and you need that space for something else. If it look bad, well maybe it's time to reencode it ^_^.

[nm]

x264 is easier to build and so easier to replace and it's for more often updated.
MPlayer and FFmpeg SVN repositories might disagree on that last point, but of course, x264 is more important when the main task is encoding H.264.

MEncoder can also be linked with a shared libx264, so you only need to rebuild MEncoder when the x264 ABI changes. Also, building MEncoder is quite easy once you have the dependencies installed.

[ScootyPuffJr]

If it look good leave it that way, unless it was too big a file and you need that space for something else. If it look bad, well maybe it's time to reencode it ^_^.

The video looks good, and the size is acceptable. My only worry was that there was something terribly wrong with the options that would cause an obvious issue in playing it on other platforms. I'm targeting the Apple devices, but if changing a couple of settings would make it play anywhere even better.

Thanks!

[ScootyPuffJr]

crf 21 and maxrate 768? (well the buffersize is not specified too...)
also why qpmin 16?QUOTE]

qpmin 16 is the setting specified in all of the MeGUI Apple Device Profiles.

I have dumped a bunch of the MeGUI profiles into a spreadsheet and am comparing them to the documentation at http://mewiki.project357.com/wiki/X264_Settings, which so far has been the most comprehensive site on options I can find. If there is a different/better source I should be looking at please let me know.

There seem to be a number of differences between the reccomendations on the site and the settings in the profiles which I'm researching in the forums now. [Updated] Apple products don't support a B-Frames, CABAC, high reference frames, etc...

[QUOTE=teddg;1233443]I even added the "-qmin 10 -qmax 51", which according to the MeGui's mousover hints are disregarded in crf mode.
Is this true? are qmin and qmax ignored in crf mode? If so why would the MeGui profiles speficy them?

[J_Darnley]

qpmin and qpmax are both respected by x264 in both bitrate and crf rate control modes.

[vmrsss]

vmrss was specifically asking how to use AviSynth.

Yes, thanks for the suggestion, it seems to work well (but I have no experience with avisynth, so I could only try the most elementary examples).

One might be able to use a win32, AviSynth-enabled build of MPlayer to replace Avs2YUV, but that's hardly easier.

No, the point is that wine is very slow, you don't want to use it for computationally intensive task. An alternative which it occurs to me may work too is to run Avisynth from an emulator like Parallels/VMWare and then avs2yuv or similar (run again from inside the virtual machine) to write a y4m pipe on the native filesystem to x264 (run from the native OS I expect this to be quicker: Parallels is better than wine, yet it still runs x264 as it were on a single core).

And for MPlayer's filtering, why not just use x264 through MEncoder instead of piping.

Because MPlayer's encoding is disastrous, only the avi container actually works, and it's got serious problems with AV sync. FFmpeg is way, way better, but then again it's got no filters... On top of that, I've got the impression that avisynth filters are definitely more refined than MPlayer's, but indeed I have never looked at them seriously.

[Dark Shikari]

No, the point is that wine is very slow, you don't want to use it for computationally intensive task.Have you actually tried running avs2yuv under Wine?

According to akupenguin, Wine runs Avisynth better than Windows ;)

[vmrsss]

x264 is easier to build and so easier to replace and it's for more often updated.

Well, MPlayer and FFmpeg use libx264 as a dynamic lib, so in this sense they are updated as often as x264 is... It is however the case that some of the advanced parameters in x264 (--psy-rd, --aq-strength, --b-adapt 2, ...) are not configurable from them (parameters in FFmpeg are in general particularly messy).

Apart from that, at the moment FFmpeg is very attentive to the developments in x264 --thanks to our own DS himself-- while I cannot say the same for MPlayer.

Finally, x264 is indeed easier to compile, and runs marginally faster than FFmpeg and MPlayer, even more.

[Dark Shikari]

Well, MPlayer and FFmpeg use libx264 as a dynamic lib, so in this sense they are updated as often as x264 is... It is however the case that some of the advanced parameters in x264 (--psy-rd, --aq-strength, --b-adapt 2, ...) are not configurable from them (parameters in FFmpeg are in general particularly messy).Mencoder passes commandlines directly to x264, so you can access all of them.

You can access b-adapt 2 from ffmpeg with -b_strategy 2.

[vmrsss]

Have you actually tried running avs2yuv under Wine?

yes, and it was certainly serving frames 5 times quicker than my typical encodes with x264, but as I said I can't yet report authoritatively on speed, because I have only tried very silly examples. I'll be back if this interest people.

[Dark Shikari]

yes, and it was certainly serving frames 5 times quicker than my typical encodes with x264, but as I said I can't yet report authoritatively on speed, because I have only tried very silly examples. I'll be back if this interest people.I highly doubt wine is slower than Windows in this case (at least significantly so), as it runs all the code natively--the only thing emulated is system calls and such, which most encoders, decoders, and Avisynth filters probably make very little use of.

[vmrsss]

Mencoder passes commandlines directly to x264, so you can access all of them.

Indeed, thx about this.

You can access b-adapt 2 from ffmpeg with -b_strategy 2.

Have you tried it personally? The fact that -b_strategy 1 and -b_strategy 2 encode at the same speed, seems to indicate there is something not quite right with this...

[Dark Shikari]

Indeed, thx about this.



Have you tried it personally? The fact that -b_strategy 1 and -b_strategy 2 encode at the same speed, seems to indicate there is something not quite right with this...Well...x4->params.i_bframe_adaptive = avctx->b_frame_strategy;

[nm]

Because MPlayer's encoding is disastrous, only the avi container actually works, and it's got serious problems with AV sync.
Personally I have never had that much problems with MEncoder, but sure, keeping sync may be difficult with VFR sources and IVTC filters. However, I was only arguing against the idea of piping video from MPlayer instead of encoding directly with MEncoder. Possible A/V sync problems are pretty much the same for both approaches.

Concerning AVI, when encoding with -ovc x264 you are supposed to also use -of rawvideo to output a H.264 elementary stream.