Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. Domains: forum.doom9.org / forum.doom9.net / forum.doom9.se |
|
|
#1 | Link |
|
Registered User
Join Date: Nov 2006
Posts: 30
|
How to optimize for speed rather than compression?
Hello,
I'm need to compress a lot of dvb mpeg2 (SD) material (usually at ~4 Mbs), and currently I'm using x264 (crf mode). I'm very satisfied with the result, however, the speed is still too slow and I could accomodate bigger files. I read a lot of intersting discussions here to tweak x264 in order to approach transparency, or to reach good compression level with high image quality. However, in my case I would like to trade a bit the compression level for the speed, at the expense of file size and not (much) at the expense of quality. I don't know if it is possible, or if ASP would be better in that use case. In this thread, http://forum.doom9.org/showthread.php?p=1014786 I have read suggested settings for transparent encoding where people are using subme at level 1 and not analyzing 4x4 partions, which speed up things. Are there other way to improves speed at the expense of compression, but not hurting too much the quality? The sessings I'm currently using are crf 22, b-frames 2, ref 3, deblock -1,-1, qcomp 0.75 and the rest are defaults. Thanks Last edited by pcat; 28th June 2007 at 16:15. |
|
|
|
|
|
#3 | Link |
|
HDConvertToX author
Join Date: Nov 2003
Location: Cesena,Italy
Posts: 6,550
|
i suppose x264 defaults..
try setting --ref 1 and subme 4 , you should gain speed and not loose to much quality BHH
__________________
HDConvertToX: your tool for BD backup MultiX264: The quick gui for x264 AutoMen: The Mencoder GUI AutoWebM: supporting WebM/VP8 |
|
|
|
|
|
#4 | Link |
|
Registered User
Join Date: Nov 2006
Posts: 30
|
Mencoder defaults, which I assume are the x264 defaults (like 0.6 for qcomp, cabac enabled, adaptive b frame allocation enabled, motion estimation algorithm is hexagon radius 2, supel refinement mode 5, etc.)
The command line I use is : mencoder $1 -oac copy -ovc x264 \ -x264encopts crf=22:8x8dct:bframes=2:frameref=3:deblock=-1,-1 \ -o temp.avi ... next is MP4Box stuff for muxing to .mp4 Pierre Last edited by pcat; 29th June 2007 at 09:26. |
|
|
|
|
|
#5 | Link | |
|
Registered User
Join Date: Nov 2006
Posts: 30
|
Quote:
|
|
|
|
|
|
|
#6 | Link |
|
HDConvertToX author
Join Date: Nov 2003
Location: Cesena,Italy
Posts: 6,550
|
one os simpliest guide for x264 are in man page of mencoder (imho)
http://www.mplayerhq.hu/DOCS/HTML-si...n/MPlayer.html and this link in particular http://www.mplayerhq.hu/DOCS/HTML-si...menc-feat-x264 you can resolve all (near) your doubt ![]() BHH
__________________
HDConvertToX: your tool for BD backup MultiX264: The quick gui for x264 AutoMen: The Mencoder GUI AutoWebM: supporting WebM/VP8 |
|
|
|
|
|
#7 | Link | |
|
Registered User
Join Date: Nov 2006
Posts: 30
|
Quote:
More precisely: the doc, as most other infos about x264 settings, says that increasing subq leads to better quality at the expense of encoding speed. What seems strange to me is that knowledgeable people here at doom9 have experimented settings focusing on detail preservation (including noise or film grain) and, to my surprise, subme is at 1. These settings also use other features such as tweaked deadzone parameters and aq-strength / sensivity, but speaking of subme, the best setting in that case seems to be 1, which means to me that a high quality can be obtain without subme >= 6 (as usually suggested). Pierre |
|
|
|
|
|
|
#8 | Link |
|
HDConvertToX author
Join Date: Nov 2003
Location: Cesena,Italy
Posts: 6,550
|
subme 1? i would like to see some test...
could be ok with some source... but with others ? i stay with subme 5 Also.. sorry i haven't read your reply.. i left the tab (of broswer) open.. so i havent a refresh ![]() BHH
__________________
HDConvertToX: your tool for BD backup MultiX264: The quick gui for x264 AutoMen: The Mencoder GUI AutoWebM: supporting WebM/VP8 |
|
|
|
|
|
#9 | Link | |
|
Silver Über Alles
Join Date: Aug 2006
Location: Australia
Posts: 269
|
Quote:
I wouldnt recommend going below subme5 because the size of the file just gets too big. Its better to use the deadzone settings and a decent matrix like the prestige one to keep grain. |
|
|
|
|
|
|
#10 | Link |
|
phjbdpcrjlj2sb3h
Join Date: Sep 2005
Location: Western Australia
Posts: 1,691
|
err, "quality" and "compression" are one and the same. If you are using target bitrate mode, you have quality. If you're talking in terms of target quantizer/ratefactor mode, you have compression.
It's better to talk only in terms of encoder efficiency. Lower submes are less efficient, and I don't believe lower submes help preserve grain. |
|
|
|
|
|
#11 | Link | |
|
Silver Über Alles
Join Date: Aug 2006
Location: Australia
Posts: 269
|
Quote:
|
|
|
|
|
|
|
#12 | Link |
|
phjbdpcrjlj2sb3h
Join Date: Sep 2005
Location: Western Australia
Posts: 1,691
|
Could you please upload a sample and two commandlines that show this? I tried a few sources back last time this was brought up and was truly unable to find any source that this help true for. It just ended up with either more blocking for higher deblock or more mosquito noise for lower deblock.
|
|
|
|
|
|
#13 | Link | |
|
Registered User
Join Date: Nov 2006
Posts: 30
|
Quote:
http://forum.doom9.org/showthread.ph...106#post996106 http://forum.doom9.org/showthread.ph...829#post996829 @R3Z: Thanks for the clarification about subme. So if understand well, lowering subme is a right way to reach my goal : increasing speed without compromizing quality. Compression efficiency will drop, and the file size will be bigger, but I have margin for that. I will experiment to find the right level. Are there other parameters that would influence in the same way (faster as the expense of compression but not visual quality) ? Thanks again! Last edited by pcat; 29th June 2007 at 09:25. |
|
|
|
|
|
|
#14 | Link |
|
Registered User
Join Date: Apr 2007
Posts: 464
|
@pcat
have you read this document : http://www.yuvsoft.com/pdf/x264_para...comparison.pdf ?? Very exhaustive. Even though it dates back to august 2006 (about 100 versions have gone by since then), it identifies the optimal options down to a fault. Personally I find --ref greatly speed & quality, so I am loath to go below 3. You could also set --merange lower than the 16 default. Also, I'd rather set a lower --subme than leave trellis out. cheers audyovydeo |
|
|
|
|
|
#16 | Link | |
|
Silver Über Alles
Join Date: Aug 2006
Location: Australia
Posts: 269
|
Quote:
Be careful with trellis though, it wont help keep grain. That document is great btw. |
|
|
|
|
|
|
#17 | Link | |
|
Registered User
Join Date: Nov 2006
Posts: 30
|
Quote:
However going too slow is a problem because of the rate I'm getting new files : in practice, I have to throw away the recordings which I haven't time to compress, which in a sense means 100% compression at 0% quality !My encodes usually end up between 0.8 GB and 1.2 GB with my current settings, at a quality level I'm happy with. So I have a good margin, and for me "decent compression" just means not going much beyond 1.5 GB. Last edited by pcat; 29th June 2007 at 11:41. |
|
|
|
|
|
|
#18 | Link | |
|
Silver Über Alles
Join Date: Aug 2006
Location: Australia
Posts: 269
|
Quote:
Problem with me is that my recording are of all different resolutions, quality and importance. The sweet spot tends to move around !
|
|
|
|
|
|
|
#19 | Link | |
|
Registered User
Join Date: Apr 2007
Posts: 464
|
Quick ! Someone hammer that line in a slab of marble !!
Quote:
Maybe your best bet is to use --bitrate single-pass. From here : http://en.wikipedia.org/wiki/H264 I see that the "max video bitrate for High Profile" for SD content (which I take to be 720x576@25 as that's my case) is 12.5Mb/s. You could try aiming at 1/3 of that bitrate with a 1pass ABR cmdline, and see what speed / metrics / filesize you get. cheers audyovydeo |
|
|
|
|
|
|
#20 | Link | ||
|
Registered User
Join Date: Nov 2006
Posts: 30
|
![]() Quote:
Quote:
Pierre Last edited by pcat; 29th June 2007 at 13:26. |
||
|
|
|
|
|
#21 | Link |
|
phjbdpcrjlj2sb3h
Join Date: Sep 2005
Location: Western Australia
Posts: 1,691
|
1pass bitrate would be a bad move from crf. It would take the same time, but quality would be far less. The one advantage would be you could predict final filesize, however the quality loss would be significant.
Anyway, the ffmpeg defaults aren't actually the same as the x264 ones last I checked, they were slightly weird in some areas. How much faster do you want the encode to be? My three general suggestions would be: use few bframes, use fewer refs, use faster m.e. algo, but if you can tell us what sort of speed increase you want this will help a lot. I can certainly see a difference between the images, I can certainly not see a large improvement. The comparison which most clearly shows a lack of improvement is the second set. Look at the fuzzy out of focus objects in the background and it is clear that no more grain is being retained from loss of precision in motion estimation. |
|
|
|
|
|
#22 | Link | |
|
Registered User
Join Date: Apr 2007
Posts: 464
|
Quote:
Last week I set about figuring out - the hard way - which ratecontrol method is most efficient. I am still at it, but preliminary results tell me that whatever the ratecontrol method chosen, the SSIM/bitrate pair is mostly the same. Here is a graph for the QP vs CRF test. I ran encodes varying crf from 1 to 39, and QP from 20 to 30. ![]() For each encode I gather the SSIM + kb/s pair. x-axis are the --crf or --qp values. left y-axis is the bitrate in 1000kb/s, right y-axis is ssim. The curves for QP and CRF are literally identical. If you shift the SSIM-QP curve to align it to the other ssim curve, the bitrates also overlap. Had I known, I would also have jotted down the fps values, but time's precious these days ... audyovydeo |
|
|
|
|
|
|
#23 | Link |
|
phjbdpcrjlj2sb3h
Join Date: Sep 2005
Location: Western Australia
Posts: 1,691
|
Neither of those modes is 1 pass ABR
. --qp/--crf and --bitrate are not equivalent at all, because they are constrained by different variables.Also, if you do some quick perceptual comparisons between --qp # and --crf # you should see that they look visually very similar. --crf works by increasing the quantizer where high visual quality is not needed, leading to a significantly smaller file at low to zero quality loss. Last edited by check; 29th June 2007 at 17:53. |
|
|
|
|
|
#25 | Link |
|
WiLD CaRD
Join Date: Mar 2007
Location: Toronto Canada
Posts: 258
|
At the risk of sounding rude, you are defeating the purpose entirely with extensive testing and comparisons. This is how you're going to make things go fast? You want great speed, not the best quality so it should be very easy when you're willing to make that sacrifice.
A quick solution to your problem, among several others, of optimizing speed for a bit less quality/compression is to simply use one of the iPod profiles and encode that way with CRF (NOT 1-pass bitrate which sucks). It's relatively good quality but many of the heavier features such as MRFs, B frames, CABAC, etc are disabled which should give you that speed you need as well in your encoding (and fast decoding too as a bonus). This is what I use when I only want a quick encode of something with decent quality instead of testing a million things. It's actually better than decent quality in my opinion with this profile. I believe you should be testing a million things when it's optimal quality you're trying to acheive, not optimal speed. Last edited by PuzZLeR; 30th June 2007 at 11:07. |
|
|
|
|
|
#26 | Link |
|
x264 Tester
Join Date: Dec 2005
Location: Austria, near Vienna
Posts: 223
|
ah, dammit, once i don't read the whole forum, there's a thread about my settings
![]() @pcat, about subme 1: as i searched the forum, for a more transparent setting, i found out that x264 uses SAD instead of SATD with subme 1. don't ask me what it is, or what it does, i just tested it ![]() at this time, i hadn't tested the adaptive quantization patch, so i worked with subme, trellis and the deadzone switches. but after AQ, my whole life changed ![]() forget about subme, or the deadzones, all you need is AQ. if you want to get a bit more speed, don't use subme 6, or trellis, use max 2 refs, hex search... i got it all wrapped up in this commandline: Code:
--crf 20.0 --level 3 --keyint 100 --min-keyint 1 --ref 2 --no-fast-pskip --bframes 2 --weightb --filter -2,-2 --analyse p8x8,b8x8,i8x8 --8x8dct --vbv-bufsize 1835 --vbv-maxrate 10000 --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "" "" --aq-strength 0.9 keyint 100 increases the bitrate by a minimum, but guarantees a faster seeking. about the levels: i like it if my encodes are level-compliant |
|
|
|
|
|
#28 | Link |
|
x264 Tester
Join Date: Dec 2005
Location: Austria, near Vienna
Posts: 223
|
no, ratetol is a different switch.
i don't know if AQ is implemented in MEncoder... guys do you know more? you could try it without AQ... depends on the source. some movies don't need it at all, some look horrible without it. |
|
|
|
|
|
#30 | Link | |
|
Registered User
Join Date: Apr 2007
Posts: 464
|
Quote:
Mhh, I dont know if you're addressing yourself to pcat or to me. Your comments aren't rude, they're welcome. My previous posting was too quick : I actually tested CRF/vs/QP/vs/2pABR, agree that 1pABR is a different story. Exactly what I'm doing, sir. I didn't even look at encoding speeds in my tests, let alone collect them. |
|
|
|
|
|
|
#31 | Link | |||
|
Registered User
Join Date: Nov 2006
Posts: 30
|
Quote:
I don't want to trade the speed for both the encoding time AND the visual quality. Quote:
Quote:
![]() Pierre |
|||
|
|
|
|
|
#32 | Link |
|
WiLD CaRD
Join Date: Mar 2007
Location: Toronto Canada
Posts: 258
|
Perhaps I was a bit hasty in determining a sacrifice of quality. It's compression you're willing to sacrifice. Although there is a relationship between the two, I do see your point.
However, my point still stands. Theoretically speaking, if you want high speed, high quality and are willing to sacrifice compression, then just disable many key quality/compression features and compensate with higher bitrate. It's that simple really. You get the quality and speed but less compression is the cost. For example, all you really need to do is turn off stuff like MRF, CABAC, B frames/pyramids to 0, or even subme from 5 to 4 and add bitrate. And you don't even need to go through the hassle of calculating this extra bitrate either. This is why I also mentioned CRF because it does exactly that - compensates for you with the exact added bitrate to the quality you want. Just to give you an example of CRF's brilliance in compensation: It is well known that CABAC has about a 10%+ increase in compression over CAVLC (10%+ better quality per file size). However, CABAC slows down encoding speed (and decoding speed too) but to many, myself included, the compression is worth it. But if you're willing to accept less compression - turn CABAC off. As well, use CRF. What will happen, paretibus ceribus, is that CRF will actually give about 10%+ more bitrate for CAVLC encoding in compensation. You get the quality, and speed, but less compression. This will happen for all features you disable for added speed - automatically and accurately adjusted for you. Anyhow, it's fun to play around with options, and this thread's ideas are informative, but from my point of view, this problem is really more simpler than it sounds. Just trying to help. |
|
|
|
|
|
#33 | Link | |
|
x264 developer
Join Date: Sep 2004
Posts: 2,392
|
The encoder optimizes between 3 parameters: quality, bitrate, and speed.
There are some complex interactions between options, but lets ignore them for now. To a first approximation, if you use CRF or CQP mode then the only option that affects quality is --crf or --qp, and all other options tradeoff speed vs bitrate (note: this does not mean that you can measure just the bitrate to determine the effectiveness of some option. I said approximation.). If you use ABR, CBR, or 2pass mode, then the only option that affects bitrate is --bitrate, and all other options tradeoff quality vs speed. I can merge those two statements to become: Ratecontrol options trade quality vs bitrate, and all other options trade speed vs bitrate. So there's two independent decisions to make: What quality you want (determines the value of --crf), and how much cpu-time you're willing to spend (determines everything else). There's more complex interactions here, but I'll ignore them too. To a first approximation, each speed-vs-bitrate option costs a certain amount of time and reduces bitrate by a certain %. You pick some ratio of how much time you're willing to spend to save any given amount of bitrate, and enable all the options whose ROI is better than that. Quote:
CABAC is the option with the best ROI. It has a significant cost in decoding speed, but insignificant cost in encoding speed. (Ok, so there's a 4th parameter to optimize: decoding speed) To be precise, the absolute amount of cpu time used to encode CABAC is the same as to decode CABAC, but the encoder does so much other stuff that it's proportionally less. Some numbers (of course they depend on content, this is just one example): CABAC: 10% bitrate / 2.8% speed = 3.6 roi 8x8dct: 7.4% bitrate / 9% speed = 0.80 roi -t1: 4.8% bitrate / 10.5% speed = 0.45 roi -Anone vs default: 12.5% bitrate / 31% speed = 0.40 roi -m1 vs -m3: 9.1% bitrate / 25% speed = 0.36 roi ... Speed measures the same thing as encoding time, so you can't trade one for the other. Last edited by akupenguin; 15th November 2007 at 10:05. Reason: typo |
|
|
|
|
|
|
#34 | Link | ||
|
WiLD CaRD
Join Date: Mar 2007
Location: Toronto Canada
Posts: 258
|
Hello Akupenguin,
My point is more general and theoretical. My CABAC example was more hypothetical. In fact, let’s just ignore encoding speed for a moment. My REAL point is that it doesn’t matter which options you disable. If you lose quality/file size as a result of these dropped features, you can “get the quality back” with more bitrate in your encoding. Result: Same quality, less compression. Now let’s bring back the speed parameter. Applying this thinking, just disable features that are known to slow down speed. Yes, you may lose quality/file size, but you can always “get the quality back” once more with added bitrate in your encoding. Result: Same quality, faster encoding speed, less compression. Now, to determine this exact extra bitrate necessary is the easiest thing of all. CRF calculates that extra bitrate to great accuracy automatically. In fact, in most cases, one needs only to do a test CRF encode with small clips, with and without the features enabled, and calculate the percentage difference and apply that to the macro picture later, even with ABR encoding. CRF is quite clever in fact, and I thank you for that feature. Well done. Result: Same exact quality, faster encoding speed, less exact compression. So, with my being on the lazier side of testing many different speed increasing options, and only use those proven, and sacrificing compression, as far as I’m concerned, problem solved, and QED. However: Quote:
Quote:
|
||
|
|
|
|
|
#37 | Link |
|
WiLD CaRD
Join Date: Mar 2007
Location: Toronto Canada
Posts: 258
|
It's a financial expression by nature but used in many contexts. A very simplified formula regarding investing in a venture is:
ROI = revenue made/money invested. An ROI of 10 would suggest that you got back $10 for every $1 of your investment. Anytime ROI > 1, you made a profit. The idea is to get the maximum ROI for your investments – to get the most money back for every dollar/euro/whatever that you input/invest. What Akupenguin was referring to was getting the most of your input parameters – choosing the right combination of x264 features that will bring back the best output whether it's best quality/file size, best quality/fastest encoding speed, or whatever necessary to serve your objective in the most optimal way. |
|
|
|
|
|
#39 | Link | |
|
x264 developer
Join Date: Sep 2004
Posts: 2,392
|
Quote:
Right, ROI is not linear, it's exponential: %speed per %bitrate. So I always visualize it with log-log graphs. |
|
|
|
|
|
|
#40 | Link |
|
Registered User
Join Date: Nov 2006
Posts: 30
|
@Akupenguin and PuZzler:
Thanks a lot for the clarification. If I understand well, basically if I'm happy with the quality achieved with CRF 22, I can safely turn off any other feature until I reach good speed with still acceptable file size ? - in particular: the number of reference frames, the number of B-frames, subme, and CABAC which seems to have the most significant impact on encoding speed. @Akupenguin, regarding MEncoder: it does not implement x264 as it just "use" the x264 engine or library, but it does not "exec" a standalone x264 binary and thus doesn't forward the command line arguments "verbatim" to x264 but has to parse them. And I don't know if the AQ patch are included in the x264 used by MEncoder... Thanks again! Pierre |
|
|
|
|
|
#41 | Link |
|
x264 developer
Join Date: Sep 2004
Posts: 2,392
|
Didn't I just say that CABAC has the least significant impact on encoding speed?
OK, so mencoder parses the options. The parsing is simply: options are separated by ':', and an option name is separate from its value by '=' (or if there isn't an '=' then the option is treated as boolean). None of the option names appear in mencoder's source code. Mencoder just passes the strings to libx264 without caring what they mean. The last time I updated mencoder for new x264 options was 2006-07-31 (r542). Several options have been added to x264 since then, but no change in mencoder was required. Likewise, no change in mencoder is required to support aq. Last edited by akupenguin; 2nd July 2007 at 17:50. |
|
|
|
![]() |
|
|