Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 29th July 2014, 14:14   #1  |  Link
friolator
Registered User
 
Join Date: Jul 2014
Posts: 10
x264: excessive memory usage with --seek

I'm trying to use the --seek and --frames options in x264 to hone in on the right settings for a section of video that's giving me some trouble. In doing so, I'm wondering if I've found a bug.

Basically, whenever I use --seek with a high starting frame number, the encoder takes quite a while to actually begin encoding (minutes). During that time memory usage steadily increases until it's at 99% usage. Then it levels off for a bit until it hits the target frame range, the memory usage drops slightly as the encode begins. When it's finished, it instantly drops back down to normal levels.

Here's the simplest version of the command I'm using that will reproduce this:
Code:
C:\Users\Owner>D:\x264_Testing\x264-r2453-ea0ca51.exe  --bitrate 23000  --seek 10000  --frames 500  -o output.avc  source.mov
At a 5000 frame starting point, the time to begin encoding is much shorter. At 20,000, it's much longer, and memory usage levels off at just under 8GB for quite some time, until it begins encoding.

My system is a core i7 2600K, 8GB RAM, Windows 7 64bit. However, on a laptop I'm using for testing (also Win7), I see similar behavior though I'm using smaller test files there.

The source Quicktime is a ProRes 422 HQ 1080p/23.98 file (91 minute feature film), but on the laptop I'm testing with an uncompressed AVI file at 1080p/2398, so I don't think the Quicktime file is the problem.

When I run the encode without --seek, the encode starts almost immediately, memory usage hovers at about 1.4-1.5GB (that's total system physical memory usage)

And I guess the question is - with --seek, why doesn't it start almost instantly, just at the target start frame (as it does when I don't use --seek)?
friolator is offline   Reply With Quote
Old 29th July 2014, 20:43   #2  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 4,406
I ran into this same issue. When using uncompressed v210 in AVI and seeking 100000+ frames my memory would actually fill and the process would be killed by Windows. I have 32GB of RAM and was using 64-bit x264 so the memory usage was quite high.

I ended up piping from AVISynth instead of using --seek.
Asmodian is offline   Reply With Quote
Old 29th July 2014, 20:52   #3  |  Link
MasterNobody
Registered User
 
Join Date: Jul 2007
Posts: 552
Can you specify what demuxer was used when you experienced this problem. ffms/lavf/avs? Also can you try to use another demuxer (with --demuxer option) to see if this is specific to one of the demuxers or all. As to why encoding doesn't start immediately is also depends from used demuxer and for some of them it need full decoding from first frame to specified with --seek to be frame accurate.
MasterNobody is offline   Reply With Quote
Old 29th July 2014, 22:05   #4  |  Link
friolator
Registered User
 
Join Date: Jul 2014
Posts: 10
Quote:
Originally Posted by MasterNobody View Post
Can you specify what demuxer was used when you experienced this problem. ffms/lavf/avs?
lavf is what it uses by default so that's what we've been using. I'll look into using one of the others and post my results.
friolator is offline   Reply With Quote
Old 29th July 2014, 22:07   #5  |  Link
friolator
Registered User
 
Join Date: Jul 2014
Posts: 10
Quote:
Originally Posted by Asmodian View Post
I ended up piping from AVISynth instead of using --seek.
Thanks - hoping to avoid that if we can help it, because we may end up running this on Mac or Linux in the near future.
friolator is offline   Reply With Quote
Old 29th July 2014, 22:16   #6  |  Link
friolator
Registered User
 
Join Date: Jul 2014
Posts: 10
Quote:
Originally Posted by friolator View Post
lavf is what it uses by default so that's what we've been using. I'll look into using one of the others and post my results.
...and it appears that the binary we downloaded (July 21 build at videolan web site: r2453) isn't compiled with ffms, so lavf is our only option at the moment.
friolator is offline   Reply With Quote
Old 30th July 2014, 20:36   #7  |  Link
friolator
Registered User
 
Join Date: Jul 2014
Posts: 10
Ok, so we're back up and running, piping the video through Avisynth. However, in order to open the Quicktime we still need to use ffms. After much research, it seems that the best results are to use the following avisynth script to open the file:

Quote:
Download the modified FFMS2 version of SAPikachu (here: http://nmm.me/vg). Use AviSynth 2.6 and open the video with:
Code:

FFVideoSource("video.mov",enable10bithack=true) #10bit in stacked format
f3kdb_dither() #flash3kyuu_deband, you can also use DitherTools
#You are in YV16 colorspace now.
(from here)

This is great because it also gets us good dithering from 10bit source - but to load up a feature film it takes forever to index, and since we'll only ever use x264 with ProRes or Uncompressed Quicktime files (never GOP-based source files), is there an alternative that gets you the same result as above, but without the wait for an index file? I can see the need to create an index file for GOP based source files before you can do any trimming, but I don't see why it's necessary for our situation, using frame-based formats like Uncompressed Quicktime.

According to the thread above, using QTSource probably isn't a good idea because it lacks proper 10->8bit dithering, which we need.

Thanks!
friolator is offline   Reply With Quote
Old 30th July 2014, 21:12   #8  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
Quote:
Originally Posted by friolator View Post
is there an alternative that gets you the same result as above, but without the wait for an index file?
lsmashvideosource("input.mov", stacked=true, format="YUV422P16")
f3kdb_dither()
sneaker_ger is offline   Reply With Quote
Old 30th July 2014, 23:19   #9  |  Link
friolator
Registered User
 
Join Date: Jul 2014
Posts: 10
Quote:
Originally Posted by sneaker_ger View Post
lsmashvideosource("input.mov", stacked=true, format="YUV422P16")
Thanks - it works, though we have to use format="YUV422P8" instead (YV16) for ProRes, at least. Haven't tested anything else with it yet. We just get garbage with YUV422P16 using ProRes.
friolator is offline   Reply With Quote
Old 31st July 2014, 08:11   #10  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
P16 works fine here. Only lsmash's P10 is incompatible with many other filters because the bit order is different. If you think you've found a bug send them a sample.
sneaker_ger is offline   Reply With Quote
Old 31st July 2014, 08:16   #11  |  Link
mp3dom
Registered User
 
Join Date: Jul 2003
Location: Italy
Posts: 1,135
The output coming from L-SMASH with stacked=true and format=YUV422P16 should be the same/very similar to the ones coming from ffms2 with enable10bithack=true.
The output with format=YUV422P8 is still dithered, so it's OK, but maybe you need a particular dither algorithm.
mp3dom is offline   Reply With Quote
Old 31st July 2014, 08:35   #12  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
Quote:
Originally Posted by mp3dom View Post
The output with format=YUV422P8 is still dithered
Yes, ffms2 dithers as well. You only need the high bit depth output if you want to use a different dither algorithm or continue working in high bit depth.
sneaker_ger is offline   Reply With Quote
Old 31st July 2014, 12:44   #13  |  Link
friolator
Registered User
 
Join Date: Jul 2014
Posts: 10
Quote:
Originally Posted by sneaker_ger View Post
P16 works fine here. Only lsmash's P10 is incompatible with many other filters because the bit order is different. If you think you've found a bug send them a sample.
What we see with a ProRes 422 HQ file and the P16 setting is that the image is about 2x taller than it should be and is split top/bottom. The top has a highly compressed version of the image, dark and blocky. The bottom is green with some motion, depending on content, but it's obviously not usable.

With P8 (which may work for our purposes since it's dithering), we get a nice looking image at the proper size - no splitting, no green.

Is what we're seeing with P16 due to the way high bit depth files are handled? (I read a little about how the way this is done in Avisynth is to kind of fake it out by having two 8-bit images merged into one image, or something like that). If so, could something be wrong with our setup?

We're using AviSynth+ since it's 64bit, if that makes a difference.

Thanks!

Last edited by friolator; 31st July 2014 at 12:47.
friolator is offline   Reply With Quote
Old 31st July 2014, 12:52   #14  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
AviSynth does not know higher bitdepths than 8. Filters like lsmashvideosource() work around it by putting 8 bits on the top and the other 8 bits on the bottom (=16 bits total). The "stacked=true" is the same hack as ffms2's "enable10bithack=true".

If you are looking for 8 bit output and you're ok with lsmash's built-in dithering it is indeed perfectly fine to simply use lsmashvideosource("input.mov", format="YUV422P8") and be done with it.
sneaker_ger is offline   Reply With Quote
Reply

Tags
seek, x264

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:03.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.