[FFmpeg] simultaneous 2-pass encodings

Klagar · 12th November 2010, 16:01

Hi !

I've made some research but couldn't Google anything really useful, though a similar issue has been adressed once on Doom10...

I am testing for encoding speed in FFmpeg, comparing source formats, destination formats, and some command line paramaters. So far, so good, I am slowly finding what best suits my needs.

But now I'm experiencing a problem : I start two simultaneous 2-pass encodings (as I don't have this problem with 1-passes), and the first pass goes all well, but when it's time for either process to start the second pass, I get error messages, including :

[libx264 @ 01c57930] failed to rename "x264_2pass.log.temp" to "x264_2pass.log"
[libx264 @ 01c57930] failed to rename "x264_2pass.log.mbtree.temp" to "x264_2pass.log.mbtree"
[libx264 @ 01c61a90] ratecontrol_init: can't open stats file
and
[libx264 @ 01c57920] statistics are damaged at line 2877, parser out=0

My semi-educated guess would be that there's some kind of conflict between .log files, but I use -passlogfile and am positively sure said file is differently named for each process.

It never happens when I do many simultaneous 1-pass encodings, nor when I do one 2-pass encoding at a time. And I am reluctant to blame it on memory, since the formulas I use for 1-passes seem much more demanding. Then again, I might be wrong...

So either my guess is wrong, or this issue adresses yet another file I just don't know about. Can anyone help me understand what's happening ?

Regards and thanks a lot !

J_Darnley · 12th November 2010, 16:18

libx264 doesn't obey ffmpeg's log file option. You will have to use a different current directory (I.e: mkdir Something; cd Something; ffmpeg ...) for each ffmpeg you run, so that the file gets written and read without interference.

Dark Shikari · 12th November 2010, 17:29

Quote:

Originally Posted by J_Darnley

libx264 doesn't obey ffmpeg's log file option.

More accurately, ffmpeg doesn't tell libx264 anything about its log file.

Klagar · 12th November 2010, 19:23

I see... too bad...
Thanks !

Klagar · 12th November 2010, 20:44

Is there really no way of circumventing this issue, other than having 15 different instances installed in different folders ? :S I wouldn't want to have to switch to 1-pass encodings just because of that !

nm · 12th November 2010, 21:09

Quote:

Originally Posted by Klagar

Is there really no way of circumventing this issue, other than having 15 different instances installed in different folders ?

You only need one installation that is run from different working directories.

Klagar · 15th November 2010, 13:28

I don't understand what you mean by that. It must not be, as I think, to have several ffmpeg.exe put in many different folders, which we call separately when another already is working ; that would be "many different instances installed in different folders". If so, what do you mean exactly ? :S

nm · 15th November 2010, 13:38

Quote:

Originally Posted by Klagar

I don't understand what you mean by that. It must not be, as I think, to have several ffmpeg.exe put in many different folders, which we call separately when another already is working ; that would be "many different instances installed in different folders". If so, what do you mean exactly ? :S

Assume ffmpeg.exe is in directory "c:\ffmpeg\". You can run it from any directory by using the full path as follows.

In one cmd.exe instance:

Code:

cd dir1
c:\ffmpeg\ffmpeg.exe -i infile1 -vcodec libx264 ...

While the first encode is running, start another one in dir2:

Code:

cd dir2
c:\ffmpeg\ffmpeg.exe -i infile2 -vcodec libx264 ...

Input and output files don't need to be in those directories dir1 and dir2 either if you only use them for the stats files.

Klagar · 15th November 2010, 13:58

But whatever the input file's directory, always the x264_2pass.log will be created in c:\ffmpeg\ when I try to make a 2-pass. And if two or more files are exploiting the same .log file, then conflict ensues and no encoding at all ends up being done.

nm · 15th November 2010, 15:49

Quote:

Originally Posted by Klagar

always the x264_2pass.log will be created in c:\ffmpeg\

Not unless that's your current directory. Change it for each encode!

Klagar · 15th November 2010, 18:57

Wow ok...
I hadn't seen the "cd dir" at the start of the command lines, no wonder I didn't get where you where heading

But I tried it, and it works marvelously well ! Thanks so much !
I've helped myself with a PHP script that generates a unique ID, which I use to create a unique folder which sole purpose is to contain the .log files and is deleted as soon as the encoding has met with success !
So thanks again, would never have found that out on my own !

12th November 2010, 16:01	#1 \| Link
Klagar Registered User Join Date: Sep 2010 Posts: 46	[FFmpeg] simultaneous 2-pass encodings Hi ! I've made some research but couldn't Google anything really useful, though a similar issue has been adressed once on Doom10... I am testing for encoding speed in FFmpeg, comparing source formats, destination formats, and some command line paramaters. So far, so good, I am slowly finding what best suits my needs. But now I'm experiencing a problem : I start two simultaneous 2-pass encodings (as I don't have this problem with 1-passes), and the first pass goes all well, but when it's time for either process to start the second pass, I get error messages, including : [libx264 @ 01c57930] failed to rename "x264_2pass.log.temp" to "x264_2pass.log" [libx264 @ 01c57930] failed to rename "x264_2pass.log.mbtree.temp" to "x264_2pass.log.mbtree" [libx264 @ 01c61a90] ratecontrol_init: can't open stats file and [libx264 @ 01c57920] statistics are damaged at line 2877, parser out=0 My semi-educated guess would be that there's some kind of conflict between .log files, but I use -passlogfile and am positively sure said file is differently named for each process. It never happens when I do many simultaneous 1-pass encodings, nor when I do one 2-pass encoding at a time. And I am reluctant to blame it on memory, since the formulas I use for 1-passes seem much more demanding. Then again, I might be wrong... So either my guess is wrong, or this issue adresses yet another file I just don't know about. Can anyone help me understand what's happening ? Regards and thanks a lot !

12th November 2010, 16:18	#2 \| Link
J_Darnley Registered User Join Date: May 2006 Posts: 957	libx264 doesn't obey ffmpeg's log file option. You will have to use a different current directory (I.e: mkdir Something; cd Something; ffmpeg ...) for each ffmpeg you run, so that the file gets written and read without interference. __________________ x264 log explained \|\| x264 deblocking how-to preset -> tune -> user set options -> fast first pass -> profile -> level Doom10 - Of course it's better, it's one more.

12th November 2010, 20:44	#5 \| Link
Klagar Registered User Join Date: Sep 2010 Posts: 46	Is there really no way of circumventing this issue, other than having 15 different instances installed in different folders ? :S I wouldn't want to have to switch to 1-pass encodings just because of that ! Last edited by Klagar; 12th November 2010 at 20:44. Reason: Spelling

12th November 2010, 19:23	#4 \| Link
Klagar Registered User Join Date: Sep 2010 Posts: 46	I see... too bad... Thanks !

15th November 2010, 13:28	#7 \| Link
Klagar Registered User Join Date: Sep 2010 Posts: 46	I don't understand what you mean by that. It must not be, as I think, to have several ffmpeg.exe put in many different folders, which we call separately when another already is working ; that would be "many different instances installed in different folders". If so, what do you mean exactly ? :S

15th November 2010, 13:58	#9 \| Link
Klagar Registered User Join Date: Sep 2010 Posts: 46	But whatever the input file's directory, always the x264_2pass.log will be created in c:\ffmpeg\ when I try to make a 2-pass. And if two or more files are exploiting the same .log file, then conflict ensues and no encoding at all ends up being done.

15th November 2010, 18:57	#11 \| Link
Klagar Registered User Join Date: Sep 2010 Posts: 46	Wow ok... I hadn't seen the "cd dir" at the start of the command lines, no wonder I didn't get where you where heading But I tried it, and it works marvelously well ! Thanks so much ! I've helped myself with a PHP script that generates a unique ID, which I use to create a unique folder which sole purpose is to contain the .log files and is deleted as soon as the encoding has met with success ! So thanks again, would never have found that out on my own !