How to make x264 with non-linear encoding

[seemees]

To x264 developers
I'm often use x264 for DVD contents and no problem - in max quality settings we get about 3 hours in one pass. And all work estimates 6 hours.
Now I'll try to encode HD contents. 720p, 1080p and so on.
First trying is very very BAD. In maximum quality one pass takes 24 more hours... And that's the questions:

1.How to make soft BREAK in encoding process (say reboot or stop encoding process for time). We can do it by pressing ESC key during encoding process. Like this

if(kbhit()){
switch(getch()){
case 0: //Extended key
getch();
break;
case 27: //ESC pressed
printf("\nAre you shure [Y/N]?:");
if(getch()=='Y'){
//Save enc state
Save_x264_EncodingState(FileName);
//Exit to os
ExitFrom_x264();
exit(0);
}
}
}

2.In start routine (ex: main) check to command line options
--continue <file.savestate>
if it exist - before run encoding process we need to do
Restore_x264_EncodingState(FileName);
This procedure can restore last state (1st,2nd pass) and
seeks into input video stream to proper position. And go encoding now...

Please help, how to implement this, and may be you implement this feature in newer version of x264. May be one
frontend GUI solutions for x264 exists to do so?

With best regards, seemees

[LoRd_MuldeR]

Use Hibernation. Optionally use a faster preset and/or get a faster encoding machine. x264 can be really fast, but if you use slow (high quality) settings, it'll take time ;)

[qyqgpower]

Split the input before encoding is also an option.
If you are using avs script, just trim() the source.
Set the output of x264 to raw and encode.
Finally, use copy/b to join the raw output.

[seemees]

LoRd_MuldeR
faster settings not option. Sorry, but only --slow and --slower. Faster machine can't using too :-)

qyqgpower
Thanks bro. How correctly use trim() to don't brake ratecontrol curve from entire movie? How to determine split point?
And how to insert raw .264 into AVI and into MKV containers?

to x264 developers
And how to save/restore all x264 state without encoding stream lost. It is an actual question too.

With best regards, seemees

[nurbs]

And how to save/restore all x264 state without encoding stream lost. It is an actual question too.
You can't. Like LoRd_MuldeR said just hibernate your computer instead of shutting it down. When you start it up again it will continue the encode from where you left it.

[seemees]

nurbs
Ok, but in x264 library (and in the x264 source) developers can make 2 function: saveEncState(filename) and restoreEncState(filename). It'is easy - believe me. I wrote
many slow programs and all of them include this simple functions - they save all data array and point of input and output positions. Cause I not familar with x264 sources,
I can not implement this correctly, but any of x264 developers can do it. Isn't it? Am I right?
And in the first pass some of the .stats and .mbtree files already written (it has an .temp extensions). Can we use partial info from this files if encoding process accidentially brakes in the middle of 1st pass?

With best regards, seemees

[Dark Shikari]

nurbs
Ok, but in x264 library (and in the x264 source) developers can make 2 function: saveEncState(filename) and restoreEncState(filename). It'is easy - believe me. I wrote
many slow programs and all of them include this simple functions - they save all data array and point of input and output posiotions. Cause I not familar with x264 sources,
I can not implement this correctly, but any of x264 developers can do it. Isn't it? Am I right?No, this would be extraordinarily difficult and require thousands of lines of save/restore code. Not to mention the fact that it's actually impossible to save/restore the exact state, since the "state" always has threads that are in progress. Stopping all the threads would require yet more code and yet more corner cases.

On the other hand, your operating system can do it all instantly and magically. So why not use that?

[seemees]

Dark Shikari
Thanks bro - but I want to break encoding not to shutdown computer. But to make some other tasks in the all power of processor, and then start the encoding process again... Say in other day, or other week... This is very good for long tasks, and all threads can stop easily by makes in the threads two more functions:
-SaveThreadState() and -RestoreThreadState(). It is error in project stage of x264 encoder. Now, we can use classes, and can implement two func myclass::saveState
and myclass::restoreState... But no one from the developers don't want to do this... I mean we need hard SEEK posibility of encoding process: this makes store/restore methods
possible. And of course in next projects, what based on x264 - some one implements this. May be it will be me :-)


With best regards, seemees

[akupenguin]

If you don't want to shutdown, then the solution is called Ctrl-Z (aka SIGSTOP). You may also want to ensure that you have a big enough swapfile so that x264's memory can all be swapped out in favor of your other apps, but that's exactly the same amount of space that would be needed for a state file, so this isn't any extra constraint.
--seek isn't a proof of concept because encoding a movie in multiple independent sections with otherwise identical commandline options doesn't give you the same result as a single encode. (multiple sections is lower quality)

Give it up. Pausing is a problem common to many many programs, which makes it the operating system's business. And OSes have solved this problem. Which doesn't leave anything for us to do.

[seemees]

akupenguin
But what if we change threads count right during encoding process?
1.We set threads=1
2.Encoding 100 frames (or over).
3.Pausing right after last call int x264_encoder_encode( x264_t *h,
x264_nal_t **pp_nal, int *pi_nal,
x264_picture_t *pic_in,
x264_picture_t *pic_out )
and save state.
4.exit from the program (x264).

In new exec of x264 we restore state (it is in 1 threads) and set threads=4 (for example).
encoder then continue using all this threads in about 20-100 frames. Isn't it?
And Ctrl-Z (SIGSTOP) not implemented on PC (windows XP and windows 7). It is not an options. Sorry...
But thank you for your words.

With best regards, seemees

[Chengbin]

And Ctrl-Z (SIGSTOP) not implemented on PC (windows XP and windows 7). It is not an options. Sorry...
But thank you for your words.

With best regards, seemees

Yes it is.

[imcold]

If you just want to redistribute the cpu power to other processes, lowering x264's process priority is also an option.

[seemees]

imcold
Even setting very low (or idle) priority for x264 process can't help. Cause it is using all cores of processor - and games and video and online web browsing slow down significantly. That's why we want to break down encoding process - and using it when we sleep (at night :-) ). And tomorrow we want to stop encoding and continue it next night :-)

With best regards, seemees

[dvy]

akupenguin
Ctrl-Z (SIGSTOP) not implemented on PC (windows XP and windows 7). It is not an options. Sorry...
But thank you for your words.


Windows have similar linux Ctrl-Z (SIGSTOP) function,it is SuspendThread/ResumeThread API function ,use procexp.exe tool,you can Suspend/Resume a process or thread in windows,after you Suspend x264,you can use full CPU power do your another job,and you can safe hibernate. :)

you can download procexp.exe in here (http://download.sysinternals.com/Files/ProcessExplorer.zip) , it is powerful windows tool.

[seemees]

dvy
It is an option! Thank you, brother! I'll using it for a while. One question: is it support Windows 7?
Only one problem still exists. How to shutdown computer without hibernate, and with restoring encoding work. It is may possible in Driver installation, or in Windows failure.
If reboot is take control without user, all works will lost.
I thought - native save/restore support from x264 developers be better...

Car players without -20db button (just mute) is only nervous.
Car players without random/repeat is very nervous.
And Car players without restore positions after pressing off/on is BAD. Isn't it?

But again very big THANKS to you...
With best regards, seemees

[dvy]

One question: is it support Windows 7?

absolutely support Win7.

Only one problem still exists. How to shutdown computer without hibernate, and with restoring encoding work.

Don't suspending hibernate,and restore encoding work,it must be supported natively by x264,this isn't easy.

[nm]

Only one problem still exists. How to shutdown computer without hibernate, and with restoring encoding work.

Run x264 in a virtual machine.

[burfadel]

Yesterday I was encoding at low priority and was playing Modern Warfare 2 at max settings without any issues regarding performance. Of course the encode slows down significantly but it didn't affect the game at all. I have a q9400 overclocked, 4gb ddr2-1066 Ram, Radeon 4850 1gb video card, Windows 7 x64. Not the highest end of computers by a long shot by today's standards, but it does give an indication how effective prioritisation is. I had been doing similar things on older configs with the latest games at the time in the past on both Vista x64 and XP x64.

[neuron2]

And Car players without restore positions after pressing off/on is BAD. Isn't it? It's not fully off. Try disconnecting power and see if the position is retained. It's a kind of hibernation.

[seemees]

Thanks all
nm
In my WMWare version wmware client does not support all 4 core of my processor. Only 2. And that in virtual machine x264 will be 2 more times slower. It is not an option :-(
burfadel
It is good, but how many threads you are using? What if you are using all 4 threads? And what do you do when computer hangs on into game or into other application?
All works lost. And all we need is soft PAUSE and EXIT from x264. And then START and RESUME slow encoding tasks...
neuron2
Correct, but again - it is not helps if Windows hangs up. Or if you are use full reboot of computer to install new software or new driver. And we will waiting waiting and
waiting (2 more days or more in the --placebo preset) :-)

With best regards, seemees

[nurbs]

When your computer hangs how exactly are you going to tell x264 to stop encoding? :rolleyes:

[bob0r]

When your computer hangs how exactly are you going to tell x264 to stop encoding? :rolleyes:

http://x264.nl/Dark_Shikari_meet_pain_and_suffering_small.png

[Audionut]

You have been advised by the main ddev of x264 as to why it will not happen.
You have been given plenty of other solutions to use.

But you only seem interested in pointing out the problems with other people suggestions and pushing for something that Aku said aint going to happen.

If you've got such a problem that "we" must have sole cpu use for other tasks, the solution is simple. Either enjoy the outside world while running your encoding task. Or stop using --preset stupid if you don't have days to spare.

[seemees]

nurbs
First way - to save state in the x264 and exit from it. And then work with dangerous tasks (game, driver install and setra).
Second way - internally supported by x264 - right after any 20 min save state - and continue work. In first start - restore last possible state (20 min lost - but it is better then erase 1 day encoding movie).

bob0r
It's cool - it is a third way, of course :-)

Audionut
I am a programmist. And I know many computer languages and techniks. And save/restore - well knowh method that implemented for long and slow tasks. It is done with BIG "CAN'T DO IT" - say for internet downloading in Opera and Mozilla. Cause many situations can happend and destroed long file up-down-loading. It is install, hang, system failure, BSOD, toilet needs :-) And all again from start - BAD BAD BAD,
I don't want it. You don't want it. We don't want it. Together we can make life easy and safe. For safe time and data.

In potentially long and slow operations developer may inserts methods of saving/restoring all arrays and internal states.

And (offtopic) why developers of x264 don't implement direct work with 64-bits DirectShow filters? Now we can't easy work with 64-bits x264, cause Avisynth is 32-bits (I know about tool which connect 32-bit Avisynth and 64-bit x264 - but it is wrong way). How about take some code (partially or full) written in existing project http://forum.doom9.org/showthread.php?t=141441?

With best regards, seemees

[Dark Shikari]

nurbs
First way - to save state in the x264 and exit from it. And then work with dangerous tasks (game, driver install and setra).
Second way - internally supported by x264 - right after any 20 min save state - and continue work. In first start - restore last possible state (20 min lost - but it is better then erase 1 day encoding movie).

bob0r
It's cool - it is a third way, of course :-)

Audionut
I am a programmist. And I know many computer languages and techniks. And save/restore - well knowh method that implemented for long and slow tasks. It is done with BIG "CAN'T DO IT" - say for internet downloading in Opera and Mozilla. Cause many situations can happend and destroed long file up-down-loading. It is install, hang, system failure, BSOD, toilet needs :-) And all again from start - BAD BAD BAD,
I don't want it. You don't want it. We don't want it. Together we can make life easy and safe. For safe time and data.

In potentially long and slow operations developer may inserts methods of saving/restoring all arrays and internal states.

And (offtopic) why developers of x264 don't implement direct work with DirectShow filters? Now we can't easy work with 64-bits x264, cause Avisynth is 32-bits (I know about tool which connect 32-bit Avisynth and 64-bit x264 - but it is wrong way). How about take some code (partially or full) written in existing project http://forum.doom9.org/showthread.php?t=141441?

With best regards, seemeesIf it's so easy, why not do it yourself and submit a patch?

[seemees]

Dark Shikari
Ok, I try it - and may be DO IT. But all technical questinos about x264 inside - I will ask. Is it possible to has developers explanation?

First question, is h->param.i_threads mean real threads counter? If so, first of all we need to create variable
h->param.i_maxtreads and all occurance of .i_threads replace with .i_maxthreads.

Second question, I want to append 2 functions: int encoder_save_state(char *filename) and int encoder_restore_state(char *filename).
In which source file that functions we can put?
This functions is set h->param.i_threads=1; and h->param.i_threads=h->param.i_maxthreads correspondently.


With best regards, seemees

[Dark Shikari]

Dark Shikari
Ok, I try it - and may be DO IT. But all technical questinos about x264 inside - I will ask. Is it possible to has developers explanation?

First question, is h->param.i_threads mean real threads counter? If so, first of all we need to create variable
h->param.i_maxtreads and all occurance of .i_threads replace with .i_maxthreads.

Second question, I want to append 2 functions: int encoder_save_state(char *filename) and int encoder_restore_state(char *filename).
In which source file that functions we can put?


With best regards, seemeesI can answer basic questions on IRC. But do note that I will not do everything for you.

[seemees]

Dark Shikari
Right now I don't have IRC and don't know it.

This is a encoding loop, isn't it?

x264.c
...
. /* Encode frames */
. for( i_frame = 0, i_file = 0, i_frame_output = 0; b_ctrl_c == 0 && (i_frame < i_frame_total || i_frame_total == 0); )
. {
. if( input.read_frame( &pic, opt->hin, i_frame + opt->i_seek ) )
. break;
.
. pic.i_pts = (int64_t)i_frame * param->i_fps_den;
.
. if( opt->qpfile )
. parse_qpfile( opt, &pic, i_frame + opt->i_seek );
. else
. {
. /* Do not force any parameters */
. pic.i_type = X264_TYPE_AUTO;
. pic.i_qpplus1 = 0;
. }
.
. i_frame_size = Encode_frame( h, opt->hout, &pic );
. if( i_frame_size < 0 )
. return -1;
. i_file += i_frame_size;
. if( i_frame_size )
. i_frame_output++;
.
. i_frame++;
.
. /* update status line (up to 1000 times per input file) */
. if( opt->b_progress && i_frame_output % i_update_interval == 0 && i_frame_output )
. Print_status( i_start, i_frame_output, i_frame_total, i_file, param );
.
. //Right here we can insert break point?
. if(kbhit()!=0){
. ///.... check to ESC, ask for Y/N, save states
. }
.
.
. }


1.Is it right place to put encoder_save_state(...)?
2.I thought that encoder_restore_state(...) can inserted right before this encoding loop. Isn't it?

With best regards, seemees

[neuron2]

Moving to development...

[DaveEL]

I doubt you need to go anywhere near the encoder core to make something like this i implemented this feature in avs2avi for xvid a few years back. Not sure about the inner workings of X264 but i would guess the process will be similar.

It basically worked as follows

first run works as normal if the process is terminated before the pass is finished the next time you run
1) read the stats file
2) find the last keyframe
3) delete the info in the stats file starting at this point
4) open the source video file at that frame
5) encode to a new stats file
6) append the new stats file on the end of the old one

This should give an identical stats file as the first pass encodes were done at constant quant as i remember.

This can then be used in the second pass. If the second pass is interrupted a similar process happens when it is resumed

1) Open output video find the last keyframe and discard everything starting with that frame
2) Create a copy of the stats file with only the info starting from the last frame found above
3) set the encoder to use that stats file and reduce the target file size by the size of the existing output
4) set the input video to start at the same frame.
5) append the frames the new encode to those which came out of the original.

This stage will produce slightly different output to a single encode and any bitrate limits could possibly be exceeded at the point when the new and the old encodes join (as the encoder will not know the bitrate of the content before the join) but in my experience with this in xvid the output file size prediction was actually better.