Is it possible to pause and resume video conversion from a temp file?

[HerpaDerp]

Hi, I have been using a few programs over the years such as XMedia Recode, Mediacoder, AviDemux, and MeGUI to convert movies to Mpeg-4 AVC, AC3 or AAC, MP4 files that can be burned to a DVD and played on the PS3.

I have had pretty good success, but I keep being plagued by a limitation in most video converters.

Most video converters do not allow you to pause your video conversion, and the ones that do, do not allow you to save your progress and continue at a later date.

There are times in which pausing video conversion simply will not suffice. Sometimes, I am forced to reboot my computer. Sometimes I want to play a game that requires a lot of RAM. Pausing video conversion does not free up the ram and is yet another running process to reduce gaming performance.

Video conversion takes a *long* time. I have a pretty decent computer (Core i7 870 quad with 8 gigs of ram). My computer is great for video conversion. However, there are certain conversions that require 4 or more hours to complete. That is a long time to tie up my computer.

I guess I'm seeking a discussion about the topic of pausing and saving your progress to your harddrive so that you can continue the video conversion at a later date. Why do no conversion programs have this feature? It seems to me like it would be *extremely* handy.

Also, I have tried installing Windows in a Virtualbox and then experimenting with closing Virtualbox during a video conversion. Virtualbox allows you to save your entire session to your harddrive so that it can be resumed at a later date. However, when I try to do this while converting videos with various video conversion programs, Virtualbox will crash upon resuming. That's not good. I'm hesitant to say it's not possible. The crashing could be limited to just my setup, but it seems like it should work in theory.

Anyway, I'm interested to know if anybody else has ever wanted this functionality in a video conversion program... and if there may be a program that does what I am seeking.

Thanks,
HerpaDerp

[J_Darnley]

Lots of people ask for this. The suggestions are: hibernate if you are going to turn the computer off; suspend if you want to pause (your computer will swap the allocated but unused memory to the hard disk); set a low priority if you want to do other things. I'm afraid that restarting will have to wait.

If you think adding resume support to x264 is easy, you are welcome to try.

[MatLz]

Script :
sourcefilter("source")
filters()



...encode running in crf mode... but...
"OMG ! I need to stop the encode because I have my reasons to do that !"
Ctrl + C
x264 log window :
aborted at input frame AAAA, output frame XXXX
encoded XXXX frames
New script :
sourcefilter("source")
filters()
trim(XXXX,0)
New CLI :
x264.exe --settings -o file_2.*

........

New encode running in crf mode... but...
"OMG ! I need to stop the encode AGAIN because I have my reasons to do that !"
Ctrl + C
x264 log window :
aborted at input frame BBBB, output frame YYYY
encoded YYYY frames
New script :
filter("source")
filters()
trim(XXXX,0)
trim(YYYY,0)
New CLI
x264.exe --settings -o file_3.*

...
.......
..........


After that, just append all the files (firstfile + file_2 + file_3 ........)
with mkvmerge, mp4box, copy/b ...

[kemuri-_9]

the proposition of trimming to start the next clip where the video stopped previously is dangerous based on the settings:

1) b-frames:
As B-frames cause coded frame order and display frame order to become different, in the scenario of processing a mini-gop such as
b1 b2 b3 b4 b5 b6 b7 P8
where you stop the encoder at frame 5 into the mini-gop, what really gets encoded is
P8 b1 b2 b3 b4
in short - this mini-gop is corrupted/incomplete.

2) VBV:
VBV requires knowing how full the current buffer is.
when x264 starts, it assumes the buffer is filled at a current percentage
(this is a command line parameter so it can be changed, but as a user you don't know how full it was on cancelling the encode anyway).
If the stream you canceled had the VBV more filled than the new initial assumption, you could violate VBV when concatenating the streams (iirc this is a "VBV overflow").
and in the reverse scenario where it's less full than assumed, the stream can empty unexpectedly (iirc this is "VBV underflow").

[MatLz]

That was the fastest/easiest way for the OP assuming the files were perfectly valid.

Best way is, and it is how I proceed, cut the streams to the last frame of the gop (with dgavcindex, mkvmerge, mp4box... and trim in the script to the next scene change (supposed to be an I frame)

So, it should be good to have an option in x264 which discarde supposed incomplete gop when the encode is aborted, no ?

[kemuri-_9]

So, it should be good to have an option in x264 which discarde supposed incomplete gop when the encode is aborted, no ?

And you're assuming that this is easy; this is more primarily a job of x264cli and not libx264 as well.

x264cli writes frames to the output file as they arrive from the encoder.
To do this 'discarding of incomplete mini-gops' it would have to rework the entire output file writing system so as to buffer mini-gops and not write the mini-gop if it was canceled midway,
as you can't just 'unwrite' what was already written to the output file.
This change could also potentially cause impacts on programs where you feed it x264cli's output through a pipe.

Overall, there is too much effort for the little benefit; similar to this pause/resume functionality that people frequently request.


Dropping the last entire GOP where you start the next stream from the GOP point is safer than straight starting from the last frame that was outputted,
but this banks on the fact you're not using open-gop and it still doesn't fix the potential for VBV issues.

[MatLz]

Dropping the last entire GOP where you start the next stream from the GOP point is safer than straight starting from the last frame that was outputted

That was what I meant... but if it isn't good...

[HerpaDerp]

Lots of people ask for this. The suggestions are: hibernate if you are going to turn the computer off; suspend if you want to pause (your computer will swap the allocated but unused memory to the hard disk); set a low priority if you want to do other things. I'm afraid that restarting will have to wait.

If you think adding resume support to x264 is easy, you are welcome to try.

Unfortunately, I do not have any programming knowledge. Based on the suggestions you mentioned, none of them really fit the bill. I have tried a number of things, including lowering priority, lowering affinity, and experimenting with Virtualbox session saving.

I have to lower my priority to idle or below normal and set my affinity to 2-4 cores (8 total because of hyperthreading). This enables me to convert the movie, but ultimately drags out the conversion time by a few hours. On top of this, if I try to play a game (like starcraft 2), my performance will suffer.. especially when playing custom maps. I never know when I'm going to be invited to play a game, my friends ask me to play and if I am in the middle of doing a long conversion, sometimes I simply cancel and wonder why I even bothered trying.

Also, I have performed a few tests. During a long conversion, pausing the conversion does not have the same performance benefit as cancelling. I haven't pinned the exact reason. I have *plenty* of ram (8 gigs)... and that means x264 has plenty of ram (while paused) and my games have plenty of ram. However, game performance is much worse while the conversion is paused. I do not understand this, if someone could shed light on why this is the case, I would appreciate it.

Of course, there's always the thought of doing a video conversion while I sleep. Lets just say that this never quite works out. I become too tired and lose interest in setting up the conversion, or I forget... which is why I try to do this during the day when I'm not completely "out of it".

The point is, I think video conversion should have a way to pause and save your progress to a temp file on your harddrive. I realize the implications of something like this are vast and may require a lot of programming... it is something that would benefit many people.

If it's not possible or way too much work to implement, I'm fine with that. I'm simply stressing my reasons for why I think it is important.

Thanks

[HerpaDerp]

the proposition of trimming to start the next clip where the video stopped previously is dangerous based on the settings:

1) b-frames:
As B-frames cause coded frame order and display frame order to become different, in the scenario of processing a mini-gop such as
b1 b2 b3 b4 b5 b6 b7 P8
where you stop the encoder at frame 5 into the mini-gop, what really gets encoded is
P8 b1 b2 b3 b4
in short - this mini-gop is corrupted/incomplete.

2) VBV:
VBV requires knowing how full the current buffer is.
when x264 starts, it assumes the buffer is filled at a current percentage
(this is a command line parameter so it can be changed, but as a user you don't know how full it was on cancelling the encode anyway).
If the stream you canceled had the VBV more filled than the new initial assumption, you could violate VBV when concatenating the streams (iirc this is a "VBV overflow").
and in the reverse scenario where it's less full than assumed, the stream can empty unexpectedly (iirc this is "VBV underflow").

I think this applies to me. I use VBV to ensure the bitrate does not exceed 14000. This is so that I can burn the video to a DVD without the risk of the movie skipping during playback.

[nm]

Also, I have performed a few tests. During a long conversion, pausing the conversion does not have the same performance benefit as cancelling. I haven't pinned the exact reason. I have *plenty* of ram (8 gigs)... and that means x264 has plenty of ram (while paused) and my games have plenty of ram. However, game performance is much worse while the conversion is paused. I do not understand this, if someone could shed light on why this is the case, I would appreciate it.

The only possible explanation that I can imagine would be GPU resource usage, if you use DGDecNV or GPU filtering in AviSynth. But even those shouldn't cause noticeable performance issues when the encoder process is paused, I think.

If you use a frontend instead of plain x264 CLI, check the CPU load of all the processes while encoding is paused. There might be something odd happening behind the scenes.