x264.exe stops encoding

[mshilly]

I have been using x264.exe to convert HD-DVDs VC1 video to h.264 with crf=20 and the PS3 slow profile. I have done half a dozen HD-DVDs and Blu-rays w/VC1 video this way.

However, just recently I bought a half dozen HD-DVDs and started to back them up but now x264.exe just stops processing the video at the same (frame) place every time for any given movie. It does not appear to be dependent upon what version of x264.exe I use nor does it matter what switches I use (ie --threads auto vs --threads 4).

I recall that the problem began after I updated files from the MeGUI auto update window. I have been playing with both ripbot264 and MeGUI as frontends for x264.exe.

I also re-installed AviSynth 2.5.8 just in case it was some issue with it but that did not change anything.

Any ideas?

[Ajax_Undone]

I noticed this as well... Ripbot264 just ran over the top of the fact that I only had a 3 kb video.264 file and muxed the audio with the non existent video to an mp4 file... I had about 50 dvd's in the que and in 12 hours they were all finished and better then half were messed up... I found that replacing the x.264 codec with the latest ones from Here (http://skystrife.com/x264/) fixed my problem in ripbot264.. I dont know about megui. 1115M is the latest.

[Chengbin]

How much RAM do you have? x264 will stop encoding as soon as you don't feed it with enough RAM. Ripping videos like that will use more than 1.5GB of RAM.

[burfadel]

I'm not sure with simple filtering and suitable x264 ref/b-frames using the latest (1120 or above) revision of x264, that 1.5gb to 2gb or more of memory would be used...

[roozhou]

x264 itsself can never use 1.5GB of RAM. I am afraid there might be serious memory leak in avisynth.

[LoRd_MuldeR]

x264 itsself can never use 1.5GB of RAM. I am afraid there might be serious memory leak in avisynth.

...or in one of the decoders/filters involved.

[squid_80]

If you see large memory usage it's not necessarily a leak in avisynth, older versions try and use up to 50% (I think) of available memory for caching. Unfortunately this can be a problem on x64 systems, for example if you've got 8GB of ram it will try and use 4GB and crash when it gets close to 2GB (due to 32-bit process limitations).

[roozhou]

...or in one of the decoders/filters involved.

IIRC no decoders has such serious memory leak But some filters do have terrible memory leak. e.g. VSFilter can eat up 4G memory in 10 secs.

And SetMemoryMax in .avs should help. Avisynth always tries to put the whole movie into memory so it allocate frame buffers as many as possible.

[LoRd_MuldeR]

IIRC no decoders has such serious memory leak

That's a pretty generalized statement. There are dozens of source filters (decoders) for Avisynth and usually each filter exists in several versions.

Who knows what memory leak may (or may not) exist in one of those source filters...

[roozhou]

That's a pretty generalized statement. There are dozens of source filters (decoders) for Avisynth and usually each filter exists in several versions.

Who knows what memory leak may (or may not) exist in one of those source filters...

I mean serious memory leaks that consumes GBs of memory. AFAIK memory leaks exist in almost every program written in C++ under windows, but most of them are too slow to be noticed.

And another example of memory leak is fft3dgpu. I haven't review its source code so dunno why it happens.

[LoRd_MuldeR]

I mean serious memory leaks that consumes GBs of memory. AFAIK memory leaks exist in almost every program written in C++ under windows, but most of them are too slow to be noticed.

And another example of memory leak is fft3dgpu. I haven't review its source code so dunno why it happens.

The number of bytes "eaten" by a memory leak doesn't depend on the type of plugin, but on the kind of memory leak.

For example: If a decoder leaks one 1 MB of memory per frame decoded, then read 1024 frames from your source and 1 GB of memory is lost.

So I wouldn't assume that a video filter tends to leak more bytes than a decoder (or vice versa).


BTW: You can pipe in the input from a separate process to x264, so Avisynth will run in it's own process.

This is a simple method to find out where the most memory is consumed: In the x264 process or in the Avisynth+Plugins process.

[squid_80]

AFAIK memory leaks exist in almost every program written in C++ under windows, but most of them are too slow to be noticed.
I highly doubt that, considering how easy it is to find memory leaks with visual c++.

[mshilly]

AMD 9550 Quad Core
Gigabyte MA78GPM-DS2H Motherboard
4GB DDR2-800 (512MB dedicated for GPU use)

While running the memory use never goes over 1.05GB and all four proceessors go from 100% to 0% utilization when the problem happens. Encoding just simply stops.

I am starting to think that it is an AviSynth thing becasue I used autox264 on the same file and it got more than 25% through the encode before I stopped it (was going to take 25 hours) where the ripbot264 based encode only ever gets to 1.8%. I thought that autox264 used a different frame server. Also, I took the x264.exe that autox264 was calling and used it with the ripbot264 generated command line and it still stiopped in the same place.

[roozhou]

I highly doubt that, considering how easy it is to find memory leaks with visual c++.

I doubt memory leak exists in M$'s C/C++ runtime and MFC. Since so far there is not an open source CRT for windows, such sort of memory leaks cannot be avoided.

Look at the following code from vsfilter. Can anybody explain why terrible memory leak occurs in this block of code?

CStringW param;
for(WCHAR c = str[++j]; c && c != ')'; param += c, c = str[++j]);
param.Trim();

while(!param.IsEmpty())
{
int i = param.Find(','), j = param.Find('\\');

if(i >= 0 && (j < 0 || i < j))
{
CStringW s = param.Left(i).Trim();
if(!s.IsEmpty()) params.Add(s);
param = i+1 < param.GetLength() ? param.Mid(i+1) : L"";
}
else
{
param.Trim();
if(!param.IsEmpty()) params.Add(param);
param.Empty();
}
}

[squid_80]

I doubt memory leak exists in M$'s C/C++ runtime and MFC. Since so far there is not an open source CRT for windows, such sort of memory leaks cannot be avoided.I don't understand. There are no memory leaks so they can't be avoided ???
Microsoft's C runtime library is open source; the source code comes with Visual Studio. What would be the point of a debug runtime library if the source wasn't available?

Look at the following code from vsfilter. Can anybody explain why terrible memory leak occurs in this block of code?I'm not going to do your homework for you. Step through it with the debugger and see where memory is allocated and released. For string objects it's normal to see them consume memory when new input is assigned/added but not release memory when they shrink; the allocated memory is not released in case they grow again. All memory should be released when they are deleted/go out of scope.
One thing that immediately jumps out is j being pre-incremented in the for loop, the first character of str will be skipped and if str is a null string who knows what might happen...

[roozhou]

All memory should be released when they are deleted/go out of scope.
One thing that immediately jumps out is j being pre-incremented in the for loop, the first character of str will be skipped and if str is a null string who knows what might happen...

The problem is memory is never released when going out of while scope. If I move the definition of "CStringW s" out of while scope, memory leak is greatly reduced. I doubt either there is a bug in MFC runtime or the M$ compiler is doing something stupid.

[squid_80]

What does params.Add() do? It's probably incrementing the reference count for s.

As of MFC version 4.0, when CString objects are copied, MFC increments a reference count rather than copying the data. This makes passing parameters by value and returning CString objects by value more efficient. These operations cause the copy constructor to be called, sometimes more than once. Incrementing a reference count reduces that overhead for these common operations and makes using CString a more attractive option.

As each copy is destroyed, the reference count in the original object is decremented. The original CString object is not destroyed until its reference count is reduced to zero.

You can use the CString member functions LockBuffer and UnlockBuffer to disable or enable reference counting.

[roozhou]

What does params.Add() do? It's probably incrementing the reference count for s.

CAtlArray<CStringW> params;

[squid_80]

Right, so if s is added to params as a new element it won't be destroyed until that element is removed/replaced, even if s goes out of scope.

[roozhou]

Right, so if s is added to params as a new element it won't be destroyed until that element is removed/replaced, even if s goes out of scope.

With MFC/ATL you never know when memory is allocated and when it is freed. Such things hardly happens with C.

[squid_80]

Why can't you override the new and delete class operators?

[roozhou]

Why can't you override the new and delete class operators?

You mean I override the new and delete operators of MFC/ATL classes?

[mshilly]

Anyone else experiencing ripbot264 hang when converting HD-DVD to h.264?

[Atak_Snajpera]

replace x264.exe with this http://skystrife.com/x264/x264.1120M.x86.exe
replace x264_x64.exe with this http://skystrife.com/x264/x264.1120M.x64.exe

[mshilly]

Thanks for the links but they still stop encoding.
I am at the end of my rope... encoding with autox264 takes too long and I don't trust it because one of the two I have done with it the audio was out of synch (Italian Job).

[burfadel]

If the audio is out of sync, run over the dvd vobs (the main movie vobs) with Projectx, keeping all the vobs as part of the same 'collection'. The m2v file then can be indexed using dgindex. Use the latest version available here:
www.oozoon.de/main_en.html

The reason why you want to use the 'new' version is it has much better handling of DVD's.

[mshilly]

burfadel, thanks for the input, however, I am dealing with HD-DVD encoding not DVDs and it is one large m2ts file.

[kemuri-_9]

have you even tried to encode the .avs to an intermediary lossless and then encode from that with x264?

you claim this is an x264 problem without having stated you tried other encoders to see if deadlocks in them as well.

[burfadel]

burfadel, thanks for the input, however, I am dealing with HD-DVD encoding not DVDs and it is one large m2ts file.

Oops!

Could still be a stream fault...?

[mshilly]

kemuri,
I agree that it could very well be an AviSynth issue as well since that is what is feeding it, just haven't been able to determine one way or the other.

burfadel,
that was my next course of action. I may need to upgrade my tsMuxer and not append the evo files together in the tsMuxer. I noticed updated tsMuxer and x264 as well as eac3to and meGui available today.