HC 0.21 - fails to start second pass

[kumi]

Is there any reason why HC would complete a 1st pass, but not start the second pass? I'm trying to re-encode a stream (I specified a target filesize, not a target bitrate) but it doesn't seem to progress beyong the end of the first pass. All I end up with is a 0 KB .m2v file...

For reference, here's the HC.INI
*INFILE g:\nightmare detective project\dvd9 demux\videofile.avs
*OUTFILE D:\NIGHTMARE DETECTIVE PROJECT\RESTORE24 OUTPUT\VideoFile.m2v
*LOGFILE D:\NIGHTMARE DETECTIVE PROJECT\RESTORE24 OUTPUT\VideoFile.log
*BITRATE 5241
*MAXBITRATE 7200
*FRAMES 0 152775
*PROFILE best
*AUTOGOP 12
*NOSCD
*BIAS 20
*MASK_SHIFT 2 0 0
*MATRIX mpeg
*LUMGAIN 1
*WAIT 0



And here's the log file:

-----------------------------------------
| HCenc - MPEG2 encoder - rel. 0.21.0.0 |
-----------------------------------------

MPEG profile@level: MP@ML
input: g:\nightmare detective project\dvd9 demux\videofile.avs
output: d:\nightmare detective project\restore24 output\videofile.m2v

--------------------
| encoder settings |
--------------------

profile: BEST
frames: 0 152775
framerate: 23.976
aspect ratio: 16:9
bitrate Kb/s: 5241
max. bitrate Kb/s: 7200
pulldown: no
closed gops: no
VBV check: yes
scene change det.: no
interlaced: no
goplen,B-pic: AUTO
dc_precision: 9
scan method: ZIGZAG
bias: 20
chapter frames: 0
time code: 0 0 0 0
CPU: MMX
matrix: MPEG, adaptive
luminance gain: yes, level 1

------------------
| source stats |
------------------

nr. of frames in source: 152776
width*height: 720x576
fps: 23.98
nr. of frames to encode: 152776
frames to encode: 0 - 152775

---------------------
| encoding - pass 1 |
---------------------

pass1 encoding time: 7:25:50 (26750 s)
fps: 5.7

[kumi]

Update: If I reduce the amount of frames to encode (I tried a 1% sample) the second pass DOES start, successfully encoding my video file. Same settings and everything. :confused:

[urvieh]

Thanks for posting this. I'm working with Cuttermaran and HC in the background. Sometimes it won't finish its encoding task, and now I know what the reason might be. So, this is not a full verification but there has to be a problem.

[Sir Didymus]

I would just add that (at least for one sample file) I am suffering for the exact same issue reported by kumi.

I also tryed to use a previous release of HC (it was the 019), but it made no difference.

I understand that this type (the mine) of bug report is of very little help for Hank315, since it gives no hints about the source of the trouble - sorry for this; however, I decided to post the comment, at least for the sake of the record...

Cheers,
SD

[kumi]

Just to follow up on this, if you set the WAIT parameter to non-zero, you'll have time to see in the hcenc.exe window whether there's an error. I found "Avisynth message: invoke environment error" everytime the second pass fails to start. A related post of mine is here: http://forum.doom9.org/showthread.php?p=1013418#post1013418

I haven't found much information about that error, but http://www.videohelp.com/forum/archive/error-using-avisynth-to-hcgui-t320514.html hints at a memory-related problem. Perhaps HC is not unloading avisynth.dll after the first pass, and that's somehow causing problem on low-memory systems? Just a guess :rolleyes:

Sir Didymus: are you running a memory intensive script, like I am?

[Boulder]

You might want to point IanB to this thread, maybe he has some ideas as an Avisynth developer.

[Sir Didymus]

Perhaps HC is not unloading avisynth.dll after the first pass, and that's somehow causing problem on low-memory systems? Just a guess...

Sir Didymus: are you running a memory intensive script, like I am?

I am not sure about this, but maybe you are in the right track, kumi. I did not repeat the test; however, now that you posted the error in the HC main window, I can easily remind to have seen the same avisynth error related to the environment invoke...

The avisynth script was generated using FAVC, and yes, I suppose it should be considered memory intensive...

@Boulder:
I can confirm that both qenc and CCE did not complain on the exact same avisynth script, so I tend to think that the issue is probably more related to HC than to Avisynth...

Cheers,
SD

[Boulder]

I just thought than IanB might provide hank with some clues on how to debug the issue.

Off the top of my head, have you tried setting SetMemoryMax in the script to a rather low value, such as 32 or 64?

[IanB]

@kumi,

I see from linked posts you are already trialing 2.5.8, as for diagnosing memory problems 1st stop is Task Manager, check the total VM size of the HC.exe process. If it's 1.5 to 2 GB then you do have problems. If it's up around the size of Physical memory, 512Mb ???, then check the free disk space on C: the page file grows into the free disk space, if you fill the disk with the page file windows tends to hang the process doing the malloc. If this is the case try manually managing the page file. Set it to minimum 3Gb or more. Again watch the HC.exe process VM size 1.5 to 2 GB is death.

From the Avisynth side run the simplest script possibleLoadPlugin("C:\Program Files\dgmpgdec\DGDecode.dll")
MPEG2Source("G:\...\VideoFile.d2v")or evenBlankClip(152776, 720, 576, Pixel_type="YV12")
KillAudio()
AssumeFPS(24000, 1001)
ShowSMPTE(size=40)Giving Hank an example failure with a Blankclip source makes his life so much easier.

[kumi]

Thanks IanB. I can confirm that my VM usage (also the Private Bytes counter) was indeed around 1.5GB when I was experiencing those crashes. I don't have time (my CPU is very slow) to test another big honking script, but next time I can, I will watch the VM size carefully.

@Boulder: The lowest SetMemoryMax value I had tried was 192. I also tried the script without a call to SetMemoryMax.

[hank315]

Giving Hank an example failure with a Blankclip source makes his life so much easier.
That would surely help :)


Here's some info how HCenc handles Avisynth input.
This is best shown by the complete source which handles the Avisynth input:
#include <windows.h>
#include "avisynth.h"

extern "C"
{
int __stdcall HC_AVS_LOADLIB(void);
int __stdcall HC_AVS_FREELIB(void);
int __stdcall HC_AVS_INIT(char *, int, int *, int *, int *, int *, int *, int *, int *, int *, char *, int);
int __stdcall HC_AVS_GETFRAME(int *, int *, int *, unsigned char *, unsigned char *, unsigned char *);
}

IScriptEnvironment *env = 0;
PClip *Video = 0;
HINSTANCE avsDLL = 0;

//////////////////////////////////////////////////////

int __stdcall HC_AVS_LOADLIB(void)
{
avsDLL = LoadLibrary("avisynth.dll");
if (!avsDLL) return 1;

IScriptEnvironment* (* CreateScriptEnvironment)(int version)
= (IScriptEnvironment*(*)(int)) GetProcAddress(avsDLL, "CreateScriptEnvironment");
if (!CreateScriptEnvironment) return 2;
env = CreateScriptEnvironment(AVISYNTH_INTERFACE_VERSION);
if (!env) return 2;

return 0;
}

//////////////////////////////////////////////////////

int __stdcall HC_AVS_FREELIB(void)
{
if (Video) delete Video; Video = 0;
if (env) delete env; env = 0;

if (avsDLL)
{
int iret = FreeLibrary(avsDLL);
if (!iret) return 7;
}
return 0;
}

//////////////////////////////////////////////////////

int __stdcall HC_AVS_INIT(char *fname, int lfname, int *setmem, int *w, int *h, int *nframes,
int *fpsn, int *fpsd, int *pixtype, int *info, char *errmsg, int lerrmsg)
{
if (!avsDLL) return 1;

if (Video) delete Video; Video = 0;

AVSValue arg[1] = { fname };
try { Video = new PClip(); *Video = env->Invoke("Import", AVSValue(arg,1)).AsClip(); }
catch (AvisynthError err) { memcpy(errmsg, err.msg, lerrmsg); return 3; }

if (!(*Video)->GetVideoInfo().HasVideo()) return (4);
if (!(*Video)->GetVideoInfo().IsYV12()) return (5);

*w = (*Video)->GetVideoInfo().width;
*h = (*Video)->GetVideoInfo().height;
*nframes = (*Video)->GetVideoInfo().num_frames;
*fpsn = (*Video)->GetVideoInfo().fps_numerator;
*fpsd = (*Video)->GetVideoInfo().fps_denominator;
*pixtype = (*Video)->GetVideoInfo().pixel_type;

*info = 0; // not used ATM

return 0;
}

//////////////////////////////////////////////////////

int __stdcall HC_AVS_GETFRAME(int *frame, int *Ypitch, int *UVpitch, unsigned char *Y, unsigned char *U, unsigned char *V)
{
PVideoFrame f;

try { f = (*Video)->GetFrame(*frame, env); } catch(...) { return 6; }

env->BitBlt(Y,*Ypitch, f->GetReadPtr(PLANAR_Y),f->GetPitch(PLANAR_Y),f->GetRowSize(PLANAR_Y),f->GetHeight(PLANAR_Y));
env->BitBlt(U,*UVpitch,f->GetReadPtr(PLANAR_U),f->GetPitch(PLANAR_U),f->GetRowSize(PLANAR_U),f->GetHeight(PLANAR_U));
env->BitBlt(V,*UVpitch,f->GetReadPtr(PLANAR_V),f->GetPitch(PLANAR_V),f->GetRowSize(PLANAR_V),f->GetHeight(PLANAR_V));

return 0;
}


The HC_AVS_LOADLIB function is executed once.
HC_AVS_INIT is executed 3 times for a 2-pass encode:
1) During the input stage, info is needed for setting global vars and allocating memory
2) At the start of the first pass
3) At the start of the second pass

For normal encodes using reliable codecs step 2) and 3), aren't needed but there are codecs out there which don't behave that nice so a new initialization is needed.
But I could create a version which skips step 2) and 3) to see if it solves this problem.

And of course, comments/advice on the source code above is more than welcome.