HCWrap - the HCEnc "ini" wrapper

[Darksoul71]

Hi all,

based on a short conversation on using the new *AQ feature of HCEnc together with DVD-RB (and RB Opt) I decided to hack together a simple "wrapper" or middleware if you like.

As HCEnc^n it´s written in AutoIt. HCWrap simply alters any given HC.ini file on the fly and add parameters of your choice (e.g. a custom matrix or something like *AQ).

Here is the readme:

How to install HCWrap:

Copy HCWrap.exe and HCWrap.txt into your DVD-Rebuilder
encoder directory for HCEnc. In a default installation
HCEnc will be name HCbatch.exe

- Rename HCbatch.exe to HCbatch_org.exe
- Rename HCWrap.exe to HCbatch.exe

Edit HCWrap.txt to suit your needings.
HCWrap will add all your HCEnc parameters found in
the HCWrap.txt to the HC.ini given via CLI. Then
the true HCEnc.exe will be called with the altered
ini-file.

If no dedicated ini-file is specified then HCWrap
will alter an existing HC.ini residing in the same
dir or a HC.ini will be generated when no one exists.

HCEnc will then read the settings from the modified
HC.ini at startup. HCWrap uses a temporary ini-file
generated during run. So your existing ini-files are
not altered.

Additional parameters could be *AQ for adaptive
quantisation, a custom matrix and so on.

HCWrap doesn´t do any input validation. So be
careful that the parameters you add to HCWrap.txt
do not have any negative impact on the original
HC.ini

Always try HCEnc without HCWrap in case any errors
occur prior reporting any bugs.

I´ve tested HCWrap with the DVD Rebuilder v0.98.2 Freeware
and RB-Optv0.38. I experienced no problems with those two
tools.


You can grab it here:
http://www.megaupload.com/de/?d=CFZHQ2QC
or here:
http://rapidshare.com/files/127679914/HCWrap.zip.html

I´m aware that the functionality is quite basic for now but can easily imagine other enhancements I can do to HCWrap if it really makes sense. May be something like adding a custom filtering AVS to your source AVS (as with QuEnc^n), automatically adding logfile with autonaming based on source file and stuff like that.

Feel free to any feedback here in the thread :thanks:

Cheers,
D$

[Boulder]

What about multiple encoder processes? If I remember correctly, DVD-RB creates two temporary folders where it copies the encoder exe files. RB-Opt probably uses the same encoder exe for all threads as it only uses the CQ mode.

[Darksoul71]

Hm, I have no chance to try since I don´t own DVD-RB Pro. You could try and report back.
Since HCEnc is multithreaded anyway there is no real need to run multiple instances although
my benchmarks with HCEnc^n versus HCEnc multithreaded showed that a multiple instances
running singlethreaded are still a good bit faster than a single instance of HCEnc running MT.

The SMP implementation of HCEnc is very effective though.

You can specify *SMP with your custom INI settings to enable multithreading anyway.

[Boulder]

I was just wondering whether HCWrap can be used with the option enabled in DVD-RB..if it requires that the HCWrap.txt is in the same directory as the actual exe file, it won't work at the moment because the exe files are copied in two temporary directories.

Then again, jdobbs has added the feature already in DVD-RB and doing the redistribution is fast in RB-Opt anyway so I can run RB-Opt with one thread and SMP enabled.

[Darksoul71]

I was just wondering whether HCWrap can be used with the option enabled in DVD-RB..if it requires that the HCWrap.txt is in the same directory as the actual exe file, it won't work at the moment because the exe files are copied in two temporary directories.

Yes, this would be my very first guess since DVD-RB will "mistake" HCWrap with the original exe and simply copy it over to a separate directory for each running instance. I do pretty much the same for HCEnc^n.

I´ve added automated logfile generation based on the source file name and custom AVISynth filters which will be added to the source AVS. After some testing I´ll upload the new version.

In regard to run multiple instances you could simply use a different "install" for RB Opt.

Simply grab HCEnc^n, copy it over to the same directory where HCWrap resides and rename HCEnc^n.exe to HCbatch_org.exe

This way HCWrap will treat HCEnc^n the same way as the original HCEnc.exe and you can do redistribution with multiple instances. Since RB Opt uses the -ini parameter as only CLI command, it´s compatible with HCEnc^n.exe.

I think having two different directories of "HCEnc" for different purposes doesn´t hurt anyway, doesn´t it ?

[blutach]

I'll give it a whirl. Until a new version of DVDRB is released, I'm mainly interested in AQ and segment logging. Thanks Darksoul.

Regards

[Darksoul71]

Hi Les,

I´ll upload the newer version with automated logging and custom AVISynth filtering tomorrow evening.

Regards,
D$

[blutach]

Thanks mate :)

EDIT: It certainly doesn't seem to be working for MEP. Just did a prepare and hit encode and keep getting:

- Failed to create M2V: V01000000001001
- The encoder doesn't appear functional.If you untick multiple encoder processses, HC is launced and HCWrap.INI is created in the Encoders folder with the correct parameters. Note, the path to logfile must exist or it will error out. An offer to create the path would be good.

All in all, if the MEP issue could be solved, then this would be very interesting.

Regards

[Darksoul71]

@Blutach:
I´m aware that MEP doesn´t work now but I´ve found a simple solution to support MEP
in near future. I´ll store the installation path of HCWrap at the first run. After
this HCWrap can easily call the original HCEnc even if it´s copied over to multiple
directories.

The automated log and AVS script support worked fine though. I think I´ll release
the new version of HCWrap either today or tomorrow.


Note, the path to logfile must exist or it will error out.
An offer to create the path would be good.

Yes, that is indeed a valid concern but since I build the name of the log file based
on the source AVS (c:\mysource.avs -> c:\mysource.log) it doesn´t seem very likely
to me that the path doesn´t exist. What do you think ?

[kumi]

Hi,

MEP and HCwrap would probably be compatible if you package HCwrap + HCenc as a single VMware ThinApp executable. If you don't have a license for that software, perhaps you could do the same with Altiris SVS (which is free for personal use.)

Thanks for HCWrap, Darksoul71

[Darksoul71]

Hi all,

sry, real life is currently keeping me very busy.
I hope to release the new version of HCWrap
(with automated logging and custom AVISynth filtering)
this evening. If not I think it will take until
weekend.

Thanks to JDobbs I´ll now be able to test with
DVD-RB Pro and MEP.
:thanks:

@kumi:
Thanks for the idea using VMware ThinApp to package
everything around HCWrap.exe but I feel this is
really overkill. Also I think that it takes me less
time to do the further modifications to HCWrap
in order to support MEP (which will require of
course some more intensive testing) than it would
take me to evaluate VMWare ThinApp or Altiris SVS.

I´ve already a good idea what to change in HCWrap
to support MEP.

Cheers,
D$

P.S.: If anyone has ideas what to functionality to
add to HCWrap feel free to post here in the thread.

[Darksoul71]

Hi *,

I´ve done my changes to HCWrap and it´s now working fine with DVB-RB Pro and multiple encoding processes (MEP) enabled. :D

I did a DVD with MEP enabled and the resulting DVD working fine and had the correct playtime.

There are still a few tests to be done but I guess I´ll upload the modified HCWrap this evening.

Cheers,
D$

[Boulder]

Thanks, I'm very much looking forward to it. Your plugin will be useful on many occasions and not only in DVD-RB related things :)

[Darksoul71]

Your plugin will be useful on many occasions and not only in DVD-RB related things
Hm, now you´ve got my interest :)
May you enlighten me what further use you have in mind ?
*curiousity killed the cat* :D

[Boulder]

Nothing really specific at the moment. Let's just say that it might be useful in any tool that utilises HC, and also in my batch encodes - I can easily change the settings for a batch encode of some TV series, no need to touch the HC.ini or the particular job ini file anymore. The best thing is that it doesn't matter what options the future may bring; as long as they are adjustable in the ini file, they can be adjusted with HCWrap.

[Darksoul71]

Hi all,

here is HCWrap v0.2.

Here is the readme:

HCWrap - the HCEnc "ini" wrapper v0.2 - Released on 2008-07-10

Purpose:
HCWrap simply alters any given HC.ini file on
the fly and add parameters of your choice
(e.g. a custom matrix or something like *AQ).

This way you can customize HCEnc even if the
frontend which calls the encoder doesn´t supports
all the features you require.

HCWrap was coded with usage for DVD-RB / RB-Opt in mind
but should work with other tools using HCEnc also.

Features:
- Generate a customized HC.ini on the fly based on a template file.
- Add custom AVISynth scripting to your source AVS automatically
- Automated log file generation based on the source AVS name

Installation:
Copy HCWrap.exe and HCWrap.txt into your DVD-Rebuilder
encoder directory for HCEnc. In a default installation
HCEnc will be name HCbatch.exe

- Rename HCbatch.exe to HCbatch_org.exe
- Rename HCWrap.exe to HCbatch.exe

After this start HCbatch.exe (renamed HCWrap.exe).
HCWrap will then write it´s current directory to
the registry, show a little message box about this
and exit. This is only required once at the first run.
HCWrap will use this registry entry to determine where
the original HCEnc resides in DVD-RB multi-instance
encoder mode.

Configuration:
Edit HCWrap.txt to suit your needings.
HCWrap will add all your HCEnc parameters found in
the HCWrap.txt to the HC.ini given via CLI. Then
the true HCEnc.exe will be called with the altered
ini-file.

If no dedicated ini-file is specified then HCWrap
will alter an existing HC.ini residing in the same
dir or a HC.ini will be generated when no one exists.

HCEnc will then read the settings from the modified
HC.ini at startup. HCWrap uses a temporary ini-file
generated during run. So your existing ini-files are
not altered.

Additional parameters could be *AQ for adaptive
quantisation, a custom matrix and so on.

In order to use custom AVISynth filtering you simply
have to generate a text file called *MYFILTER* within
the HCWrap directory. HCWrap will then add this
AVISynth script to any source AVS given by commandline.

Input Validation / bug reporting for other tools:
HCWrap doesn´t do any input validation. So be
careful that the parameters you add to HCWrap.txt
do not have any negative impact on the original
HC.ini generated by the tool you use.

Always try HCEnc without HCWrap in case any errors
occur prior reporting any bugs.

Compatibility:
I´ve tested HCWrap with the DVD-RB v0.98.2 Freeware,
DVD-RB 1.27.3 Pro and RB-Opt v0.38

I experienced no problems with those tools.

Grab it here:
http://www.megaupload.com/de/?d=TK1EJHB7
or here:
http://rapidshare.com/files/128695711/HCWrap.zip.html

Cheers,
D$

[Darksoul71]

Grmph, those things happen when you are in a hurry :eek:

The file used for AVISynth filtering must be called MyFilters.avs

I left a placeholder called *MYFILTER* in the readme since I couldn´t
remember the exact filename when updating the file. :rolleyes:

[Boulder]

For some reason, the latest version causes all of the prediction encodes (in the bitrate redistribution) to run twice in RB-Opt. The first version does not do that.

[Darksoul71]

Hm, I´ll do some investigations.....currently I have no clue.
But I must admit that I only tested with DVD-RB Free & Pro.

[Boulder]

The problem seems to be that RB-Opt doesn't get the result from the first encode of each tested Q, nothing happens in the status window when the first encode ends.

[mikenadia]

Thank you very much for your add-on.
Same problem with HCwrap 2.0 (HCWrap 1.0 works like a charm) using DVD-RB Free without Rb-opt.
Also, could you confirm that with the Free Version, HCWrap.txt is not necessary (HC.ini is sufficient ) or is one method preferable to the other.
Thx.

[Darksoul71]

@Boulder:
For some reason, the latest version causes all of the prediction encodes (in the bitrate redistribution) to run twice in RB-Opt.
Grmph, at least this behaviour has an easy explanation. I changed a small part of the code which calls HCEnc after doing my test run in order to make HCenc always run at idle priority.


RunWait ($MyCL , "", @SW_SHOW)

$PID = Run ($MyCL , "", @SW_SHOW)
ProcessSetPriority ($PID, 0)
ProcessWaitClose ($PID)

Unfortunately I left the old "RunWait" in there. That´s why HCEnc was called twice. :devil:

I simplified a few thing. This should make it more robust.

@mikenadia:
Also, could you confirm that with the Free Version, HCWrap.txt is not necessary (HC.ini is sufficient )
or is one method preferable to the other.
With DVD-RB Free HCWrap will use the settings you specifiy in HCWrap.txt and generate a temporary ini file called "HC_Temp.ini" from scratch with your settings in there.

HCEnc will then be called with the -ini parameter even if this is normally not in the commandline.
Using HC.ini as intermediate ini-file would not be a smart idea since HCWrap wouldn´t know if the file is clean or has already been altered before. This way you would add your custom parameter to HC.ini by every time HCWrap is called.

Hope this helps....

@all:
Sorry for the hassle. I hope you understand if I do some more tests prior providing a new version of HCWrap.

-D$

[Darksoul71]

Ok, here for those willing to test:
An intermediate build of HCWrap (renamed as "HCBatch.exe" already).
http://www.megaupload.com/de/?d=TP9A66T7

At least the issues with RB-Optimizer seems to be fixed. Running the OPV prediction for a second DVD right how without any hickups.

[mikenadia]

Works perfect for me (I did not use Rb-opt).
Thx a lot again.

[Darksoul71]

Same thing here:
RB-Opt fninished the OPV prediction flawlessly and DVD-RB Pro completed the whole transcoding process.
The resulting DVD structure on HDD played back fine on my PC.

Right now (I´m currently at work) DVD-RB Free is running on the same DVD. I think this version should be stable now.

One final question to you as user:
Do you HCWrap prefer to alter the original files (AVS Source, Ini-File) or making local copies, alter them and delete them after completion of the run. I would currently prefer the later one since this adds stability. Accessing and altering the source files simply means that you could add your filtering script and the HC.ini settings over and over again.

What do you think ?

[mikenadia]

IMHO. Stability first.
Thx a lot again.

[Darksoul71]

Ok, I´ll then change HCWrap after this version to use a temporary AVISynth script and a temporary HC.ini

[mikenadia]

A couple of things to be aware when using HCwrap and DVD-Rb 98.2 and Rb-opt.
1) in Rb-opt, during computation of optimal Q, if I abort the process and restart it later, it tries to delete Working\D2VAVS\prediction\VTS_01_VobID_0001.log and cannot. (I have to go to the folder and delete it).
2) if there is a Matrix in HCWrap.txt ,the matrix selected in Rb-opt for computation of Q is overriden by the one in HCWrap.txt (normal behavior but can be misleading).
3) in Rb-opt, after having computed optimal Q, it computes some redistribution bitrates (the VobID_0001.log file is either not created or deleted immediately deleted by Rb-opt when it encodes the next segment and deletes the corresponding VOB file) so I cannot check which matrix it is using (the one in Rb-opt or in HCWrap.txt : probably the latter one)

One workaround (but may lead to other issues) may be is to create another Hcbatch (the original one) for Rb-opt.

[Darksoul71]

Hi mikenadia,

thanks for posting your observations and thoughts:


A couple of things to be aware when using HCwrap
and DVD-Rb 98.2 and Rb-opt.

I tried only DVD-RB Pro together with RB-Opt. So it´s good you tested this combination also.
BTW: During my tests with DVD-RB Free I found it nice to use the *SILENT parameter in order to get rid of HCEnc GUI interupting my daily work.


1) in Rb-opt, during computation of optimal Q, if I abort the process and restart it later, it tries to delete Working\D2VAVS\prediction\VTS_01_VobID_0001.log and cannot. (I have to go to the folder and delete it).

Rb-opt tries to delete the log file and it´s blocked ? That is really strange...
Normally HCEnc generates this log (although it´s added by HCWrap to the ini-file). So if you close HCEnc after RB-Opt has shutdown HCWrap (the AutoIt tray icon will then vanish) simply close HCEnc manually and you should be done (unless you are running with the *SILENT parameter added to HCWrap.txt). If HCEnc is running hidden then you have to terminate the thread via Task Manager.
I stopped the OPV prediction phase a few times during my development and never experienced this kind of problems. Strange :confused:

If the log turns out to be the root of problem I can still add an ini for HCWrap to disable automated logging. Will do some tests on this one....


2) if there is a Matrix in HCWrap.txt ,the matrix selected in Rb-opt for computation of Q is overriden by the one in HCWrap.txt (normal behavior but can be misleading).

True, but in order to avoid this kind of problems I would have to write a complete HC.ini parser which should be pretty time consuming. There are other parameter which can cause similar problems.
I´m not shure (since I´ve never been an extreme RB-Opt user) but I think you can disable the use of any matrix in RB-Opt. If this works my route to go would be the following:
Disable any custom matrix within RB-Opt and let HCWrap always add your custom matrix to the ini-file.


3) in Rb-opt, after having computed optimal Q, it computes some redistribution bitrates (the VobID_0001.log file is either not created or deleted immediately deleted by Rb-opt when it encodes the next segment and deletes the corresponding VOB file) so I cannot check which matrix it is using (the one in Rb-opt or in HCWrap.txt : probably the latter one)

I´ll use my "debug" version of HCWrap which protocols the given parameter and ini-file in order to find this out. My first guess: If RB-Opt uses always a matrix then HCWrap will simply add the matrix specified in HCWrap.txt
I´m not shure how HCEnc then handles two matrix within an ini file......


One workaround (but may lead to other issues) may be is to create another Hcbatch (the original one) for Rb-opt.

I think this would be the easiest workaround I guess.....


Just some "sneak preview" for the next version:
I´m currently writing some sort of HCEnc CLI to HC.ini converter as part of HCWrap. This will translate all CLI parameter to an ini-file which will make things more stable and easier for me to track down problems. As side effect this will also make HCEnc^n CLI compatible with DVD-RB Free (which I had on my todo list anyway). Beside this I still have an open ear for feature suggestions and requests.

I still have a few ideas in the back of my head but are unshure wether to integrate them into HCEnc^n, HCWrap or simply write another "layer" for HCEnc, e.g. CQ prediction independently from frontend, adaptive filtering, etc.

Some sort of frontend similar to CCEFront but for HCEnc came to my mind...hmm......:cool:

Cheers,
D$

[mikenadia]

1) in Rb-opt, during computation of optimal Q, if I abort the process and restart it later, it tries to delete Working\D2VAVS\prediction\VTS_01_VobID_0001.log and cannot. (I have to go to the folder and delete it).

My mistake.It is because the Abort button in Optimal Q prediction does not kill HC (paused in the background and minimized) and that's why the error message was appearing when running another prediction. Thx for the *SILENT tip. Solves the issue.

[Darksoul71]

Hm, I´ll add the HCEnc^n exit function to HCWrap which kills HCEnc when the AutoIt script itself is closed. This should solve those problems....

[mikenadia]

I´m not shure (since I´ve never been an extreme RB-Opt user) but I think you can disable the use of any matrix in RB-Opt. If this works my route to go would be the following:
Disable any custom matrix within RB-Opt and let HCWrap always add your custom matrix to the ini-file.

Does not know if it makes sense or not.
Ability to use HCwrap with different HCwrap.txt. I am going to use Rb-opt as an example (but could be FAVC, AVStoDVD...).
Rbopt.ini is splitted between Rb-opt.ini (specific to RB-opt) and a HCwrapRbopt.ini that Rb-opt will call with specific parameters. That may be a solution to conflicting parameters (Matrixes...)

[Darksoul71]

Hm, multiple templates make sense but how should HCWrap know which template to use ?
I could add an ini-file for this but to me it seems more complicated than have 1, 2 or more
concurrent versions of HCWrap in different folders which you can reference with each tool.
What do you think ?

[mikenadia]

Agree for simplicity (different folders). Also, what do you think about this: When the output file exists already, rename the existing one to *.bak and delete it at exit (end of 2nd pass). Appreciate very much your work.

[Darksoul71]

Hm, nice idea but the new version of HCWrap will make copies of the SourceAVS and the given ini.
Additional to this there will be a debug mode which adds every ini and avs file to a log. This will make
things easier to follow even for the standard user..

I´m still not shure how to give the user control over this debug feature. I find it stupid to add an ini-file
simply because of this debug setting although an HCWrap.ini might make sense to configure future options.

[Darksoul71]

Hi all,

I´ve just uploaded the newest version of HCWrap here:
http://rapidshare.com/files/132186050/HCWrap.zip.html
and here:
http://www.megaupload.com/de/?d=M8566K3E


What´s new:
HCWrap now generates an ini file called "HCWrap.ini"
in it´s base directory. The file is pretty selfexplaining.

[Common]
Debug=0
Debugfile=C:\HCWrap.log
UseLog=1

Debug=1 enables extended logging (for debugging purposes).
Debugfile -> specifies to which log file the information is written
UseLog=1 enables the automated log-generation for HCEnc.

Additional to this both the source AVS and the given HC.ini are
copied to temporary files prior being altered. This leaves the
original files untouched and should avoid problems such as
multiple filter scripts being applied to the same file after re-run
of HCWrap.

After some testing I found out that HCEnc MT runs as fast as
HCEnc^n. Personally I do not see a big advantage running
multiple instances of HCEnc anymore (e.g. MEP with DVD-RB Pro
or Rumbaah´s Tool for DVD-RB Free). I really ask myself if it´s
really worth the hassle to do all this stuff (e.g. storing the base
path in the registry to be able to call original HCEnc after being
copied by DVD-RB).

May be I should remove this stuff. This would make HCWrap
much more flexible, e.g. having multiple versions on the same
PC for different programs.

What do you think ?

Cheers,
D$

[Boulder]

Using SMP in HC is very fast if the script is fast. However, with complex scripts, most of the time is spent in the Avisynth processing (i.e. video decoding) thread and the actual MPEG2 encoding takes very little time. In these cases, multiple encoder processes give a very big performance boost on multicore systems.

[Darksoul71]

OK, granted ! I´ve not used very complex scripts for my test :D
I´l try a more complex script and run HCEnc^n versus HCEnc MT again ;)

[mikenadia]

[QUOTE][From Darksoul71: I asked hank to output a DBS file even if you do only CQ mode. It would be even nicer if we could output a lossless file as well.

How it could work:
- Tool starts prediction
- HCEnc runs in CQ mode but outputs a lls file and a db file as well (specified by the tool)

If the resulting file overshoots the target size it could be either shrinked with Rejig / Requant (for up to 5% may be) or be transcoded in a "second pass" using the lossless file from the previous run as source.

What do you think ?/QUOTE]

1) Following your different posts and Manolito algorithm in
http://forum.doom9.org/showthread.php?t=138566&page=2,
in case of overshooting, you can keep the first 80 % of the encode (which will probably include the main movie and that % may depend on the overshooting pct) and reencode , using the lossless file , the remaining 20 % (or starting at segment No 25...) and taking into account the information of the overshooting(s).

2) It may be confusing, but you may decrease probability of overshooting by encoding the segments fron the larger (bitrate*nb_frames) to the smaller.

Edit: Would be nice if we could have a flag ( may be it is already there) that the segment has been encoded entirely (all the encodings stats are there) so when there is a batch, it can start back from the last " encoded completely" segment.

[Darksoul71]

Hi mikenadia,

hm, re-encoding just a segment of the movie to make it fit a certain size
is definitely another approach.

Based on manolito´s good explanations I´ve already coded a "segmented CQ encoder".

The approach is really simple I think I´ll discuss this with hank if I´ve done
more tests.....

Using 20% as clamp borders (as discribed by manolito) worked fine for me...
All 8 movies I did were within +/- 2% border of their target size. Quite nice.

Would be more effective if integrated into HCEnc. Some sort of size-adaptive Quantizer
mode. Essentially it´s simply one pass VBR....

Cheers,
D$