BatchEncodeM2V 00014c alpha with HCENC support

[Trahald]

Ive been coding off and on an application that will do one-pass encoding and also support apps other than cce pro sp. well, here is my first effort at it. seems to work so far, but please use it for non-mission critical applications for now.

It supports Quenc (59a, 59b). Also one-pass through cceroba (made by Tylo [d2sroba]) . and traditional multipass cce. procoder support and cce basic support will follow soon.

cceroba works best with cce 2.67.x..

quenc is supported by splitting the .avs file as it encodes then it merges it together.

This also supports adding pan_and_scan and 3:2 pulldown through use of pulldown.exe (named revpulldown.exe) that i modified that is included (run it w/out commands to see usage)

available at http://bui.dvd2dvd.org (link is on the left)

thanks again to tylo for cce roba.. dvd2svcd for the hosting and d3s7 and the gang for coding help suggestions.

<00014c> - the way i set the ecl header looked for 'version' but that didnt always exist. 'standalone' seems to be in all productnames (basic, 2.50 2.67 2.70 etc) so using that seems to have fixed the 2.50 issue.

<00014b> - fixed issue where -removesequenceendcodes being sent with -nooutput created a vif that matched the 'virtual output file' but not the source .mpv file as it should. revpulldown now ignores -removesequenceendcodes when used with nooutput. (same download filename)

<00014> - Added - VIF creation. Optional parameter (default to on i think) creates a vif file after encode through revpulldown.exe. For pan&scan or pulldown material the vif file is ALWAYS created regardless of the setting since it only takes ~1 second to create. Since for all other cases (PAL/NTSC-30fps) it is an extra step during encode so it is optional. (Worth the savings at import/compile time still IMHO.)
- HCENC is limited to 12batch(and i presume above).. .01 will work but will be wrong for bff material (which is rare) 12batch and above support setting field dominance in the .ini
- Added Support for CCEFRONT by tylo. It can be used as a replacement for CCEROBA for one pass encoding. It supports cce 2.70.. i have it properly scan the .ini to check the cce version (you still will use ccefronts cce version selecter) so run it once before using with batchencode.
- ... uhmm... im pretty sure some other stuff

<00013> - Added - Maximum GOP size param. (Default to on.) It will give an asset the longest GOP possible given its framerate (while still keeping M = 3. ) Its decision is made when the ccedata file is loaded. It makes 29.97fps items have a 18 frame gop (3/6) PAL is given a 15 frame GOP (3/5).. film can only be given the standard 12 frames( 3/4) because 3:2/2:3 pulldown process creates extra 'virtual' fields. Tested with cce seems to work so far.. hcenc/quenc/cceroba should follow suit. i'll test procoder and ccebasic later. (some say longer gops, specifically, less 'i' frames give better output )
- Fixed - repeating lines in temp avs creation. it didnt happen with doit4u's .avs created files but may happen with homemade ones.. now fixed.
- Uhmm.. other stuff

<00012c> - Fixed - allows for long PathNames. Prior long pathnames would be written to registry correctly but read back incorrectly. buffer extended to allow for more characters. Same Filename for download

<00012b> - Added support for HCENC 0.12 batch. Same filename for download

<00012> - Some small stuff. Added ability to set vaf to not overwrite (in case you wanted to do additional passes)
- You can highlight multiple items and set them as complete or delete them.
- Revpulldown version .99j included adds a -addfieldend option that will add a rff to the last temporal frame that doesnt already have one. this is for the purpose of helping with .5 errors on NTSC pulldown material. I dont actually use the new option for batchencode (addfield and 3:2 pulldown are used when needed to avoid .5 errors) but i figured i'd add it as an option..
- Other stuff (i think)

<00011c> - Im setting the ECL file header how zeul suggested (Based on the Product Name for that cce version) for all CCE versions. It seems to work fine.. please do a short test before you run a full encode. Same filename for download.

<00011b> - Found out the hard way that 2.70 treats drop_frame flag different than other versions with 23.976 fps source material (it shortens the file by 1/5) sooo.. this version fixes that by setting drop_frame=0 when encoding 23.976 (i have pulldown.exe always set dropframe ON for ntsc anyways). i tested it and appears to be ok.. i'll truely know in a few hours. this version is up for the testing with same install name (00011) on bui.dvd2dvd.org.

<00011> - 2.70 works now. since 2.70 actually lets you set field dominance i use that parameter instead of the offset line.

<00010> - added support for HCENC 0.01. it splits the file for chapter accuracy (although again you can shut the option off and encode as one file.) it uses most of the options.. any missing you can edit in the hc.ini. I'll add more in next version. HCenc encodes everything as tff. i adjust for that with pulldown (only in the rare time your mpeg is BFF). it also encodes with a start time of 01:00:00.00 which i also adjust for with pulldown. just point the exe to hc.exe. hcenc is still pretty new so use with caution.
- little gui tweaks and code tweaks

<0009> - small update..added some statistics for the AVS file including estimated file size. also a progress meter for when files are being joined (when applicable). also dropdown for changing DAR in edit mode.

<0008> - fixed issue pulldown was not always selected when it was supposed to. my aviframerate routine is fixed and should be ok now. thanks nevs5.

<0007> - includes updated pulldown 099h - fixes a long standing issue with pulldown that truncated files w/out a sequence end code. and fixed an issue with the -ps_add command (pan&scan)
- adds support for procoder (havent tested it much but seems ok)
- assures file names of encoded files are correct for what scenarist/reathorist expects.
- added option to not have encode (for procoder,quenc, cce basic) split by chapters and merged. with it on it encodes the file whole. this means chapters may be off which may be important for BOV and also may cause 0.4 errors. however its added by request (robot1) since generally it is harmless and the final file will have better quality.
- uhm... other stuff.

<0006b> - fixed PAL frame_rate_idx setting for .ECL files (hopefully)

<0006> - now supports CCE Basic.
- fixed ecl settings with PAL stuff.
- misc stuff


note: for pan and scan.. scripts created by scenaid and reathorist for pan and scan assets will flag the files as plain 16:9 (since pan&scan was never encoded in the past).. so if you decide to use the pan and scan feature you must alter the script for now.

[Darklistener]

Hi
nice tool Trahald.
I will give it a try the next few days.
Will this tool replace BatchCCE sometime?

Supports this tool CCE SP 2.70.x..

thx

[influenza]

The tool is intended to replace batchccews some time. Unfortunately the support for batchccews seems to have stopped so we should be glad people like trahald step in :)

Thanks Trahald!

[Trahald]

it probably works with 2.70.. i havent tested it yet. give it a shot :)

[buzzqw]

can it be used without the Big 3 ? i tryed with no success.

i had cofigured it on al basis, but i cannot feed avs or mb target size ... :(

thanks

BHH

[Trahald]

I really intended it to be used with big 3 however.. i did code for adding .avs individually but i did it badly so i disabled the menu option (it doesnt setup all the things it needs to).. it will be reenabled soon... for now you can make a cce-data.txt file in notepad for the file(s) you want and import that (there has to be quotes around the file name.. for now)

[buzzqw]

@Trahald

Ok, thanks anyway ;)

BHH

[Eyes`Only]

this imports standard ccedata.txt scripts (versions 1 and 2)?

[influenza]

Didn't know there were two versions, but yes :)

[Trahald]

yes.. dif4u's 1.0.0.0 or 1.0.1.5 and 1.0.1.6 (has angle parameter)

[influenza]

hmm Ok. Never too old to learn something new ;)

[D3s7]

good thing or you'd be in trouble ;D

[Trahald]

ok 0006 is up.. if you have time between mission critical backups please give it a shot for testing. thanks

[Trahald]

oh and procoder support should be in next release.. procoder uses same ecl as cce so adding should be fairly simple.

[techniquefreak]

Hi Trahald I'd like to thank you for coding such a great application. I've yet to test it thoroughly though - been a little busy lately. Anyway - thanks much and keep up the good work!

Kind regards

/Hansen

[Zenak]

I gave this a shot last night, and the results came back very impressive.

I was wondering though, I chose CCE MultiPass, but you have the single pass otption. Is that for CCE? If so what is the trade off in quality for the extra speed?

[Trahald]

glad to hear it :). if you are referring to the 1-pass option in the main screen then thats for quenc only . if you are referring to the 1-pass option in the option menu then.. yes.. that is for cce.. (its only for SP versions not basic) basically i use tylos Cceroba to have it predict the proper Q parameter and then it runs a 1-pass OPV based on that Q. if its too over or too short it runs a second pass ( i will include a setup menu for cceroba at some point but for now the .ini is editable ) i set it up to allow for decent undersizing (as low as 92% or intended size) to try to avoid the resizing second pass, but with no tolerance of oversize.

[Angelus]

Just a quick question...when I tried out BatchEncodeM2V and when I try to start a job, the CCE windows loads and starts, and in the System Toolbar the CCERoba icon pops up but says that script is paused and flashes an "X" in the tray...is this normal?

Edit: Also one request possibly in the next version...is it possible to put a minimize button on the main program? Or a minimize to tray option? Not a biggie but thanks for a great program!

[Trahald]

when you right click a program using auto-it's tray icon (as cceroba is) it automatically pauses.. so if you ever want to right click it feel free.. but make sure you un-pause it

minimize button? sounds easy... should make the next version if not the one after that. (hopefully get some time this weekend.. been working on sum stuff for batchupdateifo)

[Nevs5]

i am test out the program now. i look at batchcce and it show me that pulldown is check of when i import the ccedata.txt file and in your program i import the same ccedata.txt file and pulldown is not check off . so what one is right? yours or batchcce?

[Angelus]

Ok I was just making sure because whenever I start an encoding job, the CCERoba icon flashes between and X and its symbol, which I thought to mean that it was paused. And when I right clicked it said "script paused". So is it normal for CCERoba to flash between the X and its icon when it's working? If so then forget my ignorance. Thanks

Edit: I take that back now, it seems CCERoba is working correctly now. The only problem I have is that I can't get the result to compile using the Scenarist Script created by Scenaid. It seems that all the titles that were encoded arent the ones that the script is pointing to when it imports the files. This is a sample of the error's im getting:

Error Language name is not specified in VideoManager
Error Aspect Ratio<16:9> in this PGC<VTS1_Title_PGC1> is different from the one<Unknown Aspect Ratio> in Title:<VTS1_Title1> VTS:<VTS_1>
Error Aspect Ratio<16:9> in this PGC<VTS2_Title_PGC1> is different from the one<Unknown Aspect Ratio> in Title:<VTS2_Title1> VTS:<VTS_2>
Error Aspect Ratio<16:9> in this PGC<VTS3_Title_PGC1> is different from the one<Unknown Aspect Ratio> in Title:<VTS3_Title1> VTS:<VTS_3>
Error Aspect Ratio<16:9> in this PGC<VTS4_Title_PGC1> is different from the one<Unknown Aspect Ratio> in Title:<VTS4_Title1> VTS:<VTS_4>
Error Aspect Ratio<16:9> in this PGC<VTS6_Title_PGC1> is different from the one<Unknown Aspect Ratio> in Title:<VTS6_Title1> VTS:<VTS_6>

And the only VTS's that I've encoded are 5, 7, 8, 9, and 10. I have Scenaid set to automatically import if the assests are already encoded, and when I try to have Scenaid do this, it opens up BatchCCEWS rather than Scenarist...I guess meaning it doesn't find the encoded assets in the folders. Perhaps a different naming convention? I'm not sure... Thanks

[muppets]

First of all: Thanks Trahald for coding this application!
I've tried to encode an interlaced video with both CCE Multipass and CCERoba but it seems that it doesn't matter to BatchEncodeM2V if "Progressive" is checked or not. In every *.ecl file so far it says "progressive=1" :(
Another thing i was wondering about: When using CCERoba, there is a *.temp.AVS.ecl file created but not used by CCE, only cceroba_opv.ecl is used. Is that the way it's supposed to work?

edit: in addition to create a *.vaf file there's also created a *.mpv file (that's way bigger than the desired size) in CCERoba's first run. I haven't seen that with BatchCCEWS, so i was wondering again, if that is the correct behavior of the program?

edit2: please ignore my first edit :eek:
Thanks muppets

[Trahald]

muppets and angelus.. can you please send the log (they are in the install directory) i'll pm you the address

muppets.. interlace error is my fault. fixed in the next version.

angelus.. well.. when pulldown is used (for p/s) it renames the file to .mpv.m2v (same as when pulldown is used to change framerate to 29.97).. since scenaid isnt expecting it that could be an issue. i already corrected that in my version a few days ago. i think i'll release it today although i havent tested it as much as i normally do.

as far as how cceroba works file wise.. it makes its own .ecl from my .ecl and makes sample files until it figures out the correct 'q'. then it makes an ecl that it will use for the final encode. then creates the mpv/vaf at the same time. if mpv is not within size limits then it does a regular sizing pass using the vaf file.

[Angelus]

@Trahold

I actually did finally get the project to work in Scenarist, all thought im not sure what I did :) I'm pretty sure it was a problem with Scenaid rather than BatchEncodeM2V. I think I ran Scenaid a couple times, and on like the 3rd time when I tried importing, it worked fine. I was trying to think about what I could have done but I have no clue.

As for the whole naming convention I think it is correct. Somehow Scenaid (or my directory) got messed up and it would launch BatchCCEWS rather than Scenarist. And in Scenarist it does import the right assets that were encoded with your program, I just think it makes the file names uppercase when it displays them. Sorry for any confusion I caused. But thanks for your nifty little app, so far it's been great!

[Trahald]

New version posted 0007 alpha. change list in top post.

[Trahald]

@angelus

ahh.. cool.. hehe.. well future versions of scenaid will have option to launch batchupdate.

[Nevs5]

very nice program Trahald :)

just a little confused still, not sure if this has to do with the pan&scan. but the new version still does not check off pulldown like batchccews does, not sure if it should. i load the ccedata.txt into batchccews and pulldown is check for most of them. then i load ccedata into your program and i get some that are check with pulldown and some that are not. not sure if it's a bug or if batchccews is wrong.

thanks

[D3s7]

BatchCCEWS did a pulldown internally which is probably the switch you see

BatchEncodeM2v uses trahald's version of pulldown.. there is no flag in it (that I'm aware of) - it just does it if you encoded an asset at 23.976

what your seeing is probably just a visual thing

[muppets]

@Trahald: I sent you the log as requested. Just out of curiosity, is any valuable information for you in there?

muppets

[Trahald]

@muppets
absolutely none.. lol.. i have to fix that.. i kinda thought i did some things that i must not have gotten to :). but really.. rereading your post the main issue was resolved by the new version.. the progressive part of ecl creation was wrong.

nevs5 - easy way to check.. just get virtual dub or some derivitive and load each .avs in the cce_data. check the file information in the file menu. if the framerate is 23.976 then pulldown should be checked. if the framerate is 29.97 then it should not.

[Nevs5]

k, i will check it out.

little addition if possible:
if you could add an option to run cce at low priority and minimize to tray. there just cosmetic addons so no big deal

[Nevs5]

hey Trahald

i just check out what you said. now if it is 23.976 pulldown should be executed right? if so in your program should the pulldown box be checked off or is it done with out us knowing?

cause i am doing a rip that virtual dub is say that all my avs in ccedata are 23.976 but, then i look at your program and see if only 1 of the avs has the pulldown check next to it. there might be more to this that i don't know, i am just want to make sure that it's not a bug.

[Trahald]

if you could add an option to run cce at low priority and minimize to tray. there just cosmetic addons so no big deal

sure

[Trahald]

Originally posted by Nevs5
hey Trahald

i just check out what you said. now if it is 23.976 pulldown should be executed right? if so in your program should the pulldown box be checked off or is it done with out us knowing?

cause i am doing a rip that virtual dub is say that all my avs in ccedata are 23.976 but, then i look at your program and see if only 1 of the avs has the pulldown check next to it. there might be more to this that i don't know, i am just want to make sure that it's not a bug.
ok.. Seems that because i never use decomb (i leave things interlaced) i didnt realize the avi info sends the framerate different when decimate is used. (as compared to straight 23.976 FILM .) so i updated it to get the frame rate the right way so it works now. will be fixed in next version

[Nevs5]

thanks.. i am getting the wrong size file of my mpv but i think it will be fixed when the pulldown is fix cause the one file that had pulldown check off was the right size

[Trahald]

Originally posted by Nevs5
thanks.. i am getting the wrong size file of my mpv but i think it will be fixed when the pulldown is fix cause the one file that had pulldown check off was the right size ok 0008 is up. should fix the issue

[Nevs5]

seem to be working great now thanks for the fix

another add-on: if you can add a priview of what the size of the new m2v file would be.

[Trahald]

sure.. added to todo

[trigger638]

This is very very interesting and definatly needs some answers. I have been working on a very complex title. The only way to properly do it is by demuxing by cell id. The title is seamlessly branched, and as you know when you demux by cell id you have to set the seamless/non-seamless flags in scenarist manually. Fortunatly it is pretty obvious what is seamless and what is not.

Most of my assets were causing errors (uneven number of fields) when seamlessly connected. This can be corrected by chopping a few frames off the asset until the compile stops failing. Unfortunatly when this is done is causes a pause in playback on a settop player between the cells that were altered. I was using doitfast4u followed by batchccews to get my assets.

I did a bunch of research, at first concentrating on GOP as the possible cause of the scenarist errors. That basically got me no results. There was no doubt that somewhere after doitfast4u finished and the completion of batchccews was the problem.

I found your new tool here and had read that you had done some work with the pulldown utility. I loaded the cce data file into your tool and ran one of the problem cells. I ran a quick recompile using assets before and after the reencoded cell in the exact same config that was previously causing uneven number of fields errors. Much to my surprise the error no longer exsisted.

I will be running a full reencode using your tool tonight to see if I will get the same results with the remaining problem cells, but I suspect that I will not see that error again.

If it is possible could you please try and explain why your tool is not producing this error while batchccews appears to?

Regaurdless of your answer to my question it appears you may have solved a very common and serious problem with prepaired assets. Well done.

Thanks in advance.

[D3s7]

The answer to that question dives into the core of how pulldown functions...

put simply, every 4 original frames = 5 new frames... and can be done in a 2_3 or 3_2 pulldown
2_3 : AA BB CC DD = AA BB BC CD DD
3_2 : AA BB CC DD = AA AB BC CC DD

now that'sa complete cycle.. now what happes though if you don't have 4 originally.. what if you end up w/ 1,2 or 3.. that's where your .5 frame errors come in...

put simply. here is a grid on what it looks like
Top = original fields(2fields = 1 frame)
Next is pulldown method, what the resulting file looks like, what scenaristsees



AA
3:2 AA Ax (1) - AA - final field is dropped in Scenarist
2:3 AA (1) - AA - Ok as Is

AA BB
3:2 AA AB Bx (3) - AA BB Bx - final field is filled in, in scenarist
2:3 AA BB Bx (2) - AA BB - final field is dropped in scenarist

AA BB CC
3:2 AA AB BC CC (4) - AA AB BC CC - Ok as is
2:3 AA BB BC Cx (4) - AA BB BC Cx - final field is filled in

AA BB CC DD
3:2 AA AB BC CC DD (5) - AA AB BC CC DD - Ok as Is
2:3 AA BB BC CD DD (5) - AA BB BC CD DD - Ok as Is


If you notice, there are places that there is an "x" - this is an odd field or, .5 frames - in the example of a 3_2 pulldown w/1extra frame, scenarist seems to drop that odd field.. butif you tried tomux that.. you'd get .5 frame error.

What BatchEncode does is based off your framecount before pulldown, auto picks the 2_3 or 3_2 pulldown method.. which will elliminate about 90% of the .5 frameerrors (2 leftover is the only problemone)

Hope that helps

BTW.. whatmovie are you working on thatcould POSSIBLE HAVE to bedone by cell demux???

[recoil47]

Trahald,

A couple questions, I just saw in another thread that you have been coding this to replace BatchCCEWS in the Big3 process.

1) Does this program support CCE 2.70.x ?

2) Can I assume that when this goes from Alpha to a Beta that there will be a guide as to how to incorporate it into the Big3 (for newbies like me)? :)

I'm very interested in the single pass CCE options. Right now I use Big3 with Multipass and it takes up to 5 hours to encode. If this program helps cut that time down and keeps the quality the same, I'm all for it.

Thanks,
Chuck

[influenza]

Hmm what title needs demux by cellid? I never came across a title that needed that and am always curious of ocurse ;)

[Ripe73]

Maybe a porno :D

[trigger638]

No, actually the title is the forgotten. The details of how it is layed out are described in another thread called Chapter point question, started by me. In that thread I state that scenaid crashes when processing the asset folder. This is not totaly accurate. It was my first encounter with the new sony protection, and after reading the hang-ups with that I did get scenaid to process the asset folder.

Basically it is seamlessly branched, the title vob contains the entire theatrical version of the film followed by a couple of deleted scenes and then the alternate ending. So after the credits they just threw in a few scenes and the alternate ending. What is happening is that the chapter points are off ever so slightly. In this particular case it is critical that they are exact. In the extended playback, when it jumps to a deleted scene, I get a few frames that are out of place at both the begining and the end of the playback. Basically even if the scene times are a few frames off from the original, it will be noticed in the playback.

I tried several times to recompile with scenaid. It appears that the chapter points are not accurate enough to deal with this scenario. I also attempted to do the chapter points manually, and as it turns out I am not accurate enough not to be noticed in playback. So my conclusion is that the only way to get exact assets would be to demux by cell id. Of course getting uneven number of fields errors while attempting this was not helping.

I just finished totally recompiling my project using the assets created with BatchEncodeM2V, not one uneven field error. Had some audio length problems (too short by a few frames) but those were easily corrected.

The project plays flawlessly on my comp and on my settop with the exception of the pausing when switching between non-seamless scenes while playing the extended version on the settop (expected due to no seamless branching support).

Thanks a ton for the response to my question D3s7, I understand it much better now. This autodetect pulldown functionality is a must have, fantastic feature. As always thanks for your efforts, Trahald, on this awsome new addition.

If you need any additional info on troubleshooting why scenaid is having problems with this title I would be more than happy to assist.

Thanks for all the help.

[Trahald]

I was able to do 'the grudge' again with vobid demux.. worked fairly well.. but with some minimal scenarist tinkering
(the dummy areas audio was a tad too short. the result plays and navigates fine. make sure you have the option to have scenaid process the pgc and vobid audio with delay_cut.. this fixes some crc errors in the audio

[D3s7]

I have to agree w/ trigger...

after looking closer, VTS8 has to be done by Cell.. either that or your going to waste quite a bit of space

Guess I'll have to code in support for cell demuxes... either that or we need to come up w/ a sound way to mix types... partial PGC/VOBID partial cell when needed...

[Trahald]

ok.. batchencode 0.0.0.10 released.. info in top post.

[Trahald]

here is a replacement for revpulldown (put in your batchencode install directory) on large files (about 2.2 gb+) it would add a second sequence end code without getting rid of the old one.

[Darklistener]

Hi
I allways get this error msg popup:
vts__01_p01.i-tff.16~9_1.avs.temp.AVS.ecl is not a ecl for cce sp works version

I use CCE SP 2.70.01.05

[Trahald]

i never tested the app for 2.7.. i installed it this morning and will test when i get home. i think its a simple thing with the header.. i should have a fix out tonight.

next few things i'll be working on is to add some features specific to encoders.. ie matrix and any other options..(for any encoders that support matrix) i am thinking about implementing templates .i will allow setting the encoder options out to the front screen and you can set them for each file.. meaning you can setup doing your main movie with cce and say extras with procoder) the templates will be one per encoder option (1-pass cce, multipass cce, quenc, procoder, etc) and anything else that comes to mind or is suggested. the selection in the option menu will be your default template/encoder.

that will probably be added after i start calling it beta (in a version or 2) cce 2.7 support will be in the tonites alpha