New tool: Enc (encode, compressibility test, size prediction with almost every codec)

[jonny]

http://jonny.leffe.dnsalias.com/enc/

This tool is able to encode, make a size prediction at a given quantizer and make a compressibility test with (almost) every codec.
As input an AviSynth file is required (the file doesn't have to contain "return" or "end" in order to make compressibility test or size prediction).
It's structured in a way that make it work with every codec without the need to be modified.
You can also put multiple jobs in the joblist and save/restore both settings and joblist.
Needs VirtualDub for working (the latest VirtualDubMod version is strictly required if you are going to use size prediction or compressibility test).

First time you run this tool, be sure to open the help menu to see the nice people that helped me (i'll never be able to thanks all as i should... but i try :)).


Some important notes:

Every time you update to a new codec version, you must trash all the settings/jobs saved for the previous version of the codec (codec data structures could be changed, so previous settings doesn't work any more).
Another good idea is to restore codec defaults settings (only the first time you open the codec settings interface).

If you are going to make compressibility test or size prediction with XviD using bframes, be sure to use a version after 07-03-2003.

There are still some XviD bframes configuration where the error in the size prediction is high (something like 7%-8%), i'm working on this problem night and day :D.

Using AviSynth 2.5 + MPEG2Dec3 v1.00 seems to leak a lot of memory if you run a lot of jobs or you get AviSynth info on the file.
A reference can be found here:
http://forum.doom9.org/showthread.php?s=&threadid=47970
Nic have solved this problem, be sure to check this thread:
http://forum.doom9.org/showthread.php?s=&threadid=53164

Opening 32 times the DivX codec control panel crash the app on Win98SE (this doesn't happen with XviD or with DivX+Win2000 ... DivX bug?)

It's all for now, i hope this tool will be useful :)

[kaitsuburi]

Awesome timing! I just posted at the GKnot Dev forum last night about problems loading B-frames XviD stats files in Gknot:

http://forum.doom9.org/showthread.php?s=&threadid=50682

I used DivX Enc for quite some time before switching to XviD. I am anxious to try Enc when I get home this evening.

Many thanks for your work!!

-kaitsuburi

[kxy]

Originally posted by jonny
http://jonny.leffe.dnsalias.com/enc/
This tool is able to encode, make a size prediction at a given quantizer and make a compressibility test with (almost) every codec.
As input an AviSynth file is required (the file doesn't have to contain "return" or "end" in order to make compressibility test or size prediction).


You mean the script CANNOT "return" or "end" function?

Just tested with xvid with b-frame, and it works, give me a day to do a full encode and we see how much it deviate.

feature request: a slider for output resolution, so if the compression is too high or too low we can change it accordingly.

[hakko504]

Originally posted by kxy
You mean the script CANNOT "return" or "end" function?

Just tested with xvid with b-frame, and it works, give me a day to do a full encode and we see how much it deviate.

feature request: a slider for output resolution, so if the compression is too high or too low we can change it accordingly. Exactly: the correct line should read:As input an AviSynth file is required (the file cannot contain "return" or "end" in order to make compressibility test or size prediction).

[jonny]

@kaitsuburi:
Thanks! (i've switched to XviD too :))

@kxy:
Usually for testing i (helped a lot by betatesters) do a serie of 11 jobs, running size prediction with 1%-10% of the movie and the last at 100%.
You can find some results here:
http://jonny.leffe.dnsalias.com/enc/sp/xvid.htm (there are still some tests left and i'll go to update the page soon)
This version have an option keep/delete a special log file.
If you have the time to run 11 jobs and you are able to send me the joblist+11logfiles, this will be a really helpful for me.

[Sharro]

Originally posted by kxy
...feature request: a slider for output resolution, so if the compression is too high or too low we can change it accordingly.

That would be great!! If we could have also have:

-aspect ratio into consideration
-different resizing filters
-cropping

I would not need anything else and be a happier person :-))))

Great job for a small and simple program without any dependence on .net.

All the best,


Sharro

PS: Will run a few tests tonight as I have already a few full first pass lying on my hd.

[jonny]

Thanks Sharro.

Yep, doing an integrated AviSynth generator will be great, i know.
This could require some time, anyway i'm working on something similar now (for a different project)... so, why not? :)

[d.co]

I'm getting a strange (though I'm guessing somewhat simple to fix) error in which when I run a compressiblity test usting Xvid (b-frames on VHQ=1 Motion search=6 et al) I get an error at the end of it that says "Cannot open vdenc.log" instead of the compressiblity info.

I have Vdubmod.exe and Enc on my C: (in differenet places) and the video files (vob, d2v and avs) on my D: drive. I'm also using one of Koepi's newest unstable builds. So for the life of me I cannot figure out where this error is coming from.

Thanks for your time and consideration and thank you for this tool... I remember asking about Compressiblity tests for XVID a short while back. Nice to see it finally has it (and then some) :)

DeeCo

NOTE: Whoops sorry... I noticed I didn't have the latest VirtualDubMod version. I put that in place and everything is working now.

Out of curiosity what is it about the latest versions of VirtualDubMod that make it work with ENC that slightly older versions won't?

[kxy]

d.co,

You kind of answer that question yourself, it is vdenc.log

[ookzDVD]

@jonny,

Your enc is rock!
I love this program, thank you for your great work!

[Kamui-Dash]

nice little tool, how about adding FFVFW into the game :D

[jonny]

You kind of answer that question yourself, it is vdenc.log

:) yes, basically the latest build have an option that generate this log file (that contains informations about encoded frames size)
I use this log instead of the XviD or DivX log... so there is no need to update my tool if XviD or DivX log file format change (and the method can be applied with every codec)

@Kamui-Dash:
You can try to use FFVFW fcc in order to use it (read the docs about using custom fcc), i'm not able now to see if it works correctly (probably later in the evening i'll try)

@ookzDVD:
Thanks :)

[jonny]

@Kamui-Dash:
Yep it works, simply put fvfw in one of the custom fcc slots and you are ready to use FFVFW.
(tested with ffvfw-20030325.exe)

[Siku]

@jonny

I love this program! Thank you, keep up the good work!

Regards,
Siku

[((( atom )))]

johnny: my respect!

-very useable tool, thx - i'll use it from now on.

[kaitsuburi]

@jonny-

A big THANKS from me too. Enc has been working like a charm so far on about 4 XviD encodes. B-frames, vhq, custom matrices -- no problem, compressibility tests have given me almost perfectly accurate results.

-kaitsuburi

[jonny]

@Siku, ((( atom ))), kaitsuburi:
Many thanks for the kind words :)

There is a tiny update on my site:

1.01 (13th April 2003)
- Added ffvfw and Microsoft WMV 9 codecs
- Removed ICConfigure and ICGetState error checking (some codecs doesn't return a correct value, it's better to completely remove the check)

PS: atm size prediction accuracy is not guaranteed with ffvfw+bframes

[jonny]

Updated the notes with this really important thing:

Every time you update to a new codec version, you must trash all the settings/jobs saved for the previous version of the codec (codec data structures could be changed, so previous settings doesn't work any more).
Another good idea is to restore codec defaults settings (only the first time you open the codec settings interface).

[TNM]

@jonny

I have some questions :)

- Does Enc work "mostly" with avs2.5 and above for compressibility test? With an avs 2.08 script, unless I choose to use 100% of the movie, Enc will create a temp avs script with the function SelectRangeEvery(...) which doesn't seem to be recognized by avs2.08...

- Is it more accurate to use more % of the movie for compressibility test? Or is 1-10% enough? I did some compressibility tests with 5% movie used and the predited filesizes at quant 2 are always very near to the original filesize of the .m2v files...I wonder if it's accurate or not... if I follow the suggestion in the readme file to let the final encode filesize at 50-60%, my encode would be huge...

Really sorry if the questions are dumb...I'm quite new to all this :D Thanks a lot for creating such a useful tool anyway :)

PS: oops, forgot to mention that I did all tests with XviD latest Koepi unstable build with b-frame "extended method" :)

[hakko504]

Originally posted by TNM
@jonny

I have some questions :)

- Does Enc work "mostly" with avs2.5 and above for compressibility test? With an avs 2.08 script, unless I choose to use 100% of the movie, Enc will create a temp avs script with the function SelectRangeEvery(...) which doesn't seem to be recognized by avs2.08...

Check the box marked 'Use SelectEvery' and you should be OK. Strange though, I thought SelectRangeEvery should be included in v2.08.
- Is it more accurate to use more % of the movie for compressibility test? Or is 1-10% enough? I did some compressibility tests with 5% movie used and the predited filesizes at quant 2 are always very near to the original filesize of the .m2v files...I wonder if it's accurate or not... if I follow the suggestion in the readme file to let the final encode filesize at 50-60%, my encode would be huge...
Well, you will get slightly better prediction if you raise the percentage, but the time invested is usually not worth it. The tests are quite accurate even at 5% as shown here (http://jonny.leffe.dnsalias.com/whywork.htm) even though in jonny's test setup, 5% seems to produce the worst results. And if you think the filesize is too big if you use 50-60% in the compression test, then you could lower the resolution, add some filtering, check your resize method, check your deinterlacing method (if any) or maybe a combination of all.

[jonny]

Strange though, I thought SelectRangeEvery should be included in v2.08.

This is what i was thinking too, but seems not true :( ( i have not yet included SelectEvery option in Enc, so this option could be still useful ;) ).

So, @TNM:
To use AviSynth 2.08 you need to include mpeg2dec.dll plugin ( SelectRangeEvery is hidden here :) ).
AviSynth 2.5x include SelectRangeEvery directly in the core ( so you don't need to include a plugin in order to use this function ).


For compressibility test, i usually go for 4%-5% of the movie.

As you can see from the tests around on my site, 1% give the worst results, 10% give the better results.
Going up with the % of analisys gives usually better results, but this is true only in the general case (it's not guaranteed that 10% is always more accurate than 9% for example).
For the others doubts about accurancy, compressibility test with 100% of the movie is your best friend to verify the errors in size prediction (there are no statistics here, the result is the real video size after a full encode).


PS: as hakko says, don't be surprised when you obtain a huge predicted size, this often happen if your movie is hard to compress and you are using a big resolution, lowering resolution and using more aggressive filtering (eg. bilinear resize or some light tempsmoother) can help a lot.

[TNM]

@jonny, hakko504
Thanks a lot, I'll follow your suggestions :)

[kaitsuburi]

@jonny

I've been happily using Enc for a couple of days now and I have a very slight humble feature request for future versions.

Could you include an option to set the priority class at once for all jobs in the joblist. Vdubmod has that option but not Enc (or I haven't encountered it at least, maybe I'm blind :o ) . My W2k sucks up too much processing power on "Normal" priority and I am always forced to change it manually. I liked being able to set a default "Low" priority for the job list in Vdubmod.

Of course, this is not a big deal, just a small small request...

thanks,

-kaitsuburi

[jonny]

Actually i'm working on a standalone encoder and i could add this option here (i want to replace VDub with this encoder).
If this will take too much time, i can think to release a version in the middle that handle VDub priority.
Let's see what happen :)

[hakko504]

Originally posted by jonny
Actually i'm working on a standalone encoder and i could add this option here (i want to replace VDub with this encoder). AVS2AVI? (http://forum.doom9.org/showthread.php?s=&threadid=36768)

[jonny]

I'm taking great inspiration from avs2avi, sure :)

But i'm writing all from stratch (i'm making experiments to read frames directly via avisynth.dll, like in axenc000.zip-released by Toff in the same thread).

For the writing part i was thinking to write avi from stratch (like VDub does, so without using vfw).
But matroska will be soon released (so a better choice could be writing in this format instead of avi).

Apart from some speed gain, i think that in this way i'll be able to get a reduced error in size prediction (i'm able to fast encode each snip as a separate clip, this will delete oversized frames at the end of each snip and will help a lot in the XviD field).

If all goes well, be prepared for a new betatesting session soon :)

[hakko504]

I see. Sounds interesting. Send me a note when you want someone to beta test it for you ;)

[N_F]

Originally posted by kaitsuburi
@jonny

I've been happily using Enc for a couple of days now and I have a very slight humble feature request for future versions.

Could you include an option to set the priority class at once for all jobs in the joblist. Vdubmod has that option but not Enc (or I haven't encountered it at least, maybe I'm blind :o ) . My W2k sucks up too much processing power on "Normal" priority and I am always forced to change it manually. I liked being able to set a default "Low" priority for the job list in Vdubmod.

Of course, this is not a big deal, just a small small request...

thanks,

-kaitsuburi
No need for this. Just open VDM that Enc calls and look around in the options and you'll find a place to change default priority. I have it set at idle here as I like to be able to watch movies, play games or whatever while encoding.

@jonny - You know where to find betatesters if you need them :)

[jonny]

@jonny - You know where to find betatesters if you need them

sure! ;)

[Nazgul]

Hey Jonny, just as a note: In Divx 5.04 it seems that 100% quality has been set to Q=1, rather than the Q=2 it's been until now. I'm assuming that to run the Compressibility Tests in Enc we still use Q=2, even though it says that's 97% quality now?

[jonny]

Yep, i think quant=1 it's an overkill, so it's better to consider quant=2 as 100% quality and use it for the compressibility test (this will give you results that are "compatible" with the previous versions).

[wotef]

hi - using enc 1.01 and huffyuv ccesp patch, i get this error after trying to configure a normal encode using custom fcc "hfyu"

icgetstatesize <=0

[jonny]

It seems that huffyuv doesn't have a (correct?) get/set state implementation.
Never used huffyuv, i'll try this later in the evening

[N_F]

@jonny - Will we see an Enc version where the program window will be accessible while encoding in the near future?

[jonny]

I think in the next version... with another nice surprise (i hope in this weekend)

... nice new avatar, N_F :D

[N_F]

Originally posted by jonny
I think in the next version... with another nice surprise (i hope in this weekend)
Can't wait :)

Originally posted by jonny
... nice new avatar, N_F :D
You should see her in full size :)

[hakko504]

Originally posted by N_F
You should see her in full size :) Yes please! :)

Who is it? Your girlfriend? You? ;) :D

[N_F]

Originally posted by hakko504
Your girlfriend? In my dreams... yeah :)

Sorry to disappoint you, but it's actually not a real person. It's a painting by Jennifer Janesko, an artist who focuses on very beautiful and semi-nude women. I've uploaded the full picture at http://www.freewebs.com/franic/jennifer_janesko_018-1.jpg

I'd make all her paintings available, but I don't know what the copyright says.

[acanthis]

Jonny,

I've been using Enc 1.1 for a few days now, and I'm hooked! I was a great fan of DivX5Enc, so I missed the automatic two-pass job generation at first, but the new Enc is more flexible and it's saving me a lot of time now that I've started using the Copy To and From Job buttons.

Good work!

[jonny]

@acanthis:
Many thanks :)

@wotef:
Sorry for the delay, i've tryed huffyuv, but the codec doesn't have an ICGet/SetState implementation, so it's not possible to use it with my app. (the same thing happen to VDub, if you save the parameters in a vcf file, codec state it's not saved, so it's not possible to queue jobs with different huffyuv settings)

@N_F:
We should ask to Jennifer Janesko where have taken the inspiration :D

[nexus]

First: thanx for this great tool. But it doesn't work with latest VirtualDubMod v1.5.1.1a. At least Size prediction and Compressebility test. Would be nice if this can be fixed.

--
nexus

[soulfx]

@jonny, thanks for this awesome program. I had used DivX Enc back in my n00b days and since switchin' to pure XviD the only thing I missed was the ease of which DivX Enc could do compression tests for me. I don't think I could have gotten where I am today without your early DivX Enc work and I am happy to have Enc for my XviD encodes now.

@nexus, until jonny gets the code updated you can always stick in the old VDMod 1.4 in the same directory as 1.5 (just rename it to VDMod14 or whatever) and point to that in Enc.

Happy encoding everyone.

[nexus]

@soulfx: I already did this. Just wanted to mention this problem

--
nexus

[jonny]

@nexus & soulfx: thanks to both :)

There is a quick update on my site (Enc 1.02) that fix the problem with the latest VDubMod version (and of course works with previous VDubMod versions too)

[nexus]

Thanx :-)

--
nexus

[jonny]

Using AviSynth 2.5 + MPEG2Dec3 v1.00 seems to leak a lot of memory if you run a lot of jobs or you get AviSynth info on the file.
A reference can be found here:
http://forum.doom9.org/showthread.php?s=&threadid=47970

Nic have fixed this problem, there is a new mpeg2dec3 version ready to be tested ^^
http://forum.doom9.org/showthread.php?s=&threadid=53164

[((( atom )))]

@jonny

i am still using it :) it is reliable.

wouldn't it make sense, adding audio-dub-support to it, so one would get a all-ready second pass?

when i do vdubmod by hand, i alsways load my audio-tracks into vdubmod before doing the second pass. it saves time, klicks and space.

[jonny]

I want first to finish some things on the video encoding side (this may take a bit... the thing i'm working on it's a bit complicated and require a good amount of testing).
I have specs defined for the next 3-4 versions :).
After this i'll work on the audio side.

[((( atom )))]

cool, i'll wait..

[jonny]

Enc 1.03 is out

- Added support for AVS2AVI, at least v1.27 is required (work with normal encodes, size prediction and compressibility test, this feature is still in beta stage)
- Added an encoding window, it shows the joblist with the running job selected and the status of the terminated jobs, you can stop after the current job finish and you can change the value of the shutdown option
- Option to run only the new jobs (useful if you add jobs after encoding or if you stop encoding and you want to run the remaining jobs)
- Added some command line options (see the docs)

AVS2AVI support is still in beta stage.
At least v1.27 is required (the new versions will work too).
There are some things to take care about AVS2AVI, atm XviD output is broken (the avi header is bad for XviD encodes) and atm i'm not able to make error checking (if AVS2AVI report an error).
Compressibility test and size prediction seems to work well (on my little tests avs2avi produce the same frame sizes, compared to VDub).
Many thanks to DaveEL for adding log generation (comp.test and size prediction works with AVS2AVI thanks to this).
Be sure to check for AVS2AVI updates here:
http://daveel.leffe.dnsalias.com
http://forum.doom9.org/showthread.php?threadid=36768

Enjoy with this new version :)