HCenc 022 released

[AlanHK]

I also had a problem with CQ mode.

I got underruns when muxing the resulting file. After a long time spent making new audio files, thinking that was the problem, I finally realised that the video bitrate was too high, (the logfile showed it to be 12,000) and that was because I had not selected the "max bitrate" option on "Settings 1" . It would be nice if this was selected by default.

But later a related problem. I then ran a batch job with my usual options (2-pass) overnight. I was surprised and annoyed to find that the jobs were done in CQ mode, though I had not specified this in the command line. Of course, I found that the default ini file now had a *CQ line.
There appears to be no way to override this on the command line. So to protect myself against this I must make a minimal ini file and tack a "-ini" option to all my batch commands.

Could there be an option like "2pass=yes", or "cq=0" that will override any *CQ in the ini?
Also something like "frames=all"; I am bodging that now with
"-frames 0 999999" in case there is an old "*FRAMES" line in the ini.

[hank315]

New (beta) release of HC022.
This release (HC022.1) fixes most of the reported bugs and has some new features.

Bug fixes:
- CQ as parameter input fixed
- possible VBV underflow errors fixed
- reported time using suspend/resume fixed
- error in framelog file fixed

New stuff:
- lossless intermediate file for 2 pass encodes
- VBV progress bar changed, now shows the buffer fullness
- added return code, returns 0 on normal exit, 1 on error
- small bitrate control tweaks
- XML format chapter input
- added -2pass, -frames all, -noini and -lossless as parameter for CLI input
- user data skipped for HD encodes
- some optimizations for SSSE3, reducing the number of non-aligned memory reads

This one is released as a beta first because of a *lot* of internal changes.
It does OK running my usual tests, but some extra testing won't hurt :) so please do...

Can be downloaded here: www.mediafire.com/?9pozg2ynimz (link updated)

[Sharc]

I am getting the ERROR:
No MMX detected, aborting .....

[Guest]

What is your processor?

[Sharc]

Intel P4 2533 MHz Northwood :(

[Guest]

Ha ha, I get the same error and I have an FX-55.

[mikeytown2]

hank315
I did a quick test, it works on my box. specs in sig.

Also I'm a little curious about the lossless option. Are you using huffyuv, uncompressed, ect? You might want to have an option to put the lossless file somewhere. In my case i have multiple harddrives. And the one with the temp dir might be full. A very simple estimation of the temp file size might be helpful as well. Thanks for implementing this btw!

[AlanHK]

I am getting the ERROR:
No MMX detected, aborting .....

Me too.
I have an AMD, 2400 MHz, and quite definitely it DOES have MMX.

[Fluffbutt]

Working fine here, Q6600.

[AlanHK]

Looks like Intel is okay, AMD is no-go.

[Boulder]

Sharc has an Intel processor so that cannot be the problem.

EDIT: It looks more like a problem where older CPUs are not treated correctly.

[cweb]

Sharc has an Intel processor so that cannot be the problem.

EDIT: It looks more like a problem where older CPUs are not treated correctly.
My AMD is working in fact..

[hank315]

Small typo in the cpu_identify routine :(
It now only recognizes SSE3 and SSSE3 capable CPUs.

Fixed, package updated: www.mediafire.com/?9pozg2ynimz

Also I'm a little curious about the lossless option. Are you using huffyuv, uncompressed, ect? You might want to have an option to put the lossless file somewhere. In my case i have multiple harddrives. And the one with the temp dir might be full. A very simple estimation of the temp file size might be helpful as well. Thanks for implementing this btw!
No, it's not huffyuv. It's a just a 'quick and dirty' compression scheme, compression varies from 25 - 60% dependent of the source.
It isn't very optimized yet, think I can get more compression out of it. Speed is comparable with huffyuv.
Because the amount of compression is dependent of the source it's hard to estimate the size but I can provide a very rough estimate.
The file is now written in the same directory as the database file, maybe better to give it it's own disk/directory.

[Sharc]

Works like a charm now. :thanks:

[Nomolu]

This new version, 22.1, does indeed fix the VBV underflow error that I reported. But, Hank, you probably knew that already from using the sample I provided for your own testing. :cool:

[Sounddude]

HC 22 keeps crashing on me..the older versions do not. Any ideas?

[Boulder]

Keeps crashing - in what way?

[Sounddude]

Keeps crashing - in what way?

As soon as I add an .avs file it (HC22) closes and windows wants to send an error report.

[Boulder]

What is your script, and what Avisynth version do you have installed?

[G_M_C]

Dark Shikari is on a roll, he's adapted the X264 VAQ-patch to Xvid. So i ask myself if the same idea might be used on HCEnc (making/replacing the luminance filter/adapter, for a more teature-rich idea ?)

See my question to Dark Shikari here: http://forum.doom9.org/showthread.php?p=1103569#post1103569

About the VAQ-idea, see here: http://forum.doom9.org/showthread.php?t=132760

Mods: Sorry for the cross-posting :)

[LigH]

Dear hank315,

are you able to implement a support for the "Pan&Scan" flag (some DVD players recognise such a flagged material "4:3 P&S" in contrast to "16:9 Widescreen")?

I can imagine that it may not be as trivial as it seems... the "Picture Display Extension" specs are not easy to understand (for me). For example, I don't know if there is a "default" display window position (top left, or centered)... You could cross-check with the result of "ReStream".

This is a request of your favourite german fan lady... ;)

[peter100m]

I just want to second the request LigH has. Ability to set the sequence display extension (Pan & Scan info) would be a nice addition to fine HC.

[Lorax2161]

Many thanks to hank315 for his contribution of a high quality program such as HCEnc. I haven't missed the programs I've stopped using since I found this. As noted in the title of this post, you've seen my feature request. First, I'll say I don't program so I have no idea how much work is involved. I do respect your time and only offer this request as a suggestion for future thought.

There is already a preview window in the GUI version of the program that could possibly be used to display a bit rate/quant graph with an extra toggle option added to "OFF" and "Preview." This would not have to be incredibly detailed, just something that updated every "X" GOPs, seconds, or minutes... whichever was easier. As a bar or line graph, it would allow the end user to see in CQ mode when the bit rate spiked or plummeted, and during the second pass in the two pass method it could show the same for the quantizer.

An advantage of having this in the encoder vs running the resulting video file through a different program is that if the results are coming to you in near real time, you can act on them sooner rather than waiting for the entire encoding to finish. This could allow for a more immediate switch in CQ, filters, matrix's, or average bit rate to get the results one was looking for.

My main use for HCenc is tv captures: DV, NTSC, 720x480, 29.971fps, interlaced. Unlike a movie backup, I have natural divisions in my projects due to the commercial breaks. After removing the commercials, for my example below we will assume there are six clips of 7min 30sec duration each, comprising the 45 minute episode. I'm speculating that I could exploit the benefit of VBR even more if I knew that the bit rate or quant (depending on encoding method) spiked during most of Clip 5 of my six clips. I could then encode it as a separate movie and author it as it's own VTS. This would, as I understand it, allow me to use a different CQ, bit rate, matrix, or filter/function/script combo on this one part of the tv episode.

Example of 1 hour tv show edited, now totaling 45 min (7min 30sec each):

Clip 1 - Treatment "A" (filters, matrix, CQ), VTS 1
Clip 2 - Treatment "A" (filters, matrix, CQ), VTS 1
Clip 3 - Treatment "A" (filters, matrix, CQ), VTS 1
Clip 4 - Treatment "A" (filters, matrix, CQ), VTS 1
Clip 5 - Treatment "B" (filters, matrix, CQ), VTS 2
Clip 6 - Treatment "A" (filters, matrix, CQ), VTS 3

The purpose of doing it this way is that I could use a very slow but high quality script to denoise and/or sharpen the one segment whose scenes demanded it the most.

In an action tv show, this might be the last segment going for the "big finish" of fist fights, car chases, machine guns, explosions, etc. In a documentary about earthquakes, it might be the middle segment that features actual footage of a natural disaster or computer graphics simulating a hypothetical one. In neither case would I need to run a script of 20 or 30 times real time on a scene where a scientist was talking about seismology from behind a desk or a family was having dinner. It's overkill. But I would consider running that kind of script on the most visually demanding seven and half minutes of the show, while leaving speedier filters to run on the less demanding clips.

If nothing stood out, I could just encode the episode as one movie. If the bit rate/quant graph showed that one portion spiked above the others I could make the decision that I wanted to split the episode into multiple movies, author as separate VTS pieces, and filter that most demanding clip with a special treatment, while the other clips received a more mild and speedier treatment.

As long as the quiet "dinner scenes" or "smart guy behind a desk" scenes looked good, I don't think that 25 minutes later when the high drama/fast action took place that the average viewer would think anything had changed--just that the episode looked pretty good at all times, whether they were demanding scenes or not. And by using the time consuming treatment on only 1/6 of my episode, I've saved a considerable amount of encoding time.

There may be flaws in some of my logic and I am open to all viewpoints, but I will just part with saying that I am grateful to hank315 for the encoder as it was made available to us.

[hank315]

are you able to implement a support for the "Pan&Scan" flag (some DVD players recognise such a flagged material "4:3 P&S" in contrast to "16:9 Widescreen")?

I can imagine that it may not be as trivial as it seems... the "Picture Display Extension" specs are not easy to understand (for me). For example, I don't know if there is a "default" display window position (top left, or centered)... You could cross-check with the result of "ReStream".

This is a request of your favourite german fan lady...
Hi Ligh,
Well, I can't refuse a request from a nice lady but it isn't very straightforward:confused:, is there a software player out there which supports it? (for testing purposes)

[hartford]

I encoded 2 vids today using v.022 and when I authored them with DVDMaestro I got an error: bit rate too high. BitRateView gave rates over 11.1K.

I used 4000K ave, 9000k max, Best Profile, 9-bit dc. Lowering Max bitrate to 8500 fixed the problem.

Just providing info on my experience because this hasn't happened previously.

Audio was 192K ac3.

Just checked my last encode using BitRateView: max was 10.254k with HCenc set for 8.5K max.

[Boulder]

Regarding Sharc's post here: http://forum.doom9.org/showthread.php?p=1104478#post1104478, does the source mismatch mean that all frames starting from that frame do not match the ones that were analyzed during the 1st pass?

[LigH]

@ hank315:

We are testing hardware players (http://forum.gleitz.info/showthread.php?t=36908), and bigotti5 added a few notes about which places need to get changes or additions in the streams. There are two "extensions" with some effect here (sequence and display).

Most players will possibly only recognise the existence of P&S, but not the location, and center it.

[hank315]

@hartford
This isn't enough info :(
Please:
- re-encode at 9000 max. bitrate with framelog file enabled
- mux it with Muxman

Can't PM you, could you please enable it...

@Ligh
OK, read the stuff at the Gleitz forum, think it can be done...
BTW, about that test sequence by Bigotti5, the square didn't move using SW players :)

@Lorax2161
Nice idea...
Something to implement when I have a lot of spare time.
So it might take a while :)

[AlanHK]

A couple requests:

The name-case bug was briefly fixed, but came back a few versions ago: using command line, the output m2v filename is forced lowercase.
(Episode_One.m2v becomes episode_one.m2v)
Please make it use the case I give it.

Requested command line option: in the gui you can choose the final filesize and the bitrate is calculated.

I would like to be able to specify the target size in kb (instead of the bitrate) on the command line:

-filesize 602525

It would save loading the file in the GUI or using an external rate calculator.

And yes, the latest bugfix now recognises my 2-year-old AMD.

[fib0by]

hi
When converting an vc1 evo to mpeg2 using hc.22 im getting an error source mismatch in pass 2 starting at frame: 109 in my log.this has happened twice now. encoding completes and everything looks correct just wondering what that error means.
Thanks

heres what it looks like in my log

---------------------
| encoding - pass 2 |
---------------------

*** ERROR, source mismatch in pass 2 starting at frame: 109


I was getting the same error with 0.22
I tried "Reload Avisynth" but that causes the SoundOut plugin to be run twice instead of once (I'm using that plugin to generate the AC3 track out of the DV file).

I tried 0.22.1 with the -lossless option. It seems to eliminate the "source mismatch", but it causes a different and interesting failure mode: SoundOut runs quickly and finishes the job while HCenc is still processing the first few frames - and while SoundOut is still running, sometimes I get corrupted frames in the video stream. Badly corrupted, as in just a mess of colored squares. The corruption never occurs later, after SoundOut has finished.

I believe the source mismatch errors used to occur the same way - only for the fist frames, while SoundOut was running, but I'm not 100% sure.

I ended up running SoundOut and HCenc -lossless in two separate steps, and then there's no corruption.

By the way, with -lossless the transcoding job becomes disk-bound instead of CPU-bound.


EDIT: I am actually running all this stuff on Linux, under WINE. There seems to be no problem with the emulation, but anyway, take my findings with a grain of salt.

[LigH]

OK, read the stuff at the Gleitz forum, think it can be done...

That will make katjarella http://cosgan.de/images/smilie/froehlich/e025.gif

[Boulder]

I've also been getting source mismatch errors with HC0.22.1, even with the *AVSRELOAD option. The mismatch is always reported to happen during the first tens of frames or so.

I just left HC022 and HC022.1 to run a series of 2-pass encodes using the exact same script and ini file and see how the log files look when I get back home. IIRC I've not seen the error under the same circumstances in HC022.

[LigH]

Never use "DirectShowSource" in your script if you want to get a repeatable result. Never!

...

And if you really have no other choice, use "seek=true" or "seekzero=true" and check if they help.


If you did not, I have no other good guess about the reason.

[Boulder]

Never use "DirectShowSource" in your script if you want to get a repeatable result. Never!

...

And if you really have no other choice, use "seek=true" or "seekzero=true" and check if they help.


If you did not, I have no other good guess about the reason.Hehe, I wasn't born yesterday:) My method is MPEG2Source -> motion compensated denoising -> HC so that can probably be ruled out..but I need to doublecheck that too by using a script with only MPEG2Source in it. Then again, if HC022 shows no errors and HC022.1 does, there's something in the new HC version.

[cweb]

Never use "DirectShowSource" in your script if you want to get a repeatable result. Never!

...

And if you really have no other choice, use "seek=true" or "seekzero=true" and check if they help.


If you did not, I have no other good guess about the reason.

Hehe just encoded a short clip with directshowsource, all went well..

[Sharc]

Regarding Sharc's post here: http://forum.doom9.org/showthread.php?p=1104478#post1104478, does the source mismatch mean that all frames starting from that frame do not match the ones that were analyzed during the 1st pass?
Yes.
If you get the frame mismatch error, the output can be *very* bad, all calculated values from the first pass (Motion Vectors, quants etc.) are applied on the wrong frames in the second pass. The result will still be a valid MPEG file but bad quality... *AVSRELOAD sometimes solves the problem.
hank315 is currently investigating further.

Added:
Apparently there are cases with "small errors" caused by avs filters where only few (1 to 3) consecutive frames are affected (perhaps a couple of times during the movie), and there are "large errors" with directshowsource where everything following the first misalignment is screwed up.

[Boulder]

Unfortunately my initial test shows that both 0.22 and 0.22.1 have the error. I'm now going to run a series of tests with only MPEG2Source and Trim and see what happens.

[Boulder]

A quick test shows no errors with a simple MPEG2Source-only script:confused:

[Boulder]

For example this one causes the issue:mpeg2source("D:\TEMP\DVD-RIP\REBUILDER\D2VAVS\V03.D2V",idct=3)

source=last

fields=source.AssumeTFF().SeparateFields() # or AssumeBFF
backward_vec2 = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1)
forward_vec2 = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1)
fields.MVDegrain1(backward_vec2,forward_vec2,thSAD=400,idx=1)
Weave()

This one doesn't:mpeg2source("D:\TEMP\DVD-RIP\REBUILDER\D2VAVS\V03.D2V",idct=3)

source=last
backward_vectors = source.MVAnalyse(isb = true, truemotion=true, delta = 1, idx = 1)
# we use explicit idx for more fast processing
forward_vectors = source.MVAnalyse(isb = false, truemotion=true, delta = 1, idx = 1)
forward_compensation = source.MVFlow(forward_vectors, idx=1, thSCD1=500) # or use MVCompensate
backward_compensation = source.MVFlow(backward_vectors, idx=1, thSCD1=500) # or use MVCompensate
# create interleaved 3 frames sequences
interleave(forward_compensation, source, backward_compensation)

DeGrainMedian() # place your preferred temporal (spatial-temporal) denoiser here

selectevery(3,1) # return filtered central (not-compensated) frames only
It seems that all that include MVDegrain cause the issue - could there be some variable conflict between the encoder and the function?

[hank315]

There are no changes how the mismatch is calculated between HC022 and HC022.1.
How it's done:
- there's one check MacroBlock per frame, it's in the middle of the frame
- in the first pass, the 256 Y (luminance) values for that MB are summed, sum is stored
- in the second pass the same MB is checked and the sum is compared, should be exactly the same

So I don't check if there's a large difference which could mean a completely different frame is served (bad) or a very small difference which could be a filter hiccup.

[Boulder]

The mismatch error occurs in both HC022 and HC022.1 using the above script:( What about HC021, does it report the mismatch in the log file?

[hank315]

No, HC021 doesn't check for it at all.

[Sharc]

Never use "DirectShowSource" in your script if you want to get a repeatable result. Never!

...

And if you really have no other choice, use "seek=true" or "seekzero=true" and check if they help.


Adding "seekzero=true" has helped in my case. Anyway it seems to better not to use DirectShowSource at all.
Thanks for the hint.

[Chumbo]

Adding "seekzero=true" has helped in my case. Anyway it seems to better not to use DirectShowSource at all.
Thanks for the hint.
I've run into this with avs files that use avisource too. I've muxed vc1 streams into an avi container and a simple script like this also generates this error:avisource("file.avi")
converttoyv12()
I've noticed that the MS VC-1 decoder sometimes decodes a frame as all green when seeking randomly just trying to see what happens. Can this be the cause?

Also, and please correct me if I'm wrong, that this error can actually only impact one frame and not everything after assuming the served frames continue sequentially as they did in the first pass. So if the error is on frame 9, but frame 10, on the second pass is the same frame 10 served on the 1 pass, things should match up for the remainder of the encode right?

[phXql]

The ffdshow huffyuv implementation has the same issue, some frames are decoded wrong. If you reload the script, the same frame will then decoded right. So this should only impact one frame, and the rest is okay.

[ron spencer]

is anyone using 0.22.1 beta? Just curious how "safe" is it

[rcubed]

Hank,
If there are two instances of HC022 running on a dual processor, and the shutdown option is selected in one of them, what happens if that instance finishes before the other one (without shutdown selected) finishes? Will both be allowed to finish before the shutdown occurs or does the shutdown happen when the one checked finished regardless.

Thanks

rucbed

[cweb]

is anyone using 0.22.1 beta? Just curious how "safe" is it
I'm using it and had no problems. I don't use huffyuv at the moment.

[mparter]

Newbie question alert :eek:

I'm encoding a VC-1 source which has an aspect ratio of 2.40:1. Which aspect ration should I use, 16.9:1 or 2.21:1?

Also, I have a dual core Athlon X64, should I enable the SMP option or use the HCEnc^n front-end to ensure that both cores are used?

Thanks for an excellent free tool :)

[cweb]

Also, I have a dual core Athlon X64, should I enable the SMP option or use the HCEnc^n front-end to ensure that both cores are used?

Thanks for an excellent free tool :)
I enable the SMP option. I haven't tried the HCENC^n front-end.