HC encoder

[DrP]

I've a problem encoding interlaced content with hc 021 where interlaced video can have very poor quality at scene changes.

If I do a two pass encode, the only way I've found to stop the problem is to turn off scene change detection. If I do a single pass constant quality encode, turning off scene change detection - or - turning of the VBV - or - allowing insanely high bitrates (say 15Mbit/sec) cures the problem.

hcenc's ini file contains:

*BITRATE 4000
*MAXBITRATE 6000
*FRAMES 0 32186
*PROFILE best
*AUTOGOP 15
*DC_PREC 8
*INTERLACED
*TFF
*MATRIX mpeg


Frame from CCE (http://img182.imageshack.us/img182/6608/ccero5.png) (min 0, max 6000, av 4000)
Frame from hcenc with VBV/scene change enabled (http://img180.imageshack.us/img180/9741/badvv4.png)
Frame from hcenc without VBV/scene change (http://img211.imageshack.us/img211/2270/goodmx9.png)

[jdobbs]

I was reading posts about Bitrate Redistribution in the DVD-RB Forum. Would it be possible to include something like that in HCEnc? (plus providing option to choose sample-size and to run an extra encoding pass) Since there is already an OPV option which is used for Bitrate Redistribution calculations. And the new DVD-RB version will get an option to use HC for Bitrate Redistribution even if you want to use another encoder for the other passes. So why not just include it in HCEnc itself?
That doesn't make a lot of sense... because HC concentrates on a single file that is being encoded. How do you distribute bitrate across different files when you are only encoding one?

That's a job for DVD-RB to do -- since it tracks all the segments that are involved in a DVD.

[Potlood]

That doesn't make a lot of sense... because HC concentrates on a single file that is being encoded. How do you distribute bitrate across different files when you are only encoding one?

That's a job for DVD-RB to do -- since it tracks all the segments that are involved in a DVD.

hmmm didnt think of that.. But what are those segments exactly? HC (at least HCgui) has options to import chapterlists, to create chapterlists based on scene changes and to manually create zones to change bitrates. Maybe it can also scan for segments?

[MrC]

ATM the actual encoding threads are running at a low priority, the main program runs at normal priority.
I just implemented a priority setting in the next release (settings: idle, low, normal, high) which will set the priority for the main program and all threads.
Will be available as a c/l command too.

:thanks:

Bye
________
CB900F (http://www.cyclechaos.com/wiki/Honda_CB900F)

[hank315]

@DrP
Yep, Boulder also reported problems with the scene change detection and I've seen it also in my own encodes, there will be some changes in the next release to solve it.

[Boulder]

hmmm didnt think of that.. But what are those segments exactly? HC (at least HCgui) has options to import chapterlists, to create chapterlists based on scene changes and to manually create zones to change bitrates. Maybe it can also scan for segments?There are no segments in a single video file. DVD-RB splits the video into segments depending on the cells (chapters?) on the DVD. You could say that one video file is one big segment.

[Potlood]

Ok I understand a litte better now. But if a program can split cells/chapters into segments, why wouldnt an encoder be capable of doing that? It already has chapter support...

I think it would be an advantage for HC to have native redistribution support (if possible). So output quality becomes more independent of what program people use (like it should be). Maybe it makes HC even more popular, because not everybody uses DVD-RB.

[jdobbs]

I don't think you understand the concept. If you brought the entire film into a single file -- then distribution of available bits based upon demand is inherent in the encoding process (in two+ pass VBR, at least). The reason redistribution exists is to enable doing something similar across multiple sources.

Previous to redistribution, the bitrate allocation across segments was based upon the distribution of the original encoding. That works fine for most sources -- but for CBR or poorly encoded VBR originals it might be less-than-optimal. Redistribution was a way to improve those encodes.

[peter100m]

@hank315: Is it possible to make the HC encoding window not show up at all and take focus? For example, if I have a batch list, every new encode start makes the HC window flash and take focus from other program windows. Even though I use the *SILENT parameter. Thanks so much for your efforts!

[cweb]

@DrP
Yep, Boulder also reported problems with the scene change detection and I've seen it also in my own encodes, there will be some changes in the next release to solve it.
Is the problem only with interlaced encodes?

[foustapfe]

Hi all. First of all thanks to hank for this great encoder and to the people in this community for the great tutorials and support stuff :thanks:

I encoded the movie "I'm a Cyborg, But That's OK" with HC 0.21 using best profile and the movie produced has visible artifacts in a few scenes. Those scenes have in common that they are a shot taken from far away in which there is not much movement, like a person walking in a field shot from far away. The nature of the artifacts is a very pixelated image every few frames. By very pixelated I mean that you are not able to tell what is in the picture.

I tried the same encoding with version 0.19 and it doesn't produce the nasty blocking, but you can tell that some frames has not the same quality as the rest of the scene, so I guess the same bug is present in some way.

I can provide the original clip and the encoded clip if it's not against the rules.

---

Edit: Screenshots showing three consecutive frames in one of the problematic sequences. The one in the middle shows the "bug".

http://img404.imageshack.us/img404/867/shot0048yp7.th.png (http://img404.imageshack.us/my.php?image=shot0048yp7.png)

http://img408.imageshack.us/img408/2600/shot0049za8.th.png (http://img408.imageshack.us/my.php?image=shot0049za8.png)

http://img165.imageshack.us/img165/1148/shot0050il3.th.png (http://img165.imageshack.us/my.php?image=shot0050il3.png)

[Squeeto]

Thankyou to Hank315 and the very helpful others on this thread.

I just upgraded my version of ffmpeg from a quite old:
ffmpeg version 0.4.8, Copyright (c) 2000-2003 Fabrice Bellard
to the newer:
FFmpeg version SVN-r7945, Copyright (c) 2000-2007 Fabrice Bellard, et al.

I don't know if this is a problem with my technique, hcenc or the newer version of ffmpeg.

The m2v file created by hcenc causes the older ffmpeg to report:
Input #0, mpegvideo, from 'd:\dv2dvd\testavis\boogie.m2v':
Duration: N/A, bitrate: N/A
Stream #0.0: Video: mpeg2video, 720x480, 29.97 fps
Must supply at least one output file

This is created from a camcorder DV avi source.

The newer ffmpeg has more to say about the same m2v:
FFmpeg version SVN-r7945, Copyright (c) 2000-2007 Fabrice Bellard, et al.
configuration: --enable-mp3lame --enable-libogg --enable-vorbis --enable-faad
--enable-faac --enable-xvid --enable-x264 --enable-mingw32 --enable-a52 --enable
-dts --enable-pp --enable-amr_nb --enable-amr_wb --enable-swscaler --enable-avis
ynth --enable-gpl --enable-memalign-hack --cpu=i686 --extra-ldflags=-static
libavutil version: 49.3.0
libavcodec version: 51.32.0
libavformat version: 51.9.0
built on Feb 12 2007 12:01:35, gcc: 3.4.5 (mingw special)

Seems stream 0 codec frame rate differs from container frame rate: 29.97 (30000/
1001) -> 25.00 (25/1)
Input #0, mpegvideo, from 'd:\dv2dvd\testavis\boogie.m2v':
Duration: 00:00:26.5, start: 0.000000, bitrate: 9001 kb/s
Stream #0.0: Video: mpeg2video, yuv420p, 720x480, 9000 kb/s, 25.00 fps(r)
Must supply at least one output file


My hc.ini:
*infile D:\dv2dvd\testAvis\boogie.avs
*outfile D:\dv2dvd\testAvis\boogie.m2v
*bitrate 5000
*maxbitrate 9000
*matrix mpeg
*profile best
*aspect 4:3
*bias 0
*scanmethod zigzag
*dc_prec 10
*autogop 15
*preview

My boogie.avs:
LoadPlugin("TomsMoComp.dll")
AVISource("D:\dv2dvd\testAvis\boogie.avi")
ConvertToYV12()
TomsMoComp(0,15,1)

I need ffmpeg to do further work but it stalls here. Anyone know the fix?

Thanks again.

[Boulder]

The question that first comes to my mind is, what can ffmpeg do that HC cannot?

[Squeeto]

FFmpeg works my audio and extracts jpeg images for my menu screens.

[Boulder]

Why don't you use the original script for doing those things, it should work just fine.

[Squeeto]

Probably the best idea.
I just wondered if HCenc was not correctly setting flags (or whatever tells ffmpeg 25 or 29.97 fps).
Seems stream 0 codec frame rate differs from container frame rate: 29.97 (30000/
1001) -> 25.00 (25/1)

[gizzin]

I can't wait for a new version of HC.

[cweb]

I can't wait for a new version of HC.
Me too :)

Would like to thank Hank for the excellent program, by the way..
:thanks:

[45tripp]

I don't know if this is a problem with my technique, hcenc or the newer version of ffmpeg.
I don't see any problems.
At least you've pointed to none.

There's nothing wrong with HC output.

Probably the best idea.
I just wondered if HCenc was not correctly setting flags (or whatever tells ffmpeg 25 or 29.97 fps).
Simply seems to be the way ffmpeg treats/sees elementary mpeg2 streams.
No problem here.


I need ffmpeg to do further work but it stalls here. Anyone know the fix?

Stalls where?
When how?
Start a new post with specific details of a problem. Include commandline and it's output.

Why don't you use the original script for doing those things, it should work just fine.
"original script"?

gl

[Boulder]

The script that he feeds to HC - he could simply use the same script in ffmpeg (it does accept avs input, am I correct?)

[jdobbs]

I think it may depend on the version. Some of the releases come with an AVSDIRECT.DLL file. I included that with the version included with DVD-RB. My experience with FFMPEG and MPEG-2 seems to indicate that multiple passes "fail to converge" very often. Really low quants, too... even on low bitrates with high-bitrate matrices. I'm still trying to figure that one out (I see it on MENCODER too, causing undersizing quite often).

[hank315]

Is the problem only with interlaced encodes?
No, but it's more pronounced with interlaced encodes.
On some scenes the sensitivity for SCD is too high now which may cause very short GOPS.

every new encode start makes the HC window flash and take focus from other program windows.
I will change it so the HC window isn't created at all.

I can provide the original clip and the encoded clip
The 'buggy' image seems to have a very high quantizer, lots of blocks.
Best to upload a small sample of the original and encoded clip.

I just wondered if HCenc was not correctly setting flags
HCenc gets the framerate info from Avisynth as nominator/denominator.
If the result is within 0.5% of one of the valid MPEG2 framerates it's considered OK and the correct header is written, otherwise it will abort with an "invalid MPEG2 framerate" error message.
This code hasn't changed since the first release so I guess it should be foolproof by now.
So I don't have a clue what's going wrong in your case.

[45tripp]

The script that he feeds to HC - he could simply use the same script in ffmpeg (it does accept avs input, am I correct?)
yes.
been able for over a year and mencoder for over 2 years.

[Squeeto]

Stalls where?
When how?
Start a new post with specific details of a problem. Include commandline and it's output.


It seems with the newer version of ffmpeg, the 'singlejpeg' parameter has been dropped. 'mjpeg' is a suitable alternative (replacement?) parameter to extract images for my menu screens. I got it working.

It still complains about the codec frame rate differing from container frame rate but I trust that this is a ffmpeg issue. I have decided to just keep the older version of ffmpeg so that I don't have to see the error message.

Thanks again guys.

[Chabb]

I'm using HC021 and Avisynth 2.5.7

simple script:
colorbars(pixel_type="YV12")
trim(0,25)

this is original picture
http://forum.doom9.org/attachment.php?attachmentid=7546&stc=1&d=1188996848

and this is compressed one
http://forum.doom9.org/attachment.php?attachmentid=7547&stc=1&d=1188996853

can someone tell me what's wrong?

[foustapfe]

The 'buggy' image seems to have a very high quantizer, lots of blocks.
Best to upload a small sample of the original and encoded clip.

Here they are:

http://rapidshare.com/files/53562989/samples.rar.html

Rar file contains four clips: original from DVD9 and encodes with HC 0.21, HC 0.19 and CCE 2.7.

[Chabb]

It's not about quantizer or blocks.
I'we used default configuration.
Colours in second image are more than distinctly distorted.

[Chabb]

Ooops!
Last evening researches showed
that this problem appears to be
in internal MPEG-2 decoder of VirtualDubMod 1.5.10.1
Though it works well with mpeg-2 streams created by
other encoders (e.g. TMPEG 2.5)

[Boulder]

If you used ColorMatrix(mode="rec.601->rec.709") in the script, the colors would probably be closer to the original. This is a known "issue", see the ColorMatrix thread in the Avisynth usage forum.

[jjoshua2]

I'm having an issue with HC giving me outputs that are shorter and dropping frames. Details are here (http://forum.doom9.org/showthread.php?p=1044674#post1044674). Is there any tips on fixing this? How much higher quality is HC than Nero Vision's job of converting lossless huffy avi.

[fjhdavid]

Hello,

Thank you for this wonderful software!

does the next release will use the sse 4 instructions (the penrynn will arrive soon)?

thanks
Francois

[hank315]

does the next release will use the sse 4 instructions (the penrynn will arrive soon)?
No SSE4 instructions in the next release.
The MPSADBW instruction looks promising, it can do full search ME very fast but full search isn't used in HCenc.
IMHO SSE4 does have some nice instructions (DPPD is cool) but that's not useful for video encoding.

[Lorax2161]

This is a theoretical question I've been wrestling with in an attempt to quicken the speed of my encodes.

For the sake of argument, let's suppose HC Encoder can process unfiltered video on a given computer system from Avisynth at 30fps. Would it be correct to presume that Avisynth will be waiting for the encoder whether it feeds it at 100fps or 1000fps? If so, there is no sense leaving out an effective filter if there is no appreciable speed penalty. It could be that simple, or there could be much more at work here, which is why I pose the question.

@Hank315... phenomenal encoder, sir. A test DVD of a television crime drama looked as good as when I captured the show in the first place. Remarkable since the source was NTSC DV and it was compressed to about one third the captured size. The fast action of sports has proven a challenge, hence the experimentation with different filters in Avisynth.

Thank you.

[fjhdavid]

Dear Hank,

Does HC require the bt.709 color matrix for encoding?

thanks
francois

[Boulder]

HC assumes Rec.709 coefficients.

[pandy]

HC assumes Rec.709 coefficients.

Hm... by assumption 601 should be used in SD and 709 in HD

[Boulder]

The same behaviour is in QuEnc, mencoder and CCE..where did you actually get that information? I don't know if anybody has found an official specification anywhere.

[fjhdavid]

the problem is that during encoding, you don't know, if after, you will play the encoded DVD as it (SD resolution) or in a upscaled player (HD resolution)

but the encoder must know the color matrix during encoding, so is it bt.601 or bt.709 which is assumed for the HC encoder?

[kumi]

Like Boulder said, HC assumes Rec.709 coefficients. That's what you should feed it.

[tom942]

@Hank315

Just a quick question. Will the new version include something similar to AQM?

Regards and keep on with this great app :).

[pandy]

The same behaviour is in QuEnc, mencoder and CCE..where did you actually get that information? I don't know if anybody has found an official specification anywhere.

If Your question is for me i answer - Capabilities declaration form for HDMI certification http://www.hdmi.org/pdf/CDF1.1.pdf have such declaration:
Source_Alt_Colorimetry (description is: Will the product ever transmit video using a non-default (i.e. alternate) colorimetry under some condition? (e.g. using BT.709 for 480p or BT.601 for 1080i)). So this mean that industry by default use 601 for SD and 709 for HD.

[dragongodz]

by assumption 601 should be used in SD and 709 in HD

by real world example dvds can be 709. i just ran 3 dvds(a disney cartoon, steamboy, logans run) through dgindex to see what they were. all were 709.

i have seen others mention they have come across dvds that use 601 aswell so no i am not saying all dvds are encoded 709.

however the assumption that they all should be falls flat when you bother to look at the reality.

the example you are giving is for hd output and appears to be ignoring the dvd media produced for years. so if it is going to have trouble with content produced by these encoders its going to have the same problem with a lot of the commercial dvds people have bought, and continue to buy, aswell.

i will add this link aswell though i do not know how accurate it is
http://www.afterdawn.com/guides/archive/digital_video_fundamentals_-_color_formats_page_3.cfm

and from that
For the most part, the only two standards you need to know about are Rec. 601 (aka ITU.601, BT.601, or SMPTE 170M) and Rec. 709 (aka ITU.709 or BT.709). Rec. 601 is used for MPEG-1, MPEG-4 ASP (DiVX, XViD, and the like), and DV. MPEG-2 may use Rec. 601, Rec. 709, or SMPTE 240M (almost the same as Rec. 709). HDTV and DVD video are always supposed to use Rec. 709. Just as with nearly everything related to digital video this isn't always as simple as it should be. Some HDTV signals and DVDs are encoded with Rec. 601, and sometimes the colorimetry even changes in the middle of a video stream. In the case of MPEG-2, although the colorimetry used is stored in the file (since it supports multiple standards) sometimes it's missing so Rec. 709 is assumed.

[dragongodz]

Will the new version include something similar to AQM?


version 0.18 included the ability to change/use multiple matrices. RTFM and see if that is what you are after.

[Boulder]

Funny thing is, 99% of the Rec.601 stuff I've seen have been sloppy NTSC to PAL transfers - i.e. butchered quality with fieldblending and a very soft video. Maybe it's the hardware encoder they use, I don't know..I've just not seen much of Rec.601 in good transfers. I assume the industry mostly uses Cinema Craft's products but I may be wrong there.

[pandy]

by real world example dvds can be 709. i just ran 3 dvds(a disney cartoon, steamboy, logans run) through dgindex to see what they were. all were 709.

Probably using particular color space depend from source ie if SD is taken from HD (film tape is converted to the HD and for SD or HD is converted to the SD) then 709 will be prefered color space, if source is taken from analog SD video camera there is high probability that they use a 601.

Im not sure that we should follow industry assumption that 709 is prefered for HD source and for SD 601 is prefered. But from uknown reson industry in HDMI certification process take assumption that 709 is for HD and 601 is for SD and request clear declaration from HDMI source manufacturers that they use or not alternate color space.

[dragongodz]

Probably using particular color space depend from source ie if SD is taken from HD (film tape is converted to the HD and for SD or HD is converted to the SD) then 709 will be prefered color space, if source is taken from analog SD video camera there is high probability that they use a 601.

the problem then is its still dealing with possibilities so can never be sure.

But from uknown reson industry in HDMI certification process take assumption that 709 is for HD and 601 is for SD and request clear declaration from HDMI source manufacturers that they use or not alternate color space.

which frankly is rather stupid to me. its ignoring the precedence of current SD sources such as dvd. so if output to hd is going to be assumed those irrelevant of the real source that would be the dumbest thing ever. unless they want dvds to purposely look worse, now that is a possibility.
the only way those restrictions could make any sense is if it was limited to the hd authored format but i do not see it mentioning that.

[pandy]

the problem then is its still dealing with possibilities so can never be sure.


Hm - i think that this is no problem at all due of mandatory support for various color space inside codecs (H.262, H.264) - especially for H.262 seems that this is no problem due presence of sequence display extension also seems that for h.264 this information is to be much more optional (eg x264 implementations simply don't care about color space).


which frankly is rather stupid to me. its ignoring the precedence of current SD sources such as dvd. so if output to hd is going to be assumed those irrelevant of the real source that would be the dumbest thing ever. unless they want dvds to purposely look worse, now that is a possibility.
the only way those restrictions could make any sense is if it was limited to the hd authored format but i do not see it mentioning that.

In HDMI this is no problem due of mandatory signalization of used color space in AVI Info Frame... so if You use in content (which is HD) a 601 color space this is ok but not default. HDMI transmiter simply set proper color space signaling and everything is ok, You not loose anything... But as i say earlier from uknown to me reason SD by assumption is 601 and HD is 709.
If Your DVD uspcale SD DVD (which is 601) to the eg 1080i then simply signaling that 1080i is on 601 color space.

btw - testing equipment (HDMI analyzers) detect such situations and present AVI Info Frame as suspicious (not error but also not normal).

[tom942]

@dragongodz

I was asking just something similar to CCE's AQM, that, for instance, you use standard matrix, and if it is required it creates a halved or a quartered matrix.

I would like to use it under DVD-RB.

I don´t know Fortran and less which is the algorithm used by CCE, but if it can be included as a DLL or better as a command (like LUMGAIN), it would be great.

Regards.

[Boulder]

The problem is that no-one really knows how CCE determines the points where it changes the matrix. Hank made a decision to include an adaptive quant matrix feature which depends on the luma level, and in my opinion it works fine, helping the low-lit frames. If anyone could come up with a good algorithm (no actual code necessarily but the maths), I'm sure he would consider adding it to the encoder.

[jdobbs]

Funny thing is, 99% of the Rec.601 stuff I've seen have been sloppy NTSC to PAL transfers - i.e. butchered quality with fieldblending and a very soft video. Maybe it's the hardware encoder they use, I don't know..I've just not seen much of Rec.601 in good transfers. I assume the industry mostly uses Cinema Craft's products but I may be wrong there.I've noticed that on NTSC SD DVDs many of the menus, and a significant amount of previews are Rec.601. The features are Rec.709 with very few exceptions. One of the notable places I've seen Rec.601 is in the last (blank) segment of a feature used for command jumps.