theeo - A tool to assist in encoding HDTV

[laserfan]

Keep in mind that my templates are set up for the network affiliates in my location. My NBC station happens to broadcast in 1080i (actually, it's 1088i). Your affiliate may be totally different....No, my affiliate broadcasts in the same manner.

Maybe the particular program that I recorded was munged somehow (ER's "Time of Death" from 2004 which was re-broadcast sometime last year). I will give a different episode a try.

I was hoping to understand better tho how these things are interlaced for TV. It doesn't seem like straight telecine conversion, though IVTC seems to work the best... :confused:

[FreQi]

For anyone who's interested, I am using GordianKnot 0.35 to generate my final XviD.It sounds like the whole reason you're using GK is to calculate your bitrate. You can do this yourself using VirtualDub(Mod) and XviD's built in calculator. Here's how I do it with VDMod 1.5...It sounds complicated but I am being overly detailed. Bedsides, once you've done it, it's like second nature and you can zip through it without thinking.

- Open the .avs in VDMod then click the >| button that jumps to the last frame so you can see the full runtime of the video in VDub's Status bar.
- Hit F7 to bring up the Save As dialog box.
- Check the box for "Don't run this job now" and set the Video Mode to Fast Recompress.
- For Compression, click the Change button, select XviD and click Configure.
- Set it to "Twopass - 1st pass" and click OK, then OK to the "Select video compression" window.
- Now set a file name (I usually use something like basename-p1.avi) and click Save. Now you're back in VDMod.
- Hit F7 again to set up the second pass.
- Again, make sure "Don't run" is check and you're using fast recompress
- Click the Change button and Configure XviD.
- This time select Twopass - 2nd pass" and the "calc" button becomes enabled.
- Click "calc" and you can set your target size. Pick 716800 from the drop down to get a 700meg file (good for those 42min dramas) or use 358400 for a 350meg file (on those 22mins sitcom's). Whatever size you want, it's 1024 bytes * X Megs you want.
- Next fill in the runtime of your video, select the Frame Rate and the Audio Format (AC3 for example).
- Select the Size radio button and click the [...] button to select your -fixed.ac3 file. This will auto fill in the size of your audio file and calculate your video target size by taking into consideration the muxing overhead and runtime you filled in.
- Now just click OK and your target size is set, click OK to close the XviD settings, OK again to close the Select video compression window.
- Lastly, set the filename for pass2 (basename-p2.avi works for me) and click Save to put the job in VDMod's encode queue.

You can repeat this for every video to want to encode, and when you're ready, hit F4 or go to File -> Job Control to see your encoding queue. Click Start to begin encoding and go to bed or something.

It sounds like you guys are using HDTV capture cards to record your programming in TS format. I on the other hand use a DVB-S card to capture the programming in .mpg format, which looks a lot like the ts format with some minor differences.Transport Streams actually are MPEG2's, but like and AVI is just a container for a video and audio stream, a TS is a containter for MPEG2 streams. I am generalizing, but that's how you get "subchannels". So yeah, theeo works with anything you can generate a .D2V out of, like Transport Streams, MP2's, .MPG's, DVD Rips and whatever else.

isn't this standard res. of 720p? --> 1280x720?Yes, an NTSC broadcast of "720p" means 1280x720 at 59.96 frames per second. If you want to know all about HD, there's a couple good "ground up" sites like PBS's Crash Course (http://www.pbs.org/opb/crashcourse/) or the FAQ at HDTV.net (http://www.hdtv.net/) or AVS (http://www.avsforum.com/hdtvfaq/HDTV-FAQ.htm). You should check out the section on "Why are there so many DTV standards? (http://www.avsforum.com/hdtvfaq/HDTV-FAQ.htm#Why%20are%20there%20so%20many%20DTV%20standards?%20Isn't%20that%20overly%20confusing?%20Can%20you%20explain%20them?)".

I was hoping to understand better tho how these things are interlaced for TV. It doesn't seem like straight telecine conversion, though IVTC seems to work the best...I'm not sure if the links above will help answer that, but let me know if you don't find what you're looking for. google is pretty good at that ;]

[jthunder]

Awesome description and step-by-step on how to do this FreQi!

This is extremely helpful for someone like me that isn't really familiar with vdubmod or xvid. I will try to extrapolate a bit on your file size calculation and assume based on your recommendation that any length of file can be calculated for the final file size with this formula;

filesize (entered in xvid) = length of program (in minutes) * 16

so for a 120min feature, filesize = approx~1920 MB

Make sense?

Thanks again.

[FreQi]

...any length of file can be calculated for the final file size with this formula;

filesize (entered in xvid) = length of program (in minutes) * 16

I can't say I've really worked it out that way. My math is designed more to fit the final video to burnable media. Typically people shoot for multiples or fractions of CD sizes, where a cd is 700mb. A 30mins show trims to 22mins and an HR looks really good at 350mb. You'd actually be pretty hard pressed to make a 700meg encode from 22mins.

This is a comment I put in the code of an earlier version of theeo which includes a target size scale I made up. You should keep in mind that this scale is for MPEG2 DVD encodes. The different levels of the scale are still where I would make the cuts, but I would make the target sizes are half what's listed for here for XviD.


'Pick a target size for bitrate calculation
'
' Typically, the runtime of a 30min show without commercials
' is around 22mins. You want to fit 6 of those on a DVD5, so
' that's about 725megs. It's close enough that you may as well
' make it 700 so they fit on a CD.
'
' A 1hr show, without commercials is about 42mins. Fit 3 to
' a DVD5 and encode those to about 1450megs. That would
' make the scale go something like this...
'
' ' ' minuets target size in MB
' ' ' --------- --------------------
' S ' 0 ... 15 350 ( 50% of a CD )
' C ' 15 ... 35 700 (100% of a CD ) <-- 30min shows minus commercials
' A ' 35 ... 55 1450 ( 33% of a DVD) <-- 60min shows minus commercials
' L ' 55 ... 75 2175 ( 50% of a DVD)
' E ' 55 ... 95 2900 ( 75% of a DVD) <-- 2hr shows minus commercials
' ' ' 95 ... up 4350 (100% of a DVD)


so for a 120min feature, filesize = approx~1920 MB
Using my scale, I would assume your 120min feature is actually more like 88mins after you snip out commercials. That lands in at 2900MB for a DVD encode, so an XviD would be 1450MB. Personally, I would actually consider cutting that into two parts though so you get two 700MB files that fit on CD's...unless you expect to it on a data DVD's or just leave them on a hard drive someplace.

I hope that helps.

[TheLegend]

Freqi, thanks again for your detailed instructions, out of curiosity I tried it out in VDubMod without GK to find out if for whatever reason that solves some of my problems. One question, though, your VDubMod instructions only cover the video re-encode part, what do you use then to mux the audio and video back together? Maybe that's my problem. In GK I always used the -fixed.ac3 audio file to mux with 0 delay, since it's already trimmed and delay fixed, right?

I did another test file (FOX, 720p) and made sure this time that I'm not anywhere near the commercials where I would have those audio bit rate changes you described before. However, this time the audio is actually asynchronous from the very beginning of the file! It almost seems as if some delay was applied somewhere, even though it shouldn't. So I would like to try something else for muxing my audio and video together, and I am hoping that you are doing something different than me :)

Thanks,
Legend

[jthunder]

FreQi,

You stated earlier that vdubmod can handle the switchin of audio encodings from AC3 2.0 to 5.1 and back. Can you elaborate on this a bit?

I am recording content that starts with 2.0 audio then switches to 5.1 during the main program then switches back to 2.0 at the end (mostly trailers and other provider advertising). I cut all of these out during my vdubmod edits.

Will the trimming and fixing of the ac3 file have issues with this?

Thanks

JT

[FreQi]

I've just posted v1.03.00 for release. There's a couple new things and some fixes, but for those of you still holding out on trying theeo, I've also posted a Dependency Pack, which makes it a lot quicker to get started. Instead of downloading all the pieces and manually putting them in the right place, you can now get a single zip file with all of them ready to go. Just extract theeo and the dependency pack to the same location and you're ready to go. Here's the Changelog...

[1.03.00] - 2006-04-01
New: Dependency Pack 1.03.00
New: Auto-detects preference file revision and recreates defualts if needed
New: Setting for an alternate/older D2V Decoder (pre-DGInedex, like ProjectX)
Chg: Altered the default AC3 RegEx finder
Chg: Added .mpg and .mpv's to the default allowed source types list
Chg: Log entry made when trying to load a file that's not an allowed type
Chg: Selecting "Use Existing" project uses the alternate D2V Decoder
Chg: The default AC3 Fixer uses Delaycut in "Silence" mode instead of "Fix"
Fix: Couldn't Make Pre-Trim files when using Existing D2V Project
Fix: Crashed when Post-Trim VCF not found and Automating Audio Processing

http://freqi.net/theeo/theeo-1.03.00.png (http://freqi.net/theeo/)

Get it here here (http://freqi.net/theeo/).

Note, there was a change to the Prefrences file, so theeo will need to re-create and use it's default settings, so if you've made some changes, you'll want to make a note of them for the new version.

[FreQi]

...your VDubMod instructions only cover the video re-encode part, what do you use then to mux the audio and video back together?I use VirtualDubMod 1.5 to mux them back together. Open your second pass .avi and the -fixed.ac3 both in direct Stream Copy mode, then hit F7 to save the AVI. I used to use AVI-Mux GUI (http://www-user.tu-chemnitz.de/~noe/Video-Zeug/AVIMux%20GUI/) which has a really great and user-friendly drag-n-drop interface, but I've heard of playback problems with the AVI's it makes on non-windows based players (ie: mac, unix, xbmc...) so I stick with vdmod.

In GK I always used the -fixed.ac3 audio file to mux with 0 delay, since it's already trimmed and delay fixed, right?Yeah, when you trim and demux the AC3 from VDMod, the -trimmed.ac3 no longer requires a delay to match up with the .avs or the video you would otherwise encode from it. When you "fix" it, the .ac3 will also have a 0 delay.

I did another test file (FOX, 720p)...this time the audio is actually asynchronous from the very beginningFOX is actually pretty infamous for having these horrible desync problems when re-encoding due to bitrate chages between commercails to programming. If your recording started on a commercial, then the AC3 starts out with a different bitrate then the part of the AC3 that you wanted to keep. When VDMod first loads the AC3, it only looks at the info on the first piece of the file, and uses that for cutting the file with. So, if you have 192kbit commercials and 448kbit on the programming (like my FOX station) and your recording starts out in the lower bitrate, VDMod will cut the stream as if it's all 192kbit and the whole thing is off. If it starts out in 448kbit and you cut it to keep all the 448kbit, it seems to work just fine, for whatever reason.

So how do you get around this? When loading your source files into theeo (or DGIndex), put a clip of the programming with the audio you want to keep in first. this way, the resulting .D2V and .AC3 start out with the type of video/audio that you want to keep and it tricks VDMod into using that for processing. I record all of my shows in segmented files (about one min of video per file), so it's pretty easy to drag all my sources to theeo, then drag one in again and move it to the top. When DGIndex makes the .D2V, I just have a little more to trim off the front than normal, but my audio stays in sync.

Another option is to use ProjectX to make your .D2V and the .AC3 since it offers the ability to replace non-5.1 audio (ie: 2ch stereo) with silent 5.1. This is great if your audio is 5.1 at 384kbit, but if it's something different (higher bitrate or 2 channel) it doesn't do you much good. Hopefully DGIndex will offer a better version of this in the future. BTW, using ProjectX's Existing D2V Projects is actually possible now with v1.03.00.

You stated earlier that vdubmod can handle the switchin of audio encodings from AC3 2.0 to 5.1 and back. Can you elaborate on this a bit?As long as the bitrate doesn't change, VDMod doesn't seem to care what the number of AC3 channels are.

I am recording content that starts with 2.0 audio then switches to 5.1 during the main program then switches back to 2.0 at the end (mostly trailers and other provider advertising). I cut all of these out during my vdubmod edits.

Will the trimming and fixing of the ac3 file have issues with this?Should be fine, with the possible exception of cases where the bitrate changes (see above).

[jthunder]

Awesome work man! Keep it up, I really appreciate all of you effort that you've put into this...

[FreQi]

I've made a couple changes that are worth releasing:

[1.03.01] - 2006-04-09
New: Drag-n-Drop Source Files (AC3, D2V, Decoder DLL) when Using Existing projet
Chg: The default AC3 Fixer now uses AC3Fix instead of Delaycut
Fix: Trimmed and Fixed AC3 were named after source rather than the BaseName

Get it here (http://freqi.net/theeo).

You will need also Dependency Pack 1.03.00 if you don't already have it.

[GaveUp]

Reading through this thread I see a couple of posts relating to the sync issues common with Fox but the best solution mentioned is to 'cut close' by trial and error to avoid the bitrate change and consequently the problem. A better option, imo, is to take the uncut ac3 and patch it to CBR (see this thread (http://forum.doom9.org/showthread.php?t=92325)). After that any ac3 cutter will be able to handle the file fine. Note that you need to use a vdubmod or some such to do the cutting ... the cutter code is broken in AC3 Cutter ... at some point in the future that will change, but for patching Fox type streams to CBR the tool works fine.

[FreQi]

Using ProjectX to replace 2.0 with 5.1 is the other method. To do this, you must use ProjectX to demux the streams and make the .d2v, then in theeo "use existing" project. This is the method I use on my FOX and WB stations, but that only works because both of mine are 448kbit. If your station uses 384kbit, you need to make your own ac3.bin with a frame of 48kHz 384k 5.1 silence. Check out jsoto's "silence" (http://jsoto.posunplugged.com/audiotools.htm) tool to make that file. Just rename the silence.ac3 it makes to ac3.bin and put it in the ProjectX directory.

[GaveUp]

Using ProjectX to replace 2.0 with 5.1 is the other method. To do this, you must use ProjectX to demux the streams and make the .d2v, then in theeo "use existing" project. This is the method I use on my FOX and WB stations, but that only works because both of mine are 448kbit. If your station uses 384kbit, you need to make your own ac3.bin with a frame of 48kHz 384k 5.1 silence. Check out jsoto's "silence" (http://jsoto.posunplugged.com/audiotools.htm) tool to make that file. Just rename the silence.ac3 it makes to ac3.bin and put it in the ProjectX directory.

My problem with that is my fox often screws up the end segment and broadcasts it SD instead of HD on shows like War at Home where it's just the 30s bit at the end...with the PX method I'd end up with silence for that.

[FreQi]

Yikes, that is a problem. Even if you used the "cut close" method you'd still end up with a lot more time in the ac3 than you have video, so you're kinda screwed either way. That's a new one... I wonder if you could re-encode the AC3 before you trim it. It's not a method I would want to do as having an "untouched" ac3 is the preferred way, but it's the only way I can think of to get that last segment in the bitrate you need in order to keep it, and keep it sync'd.

[GaveUp]

Yikes, that is a problem. Even if you used the "cut close" method you'd still end up with a lot more time in the ac3 than you have video, so you're kinda screwed either way. That's a new one... I wonder if you could re-encode the AC3 before you trim it. It's not a method I would want to do as having an "untouched" ac3 is the preferred way, but it's the only way I can think of to get that last segment in the bitrate you need in order to keep it, and keep it sync'd.

That's why I mentioned the AC3 Cutter. It is able to deal with this situation because it patchs the lower bitrate frames to the highest bitrate found in the stream without reencoding.