HD-DVD to x264, how *I'm* doing it

[BLKMGK]

Okay, it seems like there are many ways to transcode from HD-DVD and Blu-ray to other formats. When I did this it took me awhile and a great deal of question asking to figure it out. Now that I have a good working process I thought I'd share back. Much of what I learned came from this (http://forum.doom9.org/showthread.php?t=129001) thread however the tutorial on the first page is incomplete and links to some bugged tools which caused me some frustration - it's a good start though. In addition I see that at least one other has a tutorial (http://forum.doom9.org/showthread.php?t=134227) up that flows slightly different than mine which ought to be read. Lastly, I also learned a HUGE amount about eac3to from an excellent Wikipedia article that some ass deleted as SPAM - the Google cache is your friend.:mad:

Hopefully between these posts, mine, and others readers will find a path that works best for them and gain some understanding as to how the process works. This particular workflow process works for me, it may work for you, others work just as well. Please understand I'm no authority at this but I'm enjoying my vids just fine now thanks:devil:

Software Tools Needed:
EVOdemux (http://www.pel.hu/down/) (free)
eac3To (http://forum.doom9.org/showthread.php?t=125966) (free)
MKVtoolnix (http://www.bunkus.org/videotools/mkvtoolnix/) (free)
meGUI (http://forum.doom9.org/showthread.php?t=96032) (free)
Graphedt (free-Microsoft (http://msdn2.microsoft.com/en-us/library/ms787460.aspx)) (Google (http://www.google.com/search?q=graphedt+download&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a)for DL)
AnyDVD HD (http://www.slysoft.com/en/anydvdhd.html) or another ripper.
ffdshow (http://sourceforge.net/projects/ffdshow) (free) I could use a source for daily builds plz
AVIsynth (http://sourceforge.net/project/showfiles.php?group_id=57023) 2.5.7 (free)
WMP11 (http://www.microsoft.com/windows/windowsmedia/player/11/default.aspx) for VC-1 decoding (Microsoft but free)
I believe that is all. X.264, Haali, and others will be loaded by meGUI as will encoding profiles.

Hardware:
XBOX 360 add-on drive
FAST CPU! I'm running an e8400 C2D @4Ghz on Vista Ultimate64
Mass disk space - if a movie is 20Gigs I use 50 for processing and work files.

Okay, install all of that. Make sure that meGUI updates and has all the tools it wants. Before diving in some information that wasn't clear to ME when I started this - HD-DVD has more than one video encoding, there are also multiple audio encodings. Some HD-DVD are encoded with H.264, others with VC-1, some trailers or Easter eggs may even be MPEG. Likewise you will see EAC3 and DTS audio encodings. The specific video encoding will change which path you take when it comes time to compress the videos - IF you choose to transcode them at all.

Here's what I do when I do *not* want to compress a video further and am happy with the encoding CODEC:

Rip DVD, delete all non EVO extension files. Of the files left I usually only keep the main movie files but sometimes choose to keep the trailers or Easter eggs. It's these smaller ones I'll often not bother to compress. Often the movie file will be named "Feature_1.EVO" and sometimes a second "Feature_2.EVO" file will exist. Sometimes they are named pretty weird but always with the _1 and _2 in the name, I rename them to Feature_# to keep things consistent when I find this. Bring up a CMD window and navigate to where eac3TO is installed. eac3to -test will test the eac3to install.

D:\Video\eac3to>eac3to -test
Sonic Audio Decoder (2.25.0.0) doesn't seem to be installed
Nero Audio Decoder (Nero 7 or older) is not working correctly
Haali Media Splitter (2007-12-29) is installed
Surcode DTS Encoder doesn't seem to be installed
MkvToolnix (v2.1.0) is installed

That's right I have no fancy CODECS installed.:eek: eac3to is very powerful and can take specific video tracks out of an EVO and place them into a MKV container for you as well as pull specific audio tracks from the video file. I suspect that it can put them both into a MKV file with one command but I've not figured out the commandline. eac3to can also give you information on the content in an EVO file.

D:\Video\eac3to>eac3to f:\Children\FEATURE_1.EVO+f:\Children\FEATURE_2.EVO
EVO, 2 video tracks, 3 audio tracks, 1:49:16
1: Joined EVO file
2: VC-1, 1080p24 (16:9) /1.001
3: VC-1, 480p30 (3:2) /1.001
4: E-AC3, 5.1 channels, 1536kbit/s, 48khz, dialnorm: -27dB
5: E-AC3, 5.1 channels, 768kbit/s, 48khz, dialnorm: -27dB
6: E-AC3, 2.0 channels, 192kbit/s, 48khz, dialnorm: -27dB, -27ms

This video is VC-1 encoded in stream #2 and has an E-AC3 5.1 soundtrack as stream #4. The following command will place stream #2 into a MKV container and encode stream #4 as a FLAC file.

D:\Video\eac3to>eac3to f:\Children\FEATURE_1.EVO+f:\Children\FEATURE_2.EVO 2: f:\Children\children-vid.mkv 4: f:\Childre
n\children-aud.flac -libav

Note you have to give it pathnames etc. and the -libav switch on the end which tells eac3to to use it's INTERNAL decoder on the EAC3 audio. This also works on DTS in my experience and I am happy with the sound. This will produce a hefty MKV and a Flac file that's probably 2+Gigs. It is possible to use many other audio extensions for the audio, AC3 would produce a soundtrack about 500Megs in size from this for instance but AC3 is lossy I'm told. It still sounds fine to my ears. There's some contention on audio formats so read some of the responses to this post plz. Note the numbers and colons - that is where I'm specifying the streams I wish to process followed by the file I wish to place them in and it's format via the extension.

Finally to merge these I use mkvmerge GUI from the MKVtoolnix install to mux them. Drag drop both onto this tool, give it an output name and path, kick it off, and in a short time you have a completed file. In this case it would still be using VC-1 encoding but if it had been H.264 as a source it would retain that as well.

But what if you want to shrink it down or encode in another CODEC?

First rip as before.

Note: The following use of EVOdemux is not needed! Bring up EVOdemux and make sure the options tab has "Continue with the next EVO checked". Drag&drop the Feature_1 file into the windows on the Status tab. This will process the EVO file and give you information about it that will look like this at the end:

1439 counted frames (0:01:00.018) in video stream 0.
1439 calculated frames in video stream 0.


Copy this data down, you will need the calculated frames number later. Use eac3to as before to create a MKV file and an audio file. Take note of whether or not the video is H.264 or VC-1 encoded.
===========Update===========
Okay, when eac3to is used to split out the video and audio it spits out a framecount, this framecount is the same as what EVOdemux spits out so use that number in the following AVIsynth script detailed below. Here's what the output will look like->
Added fps value to MKV header.
Video track 2 contains 173762 frames.
Video track 3 contains 204974 frames.
eac3to processing took 18 minutes, 28 seconds.
Done.
============================

If the video is VC-1 encoded here is what you do as learned from a previous tutorial (http://forum.doom9.org/showthread.php?p=1035422#post1035422) from Taktaal

First, we have to build a GraphEdit graph. Open the software, but don't drag&drop the EVO into it. That will open it with the default filter which will probably be Sonic Cineplayer and that's bugged and will lead to desynchronized audio.
Instead, choose "Graph" -> "Insert Filters...", then choose "DirectShow Filters" -> "Haali Media Splitter" and insert that. You'll get a popup asking you for a file to split, select the .EVO that you rebuilt earlier


Next, insert the "WMVideo Decoder DMO" Filter, and connect the EVO file with the decoder

If you get an error here, you forgot upgrading WMP to version 11.

Save the graph.

Next, create an AviSynth script to load the graph. A very basic sample:

DirectShowSource("serenity.grf", fps=23.976, framecount=171229, audio=false)

What's important here is that you have to manually set the movie to the correct framerate (always 23.976 for HD-DVDs) and set the correct framecount according to the number EVODemux calculated for you earlier or you received from eac3to. If you don't set those two options you'll get a bugged encode.
Also you can add dozens of other AviSynth options here like cropping away black bars, make sure the movie dimensions are dividable by 16, filtering away movie grain, scaling for mobile devices etc. Those options are out of scope for this guide and you should refer to the AviSynth Wiki for them.

Save the Avisynth script. (extension is .AVS)

At this point I fire up meGUI but Taktall has given some excellent X.264 encoding commandlines too. meGUI is pretty easy to use and allows for queuing of many files so I prefer it. Simply open the AVS script file from meGUI's File Open dialog and select the profile you'd like to use in meGUI. The SA-HD-DVD profile is a good starter but I've bumped my bitrate way up and tweaked some other things so my files tend to be somewhat large. One thing I'd ensure is turned on the CABAC checkbox under RC and ME as I *believe* this is what is allowing for multi-core decodes on playback for me right now. More later. Once meGUI has completed compressing your video file you still need to add the audio file, I do this as before using mkvmerge GUI. There, done!

For an H.264 encoded file you follow the SAME process as VC-1 with ONE exception. Instead of inserting the "WMVideo Decoder DMO" filter you will instead use the "ffdshow Video Decoder" DirectShow filter. Thank you Blue_MiSfit (http://forum.doom9.org/showthread.php?p=1100544#post1100544)!

I think this pretty well covers how I'm doing things as clearly as possible. I may yet buy Nero's HD-DVD plugin for V7 and I own COREAVC's decoder too but haven't used it. Before I tweaked my encoding profile some of my resulting videos were as small as 5Gigs, now with my processing turned up I see about 9Gigs or sometimes more on big movies. It will take HOURS to process and heat your machine up good so be sure you can handle the time and the machine can handle the heat. If it crashes meGUI will at least allow you to keep the first pass but midway through a 13hour long second pass is a terrible time to have it crash:eek:

I play my videos back using XBMC under Linux (http://www.xboxmediacenter.com/forum/forumdisplay.php?f=52) at full 1080 rez with digital surround sound with both cores utilized around 40% on a 3Ghz clocked C2D. It's PURTY! This is streamed from home-built unRAID NAS (http://www.lime-technology.com/) in my office via Gig ethernet, no glitches.

I'll shut-up now and hope that this is useful to someone. I will try to answer questions but realize please that I too still have much to learn.:cool:

[Blue_MiSfit]

Nice summary. It's a bit more up to date than the first page of the other big thread about this.

A few notes:

1) AviSynth and ffdshow are actually not included with MeGUI. I would suggest people use the latest daily build of ffdshow tryouts, as it has the most up to date libavcodec, which means fastest possible H.264 decoding. AviSynth 2.5.7 vanilla is reccomended. I had seeking problems with a 2.5.8 alpha.

2) WMV Decoder DMO comes with Windows Media Player 11.

I basically do the same thing, except I cut EVODemux out of the equation entirely, and use eac3to for everything.

All you're using EVODemux for is concatenating the EVOs (stripping out the audio in process), and getting the framecount. This can be easily done with eac3to by muxing the video track to an MKV. The simpler the better, I always say!

I also use the HQ-Slower profile, but again that's just me.

Finally, encoding E-AC3 to FLAC is a little silly if you ask me. E-AC3 is a lossy encoding, as is AC3. Going from a lossy format to a lossless format is a waste of space.

The only real reason to do that is to preserve every bit of quality from the E-AC3 track, but in all honesty you're probably better off encoding to 640kbit AC3 with Aften (which is what eac3to does). This will give you significant bitrate savings, which can be put towards making your 1080p video even prettier ;)

But all said and done, a nice little guide! It will help out all the newbs.

Now all we need to do is get Blu Ray drives, and wait for eac3to to be updated for m2ts processing :)

~MiSfit

[BLKMGK]

Actually I don't even use EVODemux to concatenate files, I am doing that with eac3to when it puts the files into a MKV container.:p If I cut out EVODemux altogether then I have no source for framecounts for my AVIsynth script. I will admit I'm not sure how important that is but it's trivial enough to do. Thus far every single movie I've done has remained in synch throughout so I'm pretty happy.

Not touched a Blu-Ray disk yet and probably won't until SlySoft is ready for BD+. I own too much media to be digging it out of a rack to play so I prefer it all ripped to my NAS where no sticky fingers can find it.

I appreciate the comments on ffdshow and AviSynth. MeGUI installs a wide variety of files and I didn't recall installing those separately. I'll amend my software list to reflect your recommendations. I've actually not done many H.264 movies yet, they seem to be few and far between and I do now have CoreAVC. I did know about WMP11 but on Vista it was there and I didn't want to muddy things up with more caveats, I'll add that too though as it's needed for VC-1 which is so common.:rolleyes:

I'm less sure on the FLAC stuff. If E-AC3 is lossy and AC3 is also lossy then isn't putting it into AC3 from E-AC3 much like encoding an MP3 as an MP3 - a double compression loss? When I pull it out to a FLAC it's usually well over a Gig and often 2, AC3 is seldom more than 700megs as I recall. That seems a significant loss, especially when you consider that FLAC is compressing (lossless) too!:eek: On the plus side I'm hearing that code is close for decoding E-AC3 and will be rolled into XBMC as soon as it's ready.

When not compressing further it seems like eac3to has everything needed to move both audio and video tracks into a MKV on it's own. If anyone has figured out the commandline for that I'm interested in learning how to do it, I've not been successful.:script:

Here's a question for you - what are you doing with DTS soundtracks to preserve them? I'm slapping those in FLAC too but am unsure if that's the best way to retain them.

[Blue_MiSfit]

If E-AC3 is lossy and AC3 is also lossy then isn't putting it into AC3 from E-AC3 much like encoding an MP3 as an MP3 - a double compression loss?


Well, sort of, but not quite.

The Dolby Digital formats are very resilliant to multi-generational transcoding. If you connect an HD-DVD player to your sound system with S/PDIF, it's going to re-encode to AC3 (or possibly DTS) on the fly, since S/PDIF doesn't have enough bandwidth to transfer E-AC3.

It's certainly not as bad as going from (for example) MP3 to AAC, or MP3 to MP3. The difference is indistinguishable to most, myself included, and I'm one of "those guys" that notices the smallest audio compression artifacts.

The FLAC track is probably bigger than the E-AC3 source.

Anyway, you can do it however you want, it's no biggie :)


ctually I don't even use EVODemux to concatenate files, I am doing that with eac3to when it puts the files into a MKV container. If I cut out EVODemux altogether then I have no source for framecounts for my AVIsynth script.


IIRC, when you do something like:

eac3to movie1.evo+movie2.evo movie.mkv

specifying the video track only, it will give you the frame count after it finishes muxing. I don't have my main PC in front of me to verify this... I may be hallucinating!

Specifying frame count may not even be necessary. I've done a few backups without it causing problems.


Not touched a Blu-Ray disk yet and probably won't until SlySoft is ready for BD+. I own too much media to be digging it out of a rack to play so I prefer it all ripped to my NAS where no sticky fingers can find it.


Word. I've got a Blu Ray drive on the horizon after I get back from my vacation and save up some cash after blowing $1000 on a mean Canon zoom lens for my camera ;)


Here's a question for you - what are you doing with DTS soundtracks to preserve them? I'm slapping those in FLAC too but am unsure if that's the best way to retain them.


I've yet to come across an HD-DVD that has a DTS track. If it's vanilla DTS, I would just keep it. Decoding DTS or passing it via S/PDIF is fairly simple. Actually, that's another reason to encode AC3 instead of FLAC - you can pass it via S/PDIF.

The more exotic flavors of DTS, I'm not totally sure, as I haven't tried. I'm sure the eac3to thread has details. I don't even know all what's out there :)


I've actually not done many H.264 movies yet, they seem to be few and far between and I do now have CoreAVC.


True. Most movies are VC-1.

~MiSfit

[BLKMGK]

I'll try using AC3 more, I think that FLAC still ought to be smaller than the original as it does compress. Admittedly I cannot hear the difference either!

D:\Video\eac3to>eac3to f:\Children\FEATURE_1.EVO+f:\Children\FEATURE_2.EVO 2: f:\Children\children-vid.mkv 4: f:\Childre
n\children-aud.ac3 -640 -libav

Will do it for others wondering. The .AC3 extension tells it to use AC3 and the -640 tells it the bitrate.

I do not recall the eac3to command spitting out the frames, none of the command responses in my scrollback have frames anywhere in them but it's possible. While it might not be required to use the framecount I've not yet had a single encode go out of synch:)

DTS, I've seen at least two films with this now although the names escape me - it's out there. How would you pull that out unmolested? I'll try to checkout the eac3to thread and update here if I find something helpful.

[Blue_MiSfit]

I think that FLAC still ought to be smaller than the original as it does compress


Nope. The original E-AC3 is lossy. FLAC is lossless.

Lossy -> Lossless means an increase in file size. Always.

JPEG -> TIFF means bigger. MP3/AAC/AC3 -> FLAC means bigger.

Uncompressed -> Lossless means a decrease in file size. But our source is not uncompressed :)

Lossy -> Lossy could be an increase or a decrease, depending on codec and settings.


DTS, I've seen at least two films with this now although the names escape me - it's out there. How would you pull that out unmolested? I'll try to checkout the eac3to thread and update here if I find something helpful.


I would assume the same way you pull out any track using eac3to.

~MiSfit

[Inventive Software]

Nope. The original E-AC3 is lossy. FLAC is lossless.

Lossy -> Lossless means an increase in file size. Always.

If the E-AC3 file in question is 2 channel, and uses the maximum 1536 Kbits allowed, then it's actually more than likely that FLAC will do a better job of compressing it.

However, here's where my thought comes in. Encoding the audio to FLAC for future playback only really makes sense when dealing with TrueHD and DTS-HD MA audio.

@BLKMGK: That's a good guide you got going BTW. If I ever get my hands on an HD-DVD drive, I'll be sure to follow it. :)

[BLKMGK]

Heh, and guess who just found a True-HD track? Yup, Constantine has a TrueHD soundtrack in it! Took me a little bit to get the commandline right, was half asleep!:rolleyes:

D:\Video\eac3to>eac3to f:\constantine\Feature_1.EVO+f:\constantine\Feature_2.EVO 2: f:\constantine\const-vid.mkv 5: f:\c
onstantine\const-aud.flac -libav
EVO, 2 video tracks, 7 audio tracks, 2:00:47
1: Joined EVO file
2: VC-1, 1080p24 (16:9) /1.001
3: VC-1, 480p30 (3:2) /1.001
4: E-AC3, 5.1 channels, 640kbit/s, 48khz, dialnorm: -27dB
5: TrueHD, 5.1 channels, 48khz, dialnorm: -27dB
6: E-AC3, 5.1 channels, 640kbit/s, 48khz, dialnorm: -27dB
7: E-AC3, 2.0 channels, 192kbit/s, 48khz, dialnorm: -27dB
8: E-AC3, 2.0 channels, 192kbit/s, 48khz, dialnorm: -27dB
9: E-AC3, 2.0 channels, 192kbit/s, 48khz, dialnorm: -27dB
10: E-AC3, 2.0 channels, 192kbit/s, 48khz, dialnorm: -27dB
Extracting primary video track...
Muxing video to Matroska...
Extracting audio track number 5...
Removing dialog normalization...
Encoding FLAC...
Creating/writing file "f:\constantine\const-aud.24bit.flac"...

[BLKMGK]

Looks like Blue_MiSfit is right about video frames being output by eac3to. I hadn't ever noticed it and didn't see it in my scrollback but when this last movie finished I spotted it ->

Extracting primary video track...
Muxing video to Matroska...
Extracting audio track number 5...
Removing dialog normalization...
Encoding FLAC...
Creating/writing file "f:\constantine\const-aud.24bit.flac"...
This audio track contains only 16 bit of information.
The zero bytes were successfully removed.
Added fps value to MKV header.
Video track 2 contains 173762 frames.
Video track 3 contains 204974 frames.
eac3to processing took 18 minutes, 28 seconds.
Done.

track two was the one I needed to process and that count matches what EVOdemux output. Looks like that tool isn't needed at all then.

[fitbrit]

Thanks for this... and thanks for letting the Lounge in the unRAID forums know about this too.:thanks:

[BLKMGK]

Okay, I'm not going to add this to the original post as it will get confusing but <smack> I've got a way to reduce scratch disk space used!

First, if you've got your drive hooked up to XP this will be harder. The ODF format of the HD-DVD is 2.5, XP native only understands 2.0 so you will need an ODF2.5 file driver. On Vista, which I run, ODF 2.5 is supported. That means I need not "rip" the entire disk. Since all I want is the feature EVO files and maybe the shorter trailers etc. I can just copy those files from the source disk. That saves some time and some disk space but there's an even better way!

eac3to can be pointed straight at the feature files as source and at your disk as target. Duh, this works great! Since it's been pointed out that eac3to also prints out framecounts that makes this even easier. By reading from the source disk you no longer have 20Gig of intermediate files on disk - a big help! Here's an example:

D:\Video\eac3to>eac3to y:\HVDVD_TS\FEATURE_1.EVO+y:\HVDVD_TS\FEATURE_2.EVO 2: f:\U571\u571-vid.mkv 4: f:\U571\u571-aud.A
C3 -640 -libav
EVO, 1 video track, 5 audio tracks, 1:56:10
1: Joined EVO file
2: VC-1, 1080p24 (16:9) /1.001
3: E-AC3, 5.1 channels, 1536kbit/s, 48khz, dialnorm: -27dB, 1001ms
4: DTS, 5.1 channels, 24 bits, 1536kbit/s, 48khz, 1001ms
5: E-AC3, 5.1 channels, 768kbit/s, 48khz, dialnorm: -27dB, 1001ms
6: E-AC3, 5.1 channels, 768kbit/s, 48khz, dialnorm: -27dB, 1001ms
7: AC3, 2.0 channels, 192kbit/s, 48khz, dialnorm: -27dB, 1001ms
Extracting primary video track...
Muxing video to Matroska...
Extracting audio track number 4...
Applying DTS delay...
Remapping channels...
Loading white noise (needed for dithering)...
Encoding AC3...
Creating/writing file "f:\U571\u571-aud.AC3"...
Added fps value to MKV header.
Video track 2 contains 167124 frames.
eac3to processing took 3 hours, 53 minutes.
Done.

Ya, done while other things were copying and processing so I'm hoping it wasn't I/O bound by my HD-DVD drive as nearly 4 hours is a bit long. Not that copying is very fast.:rolleyes:

Two things I've yet to fool with but am interested in:

1) MeGUI is capable of running postprocessing commands. MKVmerge has got a commandline component I think and the GUI could possibly even be called with file names. I think I've seen status in eac3to mentioning MKVmerge too. Wouldn't it be cool to have the files muxed as a postprocessing step? Writing a batch file to be called or a small helper app to do it seems possible. If anyone tries this I'd like to hear about it - I will try prowling the MeGUI thread to learn more too.

2) AVIsynth scripts can be pretty powerful. As was alluded to in Taktaal's tutorial (http://forum.doom9.org/showthread.php?p=1035422#post1035422) there are options to remove black bars and other things possible like trimming. I know pretty much squat about this scripting but something to make the video look better is obviously of interest to me and others. If anyone has some basic scripts or ideas to try fire away, I just don't want to double processing time.:rolleyes:

[BLKMGK]

I've just tried playing a movie that had a TrueHD soundtrack that I converted to FLAC with eac3to - no dice. It might be my receiver although honestly I'm not sure what eac3to converts it in exactly. I'm going to try it in AC3 to see if it plays. Just be advised that TrueHD could be an issue.

[BLKMGK]

Okay, picked up a boxed set of Harry Potter movies on HD-DVD. Looking at the first movie I'm in for a bit of a surprise. Instead of feature_1.evo etc. I've got Feature.evo and Feature_Divide.evo - BOTH at a bit over 11Gigs. the cover mentions something about including deleted scenes and indeed I see a Scene1 - 7 EVO but... How to join the files to get a single movie? Getting a copy that's whole with or without anything special is my goal. Either of the main EVO files could be the whole movie but it's not clear, has anyone else dealt with this one yet? I hope it's not like BD where you have to stich a pile opf clips together for a single movie:rolleyes: - grrr.

TIA!

[Greif]

How to extract chapter markers from HD-DVD & Blueray? I know madshi is working on including it in eac3to, but I'm just wondering what people use in the meantime.

did you figure out the Harry Potter EVOs?

[BLKMGK]

I'm not doing anything for chapters. The only Potter film that gave me issues was the first one - all of the others were normal.

[calmike79]

I've noticed that chapters ability is now in eac3to; and they showed up for me in March of the Penguins in the list of tracks; I easily extracted them to a text file and muxed it in with mkvmerge. However, my last 2 movies, Big Lebowski and Kiss Kiss Bang Band didn't even show the chapters in the list of "tracks" to extract. Looks like it's an early feature.