Abdvde (A Better DVD Encoder)

[Dinominant]

Abdvde (http://www.nathanshearer.com/modules/pages/pages/pages.php?page=28) is a bash script that automatically rips a DVD with minimal or no user input. It is ideal for backing up your DVD collection to your computer.

Features:
* NTSC and PAL support
* Physical DVD, ISO, or VIDEO_TS folder as input
* Batch mode operation: Encodes all titles, the longest title, or specified titles
* Cached operation: Dumps the DVD to your hard drive to speed up the encoding process
* Bitrate calculator: Encode to a percentage of the original bitrate or to a specific bitrate
* Automatic telecine and interlaced content detection. Supported methods are converted back into progressive material:
* 24000/1001p soft telecined to 30000/1001t
* 24000/1001p hard telecined to 30000/1001t
* 24000/1001p randomly soft and hard telecined to 30000/1001t
* 25p telecined to 25i
* 30000/1001p telecined to 30000/1001i
* 50f telecined to 25i
* 60000/1001f telecined to 30000/1001i
* VFR hard telecined to 30000/1001t (must be explicitly set)
* Automatic cropping
* Multiple audio streams
* Re-encode to you're favorite codec (supply your own arguments to the encoder if you want)
* Multiple subtitle streams
* Chapters
* Tagging (information like x264enc options, mencoder filter chain and original bitrate)

Abdvde is designed to work on linux and and doesn't require any windows tools installed with wine. Please check it out and let me know what you think: Abdvde (http://www.nathanshearer.com/modules/pages/pages/pages.php?page=28)

[setarip_old]

@Dinominant

Hi!

You refer to "rip" and "ripping" several times. With regard to DVDs, most users use the word "rip" to mean decrypt/remove copy protection and copy to a hard drive (or directly to blank burnable media).

Does your program decrypt commercial DVDs?

[Dinominant]

No, not directly. Technically mplayer does the decrypting and abdvde just calls mplayer (which is one of the dependencies). Ultimately abdvde is a wrapper script around several tools that do the grunt work (like mencoder and mplayer). But from the users perspective, you put in a DVD (or an ISO) and out comes an mkv (or several if you want to encode several titles).

[buzzqw]

nice script, but for me is a little less useful.. since i have only pal dvd (and you script threat it as vfr)

-ovc x264 -x264encopts qp=14:turbo=2:nob_pyramid:nodeblock:nocabac:me=dia:subq=0:threads=0 -of avi -o "$TMP/video.avi"
a little bit unoptimized ?

i suggest to check the froggy x264 profile for mencoder :)

BHH

[Dinominant]

Unoptimized? That's for sure. I was focusing more on quality and automation in the first release. Ultimately I want to have this script run on a headless machine in the closet/basement. Then I could simply copy an ISO onto the machine over the network and in a few hours/days it would create a bunch of mkv files, one for each title.

I'm going to have to find myself a PAL DVD so I can add support for it ^_^

PS: I just posted a new version which has some improvements like:
* bitrate calculation
* more sane defaults (for high quality rips)
* some overidable x264 settings
* better cropping
* higher quality scaling

[setarip_old]

mplayer does the decryptingI'm afraid you'll find that MPlayer will not be able to decrypt MANY, MANY more recent DVDs (Last year or two), due to the sophisticated, devilish copy protection schemes employed...

[Dinominant]

I've released a new version of Abdvde (http://www.nathanshearer.com/modules/pages/pages/pages.php?page=28) and added some documentation to configure it.

Abdvde 1.2.0.0
* Fixed several bugs affecting audio/video syncronization
* Fixed a bug in the source bitrate calculator
* Fixed a bug in the crop window calculator
* Fixed a bug in the longest title calculator
* Improved the scaling filter
* Improved the telecine detection algorithm and refactored the inverse telecine filter chain
* Added PAL support

@setarip_old

I havn't had any issues decrypting newer DVD's with libdvdcss-1.2.10 and mplayer-1.0_rc2_p28450. I could even decrypt a PAL disc with a region 1 drive.

[Quantumstate]

I tried to rip my Home Theater Tuneup disk, but it wasn't able to get the contrast and brightness pluge patterns.

These are one repetition of the pluge pattern, which is repeated a bunch of times in some kind of macro, to make it run for 3 minutes and give you a chance to make the adjustment.

What can I do? The rip is useless without the repetitive patterns

[setarip_old]

@Dinominant

To give you a better sense as to whether your program "package" will work with DVDs that have other-than-standard-CSS protection, I'd suggest you try it with some of the following DVDs (All are R1, unless otherwise stated):

Madagascar - RipGuard protection
Fool's Gold, Rendition - XProtect protection
Vince Vaughn's wild west comedy show
3:10 to Yuma
Many Disney releases - 2nd iteration of RipGuard protection
Hancock and many other 2006 - 2008 SONY releases - ArccoS protection
Grey's Anatomy S4, Eli Stone S1, Desperate Housewives S4 (All Disney-related) - 3rd iteration of RipGuard protection
The Dark Knight
WALL-E, High School Musical 3 - Yet another Disney protection scheme
The Express
My Best Friend's Girl,Bangkok Dangerous, Disaster Movie, W - LionsGate releases
Pineapple Express (PAL version)

Please let us know the results.

[setarip_old]

@Dinominant

Just curious - It's been a couple of weeks. How are you progressing?

[Dinominant]

@setarip_old

Of all those DVD's listed I was only able to find time to encode The Dark Night (Full Screen). It worked although mplayer was slow to detect the keys. I have many newer Disney/Pixar movies that I intend to try out, but I anticipate that it will decrypt them just fine based on what I saw with The Dark Night and Ratatouille.

Basically Abdvde dumps the entire disc to an ISO before mencoder starts encoding. The information displayed at the beginning of the process is read directly from the source device/iso/folder by mencoder so all the needed user input is acquired as soon as possible in the process so you don't have to babysit the application from beginning to end.

When dumping to an ISO first I run `filestat /dev/dvd` to unlock the drive. Then I run `dd if=/dev/dvd of=dvd.iso bs=2048 conv=noerror,sync` to create the ISO. The iso is then used as the source to free up the DVD drive.


@Quantumstate

I've found & fixed several bugs in the version of Abdvde that is currently released. I plan to release a newer one this week some time and maybe it'll rip those contrast and brightness pluge patterns properly with the fixes.

[Dinominant]

I've released a new version of Abdvde (http://www.nathanshearer.com/modules/pages/pages/pages.php?page=28).

Abdvde 1.3.0.0
* Fixed a bug in the VFR inverse telecine filter chain
* Fixed a bug that caused undesired frame skipping
* Added support for lpcm input
* Added support for DVD directory input
* Added support for several audio codecs: FLAC, MP3, PCM, Vorbis. More to come in later versions...
* Rebuilt the telecine detection algorithm. It can now reliably identify 30000/1001i and 60000/1001f
* Added the ability to override the detected telecine method from the command line
* Renamed some temporary files to more descriptive names

There are some new dependencies (if you want to re-encode your audio streams to another codec), so make sure you install them after you update you're config.

I've tested this version with NTSC and PAL formats and it reliably detected the following telecine methods and applied the appropriate IVTC/deinterlacing filter chain without any human input:
* 24000/1001p soft telecined to 30000/1001t
* 24000/1001p hard telecined to 30000/1001t
* 24000/1001p randomly soft and hard telecined to 30000/1001t
* 25p telecined to 25i
* 60000/1001f telecined to 30000/1001i

I do not have any discs with "30000/1001p telecined to 30000/1001i" content. However the detection algorithm should properly identify it as it does with "25p telecined to 25i" since the difference between the two methods is only the duration the frames are displayed. If you have such a disc, please test it and let me know if the algorithm worked ^_^

[Erik_Osterholm]

@Dinominant

Great utility--I just started trying it out and tweaking it for my use.

I wonder if you'd be interested in using 7z instead of poweriso, since 7z is open source and available through a couple of distribution's package managers?

The line in which you use poweriso should be directly replaceable with:
7z -e -o"$TMP" "$INPUT" VIDEO_TS/VTS_01_0.IFO > /dev/null

Also, have you considered adding turbo=1 to the first pass options? Per the mencoder man page, this option should not change the PSNR (and has a nearly unnoticeable affect on final encode quality), while yielding a significant performance boost for that first pass.

[fangorn]

Hi,

Thanks for your work. This is exactly my style. A fire and forget solution. :thanks:

Nonetheless I have a problem with a PAL DVD that does make problems. I know the source is progressive (hollywood blockbuster) but telecining detection fails and the program exits.

Now I tried to understand what your function does. I see that you are encoding two versions comparing them and based on the difference decide if the pictures are nearly the same (95 % as standard).

Now to my questions:
1. What is the criterium that tells you that it is telecined progressive?
2. What happens according to your plan, when I feed TV productions (true interlaced video source) to the script? As far as I can see, the script exits in that case.
3. Is it possible to define a criterium to detect true interlaced sources and handle them accordingly (in my case deinterlace them)? Or in a second step: TV shows that are part interlaced and part progressive (like 24 or Angel are told to be).

Again, thanks for your work,
fangorn

btw.: I am using mplayer svn and had to adjust some of the command lines to make them work (the -identify runs to be precise). Is it possible that detection fails because of a similar error? I am sorry, I don't have the changes with me, but I can post them later.

[Henrikx]

Why Closed Source poweriso?
AcetoneISO,Furius ISO Mount are good alternatives.

[fangorn]

This is the code missing to get the mplayer identify runs working
-identify -vc null -vo null -ao null

Concerning my problem with the telecining code, I lowered it down to mplayer not able to read the source. Strange thing: I echo the original command line to output and if I run the exact command line, it works flawless. When run by hand from the command line the detection algorithm works. So it's just mplayer not able to read the source.

[Dinominant]

I've released another new version of Abdvde which has several bug fixes and new features:

Abdvde 1.4.0.0 (http://www.nathanshearer.com/modules/pages/pages/pages.php?page=28)
* Fixed a bug where the telecine method override was always set to "60000/1001f telecined to 30000/1001i" then forgotten for other titles
* Fixed a bug where the input framerate was not set when the telecine method was supplied by the user
* Fixed a bug where a dvd was not unlocked when it is already in the drive
* Changed the default interlace tolerance from 5% to 20%
* Changed the default basename for ISO input to the filename
* Changed the poweriso dependency to 7z
* Changed the PAL telecine detection algorithm to assume "25p telecined to 25i" when the interlace tolerance is exceeded
* Added improved progress output
* Added wavpack support
* Added shorten support
* Added turbo=1 to the first x264 encoding pass
* Added the ability to choose the number of encoding passes for x264

There are several new variables you can override from the config files and additional new dependencies (only if you want to use wavpack and shorten). More information is available here (http://www.nathanshearer.com/modules/pages/pages/pages.php?pages[display_page][id]=68).

@Erik_Osterholm
Thanks for the suggestions. I immediatly implemented those changes into this new version.

@fangorn
The telecine detection algorithm isn't perfect, but from what testing I did it seemed to work pretty well. I've noticed that sometimes it detects "60000/1001f telecined to 30000/1001i" as "24000/1001p hard telecined to 30000/1001t". This is why I increased the interlace tolerance from 5% to 20% in this new version. Also, if the detection fails you can override the detection from the command line for the oddball DVD's.

@Henrikx
Yeah, poweriso was just the first package I found in portage that could extract an ISO in userspace. That Closed Source app is no longer needed as a dependency (replaced by p7zip) ^_^

Basically the interlace detection algorithm works like this:
It encodes 2 versions of a sample of the film. One is unmodified, and the other is passed through the phase filter. The phase filter deinterlaces on a frame-by-frame basis by delaying a field (which is usually used to deinterlace/ivtc PAL). Then it compare the various frames. If a frame has been modified then it assumes the original frame was interlaced.

Here is where the math comes in (keep in mind that 40% is 2/5 interlaced frames -- aka "24000/1001p telecined to 30000/1001t"):
* If 0% (0-20% with a 20% tolerance) are modified then it determines the input is "60000/1001f telecined to 30000/1001i". We can assume that it's not 30000/1001p because the DVD standard says that everything must be interlaced (I don't remember where I read that).
* If 100% (80-100% with a 20% tolerance) are modified then it determines the input is "30000/1001p telecined to 30000/1001i" or "25p telecined to 25i" for PAL discs
* If the tolerence is exceeded then it assumes "24000/1001p telecined to 30000/1001t". Either hard or randomly depending on CFR vs. VFR input.
* PAL discs are always detected as "25p telecined to 25i" because I have a very limited supply of PAL discs to test with and they all happen to be telecined that way.

[fangorn]

:thanks:

* PAL discs are always detected as "25p telecined to 25i" because I have a very limited supply of PAL discs to test with and they all happen to be telecined that way.

That is correct for all progressive sources like cinematic movies that were mostly just sped up from 24p.

Is it correct when I assume that the other way around the source is most likely interlaced?

[microchip8]

:thanks:



That is correct for all progressive sources like cinematic movies that were mostly just sped up from 24p.


ummm, most PAL DVDs are just sped up (and thus have a 4% higher audio pitch) and there's no telecining involved. Only a very small minority are 2:2:2:2:2:2:2:2:2:2:2:3 pulled down

You'll find much more true interlaced PAL DVDs than ones that are pulled down to 2:2:2:2:2:2:2:2:2:2:2:3

Also content produced in PAL/SECAM countries is directly shot at 25fps so no speeding up is required

[fangorn]

That brings me back to my question.

3. Is it possible to define a criterium to detect true interlaced sources and handle them accordingly (in my case deinterlace them)? Or in a second step: TV shows that are part interlaced and part progressive (like 24 or Angel are told to be).

[Dinominant]

Hi,
3. Is it possible to define a criterium to detect true interlaced sources and handle them accordingly (in my case deinterlace them)? Or in a second step: TV shows that are part interlaced and part progressive (like 24 or Angel are told to be).

Abdvde already detects 60000/1001f material and filters it into 60000/1001p. Are you talking about 50f material on PAL discs? Is there such a disc? If so I can can detect it and convert it into 50p but without any discs I wont' be able to properly test it...

[nm]

Abdvde already detects 60000/1001f material and filters it into 60000/1001p. Are you talking about 50f material on PAL discs? Is there such a disc?
If you have PAL discs around, check the extras. They are often interlaced video (sometimes badly converted from NTSC, so be careful).

[Dinominant]

I've released another new version of Abdvde:

Abdvde 1.5.0.0 (http://www.nathanshearer.com/modules/pages/pages/pages.php?page=28)

Fixed a bug where crop detection could fail with 50f and 60000/1001f content
Added support for "50f telecined to 25i" DVD's

[Erik_Osterholm]

Do you happen to notice a number of sync problems when encoding in this way? I'd say that about half of the movies I encode end up with sync problems. About half of those can be fixed by simply remuxing with an audio delay, but the rest have a progressive desync that is much harder to fix.

ETA:
Well, going back and trying a couple with 1.5.0.0, and they're in perfect sync. I'll assume for now that it's fixed. Thanks!

[fangorn]

Abdvde already detects 60000/1001f material and filters it into 60000/1001p. Are you talking about 50f material on PAL discs? Is there such a disc? If so I can can detect it and convert it into 50p but without any discs I wont' be able to properly test it...

I am talking about truly interlaced video, shot with cameras that take 50 half frames per second and coded as interlaced MPEG2. No matter how that is called in MPEG specs.

[Erik_Osterholm]

So I'm still getting some A/V sync issues on a number of DVDs. Is no one else noticing this?

I'm thinking of modifying the script to use HandBrakeCLI for a couple of reasons:
1) It can output VFR.
2) Its de-telecine filter seems to work a little better than mencoder's filters.


Combined, these seem to address my issues. When I run across a DVD on which abdvde has sync problems, encoding through the Handbrake GUI usually produces a synchronized encode. The real trick will be automatically detecting the sorts of files which give me trouble. Any tips?

[setarip_old]

Is this tool capable of ripping FULL DVDs (Menus, movies, "extras") or simply "movie only"?

[nm]

It's an encoding script for video titles on DVD, so it handles the main movie and video extras but not the menus. Same as MPlayer and MEncoder themselves without dvdnav support.

[setarip_old]

@nm

Thanks for the confirming info. I figured it was limited in that way...

[Erik_Osterholm]

I've done a little more debugging (mostly educated trial-and-error with the mencoder options) and it looks like most of my troubles have gone away with the elimination of -noskip.

The mencoder docs at http://www2.mplayerhq.hu/DOCS/HTML/en/menc-feat-dvd-mpeg4.html don't exactly provide much information on this flag, but they do imply that you shouldn't use it if you use filters which unpredictably add or drop frames--which I would think includes filmdint.

If you want to further guard against strange frame skips and duplication, you can use both -mc 0 and -noskip. This will prevent all A/V sync, and copy frames one-to-one, so you cannot use it if you will be using any filters that unpredictably add or drop frames, or if your input file has variable framerate! Therefore, using -noskip is not in general recommended.
(emphasis mine)

So it looks like -noskip should be removed at the very least in times when abdvde detects VFR input. I've tested abdvde without the -noskip flag on a couple of movies, both FILM and MIXED, and so far I've had perfect sync in all cases.

Does this make sense, or am I missing something?

[Dinominant]

I've done a little more debugging (mostly educated trial-and-error with the mencoder options) and it looks like most of my troubles have gone away with the elimination of -noskip.

The mencoder docs at http://www2.mplayerhq.hu/DOCS/HTML/en/menc-feat-dvd-mpeg4.html don't exactly provide much information on this flag, but they do imply that you shouldn't use it if you use filters which unpredictably add or drop frames--which I would think includes filmdint.


(emphasis mine)

So it looks like -noskip should be removed at the very least in times when abdvde detects VFR input. I've tested abdvde without the -noskip flag on a couple of movies, both FILM and MIXED, and so far I've had perfect sync in all cases.

Does this make sense, or am I missing something?

I have noticed a few DVD's have offset issues. Initially I added the -noskip because filmdint and pullup had problems deinterlacing on the boundries where the framerate changed. The result was a single duplicate frame (the filter dropped the wrong frame). -noskip fixed that. But if removing -noskip fixes the audio sync issue I'll remove -noskip instead and live with a single duplicate frame.

I'm thinking of writing a deinterlacing filter for mencoder anyways that can output VFR (via nullframes at 120fps) so that would fix the duplicate frame problem.

I'm testing Abdvde 1.6.0.0 with a few DVD's to iron out any bugs I may have introduced. Hopefully the new version will fix the audio sync issues for all DVD's.

[Erik_Osterholm]

Getting VFR would be utterly fantastic. Have you considered implementing your solution from elsewhere in the forums into abdvde?

[Dinominant]

Getting VFR would be utterly fantastic. Have you considered implementing your solution from elsewhere in the forums into abdvde?

Yes; In fact it's already done. Tonight I'm going to batch encode several movies and if nothing breaks I'll post Abdvde 1.6.0.0 with VFR support tomorrow afternoon ^_^

[microchip8]

I'm thinking of writing a deinterlacing filter for mencoder anyways that can output VFR (via nullframes at 120fps) so that would fix the duplicate frame problem.



That would be awesome and very much appreciated by those who want/use VFR (not me personally) and could be used by many other apps, not just yours ;)

DO IT :D http://www.youtube.com/watch?v=lmUZGdi7Ty4

[Dinominant]

Abdvde now has basic VFR support:

Abdvde 1.6.0.0 (http://www.nathanshearer.com/modules/pages/pages/pages.php?page=28)

Added support for "VFR hard telecined to 30000/1001t".
Added a switch to print information about the DVD.
Fixed a bug in the crop detection function where illegal crop values broke everything.

Currenlty only mixed 30000/1001t and 30000/1001p is supported. Mixed 60000/1001f, 30000/1001p, and 30000/1001t is planned.

[Erik_Osterholm]

Spiffy!

I'm having some problems, though:
"[avi @ 0xaaa810]Could not find codec parameters (Video: rawvideo, 720x480)"

The resulting video is pretty broken--it doesn't show anything past about 1 second in (estimating) and it just skips back and repeats those frames over and over.

What version of Mencoder are you using?

[Dinominant]

Spiffy!

I'm having some problems, though:
"[avi @ 0xaaa810]Could not find codec parameters (Video: rawvideo, 720x480)"

The resulting video is pretty broken--it doesn't show anything past about 1 second in (estimating) and it just skips back and repeats those frames over and over.

What version of Mencoder are you using?

I ran a VFR test of Babylon 5 today. It completed without any errors, but I did noticed the video was 250ms slow. Remuxing with mmg after the encode fixed it. This is because -mc 0 and -noskip are (currently) required randomly seek the input video (and feed the various pipes). Somebody recently worked on a patch to start at an arbitrary frame (not just key frames) via a command line argument. That patch would allow me to remove the -noskip since I wouldn't need to convert the framerate to 1fps anymore.

$ emerge -pv mplayer

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild R ] media-video/mplayer-1.0_rc2_p20090731 USE="3dnow 3dnowext X a52 aac alsa ass cddb cdio cdparanoia cpudetection dirac doc dts dv dvd dvdnav enca encode faac faad ftp gif iconv ipv6 jpeg ladspa live lzo mad mmx mmxext mng mp2 mp3 network openal opengl osdmenu oss png quicktime rar real rtc samba schroedinger shm speex sse sse2 tga theora tremor truetype unicode v4l vorbis x264 xanim xscreensaver xv xvid -aalib (-altivec) -bidi -bindist -bl -custom-cflags -custom-cpuopts -debug -dga -directfb -dvb -dxr3 -esd -fbcon -ggi -gmplayer -jack -joystick -libcaca -lirc -md5sum -nas -pnm -pulseaudio -pvr -radio -sdl -ssse3 (-svga) -teletext -v4l2 -vdpau (-vidix) (-win32codecs) -xinerama -xvmc -zoran" VIDEO_CARDS="nvidia vesa -mga -s3virge -tdfx" 0 kB

Total: 1 package (1 reinstall), Size of downloads: 0 kB

$ mencoder
MEncoder SVN-r29463-4.3.2 (C) 2000-2009 MPlayer Team
No file given

Exiting... (error parsing command line)

[setarip_old]

@Dinominant

Perhaps you can determine (and post back here) how successful your program is at ripping the following:

"Transformers: Revenge of the Fallen"
"StarTrek 2009"

Both appear to use variants of the "XProtect" copy protection concept.

[Dinominant]

@setarip_old

Well, I tried Transformers: Revenge of the Fallen last night and it ripped it just fine.

First I dumped the disc do my hard drive as an ISO which took about 3 hours because of some deliberate bad sectors.

The disc had 99 titles on it, most of which were fake. I watched a few seconds of the movie with VLC, noticed that the correct title to rip was 47 and ran this command.

abdvde -b 47 "Transformers - Revenge of the Fallen.iso"

This morning I opened "Transformers - Revenge of the Fallen.mkv" and it played perfectly fine.

[setarip_old]

noticed that the correct title to rip was 47So nothing in your program determines/determined the correct title to be ripped? And, again, it's not capable of ripping Full DVDs (just "movie-only")?

[Dinominant]

Abdvde 1.6.0.0 assumes that the longest title is likely the one that you want to rip. In the case of this movie there are other longer bogus titles on the disc.

If you wanted to you could rip all the titles, but it would not rip any of the menu's. The next version (the one I'm currently debugging) has the ability to dump the entire disc to an ISO which will give you a full copy of everything although it will be an ISO, not an MKV with your favorite codecs. The ISO will probably average 7GB in size for most movies.

I've been researching ways to rip the menus and have only really come up with DvdMenuXtractor (http://ffdshow.faireal.net/mirror/DvdMenuXtractor/) which appears to be 2 years old, windows only, and abandoned.

[rsalerno]

I really like the idea of your program and I like the approaches you've taken with it. Thanks so much for putting so much time into it....

I ran a disc through it with your default settings and the video quality is quite nice, but I'm having some trouble with the file it muxed together, and of course all the sources it made are cleaned up and gone.

mkvmerge rejects the file and spits this out:
Error: The demultiplexer for the file './d1.1.mkv' failed to initialize:
matroska_reader: Failed to read the headers.

And mediainfo can only report the format and size of the file.

I also get seeking problems and the chapters and subtitles didn't appear to make it, but I don't know if this is just because of the bad header or if they're actually additional problems. I'm assuming header.

Does this sound like a misconfiguration somewhere on my system? Your script finished without any notice of an error, so I wasn't sure where to start. Have you run into this behavior before?

// END vague bug report, BEGIN lame feature request

Also, would you consider adding a simple config switch for using xvid at some point? Your custom x264 args are passed to a baked in call for x264, so there's no easy override for using a different codec. I know it's not as good as x264, but I still have some standalone players I'm trying to keep in the loop as I rip my collection....

Or if you don't feel it's worth it to add all the if/else logic, maybe an abdvde-xvid version of the script? I'll play around with that myself, but you have significantly more finesse with this than I do, and I'm confident your results would be better.

[Dinominant]

@rsalerno
Thanks for the support ^_^

I'm in the process of debugging a new version that has better output and facilities to for additional video codecs like xvid. I have some rather large assignments due in the next couple weeks that I need to focus on and when those are complete I can spend some time on Abdvde and get the next version stabilized, documented, and released. If nothing else it'll help determine what is causing problems since it'll have better error handling.

Features planned for (and already implemented) in the next release:

Added a more compact printout of DVD information.
Added email notifications.
Added ability to customize MKV tags.
Added a progress bar to the dumping phase.
Added example config files with default values.
Restructed the temporary files and folders into a more logical hierarchy.
Refactored abdvde_extract_audio and abdvde_encode_audio.
Enabled downmixing 5.1 to 2.0 for mp3 since lame does not support more than 2 channels.
Added AAC support.
Added faac as an optional dependency.
Added AC3 support.
Added aften as an optional dependency.
Disabled replay gain for flac since it doesn't work on files with more than 2 channels.
Added OGM container support.
Added ogmtools as an optional dependency.
Added SRT subtitle support for the ogm container.
Added subtitle2pgm as an optional dependency.
Added ability to enable/disable video, audio, subtitles, chapters, and tags.
Added a switch to dump an ISO image.
Added support for multiple video codecs.
Added ability to copy video unmodified.
Removed ffmpeg as a dependency.
Removed transcode as a dependency.
Changed default AUDIO_CODEC to "copy".
Added lots of comments.


The next release will be able to copy all streams into an mkv container without re-encoding them.

I think it would be trivial to add support for xvid and other codecs after I release what I currently have and I plan to add support for xvid next.

[Dinominant]

I managed to find some time this weekend to test and document Abdvde 2.0.0.0 (http://www.nathanshearer.com/modules/pages/pages/pages.php?page=28). This version has a large number of changes that break backwards compatability with existing config files. The installation is a little different since there are more files, but it's still a 2 command copy and paste procedure ^_^.

Changes in Abdvde 2.0.0.0 (http://www.nathanshearer.com/modules/pages/pages/pages.php?page=28)

Added example config files with default values.
Added a more compact printout of DVD information.
Added email notifications.
Added ability to modify the niceness of the script.
Added a switch to dump an ISO image.
Added ability to enable/disable video, audio, subtitles, chapters, and tags.
Added ability to copy video unmodified.
Added ability to customize MKV tags.
Added OGM container support.
Added AAC support.
Added AC3 support.
Added a progress bar to the dumping phase.
Restructed the temporary files and folders into a more logical hierarchy.
Refactored abdvde_extract_audio and abdvde_encode_audio.
Enabled downmixing 5.1 to 2.0 for mp3 since lame does not support more than 2 channels.
Added faac as an optional dependency.
Added aften as an optional dependency.
Disabled replay gain for flac since it doesn't work on files with more than 2 channels.
Added ogmtools as an optional dependency.
Added SRT subtitle support for the ogm container.
Added subtitle2pgm as an optional dependency.
Added support for multiple video codecs.
Removed ffmpeg as a dependency.
Removed transcode as a dependency.
Changed default AUDIO_CODEC to "copy".
Added lots of comments.
Added much better logging facilities.

[rsalerno]

I managed to find some time this weekend to test and document Abdvde 2.0.0.0 (http://www.nathanshearer.com/modules/pages/pages/pages.php?pages%5Bdisplay_page%5D%5Bid%5D=68).

Awesome!

I'm looking forward to testing this out. Thanks so much for the continued work. I'll report back if I encounter anything odd.

Edited to add: Maybe I'm just a moron, but the dependency check causes a stalemate on Ubuntu. Apparently our package of libdvdread doesn't include filestat? I can't get anything useful out of google on this issue. I will try commenting out that particular check from your script and see if things break when I pass it my own iso (since that's the only part that needs filestat, right?)....

[rsalerno]

Okay, so there are a few issues on Ubuntu, apparently:

* Dependency check for filestat
Don't know how to get around this at all. Deleted check for it.

* Dependency check for libdvdread
I know this is installed in my system, and it shows in the abdvde log that it's being used, even, but for some reason, the dependency check doesn't work here. I changed the check to look for libdvdread4 (which is the name of the current package) but that didn't help. I have libdvdread.so in /usr/lib64, so maybe this is a 64bit issue? I didn't test this at all. I just deleted the check for libdvdread because I knew it was installed and working.

* Dependency check for subtitle2xxx
The suggested package didn't exist when this check failed, but I installed transcode and a couple subtitle tools and the check passed (I'm pretty sure transcode did it, but I was frustrated and sort of carpet bombed this one, so I'm not sure which package actually held the key).

After those errors were gone, it ran okay. It does give this error:
/usr/bin/abdvde: line 1652: /tmp/abdvde/info/log: No such file or directory
before doing anything else, but then it continues on just fine.

I tried running it with a disc in the drive, just for giggles, and it polled it and brought up the table of contents asking which title to rip. I picked one and it said "No more user input required." and then died thusly:

/usr/bin/abdvde: line 361: bar: command not found

Ripping title 1.
Calculating video bitrate:
/usr/bin/abdvde: line 1372: * 50 / 100 : syntax error: operand expected (error token is "* 50 / 100 ")

I'm sure this is the filestat issue, and the error reporting is just confused. I was reasonably sure it would break at the actual ripping stage, and it did. The log confirms some confusion over trying to read an iso that doesn't exist.

Next, I ripped an iso separately and fed it to the script, setting video and audio to copy (in the conf file in my home folder) for speed considerations.

It took a bit longer to poll the TOC, and spit some warnings along the way:
| 10 | 1 | 6 | 1326 | en, ac3 (stereo) | |
| | | | | en, ac3 (stereo) | |
/usr/bin/abdvde: line 1313: [: -ne: unary operator expected
/usr/bin/abdvde: line 1325: [: -ne: unary operator expected
| 11 | 1 | 1 | 0 | | |
/usr/bin/abdvde: line 1337: [: -gt: unary operator expected
/usr/bin/abdvde: line 1342: [: -eq: unary operator expected
/usr/bin/abdvde: line 1313: [: -ne: unary operator expected
/usr/bin/abdvde: line 1325: [: -ne: unary operator expected
| 12 | 1 | 1 | 0 | | |
/usr/bin/abdvde: line 1337: [: -gt: unary operator expected
/usr/bin/abdvde: line 1342: [: -eq: unary operator expected
| 13 | 1 | 1 | 10 | en, ac3 (stereo) | |
| 14 | 1 | 1 | 13 | en, ac3 (stereo) | |

(I was playing with a disc of TV shows so the files would be small and things would finish more quickly than last time.)

But it still let me me pick a title to encode. Detecting the telecine method took quite a while (especially for ostensibly just copying the video), maybe on the order of 8 - 10 minutes?

The copying went very quick and then it hit an error because I apparently didn't have dvdxchap installed (did a fresh install recently). This was in ogmtools, which was just an option dependency check, I think, but it seems to be the cause of the script exiting:
Error: Unknown chapter file format in '/tmp/abdvde/mux/chapters.txt'. It does not contain a supported chapter format.
abdvde_cleanup_title
abdvde_email
abdvde_cleanup

I manually muxed the files left over in the mux folder and everything was fine. No hitches in mediainfo like before, and it played and worked as expected.

I'll try another encode using your x264 settings and report back if there are any additional errors or issues....
Edited to add: Using your default and fast settings, I had no problems whatsoever with either file, and no additional errors during any part of the process.
Later edit: Curiously, the chapters and mkvtags files won't manually mux when I load them into mkvmerge-gui, but I know the file your script made had the correct chapters. Is this a known bug in the gui program? Also, the leftover video file in the tmp directory is an avi and has a giant pcm soundtrack attached to it. Again, it was muxed by your script correctly, but this discovery kind of confused me. Possible glitch in the mencoder routine, or intentional choice for some reason beyond me?

Thanks again for the time you're putting into this!

[Dinominant]

I've finally finished writing exams and found some time to fix a few bugs.

* Dependency check for filestat
Don't know how to get around this at all. Deleted check for it.
I was initially using filestat to unlock the drive. I've now removed it altogether since mencoder does the same. Also, there is no check for libdvdread even though without it mencoder and mplayer are quite useless (unless you are backing up unencrypted dvd's).

* Dependency check for subtitle2xxx
The suggested package didn't exist when this check failed, but I installed transcode and a couple subtitle tools and the check passed (I'm pretty sure transcode did it, but I was frustrated and sort of carpet bombed this one, so I'm not sure which package actually held the key).
Under Gentoo (the distro I use) subtitle2xxx is actually in separate packages. I found that Ubuntu has subtitle2xxx inside subtitleripper.

/usr/bin/abdvde: line 1652: /tmp/abdvde/info/log: No such file or directory
This bug is fixed in 2.0.1.0.

/usr/bin/abdvde: line 361: bar: command not found
This is because bar is not installed. I added a check so it'll dump the disc without a progress bar (http://clpbar.sourceforge.net/) when bar is not installed.

| 10 | 1 | 6 | 1326 | en, ac3 (stereo) | |
| | | | | en, ac3 (stereo) | |
/usr/bin/abdvde: line 1313: [: -ne: unary operator expected
/usr/bin/abdvde: line 1325: [: -ne: unary operator expected
| 11 | 1 | 1 | 0 | | |
/usr/bin/abdvde: line 1337: [: -gt: unary operator expected
/usr/bin/abdvde: line 1342: [: -eq: unary operator expected
/usr/bin/abdvde: line 1313: [: -ne: unary operator expected
/usr/bin/abdvde: line 1325: [: -ne: unary operator expected
| 12 | 1 | 1 | 0 | | |
/usr/bin/abdvde: line 1337: [: -gt: unary operator expected
/usr/bin/abdvde: line 1342: [: -eq: unary operator expected
| 13 | 1 | 1 | 10 | en, ac3 (stereo) | |
| 14 | 1 | 1 | 13 | en, ac3 (stereo) | |
I'm not sure what caused this. It could be a result of the previous bugs messing things up later in the process. Please let me know if this bug still happens in 2.0.1.0.

Detecting the telecine method took quite a while (especially for ostensibly just copying the video), maybe on the order of 8 - 10 minutes?
Detecting the telecine method is not required when copying streams unmodified (afaik). I'll see if I can eliminate that step in the next version (along with adding support for xvid and other video codecs).

The copying went very quick and then it hit an error because I apparently didn't have dvdxchap installed
I forgot to add a check for dvdxchap (my bad). Its fixed in this version ^_^

Curiously, the chapters and mkvtags files won't manually mux when I load them into mkvmerge-gui, but I know the file your script made had the correct chapters. Is this a known bug in the gui program?
I'm not sure why mmg isn't accepting the generated chapters file. If the command line version works I would expect the gui version to also work. Perhaps there is a bug in mmg.

Also, the leftover video file in the tmp directory is an avi and has a giant pcm soundtrack attached to it. Again, it was muxed by your script correctly, but this discovery kind of confused me. Possible glitch in the mencoder routine, or intentional choice for some reason beyond me?
When encoding the video I had to add an audio track so mencoder would add/drop frames to correct the a/v sync. It's kinda a hack because if I didn't include the audio when encoding the video stream there would be huge a/v sync issues later when muxing. I may be able to downmix the audio stream a large amount to save disk space but that would take cpu time away from encoding the video.

Please let me know if you notice any other bugs ^_^

Changes in Abdvde 2.0.1.0 (http://www.nathanshearer.com/modules/pages/pages/pages.php?page=28):

Fixed a bug where ISO images were not created correctly.
Added ability to create disc images when bar is not installed.
Eliminated the dependency on filestat.
Added an optional dependency check for dvdxchap.

[jajasdad]

Enjoyed successfully using your script. Started with Ver 1.6 a bit ago then tried 2.0 which had a few issues as noted. I'm using Ubuntu 8.04 LTS primarily which lacks the latest packages.

Following applies to 2.0 have not yet tried latest

For anyone else who might get an error on mkvmerge
My version of mkvmerge v2.0.2 balked on the following params, so had to comment out.

for FILE in "$TMP/mux/video"*; do
#MKVMERGE_ARGS[${#MKVMERGE_ARGS[@]}]="--no-audio"
#MKVMERGE_ARGS[${#MKVMERGE_ARGS[@]}]="--no-subtitles"

My older version of Ubuntu used libdvdread3 so and also could not use filestat.

iso bug ?.... I noted that mv dvd.iso command ended up creating .iso when i tried it.

Also noted the "No such file or directory" errors although they looked like they were there

Had to download command line progress bar from sourceforge (.deb file)

"-eq: unary operator expected" can induce these errors by running script on incompatible source so might be VARIABLES not being set in those cases.

Have seen some mkvmerge audio sync issues when dumping from disc to iso. Will investigate latest scripts but suspect might be outdated system packages. vobcopy on same iso will work ok incidentally

Will try out latest version

[rsalerno]

Sorry I've been slacking on testing out all your changes....

/usr/bin/abdvde: line 1652: /tmp/abdvde/info/log: No such file or directory
This bug is fixed in 2.0.1.0.
Confirmed fixed.

This is because bar is not installed. I added a check so it'll dump the disc without a progress bar (http://clpbar.sourceforge.net/) when bar is not installed.
Ahhh.... I was not aware of bar. I ran the new version without installing it, and it started ripping to iso seemingly fine, so the filestat check seems to have been the only thing holding that back previously. The disc I've been working on and/or my drive have been giving me trouble and I haven't been able to successfully rip it in linux yet. I let your script have a go for about 14 hours and then killed it unfinished. I don't think this is a bug in your script at all, but I haven't gotten around to trying a few more discs yet. I have also since installed bar, so I'm curious to see that work on a working disc as well.

* weird unary operator errors when reading/displaying TOC *
I'm not sure what caused this. It could be a result of the previous bugs messing things up later in the process. Please let me know if this bug still happens in 2.0.1.0.
| 9 | 1 | 41 | 10601 | en, ac3 (stereo) | |
| | | | | en, ac3 (stereo) | |
| 10 | 1 | 6 | 1326 | en, ac3 (stereo) | |
| | | | | en, ac3 (stereo) | |
/usr/bin/abdvde: line 1318: [: -ne: unary operator expected
/usr/bin/abdvde: line 1330: [: -ne: unary operator expected
| 11 | 1 | 1 | 0 | | |
/usr/bin/abdvde: line 1342: [: -gt: unary operator expected
/usr/bin/abdvde: line 1347: [: -eq: unary operator expected
/usr/bin/abdvde: line 1318: [: -ne: unary operator expected
/usr/bin/abdvde: line 1330: [: -ne: unary operator expected
| 12 | 1 | 1 | 0 | | |
/usr/bin/abdvde: line 1342: [: -gt: unary operator expected
/usr/bin/abdvde: line 1347: [: -eq: unary operator expected
| 13 | 1 | 1 | 10 | en, ac3 (stereo) | |
| 14 | 1 | 1 | 13 | en, ac3 (stereo) | |

It's still there, though the line numbers have moved (predictably, I'd guess). I'm not convinced this isn't a problem in the iso I'm using, since I had such trouble getting it ripped in the first place. Will report back after I've tried a few more discs.

Detecting the telecine method is not required when copying streams unmodified (afaik). I'll see if I can eliminate that step in the next version (along with adding support for xvid and other video codecs).
Yeah, I had intended to say something to the effect of "Why don't you skip that step if copy is selected?" but then didn't want to risk sounding even more demanding of you, so just reported the behavior....

RE: other codecs - I poked around and I'm really impressed with what you've done so far. I'm a bit over my head with the telecine options, though, so I never tried subbing in my own xvid parameters. I look forward to whatever you do here, whenever you find the time to do it....

I'm not sure why mmg isn't accepting the generated chapters file. If the command line version works I would expect the gui version to also work. Perhaps there is a bug in mmg.
This is totally nothing to do with your script or dvdxchap. I used a few other sources to generate the chapter files to the same spec, and they all still go in fine through mkvmerge, but mmg balks every time. Sorry for the false alarm.

When encoding the video I had to add an audio track so mencoder would add/drop frames to correct the a/v sync. It's kinda a hack because if I didn't include the audio when encoding the video stream there would be huge a/v sync issues later when muxing. I may be able to downmix the audio stream a large amount to save disk space but that would take cpu time away from encoding the video.
Ahhhhh. Interesting. Yeah, no need to compress it in my opinion. CPU cycles > disk space. Would it not be faster to just copy the audio wholesale, though, instead of including it uncompressed? Or, I guess mencoder would do that real-time, so it doesn't matter. You're the boss. I was just curious why this was, and I'm inclined to trust your expertise here.

==============

Additional bugs (?): As I've been writing this, I've noticed that it's been hanging at "Detecting crop window:" for about half an hour. The log wasn't very descriptive:The selected video_out device is incompatible with this codec.
Try appending the scale filter to your filter list,
e.g. -vf spp,scale instead of -vf spp.
Unsupported PixelFormat -1
I killed it, and tried again on the same title I encoded with the older version, and it's actually working now. I'm inclined to think I've got a more busted iso than I thought.

I'll grab a few other discs and report back in a few days. I am a useless bug reporter.

[jajasdad]

Ran a few quick easy tests succesfully on latest script
Dumping a disk to an ISO
Dumping a title (make mkv) from a disk
Creating an mkv from an existing VIDEO_TS folder

In DEBUG mode, ran a couple of tests. Seemed though to mux the 1st tests audio clip, on the 2nd test, instead of the 2nd. They both used the same temp folder abdvde/ with no postfix datetimestamp. So I guess something went awry with their being multiple clips from different tests in the same folder.

In a similar fashion to debug mode, I also found it handy to include echoing the mkv params, and not deleting the temporary info and video and audio tracks extracted so i could play around with thing afterwards.

Suggestion for a feature. Filter variable for including audio tracks. ie en for English only, or en fr for english french

On last post, the error

/usr/bin/abdvde: line 1342: [: -gt: unary operator expected which is if [ $LENGTH -gt $TITLE_LONGEST ]; then

suggests that $LENGTH is not set; ie in bash terminal typing the following yields the same error message

$ if [ -gt 0 ]; then echo "test"; fi
bash: [: -gt: unary operator expected

ie the script is unable to get the required information to set those variables