[Beta-testers needed] titleBackup

[M7S]

Quick link to "the latest and greatest" titleBackup version available:
titleBackup 0.0.1 (http://www.kolumbus.fi/msevens/titlebackup/titlebackup_0.0.1.tar.gz)


Hi. I've been busy lately with this little project I call titleBackup. It is a program for backing up a title (not the whole dvd with menues) from a dvd. It reencodes the video to make it fit on a single layer DVD+/-r(w) disc.

Features:


Backup directly from a dvd or from a dvd-backup on the harddrive (such as you get if you run vobcopy -m)
Three output modes:
- output to folder
- output to iso file
- or let titleBackup burn the new dvd for you.
titleBackup uses mencoder for encoding and let's you tweak the encoder settings (if you want to).
You can backup one title with as many of the audio and subtitle tracks as you like.
Chapters are preserved
Possibility to save, load, export and import mencoder settings profiles



System requirements
In the preference dialog you can choose where to put the temporary files (by default is this place '/tmp/'). The temporary file can get as almost 14 GB big (the size of the title you backup, max 8.5 GB + the size of the reencoded video, sound files and subtitle files, max 4.7 GB). Make sure there is at least 14 GB free space at the place where you choose to keep titleBackup's temporary directory.


Version 0.0.1

Today I'm releasing the first public beta version (0.0.1). I would really need some of you intelligent people to take this program for a test spin. I've programmed on this for exactly one month today (3.2.2007) after spending the Christmas reading a great python tutorial (dive into python). This is the first thing I've programmed in over six years, when I made a simple java applet game, so you may say that I'm a complete beginner in programming.

What I'm trying to say is: don't expect to much of this version. It can produce well working, nice looking dvd backups, but don't take it for granted. The program has incomplete errorhandling. If any of the programs that runs in the background gives an error, titleBackup will ignore it. When titleBackup syas that the work is done you better make sure that that actually is the case (and don't use burn dvd as output method if you can't afford wasting a dvd).

Known Problems

Mencoder's ratecontroll seems to give buffer underflows sometimes. I don't know if this is something mencoder does no matter what or if it's a result of titleBackup feeding it with the wrong parameters. Mplex will then try to remux the video but will fail when it comes to the buffer underflows. titleBackup don't care about this and will go on as nothing happened and you will only get a backup of the part of the title that was before the buffer underflow occurred.

Install

titleBackup depends on following programs: lsdvd, mplayer, mencoder, dvdauthor, mjpegtools, mkisofs and dvd+rw-tools.

You will need ifo_dump as well because titleBackup uses it to work around a bug in lsdvd. Ifo_dump comes with the dvd-player ogle.

titlebackup needs python 2.4 or newer, and following external python librarys: gtk, gtk.glade and xdg.

To get these programs and libraries in ubuntu, make sure you have universe and multiverse repositories enabled and run this from commandline:

sudo apt-get install lsdvd mplayer mencoder dvdauthor mjpegtools ogle mkisofs dvd+rw-tools python python-xdg python-gtk2 python-glade2

Users of ther debian flavors should hopefully be able to use the same apt-get line (except for the sudo part ofcource). Users of other distros may tell me how to install these programs on their distro with minimum effort.

When you have these dependencies all you need to do is to download the program (http://www.kolumbus.fi/msevens/titlebackup/titlebackup_0.0.1.tar.gz), unpack titleBackup somewhere and run

python titlebackup.py

Homepage

The projects current homepage is: http://www.kolumbus.fi/msevens/titlebackup/ (Maybe I should contact sourceforge...)


Things I would like you to pay extra attension to

People who are used to using mencoder for making dvd:s:

Is the options in the mencoder tab the right ones, is these the things you would like to change?
Which options could be removed and hardcoded into the program instead?
Which options would you like me add to the page?
If you are happy with the selection of mencoder options, could you make some presets for titleBackup (for now only the Default preset exists). Just tweak the options to your liking and press the 'save to file' button, save the file and attach it in a post here on this forum.
Do you know any sollutions to the buffer underflow mencoder problem I mentioned above?

For people who don't know much about mencoder:

Does the tool tip you get when you hover over the mencoder options make any sense to you?


What does titleBackup do, really?

When you load a dvd into titleBackup lsdvd will read it and give alot of important information to titleBackup. The only thing I miss is information about titles' sizes or bitrates for video and audio. Do anyone know about any program that would give this information?

This is then the backup process:

Dump the chosen title with mplayer -dumpstream
Make a pallet.yuv file needed (for getting corectly colored subtitles) from the dumped title with a ifo_dump
Get the selected subtitles (if any) from the dumped title with spuunmux
Get the selected audio tracks from the dumped title with mplayer -dumpaudio
Calculate bitrate for video using the size of the dumped audio tracks among other things
Encode the new video using mencoder
Mux video and all audio tracks using mplex - f 8 -V
Mux the subtitles to the title using spumux
Author the title using dvdauthor -t. Information about audio and subtitle languages, chapters and so on is taken from lsdvd.
Author the rest of the dvd (VIDEO_TS.IFO and stuff) with dvdauthor -T
if output mode is iso file or burn, make an iso file of the directory with mkisofs
if the mode is burn, burn the iso file with growisofs

For more information about the command lines for each step, try the program. It will tell you what you want to know.


Regards,
M7S

[danpos]

@M7S

Thanks for offer you toy to us, mate! :)

I will give it a try ASAP.

Regards,

[danpos]

You will need ifo_dump as well because titleBackup uses it to work around a bug in lsdvd. I'm not sure of which program ifo_dump comes with, but I think it's ogle. Can anyone confirm this??

I do confirm this: /usr/bin/ifo_dump is provided by ogle package (on Debian GNU/Linux based distributions).

Regards,

[danpos]

@M7S

Hi! First try with your application and I got this error:

danpos@wolf:~/Meus Documentos/Arquivos$ cd titlebackup/
danpos@wolf:~/Meus Documentos/Arquivos/titlebackup$ python titlebackup.py
Traceback (most recent call last):
File "titlebackup.py", line 496, in ?
MainWindow=titlebackup()
File "titlebackup.py", line 49, in __init__
self.settings = tbpref.tb_settings(CONFIG_XML)
File "/home/danpos/Meus Documentos/Arquivos/titlebackup/tbpref.py", line 54, in __init__
self.load_from_file(file_path)
File "/home/danpos/Meus Documentos/Arquivos/titlebackup/tbpref.py", line 57, in load_from_file
dom1 = parse(file_path)
File "xml/dom/minidom.py", line 1915, in parse
File "xml/dom/expatbuilder.py", line 922, in parse
IOError: [Errno 2] Arquivo ou diretório não encontrado: '/home/danpos/.config/titlebackup/preferences.xml'

I guess that the a preferences.xml example is missing in tarball, in order that it can to be copyied to expected directory ...

Regards,

[M7S]

@M7S

Hi! First try with your application and I got this error:



I guess that the a preferences.xml example is missing in tarball, in order that it can to be copyied to expected directory ...

Regards,
Thank you for beta testing, Danpos.
That was a silly bug (the first of many, I'm afraid). The config file should be loaded only if it exist, which it can't do the first time you run the program. I did a silent update, so you can download titleBackup again and see if things are working better now.

And thanks for confirming that ifo_dump comes with ogle.

[M7S]

Added mkisofs and dvd+rw-tools to the list of dependencies, since they are needed for the make iso and burn dvd output modes.

[M7S]

No one interested in beta testing? Just try the program and tell me what you think and if you encountered any problems.

Pretty please?

[wildchild77]

I will give this a try on my Ubuntu Edgy 6.10 system as soon as I get some free time.

[nickrout]

midentify is part of mplayer, and will give info on bitrates and title sizes, viz:

mythtv@media:~$ midentify dvd://1 -dvd-device /myth/video/elrond/PinkFloyd_BehindTheWall/
ID_AUDIO_ID=128
ID_SUBTITLE_ID=0
ID_VIDEO_ID=0
ID_AUDIO_ID=128
ID_FILENAME=dvd://1
ID_VIDEO_FORMAT=0x10000002
ID_VIDEO_BITRATE=9318400
ID_VIDEO_WIDTH=720
ID_VIDEO_HEIGHT=576
ID_VIDEO_FPS=25.000
ID_VIDEO_ASPECT=1.3333
ID_AUDIO_CODEC=a52
ID_AUDIO_FORMAT=8192
ID_AUDIO_BITRATE=384000
ID_AUDIO_RATE=48000
ID_AUDIO_NCH=2
ID_LENGTH=3208

I just tried your program on freebsd. I probably don't have all the needed bits intalled, so will try and look at it again in the weekend. I did note that it failed without python's library for xdg installed (package name py-xdg on freebsd)

if something essential is missing, will it just fail silently? Thats what is happening here.

I will also take a look at it on gentoo when I bother to get my desktop switched on.

Nick.

[M7S]

Thank you guys for taking time to beta test. It's much appreciated.

midentify is part of mplayer, and will give info on bitrates and title sizes, viz:
midentify doesn't seem to be a part of mplayer on Ubuntu Edgy :( and it's not in the repositories either. Wonder why. Thanks for the suggestion, though.


I just tried your program on freebsd. I probably don't have all the needed bits intalled, so will try and look at it again in the weekend. I did note that it failed without python's library for xdg installed (package name py-xdg on freebsd)

if something essential is missing, will it just fail silently? Thats what is happening here.
Yes. The program do silently quit if gtk or gtk.glade can't be imported. I haven't questioned that behavior since more developed program than mine do exactly the same thing and some pygtk tutorials recommend it as well. Anyway, I added a short error message that will be printed if gtk or gtk.glade can't be loaded.

I also added (hopefully) all needed python libraries to the install instructions in my first post.

[nickrout]

midentify found its way onto my system somehow, its just a wrapper around mplayer -identify. Here is the whole thing:

#!/bin/sh
#
# This is a wrapper around the -identify functionality.
# It is supposed to escape the output properly, so it can be easily
# used in shellscripts by 'eval'ing the output of this script.
#
# Written by Tobias Diedrich <ranma+mplayer@tdiedrich.de>
# Licensed under GNU GPL.

if [ -z "$1" ]; then
echo "Usage: midentify <file> [<file> ...]"
exit 1
fi

mplayer -vo null -ao null -frames 0 -identify "$@" 2>/dev/null |
sed -ne '/^ID_/ {
s/[]()|&;<>`'"'"'\\!$" []/\\&/g;p
}'


Installing py-gtk and glade now :-)

[M7S]

midentify found its way onto my system somehow, its just a wrapper around mplayer -identify. Here is the whole thing:

#!/bin/sh
#
# This is a wrapper around the -identify functionality.
# It is supposed to escape the output properly, so it can be easily
# used in shellscripts by 'eval'ing the output of this script.
#
# Written by Tobias Diedrich <ranma+mplayer@tdiedrich.de>
# Licensed under GNU GPL.

if [ -z "$1" ]; then
echo "Usage: midentify <file> [<file> ...]"
exit 1
fi

mplayer -vo null -ao null -frames 0 -identify "$@" 2>/dev/null |
sed -ne '/^ID_/ {
s/[]()|&;<>`'"'"'\\!$" []/\\&/g;p
}'


Installing py-gtk and glade now :-)
Thanks! mplayer -identify does exactly what I want (well the video bitrate seams to be max video bitrate instead of average video bitrate, but I can live with just audio bitrate). Next version of titleBackup will now be able to estimate the bitrate before starting the backup progress. I've already made the code for geting the audio bitrate and showing it so I'm half way there.

[wildchild77]

I got the program to load and start working pretty easily on Ubuntu 6.10. There were a few warnings and the program crashed between step #4 & #5. I am at work right now and will post some error logs when I do some more testing.

This error shows up after running the titlebackup.py command and the program will still open.

titlebackup.py:52: GtkWarning: Ignoring the separator setting
self.wTree = gtk.glade.XML(GLADE_FILE)

(titlebackup.py:14850): libglade-WARNING **: could not find a parent that handles internal children for `vbox'

(titlebackup.py:14850): libglade-WARNING **: could not find a parent that handles internal children for `vbox'

(titlebackup.py:14850): libglade-WARNING **: could not find a parent that handles internal children for `vbox'

[M7S]

I got the program to load and start working pretty easily on Ubuntu 6.10. There were a few warnings and the program crashed between step #4 & #5. I am at work right now and will post some error logs when I do some more testing.

This error shows up after running the titlebackup.py command and the program will still open.

titlebackup.py:52: GtkWarning: Ignoring the separator setting
self.wTree = gtk.glade.XML(GLADE_FILE)

(titlebackup.py:14850): libglade-WARNING **: could not find a parent that handles internal children for `vbox'

(titlebackup.py:14850): libglade-WARNING **: could not find a parent that handles internal children for `vbox'

(titlebackup.py:14850): libglade-WARNING **: could not find a parent that handles internal children for `vbox'

I get those warnings as well. There must be something wrong in my glade file, but it doesn't affect the functionality of the program in any way.

A crash between step #4 and #5? I would love to see a log for that one. I'm way too excited over the fact that some serious beta testing finally is happening. Thank you.

[wildchild77]

******** Edit folder name
; ) turned to images in log = mplayer: Core dumped ;)

Here is the complete log there are a lot of redundant errors in Mplayer, so I may have a problem there.

(titlebackup.py:5183): libglade-WARNING **: could not find a parent that handles internal children for `vbox'
opening preference dialog
opening preference dialog
mode: output to folder

###mplayer: Core dumped ;)
# Running mplayer
Commandline: mplayer -dumpstream -dumpfile /home/********/TBU Work/titlebackupdtDMYf/title.vob -dvd-device /dev/dvd dvd://1

mplayer: MPlayer 2:0.99+1.0pre8-0ubuntu8 (C) 2000-2006 MPlayer Team
mplayer: CPU: Intel(R) Pentium(R) 4 CPU 2.80GHz (Family: 15, Model: 3, Stepping: 3)
mplayer: CPUflags: MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
mplayer: Compiled with runtime CPU detection.
mplayer:
mplayer:
mplayer: Linux RTC init error in ioctl (rtc_irqp_set 1024): Permission denied
mplayer: Try adding "echo 1024 > /proc/sys/dev/rtc/max-user-freq" to your system startup scripts.
mplayer: Opening joystick device /dev/input/js0mplayer: Core dumped ;)
mplayer: Can't open joystick device /dev/input/js0: No such file or directory
mplayer: Can't init input joystick
mplayer: Setting up LIRC support...
mplayer: mplayer: could not connect to socket
mplayer: mplayer: No such file or directory
mplayer: Failed to open LIRC support.
mplayer: You will not be able to use your remote control.
mplayer:
mplayer: Playing dvd://1.
mplayer: Reading disc structure, please wait...
mplayer: There are 17 titles on this DVD.
mplayer: There are 28 chapters in this DVD title.
mplayer: There are 1 angles in this DVD title.
mplayer: DVD successfully opened.
mplayer: Core dumped ;)
mplayer:
mplayer: Exiting... (End of file)

###
# Running spuunmux
Commandline: spuunmux -s 0 -p /home/********/TBU Work/titlebackupdtDMYf/palette.yuv -o /home/********/TBU Work/titlebackupdtDMYf/subtitles/sub1_ /home/********/TBU Work/titlebackupdtDMYf/title.vob

spuunmux: DVDAuthor::spuunmux, version 0.6.11.
spuunmux: Build options: gnugetopt magick iconv freetype
spuunmux: Send bugs to <dvdauthor-users@lists.sourceforge.net>
spuunmux:

###
# Running mplayer
Commandline: mplayer -dumpaudio -dumpfile /home/********/TBU Work/titlebackupdtDMYf/audio/audio1.ac3 -aid 128 /home/********/TBU Work/titlebackupdtDMYf/title.vob

mplayer: MPlayer 2:0.99+1.0pre8-0ubuntu8 (C) 2000-2006 MPlayer Team
mplayer: CPU: Intel(R) Pentium(R) 4 CPU 2.80GHz (Family: 15, Model: 3, Stepping: 3)
mplayer: CPUflags: MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
mplayer: Compiled with runtime CPU detection.
mplayer:
mplayer:
mplayer: Linux RTC init error in ioctl (rtc_irqp_set 1024): Permission denied
mplayer: Try adding "echo 1024 > /proc/sys/dev/rtc/max-user-freq" to your system startup scripts.
mplayer: Opening joystick device /dev/input/js0
mplayer: Can't open joystick device /dev/input/js0: No such file or directory
mplayer: Can't init input joystick
mplayer: Setting up LIRC support...
mplayer: mplayer: could not connect to socket
mplayer: mplayer: No such file or directory
mplayer: Failed to open LIRC support.
mplayer: You will not be able to use your remote control.
mplayer:
mplayer: Playing /home/********/TBU Work/titlebackupdtDMYf/title.vob.
mplayer: MPEG-PS file format detected.
mplayer: Core dumped ;)
mplayer:
mplayer: Exiting... (End of file)

###
# Running mplayer
Commandline: mplayer -dumpaudio -dumpfile /home/********/TBU Work/titlebackupdtDMYf/audio/audio3.ac3 -aid 130 /home/********/TBU Work/titlebackupdtDMYf/title.vob

mplayer: MPlayer 2:0.99+1.0pre8-0ubuntu8 (C) 2000-2006 MPlayer Team
mplayer: CPU: Intel(R) Pentium(R) 4 CPU 2.80GHz (Family: 15, Model: 3, Stepping: 3)
mplayer: CPUflags: MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
mplayer: Compiled with runtime CPU detection.
mplayer:
mplayer:
mplayer: Linux RTC init error in ioctl (rtc_irqp_set 1024): Permission denied
mplayer: Try adding "echo 1024 > /proc/sys/dev/rtc/max-user-freq" to your system startup scripts.
mplayer: Opening joystick device /dev/input/js0
mplayer: Can't open joystick device /dev/input/js0: No such file or directory
mplayer: Can't init input joystick
mplayer: Setting up LIRC support...
mplayer: mplayer: could not connect to socket
mplayer: mplayer: No such file or directory
mplayer: Failed to open LIRC support.
mplayer: You will not be able to use your remote control.
mplayer:
mplayer: Playing /home/********/TBU Work/titlebackupdtDMYf/title.vob.
mplayer: MPEG-PS file format detected.
mplayer: Core dumped ;)
mplayer:
mplayer: Exiting... (End of file)

###
# Running mplayer
Commandline: mplayer -dumpaudio -dumpfile /home/********/TBU Work/titlebackupdtDMYf/audio/audio4.ac3 -aid 131 /home/********/TBU Work/titlebackupdtDMYf/title.vob

mplayer: MPlayer 2:0.99+1.0pre8-0ubuntu8 (C) 2000-2006 MPlayer Team
mplayer: CPU: Intel(R) Pentium(R) 4 CPU 2.80GHz (Family: 15, Model: 3, Stepping: 3)
mplayer: CPUflags: MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
mplayer: Compiled with runtime CPU detection.
mplayer:
mplayer:
mplayer: Linux RTC init error in ioctl (rtc_irqp_set 1024): Permission denied
mplayer: Try adding "echo 1024 > /proc/sys/dev/rtc/max-user-freq" to your system startup scripts.
mplayer: Opening joystick device /dev/input/js0
mplayer: Can't open joystick device /dev/input/js0: No such file or directory
mplayer: Can't init input joystick
mplayer: Setting up LIRC support...
mplayer: mplayer: could not connect to socket
mplayer: mplayer: No such file or directory
mplayer: Failed to open LIRC support.
mplayer: You will not be able to use your remote control.
mplayer:
mplayer: Playing /home/********/TBU Work/titlebackupdtDMYf/title.vob.
mplayer: MPEG-PS file format detected.
mplayer: Core dumped ;)
mplayer:
mplayer: Exiting... (End of file)
Error: title['format']=NTSC
Exception in thread Thread-1:
Traceback (most recent call last):
File "threading.py", line 442, in __bootstrap
self.run()
File "/home/********/titlework/backup_title.py", line 82, in run
bitrate = self.calculate_bitrate(title, subtitles, audios, tdir)
File "/home/********/titlework/backup_title.py", line 230, in calculate_bitrate
gops = lenght * fps / gop_size
UnboundLocalError: local variable 'fps' referenced before assignment

[M7S]

Look at that. Stupid bug #2 :D. The american video format is NTSC, not NTCS :rolleyes: . I will update that tonight when I'm at home, I can't reach my homepage from this computer. The typo is in line 225 in backup_title.py if you want to do more beta testing right away.

[M7S]

Silent update for stupid bug #2 done.

[Amnon82]

Since I did a package of AVISynth 3.0 for Paldo GNU/Linux I'll try to add your program also to my Distro. I'll let you know about my progress. Maybe I'll add it to my program DVD2HDD.

[M7S]

Wow, that would make your distro more than a little bit bleeding edge. Not one beta tester has reported a succesful backup with titleBackup yet (though I've done some myself) :p

titleBackup can be used without copying the dvd to the harddrive with programs like DVD2HDD or vobcopy. If you see a reason to add titleBackup to DVD2HDD I'm all for it, but I myself fail to see such a reason.

[wildchild77]

Silent update for stupid bug #2 done.

That seems to have worked for me, I am 50% through the first pass in Mencoder. I will post the results later on as I have to work all weekend.

[wildchild77]

@M7S what kind of times are you getting on a re-encode, mencoder has been running for 8 hrs. and it is still not complete. The GUI for the program is blank and the only thing showing in terminal is (mencoder duplicating frames). I use this same computer using DVD Rebuilder and Cinema Craft Encoder under Wine and the times are usually around 90 minutes. I will post the log and a screen shot later.

[M7S]

That doesn't sound too promising. The GUI shouldn't go blank. Just to be sure, did the progress dialog open when you started the backup?

I usually have about 15 fps in second pass and around 50 fps in first pass on a sempron 3000+.

[shevegen]

Is the latest still at
http://www.kolumbus.fi/msevens/titlebackup/titlebackup_0.0.1.tar.gz
?

Btw, since its BETA, can you move download more to the top of the page (on your webpage), and also a direct link to the latest and greatest of your application in your first post here? (Just to clarify, I use no sarcasm.)

Its easier for me to scroll to the top to get what I want. I dont mind the slightest
to help you bugtest but I want to ensure its the latest version :-)

[M7S]

Good point there. Homepage and first post now has a link to the "latest and greatest" titleBackup on top of the page/post.

Thank you for offering your bugtesting services. It's greatly appreciated.

[wildchild77]

Here are some screen shots of the issues I have encountered so far

Mencoder pass 1 no problem
http://www.geocities.com/tyedye776967/tbu3.png

Mencoder pass 2 still showing 99% pass 1, also GUI goes blank when trying to run take screen shot.
http://www.geocities.com/tyedye776967/tbu7.png

System Monitor with blanked out GUI. I do not think this is a memory problem and the program is still running.
http://www.geocities.com/tyedye776967/tbu6.png

I hope these help in some way and I will keep testing. One thing I have noticed is the GUI will blank out if I try running other things on my computer while Mencoder is running.

[M7S]

This bug was a thougher than the previous ones. Which process is taking up all the cpu power? Is it titleBackup or mencoder or something else?


One thing I have noticed is the GUI will blank out if I try running other things on my computer while Mencoder is running.
Could you explain this a bit closer as well?

[Amnon82]

Seems we need some time here. Will see when I add your program to paldo.

[wildchild77]

Which process is taking up all the cpu power?

The 2 programs running are Mencoder and Python

[M7S]

Ok. The fact that mencoder still runs narrows it down a bit. I still would need to know which one of the processes that is taking up all the cpu power. Or do they use about 50% each?

[wildchild77]

They start out pretty even but the longer the program runs Mencoder fades and Python stays around 50%

[M7S]

Wow. That's odd. If this isn't something that has to do with dual core that I don't understand, this sounds like a serious problem. It makes absolutely no sense to me that titleBackup (python) should take up half the processing power. It takes up at most 5% for me and mencoder takes up the rest.

How many times have you tried to do backups? Does this happen every time?

I don't know anything about dual core. Is it possible to 'shut down' one core to test if the same problem happens with single core?


@ danpos, nickrout and shevegen
Has anyone of you done anymore beta-testing? I would really need to know if anyone else gets the same problems as wildchild.