MKV to M2TS on Linux

[ChuckleWorthy]

The point of this post is to discuss converting HD content in MKV files to M2TS files for streaming to a PS3, via Mediatomb, on an Ubuntu Linux server.

The PS3 can be quite fussy on what video and audio formats are acceptable when it comes to streamed HD content. The main problem I have remaining is when the Video stream resolution isn’t 1920x1080 or 1280x720 - my PS3 won’t show any video on playback.

I’ve read a lot on various forums and most solutions involve using Windows GUI applications (E.g. avisynth, MeGUI). As I’m using a Linux server to store my files for streaming - I would really like to create a script to do the job in one hit if possible and just leave my server to chug away. (And thus Windows apps aren’t an option for me.)

Before I go too far I would say I’m novice in the field, (Novice = Noob + '3weeks reading forums like mad' + 'tinkering at home'), so I’m drawing together items from various threads to help other people starting out. Therefore I am happy to accept constructive criticism here to improve the content of this thread as that will only serve to help others down the line.
Here’s what I found so far - PS3 Valid Video Criteria:

Profile level: 4.1
Resolution: 1920x1080 or 1280x720
Codec: H.264
I’m sure there are other things I’ve missed off...


PS3 Valid Audio Criteria:

Codec: AC3, MP3. (However not DTS)
again I’m sure there are other things I’ve missed off...


I must credit sticky123 (http://sticky123.blogspot.com/2008_03_01_archive.html) for a great blog post which is the foundation to this posting. I am hoping to take his/her work a step or two further.

http://www.ndu.edu/nwc/nwcCLIPART/CLIP_ART(General)/Separator_bars/bargold1.gif

There are three scenarios I would like to discuss and really need help with number 3:

Scenario 1) the MKV file contains video and audio streams which are both PS3 valid.
http://img223.imageshack.us/img223/4179/remuxop1ya6.th.png (http://img223.imageshack.us/my.php?image=remuxop1ya6.png)

As you can see the (fantastic) application tsMuxer does the repackaging job without the need to re-encode any data stream. It can also force the Video profile level to 4.1 to ensure the PS3 is happy.


Scenario 2) here the Audio stream isn’t PS3 compliant. i.e. stored in a DTS format. We can ‘break’ the DTS stream out of the MKV file into a separate file on it’s own and re-encode it to AC3 format, finally re-muxing (via tsMuxer) the new AC3 file and Video stream into a M2TS file.
http://img161.imageshack.us/img161/7512/remuxop2kr4.th.png (http://img161.imageshack.us/my.php?image=remuxop2kr4.png)


Scenario 3) this time the Video stream isn’t PS3 compliant. i.e. the resolution isn’t quite right. Now this is where my knowledge/experience lets me down and why I’m looking for help from the rest of you. As far as I’m aware it should be, (he says laughably), a simple case of add black borders to the video stream to bring it up-to the correct resolution. (Obviously ensure the aspect ratio is maintained.) I would want to keep the Video data at the same quality/codec as the original source.

http://img391.imageshack.us/img391/6204/remuxop3ij8.th.png (http://img391.imageshack.us/my.php?image=remuxop3ij8.png)

From what I’ve read so far there are some powerful (free) tools available such as mencoder, ffmpeg and x264. However all of them have more command line options than you can shake a stick at and finding the correct route to take has been rather daunting. Thus I’m asking for help.


Scenario 4) obviously it’s possible for Scenario 2 & 3 to apply to one MKV file, but I didn’t think I needed to go over that individually.

http://www.ndu.edu/nwc/nwcCLIPART/CLIP_ART(General)/Separator_bars/bargold1.gif


I’m hoping some of you will find this useful, and I’m hoping more that someone will be able to help me with Scenario 3. Once I’ve got a reasonable solution for that I will post my script for all to share.

Best Regards
Chuckle Worthy


Other things to discuss at a later date:
- subtitles. (I don’t want to discuss subtitles at the moment, maybe once we’ve got the video and audio sorted then we can take things a step further.)
- ff/rewind with HD resolutions doesn’t work very well.

[neuron2]

Application for step 2 in scenario 3 is DGAVCDec + Avisynth.

What are these MKV files and where are you getting them?

[ChuckleWorthy]

Isn't Avisynth a Windows GUI application?

A pal of mine gave me a DVD with a load of film trailers on. I've just got a HD TV and I'm trying to be flash and show off the TV and video streaming etc. But as per - I'm not had too much success.

[neuron2]

Isn't Avisynth a Windows GUI application? Oops, too early in the morning. Haven't had my coffee yet.

[jeffy]

@ChuckleWorthy: OT: Please, what software did you create the images in? Thank you.

[ChuckleWorthy]

Please, what software did you create the images in? Thank you.

If you're referring to the MKV files - I didn't create any, as I said, I got copy from a mate. Is there a way of interrogating the MKV files for that kind of information?

[jeffy]

If you're referring to the MKV files - I didn't create any, as I said, I got copy from a mate. Is there a way of interrogating the MKV files for that kind of information?

The images Scenario 1, 2, 3. Sorry if I was not clear.

[ChuckleWorthy]

The images Scenario 1, 2, 3. Sorry if I was not clear.

Ohhh. Heh. I'll turn off my dullard setting...

I used Microsoft Visio 2003 Standard Edition - I always find a couple of pictures help explain things far easier then paragraphs of waffle.

[Grimmi Meloni]

I have posted my script which does this in another thread (http://forum.doom9.org/showthread.php?p=1151013#post1151013). Feel free to use it.

EDIT: Sorry, I did not realize you already have most of the scenarios covered. Just ignore my post.

[ChuckleWorthy]

Okay I've been answering my own question as best as I can. This is my first adventure in this world, so after lots and lots of tinkering later - 'mencoder' seemed like the best way forward. :)

Like I said here's what I've come up with so far. It has produced some really good results IMO.

Again I'm happy for comments/suggestions/help etc.


#!/bin/bash
#**
#** m2tsEx.sh - a script for converting MKV files to M2TS for streaming to a PS3
#** from a DLNA Media server.
#** Uses:
#** - tsMuxer: http://www.videohelp.com/tools/tsMuxeR
#** (I use the Win32 version of tsMuxer as I found the Linux version unreliable)
#** - wine: http://www.winehq.org/
#** - mkvtoolnix: http://www.videohelp.com/tools/MKVtoolnix
#** - dcadec: http://www.videolan.org/developers/libdca.html
#** - aften: http://aften.sourceforge.net/
#** - mencoder: http://www.mplayerhq.hu/
#** - x264: http://www.videolan.org/developers/x264.html
#**
#** v0.1 initial version - 24th June 2008
#**
#** Usage: m2tsEx.sh filename.mkv
#**
#** Functions ********************************************************************************
RED=$( echo -e "\e[31;40m" )
GREEN=$( echo -e "\e[32;40m" )
YELLO=$( echo -e "\e[33;40m" )
BLUE=$( echo -e "\e[34;40m" )
MAGENTA=$(echo -e "\e[35;40m" )
CYAN=$( echo -e "\e[36;40m" )
RESET=$( echo -e "\e[0m" )

dmpstart() { echo "--------------------------------------------------------" > "./process.log"; }
dmp() { echo "`date`: $*" >> "./process.log"; }

#** colorful echo functions
msg() { echo ""; echo ">> $*"; dmp "$*"; }
rmsg() { echo ""; echo ">> ${RED}$*${RESET}"; dmp "$*"; }
gmsg() { echo ""; echo ">> ${GREEN}$*${RESET}"; dmp "$*"; }
ymsg() { echo ""; echo ">> ${YELLO}$*${RESET}"; dmp "$*"; }
bmsg() { echo ""; echo ">> ${BLUE}$*${RESET}"; dmp "$*"; }
mmsg() { echo ""; echo ">> ${MAGENTA}$*${RESET}"; dmp "$*"; }
cmsg() { echo ""; echo ">> ${CYAN}$*${RESET}"; dmp "$*"; }


#** Constants ********************************************************************************
BIN_DIR="/usr/bin"
LOCAL_BIN_DIR="/usr/local/bin"
WORKING_DIR="./wrking/"
SEVEN20="1280:720"
TEN80="1920:1080"
SEVEN20_WIDTH="1280"
TEN80_WIDTH="1920"
PRIMARY_LANGUAGE="eng" #** Used in the mux.meta file, change this if you want... (obviously)
SECONDARY_LANGUAGE="und"

#** Check the passed parameter is a MKV file ************************************************
dmpstart
cmsg "Starting..."
SRC_FILE="$1"
EXTENSION=`$BIN_DIR/basename $SRC_FILE | awk -F . '{print $NF}' | tr 'A-Z' 'a-z'`
BASENAME=$($BIN_DIR/basename "$SRC_FILE" .mkv)
DEST_FILE=$BASENAME.m2ts
if [ "$EXTENSION" != "mkv" ]
then
rmsg "Not a Matroska File, exiting..."
exit 1
fi

#** EXTRACT DETAILS OUT OF THE MKV FILE ******************************************************
#** Get the info from the MKV file
cmsg "Extracting MKV Information..."
MKV_INFO=`$BIN_DIR/mkvinfo "$SRC_FILE"`

#** Check there are Streams within the MKV we're interested in
cmsg "Determine Track Id's..."
VIDEO_TRACK_NO=`echo "$MKV_INFO" | grep "V_MPEG4/ISO/AVC" -B10 | grep "Track number:" | awk '{ print $5 }'`
AC3_TRACK_NO=`echo "$MKV_INFO" | grep "A_AC3" -B10 -C3 | grep -E "Language: $PRIMARY_LANGUAGE"\|"Language: $SECONDARY_LANGUAGE" -B13 | grep "Track number:" | awk '{ print $5 }' | head -1`
DTS_TRACK_NO=`echo "$MKV_INFO" | grep "A_DTS" -B10 -C3 | grep -E "Language: $PRIMARY_LANGUAGE"\|"Language: $SECONDARY_LANGUAGE" -B13 | grep "Track number:" | awk '{ print $5 }' | head -1`

if [[ -z $VIDEO_TRACK_NO ]]
then
rmsg "No Video Stream found, exiting..."
exit 1;
fi

if [[ -z $AC3_TRACK_NO ]] && [[ -z $DTS_TRACK_NO ]]
then
rmsg "No AC3 or DTS Streams found, exiting..."
exit 1;
fi

#** AUDIO TRACK INVESTIGATION ****************************************************************
#** Okay, lets find the lowest value Track Number, as that generally is the Main audio stream
#** I'll cheat by setting any null values to a high number, so the non-nulls will win the test
cmsg "Sort Out Audio Tracks..."
if [[ -z $AC3_TRACK_NO ]]
then
AC3_TRACK_NO=999
fi
if [[ -z $DTS_TRACK_NO ]]

then
DTS_TRACK_NO=999
fi

if [[ $AC3_TRACK_NO -lt $DTS_TRACK_NO ]]
then
AUDIO_TRACK_NO=$AC3_TRACK_NO
AUDIO_TRANSCODE=1 #FALSE
gmsg "AC3 Audio Stream Found - Track Id: $AC3_TRACK_NO."
else
AUDIO_TRACK_NO=$DTS_TRACK_NO
AUDIO_TRANSCODE=0 #TRUE
gmsg "DTS Audio Stream Found - Track Id: $DTS_TRACK_NO."
fi

#** VIDEO TRACK INVESTIGATION ****************************************************************
#** Get the Video Resolution etc
cmsg "Sort Out Video Resolution..."
SRC_WIDTH=`echo "$MKV_INFO" | grep "Pixel width:" | awk '{ print $5}'`
SRC_HEIGHT=`echo "$MKV_INFO" | grep "Pixel height:" | awk '{ print $5}'`
SRC_RESOLUTION="$SRC_WIDTH:$SRC_HEIGHT"
SRC_FPS=`echo "$MKV_INFO" | grep "V_MPEG4/ISO/AVC" -A3 | grep "fps" | awk '{ print $6 }' | cut -c2-100`

#** Calculate the Average Bit Rate
SRC_SECONDS=`echo "$MKV_INFO" | grep Duration | awk '{ print $4}' | sed 's/s//' | awk -F . '{ print $1 }'`
SRC_SIZE=`echo "$MKV_INFO" | grep Segment, | awk '{ print $4}'`
SRC_AVG_BITRATE="$[ ( ($SRC_SIZE * 8) / ($SRC_SECONDS ) / 1024 ) ]"
cmsg "Source Video Details - Resolution: $SRC_RESOLUTION, Duration: $SRC_SECONDS seconds, Average Bitrate: $SRC_AVG_BITRATE, Frames Per Second: $SRC_FPS."

#** Calculate the desired Resolution
if [[ -z $2 ]]
then
if [[ "$SRC_WIDTH" != "$SEVEN20_WIDTH" ]] && [[ "$SRC_WIDTH" != "$TEN80_WIDTH" ]]
then
rmsg "Odd Resolution: $SRC_RESOLUTION. Please specify manually: 1920:1080 or 1280:720, exiting..."
exit 1;
else
if [ "$SRC_WIDTH" == "$SEVEN20_WIDTH" ]
then
DEST_RESOLUTION=$SEVEN20;
fi

if [ "$SRC_WIDTH" == "$TEN80_WIDTH" ]
then
DEST_RESOLUTION=$TEN80;
fi
fi
else
#** Use the passed paramter for the new Resolution
DEST_RESOLUTION=$2
fi

#** See if we need to transcode the video
if [[ $SRC_RESOLUTION == $DEST_RESOLUTION ]]
then
VIDEO_TRANSCODE=1 #FALSE
mmsg "Video Stream - Track Id: $VIDEO_TRACK_NO. No Transcoding Required."

else
VIDEO_TRANSCODE=0 #TRUE
mmsg "Video Stream - Track Id: $VIDEO_TRACK_NO. Transcoding To $DEST_RESOLUTION Resolution."
fi

#** START THE CONVERSION ******************************************************
#** Create the working temporary directory
mkdir -p ${WORKING_DIR}

#** Create the tsMuxer config file
echo "MUXOPT --no-pcr-on-video-pid --new-audio-pes --vbr " > ${WORKING_DIR}mux.meta

if [[ $VIDEO_TRANSCODE -eq 1 ]]
then
#** No Video Transcoding - just reference the source MKV file
echo "V_MPEG4/ISO/AVC, \"$SRC_FILE\", level=4.1, insertSEI, contSPS, track=$VIDEO_TRACK_NO, fps=$SRC_FPS, lang=$PRIMARY_LANGUAGE" >> ${WORKING_DIR}mux.meta
else
#** Video Transcoding - break out the source stream
cmsg "Extracting H264 Stream..."
$BIN_DIR/nice -n20 \
$BIN_DIR/mkvextract tracks "$SRC_FILE" $VIDEO_TRACK_NO:"${WORKING_DIR}$BASENAME.h264"

#** Start Video Transcoding
X264PASS1=""
X264PASS1="${X264PASS1}pass=1:threads=1:turbo=1:bitrate=$SRC_AVG_BITRATE:frameref=1:bframes=3:"
X264PASS1="${X264PASS1}b_pyramid:direct_pred=auto:weight_b:me=dia:mixed_refs:bime:"
X264PASS1="${X264PASS1}trellis=1:nopsnr:nossim:subq=1:level_idc=41" #** This sets the Profile Level to 4.1

X264PASS2=""
X264PASS2="${X264PASS2}pass=2:threads=1:turbo=0:bitrate=$SRC_AVG_BITRATE:frameref=1:bframes=3:" #** I found a frameref of 3 was a no-no
X264PASS2="${X264PASS2}b_pyramid:direct_pred=auto:weight_b:me=umh:mixed_refs:bime:"
X264PASS2="${X264PASS2}partitions=all:8x8dct:"
X264PASS2="${X264PASS2}trellis=1:nopsnr:nossim:subq=6:level_idc=41"

cmsg "Video Transcoding - First Pass..."
$BIN_DIR/nice -n20 \
$BIN_DIR/mencoder \
"${WORKING_DIR}$BASENAME.h264" \
-vf expand=$DEST_RESOLUTION \
-ofps $SRC_FPS \
-ovc x264 \
-x264encopts $X264PASS1 \
-nosound \
-of rawvideo \
-o /dev/null;
# -o "${WORKING_DIR}${BASENAME}-pass1.h264" > ${WORKING_DIR}pass1.txt;

# Second pass
cmsg "Video Transcoding - Second Pass..."
$BIN_DIR/nice -n20 \
$BIN_DIR/mencoder \
"${WORKING_DIR}$BASENAME.h264" \
-vf expand=$DEST_RESOLUTION \
-ofps $SRC_FPS \
-ovc x264 \
-x264encopts $X264PASS2 \
-nosound \
-of rawvideo \
-o "${WORKING_DIR}${BASENAME}-pass2.h264";
# -o "${WORKING_DIR}${BASENAME}-pass2.h264" > ${WORKING_DIR}pass2.txt;

#** Get rid of the first Pass statistics wosname
rm -f "divx2pass.log"
echo "V_MPEG4/ISO/AVC, \"${WORKING_DIR}${BASENAME}-pass2.h264\", level=4.1, insertSEI, contSPS, track=1, fps=$SRC_FPS, lang=$PRIMARY_LANGUAGE" >> ${WORKING_DIR}mux.meta
fi

if [[ $AUDIO_TRANSCODE -eq 1 ]]
then
#** No Audio Transcoding - just reference the source MKV file
echo "A_AC3, \"$SRC_FILE\", track=$AUDIO_TRACK_NO, lang=$PRIMARY_LANGUAGE" >> ${WORKING_DIR}mux.meta
else
#** Audio Transcoding - break out the source stream
cmsg "Extracting DTS Stream..."
$BIN_DIR/nice -n20 \
$BIN_DIR/mkvextract tracks "$SRC_FILE" $AUDIO_TRACK_NO:"${WORKING_DIR}$BASENAME.dts"

cmsg "Transcoding DTS To WAV..."
$LOCAL_BIN_DIR/dcadec -r -o wavall "${WORKING_DIR}$BASENAME.dts" > "${WORKING_DIR}$BASENAME.wav"

cmsg "Transcoding WAV To AC3..."
$BIN_DIR/aften "${WORKING_DIR}$BASENAME.wav" "${WORKING_DIR}$BASENAME.ac3"

echo "A_AC3, \"${WORKING_DIR}$BASENAME.ac3\", track=1, lang=$PRIMARY_LANGUAGE" >> ${WORKING_DIR}mux.meta
fi

#** Get tsMuxer on the case...
cmsg "tsMuxing To M2TS File..."
$BIN_DIR/wine $LOCAL_BIN_DIR/tsMuxeR.exe ${WORKING_DIR}mux.meta $DEST_FILE

cmsg "Tidying Up..."
#rm -r -f $WORKING_DIR

cmsg "Done."

#***********************************************************************************
#** Big Thanks To The Following:
#**
#** http://tinyurl.com/3e9x7n - This is the foundation to my script above - Huge thanks
#** http://tinyurl.com/4prnev - I got the x264encopts from here
#** http://tinyurl.com/6k6lsb - How to compile x264 and ffmpeg via Ubuntu
#** http://tinyurl.com/6k5opc
#** Plenty of other sites too...
#**
#***********************************************************************************

[RyLoS]

Very good work!
I can help you improve your script if you want.
For example, a lot of MKV in HD are not standard 1280x720 or 1920x1080....you can find many situations, like 1280x5xx etc etc.
From what i've seen your script strictly check and strictly wants to use those 2 standard definitions but PS3 can play also different ones without problems.

[ChuckleWorthy]

We'll I've been tinkering and reading even more Doom9 forum posts. Therefore I've tweaked my original script quite alot to make life easier - I hope...

I think I may have cracked the whole FF/Rewind problem thanks to other threads.

Anyhow - as before - feel free to comment.
Cheers
Chuckle