Schmendrick
19th September 2006, 17:50
How to convert an AVC/H.264-ts recording from e.g. satellite or terrestrial transmission into a directshow-avisynth usable input format e.g. for processing with VirtualDubMod
Software required (on http://www.doom9.org/ or at link given) :
1. Graphedit with installed Dump-filter (included within graphedit-package)
2. AVC/H.264-decoder, either:
a. Cyberlinks AVC/H.264-package included within PowerDVD Version 7 or PowerDVD Version 6 with AVC/H.264-package ( http://www.cyberlink.com/ )
b. Elecards AVC/H.264-decoder included within MPEG2-player package or EmuxerPro AVC/H.264-plugin-package ( http://www.elecard.com/ )
c. Coreavc H.264-decoder together with Haali-demuxer package (http://www.coreavc.com/ )
3. Yamb 1.6 or 2.0 with MP4Box (included within package)
4. VirtualDubMod
5. ProjectX
6. DVBStreamexplorer ( http://www.dvbstreamexplorer.dk/ )
7. ffdshow a recent build with enabled H.264-decoding
1. Demux ts-file with ProjectX and read from the demux protocol the listed "PES-IDs".
You will e.g. find the following:
"ok> PID 0x90E has PES-ID 0xE0 (MPEG Video) (104310 #556)
ok> PID 0x910 has PES-ID 0xC8 (MPEG Audio) (119568 #637)"
or instead of the last line:
"ok> PID 0x103 has PES-ID 0xBD (private stream 1) (39480 #211)"
This line is shown if an AC3-Dolby-Digital-Audio-stream is present. Usually either a MPEG-audio stream or an AC3-stream is present besides the MPEG-video-stream.
Near the end of the demux protocol e.g. the following is listed:
"Audio PTS: first packet 08:15:11.361, last packet 08:16:53.345"
As ProjectX is not able to produce an AVC/H.264-stream the Video PTS has to be found in a different way. To accomplish this you have to open the recorded ts-file with a hex editor like the hex editor available within VirtualDubMod under "Tools". Using the "Find"-function from the "Edit"-menu the "Find binary string" opens, then enter the following characters into the "Find"-field: "00 00 00 01 67" and click the tic-box "Hexadecimal search string" and finally click on "Find next" to find this string. For example at the bottom of the following picture the found string is shown. Now find the next "00 00 01 E0" string just ahead of this string. In this picture just a few byte above starting at memory location "E0DCD" in the right center. Then finally find the video PTS which is contained within 5 bytes starting at the 9th byte after the beginning of the last searched string. Here marked within a black box. Here it reads: "35 7D 95 01 F1". (Fig1.jpg)
6183
The video PTS is a 33 bit value where the most significant bit is a sign bit.
This 33bit value is extracted from these 5 bytes. On the first byte a bitwise AND operation has to be performed with "0E". The result has to be bitwise shifted right one bit. The resulting 3bit-value is the 33th to 31st bit (counted from left to right in the binary representation of this 3bit-value), where the 33th bit the sign bit means: "0" is positive, "1" is negative. The remaining value has to be multiplied by 2 to the power of 30. The second byte is multiplied by 2 to the power of 22. The third byte value is first converted with a bitwise AND operation with "FE". The result bitwise shifted right for one bit and the result muliplied by 2 to the power of 15. The forth byte is multiplied by 128. The fifth byte value is first converted with a bitwise AND operation with "FE". The result bitwise shifted right for one bit. All five results are summed together and fitted with the obtained sign from the 33 bit value leading to the video PTS.
The video PTS is the system 27 MHz clock divided by 300 so a 90 kHz clock value. To convert the found value into milliseconds the value has to be divided by 90. A division of the obtained millisecond results by 3600000 converts to hours, by 60000 to minutes and by 1000 to seconds. The hour, minute, second and millisecond values are obtained by deleting the corresponding value after the decimal points. In this case "35 7D 95 01 F1" results into: "8:15:13.2969777" ( 8 hours, 15 minutes, 13 seconds, 296.9777 milliseconds. This means that the audio stream is delayed by -1.9359777 seconds (calculated by subtracting the video PTS-value from the audio PTS-value).
To properly demux the ts-file into a video-AVC/H.264 stream file and an audio-AC3 stream file or an audio-MPEG stream file either graphedit or DVBstreamexplorer is used. Start graphedit use Graph - Insert Filters - Direct Show Filters - File Source (Async.). Select the ts-file which is to be demuxed. Then Insert - Direct Show Filters - Elecard MPEG Demultiplexer (empgdmx.ax (version: 1.0.75.60727), 1012 KB). Demuxing with either Cyberlinks Demuxer contained within PowerDVD 7 or 6 (with AVC/H.264-decoder pack) or with the Haali Media Splitter instead cannot be used.
Connect the "File Source (Async.)"-box with the "Elecard MPEG Demultiplexer"-Box". After that select the "Dump"-filter. After selection enter a new file name with the file extension "asf", as this is the only extension the "Dump"-filter accepts. Nevertheless the "Dump"-filter produces a plain dump of the output of the demultiplexer. Connect the "H264"-output pin of the "Elecard MPEG Demultiplexer"-box with the input pin of the "Dump"-box. Finally click on the green triangular play button. Now the red "Stop" button apears. After demultiplexing finishes the green button appears again and the red button is greyed out again. (Fig2.jpg)
6184
Instead of using graphedit with Elecards MPEG Demultiplexer also DVBStreamexplorer can be used to extract the H.264-video stream:
At first at "Device" - "Driver selection" "TS file driver" has to be selected. Then at "Device" - "Tuning" the ts-file to be demultiplexed has to be selected. Having done this at "Action" - "SI scanning" - "SI tables" on the SI tables-box click on "Defaults" and "OK". Then click on "Action" - "SI scanning" - "Current transponder". This opens the "SI Reading"-box. On this box click on "Start" to scan the ts-file for its content. After some time this box can be closed again after the "Close"-button is not greyed out anymore. In the lower window at least "Got 1 PMT section(s) for SID XXXX" is displayed (with e.g. 27D8 at XXXX). Double-click on "Transponder" within the upper window to display the content section found due to the scanning operation. Now click on the small "+"-symbol left of "PMT" and left of the following subfolders. This opens the content-service tree (Fig3.jpg):
6185
Below the stream info line as shown in this figure click on the line "ITU-T Rec. H.262..."-line. Within the top right window the stream type is shown and two lines lower the "ELEMENTARY PID" here 0xFF (255). In this case the H.264-video is encased within a H.262-MPEG2-system package. DVBstreamexplorer does not analyze its content. (Fig4.jpg)
6186
Then under stream info line as shown in the figure above click on the line:
"ITU.T Rec. H.222.....". On the right side again the stream type and the "ELEMENTARY PID" here 0x103 (259) is shown.
Now knowing the elementary PIDs of the video and audio streams demux these streams by clicking on "Plugins" - "Streaming to disk" - "Save A/V ES streams" opens the "A/V ES Streaming"-box. Here at VPID the video PID found (here 255) and a file name for this stream has to be entered. The same has to be done at APID for the audio PID found (here 259) and a file name for the audio stream. Both file by default will have "bin" as file extension. Then start demuxing by clicking on "Start". You will have to watch if the file size of the stream files keeps increasing or stops to increase as an indication that the demuxing is finished as the demuxing function does not automatically switch off at the end of the file as is was initially programmed for indefinitely streaming satellite transmissions. So by clicking on "Stop" and "OK" this box will be closed.
This example ts-file has been recorded using the KNC TV Station DVB-S2+-card and the "Globe TV Digital"-programme.
Other recording devices and the associated programmes appear to be able to detect the AVC/H.264-video stream properly like in a short clip which was transmitted by "BBC HD1" (Fig5.jpg):
6187
Now open the produced files with the Hex editor of VirtualDubMod and again search for the hex string "00 00 00 01 67" as explained above. After the string is found, open Edit - Extract segment. Within the "Extract file segment" box the "Address (hex):" field is already filled with the start address. The value for the "Length (hex):" is found by search for the address for the last byte of the file within the Hex editor. From this hex value the found start address has to be subtracted (e.g. using the calculator within the accessories of Windows XP after switching to hex mode within the scientific view mode) and the result has to be increased by one. After this value has been entered click on "OK" and enter a file name for the h264-file to be produced e.g. "Pro7HD.h264". Now an "AVC/H.264"-file is produced with the correct start code (a key frame) and can be further processed as explained following.
Guide continues on following posting as maximum number of attached attachements is reached.
Software required (on http://www.doom9.org/ or at link given) :
1. Graphedit with installed Dump-filter (included within graphedit-package)
2. AVC/H.264-decoder, either:
a. Cyberlinks AVC/H.264-package included within PowerDVD Version 7 or PowerDVD Version 6 with AVC/H.264-package ( http://www.cyberlink.com/ )
b. Elecards AVC/H.264-decoder included within MPEG2-player package or EmuxerPro AVC/H.264-plugin-package ( http://www.elecard.com/ )
c. Coreavc H.264-decoder together with Haali-demuxer package (http://www.coreavc.com/ )
3. Yamb 1.6 or 2.0 with MP4Box (included within package)
4. VirtualDubMod
5. ProjectX
6. DVBStreamexplorer ( http://www.dvbstreamexplorer.dk/ )
7. ffdshow a recent build with enabled H.264-decoding
1. Demux ts-file with ProjectX and read from the demux protocol the listed "PES-IDs".
You will e.g. find the following:
"ok> PID 0x90E has PES-ID 0xE0 (MPEG Video) (104310 #556)
ok> PID 0x910 has PES-ID 0xC8 (MPEG Audio) (119568 #637)"
or instead of the last line:
"ok> PID 0x103 has PES-ID 0xBD (private stream 1) (39480 #211)"
This line is shown if an AC3-Dolby-Digital-Audio-stream is present. Usually either a MPEG-audio stream or an AC3-stream is present besides the MPEG-video-stream.
Near the end of the demux protocol e.g. the following is listed:
"Audio PTS: first packet 08:15:11.361, last packet 08:16:53.345"
As ProjectX is not able to produce an AVC/H.264-stream the Video PTS has to be found in a different way. To accomplish this you have to open the recorded ts-file with a hex editor like the hex editor available within VirtualDubMod under "Tools". Using the "Find"-function from the "Edit"-menu the "Find binary string" opens, then enter the following characters into the "Find"-field: "00 00 00 01 67" and click the tic-box "Hexadecimal search string" and finally click on "Find next" to find this string. For example at the bottom of the following picture the found string is shown. Now find the next "00 00 01 E0" string just ahead of this string. In this picture just a few byte above starting at memory location "E0DCD" in the right center. Then finally find the video PTS which is contained within 5 bytes starting at the 9th byte after the beginning of the last searched string. Here marked within a black box. Here it reads: "35 7D 95 01 F1". (Fig1.jpg)
6183
The video PTS is a 33 bit value where the most significant bit is a sign bit.
This 33bit value is extracted from these 5 bytes. On the first byte a bitwise AND operation has to be performed with "0E". The result has to be bitwise shifted right one bit. The resulting 3bit-value is the 33th to 31st bit (counted from left to right in the binary representation of this 3bit-value), where the 33th bit the sign bit means: "0" is positive, "1" is negative. The remaining value has to be multiplied by 2 to the power of 30. The second byte is multiplied by 2 to the power of 22. The third byte value is first converted with a bitwise AND operation with "FE". The result bitwise shifted right for one bit and the result muliplied by 2 to the power of 15. The forth byte is multiplied by 128. The fifth byte value is first converted with a bitwise AND operation with "FE". The result bitwise shifted right for one bit. All five results are summed together and fitted with the obtained sign from the 33 bit value leading to the video PTS.
The video PTS is the system 27 MHz clock divided by 300 so a 90 kHz clock value. To convert the found value into milliseconds the value has to be divided by 90. A division of the obtained millisecond results by 3600000 converts to hours, by 60000 to minutes and by 1000 to seconds. The hour, minute, second and millisecond values are obtained by deleting the corresponding value after the decimal points. In this case "35 7D 95 01 F1" results into: "8:15:13.2969777" ( 8 hours, 15 minutes, 13 seconds, 296.9777 milliseconds. This means that the audio stream is delayed by -1.9359777 seconds (calculated by subtracting the video PTS-value from the audio PTS-value).
To properly demux the ts-file into a video-AVC/H.264 stream file and an audio-AC3 stream file or an audio-MPEG stream file either graphedit or DVBstreamexplorer is used. Start graphedit use Graph - Insert Filters - Direct Show Filters - File Source (Async.). Select the ts-file which is to be demuxed. Then Insert - Direct Show Filters - Elecard MPEG Demultiplexer (empgdmx.ax (version: 1.0.75.60727), 1012 KB). Demuxing with either Cyberlinks Demuxer contained within PowerDVD 7 or 6 (with AVC/H.264-decoder pack) or with the Haali Media Splitter instead cannot be used.
Connect the "File Source (Async.)"-box with the "Elecard MPEG Demultiplexer"-Box". After that select the "Dump"-filter. After selection enter a new file name with the file extension "asf", as this is the only extension the "Dump"-filter accepts. Nevertheless the "Dump"-filter produces a plain dump of the output of the demultiplexer. Connect the "H264"-output pin of the "Elecard MPEG Demultiplexer"-box with the input pin of the "Dump"-box. Finally click on the green triangular play button. Now the red "Stop" button apears. After demultiplexing finishes the green button appears again and the red button is greyed out again. (Fig2.jpg)
6184
Instead of using graphedit with Elecards MPEG Demultiplexer also DVBStreamexplorer can be used to extract the H.264-video stream:
At first at "Device" - "Driver selection" "TS file driver" has to be selected. Then at "Device" - "Tuning" the ts-file to be demultiplexed has to be selected. Having done this at "Action" - "SI scanning" - "SI tables" on the SI tables-box click on "Defaults" and "OK". Then click on "Action" - "SI scanning" - "Current transponder". This opens the "SI Reading"-box. On this box click on "Start" to scan the ts-file for its content. After some time this box can be closed again after the "Close"-button is not greyed out anymore. In the lower window at least "Got 1 PMT section(s) for SID XXXX" is displayed (with e.g. 27D8 at XXXX). Double-click on "Transponder" within the upper window to display the content section found due to the scanning operation. Now click on the small "+"-symbol left of "PMT" and left of the following subfolders. This opens the content-service tree (Fig3.jpg):
6185
Below the stream info line as shown in this figure click on the line "ITU-T Rec. H.262..."-line. Within the top right window the stream type is shown and two lines lower the "ELEMENTARY PID" here 0xFF (255). In this case the H.264-video is encased within a H.262-MPEG2-system package. DVBstreamexplorer does not analyze its content. (Fig4.jpg)
6186
Then under stream info line as shown in the figure above click on the line:
"ITU.T Rec. H.222.....". On the right side again the stream type and the "ELEMENTARY PID" here 0x103 (259) is shown.
Now knowing the elementary PIDs of the video and audio streams demux these streams by clicking on "Plugins" - "Streaming to disk" - "Save A/V ES streams" opens the "A/V ES Streaming"-box. Here at VPID the video PID found (here 255) and a file name for this stream has to be entered. The same has to be done at APID for the audio PID found (here 259) and a file name for the audio stream. Both file by default will have "bin" as file extension. Then start demuxing by clicking on "Start". You will have to watch if the file size of the stream files keeps increasing or stops to increase as an indication that the demuxing is finished as the demuxing function does not automatically switch off at the end of the file as is was initially programmed for indefinitely streaming satellite transmissions. So by clicking on "Stop" and "OK" this box will be closed.
This example ts-file has been recorded using the KNC TV Station DVB-S2+-card and the "Globe TV Digital"-programme.
Other recording devices and the associated programmes appear to be able to detect the AVC/H.264-video stream properly like in a short clip which was transmitted by "BBC HD1" (Fig5.jpg):
6187
Now open the produced files with the Hex editor of VirtualDubMod and again search for the hex string "00 00 00 01 67" as explained above. After the string is found, open Edit - Extract segment. Within the "Extract file segment" box the "Address (hex):" field is already filled with the start address. The value for the "Length (hex):" is found by search for the address for the last byte of the file within the Hex editor. From this hex value the found start address has to be subtracted (e.g. using the calculator within the accessories of Windows XP after switching to hex mode within the scientific view mode) and the result has to be increased by one. After this value has been entered click on "OK" and enter a file name for the h264-file to be produced e.g. "Pro7HD.h264". Now an "AVC/H.264"-file is produced with the correct start code (a key frame) and can be further processed as explained following.
Guide continues on following posting as maximum number of attached attachements is reached.