Log in

View Full Version : Indexing m3u8 file

FranceBB
25th February 2019, 19:29
Hi,
I have a .m3u8 metadata file which has metadata about many videos to append to make a single file.
I tried to index it using FFVideoSource, LWLibavVideoSource and DirectShowSource, but none of them recognized the .m3u8 metadata file.
I could simply index the .ts files individually and then append them together, but this is simply not practical.


LWLibavVideoSource("Z:\Distribution\segment_0.ts")
segment0=last

LWLibavVideoSource("Z:\Distribution\segment_1.ts")
segment1=last

LWLibavVideoSource("Z:\Distribution\segment_2.ts")
segment2=last

LWLibavVideoSource("Z:\Distribution\segment_3.ts")
segment3=last

LWLibavVideoSource("Z:\Distribution\segment_4.ts")
segment4=last

LWLibavVideoSource("Z:\Distribution\segment_5.ts")
segment5=last

LWLibavVideoSource("Z:\Distribution\segment_6.ts")
segment6=last

LWLibavVideoSource("Z:\Distribution\segment_7.ts")
segment7=last

LWLibavVideoSource("Z:\Distribution\segment_8.ts")
segment8=last

LWLibavVideoSource("Z:\Distribution\segment_9.ts")
segment9=last

LWLibavVideoSource("Z:\Distribution\segment_10.ts")
segment10=last

LWLibavVideoSource("Z:\Distribution\segment_11.ts")
segment11=last

LWLibavVideoSource("Z:\Distribution\segment_12.ts")
segment12=last

LWLibavVideoSource("Z:\Distribution\segment_13.ts")
segment13=last

LWLibavVideoSource("Z:\Distribution\segment_14.ts")
segment14=last

LWLibavVideoSource("Z:\Distribution\segment_15.ts")
segment15=last

LWLibavVideoSource("Z:\Distribution\segment_16.ts")
segment16=last

LWLibavVideoSource("Z:\Distribution\segment_17.ts")
segment17=last

LWLibavVideoSource("Z:\Distribution\segment_18.ts")
segment18=last

LWLibavVideoSource("Z:\Distribution\segment_19.ts")
segment19=last

LWLibavVideoSource("Z:\Distribution\segment_20.ts")
segment20=last

LWLibavVideoSource("Z:\Distribution\segment_21.ts")
segment21=last

LWLibavVideoSource("Z:\Distribution\segment_22.ts")
segment22=last

LWLibavVideoSource("Z:\Distribution\segment_23.ts")
segment23=last

LWLibavVideoSource("Z:\Distribution\segment_24.ts")
segment24=last

LWLibavVideoSource("Z:\Distribution\segment_25.ts")
segment25=last

LWLibavVideoSource("Z:\Distribution\segment_26.ts")
segment26=last

LWLibavVideoSource("Z:\Distribution\segment_27.ts")
segment27=last

LWLibavVideoSource("Z:\Distribution\segment_28.ts")
segment28=last

LWLibavVideoSource("Z:\Distribution\segment_29.ts")
segment29=last

LWLibavVideoSource("Z:\Distribution\segment_30.ts")
segment30=last

LWLibavVideoSource("Z:\Distribution\segment_31.ts")
segment31=last

LWLibavVideoSource("Z:\Distribution\segment_32.ts")
segment32=last

LWLibavVideoSource("Z:\Distribution\segment_33.ts")
segment33=last

LWLibavVideoSource("Z:\Distribution\segment_34.ts")
segment34=last

LWLibavVideoSource("Z:\Distribution\segment_35.ts")
segment35=last

LWLibavVideoSource("Z:\Distribution\segment_36.ts")
segment36=last

LWLibavVideoSource("Z:\Distribution\segment_37.ts")
segment37=last

LWLibavVideoSource("Z:\Distribution\segment_38.ts")
segment38=last

LWLibavVideoSource("Z:\Distribution\segment_39.ts")
segment39=last

LWLibavVideoSource("Z:\Distribution\segment_40.ts")
segment40=last

LWLibavVideoSource("Z:\Distribution\segment_41.ts")
segment41=last

LWLibavVideoSource("Z:\Distribution\segment_42.ts")
segment42=last

LWLibavVideoSource("Z:\Distribution\segment_43.ts")
segment43=last

LWLibavVideoSource("Z:\Distribution\segment_44.ts")
segment44=last

LWLibavVideoSource("Z:\Distribution\segment_45.ts")
segment45=last

LWLibavVideoSource("Z:\Distribution\segment_46.ts")
segment46=last

LWLibavVideoSource("Z:\Distribution\segment_47.ts")
segment47=last

LWLibavVideoSource("Z:\Distribution\segment_48.ts")
segment48=last

LWLibavVideoSource("Z:\Distribution\segment_49.ts")
segment49=last

LWLibavVideoSource("Z:\Distribution\segment_50.ts")
segment50=last

LWLibavVideoSource("Z:\Distribution\segment_51.ts")
segment51=last

LWLibavVideoSource("Z:\Distribution\segment_52.ts")
segment52=last

LWLibavVideoSource("Z:\Distribution\segment_53.ts")
segment53=last

LWLibavVideoSource("Z:\Distribution\segment_54.ts")
segment54=last

LWLibavVideoSource("Z:\Distribution\segment_55.ts")
segment55=last

LWLibavVideoSource("Z:\Distribution\segment_56.ts")
segment56=last

LWLibavVideoSource("Z:\Distribution\segment_57.ts")
segment57=last

LWLibavVideoSource("Z:\Distribution\segment_58.ts")
segment58=last

LWLibavVideoSource("Z:\Distribution\segment_59.ts")
segment59=last

LWLibavVideoSource("Z:\Distribution\segment_60.ts")
segment60=last

LWLibavVideoSource("Z:\Distribution\segment_61.ts")
segment61=last

LWLibavVideoSource("Z:\Distribution\segment_62.ts")
segment62=last

LWLibavVideoSource("Z:\Distribution\segment_63.ts")
segment63=last

LWLibavVideoSource("Z:\Distribution\segment_64.ts")
segment64=last

LWLibavVideoSource("Z:\Distribution\segment_65.ts")
segment65=last

LWLibavVideoSource("Z:\Distribution\segment_66.ts")
segment66=last

LWLibavVideoSource("Z:\Distribution\segment_67.ts")
segment67=last

LWLibavVideoSource("Z:\Distribution\segment_68.ts")
segment68=last

LWLibavVideoSource("Z:\Distribution\segment_69.ts")
segment69=last

LWLibavVideoSource("Z:\Distribution\segment_70.ts")
segment70=last

LWLibavVideoSource("Z:\Distribution\segment_71.ts")
segment71=last

part1=segment1++segment2++segment3++segment4++segment5++segment6++segment7++segment8++segment9++segment10
part2=segment11++segment12++segment13++segment14++segment15++segment16++segment17++segment18++segment19++segment20
part3=segment21++segment22++segment23++segment24++segment25++segment26++segment27++segment28++segment29++segment30
part4=segment31++segment32++segment33++segment34++segment35++segment36++segment37++segment38++segment39++segment40
part5=segment41++segment42++segment43++segment44++segment45++segment46++segment47++segment48++segment49++segment50
part6=segment51++segment52++segment53++segment54++segment55++segment56++segment57++segment58++segment59++segment60
part7=segment61++segment62++segment63++segment64++segment65++segment66++segment67++segment68++segment69++segment70
part8=segment71

encode=part1++part2++part3++part4++part5++part6++part7++part8

return encode



Basically the .m3u8 file does what I wrote above and it would be way easier and faster.
Any ideas?
videoh
25th February 2019, 21:06
Load all the .ts files at once in DGIndexNV and save project.
FranceBB
25th February 2019, 21:24
Load all the .ts files at once in DGIndexNV and save project.

I opened DGAVCIndex and I loaded all the files. (File -> Open).
I now have a file list of 71 files, but if I try to save the project [F4], it only indexes Segment 0.
videoh
25th February 2019, 22:16
DGAVCIndex is not DGIndexNV. The streams probably have EOS NALUs. If I remember correctly, DGIndexNV ignores them, but DGAVCIndex may not.
FranceBB
25th February 2019, 23:52
DGAVCIndex is not DGIndexNV. The streams probably have EOS NALUs. If I remember correctly, DGIndexNV ignores them, but DGAVCIndex may not.

I know, DGIndexNV is for paying users, is more updated and it uses Nvidia Video Cards via CUDA, but the reason why I never downloaded it it's because not all my computers have an NVIDIA GPU, as some of them are Linux boxes running Virtual Machines or dedicated servers without a GPU.
However, if it can run via software as well (CPU only), I will definitely switch from DGIndex/DGAVCIndex to DGIndexNV.
videoh
26th February 2019, 00:14
DGDecNV requires an nVidia GPU.
qyot27
26th February 2019, 02:03
Hi,
I have a .m3u8 metadata file which has metadata about many videos to append to make a single file.
I tried to index it using FFVideoSource, LWLibavVideoSource and DirectShowSource, but none of them recognized the .m3u8 metadata file.
I could simply index the .ts files individually and then append them together, but this is simply not practical.


LWLibavVideoSource("Z:\Distribution\segment_0.ts")
segment0=last

LWLibavVideoSource("Z:\Distribution\segment_1.ts")
segment1=last

LWLibavVideoSource("Z:\Distribution\segment_2.ts")
segment2=last

LWLibavVideoSource("Z:\Distribution\segment_3.ts")
segment3=last

LWLibavVideoSource("Z:\Distribution\segment_4.ts")
segment4=last

LWLibavVideoSource("Z:\Distribution\segment_5.ts")
segment5=last

LWLibavVideoSource("Z:\Distribution\segment_6.ts")
segment6=last

LWLibavVideoSource("Z:\Distribution\segment_7.ts")
segment7=last

LWLibavVideoSource("Z:\Distribution\segment_8.ts")
segment8=last

LWLibavVideoSource("Z:\Distribution\segment_9.ts")
segment9=last

LWLibavVideoSource("Z:\Distribution\segment_10.ts")
segment10=last

LWLibavVideoSource("Z:\Distribution\segment_11.ts")
segment11=last

LWLibavVideoSource("Z:\Distribution\segment_12.ts")
segment12=last

LWLibavVideoSource("Z:\Distribution\segment_13.ts")
segment13=last

LWLibavVideoSource("Z:\Distribution\segment_14.ts")
segment14=last

LWLibavVideoSource("Z:\Distribution\segment_15.ts")
segment15=last

LWLibavVideoSource("Z:\Distribution\segment_16.ts")
segment16=last

LWLibavVideoSource("Z:\Distribution\segment_17.ts")
segment17=last

LWLibavVideoSource("Z:\Distribution\segment_18.ts")
segment18=last

LWLibavVideoSource("Z:\Distribution\segment_19.ts")
segment19=last

LWLibavVideoSource("Z:\Distribution\segment_20.ts")
segment20=last

LWLibavVideoSource("Z:\Distribution\segment_21.ts")
segment21=last

LWLibavVideoSource("Z:\Distribution\segment_22.ts")
segment22=last

LWLibavVideoSource("Z:\Distribution\segment_23.ts")
segment23=last

LWLibavVideoSource("Z:\Distribution\segment_24.ts")
segment24=last

LWLibavVideoSource("Z:\Distribution\segment_25.ts")
segment25=last

LWLibavVideoSource("Z:\Distribution\segment_26.ts")
segment26=last

LWLibavVideoSource("Z:\Distribution\segment_27.ts")
segment27=last

LWLibavVideoSource("Z:\Distribution\segment_28.ts")
segment28=last

LWLibavVideoSource("Z:\Distribution\segment_29.ts")
segment29=last

LWLibavVideoSource("Z:\Distribution\segment_30.ts")
segment30=last

LWLibavVideoSource("Z:\Distribution\segment_31.ts")
segment31=last

LWLibavVideoSource("Z:\Distribution\segment_32.ts")
segment32=last

LWLibavVideoSource("Z:\Distribution\segment_33.ts")
segment33=last

LWLibavVideoSource("Z:\Distribution\segment_34.ts")
segment34=last

LWLibavVideoSource("Z:\Distribution\segment_35.ts")
segment35=last

LWLibavVideoSource("Z:\Distribution\segment_36.ts")
segment36=last

LWLibavVideoSource("Z:\Distribution\segment_37.ts")
segment37=last

LWLibavVideoSource("Z:\Distribution\segment_38.ts")
segment38=last

LWLibavVideoSource("Z:\Distribution\segment_39.ts")
segment39=last

LWLibavVideoSource("Z:\Distribution\segment_40.ts")
segment40=last

LWLibavVideoSource("Z:\Distribution\segment_41.ts")
segment41=last

LWLibavVideoSource("Z:\Distribution\segment_42.ts")
segment42=last

LWLibavVideoSource("Z:\Distribution\segment_43.ts")
segment43=last

LWLibavVideoSource("Z:\Distribution\segment_44.ts")
segment44=last

LWLibavVideoSource("Z:\Distribution\segment_45.ts")
segment45=last

LWLibavVideoSource("Z:\Distribution\segment_46.ts")
segment46=last

LWLibavVideoSource("Z:\Distribution\segment_47.ts")
segment47=last

LWLibavVideoSource("Z:\Distribution\segment_48.ts")
segment48=last

LWLibavVideoSource("Z:\Distribution\segment_49.ts")
segment49=last

LWLibavVideoSource("Z:\Distribution\segment_50.ts")
segment50=last

LWLibavVideoSource("Z:\Distribution\segment_51.ts")
segment51=last

LWLibavVideoSource("Z:\Distribution\segment_52.ts")
segment52=last

LWLibavVideoSource("Z:\Distribution\segment_53.ts")
segment53=last

LWLibavVideoSource("Z:\Distribution\segment_54.ts")
segment54=last

LWLibavVideoSource("Z:\Distribution\segment_55.ts")
segment55=last

LWLibavVideoSource("Z:\Distribution\segment_56.ts")
segment56=last

LWLibavVideoSource("Z:\Distribution\segment_57.ts")
segment57=last

LWLibavVideoSource("Z:\Distribution\segment_58.ts")
segment58=last

LWLibavVideoSource("Z:\Distribution\segment_59.ts")
segment59=last

LWLibavVideoSource("Z:\Distribution\segment_60.ts")
segment60=last

LWLibavVideoSource("Z:\Distribution\segment_61.ts")
segment61=last

LWLibavVideoSource("Z:\Distribution\segment_62.ts")
segment62=last

LWLibavVideoSource("Z:\Distribution\segment_63.ts")
segment63=last

LWLibavVideoSource("Z:\Distribution\segment_64.ts")
segment64=last

LWLibavVideoSource("Z:\Distribution\segment_65.ts")
segment65=last

LWLibavVideoSource("Z:\Distribution\segment_66.ts")
segment66=last

LWLibavVideoSource("Z:\Distribution\segment_67.ts")
segment67=last

LWLibavVideoSource("Z:\Distribution\segment_68.ts")
segment68=last

LWLibavVideoSource("Z:\Distribution\segment_69.ts")
segment69=last

LWLibavVideoSource("Z:\Distribution\segment_70.ts")
segment70=last

LWLibavVideoSource("Z:\Distribution\segment_71.ts")
segment71=last

part1=segment1++segment2++segment3++segment4++segment5++segment6++segment7++segment8++segment9++segment10
part2=segment11++segment12++segment13++segment14++segment15++segment16++segment17++segment18++segment19++segment20
part3=segment21++segment22++segment23++segment24++segment25++segment26++segment27++segment28++segment29++segment30
part4=segment31++segment32++segment33++segment34++segment35++segment36++segment37++segment38++segment39++segment40
part5=segment41++segment42++segment43++segment44++segment45++segment46++segment47++segment48++segment49++segment50
part6=segment51++segment52++segment53++segment54++segment55++segment56++segment57++segment58++segment59++segment60
part7=segment61++segment62++segment63++segment64++segment65++segment66++segment67++segment68++segment69++segment70
part8=segment71

encode=part1++part2++part3++part4++part5++part6++part7++part8

return encode



Basically the .m3u8 file does what I wrote above and it would be way easier and faster.
Any ideas?
Just cleaning up that syntax would be faster, to start with. Using variables and nesting variables together for no script-obvious reason¹ sucks up a lot of time if you're doing all of that manually (typing or copy-paste, it doesn't matter). And if you are doing it through an automatic process, there are better options to generate cleaner scripts for that, too. A one-line for loop in Bash could generate the AviSynth script to do the exact same task as the provided example, and do it in less than a second (or at least a few seconds):
for n in *.mkv ; do echo "LwLibavSource(\""$n"\") ++ \\" ; done | sed '$s, ++ \\,,'> output.avs
For FFMS2 at least, the indexing and generating the script can be done in two lines:
for n in *.mkv ; do ffmsindex -t -1 "$n" ; done

for n in *.mkv ; do echo "FFmpegSource2(\""$n"\",atrack=-1) ++ \\" ; done | sed '$s, ++ \\,,'> output.avs

The only caveat being that the number increments in the filenames need to be 0-padded so it orders them correctly.


¹ Since there's no indication that the 'part' variables are going to be filtered with differing parameters (or filtered at all), those serve no purpose except a certain style of readability. 'encode', likewise, isn't being chosen from among other very large variables with different combinations of inputs, so there's no point toward defining it as the concatenation of all those part segments, just to turn around and explicitly return 'encode'. It's a waste of space to also assign the 'segment' variables after the source invocation while using an explicit last, rather than simply assigning the source directly into the variable. And if there is no filtering to do on any of it, and it's purely just being appended together, then none of the variables are necessary at all.

In honesty, I wouldn't try to run the .m3u8 through AviSynth, or to try to keep the .ts files separate if the end result is just to have them concatenated anyway. I'd combine them using FFmpeg first, and then just use the single input as source to AviSynth rather than 71 of them. And FFmpeg understands *.m3u8 files just fine:
ffmpeg -i input.m3u8 -vcodec copy -acodec copy -bsf:a aac_adtstoasc output.mp4
(or whatever your desired output container is)
lansing
26th February 2019, 09:16
While we're at it, I just want to report that DGIndexNV output the wrong color in vapoursynth on my dvd video, while no problem in avisynth.
videoh
26th February 2019, 14:54
While we're at it, I just want to report that DGIndexNV output the wrong color in vapoursynth on my dvd video, while no problem in avisynth. If you provide a stream and your script etc. I can look into it. Your post here is too vague to do anything about. It's not clear that we don't have simple pilot error. You can open a thread at my forum too if you prefer.
FranceBB
26th February 2019, 18:04
In honesty, I wouldn't try to run the .m3u8 through AviSynth, or to try to keep the .ts files separate if the end result is just to have them concatenated anyway. I'd combine them using FFmpeg first, and then just use the single input as source to AviSynth rather than 71 of them. And FFmpeg understands *.m3u8 files just fine:
ffmpeg -i input.m3u8 -vcodec copy -acodec copy -bsf:a aac_adtstoasc output.mp4
(or whatever your desired output container is)

Got it.
I guess you are right.
I'm running them through ffmpeg with the following bat file as it's so much easier:


@echo off
setlocal disableDelayedExpansion
::
:: first assume normal call, get args from %*
set args=%*
set "dragDrop="
::
:: Now check if drag&drop situation by looking for %0 in !cmdcmdline!
:: if found then set drag&drop flag and get args from !cmdcmdline!
setlocal enableDelayedExpansion
set "cmd=!cmdcmdline!"
set "cmd2=!cmd:*%~f0=!"
if "!cmd2!" neq "!cmd!" (
set dragDrop=1
set "args=!cmd2:~0,-1! "
set "args=!args:* =!"
)
::
:: Process the first argument only
for %%F in (!args!) do (
if "!!"=="" endlocal & set "dragDrop=%dragDrop%"
rem ------------------------------------------------
rem - Your file processing starts here.
rem - Use %%F wherever you would normally use %1
rem
rem Change to drive and directory of input file
%%~dF
cd %%~pF
C:\Encoding\ffmpeg.exe -hwaccel dxva2 -i "%%~nxF" -threads 32 -vcodec copy -acodec copy "C:\Encoding\%%~nF output.mp4"
rem
rem - Your file processing ends here
rem -------------------------------------------------
)
if defined dragDrop (
pause
exit
)
lansing
26th February 2019, 18:30
If you provide a stream and your script etc. I can look into it. Your post here is too vague to do anything about. It's not clear that we don't have simple pilot error. You can open a thread at my forum too if you prefer.

Here's the sample clip (https://www.mediafire.com/file/jrncit0jhzpztu9/dgnv_color_problem.ts/file), the red color is off when opening in vapoursynth.

Okay enough hijacking for me.
videoh
26th February 2019, 20:15
Thanks for not providing your script, telling me why you think it is "off", etc. :angry: