TrimSubs - Script to cut text subtitles + AvsP macros

[vdcrim]

This Python script parses a specified Avisynth script for a line with uncommented Trims, and cuts an input text subtitle file accordingly so the subtitles match the trimmed video. It's intended as a companion to vfr.py (http://forum.doom9.org/showthread.php?t=154535), script that creates chapter files based on Trims and that also can cut audio files. It can be specially useful for Matroska ordered chapters.

There are three ways of specifying the line of the avs used:
· Parse the avs from top to bottom (default) or vice versa, and use the first line with Trims found.
· Use a line with a specific comment at the end, e.g: Trim(0,99)++Trim(200,499) # cuts
· Directly specifying the Trims line number, starting with 1.

A frame rate or timecode file (v1 or v2) is required, except for MicroDVD subtitles. A new trimmed timecode v2 file can be generated optionally.

Supported subtitle formats: ASS, SSA, SRT, SUB (MicroDVD).

usage: TrimSubs.py script.avs
[-h [{doc}]] [-V] [-v] [-r] [-l LABEL] [-g LINE] [-f FPS]
[-t [OTC]] [-i [INPUT]] [-c ENCODING] [-o OUTPUT]

Info arguments:
-h [{doc}], --help [{doc}]
Show a help message and exit. Add 'doc' to
include also the documentation
-V, --version Show program's version number and exit

Required arguments:
script.avs Avisynth script containing Trims

Optional arguments:
(--input or --otc parameter is required)

-v, --verbose Show detailed info
-r, --reversed Parse the avs from bottom to top
-l LABEL, --label LABEL
Use the Trims from the line in the avs that ends in
a commentary with LABEL
-g LINE, --line LINE Use the Trims from the line nº LINE
-f FPS, --fps FPS Frame rate or timecode file (v1 or v2). If omitted,
search for a timecode file or default to 24000/1001
-t [OTC], --otc [OTC]
Output a new timecode file. Path optional
-i [INPUT], --input [INPUT]
Input subtitle file. If INPUT is not specified,
search for a valid input file
-c ENCODING, --encoding ENCODING
Input subtitle file encoding
-o OUTPUT, --output OUTPUT
Custom path for the output subtitle file


TrimSubs.py (http://www.mediafire.com/download.php?gf1l5uadywyu6pg) (Python script, needs Python 3.2 (http://python.org/) and PySubs (http://pypi.python.org/pypi/pysubs))
TrimSubs.exe (http://www.mediafire.com/download.php?516e1crb63pfqv3) (Windows executable, without dependencies)
TrimSubs repository (https://github.com/vdcrim/TrimSubs)


I've made some AvsP (http://forum.doom9.org/showthread.php?t=153248) macros to easy the use of vfr.py and TrimSubs. These links may be out of date, latest version is always here (https://github.com/vdcrim/AvsP-macros):

· AvsP macro - Insert and format Trims (http://www.mediafire.com/download.php?ny5ui15sg4gh0vw)
Contains the following macros:
· Insert Trims from bookmarks (single-line)
· Insert Trims from bookmarks (multi-line)
· Format MeGUI Trims (single-line)
· Format MeGUI Trims (multi-line)
· AvsP macro - Insert Trims from Matroska chapter file (http://www.mediafire.com/download.php?v7m64evbxc9mbu7)
This can be useful to redo a previously non-ordered chapters encode to ordered, if the original avs is no longer available. Currently only constant frame rate is supported.
· AvsP macro - Create/join timecodes from Trims (http://www.mediafire.com/download.php?t0dmz58sqda1511)
Assign a FPS or timecode to every Trim in a specified line and create a new timecode that can span all the video range or only the Trims. It's meant to be used when obtaining a VFR video as a result of deinterlacing/IVTC various sections of the video in different ways, or joining several videos of different FPS.
. AvsP macro - Trim timecode
This functionality is already included on TrimSubs, but it's useful to have it as a stand-alone macro as well.
· AvsP macro - Matroska chapters from Trims (http://www.mediafire.com/download.php?f66a63suct3s8k6) (needs vfr.py (http://forum.doom9.org/showthread.php?t=154535) and TrimSubs)
Wrapper for vfr.py and TrimSubs.py. Generate Matroska chapters from Avisynth Trims. Additionally cut audio and text subtitles to match the trimmed video.
· AvsP macro - Chapter names from file (http://www.mediafire.com/download.php?5p9spims5f50c7l)
Replace default chapter names in the Matroska chapter files on a given directory with the ones provided in a specified file, one chapter name per line.

The macros must be put in your "AvsPmod\macros" directory. Note that the order of appearance in the macro menu can be customized by prefixing "[number]" to the filenames. Please read the complete info and change (if desired) the preferences on the first lines of each script.


Changelog:
2011-12-30: initial release
2012-01-29: see post #2 (http://forum.doom9.org/showpost.php?p=1554426&postcount=2)

[vdcrim]

TrimSubs updated to 0.2:
· Added --line parameter
· Support for negative second member of the Trim pair

'Matroska chapters from Trims' updated to v2:
· Added custom parsing order and label feature. Needs vfr.py 0.8.6.1+

'Create-join timecodes from Trims' updated to v2:
· Support for negative second member of the Trim pair

Added new macros:
· Insert Trims from bookmarks (multi-line)
· Format MeGUI Trims (single-line)
· Format MeGUI Trims (multi-line)
· Insert Trims from Matroska chapter file
· Chapter names from file

[Yellow_]

hi, I wonder if you could suggest a way of doing something similar.

I'm using AVS2pipemod + Imagemagick making big files that are high on memory and disk space usage, the whole video has to be loaded before writing image sequences out with IM. I'm using IM as it's 16bit RGB from the Dither Plugin to 16bit half float EXR.

So I'm trying to code a python macro for AVSPmod to divide the video into chunks by trimming at the bookmarks set by intervals, but creating a new .avs file instead of a multiline / single line list in one .avs as per your scripts. I've manually created the .avs scripts previously.

Looking to get an .avs script for each bookmark so that I can batch process the avs files created through AVS2pipemod sequentially.

[vdcrim]

Looking to get an .avs script for each bookmark so that I can batch process the avs files created through AVS2pipemod sequentially.

I assume you already have the bookmarks ready (probably by using the 'Bookmarks at Intervals' macro). Try this:


AvsP macro - Divide script [latest version] (https://github.com/vdcrim/AvsP-macros/blob/master/Divide script.py)'''
Divide an Avisynth script into multiple avs according to AvsP bookmarks.
The first and last frame are automatically added to the bookmarks if not
already present.
'''

# If True, save the new scripts to the same directory and with the same
# name as the input avs (if exists).
use_same_avs_dir = False

# If True, save the scripts always to this location, e.g. ur"C:\Scripts",
# using the following basename. If basename is empty take it from the
# input avs. If there's not avs, prompt for a name:
use_avs_dir = False
avs_dir = ur""
base_name = ur""


# ------------------------------------------------------------------------------


from os import makedirs
from os.path import isdir, basename, join
import codecs

bm_list = avsp.GetBookmarkList()
if not bm_list:
avsp.MsgBox("There is not bookmarks", 'Error')
return
avs = avsp.GetScriptFilename()
if not (use_same_avs_dir and avs):
if use_avs_dir and avs_dir:
if not isdir(avs_dir):
makedirs(avs_dir)
if not base_name:
if avs:
base_name = basename(avs)
else:
base_name = avsp.GetTextEntry('Introduce a basename for the new scripts',
'avs_trim', 'Divide script')
if not base_name:
return
avs = join(avs_dir, base_name)
else:
avs = avsp.GetSaveFilename('Select a directory and basename for the new scripts')
if not avs:
return
if avs.endswith('.avs'):
avs = avs[:-4]

bm_list.sort()
if bm_list[0] != 0:
bm_list[:0] = [0]
frame_count = avsp.GetVideoFramecount()
if bm_list[-1] == frame_count - 1:
bm_list[-1] = frame_count
else:
bm_list.append(frame_count)
digits = len(str(len(bm_list) - 1))
text = avsp.GetText()
avs_list = []
for i, bm in enumerate(bm_list[:-1]):
avs_path = u'{}_{:0{}}.avs'.format(avs, i+1, digits)
avs_list.append(avs_path)
with codecs.open(avs_path, 'w', 'utf-8') as f:
f.write(text + '\nTrim({},{})\n'.format(bm, bm_list[i+1] - 1))
return avs_list

[Yellow_]

ok, that's wicked, thank you so much.

I'd spent 5 hours on and off and got nothing like your code. My python skills or lack of, need serious improvement. :-)

I tested it with a 423 frame video ie: 0 - 422, bookmarks set by interval of 200 frames and recieved 3 .avs files, with trims (0, 199) , (200,399) and (400,420)

So I think I'm just loosing a few frames off the end?

I'm going to try and understand your code to see if I can resolve it, but could take a while. ;-)

[vdcrim]

So I think I'm just loosing a few frames off the end?

Yep, last frame was missing. Fixed :)

[Yellow_]

many many thanks, hope this macro is useful to others too.

I'd like to update a post on my blog, http://www.http://blendervse.wordpress.com/2011/09/16/8bit-video-to-16bit-scene-referred-linear-exrs/ from a while back, with your macro to resolve the problem I described, is that ok with you, if I credit you and link to this thread?

thanks again.

[vdcrim]

Sure. Nice blog btw, I think I've already stumbled upon it before.

[Yellow_]

Cheers. :-)

[Yellow_]

hi again, long time :-)

The script is great, but at the moment it adds the Trim() to the bottom of the script however, I wonder if it's possible for the trim(?,?) to be added at the cursor position, the reason I'd like to be able to trim a source before further processing so having control over where the trim is added would be great.

[vdcrim]

Try this new version (https://github.com/vdcrim/AvsP-macros/blob/master/Divide script.py) (raw link). The latest version is always there. If the 'split at the current cursor position' option is used then the script is only evaluated until the line the cursor is on, so posterior filters that change the framecount don't affect the splitting.

Also, the list of created scripts is saved as a global variable avs_list. Besides creating a batch file you can also start the processing of these files with a macro like this:

# run macro in new thread

import os.path
import subprocess

for avs in avs_list:
if subprocess.call(['program.exe', '--arg1', 'x', '--arg2', 'y', avs]):
avsp.MsgBox(_('Error processing "{0}"').format(os.path.basename(avs)),
_('Error'))
avsp.MsgBox(_('Done!'), _('Info'))

[Yellow_]

Great, thanks, I have a proble though, when I try using the Divide macro with either using the bookmarks or by specifying a frame step, I get a pause, then an error message 'Error loading the script". :-(

[vdcrim]

Great, thanks, I have a proble though, when I try using the Divide macro with either using the bookmarks or by specifying a frame step, I get a pause, then an error message 'Error loading the script". :-(
The pause is the macro evaluating the script in order to get the total number of frames, and the message means that there's an error in the script. My best guess is that the last line of your script (or the line the cursor is on if 'split at the current cursor position' is checked) is an assignment. In that case the script doesn't have a return value. I'm going to append 'last' to the text that is evaluated to workaround this.

[Yellow_]

Hi, thanks, although the script I was testing on had heavy processing further down, I'd thrown a return last in immediately after the initial ffmpegsource2(...) in order to test the trim addition without incurring heavy processing section of the script at this time. So correct in your analysis.

many thanks, will try again.

[vdcrim]

Instead of just appending 'last' I added a new option to always use the last evaluated expression, even if it's assigned to a variable. Also fixed the text that is evaluated when the cursor is on a multiline comment.

[Yellow_]

hi, I'm still not getting anywhere, every option causes error can't load script. :-( I must be doing something wrong.

Here's my script:

LoadPlugin("c:\Program Files\AviSynth 2.5\plugins\mvtools2.dll")
LoadPlugin("c:\Program Files\AviSynth 2.5\plugins\fft3dfilter.dll")
LoadPlugin("c:\Program Files\AviSynth 2.5\plugins\removegrain.dll")
LoadPlugin("c:\Program Files\AviSynth 2.5\plugins\mt_masktools-25.dll")
LoadPlugin("c:\Program Files\AviSynth 2.5\plugins\dfttest.dll")

SetMemoryMax(512)

source=ffmpegsource2("videofile.mkv", threads=1)

denoised=MCTD(dfttest(source, sigma=4.0, lsb=true), chroma=true, settings="low")

Dither_convert_yuv_to_rgb(denoised, matrix="601", tv_range=false, cplace="MPEG2", chromak="bicubic", lsb_in=true, output="rgb48y")
Dither_y_gamma_to_linear (tv_range_in=false, tv_range_out=false, curve="709")
Dither_convey_rgb48_on_yv12 (SelectEvery (3, 0),SelectEvery (3, 1),SelectEvery (3, 2) )


I'd like to create multiple avs files of the above with a trim(?,?) added to the end of the line:

source=ffmpegsource2("videofile.mkv", threads=1).trim(?,?)

For each bookmark set or interval set etc.

The reason for this is that I intend piping batches of around 350 to 500 16bit interleaved frames from thousands via avs2yuv to imagemagick for encoding to 16bit linear exr's, any more than about 500 frames at a time will cause memory errors on the machine I'm using for this currently. I'm manually adding trim command and trim points to multiple avs scripts currently and would like to automate it.

The problem with avs2yuv type route I believe is that all frames must be accounted for before processing starts and this is simply too many for memory + temp harddisk space, so some sort of batching required.

But alternatively if it were possible to do this directly from AVSPmod in a single avs script in a similar way to 'save to mp4' from the tools menu by calling avs2yuv or avs2pipemod and feeding batches of so many frames directly from AVSPmod negating need for multiple scripts, would this be possible because I find the more complex the processing required, the smaller the batch increments and therefore increased number of batch scripts needed to prevent memory errors.

Thanks for your time making changes to date.

[vdcrim]

every option causes error can't load script

You have to put the cursor on the 'source=...' line and then use the macro checking both 'Split at the current cursor position' and '... using the last evaluated expression' options. Works for me with your script. However the generated scripts weren't correct, that's fixed now.


The problem with avs2yuv type route I believe is that all frames must be accounted for before processing starts and this is simply too many for memory + temp harddisk space, so some sort of batching required.

But alternatively if it were possible to do this directly from AVSPmod in a single avs script in a similar way to 'save to mp4' from the tools menu by calling avs2yuv or avs2pipemod and feeding batches of so many frames directly from AVSPmod negating need for multiple scripts, would this be possible because I find the more complex the processing required, the smaller the batch increments and therefore increased number of batch scripts needed to prevent memory errors.

The 'save to mp4' tool is in fact a general script encoder that executes a batch file generated from a template file. It won't give much benefit over just writing yourself the batch, and you would still have to create the trimmed scripts. What you can do is use a macro to write your batch file, or just use the snippet I posted above instead of the batch.

Another way is to pipe directly from AvsPmod to ImageMagick. In that case all could be done from a macro, without creating temporary avs scripts and a batch file. I already made something like that on this Create GIF with ImageMagick (https://github.com/vdcrim/AvsP-macros/blob/master/Create GIF with ImageMagick.py) macro. Of course the disk space/memory issue would still be present, so the data shouldn't be piped in just one go like in that macro. Send me your command line and I'll take a look, but I can't promise you anything. EDIT: just found out that there's an example on the Dither docs, should be enough.

[Yellow_]

Thanks for your reply and suggestions, leave it with you if you get time, just wanted to say the command line in the Dither docs was provided by myself so its the one I would have provided had I responded before your edit. :-)

Thanks again.

[vdcrim]

I just posted a macro script on the AvsPmod thread (http://forum.doom9.org/showthread.php?p=1599178#post1599178). Let's keep the discussion there.

[real.finder]

hi, first thank you for wonderful macros :)

I wonder if you could made macro that create x264 qpfile from the mkv chapters with support v1 & v2 timecodes and cfr too

[vdcrim]

hi, first thank you for wonderful macros :)

I wonder if you could made macro that create x264 qpfile from the mkv chapters with support v1 & v2 timecodes and cfr too

Here's a demo (https://github.com/vdcrim/AvsP-macros/blob/master/QP file from Matroska chapter file.py) (actually just a quickly modified 'Insert Trims from Matroska chapter file'). Not really what you ask for, because:

Only CFR
May not work properly for some ordered chapters or editions

Is it worth to make a better version? I'm sure you already know that vfr.py can create a qpfile at the same time as the chapter file.

[real.finder]

thank you for demo :)

I'm sure you already know that vfr.py can create a qpfile at the same time as the chapter file.

yes, I know that, but the vfr.py create it from trims only.

And the author said: "Conversion from a different input fps to output fps is not accurate"

and I'm working on a lot of sources, most of them have output fps =\= input fps (Whether vfr or cfr)

So I think I can create the chapter by vfr.py and then convert the chapter to qpfile

Because whatever the input fps different from the output fps, the chapter is Time Stats

So when I convert it to qpfile, I can choose the new fps for new video as I want


Thank you in advance

[vdcrim]

From what I see the frame numbers are corrected before writing them to the qpfile when the output fps is different from the input. That means that vfr.py and the macro above should produce the same result if all is correct.

To avoid that possible inaccuracy the readme talks about you can instead add a new line of contiguous Trims at the bottom of the script, and then use the --ofps value as --fps and drop the former from the command line, and add --reverse as well.

[real.finder]

I see, so I have to write new trims line after the new fps :(

In this case it will not differentiate much than I wrote qpfile manually

Also, vfr.py work from the trims only, it is not convert chapter to qpfile

anyway, thank you

[vdcrim]

I see, so I have to write new trims line after the new fps :(

In this case it will not differentiate much than I wrote qpfile manually

Also, vfr.py work from the trims only, it is not convert chapter to qpfile

anyway, thank you
I've never used vfr.py's fps convertion option and can't really talk about its possible inaccuracy, but anyway as the chapter file and the qpfile are created from the same corrected frames/times the frames in the qpfile should match the chapter starting times. If I understand well the thing that could be not completely exact is those starting times. So you would still have to write new trims anyway. Better try asking RiCON/wiiaboo about this.

[real.finder]

thank you, I'll try to edit your demo myself

[vdcrim]

thank you, I'll try to edit your demo myself
What I'm saying is that a macro may not be more accurate that vfr.py's qpfile. Again, ask wiiaboo what he means by 'conversion from a different input fps to output fps is not accurate'.

[real.finder]

from vfr.py Thread

"Can output a qpfile with converted frames meant to be used for an ivtc'd encode using non-ivtc'd frames (feature inspired by automkvchapters) (not completely accurate, obviously);"

and like I say, I need a chapter to qpfile converter

Because I work on sources not need to trim and have a chapter already

thank you again

[vdcrim]

from vfr.py Thread

"Can output a qpfile with converted frames meant to be used for an ivtc'd encode using non-ivtc'd frames (feature inspired by automkvchapters) (not completely accurate, obviously);"
That doesn't say from where the inaccuracy comes from. And anyway a macro would have exactly the same input data to produce a qpfile, so there's not reason to think it would be better. Again, as I don't use --ofps I can't say if vfr.py works as expected.

and like I say, I need a chapter to qpfile converter

Because I work on sources not need to trim and have a chapter already
OK then, I'll work on it.

[vdcrim]

Timecode support added. Try this (https://github.com/vdcrim/AvsP-macros/blob/master/QP file from Matroska chapter file.py) (raw link).

[real.finder]

thank you very much :) I'll try it

[real.finder]

it work great ;)

thank you, you are life saver

[real.finder]

hi, I have two mod

Insert Trims from bookmarks MOD (single-line).py (http://pastebin.com/pSbrTQaL)

to make a full trims line from 1 bookmark or more

---------


QP file from Matroska chapter file (IDR).py (http://pastebin.com/SEg1XhpJ)

same as "QP file from Matroska chapter file" but the qp file will be I (IDR) instead of k, useful for open-gop, more details here (http://mewiki.project357.com/wiki/X264_Settings#qpfile)

----

edit: another mod macros https://forum.doom9.org/showthread.php?p=1770449#post1770449

[vdcrim]

QP file from Matroska chapter file (IDR).py (http://pastebin.com/SEg1XhpJ)

same as "QP file from Matroska chapter file" but the qp file will be I (IDR) instead of k, useful for open-gop, more details here (http://mewiki.project357.com/wiki/X264_Settings#qpfile)

i-frames with recovery point SEI (that your link shows that K uses for open-gop) should be seekable as well. Did you misread somewhere or did you really notice artifacts when using K instead of I?

[real.finder]

i-frames with recovery point SEI (that your link shows that K uses for open-gop) should be seekable as well. Did you misread somewhere or did you really notice artifacts when using K instead of I?

I did't notice anything, but for more safety, and good place to cut without re-encoding

[real.finder]

hi

I did some edit, in "time2ms" and make "QP file from Matroska chapter file" work with mp4 chapters too, I change the name to "QP file from chapter file"

edit: added .ogm too

mod 3 for fix some float cases

github https://github.com/realfinder/AvsP-macros

[JudgeDredd]

what exactly are my options for -c switch ? I tried utf-8 etc but non works

[real.finder]

I did try to make this https://github.com/realfinder/AvsP-macros/blob/master/Bookmarks%20from%20Subtitle.py

it work but sometimes it not accurate, I already try random fix but I think there are something missing, maybe peoples who has knowledge in vsfilter or libass can explain how to correctly convert subtitle time to frames

edit: I get an answer in irc.libera.chat/#libass, ceil(start timestamp in seconds * frames per second)

[real.finder]

Matroska chapters from Trims.py update https://github.com/realfinder/AvsP-macros/blob/master/Matroska%20chapters%20from%20Trims.py

there are also some other updates in https://github.com/realfinder/AvsP-macros I didn't mention here

[Emulgator]

avsp.MsgBox(stdout, _('vfr.py rrror'))
Intentional ?

[real.finder]

Intentional ?

seems typo, as vdcrim seems not even finish it, like I note that "QP file" part not there but it mention that it should output it