2 Encodes from the same script

[Atlantis]

I have an avisynth script in staxrip that is very slow because of the filters.
I need to encode 2 different files. I don't want to apply the same slow filters 2 times.

What are my options for 2 encodes and reading the script only once?
Can I do that directly from the script or should I create an intermediate lossless video file?
If yes, what are the newest lossless video codecs? And how much disk space do I need for 3 hours of 1920 x 1080 10bits?

[johnmeyer]

If you have "2 different files" and they both need to be run through the script, then you have to process each file separately.

The slowness is not in the reading of the script, but in the time it takes for each process in the script to manipulate the pixels in each frame.

If you want to make the script run faster, you can sometimes apply multi-threading and cut processing time in half (or even more). However, the success of multi-threading depends on what calls are in your script, what version of AVISynth you are using, etc.

[coolgit]

I need to encode 2 different files. I don't want to apply the same slow filters 2 times.

I am assuming you want slow filter for 1 file and another, presumably, faster filter for the other file.

If so do the slow filter and put it in the job queue. do another filter for the 2nd file and put it in the job queue. Run job. The job will store the different filters for each file.

[Atlantis]

No you didn't understand me. I think it was clear when I said I have to use an intermediate lossless video.

I have a script that is slow. I have to read the same script to do 2 jobs. I want to read that one script only once and do 2 encodes with different settings.
I know I have to encode the script into a lossless video and then do 2 encodes from that one lossless video. I was wondering if I could skip that.

[coolgit]

No you didn't understand me. I think it was clear when I said I have to use an intermediate lossless video.

Nope because earlier you said...

Can I do that directly from the script or should I create an intermediate lossless video file?

There is a difference between 'should' and 'have'.

[Atlantis]

Again, it's clear. I said I want to read the script only once.

[johnmeyer]

Actually, what you are trying to do, even after three posts, is not clear at all. Sorry, I'd like to try to provide some additional help, but I don't understand why "reading the script only once" is a requirement, nor is it clear why you think that will make any difference to the time it takes to finish the job.

[_Al_]

Have you tried this?: http://trac.ffmpeg.org/wiki/Creating%20multiple%20outputs
I just did a quick test and it worked, it looks like it created both files at the same time and perhaps pipes source simultaneously for both encodings. Tested only short file (using vapousynth, vspipe to ffmpeg which encoded two files), but it might as well work with avisynth as a source. How reliable it is , I have no idea.
vspipe script.vpy -c y4m - | ffmpeg -y -f yuv4mpegpipe -i - -s 1280x720 -c:a pcm_s24le -c:v prores output1.mkv -s 640x360 -c:a pcm_s24le -c:v prores output2.mkv
video is prores but you just might encode to h264 etc.

[Atlantis]

Thank you AI.

It's clear but I describe it again for those who want to understand.

Reading the script that applies all the filters takes 16 hours.
Encoding a 1080p file takes 1 hour. Encoding a 720p file takes 30 minutes.

If I want 2 encodes with that script I don't want to spend 2 x 16 = 32 hours
I want to spend around 16 + 1 + 1 = 18 hours

So my solution was to spend 16 hours to encode it to a lossless video and then spend 1 + 1 hours to do 2 encodes.

[StainlessS]

What he wants is to use eg "VirtualDub2" to output a single processed/filtered file via some lossless 10 bit VD2 codec.
EDIT: VD2 loads AVS scripts, [also has an avs script editor on tools menu]
He can then use that lossless file and encode two output files, one 1080p, and one 720p.
So, which is the better 10bit codec to use for VD2 ?

(I take it that above is correct requirement)
[EDIT: Of course, the 1080p encode will require a basic script to load the lossless file, and the 720p encode will require a script to downsize the lossless clip]

EDIT: If I were doing it, I'de just use UT_Video Lossless 10 bit codec, but perhaps there are better options. [I dont usually touch 10 bit]
EDIT: UT_Video Codec, [updated April 2023]:- https://www.videohelp.com/software/Ut-Video-Codec-Suite

EDIT: VirtualDub2:- https://forum.doom9.org/showthread.php?t=172021
See below UtVideo Pro codecs.
https://i.postimg.cc/1RjCS0xw/Ut-Video-Codec.jpg (https://postimages.org/)

EDIT: I think that you would need the Menu/Video/Fast Recompress option selected.
Would likely need to take look at Menu/Decode format too.
Perhaps someone else could advise here, [I only usually do 8 bit].

[Atlantis]

Thank you. Yes I remember UT Video Lossless. I remember using it many years ago. It was the best option back then. You could output directly to YV12 and no color conversion was needed.

The ffmpeg option provided above is exactly what I was looking for and the right and simplest way to do it.
However I use staxrip and not ffmpeg. I have to learn. Don't know if it is going to be as easy as staxrip.

[johnmeyer]

OK, the last explanation got through my thick skull. :)

You want to pipe the output of the script into two separate codecs, and create two video files, each one encoded with a different coder/decoder.

The most straighforward way to do this is to render to an intermediate codec. You were thinking of using a lossless codec, but I don't think this is necessary. Intermediate codecs are lossy which means they produce much smaller files than lossless codecs (less disk space, faster rendering). However, the good ones produce output that is visually "identical" to true lossless codecs. Cineform used to be the standard, but I think most people are using others at this point.

I still use Cineform, and I used to do tests to see if I could find any difference from an uncompressed file or lossless codec. I couldn't find any differences.

The render to Cineform will take some time, but it should be on the same order as the one hour you quoted that it will take you to render to 1080p. Thus, the "penalty" for doing it this way, rather than simultaneously piping to two codecs at the same time, will be one hour. If you only have to do this once, I'd recommend doing it that way because by the time you get the piping figured out, you will have killed more than an hour.

_Al_'s vspipe method could work (I have not used that tool, so I don't know what's involved). The problem with that approach is that it looks like it requires that you feed it from VapourSynth. Maybe it works with AVISynth as well, which is what you said you are using, but if not, the time to port over to VapourSynth will be far more than one hour.

So if it were me, and I was doing it with AVISynth, I'd follow StainlessS' recommendation.

[StainlessS]

Src, 1920x1080p 23.976FPS, 48KHz YV12 [EDIT: YUV420] 10Bit, 10:00.01 minutes.

VirtualDub2,
Decode Format AutoSelect.
Normal Recompress
Compression FFMPEG/Apple ProRes(Codec Pro) [P010 -> YUV422P16]
9.13GB, save time about 1:58 minutes

Complete name : D:\___BLURAY\TESTING\PRORES422.avi
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
Format settings : BitmapInfoHeader / WaveFormatExtensible
File size : 9.13 GiB
Duration : 10 min 0 s
Overall bit rate : 131 Mb/s
Frame rate : 23.976 FPS
Writing library : VirtualDub2 build 44282/release

Video
ID : 0
Format : ProRes
Format version : Version 0
Format profile : High
Codec ID : apch
Duration : 10 min 0 s
Bit rate : 126 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:2
Scan type : Progressive
Bits/(Pixel*Frame) : 2.537
Stream size : 8.81 GiB (96%)
Writing library : Lavc
Matrix coefficients : BT.470 System B/G

Audio
ID : 1
Format : PCM
Format settings : Little / Signed
Codec ID : 00000001-0000-0010-8000-00AA00389B71
Duration : 10 min 0 s
Bit rate mode : Constant
Bit rate : 4 608 kb/s
Channel(s) : 6 channels
Channel layout : L R C LFE Ls Rs
Sampling rate : 48.0 kHz
Bit depth : 16 bits
Stream size : 330 MiB (4%)
Alignment : Aligned on interleaves
Interleave, duration : 42 ms (1.00 video frame)
Interleave, preload duration : 500 ms


##########

Compression UtVideo, Fast Recompress, UTVideo Pro YUV420 10bit VCM (Configure, Predict Left)
15.00GB, save time about 1:20.00 minutes

General
Complete name : D:\___BLURAY\TESTING\UTVideo Pro YUV420_10Bit.avi
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
Format settings : WaveFormatExtensible
File size : 15.1 GiB
Duration : 10 min 0 s
Overall bit rate : 216 Mb/s
Frame rate : 23.976 FPS
Writing library : VirtualDub2 build 44282/release

Video
ID : 0
Format : UQY0
Codec ID : UQY0
Duration : 10 min 0 s
Bit rate : 211 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 23.976 (24000/1001) FPS
Bits/(Pixel*Frame) : 4.252
Stream size : 14.8 GiB (98%)

Audio
ID : 1
Format : PCM
Format settings : Little / Signed
Codec ID : 00000001-0000-0010-8000-00AA00389B71
Duration : 10 min 0 s
Bit rate mode : Constant
Bit rate : 4 608 kb/s
Channel(s) : 6 channels
Channel layout : L R C LFE Ls Rs
Sampling rate : 48.0 kHz
Bit depth : 16 bits
Stream size : 330 MiB (2%)
Alignment : Aligned on interleaves
Interleave, duration : 42 ms (1.00 video frame)
Interleave, preload duration : 500 ms


EDIT: Source was 1920x1080 borderless, ie full frame image.
EDIT: Audio was 6 channels, I did not touch it. [should be about max possible size for audio]

EDIT: i7-8700, load/save to nvme SSD.
Note, Prores is YV422-10bit, UTVideo YV420-10 bit.

EDIT:
From OP
And how much disk space do I need for 3 hours of 1920 x 1080 10bits?
So, assuming 23.976 FPS,
ProRes about 18.0 * 9.13 GB {164 GB}
UTVideo about 18.0 *15.0 GB. {270GB)
(3:00:00 = 180 mins = 18 * 10mins)

[_Al_]

It works with avisynth script directly as well:
ffmpeg -y -i test.avs -s 1280x720 -c:a pcm_s24le -c:v prores output1.mkv -s 640x360 -c:a pcm_s24le -c:v prores output2.mkv
Time for creating both output files was about 10s, whereas just one output took 1 second less.
So it must pipe that avs source for encoding of both files, thus reading avisynth script only once.
That was my tests, please test yourself, but it looks like it tremendously cuts the time for encoding and no intermediate is necessary (if script takes forever to process).
Did not test more outputs or large files. Not sure how much RAM is needed, but imagining more is needed if more outputs.
Also complex filtering might complicate things, not sure. I just used QTGMC and chained 20x blur(to slow down processing of script).

[Atlantis]

Thanks for mentioning CineForm, I did not know it. I will keep it in mind for future use. Since it's open source, why it is not on videohelp.com, software section?

[johnmeyer]

Thanks for mentioning CineForm, I did not know it. I will keep it in mind for future use. Since it's open source, why it is not on videohelp.com, software section?Cineform (https://en.wikipedia.org/wiki/CineForm) was originally developed as an intermediate codec aimed at the professional and broadcast markets. It was priced at close to $1,000.

About a decade ago it was purchased by GoPro and integrated into their GoPro CineForm Studio (https://gopro-cineform-studio.en.lo4d.com/windows) (click on that link to learn more). You can now get it for free via Studio.

I have never spent any time comparing Cineform to dnxhd, ProRes, or hqx, so I don't know if they have any advantages. They are all newer, so they probably have some advantages.

[StainlessS]

Maybe there should be some thread with list of major freebie software, da Vinci Resolve, Blender, GoPro CineForm Studio, etc.
JohnMeyer is the guy who knows where all the loot is buried.