Run x264 first pass, and output a lossless file at the same time?

[Blue_MiSfit]

Hey guys,

I had a thought today...

I have a workflow that involves running many encodes from a single source. Often, the sources require some heavy processing - i.e. mdegrain2 or something equally glacial.

Even though I'm heavily threaded (8 xeon cores), I can only push through 5-8fps on a 1080p source using MT AviSynth 2.5.8 and MVTools2, with 8-12 threads.

I wonder... would there be some way to run the firstpass for my x264 encodes, and at the same time output the result of the AviSynth script to a lossless file of some kind? This would drastically simplify and speed up my workflow.

Just a thought ;)

~MiSfit

[G_M_C]

Hey guys,

I had a thought today...

I have a workflow that involves running many encodes from a single source. Often, the sources require some heavy processing - i.e. mdegrain2 or something equally glacial.

Even though I'm heavily threaded (8 xeon cores), I can only push through 5-8fps on a 1080p source using MT AviSynth 2.5.8 and MVTools2, with 8-12 threads.

I wonder... would there be some way to run the firstpass for my x264 encodes, and at the same time output the result of the AviSynth script to a lossless file of some kind? This would drastically simplify and speed up my workflow.

Just a thought ;)

~MiSfit

The only thing i can think of is some kind of virtualisation / using VMWare or something like that. But if it will actually speed things up ... considering the overhead the virtualisation creates ?

Or did I understand your idea wrong ?

-- nvm -- misread you post. You want to output the same AVS your putting into x264 to be outputted at the same time. Sorry :(

[Blue_MiSfit]

It would probably require a modification to x264, as I don't know any other way to capture the output of AviSynth while still feeding into x264.

Maybe there's an AviSynth plugin that allows for in-script output - a-la SoundOut. If this hooked into VFW even, I could make a HuffYV12 file on-the-fly, and use this as the source for future encodes.

~MiSfit

[akupenguin]

avs2yuv in.avs -hfyu lossless.avi -raw - | x264 ...

[G_M_C]

The man is amazing, in here like a shot for a second or two with the answer. Amazing :cool:

[Blue_MiSfit]

Wow!!! I didn't know avs2yuv could do such things.

Hmm.... having avs2yuv would also let me use 64 bit x264 ;)

Much love, akupenguin!

~MiSfit

[7ekno]

Wow!!! I didn't know avs2yuv could do such things.

Just for completeness, have a look at TwriteAVI and the CachedFile() function here (http://forum.doom9.org/showthread.php?p=1073371#post1073371) ...

I invoke it in my AVS scripts with:
#Cachefile
ConvertToYUY2()
CachedFile("Z:\CACHED.AVI", "hfyu")
ConvertToYV12()

The advantage with the above is when re-running the same AVS script, it will recognise the already cached file, instead of processing the whole lot again ...

7ek

[Daiz]

Couldn't you do just do something like this?

x264 --qp 0 --pass 1 -o "lossless.mkv"

You probably wouldn't want to save a separate first-pass file anyway, so why not just use the lossless as the first pass?

[kemuri-_9]

You probably wouldn't want to save a separate first-pass file anyway, so why not just use the lossless as the first pass?

x264's lossless encoding disables a number of things, including bframes

[Dark Shikari]

x264's lossless encoding disables a number of things, including bframesAnd the bit cost calculation is inaccurate as well.

[moviefan]

I have also thought about this some time ago... Would the Avisynth way of doing it via a caching function work? And how big would the speed loss in the first pass be?

[Blue_MiSfit]

@Daiz:

I'd want to run a "normal" firstpass - i.e. write a statsfile relevant to a later secondpass (3 actually, but that's sufficiently unusual as is hehe..). The lossless file would be a source for several other encodes based on the same source (at lower resolutions).

@7enko:

So, TWriteAVI runs the script only if the "cached" version doesn't exist? Brilliant! This is a bit more in line with what I'd hoped to have!

Now... my VFW codec of choice is ffdshow's HuffYUV in YV12 mode. Is there any way to specify this? I know there's a FOURCC associated with the ffdshow encoder, but how do I tell it to use the HuffYUV encoder in YV12 mode?

Considering MJPG @ 95% quality as well...

~MiSfit

[Blue_MiSfit]

Okay, so now I'm cooking with gas :)

I downloaded TWriteAVI, and wrote a little .avsi for the CachedFile function etc..

And proceeded to test with a trim from one of my usual sources.


MPEG2Source("E:\Rework\TigersTrim.d2v", cpu=0)
ConvertToYV12

MT("""
super = MSuper(pel=2, sharp=1)
backward_vec2 = super.MAnalyse(isb = true, delta = 2, overlap=4)
backward_vec1 = super.MAnalyse(isb = true, delta = 1, overlap=4)
forward_vec1 = super.MAnalyse(isb = false, delta = 1, overlap=4)
forward_vec2 = super.MAnalyse(isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
""",8,4) # eight threads

CachedFile("cache.avi", "LAGS")


Here, my source is a YUY2 MPEG-2 file at 80mbps. As you can see, I'm applying MDegrain2 with 8 threads, at default parameters. This runs ~ 4fps on my 8 core 2.66 GHz Xeon :)

When I refresh the script, it loads from the Lagarith AVI :D :D :D

The only problem is - that damn cache file is _HUGE_ - of course..

Testing with ffdshow's MJPG encoder has produced mixed results. It's definitely throwing away more info than I'd like. Its "1 pass constant quality mode" doesn't seem terribly good - there's no difference in output between q95 and q100, and they're both chewing up the source more than I'd like. Fixed quantizer 1 is about twice the size and is noticeably better, but it's half the size of lossless at this point...

I've also played with some lossless codecs - Lagarith, FFV1, x264. File sizes are large - Lagarith and FFV1 are quite similar (but decode slowly), and x264 is much smaller, and decodes at a reasonable speed.

x264 VFW :devil: CRF10 seems pretty good, with 1 second GOPs...

~MiSfit

[thewebchat]

http://sourceforge.net/projects/x264vfw/files/ has up to r1195, but I don't think it matters for lossless. You could just use the x264 build in ffdshow.

[Blue_MiSfit]

The version in ffdshow is very old.. I ended up deciding that lossless was too big / slow to decode. CRF10 is working pretty well. I just need to ensure frame-accuracy - which could be iffy with VFW... :P
~MiSfit

[Dark Shikari]

The version in ffdshow is very old.. I ended up deciding that lossless was too big / slow to decode. CRF10 is working pretty well. I just need to ensure frame-accuracy - which could be iffy with VFW... :PVFW works fine as long as you don't use B-frames.

[Blue_MiSfit]

Gotcha. Trying an encode now without b-frames.

I know you don't have anything to do with VFW, but do you know what the "VirtualDub Hack" option is in the GUI?

Thanks DS :)

~MiSfit

[Dark Shikari]

Gotcha. Trying an encode now without b-frames.

I know you don't have anything to do with VFW, but do you know what the "VirtualDub Hack" option is in the GUI?Why are you using Virtualdub? Just use x264 and avc2avi.

[Blue_MiSfit]

Well... I'm just using VirtualDub for testing. Scan through the thread, and you'll see I'm trying to use TWriteAVI inside my AviSynth script. By simply using the FOURCC "x264" I can use the VFW encoder to output a "mezzanine" stream - I just have to pre-configure the VFW GUI.

I'm also going to try akupenguin's suggestion of using avs2yuv, but this would require the same pre-configuration.

~MiSfit

[Blue_MiSfit]

Hrm... so when I encoded like that, I get 10 frames of black in the beginning when decoded in VirtualDub, or 10 duplicate frames in the beginning when decoded in AviSynth. :p

I'm ready to throw VFW out the window, because x264 is the perfect balance of speed and coding efficiency (unsurprisingly), and encoding it via VFW is like pulling teeth..

Is there some way to use avs2yuv to feed two instances of x264 simultaneously? Then I could just toss VFW out the window, as should be the case :devil:

~MiSfit

[Dark Shikari]

Hrm... so when I encoded like that, I get 10 frames of black (when decoded in VirtualDub), or 10 seconds of freeze-frame when decoded in AviSynth. :P

I'm ready to throw VFW out the window, because x264 is the perfect balance of speed and coding efficiency (unsurprisingly).

Is there some way to use avs2yuv to feed two instances of x264 simultaneouslyman tee

[Blue_MiSfit]

That shows you how much I know about neat-o standard apps on linux etc.. :)

Fascinating... so tee essentially duplicates what it recieves via stdin. So, if I'm not being terribly foolish, it would be something like:


avs2yuv input.avs -raw - | tee x264.exe --crf 10 --preset faster --output "cache.mp4" | x264.exe --crf 23 --stats ".stats" --preset slower --output NUL

(plus all the fun stuff to help x264 with raw input of course, too lazy to look it up ATM)

Or am I not getting it? ;)

-MiSfit

[Dark Shikari]

No, that won't work. tee has to output to two separate streams. One for example might be a fifo.

[Chikuzen]

@Blue_MiSfit
I recommend you the use of ENPipe (http://ivtc.org/new/) instead of TWiteAVI.

[Blue_MiSfit]

oooooohh! neat!

[chaynik]

If this hooked into VFW even, I could make a HuffYV12 file on-the-fly, and use this as the source for future encodes.
HuffYV12, is there such a thing? I've been looking for something like that high and low!

[Dark Shikari]

HuffYV12, is there such a thing? I've been looking for something like that high and low!Yes, it's been part of ffmpeg for years (ffvhuff).

[Blue_MiSfit]

Yup, and you can use FFDShow's VFW encoder to encode to it using VirtualDub or any other VFW app. Or ffmpeg of course ;)

~MiSfit

[peter100m]

@Blue_MiSfit: Am looking for a similar thing, what did you end up using? Can't seem to find EnPipe anywhere, did you use that?