OpenDML reading in x264? (or other ways to use 32 and 64-bit together)

[omion]

Is there any chance of getting x264 to read OpenDML AVI files?

A bit of background: as those of you who have been following my previous whining may know, I have Windows x64 and a self-built 64-bit version of x264. However, in order for it to use avisynth files, I need a 64-bit version of all the filters I use (not gonna happen).

I just recently realized that I can save the .avs file to an uncompressed .avi and read that with the 64-bit x264. As the 64-bit version goes ~10% faster than its 32-bit counterpart, this extra step still decreases the total time taken.

Now, the problem is that uncompressed DVD-quality video is ~15GiB per half-hour episode. This is where the OpenDML comes in: x264 will only encode the first 2 minutes of the episode, which corresponds exactly to 2GiB. I know VirtualDubMod writes OpenDML files, as I can play them back fine using MPC. This leads me to believe than x264 is only following the "non-OpenDML" index, and therefore does not handle >2GiB AVI files.

I would also greatly appreciate any ideas of how to use 32-bit avisynth with 64-bit x264 and sidestep the whole OpenDML problem altogether. My solution is far from optimal, and currently does not even work.

[akupenguin]

Pipe.
avs2yuv foo.avs -raw - | x264 - ...
May require cygwin/msys or something; at least some versions of cmd.exe simulate a pipe by dumping to a file, and that gets hairy.

x264 doesn't deal with avi directly, it uses AviFile, so we have no control over ODML support.
But one of these days I'll fix largefile for rawyuv reading.

[omion]

Yay! That's just what I wanted! I had thought about piping stuff to x264, but couldn't think of what to use for the source. This avs2yuv is perfect.

Apparently my cmd.exe doesn't use a temp file (at least as far as I can tell) so I think that's what I'll use. Thanks again!

[squid_80]

Originally Posted by omion

However, in order for it to use avisynth files, I need a 64-bit version of all the filters I use (not gonna happen).

Which filters?

[omion]

My current project (Fullmetal Alchemist) uses the following script:

LoadPlugin("F:\MULTIMEDIA\DGindex\DGDecode.dll")
LoadPlugin("F:\MULTIMEDIA\AviSynth 2.5\plugins\undot.dll")
LoadPlugin("F:\Multimedia\AviSynth 2.5\plugins\aWarpSharp.dll")
LoadPlugin("F:\Multimedia\AVIsynth 2.5\plugins\Decomb521VFR1.3.dll")
mpeg2source("d2v.d2v",cpu=6,iPP=false)
Telecide(order=1,guide=0,post=2,back=1,bthresh=40,nt=20,ovr="over.txt",show=false).Decimate(mode=2,vfrStats="vfrStats.txt",show=false)
Levels(last, 16,1.0,235, 0,255, false)
aWarpSharp(8.0, 2, 0.5, 2)
Undot()
crop(8,0,704,480)

Other (dirtier) sources also used deen, vagueDenoiser, and TomsMoComp. Also note that although I'm not currently using the VFR feature of the decomb plugin, I have in the past and I probably will need to in the future.

[bobololo]

Originally Posted by omion

Is there any chance of getting x264 to read OpenDML AVI files?

[...]

I would also greatly appreciate any ideas of how to use 32-bit avisynth with 64-bit x264 and sidestep the whole OpenDML problem altogether. My solution is far from optimal, and currently does not even work.

Wouldn't this avs script work:

AviSource("uncompressed-huge-clip.avi")

and give this script to x264 ? This would use avisynth internal avi parser that supports open-DML.

[squid_80]

Originally Posted by omion

LoadPlugin("F:\MULTIMEDIA\DGindex\DGDecode.dll")
LoadPlugin("F:\MULTIMEDIA\AviSynth 2.5\plugins\undot.dll")
LoadPlugin("F:\Multimedia\AviSynth 2.5\plugins\aWarpSharp.dll")
LoadPlugin("F:\Multimedia\AVIsynth 2.5\plugins\Decomb521VFR1.3.dll")
mpeg2source("d2v.d2v",cpu=6,iPP=false)
Telecide(order=1,guide=0,post=2,back=1,bthresh=40,nt=20,ovr="over.txt",show=false).Decimate(mode=2,vfrStats="vfrStats.txt",show=false)
Levels(last, 16,1.0,235, 0,255, false)
aWarpSharp(8.0, 2, 0.5, 2)
Undot()
crop(8,0,704,480)

Well, DGDecode and Undot are done. I've also got regular Decomb.dll so if Decomb521VFR1.3.dll is a derivative it shouldn't take too much work. That leaves aWarpSharp, which would probably be pretty handy. I'll see what I can do and let you know.

Originally Posted by bobololo

Wouldn't this avs script work:

AviSource("uncompressed-huge-clip.avi")

and give this script to x264 ? This would use avisynth internal avi parser that supports open-DML.

I was going to suggest that too, but akupenguin's idea would be easier and save huge amounts of disk space. But does piping between 32-bit and 64-bit processes work? If it doesn't you could always drop the pipe and do it in 2 separate steps.

Edit: Whoops, avisynth64 is also missing levels. Surely that can't be too hard to fix.

[omion]

Originally Posted by bobololo

Wouldn't this avs script work:

AviSource("uncompressed-huge-clip.avi")

and give this script to x264 ? This would use avisynth internal avi parser that supports open-DML.

It would work, but I can't figure out how to have the 32- and 64-bit versions of avisynth installed on my computer at the same time. Is there any way to specify which avisynth.dll each program should use?

It would also take up a lot of space, but the good thing about it is that I'd only have to run the filters once. Now that I'm piping between programs I can see that the avisynth takes ~20% of the processing power of actually encoding with x264. It would be nice to only need to do that once.

@squid_80:
Is the "cpu=n" part of DGDecode working? I used one version of your build that didn't have that working yet.

I certainly would appreciate a 64-bit version of at least one of the postprocessing filters. I don't want to appear pushy, though. Whatever you can do is great.

Also, piping between 32- and 64-bit processes works like a charm. The pipes are simply bytes passed from one program to another, so there's nothing that bit-depth would affect (as far as I understand it... either way, it works )

The version of Decomb that I have is indeed based off Decomb 5.2.1, which adds the ability to write a Matroska-readable timecode file when decimating. The source is here. As I said, I'm not using the VFR-specific parts of it right now, so there's no rush.

[bill_baroud]

Is there any way to specify which avisynth.dll each program should use?

Place the avisynth.dll you want to use in the same directory as the program that will use it. It will be loaded first (if the current directory "." is the first in the path of course )

[foxyshadis]

omion's levels() is just coloryuv(levels="TV->PC"), if that helps. It'd probably be faster that way too.

[omion]

Originally Posted by foxyshadis

omion's levels() is just coloryuv(levels="TV->PC"), if that helps. It'd probably be faster that way too.

Well, mostly. My levels() function doesn't affect the chroma channels, whereas the coloryuv(levels) function does. Because of this, the levels() function I use is actually a bit faster (140 vs 142 fps)

Although I suppose since that is exactly what I'm doing (scaling from TV to PC) then I should also scale the chroma. But the difference between the two is so small that I don't think it matters.

[ edit: Ack! Mine DOES affect the chroma channels! That means it does it wrong, since the chroma channels' valid range is [16,240], NOT [16,235]. I guess I'm changing to coloryuv() anyway. Thanks! I never would have caught that! ]

[squid_80]

Originally Posted by omion

It would work, but I can't figure out how to have the 32- and 64-bit versions of avisynth installed on my computer at the same time. Is there any way to specify which avisynth.dll each program should use?

Well, 64-bit programs should pick up the 64-bit dll and 32-bit programs should pick up the 32-bit dll automatically. The 64-bit one lives in windows\system32 while the 32-bit dll lives in windows\syswow64. So there shouldn't be any problem - at least I've never had one and I switch programs all the time to make sure the 64-bit stuff gives identical output.

It would also take up a lot of space, but the good thing about it is that I'd only have to run the filters once. Now that I'm piping between programs I can see that the avisynth takes ~20% of the processing power of actually encoding with x264. It would be nice to only need to do that once.

@squid_80:
Is the "cpu=n" part of DGDecode working? I used one version of your build that didn't have that working yet.

No, the postprocessing was all inline assembly which was a real headache so I just dropped it. Even if I was to translate it to intrinsics it would become much slower (ever tried the resizing functions with the 64-bit build?).

I certainly would appreciate a 64-bit version of at least one of the postprocessing filters. I don't want to appear pushy, though. Whatever you can do is great.

There seems to be a big lack of interest in making 64-bit stuff. Part of that's probably my fault for not getting the avisynth core up to scratch. You could always have a go yourself seeing as you seem to know your way around a compiler...

Also, piping between 32- and 64-bit processes works like a charm. The pipes are simply bytes passed from one program to another, so there's nothing that bit-depth would affect (as far as I understand it... either way, it works )

Good to know, I thought there might be a problem getting a 32-bit process to talk to a 64-bit process but I guess there's windows magic in between. If only it were as easy to get 64-bit programs to use 32-bit dlls...

The version of Decomb that I have is indeed based off Decomb 5.2.1, which adds the ability to write a Matroska-readable timecode file when decimating. The source is here. As I said, I'm not using the VFR-specific parts of it right now, so there's no rush.

Thanks, I'll see what I can do.

Originally Posted by foxyshadis

omion's levels() is just coloryuv(levels="TV->PC"), if that helps. It'd probably be faster that way too.

Ah, coloryuv is implemented so that solves that problem. I still want levels though, it's good for removing tv station watermarks... but I need overlay for that as well

[Wilbert]

edit: Ack! Mine DOES affect the chroma channels! That means it does it wrong, since the chroma channels' valid range is [16,240], NOT [16,235]. I guess I'm changing to coloryuv() anyway. Thanks! I never would have caught that!

Hmm, i need to look at that.

[omion]

Originally Posted by Wilbert

Hmm, i need to look at that.

Well, the problem was me. I thought that levels() only affected the luma channel, when it actually affects all the channels. And since chroma and luma have different valid ranges, my command cuts off the most saturated reds and blues. So I'm now using colorYUV() as foxyshadis mentioned, which does everything right.

@squid_80: I haven't actually tried to run 32- and 64-bit avisynth at the same time, because I assumed that the .dll files would get in the way. Although now it seems obvious that Windows would only allow 32-bit programs to use 32-bit .dll files. I'll give it a try!

[Wilbert]

Well, the problem was me. I thought that levels() only affected the luma channel, when it actually affects all the channels.

I know, but that means i should correct the documentation, because it says that Levels(...) = ColorYUV(levels="TV->PC") and so on