Problem with batch encoding (x264-64 and avs2yuv)

[blubb444]

Hi,

got the following problem:
I've got 47 DV-AVI files (from a friend's Hi8 home video tapes, captured through the FireWire output of a rented Sony GV-D200), most of them are about 1 hour long and take up a lot of disk space (~600GB) so I want to batch encode them with x264. Since they're interlaced, I've batch created some avs scripts which deinterlace them from 50i to 50p using the yadif plugin from here: http://avisynth.org.ru/yadif/yadif.html.
I'm very new to this so maybe my script is faulty? Sure I could deinterlace with VirtualDub but I don't want to write huge intermediate lossless files. Or encode as interlaced, I tried it and it's rather slow and makes rather high bitrates and can only be played correctly in VLC for some reason, WMP then ignores the SAR I set. Plus according to http://mewiki.project357.com/wiki/X264_Settings#tff it's not recommended to use interlacing in x264. And the friend of mine has even less of an idea about encoding/decoding than I have so I want to "hard-encode" the deinterlacing, 25p judders to much so I deinterlace to 50p, much smoother. With my settings the files are around 5-6 MBit/s which is fine for me, better than 28 MBit/s DV-AVI.
The scripts look like this ("001.AVI" goes up until "047.AVI"):
AviSource ("001.AVI")
ConvertToYV12
LoadCPlugin("yadif.dll")
Yadif(1,0,false,3)

Now I found out with some googling that x264 64bit doesn't work with avs and that I need avs2yuv if I don't want to sacrifice the speed gain of 64bit x264 (I get ~40-43fps now, with 32bit and loading the avs directly and otherwise same params it's only ~35-37fps and that's a lot considering ~50hrs of video). So I downloaded that from here: http://akuvian.org/src/avisynth/avs2yuv/ and ran a .bat file which looks like that (it seeks to a certain frame and encodes only a certain number to remove the gray parts at the beginning and end of the files and also crops black bars around the edges):
avs2yuv.exe a015.avs - | x264 --seek 482 --frames 256552 --preset fast --crf 21 --tune film --keyint
500 --min-keyint 50 --bframes 8 --rc-lookahead 150 --chroma-qp-offset 2 --video-filter crop:8,4,12,10 --sar
59:54 --threads 8 -o mp4\015.264 - --demuxer y4m
avs2yuv.exe a016.avs - | x264 --seek 604 --frames 274982 --preset fast --crf 21 --tune film --keyint
500 --min-keyint 50 --bframes 8 --rc-lookahead 150 --chroma-qp-offset 2 --video-filter crop:8,4,12,10 --sar
59:54 --threads 8 -o mp4\016.264 - --demuxer y4m
avs2yuv.exe a017.avs - | x264 --seek 404 --frames 278846 --preset fast --crf 21 --tune film --keyint
500 --min-keyint 50 --bframes 8 --rc-lookahead 150 --chroma-qp-offset 2 --video-filter crop:8,4,12,10 --sar
59:54 --threads 8 -o mp4\017.264 - --demuxer y4m
avs2yuv.exe a018.avs - | x264 --seek 344 --frames 279132 --preset fast --crf 21 --tune film --keyint
500 --min-keyint 50 --bframes 8 --rc-lookahead 150 --chroma-qp-offset 2 --video-filter crop:8,4,12,10 --sar
59:54 --threads 8 -o mp4\018.264 - --demuxer y4m
...
Now here comes the problem: After one encoding job is done, I get an error message Output error: wrote only 590144 of 622080 bytes and the batch stops (the first number varies a little, also got 589916/590040... but the second one always seems to be 622080). The resulting .264 is perfectly fine though, when I mux into mp4 or mkv it plays completely from beginning to end. I googled around a lot for that too, and all I found was that someone here had a different avs2yuv.exe which I downloaded (can't remember the exact thread/link, file size is 84992, last modified Sep 23, 2010, 17:56:28). With that version it apparently continues with the batch despite giving the same error after each encode but after 3 or so encodes avs2yuv crashes too (the Windows message "avs2yuv doesn't respond...") and thus halts the batch again.
Any (newbie-friendly) ideas? I'm using the newest x264 from x264.nl, CPU is a Q9300 with 4x2.5GHz, 8GB RAM, Win7 Home Premium 64, AviSynth 2.58

[Asmodian]

I use avs4x264mod (http://forum.doom9.org/showthread.php?t=162656), it is similar to avs2yuv but would require a somewhat different command line.

I am not sure this is related your problem though.

[blubb444]

OK thanks, will give that a try after the current encode is over (1-2 hours left because I'm currently doing other CPU-heavy stuff which decrease the encoding rate) and report back then.

[Atak_Snajpera]

i think you should move --seek command to avs2yuv.exe section.

[Asmodian]

Ah yes! I missed that seek command, I am pretty sure your error has something to do with that. Good catch Atak_Snajpera.

Moving seek to avs2yuv and removing the --frames will allow x264 to just encode everything it gets from avs2yuv. This should stop either program from getting confused but you won't get a % completed during encoding. If you need % done you can leave the --frames in but make sure it is the total number of frames sent by avs2yuv so the data doesn't stop early and there are no frames sent to x264 but not encoded.

[blubb444]

Sorry for not replying earlier, but I was out of house... I now set up the batch file to use avs2x264mod. This somehow automatically moved the --seek command to avs but I don't know what it did to the frames command, because it's missing now from the x264 options but it is displayed correctly on the "current encoding status" of x264:

L:\Multimedia\Videos\Video8\*NAME REMOVED*>avs4x264mod --x264-binary "L:\Multimedia\Vide
os\Video8\*NAME REMOVED*\x264.exe" --seek 454 --frames 184900 --preset fast --crf 21 --t
une film --keyint 500 --min-keyint 50 --bframes 8 --rc-lookahead 150 --chroma-qp
-offset 2 --video-filter crop:8,4,12,10 --sar 59:54 --threads 8 -o mp4\028.264
a028.avs
avs [info]: Avisynth version: 2.58
avs [info]: Video colorspace: YV12
avs [info]: Video resolution: 720x576
avs [info]: Video framerate: 50/1
avs [info]: Video framecount: 185414
avs4x264 [info]: Convert "--seek 454" to internal frame skipping
avs4x264 [info]: "L:\Multimedia\Videos\Video8\*NAME REMOVED*\x264.exe" - --preset fast -
-crf 21 --tune film --keyint 500 --min-keyint 50 --bframes 8 --rc-lookahead 150
--chroma-qp-offset 2 --video-filter crop:8,4,12,10 --sar 59:54 --threads 8 -o mp
4\028.264 --frames 184900 --fps 50/1 --input-res 720x576 --input-csp i420
raw [info]: 720x576p 59:54 @ 50/1 fps (cfr)
crop [info]: cropping to 700x562
x264 [info]: using SAR=59/54
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.1 Cach
e64
x264 [info]: profile High, level 3.1
[12.8%] 23662/184900 frames, 37.46 fps, 4517.13 kb/s, eta 1:11:43

And I also wonder if it has moved the --seek command to after the yadif step so that the doubled framerate is respected. Well I'll see in about 1 hour when encoding will be done (muxing will take some extra time)

I'd also like to use this opportunity to ask some slightly OT questions:
Was it a good decision to use chroma qp offset? I did so because according to Wikipedia's Hi8 article:
Both Hi8 and S-VHS were officially rated at a luminance resolution of 420 horizontal TV/lines (560×480 in today's digital terms), a vast improvement from their respective base formats of 240 lines and roughly equal to laserdisc quality. Chroma resolution for both remained unchanged, at approximately 30 lines horizontal.
And since the tapes are PAL, I guess the luma resolution is even a bit higher(?). This raises the additional question: Is there an option in x264 to use an even lower chroma subsampling, like let's say setting only one U and V value for a 4x4, 6x6... block?

[sneaker_ger]

Is there an option in x264 to use an even lower chroma subsampling, like let's say setting only one U and V value for a 4x4, 6x6... block

That's not possible. Only i420, i422, i444, rgb are available.

[blubb444]

That's not possible. Only i420, i422, i444, rgb are available.

OK then, there's apparently nothing to do about it...
I'm now finished with encoding (without error) and muxing the file (mp4box), and it plays fine in VLC. But for some strange reason, MPC-HC x64 now reports that the file is 38 hours long? Windows Explorer, WMP-32, WMP-64 and VLC all report the correct length of 1:01:38. I've muxed exactly the same way as before and MPC-HC uses LAV splitter and decoder, strange...

[Asmodian]

And I also wonder if it has moved the --seek command to after the yadif step so that the doubled framerate is respected.

avs4x264mod will not have moved seek into the avs but will just start sending frame 454 as frame 1. I think it doesn't actually look inside the avs script at all and just treats it as raw video while getting basic information through the avisynth api (resolution, fps, color space, etc.).

Was it a good decision to use chroma qp offset?

I personally wouldn't use a chroma qp offset unless you needed more compression and liked the look of a higher chroma qp more than the higher crf needed for the same increase in compression. I have been doing SVHS capture and compression, not Hi8, but even though the chroma is really bad that is no reason to make it even worse. Digitizing analog video isn't exactly coping the source but more trying to get as good a representation of the analog image into the digital file as you can. By setting a chroma offset you are dropping some extra information from the already limited chroma information. Of course if you cannot tell that you did it maybe you don't care about the chroma information that was dropped (it could be mostly not real information, just noise etc.). Also qp and resolution are different concepts, usually I want a lower qp at lower resolutions not higher.

For PAL both luma and chroma will be a bit higher resolution.

Sorry I have no idea why MPC-HC x64 would be reporting wrong file lengths. I would try the 32 bit MPC-HC, no reason to want a 64 bit media player as far as I know?

[blubb444]

I personally wouldn't use a chroma qp offset unless you needed more compression and liked the look of a higher chroma qp more than the higher crf needed for the same increase in compression.
Well yeah, of course I played around with the settings before and I personally find crf 21 + chroma offset better than crf 22. Sure, even at crf 21 fast there are some minor artefacts, but those are almost only visible on paused video. I got my PC hooked up to a 46" Sharp LED TV which has quite high brightness and contrast, and I'm sitting 2m away from it. The friend of mine for whom I captured the videos will watch on a smaller TV and sit further away, so I guess that's OK.

I have been doing SVHS capture and compression, not Hi8, but even though the chroma is really bad that is no reason to make it even worse. Digitizing analog video isn't exactly coping the source but more trying to get as good a representation of the analog image into the digital file as you can. By setting a chroma offset you are dropping some extra information from the already limited chroma information. Of course if you cannot tell that you did it maybe you don't care about the chroma information that was dropped (it could be mostly not real information, just noise etc.).

I fully understand what you're saying there, but it's also a quality/size question. Sure one could capture, let's take an intentionally exaggerated example, Super 8 film to 32bit RGB 4K video (with appropriate HDR equipment), but thanks to the high noise level of the analog signal (both in terms of resolution and dynamic range), the tiny quality gain over a normal 4:2:0 1024x768 capture wouldn't justify the huge size difference, at least for me.

Also qp and resolution are different concepts, usually I want a lower qp at lower resolutions not higher.
Yeah, that's what I'm thinking too therefore my question about lower chroma subsampling. But since that not possible yet... maybe that'd be something for "h.265", setting U and V resolutions completely independently from Y, or maybe even adaptive variable subsampling, but I don't know if the overhead would kill any benefits from that (I'm not that deep into the internals)

For PAL both luma and chroma will be a bit higher resolution.

Sorry I have no idea why MPC-HC x64 would be reporting wrong file lengths. I would try the 32 bit MPC-HC, no reason to want a 64 bit media player as far as I know?

Yup, 32-bit MPC detects it fine. The reason for using x64 is..well.. I don't want to start a big discussion here but I'm personally starting to view 32bit PC software as legacy for the few remaining users. Sure it's compatible to 64bit systems, but there's a lot of potential speedup wasted just to stay compatible, I guess. For example on test runs, x264-64 is up to 30% faster for me than 32bit. So I figured the same would apply to decoding "advanced" stuff like h.264, seeking can actually take a while, especially when I once encoded to open-gop 16 bframes it took a few seconds to start playing again. I didn't measure exactly how long the 32 and 64bit version take for that task, but I do know that when playing intense stuff like 4K h.264, the 64bit version of WMP (haven't tested for MPC-HC yet) uses considerably less CPU than the 32bit version.
EDIT: Did some testing with the same 4K h.264 file, with the following results (CPU time reported by Windows task manager):
MPC-HC (64 vs 32): 290s vs 303s (32bit version is 4,5% slower)
WMP (64 vs 32): 282s vs 289s (32bit version is 2,5% slower)
So the difference isn't as big as I remembered, maybe because I installed some new codecs meanwhile... (previously the 32bit WMP even dropped frames on another 4K vid while 64bit played it fine)