XVID crash w/ multi-core CPU and 2-pass encode

[scooby108]

I am having problems encoding with AutoGK on a new computer. VirtualDubMod keeps crashing either right away or when it reaches 99%. I think I have figured out why this is happening with AutoGK.

My setup:
64bit Intel Xeon E5520 Quad-core WITH Hyperthreading ENABLED
Windows 7 Enterprise 64-bit
4GB RAM
Latest stable releases of AutoGK and Gordian Knot

I believe the problem is with the number of threads used to encode with XVID. AutoGK automatically detects the number of threads to use at each encoder pass. And each time it sets the number of threads equal to the number of virtual CPUs. Because I'm using a quad-core with HT enabled, this works out to be 8 virtual CPUs. VirtualDubMod crashes whenever threads is set to 8.

By opening the XVID encoder settings before/after each pass and changing "threads" to 4 I am able to get AutoGK to limp along far enough to have it build its AviSynth script. Once I get the script I can set threads back to 4 and do a manual encode using that script. Single-pass encodings work fine if I can set the number of threads to 4.

Can there be a cap to the number of threads or a registry setting to limit # of threads? I read that you shouldn't need any more than 3-4 threads when encoding with XVID anyway because after that the gain is marginal. Is that true?

[nurbs]

I read that you shouldn't need any more than 3-4 threads when encoding with XVID anyway because after that the gain is marginal. Is that true?

Don't know how you can limit the number, but this is true. Xvids threading is pretty inefficient.

[netmask]

My setup:
64bit Intel Xeon E5520 Quad-core WITH Hyperthreading ENABLED
Windows 7 Enterprise 64-bit
4GB RAM
Latest stable releases of AutoGK and Gordian Knot

Every box running Windows 7 64 bit I've had any experience with has had a minimum of 6GB RAM. Maybe it's a memory problem?

[LoRd_MuldeR]

Every box running Windows 7 64 bit I've had any experience with has had a minimum of 6GB RAM. Maybe it's a memory problem?

Unless he is using 64-Bit Xvid (which would also require 64-VirtualDub), he is using 32-Bit Xvid. So there will be a 2 GB per-process memory limit anyway!

No matter how much RAM you have or how much memory your OS can address, the per-process limit for 32-Bit processes remains ;)

[netmask]

Unless he is using 64-Bit Xvid (which would also require 64-VirtualDub), he is using 32-Bit Xvid. So there will be a 2 GB per-process memory limit anyway!

No matter how much RAM you have or how much memory your OS can address, the per-process limit for 32-Bit processes remains ;)

But the OP says he using 64 bit.

[LoRd_MuldeR]

But the OP says he using 64 bit.

He says he's using 64-Bit Windows. He doesn't say he's using 64-Bit Xvid. That's a fundamental difference ;)

But he mentions AutoGK/VirtualDubMod, which, as far as I know, doesn't have a 64-Bit version available. So I must assume he's using 32-Bit Xvid.

And as said before: Even on 64-Bit Windows with 4 GB of RAM or more, each 32-Bit process is limited to 2 GB of memory :sly:

My advice for his problem would be: Instead of using the discontinued/outdated VirtualDubMod he should update to an up-to-date VirtualDub!

[scooby108]

He says he's using 64-Bit Windows. He doesn't say he's using 64-Bit Xvid.

True indeed.

My advice for his problem would be: Instead of using the discontinued/outdated VirtualDubMod he should update to an up-to-date VirtualDub!

I have the latest stable VirtualDub (1.9.8) that I use for AVI processing, however, I cannot use it this case since AutoGK is converting from MPEG-2 sources. I must use VirtualDubMod for its MPEG-2 capabilities. :(

And I can use the scripts created by AutoGK (.avs and .vcf) to process, encode, and mux, as long as I set the threads back to 4 before I start. Makes having a queue useless since I have to babysit it and manually load the scripts into VDM to get the encoding done.

[LoRd_MuldeR]

You know that up-to-date VirtualDub supports Input-Plugins and that there's a MPEG-2 Plugin (http://fcchandler.home.comcast.net/~fcchandler/Plugins/MPEG2/index.html) a available ???

No need to use an old and outdated fork of VirtualDub for MPEG-2 input ;)

Also, as an alternative to VirtualDub(Mod) and VFW Codecs, you may want to give Avidemux (http://mulder.dummwiedeutsch.de/home/?page=projects#avidemux) a try...

[scooby108]

You know that up-to-date VirtualDub supports Input-Plugins and that there's a MPEG-2 Plugin (http://fcchandler.home.comcast.net/~fcchandler/Plugins/MPEG2/index.html) a available ???

I had no idea that there was a plugin for MPEG-2! Awesomeness!
However it still doesn't help me with AutoGK. I have used many tools for transcoding, but none as quick and easy as AutoGK. It takes 5-10 seconds to add a video to the queue in AutoGK whereas Avidemux takes 1-2 minutes. I really value the speed and simplicity.

Anyway, I tried to plug an up-to-date VirtualDub into my AutoGK install and get an error:
http://forum.doom9.org/attachment.php?attachmentid=10620&stc=1&d=1262716616

I looked at the switches for the GUI and command line versions of VirtualDub and neither of them have the "nowrite" switch.

Also, as an alternative to VirtualDub(Mod) and VFW Codecs, you may want to give Avidemux a try...

I'm not looking for a replacement for AutoGK, just an improvement. If I have to do things the hard (pronounced "slightly more time-intensive and not automatic") way, then I will. I was only hoping for an easy solution. I really like Avidemux, BTW. :)

Thanks for the help, MuldeR.

[LoRd_MuldeR]

As far as I know, AutoGK builds on top of Gordian Knot. And Gordian Knot uses VirtualDubMod. I don't think it can use up-to-date VirtualDub.

So if VirtualDubMod fails, your only option will be droppoing AutoGK/VDubMod and move the a less antiquated tool chain.

But most important: Did you try to encode manually from an up-to-date VirtualDub? Does it still crash or is the problem fixed now ???

[scooby108]

But most important: Did you try to encode manually from an up-to-date VirtualDub? Does it still crash or is the problem fixed now ???

I direct you to my other comment:

And I can use the scripts created by AutoGK (.avs and .vcf) to process, encode, and mux, as long as I set the threads back to 4 before I start. Makes having a queue useless since I have to babysit it and manually load the scripts into VDM to get the encoding done.

By VDM I meant VirtualDubMod. So, yes. The scripts work fine, VirtualDubMod works fine, VirtualDub works fine, XVID works fine, all fine EXCEPT when using 8 threads to encode.

Maybe Len0x will run across this post and put a option for the number of threads into AutoGK. I believe AutoMKV has the option and stores it with each item in the queue. And Avidemux has the multithreading option at least when using XVID or x264 (Disable, Auto, Custom). Something like that in AutoGK would be great.

I guess I'll have to go with Avidemux. That's what I use most of the time on Linux anyway. :D

Thanks again, MuldeR.

[88keyz]

Seems like a stupid suggestion but switching to a single threaded version of Xvid will help. I was having the same problem with AutoGK after upgrading to an i7 processor and I solved it by going back to an older version of Xvid that didn't support multi-threaded encoding. Strangely thanks to Intel Turbo Tech the single threaded encoding isn't that much slower on my i7-860 than the multi-threaded Xvid. However it doesn't crash VirtualDubMod and your encodes will actually finish, so I guess in that sense its actually faster.

Xvid_1.1.3-27042008 (http://rapidshare.com/files/332696017/Xvid_1.1.3-27042008.exe)

On a side note, if anyone has Koepi's 1.1.3 VAQ build dated 27 Apr 2008 (filename Xvid-1.1.3-VAQ-27042008.exe) and would care to post it for download I would be grateful. I looked high and low for it on the web and all download links point back to Koepi's site and he has removed it for download.

[88keyz]

A quick follow-up. I found that Jawor's 1.3.-127 build works just fine with up to 8 threads in AutoGK. Not sure what the difference is but since I switched to this build I have not had a crash in AutoGK.

Get Jawor's Xvid builds here: Jawor's Xvid Binaries (http://jawormat.republika.pl/xvid.html)

or here: Jawor's Xvid 1.3.-127 (http://rapidshare.com/files/336852487/Xvid_1.3.127-06122009.exe)

[szabi]

Seems like a stupid suggestion but switching to a single threaded version of Xvid will help. I was having the same problem with AutoGK after upgrading to an i7 processor and I solved it by going back to an older version of Xvid that didn't support multi-threaded encoding. Strangely thanks to Intel Turbo Tech the single threaded encoding isn't that much slower on my i7-860 than the multi-threaded Xvid. However it doesn't crash VirtualDubMod and your encodes will actually finish, so I guess in that sense its actually faster.

I hope it will help me, because today I got a problem during encoding vdub and XviD 1.2.2.
I tried to use newer vdubmod (http://www.virtualdub-fr.org/Fichiers/VDMOD/VirtualDubMOD.exe), also the newest vdub (v1.9.9) but I got crash in both case.
I will try this thread control tomorrow.
Thnx for the idea. :)

EDIT: Ok it worked fine. :)

Bye
szabi