VirtualDubMod crashes at certain output resolutions

[Kylenol]

I'm using StaxRip (the latest beta version) to encode on BootCamp (I have Windows Vista Ultimate SP1 32-bit installed, btw). And for some reason, my encoding will freeze on the at the start of 2nd pass IF the output resolution is for instance 512x384.However, if I encode the output at a higher output resolution such as 624x352 to encode HD, it will encode successfully. I've tried re-installing XviD, VirtualDubMod, AVISynth and installed older versions of StaxRip but nothing will help. I never had this problem on my older PC on Windows XP SP3. I also don't have a problem with this using a different XviD build such as xvid_encraw using MeGUI.

The only way I it SEEMS to work better is when I use an older version of VirtualDubMod, but even then at the end of 2nd pass it will freeze at 99%. If I use the current version of VirtualDubMod, and clear the stats file from another encode, I have found that it works in the same way as an older version of VirtualDubMod would.

I would like to encode successfully, without any crashes!

[Firebird]

Don't use VirtualDubMod. Use VirtualDub.

[Kylenol]

Don't use VirtualDubMod. Use VirtualDub.

It still crashes with VirtualDub.

[Kylenol]

Here's the crash report


VirtualDub crash report -- build 28346 (release)
--------------------------------------

Disassembly:
772566a0: c3 ret
772566a1: 07 pop es
772566a2: 0f8571950000 jnz 7725fc19
772566a8: 8d43f8 lea eax, [ebx-08h]
772566ab: 80780705 cmp byte ptr [eax+07h], 05h
772566af: 0f844d950000 jz 7725fc02
772566b5: f640073f test [eax+07h], 3fh
772566b9: 0f8451950000 jz 7725fc10
772566bf: 3bc7 cmp eax, edi
772566c1: 894510 mov [ebp+10h], eax
772566c4: 0f84b9950000 jz 7725fc83
772566ca: 807bff05 cmp byte ptr [ebx-01h], 05h
772566ce: 0f8458950000 jz 7725fc2c
772566d4: 8b4510 mov eax, [ebp+10h]
772566d7: f6400780 test [eax+07h], 80h
772566db: 0f84530e0000 jz 77257534
772566e1: 8bd3 mov edx, ebx
772566e3: 8bce mov ecx, esi
772566e5: e823000000 call 7725670d
772566ea: 84c0 test al, al
772566ec: 0f84420e0000 jz 77257534
772566f2: 803d8003fe7f00 cmp byte ptr [7ffe0380], 00h
772566f9: 0f85ad950000 jnz 7725fcac
772566ff: b001 mov al, 01h
77256701: 5e pop esi
77256702: 5f pop edi
77256703: 5b pop ebx
77256704: 5d pop ebp
77256705: c20c00 ret 000c
77256708: 90 nop
77256709: 90 nop
7725670a: 90 nop
7725670b: 90 nop
7725670c: 90 nop
7725670d: 8bff mov edi, edi
7725670f: 55 push ebp
77256710: 8bec mov ebp, esp
77256712: 83ec28 sub esp, 28h
77256715: 53 push ebx
77256716: 56 push esi
77256717: 57 push edi
77256718: 8d7af8 lea edi, [edx-08h]
7725671b: 807f0705 cmp byte ptr [edi+07h], 05h
7725671f: 0f8415c20100 jz 7727293a
77256725: 8b07 mov eax, [edi]
77256727: 8bf7 mov esi, edi
77256729: c1ee03 shr esi, 03h
7725672c: 33f0 xor esi, eax
7725672e: 333544422b77 xor esi, [772b4244]
77256734: 897dfc mov [ebp-04h], edi
77256737: 33f1 xor esi, ecx
77256739: 8b4604 mov eax, [esi+04h] <-- FAULT
7725673c: 8945f4 mov [ebp-0ch], eax
7725673f: c6470780 mov byte ptr [edi+07h], 80h
77256743: c6470600 mov byte ptr [edi+06h], 00h
77256747: 8b5e08 mov ebx, [esi+08h]
7725674a: 8b4e0c mov ecx, [esi+0ch]
7725674d: 895de0 mov [ebp-20h], ebx
77256750: 83c301 add ebx, 01h
77256753: 894de4 mov [ebp-1ch], ecx
77256756: 83d101 adc ecx, 01h
77256759: 81e3ff7f0000 and ebx, 00007fff
7725675f: 663b5e14 cmp ebx, [esi+14h]
77256763: 0f84b3040000 jz 77256c1c
77256769: 8b45e0 mov eax, [ebp-20h]
7725676c: 8b55e4 mov edx, [ebp-1ch]
7725676f: 0facd010 shrd eax, edx, 10h
77256773: 66894708 mov [edi+08h], ax
77256777: 2b7df4 sub edi, [ebp-0ch]
7725677a: 33c0 xor eax, eax
7725677c: c1ef03 shr edi, 03h
7725677f: 0fa4f810 shld eax, edi, 10h
77256783: c1e710 shl edi, 10h
77256786: c1ea10 shr edx, 10h
77256789: 0bdf or ebx, edi
7725678b: 0bc8 or ecx, eax
7725678d: 8b06 mov eax, [esi]
7725678f: c745f801000000 mov dword ptr [ebp-08h], 00000001
77256796: 8930 mov [eax], esi
77256798: 8b45e0 mov eax, [ebp-20h]
7725679b: 8b55e4 mov edx, [ebp-1ch]
7725679e: 8d db 8dh
7725679f: 7e db 7eh

Built on KOS-MOS on Tue Feb 12 22:09:24 2008 using compiler version 1400

Windows 6.0 (Windows Vista x86 build 6002) [Service Pack 2]

EAX = 81818181
EBX = 086d4900
ECX = 07f50000
EDX = 086d4900
EBP = 0012deac
ESI = b9187969
EDI = 086d48f8
ESP = 0012de78
EIP = 77256739
EFLAGS = 00210286
FPUCW = ffff027f
FPUTW = ffffffff

Crash reason: Access Violation

Crash context:
An out-of-bounds memory access (access violation) occurred in module 'ntdll'...

...reading address B918796D.

Pointer dumps:

EBX 086d4900: 07f500c4 085b72d0 00000000 00000000 00000000 00000000 00000000 00000000
ECX 07f50000: b93dc2a0 0100b47c ffeeffee 00000000 07f80010 07f500a8 07f50000 07f50000
EDX 086d4900: 07f500c4 085b72d0 00000000 00000000 00000000 00000000 00000000 00000000
EDI 086d48f8: 81818181 82824563 07f500c4 085b72d0 00000000 00000000 00000000 00000000
ESP 0012de78: 00000000 07f50000 086d4900 07f56f08 01010118 0012ddc4 00000000 0012df0c
0012de98: 771f99fa 00c1bbdd fffffffe 77257512 086d48f8 0012dec0 772566ea 086d4900
0012deb8: 07f55e00 07f55d40 0012ded4 76529a26 07f50000 00000000 086d48f8 0012df1c
0012ded8: 07e884ca 07f50000 00000000 086d4900 07f55e00 ffffd800 07f55d40 07f563f0
EBP 0012dea8: 086d48f8 0012dec0 772566ea 086d4900 07f55e00 07f55d40 0012ded4 76529a26
0012dec8: 07f50000 00000000 086d48f8 0012df1c 07e884ca 07f50000 00000000 086d4900
0012dee8: 07f55e00 ffffd800 07f55d40 07f563f0 ffffd800 00000002 07f55d40 07f55d40
0012df08: 0012da9c 0012ee10 07e88c20 07eb70f8 ffffffff 07f56600 07e3c448 086d4900

Thread call stack:
77256739: ntdll!RtlFreeHeap [771f0000+66670+c9]
77257512: ntdll!RtlEnumerateGenericTableWithoutSplaying [771f0000+66dc2+750]
772566ea: ntdll!RtlFreeHeap [771f0000+66670+7a]
76529a26: kernel32!HeapFree [764e0000+49a12+14]
07e884ca: xvidcore!xvid_plugin_dump [07e10000+6bff4+c4d6]
07e3c448: xvidcore!xvid_encore [07e10000+2acc8+1780]
07e3b1d0: xvidcore!xvid_encore [07e10000+2acc8+508]
07e3af34: xvidcore!xvid_encore [07e10000+2acc8+26c]
07e3acf9: xvidcore!xvid_encore [07e10000+2acc8+31]
03b37134: xvidvfw!DriverProc [03b30000+6e04+330]
76c999cb: USER32!GetWindowLongA [76c80000+19994+37]
004ab09d: VDProjectUI::WndProc()
004baa3e: VDUIFrame::StaticWndProc()
7646d60f: USP10!ScriptGetCMap [76460000+b3e4+222b]
005a192c: _fputwc_nolock()
005a0b12: _mbtowc_l()
005a192c: _fputwc_nolock()
76c9fd72: USER32!GetWindowLongW [76c80000+1f8bf+4b3]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c9fdf3: USER32!GetWindowLongW [76c80000+1f8bf+534]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
7646ed0b: USP10!ScriptTextOut [76460000+ea95+276]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c99d6a: USER32!ReleaseDC [76c80000+19ced+7d]
76c99d75: USER32!ReleaseDC [76c80000+19ced+88]
76c99d75: USER32!ReleaseDC [76c80000+19ced+88]
76c99f8d: USER32!WCSToMBEx [76c80000+19ec1+cc]
76c99f9d: USER32!WCSToMBEx [76c80000+19ec1+dc]
77255dae: ntdll!KiUserCallbackDispatcher [771f0000+65d80+2e]
76c98e7d: USER32!InflateRect [76c80000+18d4f+12e]
76ca06b8: USER32!IsWindow [76c80000+2067b+3d]
76ca03ab: USER32!DispatchMessageW [76c80000+2021c+18f]
7456af85: uxtheme!GetThemeBool [74560000+a886+6ff]
745625a7: uxtheme!000025a7
745625f4: uxtheme!000025f4
76ca0245: USER32!DispatchMessageW [76c80000+2021c+29]
76ca0454: USER32!DefWindowProcW [76c80000+203b4+a0]
76ca041f: USER32!DefWindowProcW [76c80000+203b4+6b]
76ca041f: USER32!DefWindowProcW [76c80000+203b4+6b]
004ba4c3: VDUIFrame::DefProc()
004abaca: VDProjectUI::DubWndProc()
76c999cb: USER32!GetWindowLongA [76c80000+19994+37]
004ab09d: VDProjectUI::WndProc()
004baa3e: VDUIFrame::StaticWndProc()
7723127c: ntdll!RtlImageNtHeader [771f0000+41261+1b]
7725661f: ntdll!RtlAllocateHeap [771f0000+66570+af]
76c9fd72: USER32!GetWindowLongW [76c80000+1f8bf+4b3]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
772566ea: ntdll!RtlFreeHeap [771f0000+66670+7a]
76529a26: kernel32!HeapFree [764e0000+49a12+14]
0058bb72: free()
0058bb91: free()
0058bb91: free()
00483fb3: guiSetTitleW()
0044006c: VDFilterGraphControl::RequeryFormats()
76cb280d: USER32!GetClipboardOwner [76c80000+326ef+11e]
76c99560: USER32!DrawTextExW [76c80000+191ce+392]
76c93c42: USER32!GetWindowDC [76c80000+13ba7+9b]
76ca18ce: USER32!IsDialogMessage [76c80000+21847+87]
76c9fd72: USER32!GetWindowLongW [76c80000+1f8bf+4b3]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c9fdf3: USER32!GetWindowLongW [76c80000+1f8bf+534]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76ca0943: USER32!CallMsgFilterW [76c80000+20801+142]
76c9f860: USER32!GetClipboardFormatNameA [76c80000+1a552+530e]
76c999bf: USER32!GetWindowLongA [76c80000+19994+2b]
76c999cb: USER32!GetWindowLongA [76c80000+19994+37]
768223a6: MSCTF!TF_GetAppCompatFlags [76820000+2361+45]
00476491: DubStatus::StatusTimerProc()
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c991ec: USER32!DrawTextExW [76c80000+191ce+1e]
76cb193e: USER32!DrawStateW [76c80000+31825+119]
76cb1955: USER32!DrawStateW [76c80000+31825+130]
76cb20ab: USER32!DrawStateW [76c80000+31825+886]
76cb20bd: USER32!DrawStateW [76c80000+31825+898]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c999bf: USER32!GetWindowLongA [76c80000+19994+2b]
00476575: DubStatus::StatusMainDlgProc()
76c9fd72: USER32!GetWindowLongW [76c80000+1f8bf+4b3]
76c94a0b: USER32!ActivateKeyboardLayout [76c80000+1478c+27f]
76c949c8: USER32!ActivateKeyboardLayout [76c80000+1478c+23c]
76c9495f: USER32!ActivateKeyboardLayout [76c80000+1478c+1d3]
76c94a0b: USER32!ActivateKeyboardLayout [76c80000+1478c+27f]
76ca0b5a: USER32!NotifyWinEvent [76c80000+20b4d+d]
728b6b9f: COMCTL32!_TrackMouseEvent [72870000+35a6e+11131]
76c9fd72: USER32!GetWindowLongW [76c80000+1f8bf+4b3]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c9fdf3: USER32!GetWindowLongW [76c80000+1f8bf+534]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76ca0943: USER32!CallMsgFilterW [76c80000+20801+142]
0046d842: Dubber::UpdateFrames()
76c98e7d: USER32!InflateRect [76c80000+18d4f+12e]

-- End of report

[LoRd_MuldeR]

Try VirtualDub 1.9.6 (Build 32618) instead of that old version!

Anyway, it seems like the crash originates from Xvid, not from VirtualDub(Mod) itself. So probably Xvid is to blame:

Thread call stack:
77256739: ntdll!RtlFreeHeap [771f0000+66670+c9]
77257512: ntdll!RtlEnumerateGenericTableWithoutSplaying [771f0000+66dc2+750]
772566ea: ntdll!RtlFreeHeap [771f0000+66670+7a]
76529a26: kernel32!HeapFree [764e0000+49a12+14]
07e884ca: xvidcore!xvid_plugin_dump [07e10000+6bff4+c4d6]
07e3c448: xvidcore!xvid_encore [07e10000+2acc8+1780]
07e3b1d0: xvidcore!xvid_encore [07e10000+2acc8+508]
07e3af34: xvidcore!xvid_encore [07e10000+2acc8+26c]
07e3acf9: xvidcore!xvid_encore [07e10000+2acc8+31]
03b37134: xvidvfw!DriverProc [03b30000+6e04+330]