ffmpeg FFV1 vs ffdshow FFV1

[Dogway]

I normally use ffdshow in Vdub for encoding FFV1, but Ive been having memory issues due to a heavy avs script. So I tried a GUI-less method, encoding through ffmpeg, I tried the next code:
ffmpeg -i input.avs -vcodec ffv1 -g 24 -an "ouput.avi"

The funny thing is it differs from the ffdshow encode in size (higher) and quality (slightly lower, but it does), it was also a bit slower. Can anyone help me to configure ffmpeg to encode as ffdshow's one? That is CABAC, context SMALL. Or a command line for ffdshow?

Another alternative would be a less hungry codec in terms of RAM, but also in size.

[Dark Shikari]

No, that's not CABAC, CABAC would be with -coder 1.

[poisondeathray]

The funny thing is it differs from the ffdshow encode in size (higher) and quality (slightly lower, but it does)


How can there be a difference in quality ? How are you measuring this ?

Are you using YV12 for both ?

[Dogway]

that's why its funny, and the ffmpeg one seems to be the one failing, I tested with ssim.
Im gonna try with -coder 1 now.

But still Im unsure if Im gaining something encoding through ffmpeg instead of ffdshow, any good alternative for low RAM consumption?

[GodofaGap]

It's Avisynth that's consuming your memory, not VirtualDub or ffmpeg. Did you try setmemorymax()?

[Dogway]

@GodofaGap: sure, Im aware of that, thus Im trying to lower memory in other areas as Im not able to optimize my avs script any further. I tried setmemorymax(512), it encoded 10000 frames, then I changed it to 256 and only encoded 3400. So now Im trying with 768, which will essentially make available all my RAM (3Gb minus 500Mb).

-coder 1 worked and compressed as much as ffdshow did, just much slower encoding. Thanks for the flag, ffmpeg documentation is very messy.

[poisondeathray]

I'd be more concerned about it not being lossless. Otherwise you might as well use a lossy format that encodes faster, smaller filesizes, decodes faster (for editing) . You might have an issue somewhere in your workflow

[Dogway]

It failed at 768 too, around 3000 something frames, with this crashlog:

VirtualDub crash report -- build 33848 (release)
--------------------------------------

Disassembly:
100025c0: 5c pop esp
100025c1: 0e push cs
100025c2: d80f fmul dword ptr [edi]
100025c4: 6f outsd
100025c5: 640e push cs
100025c7: e00f loopnz 100025d8
100025c9: 6f outsd
100025ca: 6c insb
100025cb: 0e push cs
100025cc: e80f6f740e call 1e7494e0
100025d1: f0 lock
100025d2: 0f6f7c0ef8 movq mm7, [esi+ecx-08h]
100025d7: eba7 jmp 10002580
100025d9: 83e940 sub ecx, 40h
100025dc: 8d642400 lea esp, [esp+00h]
100025e0: 83c108 add ecx, 08h
100025e3: 3bca cmp ecx, edx
100025e5: 770c ja 100025f3
100025e7: 0f6f440ef8 movq mm0, [esi+ecx-08h]
100025ec: 0fe7440ff8 movntq [edi+ecx-08h], mm0
100025f1: ebed jmp 100025e0
100025f3: 0f6f4c16f8 movq mm1, [esi+edx-08h]
100025f8: 0fe74c17f8 movntq [edi+edx-08h], mm1
100025fd: 2b7514 sub esi, [ebp+14h]
10002600: 2b7d0c sub edi, [ebp+0ch]
10002603: 4b dec ebx
10002604: 0f8526ffffff jnz 10002530
1000260a: 0faef8 sfence
1000260d: 0f77 emms
1000260f: 5f pop edi
10002610: 5e pop esi
10002611: 5b pop ebx
10002612: 8be5 mov esp, ebp
10002614: 5d pop ebp
10002615: c3 ret
10002616: 8b75f8 mov esi, [ebp-08h]
10002619: 8b7dfc mov edi, [ebp-04h]
1000261c: 8b5d1c mov ebx, [ebp+1ch]
1000261f: 8b5518 mov edx, [ebp+18h]
10002622: eb0c jmp 10002630
10002624: 8da42400000000 lea esp, [esp+00]
1000262b: eb03 jmp 10002630
1000262d: 8d4900 lea ecx, [ecx+00h]
10002630: 8bca mov ecx, edx
10002632: 49 dec ecx
10002633: 03ce add ecx, esi
10002635: 83e1c0 and ecx, 0c0h
10002638: eb06 jmp 10002640
1000263a: 8d9b00000000 lea ebx, [ebx+00]
10002640: 668b01 mov ax, [ecx] <-- FAULT
10002643: 83e940 sub ecx, 40h
10002646: 3bce cmp ecx, esi
10002648: 73f6 jnc 10002640
1000264a: 8bc7 mov eax, edi
1000264c: 33c9 xor ecx, ecx
1000264e: f7d8 neg eax
10002650: 83e03f and eax, 3fh
10002653: eb0b jmp 10002660
10002655: 8da42400000000 lea esp, [esp+00]
1000265c: 8d642400 lea esp, [esp+00h]
10002660: 3bc8 cmp ecx, eax
10002662: 7444 jz 100026a8
10002664: 0f6f3c0e movq mm7, [esi+ecx]
10002668: 0fe73c0f movntq [edi+ecx], mm7
1000266c: 83c108 add ecx, 08h
1000266f: ebef jmp 10002660
10002671: eb0d jmp 10002680
10002673: 8da42400000000 lea esp, [esp+00]
1000267a: 8d9b00000000 lea ebx, [ebx+00]
10002680: 0fe7440fc0 movntq [edi+ecx-40h], mm0
10002685: 0fe74c0fc8 movntq [edi+ecx-38h], mm1
1000268a: 0fe7540fd0 movntq [edi+ecx-30h], mm2
1000268f: 0fe75c0fd8 movntq [edi+ecx-28h], mm3
10002694: 0fe7640fe0 movntq [edi+ecx-20h], mm4
10002699: 0fe76c0fe8 movntq [edi+ecx-18h], mm5
1000269e: 0fe7740ff0 movntq [edi+ecx-10h], mm6
100026a3: 0fe77c0ff8 movntq [edi+ecx-08h], mm7
100026a8: 83c140 add ecx, 40h
100026ab: 3bca cmp ecx, edx
100026ad: 772a ja 100026d9
100026af: 0f6f440ec0 movq mm0, [esi+ecx-40h]
100026b4: 0f6f4c0ec8 movq mm1, [esi+ecx-38h]
100026b9: 0f6f540ed0 movq mm2, [esi+ecx-30h]
100026be: 0f db 0fh
100026bf: 6f outsd

Built on Aegis on Fri Dec 24 13:11:24 2010 using compiler version 1400

Windows 5.1 (Windows XP x86 build 2600) [Service Pack 3]

EAX = 53670520
EBX = 000002b0
ECX = 000004c0
EDX = 00000500
EBP = 1d7de93c
ESI = 00000000
EDI = 53746b20
ESP = 1d7de928
EIP = 10002640
EFLAGS = 00010206
FPUCW = ffff0e7f
FPUTW = ffffffff

Crash reason: Access Violation

Crash context:
An out-of-bounds memory access (access violation) occurred in module 'AviSynth'...

...reading address 000004C0.

Pointer dumps:

EAX 53670520: 34383634 31313030 35333030 35353636 33333435 2f2e3131 2f30302f 32302e2d
EDI 53746b20: 98989898 98999999 99999899 9a9a9a9a 9a9b9a9b 9d9d9b9b 9d9d9d9e 9d9d9d9d
ESP 1d7de928: 000002b0 000002b0 00000500 00000000 53746b20 00000500 100047db 53670020
1d7de948: 00000500 00000000 00000000 00000500 000002b0 012b4678 00000500 100048be
1d7de968: 53670020 00000500 00000000 00000000 00000500 000002b0 061c0ec0 012b4678
1d7de988: 1d7dee58 03c94aba 012b4678 53670020 00000500 00000000 00000000 00000500
EBP 1d7de938: 53746b20 00000500 100047db 53670020 00000500 00000000 00000000 00000500
1d7de958: 000002b0 012b4678 00000500 100048be 53670020 00000500 00000000 00000000
1d7de978: 00000500 000002b0 061c0ec0 012b4678 1d7dee58 03c94aba 012b4678 53670020
1d7de998: 00000500 00000000 00000000 00000500 000002b0 1d7dee58 0132eeb0 061c0ec0

Thread call stack:
10002640: AviSynth!00002640
100047db: AviSynth!000047db
100048be: AviSynth!000048be
03c94aba: ffms2!_AvisynthPluginInit2@4 [03c70000+239d0+10ea]
03c954c3: ffms2!_AvisynthPluginInit2@4 [03c70000+239d0+1af3]
7c80d10e: kernel32!LCMapStringW [7c800000+cd48+3c6]
7c80d04c: kernel32!LCMapStringW [7c800000+cd48+304]
7c80d20c: kernel32!CompareStringA [7c800000+d117+f5]
7c80d245: kernel32!CompareStringA [7c800000+d117+12e]
7c91df5a: ntdll!NtWaitForSingleObject [7c910000+df4e+c]
7c92b24b: ntdll!RtlpWaitForCriticalSection [7c910000+1b1bf+8c]
7c80bb64: kernel32!lstrcmpi [7c800000+bb41+23]
10002c6e: AviSynth!00002c6e
10005010: AviSynth!00005010
10005019: AviSynth!00005019
1000a678: AviSynth!avs_release_value [10000000+75d0+30a8]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
10035d19: AviSynth!DllGetClassObject [10000000+da50+282c9]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
049c8cbb: SmoothAdjust-ICC-x86!00008cbb
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
10040778: AviSynth!DllGetClassObject [10000000+da50+32d28]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
04904afa: RepairSSE3!00004afa
77bfc2e3: msvcrt!free [77be0000+1c21b+c8]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
1003b91f: AviSynth!DllGetClassObject [10000000+da50+2decf]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
04922e82: -DePanEstimate!00002e82
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
058d1f90: -DePan!00001f90
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
1003bd40: AviSynth!DllGetClassObject [10000000+da50+2e2f0]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
059014a1: TTempSmooth!000014a1
15b672c7: mvtools2!_AvisynthPluginInit2@4 [15b10000+6340+50f87]
05901159: TTempSmooth!00001159
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
15b6241f: mvtools2!_AvisynthPluginInit2@4 [15b10000+6340+4c0df]
100045ae: AviSynth!000045ae
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
15b195f0: mvtools2!_AvisynthPluginInit2@4 [15b10000+6340+32b0]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
15b186b8: mvtools2!_AvisynthPluginInit2@4 [15b10000+6340+2378]
15b3ae8d: mvtools2!_AvisynthPluginInit2@4 [15b10000+6340+24b4d]
10004631: AviSynth!00004631
1000190b: AviSynth!0000190b
10009dd8: AviSynth!avs_release_value [10000000+75d0+2808]
10009e09: AviSynth!avs_release_value [10000000+75d0+2839]
15b5d5c2: mvtools2!_AvisynthPluginInit2@4 [15b10000+6340+47282]
15b1fff3: mvtools2!_AvisynthPluginInit2@4 [15b10000+6340+9cb3]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
1003b91f: AviSynth!DllGetClassObject [10000000+da50+2decf]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000a8e5: AviSynth!avs_release_value [10000000+75d0+3315]
059e81ee: dfttest!_AvisynthPluginInit2@4 [059d0000+17f10+2de]
059e89f0: dfttest!_AvisynthPluginInit2@4 [059d0000+17f10+ae0]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
1003bd40: AviSynth!DllGetClassObject [10000000+da50+2e2f0]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
1006492e: AviSynth!DllGetClassObject [10000000+da50+56ede]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
1e0bc753: mt_masktools-25!0002c753
100045ae: AviSynth!000045ae
10004631: AviSynth!00004631
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
06ca1c47: dither!00001c47
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
19a93084: aWarpSharp2!00003084
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
1000e2cd: AviSynth!DllGetClassObject [10000000+da50+87d]
7c80b729: kernel32!GetModuleFileNameA [7c800000+b56f+1ba]

-- End of report

in task manager 1700Mb were being used, not that much.
Im going to test Vdub buffer.
This is my first HD source processing, so it is a reason for it consuming memory, but I think it might be a bootleneck inside my script, rather getting out of memory.

@poisondeathray: one of the reasons Im encoding to FFV1 instead of mp4 directly is because x264 takes lot of ram, and speed gets very low, normally Im faster in this 2 stage encoding than directly to mp4.

[poisondeathray]

@poisondeathray: one of the reasons Im encoding to FFV1 instead of mp4 directly is because x264 takes lot of ram, and speed gets very low, normally Im faster in this 2 stage encoding than directly to mp4.

Yes I know... My earlier concern was about your "quality" differences.

How is that possible with a lossless codec in the absence of colorspace differences ? This suggests something else is wrong as well

You can try UT video codec - it's way faster, but ffmpeg doesn't support it

[Dogway]

Ah yess sorry poisondeathray, 2 times in a row! I again forgot Im using addgrainC at sime point, 2 encodes will never be the same.
It might be that, anyway ffdshow is much faster and it doesnt look like any big change in RAM usage.

I have UT codec installed, does it load with avisource, DSS2?
Also has Vdub any problem outputting >4Gb files?

[poisondeathray]

I have UT codec installed, does it load with avisource, DSS2?
Also has Vdub any problem outputting >4Gb files?

AVISource() ; no problems with large files