How to disable de-interlace in ffms2 source

[t_2]

Searched, but couldn’t find an answer to what must be a silly question. I have an archival .avi file (mjpeg/wav) that has become corrupted after 25 years. Nothing worked, (neither AviSource, DirectShowSource, nor LSMASHsource) until I was finally able to open it correctly in AviSynth using ffms2 source (ffms2-5.0-msvc). Hurray! Now I can save to lossless HuffyUV! Unfortunately, I have one more problem: ffms2 de-interlaces the video by default, but I want to preserve the original de-interlacing (Pal 50i). What’s more, the de-interlacing has introduced some artifacts that distort the video. Can ffms2 pass a command from an avs scripy to disable de-interlacing? My sincere thanks in advance to anyone who can help me with this problem. This video is very dear to me, and I’d hate to lose it.

[Sharc]

Originally Posted by t_2

Searched, but couldn’t find an answer to what must be a silly question. I have an archival .avi file (mjpeg/wav) that has become corrupted after 25 years. Nothing worked, (neither AviSource, DirectShowSource, nor LSMASHsource) until I was finally able to open it correctly in AviSynth using ffms2 source (ffms2-5.0-msvc). Hurray! Now I can save to lossless HuffyUV! Unfortunately, I have one more problem: ffms2 de-interlaces the video by default, but I want to preserve the original de-interlacing (Pal 50i). What’s more, the de-interlacing has introduced some artifacts that distort the video. Can ffms2 pass a command from an avs scripy to disable de-interlacing? My sincere thanks in advance to anyone who can help me with this problem. This video is very dear to me, and I’d hate to lose it.

ffms2 does not deinterlace by default. I understand that you want to preserve the original interlaced format. Are you sure that your mjpeg sources are interlaced at all?

[t_2]

My film was archived in 28 separate .avi files each about 500MB on a single layer writable DVD. Only the first file is corrupt. The other 27 files open in VirtualDub2 and show clearly, in high motion areas, the interlaced fields. Please see the attached jpeg named “VirtualDub2.jpeg.” Compare this to a frame named “ffms2.jpeg,” which was from the corrupted file that I managed to open in AviSynth+ with ffms2 source. All the fields have been de-interlaced.

[Sharc]

Can you upload short snippets (few seconds) of the 2 original files?
(Maybe upload to a file hoster like Mega or similar).
Were the 2 clips shot with the same camera? MJPEG is poorly standardized. It can be progressive or interlaced depending on codec/camera or camera setting options.
Any video processing in place when the files were archived to the DVD?

Edit: Try to set the framerate in ffms2(), something like ffms2("your_source",fpsnum=25,fpsden=1). Maybe it picks only fields from the corrupted .avi

[t_2]

Sorry about the long post! The corrupted avi is named “001.avi”. The weird thing is that I can open it in VirtualDub2, where it plays perfectly. I can even grab a frame using “export” command. See “VirtualDub2_001.jpeg.” There you can see that the fields have not been de-interlaced. But when I try to save “001.avi” in VirtualDub2 (using direct stream copy or some compression codec), I get the error message: “File format is incompatible.” AviUtl will open, play and save the file, but the output looks like: “AviUtl.jpeg.” The video looks the same as “AviUtl.jpeg” when I open the file in AviSynth+ using the following script:

A=LWLibavAudioSource("001.avi")
V=LWLibavVideoSource("001.avi")
AudioDub(V,A)

The weird thing is that half the time, when I open the file with this script, it looks and plays normally and doesn’t de-interlace the video in VirtualDub2’s playback window, then a frame grab looks like: “VirtualDub2_001.jpeg” – i.e., no de-interlacing! Even then, however, when I save the file in VirtualDub2, (regardless of whether it’s a direct stream copy of video/audio, or compressed with some codec), the output video always looks like: “AviUtl.jpeg.” The other half of the time, the above avi script even looks like “AviUtl.jpeg” in VirtualDub2’s playback window!

I tried your “fpsnum=25,fpsden=1” in ffms2 source and the output was still de-interlaced. When I saved the film in 28 parts to 4 DVD-R disks, I simply cut the film in the timeline of Adobe Premier and exported each section separately as an avi file, which I simply copied to the root of the DVD-R disk.

Do you have a Gmail account? If so, I have uploaded the corrupted file, “001.avi” as well as a non-corrupted file from the same camera/capture board, called: “003.avi” Please find the two links below!

JFYI, the file was recorded with a PAL MiniDV camcorder, but because my tech advisor was afraid that our Pentium III workstation couldn’t handle the editing demands, we imported the video into the workstation using the camera’s analog SVHS- PAL output port. That way, playback in the timeline was accelerated by the hardware capture card. The original DV footage, was, I believe progressive PAL 25p, (720x576) but I no longer have the cassettes to verify. I believe the camcorder interlaced the progressive video when it outputted via its SVHS port.

Links to Google Drive:

001.avi – corrupted file – opens, plays but doesn’t save properly.

https://drive.google.com/file/d/17Xb...ew?usp=sharing

003.avi – good file – opens, plays and saves/compresses properly.

https://drive.google.com/file/d/1wIS...ew?usp=sharing

[Sharc]

Something is strange.
Here a workaround. Reencoding it with x264 into .mp4 container seems to work. It is encoded interlaced.
https://mega.nz/file/6R1HgJ6K#LL1zOq...U4vAaJV-Me_J9Y

(You can encode lossless with x264 as well, just set --crf 0)

Edit:
It works using the ffmpeg Huffyuv codec. Here the Huffyuv interlaced lossless file
https://mega.nz/file/nQFgHBgR#o61KPA...iVv2EXxOdZ3jT4

ffms2 as a source filter seems to fail. You need to install the Picvideo codec and open the file with AviSource() rather than with ffms2.

Here the Picvideo codec (trial version):
https://mega.nz/file/TINWgZrA#z2vzac...ZJn80LxB-kds0E

It also works with Virtualdub2.

[hello_hello]

ffms2 r2390 and ffms2 version 5.0 and lsmash opened the AVI in Avisynth for me. BSVideoSource didn't know what to make of it.
There was a lot of aliasing when ffms2 was decoding as it appeared to be de-interlacing but not for LWLibavVideoSource. I'm not sure why, but VirtualDub2 often displays nothing but blank frames for a while when opening the AVI with the AVI Driver (taking a long time to display something, and it upsets AVIFile compatibility mode).



It's over my head, but it's as if the fields are stored together at the beginning but separately later on. SMPlayer starts off displaying frames then switches to displaying fields as frames, I assume.

[poisondeathray]

Originally Posted by hello_hello

I'm not sure why, but VirtualDub2 often displays nothing but blank frames when opening the AVI directly.

Depends on what decoder vdub2 is configured to use. If you have "prefer internal video decoders over installed third-party codecs (MJPEG...etc...)" enabled , it decodes ok .

With the file directly loaded, file=>file information will tell you what is being used to decode

[t_2]

@Sharc

Wow! That PICVideo MJPEG codec was the answer to all my problems! Thank you so much for taking the time to help me out! I really appreciate it! You are a life saver. I mean it! Unlike FFMpeg, the PicVideo codec even opens the corrupted file perfectly in AviSynth+, as you said. For that matter, VirtualDub2 also opens it correctly – wonder what it uses to do that?

I tried to buy the PICVideo codec, however, their web site says it’s no longer for sale! I notice that the trial version counts how many times you use it. So I did an end task on VirtualDub2 to save one instance of my limited uses. What a mistake! Unfortunately, after that the codec no longer works on my system. Whenever I click on continue, it counts another use, but won’t let the program access the codec. I have to do another end task to close VirtualDub2.

I tried uninstalling and reinstalling the PICVideo codec, but that didn’t solve the problem. Unfortunately, system restore was turned off on the computer that I am using, so I wasn’t able to roll back to an earlier restore point. No worries, I can use another computer to get my job done. I was, however, wondering what exactly the limitations are on the trial version? Is there any workaround to these limitations? Thanks again for solving my problem!

@hello-hello

With regard to that error message about a frame height of 272 vs 270, I noticed that VirtualDub2 opens the file and plays it well, but for some reason it opens it as with a file size of 720x544 when the original file is actually 720x540. VirtualDub2 also saves it with the same enlarged dimensions! Does that mean that it automatically resizes the fields? Since this copy is for archival purposes, I’d hate to suffer the slight quality loss that an unnecessary resize would occasion.

@poisondeathray

I do have “prefer internal video decoders” enabled in VirtualDub2. With the exception of the resize mentioned about, it opens and plays the avi files correctly, without de-interlacing and introducing all those nasty articacts. The PICVideo codec even leaves the file as is, i.e.: 720x540.

@everyone: Thanks for your help!

[Sharc]

Originally Posted by t_2

@Sharc

I tried to buy the PICVideo codec, however, their web site says it’s no longer for sale! I notice that the trial version counts how many times you use it. So I did an end task on VirtualDub2 to save one instance of my limited uses. What a mistake! Unfortunately, after that the codec no longer works on my system. Whenever I click on continue, it counts another use, but won’t let the program access the codec. I have to do another end task to close VirtualDub2.

I tried uninstalling and reinstalling the PICVideo codec, but that didn’t solve the problem. Unfortunately, system restore was turned off on the computer that I am using, so I wasn’t able to roll back to an earlier restore point. No worries, I can use another computer to get my job done. I was, however, wondering what exactly the limitations are on the trial version? Is there any workaround to these limitations? Thanks again for solving my problem!

Uninstall PicVideo using the Revo Uninstaller. Then re-install Picvideo.
Even the free version of Revo Uninstaller seems to reset the Picvideo counter (use the "Scan" for a complete uninstall).
https://www.revouninstaller.com/de/r...free-download/
https://www.snapfiles.com/get/revouninstaller.html

[Sharc]

... and ffms2() seems to work as well when you just trim (cut) the first 564 frames of your 001.avi. There seems to be something odd is going on at the beginning of the clip which fools most codecs, so

ffms2("001.AVI")
trim(564,0)
assumeTFF()

seems to solve the issue as well.

Edit:
Here a workaround for the full 001.avi file, splitting it into 2 parts:

v1=ffms2("001.AVI")
v2=ffms2("001.AVI")
v1=v1.trim(0,563)
v2=v2.trim(564,0)
v=v1++v2
v=v.assumeTFF()
return v

https://mega.nz/file/3BlwxJqI#mbGeCP...w8hRCXpY3HVmBM

[Emulgator]

https://sourceforge.net/projects/vdf.../version%2020/
44282 zip contains both 32 and 64 bit versions. You just run the 32bit version (VirtualDub.exe) 4008KB

[Selur]

Small suggestion: maybe use 'code'-tags to make your posts more readable to others.

[Emulgator]

Jó estét ! I would welcome a sample of the source MJPG file.

[t_2]

Köszönöm érdeklődését!


https://drive.google.com/file/d/1zFG...ew?usp=sharing


If you’re interested in knowing what the actors are saying, please find below a link to the entire 61 min drama with English subtitles. The young ladies’ conversation starts at 14 min 30 sec and lasts for 2 min 17 sec. The conversation stands on its own. SPOILER ALERT! The film is religious content on steroids!

https://drive.google.com/file/d/1Hhj...ew?usp=sharing

[t_2]

@ Emulgator

Thanks again for your interest in my problem! That was so kind of you! In the meantime, I have been working further on the problem. I took your cue and started working with the raw archive footage before passing it through the de-interlacer. Alas, the strobing problem – lightening and darkening of the background – wasn’t fixed. Then, I decided to put white borders around the clips (settled on 200 pixels all around) before passing them to the Retinex filter and cropping them off afterwards. That seems to have done the trick.

Please find here a Google Drive link to the video output of the script which I have cut and pasted below.

https://drive.google.com/file/d/1Hs5...ew?usp=sharing

I was even able to use much stronger settings (lowered “sigma” from 120 to 60) for the effect. This did produce another distortion effect, a sort of halo that surrounds the two girls in scenes 2 and 4. I think I can live with that. Do you think there is any way to improve on these results? I must admit that I am pleasantly pleased. Really interested in your feedback.

##############################################
AudioX=FFAudioSource("LumaMJPEG.avi")
Video1=FFVideoSource("LumaMJPEG.avi")

y=200


Scene001a=Video1.Trim(0, 900).SeparateFields().AddBorders(y, y, y, y, $FFFFFF).ConvertToYUV444.MSRCP(sigma=80, fulls=true, fulld=true).ConvertToYUV422().Crop(y, y, -y, -y). Weave() # Two girls - Bogi and Reka are common Hungarian girl's names.
Scene002a=Video1.Trim(901, 1046).SeparateFields().AddBorders(y, y, y, y, $FFFFFF).ConvertToYUV444.MSRCP(sigma=80, fulls=true, fulld=true).ConvertToYUV422().Crop(y, y, -y, -y). Weave() # Bogi - First close up
Scene003a=Video1.Trim(1047, 1257).SeparateFields().AddBorders(y, y, y, y, $FFFFFF).ConvertToYUV444.MSRCP(sigma=80, fulls=true, fulld=true).ConvertToYUV422().Crop(y, y, -y, -y). Weave() # Reka - First close up
Scene004a=Video1.Trim(1258, 2352).SeparateFields().AddBorders(y, y, y, y, $FFFFFF).ConvertToYUV444.MSRCP(sigma=120, fulls=true, fulld=true).ConvertToYUV422().Crop(y, y, -y, -y). Weave() # Two girls together.
Scene005a=Video1.Trim(2353, 2927).SeparateFields().AddBorders(y, y, y, y, $FFFFFF).ConvertToYUV444.MSRCP(sigma=80, fulls=true, fulld=true).ConvertToYUV422().Crop(y, y, -y, -y). Weave() # Reka - 2nd close up
Scene006a=Video1.Trim(2928, 0).SeparateFields().AddBorders(y, y, y, y, $FFFFFF).ConvertToYUV444.MSRCP(sigma=80, fulls=true, fulld=true).ConvertToYUV422().Crop(y, y, -y, -y). Weave() # Bogi - 2nd close up

Video2=UnalignedSplice(\
Scene001a,\
Scene002a,\
Scene003a,\
Scene004a,\
Scene005a,\
Scene006a)

Video2=AudioDub(Video2, AudioX).AssumeTFF()

Scene001b=Video2.Trim(0, 900)
Scene002b=Video2.Trim(901, 1046)
Scene003b=Video2.Trim(1047, 1257)
Scene004b=Video2.Trim(1258, 2352)
Scene005b=Video2.Trim(2353, 2927)
Scene006b=Video2.Trim(2928, 0)

Luma=UnalignedSplice(\
Scene001b,\
Scene002b,\
Scene003b,\
Scene004b,\
Scene005b,\
Scene006b)

#Return Scene001b
#Return Scene002b
#Return Scene003b
#Return Scene004b
#Return Scene005b
#Return Scene006b

Return Luma

#################################################

[Emulgator]

Downloading now. Ah, I get better shadow details fom the source MJPEG. Expected that. Stay tuned.

# +++++ Start of AutoPath 1 Videostream + 0 or 1 Audiostream  +++++
SetFilterMTMode("DEFAULT_MT_MODE", 2)
SetFilterMTMode("LWLibavVideoSource", 3)
SetFilterMTMode("LWLibavAudioSource", 3)
SetFilterMTMode("FFVideoSource", 3)
SetFilterMTMode("FFAudioSource", 3)
SetFilterMTMode("DGSource", 3)
SetFilterMTMode("InpaintDelogo", 3)
scriptextlen=FindStr(RevStr(ScriptFile()),".")
avextlen=FindStr(RevStr(LeftStr(ScriptFile(),StrLen(ScriptFile())-scriptextlen)),".")
avfolder=ScriptDir()
audfolder=avfolder
avfile0=LeftStr(ScriptFile(),(StrLen(ScriptFile())-scriptextlen-avextlen))
avext0=LeftStr(RightStr(ScriptFile(),scriptextlen+avextlen),avextlen)
vidsrc= avext0==".d2v" ? "MPEG2Source" : avext0==".dga" ? "AVCSource" : avext0==".dgi" ? "DGSource" : avext0=="mpg" ? "FFVideoSource" : "LWLibavVideoSource"
#vidsrc="LWLibavVideoSource"
#vidsrc="FFVideoSource"
vidformatLWLibav="YUV444P8"
vidformatFF="YV24"
vidargDG158="idct=5, ipp=true, cpu=6, cpu2=String(xxxxxx), moderate_h=40, moderate_v=80"
audfile=avfile0
audext=avext0==".d2v" ? ".ac3" : avext0==".dgi" ? ".ac3" : avext0
#audext=avext0==".d2v" ? ".dts" : avext0==".dgi" ? ".dts" : avext0
#audext=avext0==".d2v" ? ".wav" : avext0==".dgi" ? ".wav" : avext0
#audext=".mpg"
audpid= avext0==".d2v" ? " T80 2_0ch 224Kbps DELAY 0ms" : avext0==".dgi" ? " PID 1100 2.0ch 48KHz 192Kbps DELAY 0ms" : avext0==".mpg" ? " Tc0 L2 2ch 44.1 224 DELAY 0ms" : ""
#audpid= avext0==".d2v" ? " T81 3_2ch 448Kbps DELAY 0ms" : avext0==".dgi" ? " T81 3_2ch 48KHz 448Kbps DELAY 0ms" :""
audpid=""
audsrc= audext==".ac3" ? "NicAC3Source" : audext==".mp2" ? "FFAudioSource" : "LWLibavAudioSource"
#audsrc="BestAudioSource"
#audsrc="FFAudioSource"
ffaudtrk=1
audsrc="LWLibavAudioSource" #Asd-g versions only, here 1194
Libavaudtrk=-1 #int stream_index = -1 #-1: default stream
Libavaudcache=true #bool cache = true
Libavaudcachefile=String(audfolder)+String(audfile)+String(audpid)+String(audext)+".lwi"#string cachefile = source +".lwi"
Libavaudavsync=false #bool av_sync = false
Libavaudlayout="FL+FR"#"0x3f"#"FL+FR+FC+LFE+SL+SR" #string layout = ""  | CN ? No Audio |  5.1 -> "5.1" or "FL+FR+FC+LFE+SL+SR" or "0x3f"
Libavaudrate=0 #int rate = 0
Libavauddecoder="" #string decoder = ""
Libavaudffloglevel=0 #int ff_loglevel = 0
Libavaudcachedir="" #string cachedir = ""
Libavindexingpr=true #bool indexingpr = true
Libavauddrcscale=0.0 #float drc_scale = 1.0 #0.0 deactivates DRC, -values invert the DRC processing
Libavaudffopts="" #string ff_options = ""
Exist(String(avfolder)+String(avfile0)+String(avext0)) ? Apply(vidsrc,String(avfolder)+String(avfile0)+String(avext0))\
: Assert(false, "! NO SUCH AV FILE -> " + String(avfolder)+String(avfile0)+String(avext0) + "<- NO SUCH AV FILE !")
#try {AudioDub(last,Apply(audsrc,String(audfolder)+String(audfile)+String(audpid)+String(audext)))}
try {AudioDub(last,Apply(audsrc,String(audfolder)+String(audfile)+String(audpid)+String(audext),Libavaudtrk,Libavaudcache,Libavaudcachefile,Libavaudavsync,Libavaudlayout,Libavaudrate,Libavauddecoder,Libavaudffloglevel,Libavaudcachedir,Libavindexingpr,Libavauddrcscale,Libavaudffopts))}
#Blankclip(last).Subtitle(String(audfolder)+String(audfile)+String(audpid)+String(audext))
catch (err) {last}
# -----   End of AutoPath 1 Videostream + 0 or 1 Audiostream   -----
# source is 720x544 TFF
ConvertToYUV444()
smoothdebug=0
SmoothCurve16(ConvertTo16bit().ConvertToStacked(),\
Ycurve="0-1023;12000-14000;38000-35000;55000-42000;60000-48000;65535-61000",\
Ucurve="0-0;16383-16383;32767-32767;49151-49151;65535-65535",\
Vcurve="0-0;16383-16383;32767-32767;49151-49151;65535-65535",\
limiter=false, HQ=true, interp=100, debug=smoothdebug==0 ? false : true, screenW=1920, screenH=1080).ConvertFromStacked()#.Convertbits(8)
propSet("_FieldBased",2) #QTGMCp needs propSet from a certain version on. 0=frame based (progressive), 1=bottom field first, 2=top field first 
QTGMCp()
ConvertBits(32).ConvertToPlanarRGB().MSRCR(sigma=[300,400,500], lower_thr=0.001, upper_thr=0.001, fulls=false, fulld=false, restore=125.0).ConvertToYUV444().ConvertBits(16)
#return(last)
SmoothCurve16(ConvertTo16bit().ConvertToStacked(),\
Ycurve="0-0;12000-13000;32767-32767;55000-48000;65535-65535",\
Ucurve="0-0;16383-18000;32767-32767;49151-47000;65535-65535",\
Vcurve="0-0;16383-16383;32767-32767;49151-46000;65535-65535",\
limiter=false, HQ=true, interp=100, debug=smoothdebug==0 ? false : true, screenW=1920, screenH=1080).ConvertFromStacked().ConvertBits(10)
SMDegrain(tr=6, thSAD=400, RefineMotion=true, contrasharp=true, interlaced=false, plane=4, prefilter=8, chroma=true, Show=false) #5: DFTTest, 6: KNLMeansCL, 7: DGDenoise, 8: BM3D
GrainFactory3(2*Round(Pow(2,BitsPerComponent-8)),2*Round(Pow(2,BitsPerComponent-8)),1*Round(Pow(2,BitsPerComponent-8)),100,100,100,0.9*sqrt(width()/720),0.7*sqrt(width()/720),0.5*sqrt(width()/720),0,0,0,0,0,24*Round(Pow(2,BitsPerComponent-8)),56*Round(Pow(2,BitsPerComponent-8)),128*Round(Pow(2,BitsPerComponent-8)),160*Round(Pow(2,BitsPerComponent-8)))
#portalscope()

This gives 720x544x50p YUV444P10 with lots of detail.
It still needs colour matching to sunlight per shot, your turn (MSRCR amplifies sky casts: sometimes pinkish tint, sometimes cyan tint).

P.S. SMDegrain to your liking, ReGrain...well, there they are. Couldn't resist.

[t_2]

@ Emulgator

Wow! Very impressive script. I can't make heads nor tails of it. I am getting the following error: "No such video file" at line 47, but can't figure out what file it is looking for. Also, I noticed in your script that you call QTGMCp(). I only have QTGMC() installed. Will that give me problems? I can't wait to see your results, but I will have to since I am out of town the next 4 days without Internet!

[Emulgator]

Ah, forgot to mention: Name the script after your file and put it side by side like that: "LumaMJPEG.avi.avs"
Or just cut off all the intro including # ----- End of AutoPath 1 Videostream + 0 or 1 Audiostream -----
and start with LWLibavVideoSource(), LWLibavAudioSource() as usual.

Why QTGMCp() ? It seems to be the gold standard for me now, gets so much details out of interlaced sources, be it VHS or any other.
For trying your script QTGMC is fine too.

Rendering 6850frames @ 2,4fps now into a H.264 4:2:0 10bit .mkv
BTW, just as I saw the scene in the completed movie:
My result is not that much different from what you already achieved in the final movie. Maybe less strobing..
I just prefiltered before Retinex to make it less hard for it, and postfiltered again.

[Sharc]

A luma mask may help to prevent the clipped bright sky from just turning occasionally into grey by the filtering (file fx.mp4 of post#28).
Something like

lmask=clip.mt_binarize(threshold=58,upper=true).converttoYV12(interlaced=true).GaussianBlur(varY=2000.0).grayscale()    
overlay(<your filtering>,mask=lmask)

(I usually apply Retinex with a mask, for example)