Need help with encoding DVB-T PAL signal

[marek1712]

Hello dear friends.

I know there are lots of questions like this, but I'm stuck.
I'm recording some anime from the DVB-T broadcasted in Poland (which means 576i50).

So - the first thing is my decoder. It's Ferguson FT-8100 which records .pvr files (WHICH turn out to be AVC in MPEG2-TS) with ~3000kb/s bitrate. I don't know any program that successfully cuts M2TS (so I can get rid of commercials, etc) - that's why I didn't bother with it too much. I've figured out I can mux it to the MKV with MKVmerge. Output file is then ~100-150MB smaller. How come? For example - 25 minutes long record is ~700MB. After muxing it weighs ~500MBs.
But then - I can use Avidemux or VirtualDub to operate on the file.
Output files look nicely in the VLC AFTER performing Yadif 2x deinterlace and little sharpening. But I'd like it to be independent from the player - don't need interlaced files.
Getting to the point...

I'd like to deinterlace the file, sharpen it, reduce the size (I've seen files half the size that look the same if not better) and possibly denoise. You can find RAW sample attached below (already muxed into MKV).
VirtualDub always makes it problematic for me so I've been trying to use Avidemux 2.6.0 to recompress it. The thing is - it crashes just after first pass. Speaking of which - I'd need some help with trascoding - not to make the quality worse or the file bigger in size.

SAMPLE (http://d01.megashares.com/dl/Regqrlr/SM_sample.mkv)

[nm]

So - the first thing is my decoder. It's Ferguson FT-8100 which records .pvr files (WHICH turn out to be AVC in MPEG2-TS) with ~3000kb/s bitrate. I don't know any program that successfully cuts M2TS (so I can get rid of commercials, etc) - that's why I didn't bother with it too much. I've figured out I can mux it to the MKV with MKVmerge. Output file is then ~100-150MB smaller. How come?

TS format itself has lots of overhead and there may also be some padding that gets stripped in the process.

I've been trying to use Avidemux 2.6.0 to recompress it. The thing is - it crashes just after first pass.

Any messages about the reason of that crash?

Why 2-pass? CRF would probably suit you better.

SAMPLE (http://d01.megashares.com/dl/Regqrlr/SM_sample.mkv)

That's field-blended video (poor NTSC to PAL conversion). You need AviSynth and SRestore to recover what's left of the original progressive video.

There might have been some truly interlaced scenes (usually camera pans) too in the original, but I didn't spot any in that sample. They can't be fully restored, and bob-deinterlacing might be the easiest option to handle those parts. Try QTGMC for that. I don't know if there's any tool that can reliably detect such sequences within field-blended video.

[marek1712]

Thanks for some answers.
ad.1 - it only crashes with 2-pass encoding. You've mentioned I should use CRF - with it I get the error File too short (http://www.avidemux.org/smf/index.php?topic=10183.0) which probably is a side effect of my poor cutting abilities (the sample probably doesn't start with key frame; couldn't find proper indicator in Avidemux - it shows only P and B frames on the scale). BTW - I always thought 2-pass encoding offered the best quality. Any article I can read about it?
ad.2 - I'll check this Avisynth script later. Meanwhile I've tried following filters in avidemux (in this order):
- Yadif (Bob, temporal & spatial check)
- asharp (threshold/strength/block adaptive = 2.0/4.0/2.0)
- MPlayer Denoise 3D HQ - and it crashes on this one - only with these muxed MKVs.

Now - without denoise filter, after launching it in VLC and performing MPlayer Denoise 3D HQ post processing filter there, with maximum PP strength, the video looks quite nice. The only problem is the size doubles with quantizer value=22.

Anyway - when I'll have some free time I'll definitely check this and other AVS scripts. Still got some free space (=time) on the HDD :)

[nm]

Thanks for some answers.
ad.1 - it only crashes with 2-pass encoding. You've mentioned I should use CRF - with it I get the error File too short (http://www.avidemux.org/smf/index.php?topic=10183.0) which probably is a side effect of my poor cutting abilities (the sample probably doesn't start with key frame; couldn't find proper indicator in Avidemux - it shows only P and B frames on the scale). BTW - I always thought 2-pass encoding offered the best quality. Any article I can read about it?

At the same bitrate, 2-pass and CRF give about the same quality. With 2-pass, the encoder targets a specific average bitrate (or file size) but quality depends on the content and encoder settings. With CRF the encoder targets a quality level but file size depends on the content and settings.

In practice CRF is usually the better choice if you aren't targeting a specific file size. You don't need to guess a suitable bitrate to reach the quality you want. CRF is also faster since it only needs one pass.

http://www.avidemux.org/admWiki/doku.php?id=tutorial:h.264#general



ad.2 - I'll check this Avisynth script later. Meanwhile I've tried following filters in avidemux (in this order):
- Yadif (Bob, temporal & spatial check)
- asharp (threshold/strength/block adaptive = 2.0/4.0/2.0)
- MPlayer Denoise 3D HQ - and it crashes on this one - only with these muxed MKVs.

Now - without denoise filter, after launching it in VLC and performing MPlayer Denoise 3D HQ post processing filter there, with maximum PP strength, the video looks quite nice. The only problem is the size doubles with quantizer value=22.

Yep, noise takes a lot of space, and you're encoding what is essentially 24p video as blended 50p. Yadif also causes artifacts that reduce compression ratio.

Here's my first try: sm_mdegrain_lsfmod.mkv (http://ubuntuone.com/1UILGtKII6Fel734mMfSxD?attachment=true)
- Yadifmod with nnedi3 to bob the video for SRestore
- SRestore to recover the 24p (24000/1001 fps) video
- Stabilise with Depan
- Sharpen with LSFMod
- Denoise with MDegrain2 (Some lower frequency moving noise is left over though. Would need something heavier.)
Encoding at CRF 20 reduced video size by 26 % compared to the original sample (http://ubuntuone.com/5GrdpvNJvxtl0a05tvNRiB?attachment=true) (copied here for direct downloads).

Another, completely denoised version with reconstructed lines: sm_restored.mkv (http://ubuntuone.com/0dWMqSTB84cz4PWqzwPoky?attachment=true)
- Used this script (http://forum.doom9.org/showthread.php?t=134078) after srestore & depan
Encoding at CRF 20 reduced video size by 41 % compared to the original sample.

My encodes have a couple bad frames in the beginning. I had to use ffmpeg to make a lossless intermediate since ffms2 has difficulties with interlaced H.264 and I can't use other AviSynth source filters on Linux. Ffmpeg had some problems with the original sample too, although MPlayer and VLC seem to play it fine.

[marek1712]

Hello again. Sorry for such a delay with a reply.

I'd like to say it first - you clip looks ASTONISHING! I didn't know it's possible to achieve such quality with a original source like this.
I wanted to follow your guideline but it seems I'm stuck.
This is my script:
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\aWarpSharp.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\Deen.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\Depan.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\MaskTools-v1.5.5.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\mt_masktools-26.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\Mvtools2.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\dfttest.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\deblock.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\nnedi3.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\yadifmod.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\ffms\ffms2.dll")
Import("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\FastLineDarken.avs")
Import("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\DeHalo_alpha.avsi")
FFVideoSource("C:\Users\Markus\Desktop\SM_sample.mkv", threads=1)

yadifmod(edeint=nnedi3(field=-1))
Import("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\SRestore.avsi")
srestore()
Depan()

super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)

Deblock()
dfttest()
FastLineDarken()
deen("a3d",4,8,9)
Spline36Resize(last.width*2,last.height*2)
AddBorders(4, 0, 4, 0)
aWarpSharp(depth=12,blurlevel=4,thresh=0.2,cm=1)
FastLineDarken()
aWarpSharp(depth=6,blurlevel=4,thresh=0.7,cm=1)
Crop(4,0,-4,0)
FastLineDarken()
DeHalo_Alpha()
Spline36Resize(last.width/2,last.height/2)

Import("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\LSFmod.v1.9.avsi")

MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
MeGUI's preview window crashes once I add the Depan() function (Event Viewer states that it's depan.dll's fault).
Besides - am I doing it right (loading scripts, DLLs, order of things and parameters)? I've encoded the video without the Depan() function and it looks like it isn't deinterlaced :/ No to mention the 0.5 fps on my C2D E6300...

[Atak_Snajpera]

sometimes less is more.

you are using awarpsharp twice.
fastlinedarken three times.
resizer two times.
first you add borders then you remove them.

you should use filters which are really really really needed. no wonder you get only 0.5 fps

[marek1712]

Uhm - I've mindlessly used the script suggested earlier. Once I get the quality of the latest sample provided by the nm - I'll try to organize it a bit and experiment - what is what and do I really need it.

[Blue_MiSfit]

Regarding your original question - MPEG-TS almost always contains a good deal of padding / nulls. This is because the final muxes usually need to be strictly CBR, and the video is basically never strictly CBR. Even pseudo CBR usually has a lot of variation on a frame by frame basis (which is a very good thing for quality). MPEG-TS itself also has inherent overhead. Transmitting things like the PCR, PMT, descriptors etc does use some bitrate.

For example, if you were shooting for a 5625kbps mux (commonly used in IPTV applications), you might encode the video at 4700kbps pseudo-CBR, and the audio at 384kbps pure CBR. This leaves an average of 541 kbps, which is about 9.6% of the full mux. The overhead of MPEG-TS will get you close to the full mux rate, and the nulls will pad you up to the full rate. I'd say this example is fairly aggressive with usage of the available bitrate. Your example is obviously from a much more conservative deployment. There may be additional bitrate consumed by error correction with things like DVB-T, but I'm really not sure.

It's ALWAYS better to err on the side of caution than it is to accidentally cause broadast issues!

Derek

[Ghitulescu]

Not all MPEG-PS streams carry a "good load" of NULLs.
DVB-T has in Europe typically 2-3Mbps per video PID. Which is at most half of what a DVB-S stream usually carries (apart from teleshopping channels, but who records teleshopping?). A full movie (90min) can be broadcast in around or less than 2GB. Because of this lower bitrate post-processing must be performed, in particular deblocking.

[nm]

MeGUI's preview window crashes once I add the Depan() function (Event Viewer states that it's depan.dll's fault).

Depan is a bit more complex than a single function call. You need motion data from DepanEstimate too.

I've encoded the video without the Depan() function and it looks like it isn't deinterlaced :/

You need to bob the video with something like yadifmod(mode=1,edeint=nnedi3(field=-2))

It's also possible that direct input of interlaced H.264 video doesn't work properly with FFMS2. It didn't for me, and I had to encode the video losslessly to FFV1 with ffmpeg and then open it in AviSynth with FFMS2.

No to mention the 0.5 fps on my C2D E6300...

Yep, that reconstruction script by Dark Shikari is pretty slow.

I used this script. Just a quick try, not optimal:

LoadPlugin("ffms2.dll")
LoadPlugin("mvtools2.dll")
LoadPlugin("removegrainbeta\RemoveGrainSSE2.dll")
LoadPlugin("mt_masktools-25.dll")
LoadPlugin("yadifmod.dll")
LoadPlugin("nnedi3.dll")
LoadPlugin("RepairSSE2.dll")
LoadPlugin("deblock.dll")
LoadPlugin("dctfilter.dll")
LoadPlugin("dfttest.dll")
LoadPlugin("DePan.dll")
LoadPlugin("DePanEstimate.dll")
LoadPlugin("Deen.dll")
LoadPlugin("aWarpSharp.dll")
LoadPlugin("fft3dfilter.dll")
import("Srestore.avsi")
import("deblock_qed.avsi")
import("dehalo_alpha_mt.avsi")
import("fastlinedarkenmod.avsi")

setmemorymax(1024)

# FFVideoSource("input.mkv",seekmode=0,threads=1,fpsnum=25,fpsden=1)
FFVideoSource("input.avi")
AssumeFPS(25)
AssumeTFF()

# Remember to crop vertically by mod 4 amounts when dealing with interlaced video
crop(12,4,696,572)

# Bob and deblend
yadifmod(mode=1,edeint=nnedi3(field=-2))
srestore()


# Stabilize
clp=last
temp = clp.TemporalSoften(7,255,255,25,2)
inter = Interleave(temp.Repair(clp.TemporalSoften(1,255,255,25,2)),clp)
mdata = DePanEstimate(inter,range=1,trust=0,dxmax=16,dymax=16)
DePan(inter,data=mdata,offset=-1,mirror=15)
SelectEvery(2,0)

# Remove dust and noise
o=last
osup = o.MSuper(pel=2,sharp=2)
bv2 = MAnalyse(osup, isb=true, delta=2, blksize=8, overlap=4, search=4)
bv1 = MAnalyse(osup, isb=true, delta=1, blksize=8, overlap=4, search=4)
fv1 = MAnalyse(osup, isb=false, delta=1, blksize=8, overlap=4, search=4)
fv2 = MAnalyse(osup, isb=false, delta=2, blksize=8, overlap=4, search=4)
bc1 = o.MCompensate(osup, bv1)
fc1 = o.MCompensate(osup, fv1)
Interleave(fc1,o,bc1)
Clense(reduceflicker=false)
SelectEvery(3,1)
MDegrain2(osup, bv1, fv1, bv2, fv2, thSAD=400)

# Dark Shikari's anime reconstruction script
dfttest()
FastLineDarkenMod()
deen("a3d",4,8,9)
Spline36Resize(last.width*2,last.height*2)
AddBorders(4, 0, 4, 0)
aWarpSharp(depth=12,blurlevel=4,thresh=0.2,cm=1)
FastLineDarkenMod()
aWarpSharp(depth=6,blurlevel=4,thresh=0.7,cm=1)
Crop(4,0,-4,0)
FastLineDarkenMod()
dehalo_alpha_mt()
fft3dfilter(bt=3,sigma=8,sharpen=1.3)
Spline36Resize(last.width/2,last.height/2)


I had to use the slower fft3dfilter instead of fft3dgpu because of compatibility issues with my Wine environment.

[kieranrk]

There may be additional bitrate consumed by error correction with things like DVB-T, but I'm really not sure.


Error correction is done at a different layer for these services and the bitrate you would be allocated by the operator would be before error correction was added.

[Blue_MiSfit]

That makes good sense! It's funny, I was just reading ATSC specs the other day for fun, and came to the same exact conclusion regarding where error correction happens.

[manolito]

Not all MPEG-PS streams carry a "good load" of NULLs.
DVB-T has in Europe typically 2-3Mbps per video PID. Which is at most half of what a DVB-S stream usually carries (apart from teleshopping channels, but who records teleshopping?). A full movie (90min) can be broadcast in around or less than 2GB. Because of this lower bitrate post-processing must be performed, in particular deblocking.
Can you recommend a good "one size fits all" deblocker and settings for exactly this purpose?

I usually reencode my DVB-T captures to remove the station logo. So far I have never used any deblocker mainly because I still watch my movies on a CRT TV which is very forgiving when it comes to blocking.

Recently I watched one of my reencoded captures at a friend's house on an LCD, and I was quite shocked about the very visible blocking. So it might be a good idea to add deblocking routinely. But since I cannot judge the result on my CRT I need some recommendations.

The settings should be a good compromise between enough deblocking and not destroying all the details. And it should be fast.

Deblock_QED would probably be too slow for me. MPEG2Source ("", cpu=4) or Deblock(30) sound reasonable. Some people recommend the MSU Smart Deblocker, but I think this would also be too slow.


Thanks
manolito

[nm]

Deblock_QED would probably be too slow for me.

Shouldn't be too slow for reencoding. With a decent CPU (2.4 GHz Core 2 or better) it runs easily in real time for SD videos, in a single thread.

It's the only deblocking filter I've found satisfactory for MPEG-2 DVB sources.

[marek1712]

@nm - I'd like to ask you another question. You've mentioned that ffms2 doesn't like interlaced H.264 (it's even mentioned in the readme). Is the symptom similar to video going back like 0.2s every second or two?
I've made simple recompression to FFV1 with ffmpeg (with vcodec ffv1 parameter). Is it enough or should I add something else?
BTW - nonOCed i7-2600k reaches as much as 1.8fps with this heavy script (and CPU utilization is 40% max) :)

[nm]

@nm - I'd like to ask you another question. You've mentioned that ffms2 doesn't like interlaced H.264 (it's even mentioned in the readme). Is the symptom similar to video going back like 0.2s every second or two?
Yep, that's what I usually get even with fpsnum and fpsden set. I guess this is what the documentation describes as:
one of the fields (odd or even) may "jump around"

I've made simple recompression to FFV1 with ffmpeg (with vcodec ffv1 parameter). Is it enough or should I add something else?

That's enough.

[manolito]

It's the only deblocking filter I've found satisfactory for MPEG-2 DVB sources.

Anyone compared Deblock_QED to SmoothD2 ?


Cheers
manolito

[marek1712]

Sorry for dragging it for so long but there's another problem.
After some time the resolution of the source material has changed to something strange... 1047x576 (something which isn't even a 16:9 aspect ratio).
Do I have to crop in the H and V part (and get rid of the vertical part of the image) or can I remove the black borders to make it 4/3 somehow.
I've tried to calculate the aspect ratio of 1.333333333333 and came with something like 768x576.
Sample:
http://s17.postimage.org/ow5acabe3/vlcsnap_2012_05_28_15h43m54s233.jpg (http://postimage.org/image/ow5acabe3/)

[nm]

Sorry for dragging it for so long but there's another problem.
After some time the resolution of the source material has changed to something strange... 1047x576

That's not the actual resolution of source frames but size of frames scaled according to the SAR (sample aspect ratio) of the stream. Apparently the broadcaster used the ITU-R BT.601 compatible SAR of 16:11, which is calculated for 704x576 video: 704*16/11 = 1024. Basically that means that at least 8 pixels (more if there's overscan) would be cropped from each side of the source frame when the video is displayed on a 16:9 screen.

I'd keep the SAR and crop the black borders of 4:3 content that was encoded anamorphically. If the resulting non-standard frame size is a problem, resize to 720x576 and change the SAR to either 16:11 or 64:45.