Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 14th April 2012, 21:15   #1  |  Link
marek1712
Apprentice
 
Join Date: Apr 2012
Location: Poland
Posts: 6
Need help with encoding DVB-T PAL signal

Hello dear friends.

I know there are lots of questions like this, but I'm stuck.
I'm recording some anime from the DVB-T broadcasted in Poland (which means 576i50).

So - the first thing is my decoder. It's Ferguson FT-8100 which records .pvr files (WHICH turn out to be AVC in MPEG2-TS) with ~3000kb/s bitrate. I don't know any program that successfully cuts M2TS (so I can get rid of commercials, etc) - that's why I didn't bother with it too much. I've figured out I can mux it to the MKV with MKVmerge. Output file is then ~100-150MB smaller. How come? For example - 25 minutes long record is ~700MB. After muxing it weighs ~500MBs.
But then - I can use Avidemux or VirtualDub to operate on the file.
Output files look nicely in the VLC AFTER performing Yadif 2x deinterlace and little sharpening. But I'd like it to be independent from the player - don't need interlaced files.
Getting to the point...

I'd like to deinterlace the file, sharpen it, reduce the size (I've seen files half the size that look the same if not better) and possibly denoise. You can find RAW sample attached below (already muxed into MKV).
VirtualDub always makes it problematic for me so I've been trying to use Avidemux 2.6.0 to recompress it. The thing is - it crashes just after first pass. Speaking of which - I'd need some help with trascoding - not to make the quality worse or the file bigger in size.

SAMPLE
marek1712 is offline   Reply With Quote
Old 15th April 2012, 00:32   #2  |  Link
nm
Registered User
 
Join Date: Mar 2005
Location: Finland
Posts: 2,641
Quote:
Originally Posted by marek1712 View Post
So - the first thing is my decoder. It's Ferguson FT-8100 which records .pvr files (WHICH turn out to be AVC in MPEG2-TS) with ~3000kb/s bitrate. I don't know any program that successfully cuts M2TS (so I can get rid of commercials, etc) - that's why I didn't bother with it too much. I've figured out I can mux it to the MKV with MKVmerge. Output file is then ~100-150MB smaller. How come?
TS format itself has lots of overhead and there may also be some padding that gets stripped in the process.

Quote:
Originally Posted by marek1712 View Post
I've been trying to use Avidemux 2.6.0 to recompress it. The thing is - it crashes just after first pass.
Any messages about the reason of that crash?

Why 2-pass? CRF would probably suit you better.

Quote:
Originally Posted by marek1712 View Post
That's field-blended video (poor NTSC to PAL conversion). You need AviSynth and SRestore to recover what's left of the original progressive video.

There might have been some truly interlaced scenes (usually camera pans) too in the original, but I didn't spot any in that sample. They can't be fully restored, and bob-deinterlacing might be the easiest option to handle those parts. Try QTGMC for that. I don't know if there's any tool that can reliably detect such sequences within field-blended video.

Last edited by nm; 15th April 2012 at 01:13.
nm is offline   Reply With Quote
Old 15th April 2012, 11:30   #3  |  Link
marek1712
Apprentice
 
Join Date: Apr 2012
Location: Poland
Posts: 6
Thanks for some answers.
ad.1 - it only crashes with 2-pass encoding. You've mentioned I should use CRF - with it I get the error File too short which probably is a side effect of my poor cutting abilities (the sample probably doesn't start with key frame; couldn't find proper indicator in Avidemux - it shows only P and B frames on the scale). BTW - I always thought 2-pass encoding offered the best quality. Any article I can read about it?
ad.2 - I'll check this Avisynth script later. Meanwhile I've tried following filters in avidemux (in this order):
- Yadif (Bob, temporal & spatial check)
- asharp (threshold/strength/block adaptive = 2.0/4.0/2.0)
- MPlayer Denoise 3D HQ - and it crashes on this one - only with these muxed MKVs.

Now - without denoise filter, after launching it in VLC and performing MPlayer Denoise 3D HQ post processing filter there, with maximum PP strength, the video looks quite nice. The only problem is the size doubles with quantizer value=22.

Anyway - when I'll have some free time I'll definitely check this and other AVS scripts. Still got some free space (=time) on the HDD
marek1712 is offline   Reply With Quote
Old 16th April 2012, 17:09   #4  |  Link
nm
Registered User
 
Join Date: Mar 2005
Location: Finland
Posts: 2,641
Quote:
Originally Posted by marek1712 View Post
Thanks for some answers.
ad.1 - it only crashes with 2-pass encoding. You've mentioned I should use CRF - with it I get the error File too short which probably is a side effect of my poor cutting abilities (the sample probably doesn't start with key frame; couldn't find proper indicator in Avidemux - it shows only P and B frames on the scale). BTW - I always thought 2-pass encoding offered the best quality. Any article I can read about it?
At the same bitrate, 2-pass and CRF give about the same quality. With 2-pass, the encoder targets a specific average bitrate (or file size) but quality depends on the content and encoder settings. With CRF the encoder targets a quality level but file size depends on the content and settings.

In practice CRF is usually the better choice if you aren't targeting a specific file size. You don't need to guess a suitable bitrate to reach the quality you want. CRF is also faster since it only needs one pass.

http://www.avidemux.org/admWiki/doku...:h.264#general


Quote:
Originally Posted by marek1712 View Post
ad.2 - I'll check this Avisynth script later. Meanwhile I've tried following filters in avidemux (in this order):
- Yadif (Bob, temporal & spatial check)
- asharp (threshold/strength/block adaptive = 2.0/4.0/2.0)
- MPlayer Denoise 3D HQ - and it crashes on this one - only with these muxed MKVs.

Now - without denoise filter, after launching it in VLC and performing MPlayer Denoise 3D HQ post processing filter there, with maximum PP strength, the video looks quite nice. The only problem is the size doubles with quantizer value=22.
Yep, noise takes a lot of space, and you're encoding what is essentially 24p video as blended 50p. Yadif also causes artifacts that reduce compression ratio.

Here's my first try: sm_mdegrain_lsfmod.mkv
- Yadifmod with nnedi3 to bob the video for SRestore
- SRestore to recover the 24p (24000/1001 fps) video
- Stabilise with Depan
- Sharpen with LSFMod
- Denoise with MDegrain2 (Some lower frequency moving noise is left over though. Would need something heavier.)
Encoding at CRF 20 reduced video size by 26 % compared to the original sample (copied here for direct downloads).

Another, completely denoised version with reconstructed lines: sm_restored.mkv
- Used this script after srestore & depan
Encoding at CRF 20 reduced video size by 41 % compared to the original sample.

My encodes have a couple bad frames in the beginning. I had to use ffmpeg to make a lossless intermediate since ffms2 has difficulties with interlaced H.264 and I can't use other AviSynth source filters on Linux. Ffmpeg had some problems with the original sample too, although MPlayer and VLC seem to play it fine.

Last edited by nm; 16th April 2012 at 17:16.
nm is offline   Reply With Quote
Old 1st May 2012, 20:26   #5  |  Link
marek1712
Apprentice
 
Join Date: Apr 2012
Location: Poland
Posts: 6
Hello again. Sorry for such a delay with a reply.

I'd like to say it first - you clip looks ASTONISHING! I didn't know it's possible to achieve such quality with a original source like this.
I wanted to follow your guideline but it seems I'm stuck.
This is my script:
Code:
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\aWarpSharp.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\Deen.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\Depan.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\MaskTools-v1.5.5.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\mt_masktools-26.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\Mvtools2.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\dfttest.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\deblock.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\nnedi3.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\yadifmod.dll")
LoadPlugin("E:\Programy\Multimedia\MeGUI\tools\ffms\ffms2.dll")
Import("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\FastLineDarken.avs")
Import("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\DeHalo_alpha.avsi")
FFVideoSource("C:\Users\Markus\Desktop\SM_sample.mkv", threads=1)

yadifmod(edeint=nnedi3(field=-1))
Import("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\SRestore.avsi")
srestore()
Depan()

super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)

Deblock()
dfttest()
FastLineDarken()
deen("a3d",4,8,9)
Spline36Resize(last.width*2,last.height*2)
AddBorders(4, 0, 4, 0)
aWarpSharp(depth=12,blurlevel=4,thresh=0.2,cm=1)
FastLineDarken()
aWarpSharp(depth=6,blurlevel=4,thresh=0.7,cm=1)
Crop(4,0,-4,0)
FastLineDarken()
DeHalo_Alpha()
Spline36Resize(last.width/2,last.height/2)

Import("E:\Programy\Multimedia\MeGUI\tools\avisynth_plugin\LSFmod.v1.9.avsi")

MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
MeGUI's preview window crashes once I add the Depan() function (Event Viewer states that it's depan.dll's fault).
Besides - am I doing it right (loading scripts, DLLs, order of things and parameters)? I've encoded the video without the Depan() function and it looks like it isn't deinterlaced :/ No to mention the 0.5 fps on my C2D E6300...

Last edited by marek1712; 1st May 2012 at 20:31.
marek1712 is offline   Reply With Quote
Old 1st May 2012, 21:08   #6  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 7,915
sometimes less is more.

you are using awarpsharp twice.
fastlinedarken three times.
resizer two times.
first you add borders then you remove them.

you should use filters which are really really really needed. no wonder you get only 0.5 fps

Last edited by Atak_Snajpera; 1st May 2012 at 21:22.
Atak_Snajpera is online now   Reply With Quote
Old 1st May 2012, 22:01   #7  |  Link
marek1712
Apprentice
 
Join Date: Apr 2012
Location: Poland
Posts: 6
Uhm - I've mindlessly used the script suggested earlier. Once I get the quality of the latest sample provided by the nm - I'll try to organize it a bit and experiment - what is what and do I really need it.
marek1712 is offline   Reply With Quote
Old 3rd May 2012, 07:49   #8  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,998
Regarding your original question - MPEG-TS almost always contains a good deal of padding / nulls. This is because the final muxes usually need to be strictly CBR, and the video is basically never strictly CBR. Even pseudo CBR usually has a lot of variation on a frame by frame basis (which is a very good thing for quality). MPEG-TS itself also has inherent overhead. Transmitting things like the PCR, PMT, descriptors etc does use some bitrate.

For example, if you were shooting for a 5625kbps mux (commonly used in IPTV applications), you might encode the video at 4700kbps pseudo-CBR, and the audio at 384kbps pure CBR. This leaves an average of 541 kbps, which is about 9.6% of the full mux. The overhead of MPEG-TS will get you close to the full mux rate, and the nulls will pad you up to the full rate. I'd say this example is fairly aggressive with usage of the available bitrate. Your example is obviously from a much more conservative deployment. There may be additional bitrate consumed by error correction with things like DVB-T, but I'm really not sure.

It's ALWAYS better to err on the side of caution than it is to accidentally cause broadast issues!

Derek
__________________
These are all my personal statements, not those of my employer :)
Blue_MiSfit is offline   Reply With Quote
Old 3rd May 2012, 10:18   #9  |  Link
Ghitulescu
Registered User
 
Ghitulescu's Avatar
 
Join Date: Mar 2009
Location: Germany
Posts: 5,773
Not all MPEG-PS streams carry a "good load" of NULLs.
DVB-T has in Europe typically 2-3Mbps per video PID. Which is at most half of what a DVB-S stream usually carries (apart from teleshopping channels, but who records teleshopping?). A full movie (90min) can be broadcast in around or less than 2GB. Because of this lower bitrate post-processing must be performed, in particular deblocking.
__________________
Born in the USB (not USA)
Ghitulescu is offline   Reply With Quote
Old 3rd May 2012, 13:58   #10  |  Link
nm
Registered User
 
Join Date: Mar 2005
Location: Finland
Posts: 2,641
Quote:
Originally Posted by marek1712 View Post
MeGUI's preview window crashes once I add the Depan() function (Event Viewer states that it's depan.dll's fault).
Depan is a bit more complex than a single function call. You need motion data from DepanEstimate too.

Quote:
Originally Posted by marek1712 View Post
I've encoded the video without the Depan() function and it looks like it isn't deinterlaced :/
You need to bob the video with something like yadifmod(mode=1,edeint=nnedi3(field=-2))

It's also possible that direct input of interlaced H.264 video doesn't work properly with FFMS2. It didn't for me, and I had to encode the video losslessly to FFV1 with ffmpeg and then open it in AviSynth with FFMS2.

Quote:
Originally Posted by marek1712 View Post
No to mention the 0.5 fps on my C2D E6300...
Yep, that reconstruction script by Dark Shikari is pretty slow.

I used this script. Just a quick try, not optimal:

Code:
LoadPlugin("ffms2.dll")
LoadPlugin("mvtools2.dll")
LoadPlugin("removegrainbeta\RemoveGrainSSE2.dll")
LoadPlugin("mt_masktools-25.dll")
LoadPlugin("yadifmod.dll")
LoadPlugin("nnedi3.dll")
LoadPlugin("RepairSSE2.dll")
LoadPlugin("deblock.dll")
LoadPlugin("dctfilter.dll")
LoadPlugin("dfttest.dll")
LoadPlugin("DePan.dll")
LoadPlugin("DePanEstimate.dll")
LoadPlugin("Deen.dll")
LoadPlugin("aWarpSharp.dll")
LoadPlugin("fft3dfilter.dll")
import("Srestore.avsi")
import("deblock_qed.avsi")
import("dehalo_alpha_mt.avsi")
import("fastlinedarkenmod.avsi")

setmemorymax(1024)

# FFVideoSource("input.mkv",seekmode=0,threads=1,fpsnum=25,fpsden=1)
FFVideoSource("input.avi")
AssumeFPS(25)
AssumeTFF()

# Remember to crop vertically by mod 4 amounts when dealing with interlaced video
crop(12,4,696,572)

# Bob and deblend
yadifmod(mode=1,edeint=nnedi3(field=-2))
srestore()


# Stabilize
clp=last
temp  = clp.TemporalSoften(7,255,255,25,2)
inter = Interleave(temp.Repair(clp.TemporalSoften(1,255,255,25,2)),clp)
mdata = DePanEstimate(inter,range=1,trust=0,dxmax=16,dymax=16)
DePan(inter,data=mdata,offset=-1,mirror=15)
SelectEvery(2,0)

# Remove dust and noise
o=last
osup = o.MSuper(pel=2,sharp=2)
bv2  = MAnalyse(osup, isb=true,  delta=2, blksize=8, overlap=4, search=4)
bv1  = MAnalyse(osup, isb=true,  delta=1, blksize=8, overlap=4, search=4)
fv1  = MAnalyse(osup, isb=false, delta=1, blksize=8, overlap=4, search=4)
fv2  = MAnalyse(osup, isb=false, delta=2, blksize=8, overlap=4, search=4)
bc1  = o.MCompensate(osup, bv1)
fc1  = o.MCompensate(osup, fv1)
Interleave(fc1,o,bc1)
Clense(reduceflicker=false)
SelectEvery(3,1)
MDegrain2(osup, bv1, fv1, bv2, fv2, thSAD=400)

# Dark Shikari's anime reconstruction script
dfttest()
FastLineDarkenMod()
deen("a3d",4,8,9)
Spline36Resize(last.width*2,last.height*2)
AddBorders(4, 0, 4, 0)
aWarpSharp(depth=12,blurlevel=4,thresh=0.2,cm=1)
FastLineDarkenMod()
aWarpSharp(depth=6,blurlevel=4,thresh=0.7,cm=1)
Crop(4,0,-4,0)
FastLineDarkenMod()
dehalo_alpha_mt()
fft3dfilter(bt=3,sigma=8,sharpen=1.3)
Spline36Resize(last.width/2,last.height/2)
I had to use the slower fft3dfilter instead of fft3dgpu because of compatibility issues with my Wine environment.
nm is offline   Reply With Quote
Old 3rd May 2012, 17:12   #11  |  Link
kieranrk
Registered User
 
Join Date: Jun 2009
Location: London, United Kingdom
Posts: 707
Quote:
Originally Posted by Blue_MiSfit View Post
There may be additional bitrate consumed by error correction with things like DVB-T, but I'm really not sure.
Error correction is done at a different layer for these services and the bitrate you would be allocated by the operator would be before error correction was added.
kieranrk is offline   Reply With Quote
Old 5th May 2012, 02:51   #12  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,998
That makes good sense! It's funny, I was just reading ATSC specs the other day for fun, and came to the same exact conclusion regarding where error correction happens.
__________________
These are all my personal statements, not those of my employer :)
Blue_MiSfit is offline   Reply With Quote
Old 12th May 2012, 12:05   #13  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,113
Quote:
Originally Posted by Ghitulescu View Post
Not all MPEG-PS streams carry a "good load" of NULLs.
DVB-T has in Europe typically 2-3Mbps per video PID. Which is at most half of what a DVB-S stream usually carries (apart from teleshopping channels, but who records teleshopping?). A full movie (90min) can be broadcast in around or less than 2GB. Because of this lower bitrate post-processing must be performed, in particular deblocking.
Can you recommend a good "one size fits all" deblocker and settings for exactly this purpose?

I usually reencode my DVB-T captures to remove the station logo. So far I have never used any deblocker mainly because I still watch my movies on a CRT TV which is very forgiving when it comes to blocking.

Recently I watched one of my reencoded captures at a friend's house on an LCD, and I was quite shocked about the very visible blocking. So it might be a good idea to add deblocking routinely. But since I cannot judge the result on my CRT I need some recommendations.

The settings should be a good compromise between enough deblocking and not destroying all the details. And it should be fast.

Deblock_QED would probably be too slow for me. MPEG2Source ("", cpu=4) or Deblock(30) sound reasonable. Some people recommend the MSU Smart Deblocker, but I think this would also be too slow.


Thanks
manolito
manolito is offline   Reply With Quote
Old 12th May 2012, 12:51   #14  |  Link
nm
Registered User
 
Join Date: Mar 2005
Location: Finland
Posts: 2,641
Quote:
Originally Posted by manolito View Post
Deblock_QED would probably be too slow for me.
Shouldn't be too slow for reencoding. With a decent CPU (2.4 GHz Core 2 or better) it runs easily in real time for SD videos, in a single thread.

It's the only deblocking filter I've found satisfactory for MPEG-2 DVB sources.
nm is offline   Reply With Quote
Old 13th May 2012, 16:25   #15  |  Link
marek1712
Apprentice
 
Join Date: Apr 2012
Location: Poland
Posts: 6
@nm - I'd like to ask you another question. You've mentioned that ffms2 doesn't like interlaced H.264 (it's even mentioned in the readme). Is the symptom similar to video going back like 0.2s every second or two?
I've made simple recompression to FFV1 with ffmpeg (with vcodec ffv1 parameter). Is it enough or should I add something else?
BTW - nonOCed i7-2600k reaches as much as 1.8fps with this heavy script (and CPU utilization is 40% max)
marek1712 is offline   Reply With Quote
Old 13th May 2012, 17:33   #16  |  Link
nm
Registered User
 
Join Date: Mar 2005
Location: Finland
Posts: 2,641
Quote:
Originally Posted by marek1712 View Post
@nm - I'd like to ask you another question. You've mentioned that ffms2 doesn't like interlaced H.264 (it's even mentioned in the readme). Is the symptom similar to video going back like 0.2s every second or two?
Yep, that's what I usually get even with fpsnum and fpsden set. I guess this is what the documentation describes as:
Quote:
one of the fields (odd or even) may "jump around"
Quote:
Originally Posted by marek1712 View Post
I've made simple recompression to FFV1 with ffmpeg (with vcodec ffv1 parameter). Is it enough or should I add something else?
That's enough.
nm is offline   Reply With Quote
Old 15th May 2012, 01:16   #17  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,113
Quote:
Originally Posted by nm View Post
It's the only deblocking filter I've found satisfactory for MPEG-2 DVB sources.
Anyone compared Deblock_QED to SmoothD2 ?


Cheers
manolito
manolito is offline   Reply With Quote
Old 28th May 2012, 16:01   #18  |  Link
marek1712
Apprentice
 
Join Date: Apr 2012
Location: Poland
Posts: 6
Sorry for dragging it for so long but there's another problem.
After some time the resolution of the source material has changed to something strange... 1047x576 (something which isn't even a 16:9 aspect ratio).
Do I have to crop in the H and V part (and get rid of the vertical part of the image) or can I remove the black borders to make it 4/3 somehow.
I've tried to calculate the aspect ratio of 1.333333333333 and came with something like 768x576.
Sample:
marek1712 is offline   Reply With Quote
Old 28th May 2012, 16:24   #19  |  Link
nm
Registered User
 
Join Date: Mar 2005
Location: Finland
Posts: 2,641
Quote:
Originally Posted by marek1712 View Post
Sorry for dragging it for so long but there's another problem.
After some time the resolution of the source material has changed to something strange... 1047x576
That's not the actual resolution of source frames but size of frames scaled according to the SAR (sample aspect ratio) of the stream. Apparently the broadcaster used the ITU-R BT.601 compatible SAR of 16:11, which is calculated for 704x576 video: 704*16/11 = 1024. Basically that means that at least 8 pixels (more if there's overscan) would be cropped from each side of the source frame when the video is displayed on a 16:9 screen.

I'd keep the SAR and crop the black borders of 4:3 content that was encoded anamorphically. If the resulting non-standard frame size is a problem, resize to 720x576 and change the SAR to either 16:11 or 64:45.
nm is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 21:48.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2025, vBulletin Solutions Inc.