DV to Xvid or h264 contest

[Terka]

Some of us convert DVD, others DV, ... as input format.
We usually do:
1.deinterlace
2.denoise
3.resize
4.encode
I wil upload an DV video to let u try get following output:
6000kbps as good quality as you can get
encoding speed cca 1fps on 3GHz machines (this means not 5 or 10 times slower)
50p output

why doing this?
overall strengh of chain ... you know
Many of us are doing similar conversions. How to divide the overall time? Is it better to use slower denoiser or better deinterlacer, or more pass compression, some matrix...

[Garda]

Where is the dv file ? :)

[Blue_MiSfit]

So, you're looking for a workflow that will output 50p MPEG-4 @ 6000kbit from a 25i PAL DV source, at speeds no less than 1fps on an average 3GHz box?

Definitely doable. Let's see the source. Gentlemen, start your CPUs.

~MiSfit

[Terka]

there is the file
http://rapidshare.com/files/25061661/Psi.rar
tried:
tdeint(mode=1,order=0,emask=TMM(mode=1,order=0))
fft3d
xvid
------------
1.is there any fast filter to remove bob() "flickering"
2.how to remove small camera shakin - via deshaker or depan or something similar - without resize, just crop the black borders. I think it could improve the compressibility.

[Garda]

http://rapidshare.com/files/25096331/doom.mkv.html

1.deinterlace -> mvbob
2.denoise -> DeGrainMedian
3.resize -> Spline36Resize
4.encode -> @ x264 @ MeGui @ HQ-Insane

[*.mp4 guy]

script:
avisource("psi.avi").coloryuv(levels="tv->pc")

yadif(mode=1).assumefps(50)

pointresize(1440, 576)#
dctlimit(dct4=0.75)#vertical lowpass to remove flickering
pointresize(720, 576)#

spline36resize(640, 480)
fft3dfilter(sharpen=1, bw=16, bh=16, ow=8, oh=8, bt=1, sigma=0.5, plane=4)
gradfun2db()

X264:
--pass 2 --bitrate 6000 --stats ".stats" --keyint 240 --min-keyint 48 --ref 8 --mixed-refs --no-fast-pskip --bframes 8 --b-pyramid --b-rdo --bime --weightb --direct none --filter -2,-4 --subme 6 --analyse all --8x8dct --merange 24 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "C:\Documents and Settings\\Desktop\Psi\script.mkv" "C:\Documents and Settings\\Desktop\Psi\script.avs" --deadzone-inter 6 --deadzone-intra 8

This should run at ~3+ fps first pass, ~2+ fps second pass on your system, and look like this (http://www.mytempdir.com/1289345).

If you have trouble with banding with these settings you shold consider lowering the resolution to 512*384 and using a cqm.

[Terka]

mp4 guy,
please comment your script-
why:
1.coloryuv(levels="tv->pc") ?
2.is the resize necesary ?
or if yes, is useful to remove the pointresize(720, 576)#
(because of next resize-spline)
-------------
one more question, why not to leave at original size 720x576 and if need resize while playing?

[*.mp4 guy]

1. Because the source is tv levels, which look ugly, so I switched to pc levels.

2. The pointresize(720, 576) call is neccesary, it is part of the lowpassing.

2. The spline36resize(640, 480) call is not necesary, but will help give you better results after compresion, because your source doesn't have any real detail that can't be kept at that resolution, and because your source isn't very compressable, so downsizing would look better even if you lost apriciable detail, which in this case, you won't.

[Terka]

i see.
2more questions:
------------------
so if i understand, there are 2 possibilities to improve final quality:
1.not to resize - incerease bitrate
2.resize - keep same bitrate
you mean that method 2 has similar results to 1. because of
not high quality source
------------------
resize to 640x480 has not the same ratio as original.
if i film a circle and play the dv source on monitor, it wont be
circle?-and what about playing on tv?
thank you!:thanks: :thanks:

[*.mp4 guy]

1. That would work, but it would be better to resize, AND increase bitrate.

2. yep, lower resolution+lower bitrate will look similar to higher resolution+higher bitrate (as long as the bitrates aren't too far apart) becuase not all of your sources resolution has usefull information in it.

3. 720*576 is an anamorphic resolution (non-square pixels), so in order for a circle to look like a circle it has to be resized to a square pixel resolution during playback (usually done automagically, though not always). I resized it to a "square pixel" resolution, so no resizing is needed during playback to get a correct looking picture, I did this because square pixels compress better then non-square ones, they also look subjectively better at equivelent resolutions, and they have less compatibility issues. It is possible that your camera records in 720*576 square pixels, in which case what I did would mess up the image, but I have never heard of a camera that used a 720*576 square pixel format to record video.

[Terka]

thank you.:thanks:
increase bitrate-(if it is possible to say) how much?
i tried use xvid constant quantitizer-usually Q3. i do not need to match exact filesize- can this spare time because of the 2nd pass is not needed?
Is it possible to avoid artefact around moving objects-for example using some special matrix?
--------------------------------
some more thoughts
if you look at the clip - the car and rail in background.
it is jumping up/down-wasting bitrate
look at http://forum.doom9.org/showthread.php?t=96344
a long time ago ive seen some filter (special for anime)
which works like:
if 2 frames are less different then treshold, make them the same.
this could be used for the idea in http://forum.doom9.org/showthread.php?t=96344.
big object vs small, center of picture vs borders .
deshaking could help too?

[wozio]

1. Because the source is tv levels, which look ugly, so I switched to pc levels.

Shouldn't we don't change levels during encoding, leave at they are in source and let renderer or display do it's job of getting this properly displayed? All real encoded sources are in tv scale (by "real" I mean dvds, tv broadcasts, and so on). If we change to pc scale during encodes then all blacks below 16and whites above 235 will be cut on properly calibrated display and contrast will be higher.

[*.mp4 guy]

Tv levels always have black at 16, and white at 235 when displayed on a PC, unless the specific decoder (IE, a DV or Mpeg2 decoder) fixes them preemptively, H.264 and Mpeg4-asp decoders never preemptively fix tv levels for display on a pc. Most HDTV's are capable of correctly displaying pc colors.

@ Terka

1 - bitrate is all about how much space or quality you are willing to give up, I can't tell you what bitrate to use.

2 - If you do not need to achieve a certain filesize, then you don't need to use two passes, one pass will have almost exactly the same quality as 2 passes at the same filesize

3 - maybe, if the artifacts you are talking about are made during compression, it is atleast possible to reduce them.

4 - that is from the deinterlacing, mcbob might remove it, but it is very very slow, which is why I used Yadif()+lowpassing instead, I tried mvbob, but It didn't remove any more of the interlacing artifacts, and was slower.

[wozio]

Tv levels always have black at 16, and white at 235 when displayed on a PC, unless the specific decoder (IE, a DV or Mpeg2 decoder) fixes them preemptively, H.264 and Mpeg4-asp decoders never preemptively fix tv levels for display on a pc. Most HDTV's are capable of correctly displaying pc colors.
Yes, decoders don't expand and should not do it (I've never seen any which does by default) but on my pc VMR9 always does. We may argue if this is correct behaviour (instead of leaving at tv levels and let display handle this via calibration) but when yuv video is encoded at 0-255 and renderer or display expects 16-235 then we loose details and contrast is shifted.

In my opinion we must stick with video/tv standard (16-235) on our encodings. Otherwise we will have different results from orginal and recoded video.

Would it be better when we would have all possible content encoded at the same standard (tv levels from tv, dvd, hd dvd, dv captures) and also the same standard in our reencodings? Then we will have consistent and expactable results in all possible scenarios (how standalone players decode such yuv data with 0-255 levels?).

TV levels can be expanded to match pc levels during playing at very good quality, this is done very well by vmr9, haali renderer, ffdshow with rgb output at least...

[henryho_hk]

Should we standardize the DV "decode" codec also? They behave vastly differently.... luma range, graininess, chroma interpolation, etc.

[*.mp4 guy]

Overlay rendering is faster then VMR9, and usually doesn't change levels, my overlay render is exactly the same quality as VMR9 (aparently some grafics cards have bad overlay renderers, though) and does not perform any levels adjustment, which is exactly the way I wan't it. I keep everything in pc levels, because this allows me to keep more of the source then keeping everything in tv levels, converting pc->tv levels is much more lossy then converting tv->pc levels (which is very nearly lossless). Quite a few dvd's I have, have color ranges that excede the tv range (though they don't quite use the full pc range either), If I wanted to comply with tv level standards i would have to truncate, or squich there color range, so I choose to use the pc range instead, it is IMO an all around better system, having 36 unused colors that you encode anyway is braindead, and leads to interoperability nightmares.

[henryho_hk]

XviD Teegdeck's >45% HQ Preset @ 5474kbps ... a bit undersized as the clip is too short.

Download link: http://rapidshare.com/files/25911903/psi_yadif_TGHQ-45_br_6000.zip.html

Command-line:

xvid_encraw_20070203.exe -zones 0,q,3,KO -threads 1 -progress 100 -max_key_interval 500 -packed -quality 5 -vhqmode 1 -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -qtype 1 -qmatrix eqm_v3hr_rev1.xcm -nochromame -turbo -pass1 psi_yadif.pass -type 2 -i psi_yadif.avs
xvid_encraw_20070203.exe -zones 0,w,1.0,KO -threads 1 -progress 100 -max_key_interval 500 -packed -quality 6 -vhqmode 4 -qpel -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -qtype 1 -qmatrix eqm_v3hr_rev1.xcm -imin 3 -imax 4 -pmin 3 -pmax 5 -bmin 3 -bmax 5 -chigh 20 -clow 7 -bvhq -bitrate 6000 -pass2 psi_yadif.pass -stats psi_yadif.stats -type 2 -i psi_yadif.avs -avi psi_yadif_TGHQ-45_br_6000.avi


Script:

import("Soothe_mt2.avsi")
import("LimitedSharpenFaster.avsi")
loadplugin("degrainmedian.dll")
loadcplugin("yadif.dll")
avisource("psi.avi",audio=false,fourcc="cdve",pixel_type="YV12") # Cedocida 0.1.7 DV codec
yadif(order=0,mode=1)
a=last.spline36resize(640,480,8,0,704,576).degrainmedian(mode=1)
b=a.limitedsharpenfaster()
Soothe_mt2(b,a,20)
degrainmedian(mode=2)

[wozio]

So you have dvd which is coded in 0-255 but blacks are referenced in 16 and whites are in 235. During your encode you are extending this 16-235 to 0-255 so everything below 16 and above 235 is lost. Is it correct? Now you are sending this information to renderer which... does what? Expects on input signal that blacks are at 16 and whites are at 235 so once more it extends levels and one more time information is lost this time with real image information??? Or maybe renderer expects that image is encoded in 0-255 and leave it as it is. But then when you send to renderer signal which IS NOT coded at 0-255 but in 16-235 (for example any dvd) you will have original levels and result will be different than your reencode.

I'm not getting it, maybe some drawin would be better ;)

This is industrial standard that yuv video data is coded at 16-235, we SHOULD NOT change it in our encodings. If renderer expects on input that yuv image is in full 0-255 then it is not
following general rules and should not be used... In fact all current renderers excpects 16-235 range on yuv input and extends it to 0-255 so you have incorrect double conversion in your encodings (this applies to all: VMR9, haali renderer, VMR7, overlay mixer - at least on my system results from all of this are almost identical).

http://www.fourcc.org/fccyvrgb.php#mikes_answer

Frome there:

defined range for Y is [16,235] (220 steps) and the valid ranges for Cr and Cb are [16,239] (235 steps) These are normalized ranges and when the data is used, 16 is added to Y and 128 is added from Cr and Cb to de-normalize them. The reason for this is that control and colorburst information is stored in the components along with the luminence and chromiance data which is why the full range of [0,255] is not used. There is no "rumor" about this being correct. It is a defined standard for YCrCb video. I wouldn't be surprised if certain PC programs are using a variant where the entire range of [0,255] is used but technically it is not YCrCb. If you use YCrCb for video purposes, its probably going to be CCIR 601 compliant.

[henryho_hk]

XviD Teegdeck >45% Preset again. 5894kbps. This AVS is much more luxurious (MCBob+fft3dfilter).

Download link: http://rapidshare.com/files/25915731/psi_mcbob_fft3dfilter_TGHQ-45_br_6000.zip.html

Script:

import("mcbob.avsi")
loadplugin("fft3dfilter.dll")
avisource("psi.avi",audio=false,fourcc="cdve",pixel_type="YV12")
assumebff()
mcbob()
spline36resize(640,480,8,0,704,576)
fft3dfilter(sharpen=0.5, sigma=0.5, plane=4)

[*.mp4 guy]

So you have dvd which is coded in 0-255 but blacks are referenced in 16 and whites are in 235. During your encode you are extending this 16-235 to 0-255 so everything below 16 and above 235 is lost. Is it correct? Now you are sending this information to renderer which... does what? Expects on input signal that blacks are at 16 and whites are at 235 so once more it extends levels and one more time information is lost this time with real image information??? Or maybe renderer expects that image is encoded in 0-255 and leave it as it is. But then when you send to renderer signal which IS NOT coded at 0-255 but in 16-235 (for example any dvd) you will have original levels and result will be different than your reencode.

I'm not getting it, maybe some drawin would be better ;)

This is industrial standard that yuv video data is coded at 16-235, we SHOULD NOT change it in our encodings. If renderer expects on input that yuv image is in full 0-255 then it is not
following general rules and should not be used... In fact all current renderers excpects 16-235 range on yuv input and extends it to 0-255 so you have incorrect double conversion in your encodings (this applies to all: VMR9, haali renderer, VMR7, overlay mixer - at least on my system results from all of this are almost identical).

http://www.fourcc.org/fccyvrgb.php#mikes_answer

Frome there:

No, here is what happens (and don't be thick and purposefully missinterpret me this time). I have DVD's that have colors outside of the 16-235 range, and I leave them alone, My renderer expects yuv video to have the full range, and renders it without any changes, just like RGB, and I am happy with this because it keeps things simple; I would rather make sure that the colors are correct myself before encoding, and be confident that they will displayed as I encoded them.

[edit]
anyway, this is all moot, as the reason I put the levels conversion in originally is because the original video looked like it had been put through two 16-235 clamp/squish operations instead of just one (I can't be sure thats the case, which is why I didn't mention it before, all I can be confident of, is that the colors look closer to the range I would expect out of a dv camer after the tv->pc levels switch).

[wozio]

I have DVD's that have colors outside of the 16-235 range
These are not colors, just trash to be ignored.

and be confident that they will displayed as I encoded them.
You can't be confidend couse on other machine with correct renderer ;) result will be different.

the colors look closer to the range I would expect out of a dv camer after the tv->pc levels switch).
So which one is better or maybe I must name it not better but pure, original, as it was from the source, whatever:

- without levels tv->pc:
http://images6.theimagehosting.com/org.79a.th.jpg (http://server6.theimagehosting.com/image.php?img=org.79a.jpg)

- or this one with levels tv->pc applied:
http://images6.theimagehosting.com/levels.th.jpg (http://server6.theimagehosting.com/image.php?img=levels.jpg)

Take a look how many information is lost, on dog's black fur and girl's white shirt.

I must admit, second looks BETTER but this is not levels fix, this is processing by change of the levels of the original image to boost contrast.

[*.mp4 guy]

There isn't any lost information (well, actually there is, some minor highlight detail was removed, but shadow detail was untouched), Your monitor has poor black level detail (or you have bad eyesight), so it looks like information has been lost, but I assure you, its still there.

can't be confidend couse on other machine with correct renderer ;) result will be different.
I don't give any of the video I've got to anyone else.... So I don't care if they would look bad on your PC.

[wozio]

There isn't any lost information (well, actually there is, some minor highlight detail was removed, but shadow detail was untouched), Your monitor has poor black level detail (or you have bad eyesight), so it looks like information has been lost, but I assure you, its still there.

Oh come on :) my monitor is OK, I checked actual levels graph in photoshop of both screenshots and really you can believe me, info below 16 and above 135 is lost.

I don't give any of the video I've got to anyone else.... So I don't care if they would look bad on your PC.
In that case de gustibus non est disputandum

[*.mp4 guy]

The information below 16 that is lost is due to haloing, seriously did you even look at the results.
http://img411.imageshack.us/img411/5909/org79a2gg1.png

1. Minor detail above 235, that is lost, possibly due to ccd noise, or quantization rounding.

2,3,4. Very obviouse blacker then black haloing, which accounts for all of the values below 16 that I could find in that area.

In that case de gustibus non est disputandum
All bow before the amazing highdchool graduate :shock: :Awe:

[wozio]

There isn't any lost information (well, actually there is, some minor highlight detail was removed, but shadow detail was untouched)

The information below 16 that is lost is due to haloing, seriously did you even look at the results.

1. Minor detail above 235, that is lost, possibly due to ccd noise, or quantization rounding.

2,3,4. Very obviouse blacker then black haloing, which accounts for all of the values below 16 that I could find in that area.

Could you please decide if there is any information lost or not? I'm quite lost in this academic-after-highschool-graduate argumentation. ;)

I know that on this particular sample all clipped details in shadows and highlights are not important. That's not my point here.

I'm saying that we should not do tv->pc conversion during encoding because this will lead to, how did you say this, "interoperability nightmares". How do you play dvds on your pc when your renderer does not extend tv to pc and dvds are in fact coded in tv standard? Are you changing levels on the fly by ffdshow or change renderer? And when you playing your encodes you disable ffdshow or change back renderer? Will you call it interoperability heaven?

Do how you like at your own encodes but you will teach all people here to extend levels during encoding and they will go back here and ask why the hell their rips look different than source dvds, when playing on the same standalone dvd/divx player where it is quite hard to change renderer...

[*.mp4 guy]

This is pointless, I shouldn't have kept this going even this long, sorry for disrupting your thread terka.

[henryho_hk]

Some DV codecs perform kinda brightness conversion too. That's why I propose to standardize the DV codec (e.g. open source Cedocida 0.1.7) at the very beginning. Unfortunately, no one seems care about it.

[Terka]

*.mp4 guy > is it possible to open the *.mkv done by you in vdub?
if yes, please how?

[*.mp4 guy]

You can load it with Directshowsource() in an avisynth script, then load that script with virtualdub. Or you could demux the mkv, and extrac the raw avc track, then load it using dgavcdec, in an avisynth script.

[Terka]

thank you, ill try Directshowsource()..
..tried, but no success

-----------------vdub err-----------------
Directshowsource. Could not open as video or audio.

Avisynth open failure
DirectShowSource: Could not open as video or audio.
Video returned: "DirectShowSource: couldn't open file ...
Cannot play back the file. The format is not supported."

same for audio
--------------------------------------

i tried load xvid this way-successfully

[*.mp4 guy]

it works for me, you probably just don't have the needed directshow filters.

[Terka]

*.mp4 guy
after reinstaling the pc i am not able to load the dctlimit function. What dlls are needed to load - i used dctfilter, but
got error: dctfilter does not have a nemed argument "chroma"
Could you please post the whole script including dlls as attachement? Thank you very much! Terka

[*.mp4 guy]

Dctlimit requires foxyshadis' modified dctfilter (http://foxyshadis.slightlydark.com/random/DctFilter.zip) (modified version can discard chroma, and handles memory better).

[Terka]

thank you.
but after loaded version you linked, another error appears:
there is no function named "average" :mad:

[Dark Shikari]

thank you.
but after loaded version you linked, another error appears:
there is no function named "average" :mad:
You need the average filter, search doom9 for it :p

[Terka]

so simple. thank you, i thought it a function