Problem: Transcoding VC1 > x264

[Rodger]

Hi all,

I have a problem when trying to transcode from VC1 to x264.
Pretty early in the movie the encoded Video goes from everything perfect to this:

http://www.bilder-space.de/thumb/08lJoNh7yFNEX2P.jpg (http://www.bilder-space.de/show.php?file=08lJoNh7yFNEX2P.jpg)

My avs-script just looks like that...
DirectShowSource("E:\Die Fremde In dir\4355.m2ts")
#AviSource("6974v1.avi")
#changefps(25, 1)
#AddBorders(0, 96, 0, 96)
#LanczosResize(1280,720) # Lanczos (Sharp)

It is to be said that the video has correct length, but is MILES from the filesize it should be max. This time for example: 44,4GB
It should be around 7,4GB!!!

Any Idea what´s wrong?

Greetings
Rodger

[Dark Shikari]

The bitrate goes up because once your script starts feeding x264 corrupt frames, x264 is no longer able to stay within the bitrate bounds (since it takes loads of bits to encode those random-noise frames).

[LoRd_MuldeR]

If you open the source AVS script in VirtualDub and let it play, does it show up correctly all the way to the end?

[Rodger]

If you open the source AVS script in VirtualDub and let it play, does it show up correctly all the way to the end?

No It doesn´t! The same (time-)point when it "crashes" to that noise-picture.
I removed the audio from the stream and still it "crashes".

This is the second time I experience THAT Problem with VC1-Streams. Last Time I thought it was a corrupt vc1 stream.

And again...it is corrupt. What the hell does AnyDVD do there?! :mad:
H264 Titles work perfectly!

Sorry to bother you guys...should have checked this myself before. But anyway...anybody else experiencing this...fault/error?

[CruNcher]

Rodger for me it looks like the Hard Drive was damaged where the Frame was written :( and it couldn't write it fast enough when it reallocated (sometimes half the slice is written and the rest like yours showing this pattern) you should really check your Hard Drive (tough that is when this happens for writting H.264 dunno if the patern looks the same for VC-1 if you say so then something must have been gone wrong @ transfering from Blu-Ray to Hard Drive :(

[Adub]

Yeah, it's possible you have a damaged rip. Make sure you are using the latest AnyDVD, and read the changlog for possible bug fixes that concern you. Frankly, I think the harddrive idea is a little extreme, but is not outside of the realm of possibility.

[Rodger]

This ONLY happens to VC1 streams?
I don´t think so! I reapeat...ALL h264-titles work 100% perfectly!
2 rips created on the same day and only the one created in vc1 has the error...it is impossible this repeats each time.

Anyway...I won´t blow this hint...chkdsk is already running on this 384GB big partition :rolleyes:

And YES...I´m using the latest Version of AnyDVD HD.

[CruNcher]

Rodger after that check your harddrive with a hdtune http://www.hdtune.com/ error scan you imediatly see where problems occoured graphicaly (on the physical disk)
Sure tough it could be also a Problem of AnyDVD HD look for someone who tried to process the same Blu-Ray and if he got the same errors if that is the case you can be almost sure it's a AnyDVD HD Problem (maybe something in the encryption of this Blu-Ray has changed).

[Rodger]

"Auf dem Datenträger wurden keine Probleme festgestellt."
Means: No Errors found! :p

By the way...the Microsoft own tool is MUCH more sensible to hardware errors.

So....it´s definetly NOT the harddisk.

[Revgen]

Try FFMPEGSource?

Directshow may be the problem.

[Rodger]

No, I wished that was the problem...it´s sadly the original stream...right after the ripping.

I have a certain idea what it could possibly be.
I DEMUX via tsMuxer and Mux the used stream back together with this program. I´ll have an eye on this....

[jeffy]

"Auf dem Datenträger wurden keine Probleme festgestellt."
Means: No Errors found! :p

By the way...the Microsoft own tool is MUCH more sensible to hardware errors.

So....it´s definetly NOT the harddisk.
You did a full scan, not a quick one, right? A quick one is not a reliable test.

[Rodger]

You did a full scan, not a quick one, right? A quick one is not a reliable test.

Coming to the Microsoft Tool...there IS NO quick scan option! ;)
And of course I didn´t use the quick scan option of HDtune.

By the way...It´s a Raid0-Array of two pretty new WD3000GLFS...the VelociRaptors :D

[jeffy]

@Rodger: thanks for explaining, yes, I meant the option in HDTune. BTW, what tool it this? By the way...the Microsoft own tool is MUCH more sensible to hardware errors. And one more off-topic question, please: how fast, how hot and how quick -edit- quiet such the VelociRaptor RAID array is?

Thank you.

[Rodger]

Well...it has no real name anymore...it was once "chkdsk" alias CheckDisk, then there was a tool named "Scan Disk". Now it´s more or less a snap-in tool of vista...
http://www.bilder-space.de/thumb/GCeEJTbFBxEXVDU.png (http://www.bilder-space.de/show.php?file=GCeEJTbFBxEXVDU.png)

Here...have yourself a view of what a "VelociRaptor-Raid" is able of.
http://www.bilder-space.de/thumb/Fp81PKAUg67xNOH.png (http://www.bilder-space.de/show.php?file=Fp81PKAUg67xNOH.png)
Addionally I may say...THIS is the result of a Raid at a Intel ICH8R-Controller...a "real" controller would bring even better results.
What you can´t see off the benchmark is...you FEEL how fast it is...you can feel how much faster Vista boots without measuring.
This is a result of the massively improved AccessTime, which also shows up in the everyday work. Faster than that can only be achieved with SSDs...but that´s on another planet when it comes to payment ;)

[Blue_MiSfit]

What happens if you use eac3to to remux the M2TS into an MKV, containing only a VC1 video track?

~MISfit

[jeffy]

@Rodger: A big thank you for your answers. Now back on the topic, does the ripped M2TS play at the point where the error occurs without any problem?

[Rodger]

I FOUND "THE BAD GUY"!

It really is tsMuxer! Don´t Rip+Demux with tsMuxer from Blu-Ray.
You really need to copy the original m2ts-file(s) first before you can select the single Streams you want/need.

What a shame! Don´t ask me why...but to me that really has quiet an impact on the usuability.

Now I need to save the whole lot of Audio-Streams+Subtitles I really don´t need. But anyway...first there comes the final product.

[CruNcher]

I would say it is a interoperability problem with tsmuxer and VC-1 ES streams, as you said H.264 titles are working

[Selur]

btw. just tested a small method some days ago an a .m2ts(vc-1,ac3) Trailer which worked fine:
1. remux .m2ts with gdsmux to .mkv
2. decode with ffmpeg and pipe to x264
ffmpeg -i "Path to .mkv" -v 0 -vcodec rawvideo -pix_fmt yuv420p -f rawvideo - | X264 ... (remember to specifiy resolution&framerate)
worked fine. It also seems to work when not remuxing and halfing the framerate in ffmpeg (i.e. -r 12 for a 24fps clip), which is needed since ffmpeg doubles each frame,..
Not totally sure how well this works for whole movies, since I one tested it on <5min trailers but may be it's worth a try. :)
(tried this also under Linux, but couldn't get it to work, which is probably due to missing patches for ffmpeg)

Cu Selur

[CruNcher]

Selur try -copyts that should stop doubling the frames :)

[Selur]

thanks :) Must have overlooked that option. :) (ffmpeg's docu is a hell for it's own ;))
-> doesn't work when piping to x264 gives a 'av_interleaved_write_frame(): Error while opening file' but it works when encoding directly with ffmpeg.

[Selur]

found a solution '-vsync 0' saves the day :)

[lithiumus]

I have a different problem with VC-1 transcoding to x264... I'm using an Anime trailer from my HD-DVD to test. The problem is that at the same approx. cluster of frames, I get some serious blocking artifacts. I've tried versions of x264 from 837 to 938 and many different variations of profiles all resulting in some form of blocking. I've used WMV9 and libavcodec with similar results and I've tried Megui and Ripbot.

I must have done 20+ encodes all with similar results.

I have a friend who has Mainconcept and I'll have him give it a try.

This is the distorted frame
http://www.nsxprime.com/photopost/data/500/Cap1.JPG

This is the next frame
http://www.nsxprime.com/photopost/data/500/Cap2.JPG

Any suggestions? I'm open to trying anything I haven't tried before...

[Dark Shikari]

Are you sure there's no such problem in the source?

[poisondeathray]

@lithimus:

If you use the same script with, say, XviD, does the blocking still occur (i.e. is it an encoding to x264 problem or a VC-1 decoding problem)?

What are you using for your script?

Maybe you could post a small sample?

[Ranguvar]

Try FFmpegSource (seekmode=-1 may help stability), or the Windows DMO decoder.

[lithiumus]

@lithimus:

If you use the same script with, say, XviD, does the blocking still occur (i.e. is it an encoding to x264 problem or a VC-1 decoding problem)?

What are you using for your script?

Maybe you could post a small sample?


Great questions and thanks for the help. This is a sample. I'm trying to convert my HD-DVD to Blu-ray without loss of quality. The bit-rate is the same as the source...

program --pass 2 --bitrate 10700 --stats ".stats" --level 4.1 --keyint 24 --min-keyint 2 --ref 3 --mixed-refs --no-fast-pskip --bframes 3 --b-rdo --bime --weightb --direct auto --subme 7 --trellis 1 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 14000 --vbv-maxrate 40000 --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input" --psy-rd 0 --mvrange 511 --aud --nal-hrd --sar 1:1

I've never tried Xvid so I'll try that... will post results when it's done. I'll find a way to host a clip.

[lithiumus]

Are you sure there's no such problem in the source?

I'm pretty sure but I'm only saying that from visual inspection, I don't know if there is something strange in the VC-1 encode that is causing an encoder to misfire or something. The raw stream after a demux looks visually clean.

Here is that same distorted from taken from the source...

http://www.nsxprime.com/photopost/data/500/cap3.JPG

[Dark Shikari]

An image is rather useless... a clip of the stream (uploaded to Mediafire or similar) would be dramatically more helpful.

I'll make a guess that its VBV doing it, by the way.

[lithiumus]

Decoding seems fine. I can view the source via MPC or Total Media Theatre and it's flawless. Once I encode it using x264 this happens. Both with libavcodec and WMV9 (which I assume is the windows DMO decoder). When I watch the source Gspot tells me that I'm using DMO and it plays flawlessly.

[lithiumus]

Thanks. Let me get something uploaded for testing...

I tried... VBV Max at 24000 and 40000 and VBV buffer at 9000 and 14000.

[lithiumus]

Uploading, will take a few hours...

[CruNcher]

it looks Visualy exactly like the VBV error i reported once :)
tough i think Gabriel fixed that and now it's back ??

[Dark Shikari]

it looks Visualy exactly like the VBV error i reported once :)
tough i think Gabriel fixed that and now it's back ??I fixed that in r930 I think...

[CruNcher]

Huh no it was much older i think the initial fix was this one of yours http://pastebin.com/f4f4d47ea after that i think Gabriel added something to it and then the problems where gone, at least for me @ that time i didn't experienced this anymore. Tough i didn't use VBV for a long time since then anymore only just recently for the ETI compare and their i had not such problems @ all, but true this problem is really a lucky thing to get i had it with some source in some scenes but on other stuff there where non of these borked frames @ all it felled like it happens randomly without any warning.

[lithiumus]

I rar'd the MKV that I was working from (2 parts) I eac3to to demux from the original EVO (video only) and strip the pulldown. Let me know if you'd rather the original EVO and I can post that too.

Here is the first part
http://www.mediafire.com/?smktpkveyia

Second part and sample will be done and posted after I wake up...

The Xvid encoding completed and is flawless. I'll post that sample as well.

[Dark Shikari]

I didn't want the source file--just a small clip of the encoded h.264 file with the problem. You don't need to upload the whole clip, just use mkvmerge to split out the few problematic megabytes.

[lithiumus]

Got it. That's simple enough...

[lithiumus]

Here is the encoded clip with the problem at about the 2 second mark. You'll recognize the frames...

http://www.mediafire.com/?5urzoun7euf

[Dark Shikari]

Yup, its VBV... but it doesn't make any sense why its acting up like this.

Can you post a clip of the source (short one), which, when encoded independently with a specific set of first and second pass settings (pasted with the clip), results in these issues?

[lithiumus]

Sure thing. I'll grab the approx. same clip (source), post that and since there were a lot of recent Megui x264 updates, I'll use the most recent build and do 2 pass and post both 1st and 2nd pass settings along with the resulting encode. I use Megui's Automated 2 pass.

What I noticed when I tested different x264 releases was that certain releases would distort certain frames i.e. that one posted or one after or one before, etc. I'm not sure if it was random because I did so many tests, they became less and less scientific as I past the 2am mark...

[poisondeathray]

Did you use the same .avs script with the successful XviD encode? or did you use another process? Was the filter chain identical?

[lithiumus]

Here is the 5 second clip of the source
http://www.mediafire.com/?pvzelbhjmyd

Here is the 5 second clip of the encode using x264 build 947
http://www.mediafire.com/?nmnalalmrto


This is the first pass command

"C:\Program Files\megui\tools\x264\x264.exe" --pass 1 --bitrate 10700 --stats "F:\Burn\Testing\Extra01.stats" --level 4.1 --keyint 24 --min-keyint 2 --bframes 3 --direct auto --subme 1 --partitions none --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 14000 --vbv-maxrate 40000 --me dia --threads auto --thread-input --sar 1:1 --progress --no-ssim --output NUL "F:\Burn\Testing\Extra01.avs" --mvrange 511 --aud --nal-hrd --sar 1:1

This is the second pass command

"C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 10700 --stats "F:\Burn\Testing\Extra01.stats" --level 4.1 --keyint 24 --min-keyint 2 --ref 3 --mixed-refs --no-fast-pskip --bframes 3 --b-rdo --bime --weightb --direct auto --subme 7 --trellis 1 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 14000 --vbv-maxrate 40000 --me umh --threads auto --thread-input --sar 1:1 --progress --no-ssim --output "F:\Burn\Testing\Extra01.264" "F:\Burn\Testing\Extra01.avs" --mvrange 511 --aud --nal-hrd --sar 1:1

Let me know if you need any of the other log outputs. I'll keep them handy.

[lithiumus]

Did you use the same .avs script with the successful XviD encode? or did you use another process? Was the filter chain identical?

Same script. I used Megui. Opened up the same AVS script and just changed to DivX and it worked.

Here is an DivX AVI I just created with the same sample without problem...

http://www.mediafire.com/?16xmtg4o6af

[poisondeathray]

I tested your source clip, and it worked fine, no distortion.

I used default settings in MeGUI "unrestricted 1pass const. quality"

My script used DirectShowSource()

Hopefully Dark S. will examine your 2pass settings and figure out why

Edit: I just tried "unrestricted 2passHQ" and it worked fine too...

[lithiumus]

OK, I just tried another encode with the VBV buffer left blank similar to the "unrestricted 1pass const. quality" and "unrestricted 2passHQ" profiles and it worked. My settings are exactly as above for both 1st pass and 2nd pass except minus "--vbv-bufsize 14000" and it work perfectly.

[lithiumus]

it looks Visualy exactly like the VBV error i reported once :)
tough i think Gabriel fixed that and now it's back ??

Was your source also Anime or live action? I'm wondering if this issue affects ALL sources or just prone to Anime sources...

[CruNcher]

http://forum.doom9.org/showthread.php?t=134656
also trying to lower X264 automatic threading wich i think does core+1 to the actuall cores --threads = cores seemed to help avoiding this issue with VBV (ofcourse with a small slowdown but better then turning threading of completely)

@lithiums
i know you workaround that problem yourself but just for fun could you try to set threading i guess @ least for the 1st pass to the same amount of cores you have (manualy) and see if you still get those borked frames with the same vbv settings ? (also try once for both passes)

[kemuri-_9]

also trying to lower X264 automatic threading wich i think does core+1

--threads auto = --threads cores * 3/2 not cores + 1

[CruNcher]

ahh yeah sorry for dualcore it endsup the same tough hehe both 3 ofcourse it's not the same ;)

[lithiumus]

http://forum.doom9.org/showthread.php?t=134656
also trying to lower X264 automatic threading wich i think does core+1 to the actuall cores --threads = cores seemed to help avoiding this issue with VBV (ofcourse with a small slowdown but better then turning threading of completely)

@lithiums
i know you workaround that problem yourself but just for fun could you try to set threading i guess @ least for the 1st pass to the same amount of cores you have (manualy) and see if you still get those borked frames with the same vbv settings ? (also try once for both passes)

Thanks Cruncher, I'll give that a try. I wouldn't conclude that the workaround works 100% as I'll need to ensure that my HD-DVD to Blu-ray conversion will actually play properly on my BD Player and PS3. It's possible that the "lack" of VBV buffer may have other adverse effects...

[lithiumus]

So put 3 under threads? Not 2 for Core2Duo?

[CruNcher]

for dualcore 2 then --threads 2 :)

[qyqgpower]

this VBV issue would only take place when encoding the whole clip. if you encode the problem part separately, the issue may not be reproduced.
I've encountered same problem, and fixed the broken parts by reencode them separately.

[lithiumus]

this VBV issue would only take place when encoding the whole clip. if you encode the problem part separately, the issue may not be reproduced.
I've encountered same problem, and fixed the broken parts by reencode them separately.

Maybe a fluke, but regardless of how the clips I posted are encoded i.e. split up or the entire clip, the issue is reproduced. Which I guess makes it a good candidate for debugging purposes.

The entire trailer is 1.5 mins and I split it up to 5 second clips. Unless you are talking about isolating a 10-20 frame sequence that is less than a second, what use is that? Maybe if I just encoded those 5 frames in isolation the problem would not happen... but is that useful?

[Dark Shikari]

Maybe a fluke, but regardless of how, the clips I posted are encoded i.e. split up or the entire clip, the issue is reproduced. Which I guess makes it a good candidate for debugging purposes.It is. I've already sent it off to Gabriel, the author of the VBV code, for consideration.

[Dark Shikari]

Now that was a very subtle VBV problem, related to the fact that your frames were duplicates of each other (one frame takes loads of bits, the next takes zero, since its the same as the previous frame). This was confusing the hell out of VBV.

Fixed in latest revision.

[jefrey]

finally i've encoded my movie 3times again an again:D and the error appears allways on the same frame/timestamp.
what can prefere such damned thing?, that are 3 movies where that appears

[Ranguvar]

With r949?

[lithiumus]

Now that was a very subtle VBV problem, related to the fact that your frames were duplicates of each other (one frame takes loads of bits, the next takes zero, since its the same as the previous frame). This was confusing the hell out of VBV.

Fixed in latest revision.

Nice work! I reverified it with the new r949 build and it works a charm! Sounds like this would only affect Anime sources since they do have duplicate frames and most live action won't.

Thanks again!

[jefrey]

no i used the old one, 940 ill test later the new build

[qyqgpower]

It seems that DA have fixed the previous VBV issue in 949, but I still get random blocky pictures with 949 & 951

The bad news is that I couldn't reproduce the issue with same source (at least for re-encoding a few GOPs around the blocky point), so it's hard to tell whether this is a bug of x264 or my fault.

here's a sample, but I wonder if anyone could figure out things from this isolated sample. Anyway, I'm expecting some kind of advice.

http://www.mediafire.com/?hwmmpgc5pfd
command line:
--pass 2 --bitrate 4000 --stats ".stats" --level 4.1 --keyint 240 --min-keyint 24 --ref 4 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 30000 --vbv-maxrate 38000 --me tesa --merange 32 --threads auto --thread-input --aq-mode 1 --progress --no-psnr --no-ssim --output "output" "input" --psy-rd 0:0 --sar 1:1

[CruNcher]

@qyqgpower
try this http://forum.doom9.org/showthread.php?p=1173808#post1173808 explained there maybe it helps you have nothing to lose (except some speed) :)