Multithreaded XviD - official thread

[LoRd_MuldeR]

One could argue that the so-called "Xvid era" isn't "ending" at all; it's merely becoming compartmentalized in the grand scheme of things. Thus, improvement would indeed yet benefit many.

For what would you use Xvid/DivX today, except for stand-alone compatibility? :confused:

And stand-alone support for H.264 is spreading out these days, so even stand-alone compatibility won't be a noteworthy reason for Xvid/DivX soon...

[DeathTheSheep]

Standalones (read previous posts about price and availability points) and divx TV's and divx connected boxes, and device support (like my x51v in powersave), and support for slower systems in full res (netbooks and eee pc in powersave), widespread net support (xvid/divx files are still more widespread and entrenched than h264's), easy frame-accurate extensible virtualdub and vfw editing support, highly customizable post-processing available (look at all the PP methods, some of which are superb and highly tweakable for desired effect decoder-side), more 'official' VfW support and encoding in all capable applications (the AVI's will work everywhere thanks to a long history of divx support), a faster asp decoder native to windows 7 (the avc one is laughable to say nothing of the unsupported mkv and aac plus and embedded subtitles etc), and more...

In a nutshell, one could say speed and convenience, but that wouldn't hit the mark since AVC junkies have an arsenal of canned and superficial comebacks at the ready to convince themselves with. Sure x264 can be faster [and still be leagues better in quality] at encoding. Such a narrow definition of speed is useless for most people who won't do the darn encoding and have to put more crap on their computers and more stupid players or filters or splitters. Sure support is growing in the HD market and in funny niche things like Popcorn hour [only available in certain countries for any reasonable price anyway], but we 'geeks ahead of the game by a few years' really need to take a look at people who don't have, want, or give a crap about something they don't or really can't use for their purposes, or would be simply impractical in comparison.

It was so amazing to install Windows 7 on the class eee pcs and they play xvids out of the box in the cool new WMP12. Bring AVC into the picture and all hell breaks loose. If you like, follow this little skit below:
"It won't play right on my WMP. I want the avi files."
Too bad--I make the rules, loser. Get this and this splitter, decoder, vobsub filter.
"Now it crashed!"
That's the damn OS or your config or your file or that POS WMP's fault, get this other random awesome player.
"But I like my old one much better than this ugly one, plus all my library is on it and it worked just fine with the avi files!"
Shut up and go with it because I said it's better.
"Fine... Hey, it doesn't play right, and keeps stuttering."
Put your processor in turbo mode, dufus.
"...The battery died halfway through!"
Buy a new clunky battery if you wanna watch movies you idiot.
"I don't want to spend more money and carry more around and have to switch the ba--"
SHUT UP. Then buy CoreAVC or sign up to use some beta decoder with the DivX logo on it and in either case pray it works with all your AVCs in all the containers.
"That's still spending money and filling up my limited hard-drive with crap!"
Listen, you have to make some sacrifices and learn for the sake of technology.
"Isn't the evolution of technology supposed to make things easier, faster, and more intuitive?"
Uhm, you're just an idiot and got unlucky and your stuff just can't handle it.
"It's also supposed to be 'easier' and more accessible. Just because I'm not a video file nerd doesn't mean I have to put up with all this!"
Then just stop living in the past and get new, expensive stuff!
"These are all brand new computers and devices, and I'm not stupid enough to spend all that money for something I don't need when the avis worked fine. Now please go away."
ARGHHH!!
Now that situation is a lot more typical than you might think out here in the "real world" (replacing the "Windows 7" native support with "install this one 446K xvid.exe (http://ffdshow.faireal.net/mirror/XviD/XviD.cvs.head.MTKVAQ.exe)), and while it might not be a powerful case to get people back to working on Xvid, it at least needed to be said in light of some recent misunderstandings in this regard.

You probably don't want to hear the story about my friend's DivX supporting TV, or my pocket pc which hums along with SD XviD in powersave but can't bench 50% on VGA baseline AVC (even in max performance), or my uncle's new DivX capable DVD player, right? Or any of the multitudes of others? In light of this, I honestly can't care for some self-conceived "noteworthy reasons" being so easily thrown around; you simply don't understand that there are a vast number of people in and out of the nerd sphere who would really benefit from better Xvid--and not only for the compat/powersave/decoder/editability/etc/etc/etc stuff above. Just because you don't understand the hassles of AVC for both people and hardware (or are immune from it with your own setups), doesn't mean it's not there and you can self-righteously claim that the "Xvid era is over." I love AVC for my own purposes. I've been hacking around with some x264 code myself these last few weeks here... and XviD too, but I'm simply not really a programmer. I always knew XviD was better for most people outside of my computer, but only recently had cause to consider regularly using it for myself (I'm also getting an eee pc now along with my VGA pocket pc) and the reality hits me like a slap in the face. The need for Xvid as is strong as ever in many spheres of users.

Now I hope you can see what I was referring to regarding the "compartmentalization" I spoke of. I'm not the best at getting ideas across, but if you could look beyond my pitiful writing and try to see where I'm coming from, it would help not only me, but all the people I spoke of. So please don't just declare Xvid dead and walk away, or worse, withhold the [literally] few minutes worth of coding that comes so easily to you devs (especially DS et al) which would really make things that much better for the rest of us. Please, at least consider spending that metaphorical (or literal?) "15 minutes" making Xvid that much better, instead of spending the same on x264 for less dramatic improvement. Hack in some better B-frame detection, some sort of primitive UMH or ESA, some form of decent RD, and just maybe some basic speedups. I mention these because all of them can be found in some form or another in x264, so porting shouldn't be that hard, right? (The breathtaking legendary 15-minute VAQ comes to mind.)

[Sagekilla]

Erm, that's an awfully long list of problems and quirks.

For me it was:
AVC --> Haali Media Splitter + CoreAVC
ASP --> XviD or DivX decoder

Nothing more beyond that was needed. The other thing? My desktop runs on an Opteron 170 (2 GHz) and I tried using XviD @ 1080p once. I don't know if I was using the wrong decoder or something, but my playback stuttered unless I turned off all PP (inc. deblocking). Transcode the material over to AVC @ 1080p and coreAVC played it back perfectly with deblocking.

That's just my experience. I haven't seen an XviD decoder so far that offers similar speed to an AVC decoder for the same resolution. I wasn't making any "special cases" either by forcing certain options off on the encoder for compatibility or performance (beyond disabling PP for the ASP decoder to allow for fluid playback).

I agree that so far the hardware for AVC isn't great, but that's to be expected this early in the mainstream life cycle. Stuff like this only improves over time.

[DeathTheSheep]

A separate splitter + a paid MT program, so not quite the single 450k exe. Your results might be strange nonetheless; if Windows 7 could do it bareboned on the eee (or one xvid.exe for xp), I'd say maybe you screwed something up with all the decoders and software and settings you added/tried out? Besides, if you're going to watch 1080p, and/or have a super dual-core opteron desktop system, that's the perfect scenario for x264; you pretty much exemplify the "other audience" I speak of, the very type of person and system x264 (and the paid multithreaded decoders) are designed and optimized for. That "awfully long list of problems and quirks" manifest for systems and scenarios essentially opposite of yours. :)

[PatchWorKs]

Wait, the "Xvid era is ending"... so you want to switch to something inferior to Xvid?! (Theora)

Err... well, I'm not alone: Upcoming versions of Firefox and Opera will play natively Ogg/Theora videos with the new HTML5 element (http://news.slashdot.org/article.pl?sid=08/11/04/136220)...

Theora should be the flash replacer, but for quality encodings i would go with Dirac ! ;)

BTW, the best approach would be expand/optimize the FFMpeg multithreading, IMHO. :devil:

[LoRd_MuldeR]

A separate splitter + a paid MT program, so not quite the single 450k exe.

Not needed, ffdshow-mt is on the way:
http://forum.doom9.org/showpost.php?p=1215370&postcount=5157

[Lenny_Nero]

I am sure that all that happened when I watched H264 was that FFDshow kicked in, its what I say to people that are having problems watching H264, if that does not work most of the time their computer does not have the power to play them.

As for settings I can make MPC use 2~3% or 100% CPU and have problems unable to get over 15 fps for the same file. This seems to be more of a problem for Windows XP users and VMR7 and their vid cards having AA (anti aliasing) turned on, like most things computer its all about the settings.

As for fancy media playing HDD's and that sort of media streamer like popcornH I dont need them as I can work out how to run a cable from one of my computers to the TV, so in fact I have the ability to do everything that popcornH can for the £10ish I gave for the cable. Also the £20 quid Xvid DVD players have USB (and SD card slots) plugs on them so I might have to have a play via them. But with my quality encodes I can get a nights TV on a 4 GB USB stick no problem at all.

As for people trying to making Xvid better I could care less as I said I get very good encodes and I dont feel the need for a massive TV, but then I dont have the small body part that means some have to have big TV's, fast cars and the newest computers :)

[seggitek]

What is the reason of using threads at the codec level, if you can do it in your AviSynth script?

[clsid]

You can have a trillion threads in your AviSynth script, but if just one of those threads is doing all the resource intensive work, the performance will not scale.

[alexins]

xvid 1.2-127.08.11.27-VAQ x86 (http://www.xvidvideo.ru/content/view/436/1/)

Brightness control fix; GUI controls for SSE3/SSE4; Updated about box and messages

[squid_80]

Looking in xvid's cvs, it appears win64 support has been officially added.

Edit: Hmm doesn't look right to me - XMM6 and XMM7 aren't treated as non-volatile registers.

[seggitek]

You can have a trillion threads in your AviSynth script, but if just one of those threads is doing all the resource intensive work, the performance will not scale.

First of all thanks for the reply. But why should 1 thread do the resource intensive work. Normally the frame is split up evenly among the cores using the MT function of AviSynth.

I assume the XviD multithreading only affects the compression of the frame, right? And not how the frame is built up, because that is part of AviSynth.

So it goes like this:

1.) AviSynth creates the frame (maybe using MT or SetMTmode multithreaded)
2.) XviD compresses the resulting frames (maybe using multiple threads, but this only speeds up the work of the codec)

[clsid]

A frame is not split up. Xvid encodes whole frames.

[olnima]

Btw - any news about Syskin? Is he doing well? Maybe he can code some tweaks?

Olnima

[seggitek]

A frame is not split up. Xvid encodes whole frames.

Sure, but the work (compression) can be split up. Or what do the 4 threads I start for XviD do? Does each one write one single frame?

[Dark Shikari]

Sure, but the work (compression) can be split up. Or what do the 4 threads I start for XviD do? Does each one write one single frame?That's what they would do if XviD was multithreaded well (frame-based threading, a'la x264).

XviD itself uses a quite interesting threading model that results in the exact same output as unthreaded encoding (IIRC) but is not as efficient as full-frame threading.

[Blue_MiSfit]

A separate splitter + a paid MT program, so not quite the single 450k exe. Your results might be strange nonetheless; if Windows 7 could do it bareboned on the eee (or one xvid.exe for xp), I'd say maybe you screwed something up with all the decoders and software and settings you added/tried out? Besides, if you're going to watch 1080p, and/or have a super dual-core opteron desktop system, that's the perfect scenario for x264; you pretty much exemplify the "other audience" I speak of, the very type of person and system x264 (and the paid multithreaded decoders) are designed and optimized for. That "awfully long list of problems and quirks" manifest for systems and scenarios essentially opposite of yours. :)

I hear what you're saying about MPEG-4 ASP being very important, particularly where hardware devices and power savings are concerned.

It makes perfect sense, and if a device works well with ASP, why bother with AVC if it burns significantly more battery power?

That being said, your discussion of Windows 7 isn't really valid. It's not even an officially released operating system!

~MiSfit

[alexins]

xvid1.2-127.CVS - 08.11.28 - VAQ (http://www.xvidvideo.ru/content/view/438/1/)

Enable SSE4 GMC code;
Encraw - fix for -ssim option;
More ssim fixes;
Bugfix: prevent access violation if width/height is not multiple of 2;
Auto SMP.

[johnsonlam]

xvid1.2-127.CVS - 08.11.28 - VAQ (http://www.xvidvideo.ru/content/view/438/1/)

Enable SSE4 GMC code;
Encraw - fix for -ssim option;
More ssim fixes;
Bugfix: prevent access violation if width/height is not multiple of 2;
Auto SMP.

Wow! Thank you very much Alexins!

You're our binary savior!

[clsid]

What is the preferred build? MSVC9 or GCC?

[_xxl]

@ alexins
I have seen that you compile ffdshow's libs using MinGW GCC with -msse. If I remember correctly forcing the compiler to use SSE won't make them faster, but slower.

[alexins]

What is the preferred build? MSVC9 or GCC?
I build in MSVC9 sp1

[clsid]

Yes, Xvid already has handwritten assembly optimizations. Using -msse has little or no benefit.

[alexins]

clsid,
I compared the assembly was made in the GCC and MSVC.
Assembling the GCC - the first passage: 124.62 fps, the second pass: 52.75 fps.
Assembling MSVC - first pass: 142.04 fps, the second pass: 61.02 fps.

[olnima]

clsid,
I compared the assembly was made in the GCC and MSVC.
Assembling the GCC - the first passage: 124.62 fps, the second pass: 52.75 fps.
Assembling MSVC - first pass: 142.04 fps, the second pass: 61.02 fps.

So maybe You can compile and upload a new MSVC-bulid?

Olnima

Great to see that xvid isn't dead!

[alexins]

XviD-1.3.0 (CVS)-081129.05.22-VAQ (http://www.xvidvideo.ru/content/view/441/1/) - MSVC9-bulid

add: alternative multicore detection;
pump up HEAD version numbers;
installer is added

[olnima]

Dear alexins,
thank You very much !!

Olnima

[cweb]

Dear alexins,
thank You very much !!

Olnima
thanks! this seems like a very good build, testing it right now.

[seggitek]

Thanks for your work. Does this mean that this version should be a bit faster? :)

[Lugia25000]

When i use Megui, the framerate is wrong. Instead of 23.976 is detected 25.00. But it works with vfw.

And with Mediainfo the Version is not right:
with Dark Shikaris 1.2 Build http://img19.myimg.de/Aufnahme253a7a.jpg

with the new 1.3 Build http://img19.myimg.de/Aufnahme19614a.jpg

And with Megui, the width and height are swapped in WinXP
http://img19.myimg.de/0141762.jpg
normal:
http://img19.myimg.de/023d1a6.jpg

But I found no major errors :)

:thanks: for the new Builds and sorry for my bad english.

[avivahl]

So the new XviD-1.3.0-(CVS)-081129.05.22-VAQ build (from XvidVideo.ru) sounds quite buggy. :( Can anybody confirm?

[alexins]

Problem appears with new in xvid_encraw.exe. Old xvid_encraw.exe and new version xvidcore.dll work normally.

[avivahl]

So does that mean the DirectShow filter (of XviD-1.3.0-(CVS)-081129.05.22-VAQ) should work fine?

[Sharc]

Works fine here - so far. >85% CPU on Quadcore.
Thanks alexins.
Any plans to improve VBV / peak bitrate control? IIRC this is still an issue with Xvid for standalone compliance.

[Dark Shikari]

Works fine here - so far. >85% CPU on Quadcore.
Thanks alexins.
Any plans to improve VBV / peak bitrate control? IIRC this is still an issue with Xvid for standalone compliance.Two random thoughts on this topic from looking at Xvid ratecontrol code:

1. There's no row-based ratecontrol; be this as it will for VBV compliance (bad).
2. The 2pass algorithm has a very very very fine threshold for underflows; from what I can tell it literally says "as long as this is less than 100% of the max bitrate, its fine." This obviously fails if the prediction isn't exact--which it surely isn't. Odds are it'll perform better if the limit is 95% or 90% or something.
3. The first pass in Xvid is always run at CQ2--this is probably bad for VBV compliance that relies solely on what is described in 2), because it makes the prediction from the firstpass less accurate.

[Sharc]

Hmmm, this looks to me - as a layman - that improvements in this area would require substantial re-engineering.

[Lenchik]

Old xvid_encraw.exe and new version xvidcore.dll work normally.

Is this working version bundled with your build's "coming without installer" version?

[Amefurashi]

Hi guys, just got this mail:

Hello!

This is Xvid 1.2.0 release.

This release is Xvid 1.2.0 stable release. It is API compatible with
the previous 1.1.3 stable release.

Changes since 1.1.3:

* xvidcore library
- Complete AMD64/EM64T 64-bit support
- Added support for WIN64 platform
- Multi-threaded encoding support
- SSE3/SSE4 optimizations
- Faster and more precise mpeg intra quantization
- Fixed bug in packed pixel format colorspace conversion
- Noexec-stack security patch
- Fix for bad resync marker length
- Improved decoder robustness for broken streams containing B-frames
- Fix for potential out-of-bound access to MV bits table
- Added SSIM quality-metric plugin


* VFW frontend
- WIN64 compatibility
- Added widgets for SSE3/SSE4
- Auto-detection of available processor cores
- Minor GUI cosmetics


* DShow frontend
- WIN64 compatibility
- Minor GUI cosmetics


The files are available in the download section of Xvid.org:
http://www.xvid.org/downloads.html

-- The "Xvid Team"

Can't wait to try this out... :helpful:

[olnima]

...and Isibaar has added new code again yesterday...

Olnima

//edit: maybe Koepi will build THE official 1.2-release

[buzzqw]

yep.. i hope soon!

BHH

[totya]

Hi guys, just got this mail:
Can't wait to try this out... :helpful:

xvid is free, but the recommended compiler...not :)

MS VisualDev 6 Processor Pack 5 or MS VisualDev 7

This is slightly joke.

[squid_80]

The visual studio project files have been updated to VC8 so the freely available VC++ 2005/2008 Express Editions should work.

[alexins]

XviD-1.2.0 x86 / x64 Stable release (http://www.xvidvideo.ru/content/view/452/1/)

[LordIntruder]

I'm a bit lost.

Amefurashi tells us he just got an email about Xvid 1.2.0 release:

http://www.xvid.org/downloads.html

But on Koepi site there is already a 1.2.0 branch for SMP for months now:

http://koepi.info/

So what is the difference between them? Same? The XviD.org is a brand new one just released and better than the one on Koepi site?


alexins released XviD-1.3.0 (CVS)-081129.05.22-VAQ - MSVC9

Today he announced XviD-1.2.0 x86 / x64 Stable release for windows based on the xvid.org.

Great but again what build to use? What is the difference between 1.3.0 and the 1.2.0?

[clsid]

The 1.2.0 branch has been around for a long time. But today the final version has been released. Previous builds were just builds from current SVN repository, consider them as "work in progress". The 1.3.0 branch has been created a few days ago. I don't think that has anything new compared to 1.2.0 final. So just use the final version.

[olnima]

XviD-1.2.0 x86 / x64 Stable release (http://www.xvidvideo.ru/content/view/452/1/)

Again MSVC9 sp1 ?

Thanks anyway,
Olnima

[alexins]

Again MSVC9 sp1 ?

Thanks anyway,
Olnima

Assembling version 1.2.0 I have done in msvc8sp1, as it was conceived by developers xvid, no change in the code. In msvc9sp1 I will assemble the 1.3-dev.

[totya]

The visual studio project files have been updated to VC8 so the freely available VC++ 2005/2008 Express Editions should work.

Thanks.

[olnima]

Assembling version 1.2.0 I have done in msvc8sp1, as it was conceived by developers xvid, no change in the code. In msvc9sp1 I will assemble the 1.3-dev.

Thanks again, alexins :o

[LordIntruder]

Assembling version 1.2.0 I have done in msvc8sp1, as it was conceived by developers xvid, no change in the code. In msvc9sp1 I will assemble the 1.3-dev.

You mean 1.2.0 and 1.3.0 are exactly the same XviD but:

1.2.0 is done with msvc8sp1
1.3.0 is done with msvc9sp1

or am I wrong as usual? :D

That would mean 1.3.0 faster than 1.2.0? Sorry it may be obvious for some but I'm totally confused, I have no knowledge about what is msvc8sp1 and such stuff. It is chinese for me.

I suppose I should use 1.2.0 as clsid suggested me, this is official but maybe 1.3.0 is much faster? Alexins could you just clarify that? What 1.3.0 has than 1.2.0 has not? Thanks :)