suggestion for standardized testing

[Doom9]

Let's collect ideas for a standardized DVD/DivX player test here.
Considering that there are many ressources for basic DVD player testing already, I'd like to focus this on DVD backups, and specifically on DivX/XviD playback.

Test 1: Basic format support
DVD-R/W
DVD+R/W
VCD
SVCD
miniDVD

Test 2: SVCD testing

Multiple audio streams
SVCD / CVD subtitles
Menus

Test 3: MPEG-4 Containers

AVI
Ogg
Matroska
MP4

Now for the time being it makes no sense to test for special features in non AVI formats because they are most likely not supported.

Test 4: Codecs

DivX3
DivX4
DivX5
XviD

Here we test multiple things: First a basic 2 CD rip (bitrate in the 900-1200kbit/s range) having one 128kbit/s VBR MP3 audio stream using no Advanced Simple Profile features (GMC, QPel, B-frames). Then the ASP features are progressively added. Using the same source and bitrate, GMC, QPel and B-frames are tested. In the case of XviD, multiple b-frames are also to be tested (max 3 consecutive b-frames). As test scene, we need something that has bitrate peaks, so an action scene (I found that the lobby shootout in the matrix was a good test for multiple b-frames, even at 1 CD bitrates).
Once the ASP features have been tested, the ones supported should be tested in an extreme situation (1 pass quantizer 2) to see if the player starts to stutter.
For DivX3 we should perform the same extreme testing to see how powerful the transcoding engine is.

Test 5: Audio

1x CBR MP3 @ 192kbit/s
1x VBR MP3 @ 160kbit/s
1x AC3 @ 448kbit/s
2x VBR MP3 @ 160kbit/s
2x AC3 @ 448kbit/s
1x AC3 @ 448kbit/s, 1x AC3 @ 192kbit/s
1x AC3 @ 448kbit/s, 1x VBR MP3 @ 160kbit/s

Test 6: DivX subtitles

AVI with integrated srt
External srt

Also, if there are any other formats the manufacturer claims to support, those should be tested as well.

Test 7: Multiple content on the same disc

VCD mpg burned as ISO
SVCD mpg burned as ISO
DVD VOB burned as ISO
JPEG images burned as ISO

And combinations thereof, for instance a VOB trailer, an SVCD mpg and a DivX on the same disc.

Test 8: audio playback

CBR MP3
VBR MP3
Ogg

Other formats should only be tested if the manufacturer claims to support anything else.

Test 9: playback facilities

Playlists
fast forward / backward for DVD and DivX (how many levels, how much is skipped with DivX, how long does it take for DivX)
Zooming capabilities
Proper aspect ratio (take a 1:1 AVI showing a perfect circle.. see how this is displayed on the TV).

Other codecs should only be tested for if specifically mentioned as supported by the manufacturer (also depending on the alternative container support.. if there's no ogg/Matroska support it makes no sense to even test for Ogg Vorbis audio).



Now to allow anybody to perform these tests without too much hassle, sample images might be advantageous, but before we go into that (or source selection), I'd first like to have an established test procedure, then we can deal with the rest.

[bond]

great idea, this hardware test!!!

i really second that you want to test asp features like qpel and gmc too! great! imho every standalone player who has a big mpeg-4 sticker on it should be able to play these features

the same goes for the mp4 logo
many manufacturers list "mp4" as their features, so i would also test the .mp4 container if possible and not too much work


suggestion for the procedure:
1) perhaps we first should list what players are on the market which support divx...
2) than we should decide about very specific a/v samples to test (you already wrote about the codec settings) and upload these samples somewhere (perhaps already the final isos for burning to avoid problems here)
3) than people who own a standalone player from the list can download these specific video files and test them on their player and give us feedback

i know that this is a little bit "user dependend" but...

[SeeMoreDigital]

I think it may also be useful to know which standalone DVD players can spin RAW Mpeg1 and 2 files. As not everyone can be bothered with authoring!

Useful too for users who might like to store 30mins episodes of a TV series in Mpeg2 direct to a CD-R (after capturing) instead of using a DVD R/RW

Cheers

[Zhnujm]

I think such a test would be a great idea, as there is absolutly no reliable source for such information in the net.
I would add a DivX3 test, not really important but for completeness.
The procedure bond describes sounds good but i wonder if such sample files would create to much traffic.

[Doom9]

I've added DivX subs, DivX3 and raw content tests.
Let's first define the test, then we define the source and only then we worry about the distribution.. trying to solve everything at once will only get us stuck.

[SeeMoreDigital]

Doom9

It's just not fair that you can edit your posts without the little 'Last edited....' tag appearing below your signature.

No it's just not fair at all!

Cheers

[zag2me]

Ok, great idea Doom9.

I was going to talk to the xvid developers about some kind of hardware certification but we will start with this.

I will post this test on our KISS forums, and get back to you with some preliminary results from the DP-450 and DP-500. This could show if the tests are feasible in a review situation.

My ideas for testing criteria.

1) Network streaming capabilities
2) Codecs - support for 3IVX
3) Audio - OGG audio
4) Special additional features - Such as webradio, video on demand ect.

Cheers.

[zag2me]

Some more ideas from BBDOC on our forums:

Some stuff I think should be added to this test:

In Multicontent part:
- Mpeg + Avi
- Video + Images
- Video + MP3
- Images + MP3


in MPEG-4 Containers
- Qpel support

in Audio part:
- wma support

+ somewhere else (don't know where!):
- Manual Zoom function
- Automatic Zoom to fullscreen
- FF/RW functionnality

These tests are based on my personal experience with the player (things that doesn't work with my DP-450!)... as well as on rumours I heard about other players (it seems the Philips DVD-737 doesn't allow you to zoom, which I find is a BIG problem!).

[SeeMoreDigital]

Originally posted by zag2me
Some more ideas from BBDOC on our forums:
in MPEG-4 Containers
- Qpel support....
Do any of the Kiss players provide this yet?

Cheers

[zag2me]

The new sigma chips will support Qpel I am lead to beleave, but no players do at the moment.

I'v just had another idea

A test for multiple audio tracks, on all codecs.

[SeeMoreDigital]

Thanks for the update zag2me

Maybe, when such players arrive I'll take a look at them. But at the moment I'll stick with my Xcard (or should I say Xcards as we have 4 'Hire Out' PC's with them)!

I think if we are not careful this thread will end up being full of 'wishful thinking' capabilities and as a result become impossible finalize.

Where will it end.... With RealMedia, QuickTime, WinMedia9, RAW DV.AVI audio and video file playback!!!!!

Cheers

[Zhnujm]

Maybe a test for UDF DVD and filesizes >2GB would be a good idea. My first player did not support UDF at all and the current cannot ff/rw AVIs on UDF DVDs :confused:

[masken]

Regarding subtitles:

- multiple subtitles according to de-facto standard, like:
\somemovie.avi
\somemovie.English.srt
\somemovie.Swedish.srt
\somemovie.Dutch.sub

- Subtitle formatting (italics, bold, underlined text)
...in their different formats

- subtitle formats
Subrip (*.srt)
MicroDVD (*.sub)
Substation Alpha (*.ssa)
SAMI (*.smi)
SubViewer 1/2 (*.sub, *.txt)

... I guess vobsubs won't be interesting to test at all?

Testing the CDFS would also be interesting for bugs I guess. ISO level 1/2, character set support, Joliet, pathdepths, more than 255 characters in path?

[Lord_KiRon]

Yes and I would add OGM WMV and ASF categories for testing.
I do not know any players yet that suppot them but even putting them into a test bench would create a pressure for DVD makers to support

MS codecs support in AVI is another thing we can test.

Also don't forget ABR for audio part.

Btw: KiSS does not support UDF DVDs at all !!!!

[SeeMoreDigital]

Originally posted by Lord_KiRon
Yes and I would add OGM WMV and ASF categories for testing.
I do not know any players yet that suppot them but even putting them into a test bench would create a pressure for DVD makers to support

MS codecs support in AVI is another thing we can test.

Also don't forget ABR for audio part.

Btw: KiSS does not support UDF DVDs at all !!!! Personally, I don't think we will see Microsoft WMV9 .wmv or .avi hardware players in the near future.

I think that all M$'s content will be encrypted. In much the same way it's T2 Extreme DVD content is encrypted and requires authorization prior to viewing. So in my opinion M$ will only allow it's codec to be used in DVD players that are connected to a network!

Cheers

[Doom9]

@zag2me: QPel was already there
I've added audio-only testing, but as with the codecs, it makes no sense to establish a test that's never used (exotic audio or video formats.. only if a manufacturer claims to support should it be tested). For instance, so far no other codecs are supported so I don't think that belongs into a standard test (btw: the elta plays nerodigital if you use the divx fourcc ;)

I've also added jpeg images, and special features (zoom and such).

Since DVDs should be in a bridge ISO/UDF format, I don't think UDF DVD testing is warranted.. no burning prog creates UDF only DVDs onless you force it to.

@masken: you're overshooting the goal.. subs are currently not even supported, and all the formats you listed will never be supported. This isn't a test to cover 100%, but the 80% that matter for most people.

@Lord_KiRon: ABR is VBR with some limits ;) And I agree with zag2me that ASF and WMV testing is not useful. Don't forget that this test is designed to compare existing players... we should not devise tests that cover whatever combination we can imagine (the test as it is is already extremely complex and will take a long time to complete.. in the end I believe we'll have to cut a few corners to get something useful).

@all: when making suggestions, try to imagine that you're the person who has to perform those tests... only then can you decide what's really worth testing ;) I only performed a small part of those tests on my elta player and it already took me hours, preparing sources and watching them.

[SeeMoreDigital]

Hi Doom9,

I've read good things about the elta player. Shame it's not available in the UK (or not as I've been able to find)!

So am I correct in assuming that you're able to test the new NeroDigital application. If that's the case is it worth waiting for?

Cheers

[Stux]

I would suggest doing at least two tests for "MP4 Support"

1) AAC.MP4 (SP video + LC-AAC audio in an MP4)
2) MP3.MP4 (SP video + CBR-MP3 audio in an MP4)

You can go further with this test if you want

HE-AAC, Multi-Channel, VBR MP3, ASP features, etc

But its important to test both AAC and MP3 support in a player's MP4 support.

Other issues which can crop up with MP4 support are negative/positive track offset support and VFR (variable framerate) support

[masken]

Originally posted by Doom9
@masken: you're overshooting the goal.. subs are currently not even supported, and all the formats you listed will never be supported. This isn't a test to cover 100%, but the 80% that matter for most people.
Doom9, I don't think that's overshooting anything at all. The KiSS player already has support for text-based subtitles. It doesn't support formatting though (italics, bold, underlined etc), and thereby display those tags wrong (ie; a MicroDVD italic marker - {y:i} is displayed in the subtitle instead of the text being italic - or as would be preferred - just regular text), but I think it actually supports most formats.

Testing this would be easy, just burn a disc for each of the most common formats (the one's mentioned for example) and try to play it. Then have one disc with multiple subtitles on it (and perhaps one or two more for testing naming conventions).

It's just a case of really easy firmware programming, and I think there will be support for this once the manufacturers gets heads up for it. And it has importance, since all the worlds Level 2 + EFL English speakers (~2/3 of the worlds population) does use subtitles when watching features. Have a look at this (http://www.vcdquality.com/forum/showthread.php?s=&threadid=10702&highlight=subtitles) too for reference...

[zag2me]

I have to agree that the tests should be as simple as possible, the more complicated they get the more time it takes to test and more errors sneak into the results. I dont want to have to burn 5 different disks just to test what subtitles are supported, when it is clear from the product docs that they are not supported.

One test is fine for subtitles, most players dont even support subs yet, :)

This test is not about getting new features or bugs out of the current crop of players, it should be about comparing them against each other in a standard way. There is no need to complicate things.

[Soulhunter]

Whats with MPA/MP2 ? (audio files) :rolleyes:

Because lot of DVD Players are able to play MP3, but not MPA/MP2...

Bye

[Doom9]

mp2 has no real significance as standalone format, has it? mp3 is by far the most widely used format for standalone audio.

[Tuning]

I think mp2 is the audio part in Video CDs[MPEG-1,mp2@224kbps]

[Stux]

MP2 is also part of the DVD standard, at least in the PAL world

But Doom9 is correct that the format is more or less irrelevant from a standalone playback perspective

[Doom9]

@tuning: mp2 playback is tested in the form of VCD and SVCD playback. But nobody would encode his audio CDs to mp2.. they'd pick MP3, Ogg Vorbis (or perhaps AAC).

[SeeMoreDigital]

I know this is off topic, but I seem to remember at the begining of DVD there was some talk about a totally different 6Ch audio system being used (not Dolby Digital or even DTS) in Europe.

I can't for the life of me remember what this format was. Or even if 'surround sound' amplifiers were ever made!

Can anyone oblige?

Cheers

[Doom9]

mpeg2 multichannel. It has no significant commercial importance, hence it should not be in a standard test.

[SeeMoreDigital]

I think you've just about covered it all now Doom9.

I guess the only other thing I can think of is to make sure players can spin some form of DTS encodes!


Cheers

[zag2me]

OK, i think we have covered most aspects of these tests, how do we proceed from here? What content should be used in these tests?

[ronnylov]

Extended basic functions like:
XSVCD (maximum bitrate above standard and non-standard resolutions.
SVCD on DVD (DVD with non-standard resolutions)

would be interesting to know. Especially the non-standard resolutions on DVD because when using DVB capturing I get non-standard resolutions like 544x576.

Possibility to play mp3 files or divx/xvid from DVD-R disc. Many players only support mp3 on CD, not on DVD.

[bond]

Originally posted by ronnylov
Possibility to play mp3 files or divx/xvid from DVD-R disc. Many players only support mp3 on CD, not on DVD.i second that

[saptecai]

There hasn't been much activity on this thread for a while...
I hope the project is not abandoned. I am waiting for this before buying a machine...

All the best

[Starcraftfreak]

Originally posted by ronnylov
Possibility to play mp3 files or divx/xvid from DVD-R disc. Many players only support mp3 on CD, not on DVD.
Thats dumb. Why not just reading the files? Some vendors are really stupid.

I think we should stick to the really important formats. DivX3 isn't really important for example.

[Doom9]

@saptecai: well.. if I can get some players for testing (I'm talking to some companies) I might give it a shot.. it would also be a sort of feasability test to see how much time it takes.
I wasn't planning on using half a dozen CDs for a test, but rather some major brand DVD-Rs to put the content to (I tested my current DVD/DivX player that way). To day I think we can safely assume that any player should have no problem with CD media so testing with DVD media makes it more interesting.

[dede95064]

i posted a my review of the liteon lvd-2001 player a while back. i have begun these tests but was wondering where should the results be posted? should i put them with my review and/or somewhere else?

also, liteon released their newest firmware 030 but it was beta. this beta update allows for .srt external subs, ogg audio, and improved divx3 playback, folder support, mp3/cd shuffle play. now they took it off because some people were having problems with it. my player still plays fine for the most part. for the time being, i can hold off on the results until the official firmware version comes out.

any suggestions?

[zag2me]

Yep lots of new Kiss players being released at the moment, We could carry out the tests if we have some standard material to work with.

[Wolfman]

there are some free mpeg2 test clips at the mpeg.org including some nice juicy clips encoded at 40mbs per second. My player couldnt play these!! they include many BBC refernce clips. Also people might use mp2 as they might also use LPCM or AAC, not necessarily sensible but then who knows.. I had a clip encoded with 400mbs video data and 400mbs AAC (simply because the software looked at target size greater than 700mb, then defaults to AAC sound, luckily it was only 2ch AAC not 6ch, as that would have been 780mb, leaVING 20mb for video!! gkif4u!).

[bond]

seems someone already tried to push some sort of standardized tests

read about it here (http://www.dvdboard.de/forum/showthread.php?s=&threadid=60788) (in german)

the thread also mentiones a Test CD (ftp://zhnujm.dyndns.org/TestCD.txt)

edit: huh seems that zhnujm, who already posted here, did that :)

[nesskiel]

hellon

sometime ago i made few test regarding the MP3 support for the Philips DVD737.here the results:

the Player documentation claims that it is only able to decode mp3 up to 256Kb/s
i encoded a song from ones of my orginal CDs with LAME 3.90.3 with the following settings:

CBR:
-b 192 -m s -h --lowpass 19.5 -q 0
-b 224 -m s -h --lowpass 19.5 -q 0
-b 256 -m s -h --lowpass 19.5 -q 0
-b 320 -m s -h --lowpass 19.5 -q 0

VBR:
-b 128 -m s -h -V 0 -B 320 --lowpass 19.5 -q 0

ABR:
-b 128 -m s -h --abr 310 -B 320 --lowpass 19.5 -q 0

results: all Ok

Ness