suggestion for standardized testing

Doom9 · 27th September 2003, 15:48

Let's collect ideas for a standardized DVD/DivX player test here.
Considering that there are many ressources for basic DVD player testing already, I'd like to focus this on DVD backups, and specifically on DivX/XviD playback.

Test 1: Basic format support
DVD-R/W
DVD+R/W
VCD
SVCD
miniDVD

Test 2: SVCD testing

Multiple audio streams
SVCD / CVD subtitles
Menus

Test 3: MPEG-4 Containers

AVI
Ogg
Matroska
MP4

Now for the time being it makes no sense to test for special features in non AVI formats because they are most likely not supported.

Test 4: Codecs

DivX3
DivX4
DivX5
XviD

Here we test multiple things: First a basic 2 CD rip (bitrate in the 900-1200kbit/s range) having one 128kbit/s VBR MP3 audio stream using no Advanced Simple Profile features (GMC, QPel, B-frames). Then the ASP features are progressively added. Using the same source and bitrate, GMC, QPel and B-frames are tested. In the case of XviD, multiple b-frames are also to be tested (max 3 consecutive b-frames). As test scene, we need something that has bitrate peaks, so an action scene (I found that the lobby shootout in the matrix was a good test for multiple b-frames, even at 1 CD bitrates).
Once the ASP features have been tested, the ones supported should be tested in an extreme situation (1 pass quantizer 2) to see if the player starts to stutter.
For DivX3 we should perform the same extreme testing to see how powerful the transcoding engine is.

Test 5: Audio

1x CBR MP3 @ 192kbit/s
1x VBR MP3 @ 160kbit/s
1x AC3 @ 448kbit/s
2x VBR MP3 @ 160kbit/s
2x AC3 @ 448kbit/s
1x AC3 @ 448kbit/s, 1x AC3 @ 192kbit/s
1x AC3 @ 448kbit/s, 1x VBR MP3 @ 160kbit/s

Test 6: DivX subtitles

AVI with integrated srt
External srt

Also, if there are any other formats the manufacturer claims to support, those should be tested as well.

Test 7: Multiple content on the same disc

VCD mpg burned as ISO
SVCD mpg burned as ISO
DVD VOB burned as ISO
JPEG images burned as ISO

And combinations thereof, for instance a VOB trailer, an SVCD mpg and a DivX on the same disc.

Test 8: audio playback

CBR MP3
VBR MP3
Ogg

Other formats should only be tested if the manufacturer claims to support anything else.

Test 9: playback facilities

Playlists
fast forward / backward for DVD and DivX (how many levels, how much is skipped with DivX, how long does it take for DivX)
Zooming capabilities
Proper aspect ratio (take a 1:1 AVI showing a perfect circle.. see how this is displayed on the TV).

Other codecs should only be tested for if specifically mentioned as supported by the manufacturer (also depending on the alternative container support.. if there's no ogg/Matroska support it makes no sense to even test for Ogg Vorbis audio).

Now to allow anybody to perform these tests without too much hassle, sample images might be advantageous, but before we go into that (or source selection), I'd first like to have an established test procedure, then we can deal with the rest.

bond · 27th September 2003, 16:49

great idea, this hardware test!!!

i really second that you want to test asp features like qpel and gmc too! great! imho every standalone player who has a big mpeg-4 sticker on it should be able to play these features

the same goes for the mp4 logo
many manufacturers list "mp4" as their features, so i would also test the .mp4 container if possible and not too much work

suggestion for the procedure:
1) perhaps we first should list what players are on the market which support divx...
2) than we should decide about very specific a/v samples to test (you already wrote about the codec settings) and upload these samples somewhere (perhaps already the final isos for burning to avoid problems here)
3) than people who own a standalone player from the list can download these specific video files and test them on their player and give us feedback

i know that this is a little bit "user dependend" but...

SeeMoreDigital · 27th September 2003, 18:52

I think it may also be useful to know which standalone DVD players can spin RAW Mpeg1 and 2 files. As not everyone can be bothered with authoring!

Useful too for users who might like to store 30mins episodes of a TV series in Mpeg2 direct to a CD-R (after capturing) instead of using a DVD R/RW

Cheers

Zhnujm · 27th September 2003, 19:02

I think such a test would be a great idea, as there is absolutly no reliable source for such information in the net.
I would add a DivX3 test, not really important but for completeness.
The procedure bond describes sounds good but i wonder if such sample files would create to much traffic.

Doom9 · 27th September 2003, 21:01

I've added DivX subs, DivX3 and raw content tests.
Let's first define the test, then we define the source and only then we worry about the distribution.. trying to solve everything at once will only get us stuck.

SeeMoreDigital · 27th September 2003, 21:13

Doom9

It's just not fair that you can edit your posts without the little 'Last edited....' tag appearing below your signature.

No it's just not fair at all!

Cheers

zag2me · 29th September 2003, 11:20

Ok, great idea Doom9.

I was going to talk to the xvid developers about some kind of hardware certification but we will start with this.

I will post this test on our KISS forums, and get back to you with some preliminary results from the DP-450 and DP-500. This could show if the tests are feasible in a review situation.

My ideas for testing criteria.

1) Network streaming capabilities
2) Codecs - support for 3IVX
3) Audio - OGG audio
4) Special additional features - Such as webradio, video on demand ect.

Cheers.

zag2me · 29th September 2003, 13:21

Some more ideas from BBDOC on our forums:

Quote:

Some stuff I think should be added to this test:

In Multicontent part:
- Mpeg + Avi
- Video + Images
- Video + MP3
- Images + MP3

in MPEG-4 Containers
- Qpel support

in Audio part:
- wma support

+ somewhere else (don't know where!):
- Manual Zoom function
- Automatic Zoom to fullscreen
- FF/RW functionnality

These tests are based on my personal experience with the player (things that doesn't work with my DP-450!)... as well as on rumours I heard about other players (it seems the Philips DVD-737 doesn't allow you to zoom, which I find is a BIG problem!).

SeeMoreDigital · 29th September 2003, 13:39

Quote:

Originally posted by zag2me
Some more ideas from BBDOC on our forums:
in MPEG-4 Containers
- Qpel support....

Do any of the Kiss players provide this yet?

Cheers

zag2me · 29th September 2003, 13:42

The new sigma chips will support Qpel I am lead to beleave, but no players do at the moment.

I'v just had another idea

A test for multiple audio tracks, on all codecs.

SeeMoreDigital · 29th September 2003, 14:01

Thanks for the update zag2me

Maybe, when such players arrive I'll take a look at them. But at the moment I'll stick with my Xcard (or should I say Xcards as we have 4 'Hire Out' PC's with them)!

I think if we are not careful this thread will end up being full of 'wishful thinking' capabilities and as a result become impossible finalize.

Where will it end.... With RealMedia, QuickTime, WinMedia9, RAW DV.AVI audio and video file playback!!!!!

Cheers

Zhnujm · 29th September 2003, 15:29

Maybe a test for UDF DVD and filesizes >2GB would be a good idea. My first player did not support UDF at all and the current cannot ff/rw AVIs on UDF DVDs

masken · 29th September 2003, 18:12

Regarding subtitles:

- multiple subtitles according to de-facto standard, like:
\somemovie.avi
\somemovie.English.srt
\somemovie.Swedish.srt
\somemovie.Dutch.sub

- Subtitle formatting (italics, bold, underlined text)
...in their different formats

- subtitle formats
Subrip (*.srt)
MicroDVD (*.sub)
Substation Alpha (*.ssa)
SAMI (*.smi)
SubViewer 1/2 (*.sub, *.txt)

... I guess vobsubs won't be interesting to test at all?

Testing the CDFS would also be interesting for bugs I guess. ISO level 1/2, character set support, Joliet, pathdepths, more than 255 characters in path?

Lord_KiRon · 29th September 2003, 18:47

Yes and I would add OGM WMV and ASF categories for testing.
I do not know any players yet that suppot them but even putting them into a test bench would create a pressure for DVD makers to support

MS codecs support in AVI is another thing we can test.

Also don't forget ABR for audio part.

Btw: KiSS does not support UDF DVDs at all !!!!

SeeMoreDigital · 29th September 2003, 19:03

Quote:

Originally posted by Lord_KiRon
Yes and I would add OGM WMV and ASF categories for testing.
I do not know any players yet that suppot them but even putting them into a test bench would create a pressure for DVD makers to support

MS codecs support in AVI is another thing we can test.

Also don't forget ABR for audio part.

Btw: KiSS does not support UDF DVDs at all !!!!

Personally, I don't think we will see Microsoft WMV9 .wmv or .avi hardware players in the near future.

I think that all M$'s content will be encrypted. In much the same way it's T2 Extreme DVD content is encrypted and requires authorization prior to viewing. So in my opinion M$ will only allow it's codec to be used in DVD players that are connected to a network!

Cheers

Doom9 · 29th September 2003, 19:55

@zag2me: QPel was already there
I've added audio-only testing, but as with the codecs, it makes no sense to establish a test that's never used (exotic audio or video formats.. only if a manufacturer claims to support should it be tested). For instance, so far no other codecs are supported so I don't think that belongs into a standard test (btw: the elta plays nerodigital if you use the divx fourcc

I've also added jpeg images, and special features (zoom and such).

Since DVDs should be in a bridge ISO/UDF format, I don't think UDF DVD testing is warranted.. no burning prog creates UDF only DVDs onless you force it to.

@masken: you're overshooting the goal.. subs are currently not even supported, and all the formats you listed will never be supported. This isn't a test to cover 100%, but the 80% that matter for most people.

@Lord_KiRon: ABR is VBR with some limits

And I agree with zag2me that ASF and WMV testing is not useful. Don't forget that this test is designed to compare existing players... we should not devise tests that cover whatever combination we can imagine (the test as it is is already extremely complex and will take a long time to complete.. in the end I believe we'll have to cut a few corners to get something useful).

@all: when making suggestions, try to imagine that you're the person who has to perform those tests... only then can you decide what's really worth testing

I only performed a small part of those tests on my elta player and it already took me hours, preparing sources and watching them.

SeeMoreDigital · 29th September 2003, 20:15

Hi Doom9,

I've read good things about the elta player. Shame it's not available in the UK (or not as I've been able to find)!

So am I correct in assuming that you're able to test the new NeroDigital application. If that's the case is it worth waiting for?

Cheers

Stux · 30th September 2003, 07:59

I would suggest doing at least two tests for "MP4 Support"

1) AAC.MP4 (SP video + LC-AAC audio in an MP4)
2) MP3.MP4 (SP video + CBR-MP3 audio in an MP4)

You can go further with this test if you want

HE-AAC, Multi-Channel, VBR MP3, ASP features, etc

But its important to test both AAC and MP3 support in a player's MP4 support.

Other issues which can crop up with MP4 support are negative/positive track offset support and VFR (variable framerate) support

masken · 30th September 2003, 12:10

Quote:

Originally posted by Doom9
@masken: you're overshooting the goal.. subs are currently not even supported, and all the formats you listed will never be supported. This isn't a test to cover 100%, but the 80% that matter for most people.

Doom9, I don't think that's overshooting anything at all. The KiSS player already has support for text-based subtitles. It doesn't support formatting though (italics, bold, underlined etc), and thereby display those tags wrong (ie; a MicroDVD italic marker - {y:i} is displayed in the subtitle instead of the text being italic - or as would be preferred - just regular text), but I think it actually supports most formats.

Testing this would be easy, just burn a disc for each of the most common formats (the one's mentioned for example) and try to play it. Then have one disc with multiple subtitles on it (and perhaps one or two more for testing naming conventions).

It's just a case of really easy firmware programming, and I think there will be support for this once the manufacturers gets heads up for it. And it has importance, since all the worlds Level 2 + EFL English speakers (~2/3 of the worlds population) does use subtitles when watching features. Have a look at this too for reference...

zag2me · 30th September 2003, 18:29

I have to agree that the tests should be as simple as possible, the more complicated they get the more time it takes to test and more errors sneak into the results. I dont want to have to burn 5 different disks just to test what subtitles are supported, when it is clear from the product docs that they are not supported.

One test is fine for subtitles, most players dont even support subs yet,

This test is not about getting new features or bugs out of the current crop of players, it should be about comparing them against each other in a standard way. There is no need to complicate things.

27th September 2003, 15:48	#1 \| Link
Doom9 clueless n00b Join Date: Oct 2001 Location: somewhere over the rainbow Posts: 10,258	suggestion for standardized testing Let's collect ideas for a standardized DVD/DivX player test here. Considering that there are many ressources for basic DVD player testing already, I'd like to focus this on DVD backups, and specifically on DivX/XviD playback. Test 1: Basic format support DVD-R/W DVD+R/W VCD SVCD miniDVD Test 2: SVCD testing Multiple audio streams SVCD / CVD subtitles Menus Test 3: MPEG-4 Containers AVI Ogg Matroska MP4 Now for the time being it makes no sense to test for special features in non AVI formats because they are most likely not supported. Test 4: Codecs DivX3 DivX4 DivX5 XviD Here we test multiple things: First a basic 2 CD rip (bitrate in the 900-1200kbit/s range) having one 128kbit/s VBR MP3 audio stream using no Advanced Simple Profile features (GMC, QPel, B-frames). Then the ASP features are progressively added. Using the same source and bitrate, GMC, QPel and B-frames are tested. In the case of XviD, multiple b-frames are also to be tested (max 3 consecutive b-frames). As test scene, we need something that has bitrate peaks, so an action scene (I found that the lobby shootout in the matrix was a good test for multiple b-frames, even at 1 CD bitrates). Once the ASP features have been tested, the ones supported should be tested in an extreme situation (1 pass quantizer 2) to see if the player starts to stutter. For DivX3 we should perform the same extreme testing to see how powerful the transcoding engine is. Test 5: Audio 1x CBR MP3 @ 192kbit/s 1x VBR MP3 @ 160kbit/s 1x AC3 @ 448kbit/s 2x VBR MP3 @ 160kbit/s 2x AC3 @ 448kbit/s 1x AC3 @ 448kbit/s, 1x AC3 @ 192kbit/s 1x AC3 @ 448kbit/s, 1x VBR MP3 @ 160kbit/s Test 6: DivX subtitles AVI with integrated srt External srt Also, if there are any other formats the manufacturer claims to support, those should be tested as well. Test 7: Multiple content on the same disc VCD mpg burned as ISO SVCD mpg burned as ISO DVD VOB burned as ISO JPEG images burned as ISO And combinations thereof, for instance a VOB trailer, an SVCD mpg and a DivX on the same disc. Test 8: audio playback CBR MP3 VBR MP3 Ogg Other formats should only be tested if the manufacturer claims to support anything else. Test 9: playback facilities Playlists fast forward / backward for DVD and DivX (how many levels, how much is skipped with DivX, how long does it take for DivX) Zooming capabilities Proper aspect ratio (take a 1:1 AVI showing a perfect circle.. see how this is displayed on the TV). Other codecs should only be tested for if specifically mentioned as supported by the manufacturer (also depending on the alternative container support.. if there's no ogg/Matroska support it makes no sense to even test for Ogg Vorbis audio). Now to allow anybody to perform these tests without too much hassle, sample images might be advantageous, but before we go into that (or source selection), I'd first like to have an established test procedure, then we can deal with the rest. __________________ For the web's most comprehensive collection of DVD backup guides go to www.doom9.org

27th September 2003, 16:49	#2 \| Link
bond Moderator Join Date: Nov 2001 Posts: 9,780	great idea, this hardware test!!! i really second that you want to test asp features like qpel and gmc too! great! imho every standalone player who has a big mpeg-4 sticker on it should be able to play these features the same goes for the mp4 logo many manufacturers list "mp4" as their features, so i would also test the .mp4 container if possible and not too much work suggestion for the procedure: 1) perhaps we first should list what players are on the market which support divx... 2) than we should decide about very specific a/v samples to test (you already wrote about the codec settings) and upload these samples somewhere (perhaps already the final isos for burning to avoid problems here) 3) than people who own a standalone player from the list can download these specific video files and test them on their player and give us feedback i know that this is a little bit "user dependend" but... __________________ Between the weak and the strong one it is the freedom which oppresses and the law that liberates (Jean Jacques Rousseau) I know, that I know nothing (Socrates) MPEG-4 ASP FAQ \| AVC/H.264 FAQ \| AAC FAQ \| MP4 FAQ \| MP4Menu stores DVD Menus in MP4 (guide) Ogg Theora \| Ogg Vorbis use WM9 today and get Micro$oft controlling the A/V market tomorrow for free Last edited by bond; 27th September 2003 at 16:51.

27th September 2003, 18:52	#3 \| Link
SeeMoreDigital Life looks better in HDTV Join Date: Jun 2003 Location: UK Posts: 10,009	I think it may also be useful to know which standalone DVD players can spin RAW Mpeg1 and 2 files. As not everyone can be bothered with authoring! Useful too for users who might like to store 30mins episodes of a TV series in Mpeg2 direct to a CD-R (after capturing) instead of using a DVD R/RW Cheers __________________ Still confused about anamorphic images? Then look here! \| Main Encoding Interests: Hardware device playback \| My Gear \|

27th September 2003, 21:01	#5 \| Link
Doom9 clueless n00b Join Date: Oct 2001 Location: somewhere over the rainbow Posts: 10,258	I've added DivX subs, DivX3 and raw content tests. Let's first define the test, then we define the source and only then we worry about the distribution.. trying to solve everything at once will only get us stuck. __________________ For the web's most comprehensive collection of DVD backup guides go to www.doom9.org

27th September 2003, 21:13	#6 \| Link
SeeMoreDigital Life looks better in HDTV Join Date: Jun 2003 Location: UK Posts: 10,009	Doom9 It's just not fair that you can edit your posts without the little 'Last edited....' tag appearing below your signature. No it's just not fair at all! Cheers __________________ Still confused about anamorphic images? Then look here! \| Main Encoding Interests: Hardware device playback \| My Gear \|

27th September 2003, 19:02	#4 \| Link
Zhnujm Registered User Join Date: Oct 2001 Posts: 983	I think such a test would be a great idea, as there is absolutly no reliable source for such information in the net. I would add a DivX3 test, not really important but for completeness. The procedure bond describes sounds good but i wonder if such sample files would create to much traffic.

29th September 2003, 11:20	#7 \| Link
zag2me Registered User Join Date: May 2003 Posts: 67	Ok, great idea Doom9. I was going to talk to the xvid developers about some kind of hardware certification but we will start with this. I will post this test on our KISS forums, and get back to you with some preliminary results from the DP-450 and DP-500. This could show if the tests are feasible in a review situation. My ideas for testing criteria. 1) Network streaming capabilities 2) Codecs - support for 3IVX 3) Audio - OGG audio 4) Special additional features - Such as webradio, video on demand ect. Cheers. Last edited by zag2me; 29th September 2003 at 11:34.

29th September 2003, 13:42	#10 \| Link
zag2me Registered User Join Date: May 2003 Posts: 67	The new sigma chips will support Qpel I am lead to beleave, but no players do at the moment. I'v just had another idea A test for multiple audio tracks, on all codecs.

29th September 2003, 14:01	#11 \| Link
SeeMoreDigital Life looks better in HDTV Join Date: Jun 2003 Location: UK Posts: 10,009	Thanks for the update zag2me Maybe, when such players arrive I'll take a look at them. But at the moment I'll stick with my Xcard (or should I say Xcards as we have 4 'Hire Out' PC's with them)! I think if we are not careful this thread will end up being full of 'wishful thinking' capabilities and as a result become impossible finalize. Where will it end.... With RealMedia, QuickTime, WinMedia9, RAW DV.AVI audio and video file playback!!!!! Cheers __________________ Still confused about anamorphic images? Then look here! \| Main Encoding Interests: Hardware device playback \| My Gear \|

29th September 2003, 15:29	#12 \| Link
Zhnujm Registered User Join Date: Oct 2001 Posts: 983	Maybe a test for UDF DVD and filesizes >2GB would be a good idea. My first player did not support UDF at all and the current cannot ff/rw AVIs on UDF DVDs

29th September 2003, 18:12	#13 \| Link
masken uhm... ? Join Date: Oct 2001 Location: Gothenburg, Sweden Posts: 276	Regarding subtitles: - multiple subtitles according to de-facto standard, like: \somemovie.avi \somemovie.English.srt \somemovie.Swedish.srt \somemovie.Dutch.sub - Subtitle formatting (italics, bold, underlined text) ...in their different formats - subtitle formats Subrip (.srt) MicroDVD (.sub) Substation Alpha (.ssa) SAMI (.smi) SubViewer 1/2 (.sub, .txt) ... I guess vobsubs won't be interesting to test at all? Testing the CDFS would also be interesting for bugs I guess. ISO level 1/2, character set support, Joliet, pathdepths, more than 255 characters in path?

29th September 2003, 18:47	#14 \| Link
Lord_KiRon Registered User Join Date: Aug 2002 Posts: 151	Yes and I would add OGM WMV and ASF categories for testing. I do not know any players yet that suppot them but even putting them into a test bench would create a pressure for DVD makers to support MS codecs support in AVI is another thing we can test. Also don't forget ABR for audio part. Btw: KiSS does not support UDF DVDs at all !!!!

29th September 2003, 19:55	#16 \| Link
Doom9 clueless n00b Join Date: Oct 2001 Location: somewhere over the rainbow Posts: 10,258	@zag2me: QPel was already there I've added audio-only testing, but as with the codecs, it makes no sense to establish a test that's never used (exotic audio or video formats.. only if a manufacturer claims to support should it be tested). For instance, so far no other codecs are supported so I don't think that belongs into a standard test (btw: the elta plays nerodigital if you use the divx fourcc I've also added jpeg images, and special features (zoom and such). Since DVDs should be in a bridge ISO/UDF format, I don't think UDF DVD testing is warranted.. no burning prog creates UDF only DVDs onless you force it to. @masken: you're overshooting the goal.. subs are currently not even supported, and all the formats you listed will never be supported. This isn't a test to cover 100%, but the 80% that matter for most people. @Lord_KiRon: ABR is VBR with some limits And I agree with zag2me that ASF and WMV testing is not useful. Don't forget that this test is designed to compare existing players... we should not devise tests that cover whatever combination we can imagine (the test as it is is already extremely complex and will take a long time to complete.. in the end I believe we'll have to cut a few corners to get something useful). @all: when making suggestions, try to imagine that you're the person who has to perform those tests... only then can you decide what's really worth testing I only performed a small part of those tests on my elta player and it already took me hours, preparing sources and watching them. __________________ For the web's most comprehensive collection of DVD backup guides go to www.doom9.org

29th September 2003, 20:15	#17 \| Link
SeeMoreDigital Life looks better in HDTV Join Date: Jun 2003 Location: UK Posts: 10,009	Hi Doom9, I've read good things about the elta player. Shame it's not available in the UK (or not as I've been able to find)! So am I correct in assuming that you're able to test the new NeroDigital application. If that's the case is it worth waiting for? Cheers __________________ Still confused about anamorphic images? Then look here! \| Main Encoding Interests: Hardware device playback \| My Gear \|

30th September 2003, 07:59	#18 \| Link
Stux Registered User Join Date: Dec 2002 Location: Sydney, Australia Posts: 252	I would suggest doing at least two tests for "MP4 Support" 1) AAC.MP4 (SP video + LC-AAC audio in an MP4) 2) MP3.MP4 (SP video + CBR-MP3 audio in an MP4) You can go further with this test if you want HE-AAC, Multi-Channel, VBR MP3, ASP features, etc But its important to test both AAC and MP3 support in a player's MP4 support. Other issues which can crop up with MP4 support are negative/positive track offset support and VFR (variable framerate) support __________________ http://www.3ivx.com

30th September 2003, 18:29	#20 \| Link
zag2me Registered User Join Date: May 2003 Posts: 67	I have to agree that the tests should be as simple as possible, the more complicated they get the more time it takes to test and more errors sneak into the results. I dont want to have to burn 5 different disks just to test what subtitles are supported, when it is clear from the product docs that they are not supported. One test is fine for subtitles, most players dont even support subs yet, This test is not about getting new features or bugs out of the current crop of players, it should be about comparing them against each other in a standard way. There is no need to complicate things.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode