Trusted Testers ?

[vio_man]

I have an idea:

Create a tool which creates an unique hardware ID for each computer (based on RAM, processor, Windows, etc) and create a customized version for each tester which will only run in the tester's computer. Even if leaked it will have to be cracked in order to run in another system. Also if the tester changes the hardware in the mean time, you could send him/her a new version for the new hardware ID and block the older one.

[DeeGee]

I would also like to be part of this, but I don't think I'll be so lucky... But anyway exited to hear what feature have you added.

Too bad about the leak before. Hopefully this time these strict rules stop it from happening.

[Sirber]

Originally posted by vio_man
I have an idea:

Create a tool which creates an unique hardware ID for each computer (based on RAM, processor, Windows, etc) and create a customized version for each tester which will only run in the tester's computer. Even if leaked it will have to be cracked in order to run in another system. Also if the tester changes the hardware in the mean time, you could send him/her a new version for the new hardware ID and block the older one.
Complicated. less work to trust people :)

[Neo Neko]

Originally posted by Sagittaire
[Mode joke On]
it's very simple ... don't believe Russian tester ... lol
[/Mode joke Off]

&lt joke&gt
Your XML structure is invalid. Hopefully no MS products were involved. :p

The syntactically correct XML enclosure would be &lt mode joke="on"&gt &lt /mode joke="on"&gt or &lt joke&gt &lt /joke&gt . Can't have computer geeks going around without propperly encapsulating their moods etc. :D
&lt /joke&gt

[dragongodz]

In my situation, I have a lot of income that could be sued for and I am not willing to put my lively hood on the line for anyone. This is more than a beta test for some of us, especially when you figure in legal liabilities and financial ramifications.
whats interesting is that bobololo did not threaten to sue anyone, others started that. all he said was the reccomender would also be fired from the testing. but hey if you trust absolutly nobody on this forum thats up to you.

When criticising this whole approach don't forget that our friends at Ateme inevitably had to come up with some kind of security scheme as a precondition if they were to justify getting their beta-testers from this community once more.
of course. i dont think anyone is saying give out the beta to every man and dog. on the same hand the extremes some people seem to want to go to is a different story. i am waiting for the retina scan, dna test and anal probe suggestions. they cant be far away. :D

What they came up with is as good and non-formal as any solution based on trust goes. Time to let it go...
sorry Teegedeck but to quote bobololo
This thread wasn't intended to give the kick off of a new beta but instead a discussion about how it could be done
so infact this thread is for peoples opinions on whats good, whats bad and how it could be done differently. thats the whole point of this thread.

[vahegan]

Hi, I would like to participate, if possible. I do lots of encodings daily and I am eager to try new stuff from Ateme. I would be very happy if I am selected as a beta tester for this project.
I think what vio_man suggested is simple and efficient, and can provide good enough protection from leaks.

[plonk420]

i'm in and wouldn't mind printing/signing/scanning an NDA. and now that Tiger's unleashed i have 3 platforms to test on... i may even have access to a HD camera (sony's HDV unit). all i'd need to figure out is how to get footage off the unit using software the workplace owns... (avid xpress pro, fcp hd, and adrenaline) preferrably without transcoding...


edit: oops, just noticed that the NDA was someone else's suggestion. as for original testers' subset testers, would they have to be from this board or just people they "know" and trust from anywhere else?

[plonk420]

Originally posted by Neo Neko
&lt joke&gt
Your XML structure is invalid. Hopefully no MS products were involved. :p

The syntactically correct XML enclosure would be &lt mode joke="on"&gt &lt /mode joke="on"&gt or &lt joke&gt &lt /joke&gt . Can't have computer geeks going around without propperly encapsulating their moods etc. :D
&lt /joke&gt

in Soviet Russia, software tests YOU :p

*hides, chastising self for spreading anandtech forum post jokes*

[thegeby]

Happy to see that Ateme is gearing up for more testing. I think the sponsor option is a valid one. An additional possibility would be to require real names in the beta forum. I know some of us (not me, I really am Thegeby, and really ain't not got none to begin with) would lose some street cred, but it does put a bit more on the line...

[Selur]

just a little thought:
Verification could be made via snail mail. A german mail provider does it via snail mail. You register online, they send you a postcard/letter you have to sign (iirc) and then send it back to them, so they know where you live,...

Cu Selur

[Andrey]

>>so they know where you live
Mafia ? :P

[bill_baroud]

I know this post is to discuss how this new test could be done, but if i could participate again, i would be delighted (now that i've time). And i could come in person to Ateme (Rennes) to sign any NDA or anything...

[Teegedeck]

Originally posted by dragongodz

sorry Teegedeck but to quote bobololo

so infact this thread is for peoples opinions on whats good, whats bad and how it could be done differently. thats the whole point of this thread.
You are, of course, correct. So, on to business.

The whole problem is that what Ateme probably aims for is not the ability to sue anyone who doesn't adhere to the conditions of the test (would be costly, never a threat certain enough to deter everyone, and bad rep in the community), after the fact becomes known, but to prevent anyone from violating those conditions beforehand.

So in order to build a feeling of obligation they can a) deter or b) generate commitment. Verifying the participants' identity goes more into the direction of a), while the scheme of mutual responsiblity and trust described earlier on goes more into the direction of b).

I would prefer b); mainly because it is easier to do, and immediately. Now, in order to allow for including 'newbies' I would propose that every already 'trusted' tester is obliged to certify, say, one new participant as 'trusted'. If that new participant would leak material, the one who vouched for him gets of course fired, also, just like initially described. That would be a very unfair scheme, yeah, but it would be probable to produce a stronger commitment of the new participant. Or not? ;)

[Doom9]

< mode joke="on"> </mode joke="on">Actually, attributes are to be placed only in the opening tag, so it would have to be <mode joke="on">blahblah</mode>

[Neo Neko]

Originally posted by Doom9
Actually, attributes are to be placed only in the opening tag, so it would have to be <mode joke="on">blahblah</mode>

Indeed. DOH! :p

[IgorC]

bad_joke function (sagitaries joke)
{ do
{ printf(" Keep your jokes with yourself")
} while (joke!=´/0´);
return rebad_joke;
}; :p

[Sirber]

Originally posted by IgorC
bad_joke function (sagitaries joke)
{ do
{ printf(" Keep your jokes with yourself")
} while (joke!=´/0´);
return rebad_joke;
}; :p
Compiler error :p

[Gabriel_Bouvigne]

The problem is that you will perhaps have a hard time finding new testers.
Myself, as an example, would not recommand anyone in this scheme (except people already in your list). I am probably trustworthy (at least I think so), but as I am mainly an audio guy, I do not know video people "deeply" enough. I think that I am probably not alone in a similar situation.
I think that you would have new testers only if the recommandation by 2 people do no imply to be "liable" for the behavior of the recommanded.

This could be handled using the usual "circle" way, as in the IT Who's Who, as a example: new testers can only be suggested by existing testers, but should not ask themselves to become a tester.


A snail mail NDA could also be fine, IMHO.

[Bulletproof]

I'd love to give the new version a go, it was really great to be in the previous beta and just like the last test I would try my best to provide useful feedback to improve your product. One thing in particular that I'm really hoping for is the pause button to not actually cause the encoder to become deadlocked after running a CPU intensive application like a game. Not having to use your computer for 10+ hrs. is really hard :p and I'm really tempted to play a game and pause my encoding.

[Doom9]

I've been playing games while encoding in the background.. even did that when I was encoding for the codec comparison. While encoding, I finished NFSU2 and played through Half Life 2.

[Sirber]

Makes the encoding encode at 0 FPS. Best have a second computer or do it during the night.

[Doom9]

games hardly use up all cpu resources.. the GFX card is usually the limiting factor. And obviously if you need your CPU power for something else.. don't expect fast encoding. You can have a 8-way system and still use 100% cpu resources. And you'll never have a fast enough CPU ever.

let's get back on topic now. The idea of having to vouch for testers is a good thing.. it's quite harsh that the referer is also kicked, but having a personal stake ensures that only really trusthworthy people get in. And it's not like it's not possible to that encoder gets into hands it's not supposed to.. you could theoretically even share it with family and friends that are trustworthy.. you just need to be sure they're not going to screw you.

[Joe Fenton]

Okay, no NDA, no postcards, no IDs, no dongles. What you do is make the codec ask for a password each week. The password is programmed to change by a complicated set of rules partly based on the date. Each week, email new passwords to the testers. If a codec leaks, you stop sending that person the passwords. Anyone using the codec is stuck setting their clock back anytime they wish to use the codec, so they won't bother. Too much hassle for a beta codec. Sound good?

[IgorC]

Too much extra job for testers and developers. Some people will try to change current date to prolong ability of codec.

[bobololo]

Some of you suggested the beta to be covered by a NDA. To me this NDA would theoricaly offer us the possibility to start a legal action against a leaker. Considering that fact, all testers would then be afraid of getting sued and therefore won't leak. That's a good point.

However I've still a concern. How can we be sure that a NDA isn't executed using false personal info ? If I was a malicious testers, I would fill the NDA with John Doe, living somewhere in US and send it back by snail mail. The NDA would therefore be totally useless and I'd free to leak the beta without any risks.

How can we circumvent this ?

[bobololo]

Originally posted by Joe Fenton
Okay, no NDA, no postcards, no IDs, no dongles. What you do is make the codec ask for a password each week. The password is programmed to change by a complicated set of rules partly based on the date. Each week, email new passwords to the testers. If a codec leaks, you stop sending that person the passwords. Anyone using the codec is stuck setting their clock back anytime they wish to use the codec, so they won't bother. Too much hassle for a beta codec. Sound good?

To be honest, I don't very like the idea of software lock or limitation. IMHO, if we don't take care to whom we give the beta material away, whatever protection you could implement, if it happens to be spread on the net, it will be cracked by a way or another. That's why our rule n°1 is "the beta material must not leak".

[bobololo]

Just to have an idea of how things could take place.

Refering to my first post rules, if you were selected as an initial trusted tester, would you give your consent to any other volunteers and if so how many forum users would you personaly trust ?

[dragongodz]

all testers would then be afraid of getting sued and therefore won't leak. That's a good point.
the only problem there is if the person making a reccomendation is also afraid of being sued for someone else leaking. there is 1 example in this thread already of a person so afraid they wont reccomend anyone. thats pretty sad considering how many good and trustworthy people there are on this forum that someone is so afraid they basically say they wont trust anybody.

To be honest, I don't very like the idea of software lock or limitation.
fair enough. how about each version of the codec has a personnal code hidden in it ? nothing obvious such as a name or something easy to search for but a small sequence that would enable it to be identified from whom it came ?

[P0l1m0rph1c]

Originally posted by dragongodz

fair enough. how about each version of the codec has a personnal code hidden in it ? nothing obvious such as a name or something easy to search for but a small sequence that would enable it to be identified from whom it came ?

That was how they found out who leaked the last time.

[LigH]

Originally posted by bobololo
If I was a malicious testers, I would fill the NDA with John Doe, living somewhere in US and send it back by snail mail. The NDA would therefore be totally useless and I'd free to leak the beta without any risks.

How can we circumvent this ?
If you are so concerned, then it may require several consecutive steps:

1) Tester orders a printed NDA using a valid postal address (no PO-boxes), e.g. via e-Mail
2) Ahead sends a letter containing the printed NDA and a code via e-Mail
3) Tester sends the signed NDA with code back to their next local Ahead branch office

Sounds like

a) almost too much efforts
b) still no guarantee for a "summonable/citatory" (?) address, though

[dragongodz]

That was how they found out who leaked the last time.
sorry it wasnt clear that it was the codec that was tagged since i just saw it mentioning watermarking. besides i was asking bobololo's opinion on it. since it did work first time then why not simply use that plus the reccomendations etc this time.

:sigh:
is that suposed to be clever ? dont be a smart-arse all your life mate.

[Gabriel_Bouvigne]

However I've still a concern. How can we be sure that a NDA isn't executed using false personal info ? If I was a malicious testers, I would fill the NDA with John Doe, living somewhere in US and send it back by snail mail. The NDA would therefore be totally useless and I'd free to leak the beta without any risks.
No, it should be the other way:
YOU send the NDA by snail mail to the tester, using the address he provided. Then he sends it back with a signature, and could even send it back by e-mail (photography of the signed NDA).

This does not specially provides way to sue people, but rather to know the real identity of testers. In case of leaking, to sue or not is only your decision. An NDA does not means that if you do not follow it you will be sued, only that you agreed to follow it. That is why I think signing an NDA does not put extra pressure on "normal" testers. It just provides identity info, which we have no need to hide anyway.

[Sharktooth]

Originally posted by bobololo
However I've still a concern. How can we be sure that a NDA isn't executed using false personal info ? If I was a malicious testers, I would fill the NDA with John Doe, living somewhere in US and send it back by snail mail. The NDA would therefore be totally useless and I'd free to leak the beta without any risks.

How can we circumvent this ?
Easy, send a digital document (pdf or whatever) containing the NDA by email to the potential tester.
The tester prints the docs, signs the NDA and includes a copy of an identity document and send the stuff back to you by UPS.
Once you get the stuff, you send the software by UPS at the given address.

[P0l1m0rph1c]

Originally posted by dragongodz

is that suposed to be clever ? dont be a smart-arse all your life mate.

Hey, man, don't take me wrong. It was just because it was already mentioned in thread a while back. I'll remove it anyhow.

[IgorC]

NDA + personal information is the best choice. Mayor of us hadn´t meet
each other in real life that´s why to obtain 2 votes from trusted testers is just impossible.

[Sirber]

UPS cost money. Email with download link is less expensive...

[edit]

Ha... to verify the address :rolleyes:

[bobololo]

After reading your latest comments, here is a new suggestion :

- Like previously we have some initial trusted testers.
- Those testers can recommend new volunteers but they won't be responsible for the recommended persons.
- A trusted tester can recommend at most 4 volunteers and a volunteer need 2 votes to get in.
- All testers (trusted & new ones) will receive by email a NDA template they have to execute and send back by snail mail to us. The testers will have to provide their personal info (name & address).
- Once received, we send by snail mail to the address provided by the tester a personal login & password required to proceed the beta program.

What do you think ?

[Gabriel_Bouvigne]

- All testers (trusted & new ones) will receive by email a NDA template they have to execute and send back by snail mail to us. The testers will have to provide their personal info (name & address).
- Once received, we send by snail mail to the address provided by the tester a personal login & password required to proceed the beta program.
I think that asking testers to send snail mail could be annoying for them, especially as some will have to send this mail in foreign countries.
As anyway you plan to send them snail mail, why not changing this part into:

*tester provide real ID and snail mail address.
*once received, Ateme sends by snail mail the NDA to fill and sign.
*testers send back the NDA by whatever mean they want, provided that there is at least a "manual" signature. Ways to send it back: by snail mail, by e-mail, by fax
*You then process as usual by e-mailing login and password

[Latexxx]

That sounds pretty reasonable. Though I'm not sure whether you are going to count me as trusted because I didn't sent much comments during the last test round.

[bond]

i doubt that lot of people will do this (snail mail sending) :p
i mean after all the testers are helping ateme, not the other way round, why should they mess around with snail mail, buying stamps and what not...

and why did the leak happen last time? because the beta has been sent to nearly everyone.
i think the easiest way would be to simply give it only to people who have been around for a long time on doom9 and other forums, like hydrogenaudio, and should be pretty reliable anyways
i think its manageable to permanently ban someone from doom9 if he leaks. any long time member wanting this?

after all even if the last beta has been leaked, i dont think that it really gets used (eg so you can see it as real competitor to nerodigital for example, threatening the business)
i mean its commandline (simply look at how many people cant use mp4box or mp4creator without gui) :D

[SeeMoreDigital]

Originally posted by Latexxx
That sounds pretty reasonable. Though I'm not sure whether you are going to count me as trusted because I didn't sent much comments during the last test round. Me too I'm afraid...

Coz I'm not testing via command line :(


Cheers

[bobololo]

Originally posted by bond
i doubt that lot of people will do this (snail mail sending) :p

If a tester can't even make the effort to go to the post office and send an airmail, I doubt he would be ready to make much effort during the beta ;) After all that only needs to be done once for all.

Considering the excitement and motivation demonstrated by some of the previous posts, I've the feeling it won't be a major issue. Tell me if I'm wrong.

ps: I didn't think about it, but that could be also a way to verify testers motivation ;)

[bobololo]

Originally posted by SeeMoreDigital
Me too I'm afraid...

Coz I'm not testing via command line :(

And it'll be CLI again. I couldn't imagine running a cli app could be an issue for a tester ?? What's the matter ???

[bond]

Originally posted by bond
i mean after all the testers are helping ateme, not the other way round, why should they mess around with snail mail, buying stamps and what not... :p

[dragongodz]

bobololo - thats seems much better. yes cost of mail is a bit of a pain but for plain airmail shouldnt be much.

the only thing missing is that there would also be a privacy agreement that peoples names and addresses would not be used for anything outside of the test or divulged to any third party etc etc etc. :)

[Gabriel_Bouvigne]

If a tester can't even make the effort to go to the post office and send an airmail
Have you tryed to go to a french post office recently? It usually means loosing 1 hour and a half saturday morning. This is 1.5 less for coding or testing. My personnal opinion is that you are asking too much in this case. There is a risk that some people would consider that they do not have this extra time to offer, as they are already offering some of their time for testing.

Even for contracting job you usually do not need to send back the contract and nda by snail mail, so why would a tester need to do it?

Of course this is only my personnal opinion...

[IgorC]

It´s not clear how trusted tester will select a new testers. Wich will be a criteries and requirements?

1. Operations with AVS, CLI etc.
2. Ability to post screenshots and upload big or even quite huge video files.
3. SSIM, PSNR and others tests for differnets options
4. Hardware
5. Software
6.. ?

Who are the trust testers?
Have the new testers to pm to trusted testers or just wait to be selected? Untill now I think I can obtain only 1 vote and have no information about how to obtain the second one.

[Doom9]

and why did the leak happen last time? because the beta has been sent to nearly everyone. Actually, a tester gave it to somebody else who then leaked it if I recall correctly.

i think its manageable to permanently ban someone from doom9 if he leaks. any long time member wanting this?as much as I despise people who do not honor the trust placed in them, such a measure would be highly problematic. It would mean an independent entity (me) would suddenly enforce ateme policy.

I'm in a position where I already have to trust more than 50 people implicitly (also known as moderators), so I'd definitely be able to make a recommendation, without having even met a single one in person. A potential moderator is usually suggested by an existing moderator, and they qualify much more by their actions than by offering their services. I figure it wouldn't be much different for beta testers. Beta testing, when properly done, is more a waste of your time as service to whomever wrote the software you're testing, than anything else. It's not fun, it's supposed to be boring and you ought to be able to do better things with your time. So, those that scream out "me me" usually have a different agenda.

[eb]

A little OT.
What we are going to test here, A-:devil: project?
I am/was physicist, may be I can help, joking cos not expecting to get to testers circle.

eb

[Sirber]

I have no clue on what snail mail is, and I have no intention of using it. This thing is by far too troublesome. Until this change, count me off.