Mozilla creates Shield study rules to avoid another Mr.Robot disaster

Mozilla created a set of guiding principles for Shield studies after it launched an analysis of the Looking Glass Shield study which went wrong on several levels.

Looking Glass was released as a system add-on to Firefox which meant that users saw the add-on appear in the browser's add-on manager without them initiating the installation.

This appearance out of thin air was arguably the biggest complaint that users had and something that got them alarmed because it shared the characteristics of malware. The fact that the initial description and add-on name did not reveal anything about the add-on's purpose added to the confusion as well.

The description of the add-on read "MY REALITY IS JUST DIFFERENT THAN YOURS" and "PUG Experience Group" was listed as the creator. Nothing linked the add-on to Mozilla in Firefox's add-on manager.

Mozilla announced shortly after the study blew up in the company's face that it had pulled the study, uploaded the add-on to Mozilla's Add-on repository, and started an investigation to "better understand how and why this happened", and how the company "could do better" in the future.

Shield studies are used to A-B test things in the Firefox web browser. The studies are used to test small and big changes to Firefox, from simple icon or color changes to new features, to find out whether the majority of users that have been selected for the study like the changes, or not.

Studies help Mozilla make better decisions in regards to future changes and features in the Firefox browser.

This platform helps us make decisions on new product features, evaluate whether or not a technology update is stable, and generally helps us make sure that we can make good decisions in a responsible way.

Looking Glass did pass the privacy review as it did not collect any data. The fact that it did not should have been a red flag, as it is impossible to evaluate a feature without collecting a bare minimum of data.

Mozilla create a set of principles for Shield studies so that something like Looking Glass won't happen again.

1. All Shield studies must answer specific questions.
2. Shield studies will always respect user privacy.
3. All Shield studies adhere to the "scientific method for answering complex questions".
4. All Shield studies require a Product Hypothesis Doc which outlines the research question the study is trying to answer.
5. All Shield studies must be named accurately.

Looking Glass would fail in all but the second.

Closing Words

It is definitely a good thing that Mozilla created a set of guiding principles for Shield Studies. I would have preferred if the organization revealed a bit more about Looking Glass itself: was Mozilla paid for the promotion and how did not anyone object to the study by pointing out the, rather obvious, issues it had? (via gHacks)