Not OkCupid


The match-making service OkCupid recently ran several experiments on their users. The experiments ranged from ethically neutral to blatantly dishonest.

Experiment #1

OkCupid hid user profile pictures and measured how site usage changed. They monitored responses to messages, how deep conversations went, and what information was exchanged. The sole modification done for this experiment was disabling a feature (profile pictures), and it lasted less than a day. No harm done, and they got some interesting data.

Experiment #2

Next they wanted to see how much users based their opinion of someone else on the picture versus the actual information about them. To do this, they hid the text and monitored how perceptions changed (they didn’t). This is a little sketchy because they were temporarily modifying users’ profiles. But they weren’t adding anything, just omitting information. It makes me a little uneasy, but it could still fall under the “disabling a feature” category, so I’m not grabbing my pitchfork.

Experiment #3

OkCupid then decided they wanted to measure how much their matching percentages were affecting the user interactions. They decided to gauge the power of suggestion by giving users incorrect information. People who were a bad match (by OkCupid’s formula) were told they were a good match, and people who were a good match were told they were a bad match. To me, this falls under very sketchy territory.

Presidential Indifference

OkCupid’s President Christian Rudder went on All Things Considered to do some damage control. Give it a listen here. His attitude towards his customers was extremely alarming. He compared experiment #3 to taking part in a psychology experiment at college for $20. You know, the kind of experiment where you have to sign a consent form at the start. Rudder compared that to OkCupid giving their customers exactly the opposite of what they signed up for. He said OkCupid has a “tremendous amount of empathy” for their users, but also called the site’s terms and conditions a “charade of consent”. If he doesn’t believe his customers actually gave consent, what makes the experiment permissible?

UI Experimentation

Experimentation can mean many things. Many companies frequently run UI and UX experiments (slight modifications), monitoring how it affects the user’s behavior (like this Google experiment in 2008). OkCupid tried to equate their “experiment” to testing a new feature, which is disingenuous. They did it to run a sanity check on their match-making algorithm, at the cost of unsuspecting users. They defended this by pointing to the fact that they told the users involved- after the experiment was over.

Betrayed

OkCupid founder Christian Rudder says users who feel betrayed have a “misunderstanding”. How should they feel? They signed up for a site claiming to give them match ratings for a date, and the site purposefully gave them incorrect ratings. That sounds like a textbook definition of misplaced trust and betrayal to me. This type of behavior is disappointing, even if done in the name of verifying their product.

Alternatives

OkCupid claim the alternative to their experiment was having an “algorithm that was garbage”. This is untrue, since this experiment didn’t alter the actual algorithm. The true cost of not running the experiment would have been a lack of confidence in the algorithm. Were there no other options to verify the formula?


comments powered by Disqus

About

Hi, I'm Dominic. I'm a software developer working mostly with .NET and other web technologies. Check out some of my projects, or get in touch with me.