We’ve entered a new era of data-informed decision-making. The capabilities of data storage continue to grow as the cost of storing data goes down. And you don’t have to be harnessing “Big Data” to perform market research, competitive analysis, or user testing.
The research we need to drive business decisions is often at our fingertips, and that usually includes compiling other companies’ research to make our point. The problem is: as the research we use gets further away from its original context, actual data can get mixed with interpretations—sometimes until it becomes some different entirely.
Do you know how to vet research? Are you willing to do so even when it supports your opinion? Are you accidentally misrepresenting data publicly?
From Changing Features to Changing Products
This week, I read an article from Voxa about logging data in a CRM. At the time, they were directly quoting a statistic that seemed…off to me. They linked over to a SalesLoft post that had this in an infographic:
72% of all your users said they would trade in all of the functionality they hold so highly for a CRM that’s just easier to use.
I love both of these companies, but that statistic makes it sound as though nearly 3 out of 4 CRM users would literally change platforms, regardless of features, if they were promised a better user experience. As a UX designer in a related field, I enjoy seeing the importance of design highlighted—but that seemed far too good to be true.
Luckily (and wisely), SalesLoft linked to their sources. When I found the statistic they were citing on their linked source, it actually said:
72 percent indicated they would trade functionality for ease of use
Well, that’s already not the same thing at all. The phrase “trade functionality for ease of use” means something totally other than “would trade in all of the functionality they hold so highly for a CRM that’s just easier to use”. Users across all sorts of platforms would trade functionality for ease of use, because that’s often the perceived trade-off.
I wanted to see how people “indicated” the above, though, so I kept digging further. It was no small task, as many sites had been using the second quote without any attribution. I finally found the original research from 2007 (and plenty has changed in the CRM space in 7 years), but there’s no link to the actual data. After engaging with the company directly, they graciously sent me a copy of the survey and the results. Imagine my surprise to see the actual wording of the question:
Generally speaking I would be prepared to trade some functionality in my CRM system for greater ease of use
Participants answered on a scale from “strongly disagree” to “strongly agree”, and the 72% was calculated by adding the number of “agree” and “strongly agree”.
In that same survey, about 43% said they use less than half of the functionality in their current CRM.
So—ahem—of course most people would trade “some functionality” for “greater ease of use”. They’re not using all of it as it is! Wouldn’t you trade functionality you weren’t using to get ease of use in the parts you do?
Though the actual data wasn’t published, I think the original company (Really Simple Systems) still fairly represented their results with the phrasing they used. All it took was a few citings of the data and a desire to infuse a bit of personality to get one thing to mean another.
Get Source
I recommend that you always trace important research back to its source. Yes, it will take more time, and you will have to be willing to seek out contradictory data. But, having the actual data both fully informs and exonerates you.
It’s not especially fun to read academic papers or spreadsheets, but it’s worth the hassle. Don’t get stuck hinging your argument on a rephrase of a rephrase of a rephrase of research.