Erika Hall smacks down surveys that aren't designed
Like requirements that are merely gathered, and questionaires that aren't really thought out surveys can go all kinds of nasty if you don't know what you are doing with them and most importantly what you are trying to achieve with them.
It is similar to the finance industry relying on Excel to generate complex and accurate representation of important data... cough cough
J.P Morgan Chase & Co. and the 'London Whale Incident'| Why do projects fail site | May 2013 .
The 'magic' of Excel is that allows programmatic functions to be created by anyone. Many a shadow IT system has been built in Excel and delivered a good solution for business. The bad side is that unlike languages developers use everyday Excel is hopeless for debugging. Be careful or you could lose in excess on 2 Billion like J.P Morgan Chase & Co.
In exactly the same way as Erika puts it in her excellent Medium article
On Surveys | Medium | Erika Hall | Feb 2015
"Bad surveys don't smell" [i.e It is hard to recognise when you have made a mistake in survey design]
Most seductively, surveys yield responses that are easy to count and counting things feels so certain and objective and truthful.
Even if you are counting lies.
A lot of people in a lot of conference rooms are pretending that the easiest to gather data is the most useful.
Ever worked with analytics on web pages from more than one source. How fuzzy are they? Especially ones on the old skool metrics of traffic, hits, and visits. I guess what I aim saying is that you need to ensure you are measuring the right things and have someone appropriately skilled to interpret the data.
One of the issues with measuring in the digital environment is that often things are on a continuum and not well expressed in a binary measure. Take what your site will look like for instance. You might be able to say how it will be viewed in a particular version of a browser on your machine... but others may have different settings, browser versions, modes enabled, data cached, display size, etc. It gets complex really quickly. That is why you need to have a good test bench process and include responsive design in your web build and progressive enhancement and graceful degradation in your code.
My favourite bits from the article
Emphasis is mine.
never ask people what they like or don’t like. Liking is a reported mental state and that doesn’t necessarily correspond to any behavior.
Avoid asking people to remember anything further back than a few days. I mean, we’ve all been listening to Serial, right? People are lazy forgetful creatures of habit. If you ask about something that happened too far back in time, you are going to get a low quality answer.
And especially, never ask people to make a prediction of future behavior. They will make that prediction based on wishful thinking or social desireability. And this is the most popular survey question of all, I think
Making bets based on insights from observed human behavior can be far more effective that basing bets on bad surveys. So go forth, be better, and be careful about your data gathering. The most measurable data might not be the most valuable.
YEEEEES!!!! Although user researchers have been telling us this for years.
I also recently discovered that Erika also is behind the awesome business jargon and cliche busting site Unsuck It and wrote a handy book on user research 'Just enough research'.