The Importance of Valid Survey Data

How Much Do You Trust Your Survey Results? 

At Insightlink, we live and breathe data. It's what we collect when we run surveys and it's what we analyze when we write reports. Without data, we'd all be looking for new jobs!

We often get asked by our clients how we can ensure the data from their surveys is accurate and valid. And it is a very important question. For employee surveys, however, the answer is a little bit different than for other types of surveys like politcal polls or consumer opinion surveys. In polling and consumer surveys, the data you collect is almost always going to be a small "sample" of the entire population. Sampling is a statistical method of obtaining representative data or observations from a larger group. In simple terms that means you want the small number to accurately represent the larger population from which they are chosen. There are many different ways to sample, with simple random sampling being one of the most common, so it's imortant to choose the right method for the purpose at hand. A national sample of as small as 1,000 people can be used to predict outcomes in a much larger population. You'll commonly see this on political polls. Will Donald Trump win the GOP nomination? The bigger the sample, the higher the confidence the poll will be accurate, and the smaller the sample, the higher the "margin of error". 

Survey Technology Today

Technology has made it easier and easier to collect survey data. Mostly gone are the days of telephone surveys or pen and paper surveys that required transcription and copying by rooms full of data entry clerks before any reports could be run. And even the once highly used scannable forms are pretty much a relic of the past. Nowadays it's PCs, laptops, tablets, phones and apps. Available cheaply everywhere, easy to use, always connected to the internet, they make data collection a snap. The technology is undeniably great and has revolutionized many aspects of research. That doesn't mean, though, that everything about the revolution is good. There is both good and bad, and the bad can be an ugly problem for researchers. Why? Because, along with the easy-to-use survey tools have come a plethora of apps, programs and browser extensions that will let respondents fill your survey with random data in a few minutes, with a few mouse clicks, and without ever reading a single word in your survey!

A quick search on YouTube will turn up hundreds of video tutorials on how to find, install and use these tools. Want to make money filling out surveys? No problem! Sign up here and click, click, click. Easy money in your wallet. It's so ridiculously easy to cheat on surveys using robot survey fillers that it's become a serious and widespread problem in the industry. But no-one is talking about it. Survey programmers and data analysts now spend almost as much time trying to thwart the cheaters, and then cleaning and validating survey data as they used to do managing data entry in the old days. Data collection might be cheaper today, but the costs of cleaning and validating it are significantly higher. If you care, that is.

Fortunately for us, when you are looking at employee survey data, things are quite different. In a typical employee survey you're not surveying a "sample" of your employee population, you are taking a census poll. In other words, you're surveying everyone in the target population, and assuming you get 80-90% participation which is common, that means there is really no margin of error in the results. Also, since employees are extremely invested in the outcome of the employee survey it's highly, highly unlikely that they will rush through the survey or attempt to cheat. You can be confident, then, that the data reflects exactly what the population has told you. In 15 years of analyzing employee survey data, we've never come across suspicious or spurious responses and we check the data in every survey we do.

The Dark Side of Easy Data

Cheating, though, has become a significant problem in some of the other surveys we conduct. Each year, Insightlink conducts an "Annual Survey of the American Workforce" which we've been doing for 15 years now. The survey is a large, balanced sample of U.S. workers across many industries that is the basis of our employee survey benchmark database. In order to collect the data for this survey we use several outside companies who manage very large panels of survey respondents. (i.e. people who have signed up to receive incentives or money to complete surveys). The companies who run the panels are reputable and highly respected in the industry. The millions of people who are on these panels are screened, sorted and categorized to be statistically representative of the U.S. population as a whole. In spite of this, the cheating problem has gotten worse every year. In our most recent panel survey we had to discard almost 60% of the survey responses we got. Ten years ago, we might have discarded 2 or 3%. See the problem? We now have to use sophisticated tools to sweep the data and flag and remove cheaters otherwise our results would be meaningless. We check how fast each survey was completed, we add trick questions to gauge attention, we analyze response patterns to find "flatliners" and once we employ all these tools to get rid of the majority of the cheaters, we go further and manually review every single survey for non-sensical or garbage responses before committing it to our database.

So on the one hand, technology has made a lot of things much easier, but it's also created a whole new set of problems that researchers have to tackle. Our company founder Robert Gray expressed his frustration recently in an article on MarketResearch.com. It's become a major headache for professional researchers, but the bigger problem is that so many surveys these days are done using DIY survey tools like SurveyMonkey and Qualtrics by people with little or no research experience and with absolutely no idea that the data they are collecting is mostly random garbage generated by cheaters using automated survey bots. It's the dirty little secret of the online survey industry that so far, few are willing to admit. They are making boatloads of money while you are potentially making critical business decisions based on worthless data.

We take this issue very seriously. Solid, well-designed surveys are critical to the success of any research project, be it an employee survey, a customer survey or something else, but the source and quality of the data is absolutely critical too. As they say: Garbage in, garbage out!

If you want to learn more about this, please contact us directly at 866-802-8095 ext. 705 or info@insightlink.com