Often, people try to create virtual clones of existing paper survey instruments
with eSurveysPro. While there are huge advantages in starting off with an
existing instrument, it is very important to understand the differences between
these methods. Read on for valuable information on usability, response rates,
question structure and instrument bias.
Usability
One of the most obvious differences between paper and web surveys is how the
survey is presented to the audience. Most printed surveys are designed to work
on one or more pages that are printed at an incredibly high resolution compared
with what we see on our computer screens. Not only can you fit more information
on one piece of paper than on a comparably sized monitor, the audience always
sees the paper version in the same size. With web surveys, though, people use
widely different screen resolutions, font sizes and monitor sizes. Some people
run their web browsers full screen, others in a relatively small window. This
means that your survey will look different to different readers.
In general paper surveys are designed to take advantage of all the real estate
on the paper. A question can run the entire width of the page without any
problems. Web surveys have different requirements, though. While most users are
relatively comfortable scrolling a screen up and down, they are usually
resistant to scrolling a screen right and left. Matrix questions are the
easiest to make too wide for a good survey. In general you want your matrix
questions to have response option columns that are very thin. Numbers in a
rating scale work very well, as well as keeping the number of response options
very low.
The best way to test your survey for visibility is to change your monitor
resolution temporarily to a very low setting (640x480 or 800x600). This will
give you an idea of what many of your users will see.
Another issue is that of aesthetics. Most paper surveys are on plain white paper
and use a very easy to read font. While it is easy to create a web survey with
a deep purple background, bright yellow text and flashing graphics, it is not a
good idea if you are interested in collecting valid survey data. A survey with
a white background and a Verdana, Arial or Helvetica font works best. Subtle
background graphics can add a professional tone to your survey. Graphics are
very easy to insert into eSurveysPro surveys - just make sure that your
graphics have a specific purpose. Many users implement graphics effectively by
adding logos, inserting graphics to match their web site or separating sections
of their surveys using an image. Unless it serves a purpose, though, avoid
placing graphics into your survey just because you can.
Speed and Response Rates
Very often first time users are shocked by how quickly the results start pouring
in. Depending on the audience, the purpose of the survey and the notification
method used, responses can start to accumulate in minutes - complete surveys
can be conducted in a few days. On the other hand, paper-based surveys take
from several weeks to several months to distribute, collect and encode for
analysis.
This is an important difference to recognize because it can change your survey
objectives. Many people who have conducted paper-based surveys previously ask
as many questions as they think their users will tolerate. Because web-based
surveys are more economical and provide data so much faster, a better approach
is to conduct much briefer web surveys more often. Not only does this approach
keep from burdening your survey audience, the data you collect with subsequent
surveys is more timely.
Structuring Questions
Web surveys provide better the control surfaces than do paper surveys. While
radio buttons (select one) and check boxes (check all that apply) are common to
both, web surveys can also take advantage of pull down lists. This compact form
of option selection allows you to present a very large number of responses to
the user on only one line of the screen. Very often a survey is laid out based
on the medium it will use; simply taking the question styles from a paper
survey doesn't leverage the advantages that web surveys offer.
Asking people to provide text based subjective feedback is another area where
web surveys have a distinct advantage. Getting people to write in text is
difficult because most people do not want to take the time. In addition,
collection of that type of data is highly error-prone.
In web surveys it is very easy to add multiple verbose text areas that the user
can fill in. Data entry accuracy is very high, subject to misspellings and
mistakes that the user may make. One huge advantage is when you analyze your
survey results. Using the proper filtering ("Only show me the people that said
they would purchase again"), you can read through a group of comments from
people that answered a question a specific way, providing a targeted view of
subjective feedback.
Understanding Instrument Bias
All surveys introduce bias. Telephone surveys can be biased by the tone and
demeanor of the person asking the questions - they also limit your audience to
people that have phones. Paper surveys introduce different types of bias - at a
minimum the audience must be able to read and write, making surveys about
literacy levels rather useless. Web surveys require respondents that are
literate, have at least basic computer skills and access to the Internet. This
means that you need to qualify any results you get in your web survey with that
information.
Certain surveys are not affected as much by the instrument bias a web survey
introduces. If you are looking to determine the job satisfaction of high
technology workers, the bias has a very minimal impact. Getting feedback from
employees on a benefit package can have a slightly higher bias if not all
employees have computer access. However, trying to determine whether or not the
United States should invade Afghanistan would be highly biased because the only
opinions you could register would be computer literate people with Internet
access that you are able to reach. Any results from that type of survey would
need to be qualified.
The good news is that the number of people with Internet access in the United
States and most modern countries is quickly approaching a representative cross
sampling of those societies. Based on a recent NetRatings study, 162.5 million Americans have Internet access at home,
while 35 million Americans have access at work.