White Paper: Replacing Online Non-Probability Panels with Short Message Service (SMS)

White Paper: Replacing Online Non-Probability Panels with Short Message Service (SMS)

Introduction

On April 20, 2020 Precision Opinion released two sample databases to begin the comparison of Online Non-Probability Panels versus SMS Text-to-Web invites to mobile phones. Three markets were targeted. The first was a national poll of adults eighteen plus. The second a statewide poll of adults eighteen plus for the state of Ohio. The third was a poll of adults eighteen plus in Jefferson County, in the 6th Congressional District of the State of Ohio.

The survey was global in design with each market receiving identical invites as well as matching surveys. A copy of the survey may be found in Appendix A. Tabulations of the survey may be found in Appendix B.

Targeted Sample Size By Market

Background

Historically there has been an ongoing debate regarding the use of Non-Probability Based Samples provided by Panel Houses. The core issues revolve around representation, especially when targeted populations are small (normally considered < 30,000). Each panel supplier has different modes for validating the authenticity of members as well as placing limits on the totalsurvey invites a participant is allowed to take. In some, the allowable figure is 10 per day (or more).

Another issue described by the American Association of Public Opinion Research (AAPOR), as part of a comprehensive online journalism polling course created in partnership with NewsU, a project of the Poynter Institute and funded by the Knight Foundation found that internet surveys often have very low participation rates, often in the single digits. Many are invited to complete the survey, but few choose to do so. (This is also a problem with many telephone surveys.) As a result, the sample respondents may be an unrepresentative subset of the panel members initially invited for the survey.

Another aspect is one of representativeness and the use of “volunteer samples” rather than probability samples. Online panels remain a popular tool for data collection and they have proven their value for many purposes. However, evaluations have found that volunteer online panels are less representative of the general population than probability samples.

The Issue To Be Tested

Today’s online surveys can replicate branching logic found in CATI programs, and can also provide the ability to stream media, offer voice overs and conduct heat-mapping tests (just to name few).

An online programmed survey is not the issue that SMS Text-to-Web is attempting to solve. The issue is the inability to obtain a truly representative sample for market research surveys from open registration online panel houses versus those who are not members of a panel.

Even though most people in the U.S. have access to the Internet, the ones who don’t differ systematically from those who do. People without access to the Internet tend to be older, less educated, and poorer than those with internet access.

And, if one has access to the internet along with a computer or tablet, if they are not a member of a panel, they can never be representative.

Currently those issues are handled by weighting. The question then becomes how much weighting is allowed before the results are skewed?

AAPOR as well as other well-respected organizations have conducted evaluations and concluded that volunteer online panels are less representative of the general population than probability samples.

The Objective

The objective of this White Paper is to analyze a viable option to Non- Probability Panels by implementing a TCPA compliant SMS Text-to-Web methodology.

Today, most of America has a device capable of taking online surveys in their hand. As of February 2019, 74% of Americans owned a desktop or laptop computer. In the same time period Pew Research found the vast majority of Americans (96%) own a cellphone of some kind. The share of Americans that own smartphones is 81%.

By implementation of a TCPA compliant service such as PrecisionSMS, text messages with an online survey link embedded in the message, can be sent directly to one’s desired target audience.

The Methodology

On April 20, 2020 three identical surveys were launched using two sample bases. One sample was a random pull using one of the largest Online Non-Probability Panels companies. The second was a sample pull using a random selection of mobile telephone numbers. Both samples were drawn for three markets: national, the state of Ohio, and Jefferson County in Ohio’s 9th Congressional District. The responses were allowed to fall naturally with no quotas for gender nor age required.

The panel house’s sample was sent using a router. That is when a member of the panel was found to be online, they received an invite for the study. The cell phone sample invites were sent using PrecisionSMS Peer-to-Peer TCPA Compliant technology.

The study was designed to stay in the field no longer than 3-days (April 20 through April 23, 2020)

Key Discoveries

Both the Non-Probability Panel and the SMS Text-to-Web national and state of Ohio studies returned results that delivered an accurately represented sample.

The key discovery was PrecisionSMS Text-to-Web methodology reaches an untapped market of respondents not found in Online Non-Probability Panels. Only 6% of SMS respondents to the national study were members of a panel. For the state of Ohio 8% of the SMS sample were members of a panel while for Jefferson County only 4% belonged to a panel.

Non-probability panels for small markets of (in this study 30,000 households) were ineffective at best. Using multiple Non-Probability Panels over a 3-day period 21 completes were generated. The SMS Text-to-Web produced 116 completed interviews with 91 obtained in the first 6 hours.

Conclusions

  1. The research conducted for this White Paper found the SMS methodology delivered a higher participation rate than non-probability panels, including areas of low population.
  2. SMS Text-to-Web delivers an un-tapped representative market.
  3. SMS Text-to-Web effectively reaches small markets that are underserved by panels.
  4. SMS Text-to-Web delivers an accurately representative sample in all markets tested significantly faster than online panels.
  5. The SMS Text-to-Web methodology outperformed Online Non-Probability Panels in field time. In all three markets quotas were met in the national and state studies in +/- 6 hours.
  6. In all 3 studies Online Panels did not reach n= quotas.
  7. The SMS Text-to-Web performed significantly better in the small market of Jefferson County as the panels were under representative. The Jefferson County market with a population of 65,235 representing 29,109 households proved to be problematic for Online Panels. After 53 hours of field time the Online Panel completed 21 surveys. The SMS Text-to-Web delivered 116.
  8. In the core market of 25-54 SMS Text-to-Web reflected the US Census in both the age and gender categories.

Frequency of Participation in Online Surveys

The SMS Text-to-Web identified a new participant base with a limited duplica- tion of respondents found in Online Non-Probability Panels sample.

  • The National Study identified 6% of SMS respondents were also panel members.
  • The State of Ohio Study identified 8% of SMS respondents were also panel members.
  • The Jefferson County Study identified 4% of SMS respondents were also panel members.

A review of the Online Non-Probability Panel respondent’s frequency of participating revealed the following:

  • The National Study identified 74% of panel respondents belonged to at least 2 panels with 57% reported taking 10 or more online surveys in the past 3-months. Thirty-four percent reported taking 21 or more online surveys in the past 3 months.
  • The State of Ohio Study identified 76% of panel respondents belonged to 2 or more panels with 56% reported taking 10 or more online surveys in the past 3-months. Thirty-two percent reported taking 21 or more online surveys in the past 3 months.
  • The Jefferson County Study identified 71% of the 21 panel respondents belonged to 2 or more panels with 44% reported taking 10 or more online surveys in the past 3-months. Twenty-nine percent reported taking 21 or more online surveys in the past 3 months.

PrecisionSMS Performance

Overall using the SMS to Web Peer-to-Peer methodology for the three target markets the incidence rate was 88.7%. A total of 221,033 sample records were sent resulting in 1,979 completes and a contact rate of 1.46%. Comparable data is not available for the panel sample.

SMS-To-Web Survey Performance

Survey Results

For the National study with a target n= 800 the SMS study was completed in under 6 hours with 814 completes. At the end of the 3-day fielding period a total of 1,046 surveys were completed. The panel portion of the survey did not reach the target of 800 completes. At the end of the 3-day fielding period a total of 795 surveys were completed.

For the state of Ohio study with a target n= 650 the SMS study was completed in a little over 6 hours with 680 completes. At the end of the 3-day fielding period a total of 817 surveys were completed. The panel portion of the survey fell short of the target of 650 completes. At the end of the 3-day fielding period a total of 647 surveys were completed.

For the Jefferson County (Ohio) study with a target n= 400 the SMS study was not completed in the three-day fielding resulting in 116 completes. At the end of the 3-day fielding period the panel portion of the survey reached 21 completes.

Comparison Of Age Groupings For SMS And Panel Against US Census

Comparison Of Race Groupings For SMS And Panel Against US Census

Both the SMS Text-to-Web and the Online Non-Probability Panels Sample delivered comparative percentages by race against the US Census for White/Caucasian, Black/African and the Asian demographics. Underrepresentation was found in both samples for Hispanics.

Comparison Of Household Income Versus National Income Levels

Versus the national income levels both the SMS and panel sample performed well against the average except in the income range $15,000 and under where the SMS panel was underrepresented.

Tired of not being able to reach target audiences? Looking to tap into un-touched respondents for insights? PrecisionSMS will surprise you with all that it offers. Contact us at sales@PrecisionOpinion.com