Tải bản đầy đủ (.pdf) (33 trang)

Hackers Guide to Advanced Research Methodologies

Bạn đang xem bản rút gọn của tài liệu. Xem và tải ngay bản đầy đủ của tài liệu tại đây (1.45 MB, 33 trang )

 

 

 
Hacker's Guide  
to Advanced Research 
Methodologies 
 

Derive the research insights that matter with the most powerful suite 
of research tools to help you make better decisions! 
 

 

 

Author:​ Dan Fleetwood - President, Research & Insights Platform, QuestionPro​
 


 
Hacker’s Guide to Advanced Research Methodologies 

Table of Contents 


Introduction 




Advanced research methodologies & techniques - survey platform 












Advanced research question types 


Conjoint analysis 



MaxDiff analysis 



Anchored MaxDiff scaling 



Card sorting 




Text highlighter 



Van Westendorp 



Semantic differential scale 



Heatmap analysis 



Hotspot testing 

Advanced research logic 


Advanced question & answer randomization 



Block randomization 




Distributed logic quotas 

Advanced research report management 


Weighting & balancing 



Data quality 

Advanced research analysis 


Sentiment analysis 



TURF analysis 



Gap analysis 



Correlation analysis 


Additional advanced research software tools 


Auto translate 



Integrations 

Clients we have served 


 
Hacker’s Guide to Advanced Research Methodologies 

Introduction 
With no ability to foresee the COVID-19 pandemic and its effects on all of us in both our personal and 
work life before the new year, organizations have had to be agile to navigate, 2020. Despite the halt, 
perhaps even a ‘reshuffling of the deck’ of what we call our day-to-day life, businesses’ rules have not 
changed: stay up to speed on major trends and avoid the risk of being left behind. That rule still 
strongly applies to market research, as we need to meet other organizations’ needs to understand and 
understand how to maximize growth for our customers and partners. 
 
Let’s look back on some of the new market research trends that have emerged. *Note that while many of 
these trends have come as a direct response to COVID-19, many trends, before the new year, were 
predicted to thrive, both in 2020 and beyond. These trends are despite COVID-19’s direct impact on the 
marketplace and based on certain market research patterns from the 2010s. 
 

Shift of market research towards new methods 

There has been a marked shift in market research to include newer research techniques to unlock 
insights. Some of the major shifts in market research in 2020 and beyond, are: 
 
1)

Longitudinal research​ - Researchers and insights teams now need to realize that there is now a 
lot of data collected on things that have now gone ‘tone-deaf’ as a result of economic certainty 
and conservative consumerism. While longitudinal (or tracking) studies have already played 
such a significant role in capturing insights on awareness and purchase consideration, many 
factors in 2020 could affect the overall demand for particular products. 
 
Regardless of how well and business or industry is thriving during these times, researchers will 
need to continue monitoring and tracking data to see and understand what is changing over 
time (and what is NOT) for businesses to better prepare for the future. 
 

2) Measuring perception and sentiment​ - Certain brands and products will thrive due to these 
times, while others won’t. This is a result of the sweeping changes in attitudes and behaviors 
among marketplace consumers. Something vital to consider is that many of these changes won’t 
be temporary; purchase behaviors in the masses could change for good. Since customers will be 
exposed to new ways of purchasing products, many of these consumers have the potential to 
find out that these new ways may be better suited for their day-to-day lives. 
 


 
Hacker’s Guide to Advanced Research Methodologies 

It will be essential for researchers to consider all of the factors that affect purchasing processes 
and brand perception and awareness to infer the next steps and directions for clients to thrive 

during these times. 
 
3) Surge in online qualitative research & video focus groups​ - Over the last decade, online 
qualitative research has gone from a novelty or a ‘nice-to-have’ into an essential research 
methodology across the market research industry. In general, traditional qualitative research is 
declining because it is so time-consuming and expensive to designate physical spaces to host 
these qualitative research activities.  
 
With added features such as being able to share video responses and recording participants’ 
interactions on the platform, market researchers can now reach out to their audiences in a 
shorter period and probe the ‘why’ about a brand or a product now faster than ever. Plus, live 
explanations of thoughts and feelings are more natural, leading to even more feeling, sharing, 
and connecting with your audience. 
 
4) Shorter but smarter surveys -​ Short surveys and quick polls, more than ever, are popping up on 
websites, phones, social media, and chatbots. One reason for this is the growing fatigue for 
respondents to answer longer surveys and a continued increase in utilization in social media or 
communication apps on electronic devices, particularly mobile phones. Apps for social media 
and communication account for at least 50% of all apps used worldwide. With that, researchers 
can intercept and collect more data in real-time at a higher rate.  
 
5) Incorporation of Artificial Intelligence in data collection​ - Researchers are pushing for new 
data collection methods to be more seamless and automated processes. As mentioned, shorter 
surveys are a direct result of this. But more than that, the demand for data delivery to go 
straight from laptops to high-level decision-makers is rising. 
 
Areas for data collection to become more seamless include analysis at open-ended text 
responses. These capabilities will enhance time-efficiency for researchers and create more 
value for clients and break down the ‘why’ when purchasing behaviors. Another example of this 
is in Sentiment Analysis – where marketers and researchers will be able to decipher positive or 

negative responses better. 
 
QuestionPro is at the forefront of facilitating research and has consistently stayed above the curve to 
bring simpler but powerful research to brands, organizations and researchers alike. 


 
Hacker’s Guide to Advanced Research Methodologies 

Advanced research methodologies & techniques - 
survey platform  
Using our research platform will provide you unmatched flexibility in how you collect actionable insights 
for your brand. Leverage our DIY research tools, smart survey logic, and more, that solve your biggest 
research challenges and align with your brand and research goals.  
 

 
 
Our research platform solution matrix will help you with: 
 


Choice-based analysis​ - Simulate market conditions and track and monitor preferences by 
using conjoint analysis, max diff analysis, card sorting, etc. 



Market trends​ - Monitor market dynamics and stay insulated from shocks which will allow you 
to constantly be above the curve. 



 
Hacker’s Guide to Advanced Research Methodologies 



Market segmentation​ - Monitor behavior across various geographies and demographics. Make 
informed decisions on positioning based on value. 



Competitive benchmarking​ - Identify gaps in relation to your biggest competitors and clamp 
down on your biggest differentiators. 



Purchase behavior​ - Identify purchasing behavior including various aspects such as price 
sensitivity and trade-offs to assign optimum pricing models. 



Product & service research ​- Listen and act on what your customers and audience want, to stay 
above the competition and reduce churn. 



Academic research -​ Capture academic insights as an academician or a student with unlimited 
flexibility that no other tool provides. 




Ad testing -​ Monitor how your brand’s messaging resonates with your audience and impacts 
your referencability. 



A/B tests​ - Test everything across multiple vectors to make the most informed decisions that 
are representative of the larger population. 

 
Advanced research question types 
 
The QuestionPro Research suite provides you the ability to use advanced research question types to be 
able to solve any research problems that you may have. 
 

Conjoint analysis 
Conjoint analysis is a choice modeling research method to understand how people make purchasing 
decisions. In the real world, we often encounter situations when we have to make tough choices 
between various alternatives. The conjoint analysis question helps us understand what is essential for 
your target audience. It involves how they make trade-offs and what essential features they are not 
willing to let go. 
 
The conjoint survey question is an advanced question type that market researchers use to present many 
combinations of product attributes like features, cost, brand, etc. Based on the respondents' answers, 
market researchers can find out the most liked features by customers and get an idea of pricing. Many 
times a purchase involves evaluating several parameters that make it complicated. In such a situation, 
running a conjoint analysis survey can help understand customer psychology. 
 



 
Hacker’s Guide to Advanced Research Methodologies 

Types of conjoint analysis commonly used in surveys 
Choice-based conjoint analysis:​ This type of analysis question asks respondents to imitate their 
purchasing behavior while answering the survey. The respondents submit responses based on the 
actual products they would choose in real-life, given specific prices and features. 

Types of designs for the discrete choice model 
QuestionPro offers the below design types for conjoint analysis using the discrete choice model: 
 


Random:​ This design is a random sample of the possible attribute levels. The survey software 
will create a unique combination of attributes for the number of tasks per respondent. To know 
what choices will be presented when your survey is deployed, you can run a conjoint concept 
simulator. 



D-Optimal:​ While designing experiments for estimating statistical models, optimal designs 
estimate parameters without bias, and with minimum-variance. D-optimal design runs a set of 
tests to optimize or investigate the subject under study. The algorithm creates an optimal 
design for the tasks per respondent and sample size. 



Import design: T​ his design type allows designs in the SPSS format to be imported and used by 
the discrete choice module. For instance, you can import fractional factorial orthogonal designs 

and use them in QuestionPro surveys. 

 
Adaptive conjoint analysis:​ This type of conjoint analysis is used in surveys when there are many 
product features. Researchers generally use it to identify key features that should be included in the 
product and not the best choice for determining the price. For instance, the surveyor asks respondents 
to select their relative preference from several attributes. They assess each pair on a grade point scale. 
 
The choice-based conjoint analysis, also known as discrete-choice conjoint analysis, is the most 
commonly used type of conjoint analysis survey question. 

Data analysis of conjoint survey question 
With QuestionPro surveys, you can generate a conjoint analysis report and filter the survey data. The 
statistical analysis report consists of the below tabs. 
 
1)

Attribute importance​ - This tab shows which attributes are more important than others and by 
what percentage.  


 
Hacker’s Guide to Advanced Research Methodologies 

 
Learn more about a
​ ttribute importance​.  
 

 

 
2) Profiles​ - It is a set of attributes with different levels. The conjoint analysis software shows 
respondents various combinations of product features, prototypes, mockups, or pictures 
created from a combination of levels. Each example is similar enough to be close substitutes 
but different enough to be distinguishable.  
 
Learn more about c​ onjoint analysis profiles​.  
 
3) Market simulation ​- Using this feature, you can forecast the market share of new products that 
don't exist today. You can also measure the gain or loss in the market share based on the 
existing products' changes. The conjoint analytics tool simulates the market share of the 
products to establish a baseline. Then, you can see how the market share changes depending 
on new products and configurations.  
 
Learn more about m
​ arket segmentation simulators​.  
 


 
Hacker’s Guide to Advanced Research Methodologies 

4) Estimated brand premium​ - In many cases, customers are willing to pay extra for a product with 
the same features as others but with a different brand. This report finds out how much 
premium a customer will pay for a brand. 
 
Learn more about e
​ stimated brand premium​.  
 


 
 
5) Price elasticity​ - It is the proportional change in demand for a product for change in attributes 
and price. To view this report, map attribute type to brand & price for each level. 
 
Learn more about f​ eature attribute type​.  
 
Additional asset: T​ he Hacker's Guide to Conjoint Analysis 
 

MaxDiff analysis
Maximum Difference Scaling is a very effective method of establishing the relative priority attached by 
an audience to a large set of items (up to 30). These items might be: 


 
Hacker’s Guide to Advanced Research Methodologies 

1)

Determine the features or benefits of a service​ - Before investing time and money into a new 
feature or rolling out a new service, ensure that the target market is likely to buy and use the 
new offerings. 

2) Discern areas for potential investment of resources​ - Take stock of the market trends without 
spending excess money and time on an expensive market research expert—just ask the target 
consumers! 
3) Learn about consumer interests and activity preferences​ - There’s no money in attempting to 
manufacture genuine interest; ensure the business offers products, services, and activities 
based on consumer attraction. 

4) Test potential marketing messages for a new product​ - The organization’s messaging concepts 
should resonate with the entire market for maximum impact—test it out with a statistically 
significant sample size to see if the brand voice resonates with the target audience. 
5) Know which products or services are used​ - Reinvest in the products and services that the 
organization’s customer base is already investing in for maximum ROI. 
 

Key terms and concepts 
MaxDiff scaling is a powerful choice-based modeling method. To be able to harness the full potential of 
this analysis method, it is important to know what these key terms and concepts, mean: 
1)

MaxDiff Analysis​ - MaxDiff Analysis is shorthand for Maximum Differential Analysis, also known 
as Best-Worst scaling, wherein respondents choose the best value and worst value from a set of 
attributes (and the attributes repeat in intentional permutations based on the logit model).  
I.e. a large Bank determining which major credit card company to partner with for their 
customer credit card offering. 

2) Attributes​ - The options that need to be chosen from (features of a product, answer options).  
I.e. Preference of use (most preferred to use / least preferred to use). 
3) Maximum attributes​ - The number of attributes that will be tested in the survey. 
4) Task​ - The current screen the respondent sees (you may have up to tasks per survey). 
5) Attributes per task​ - the number of attributes that will be shown on the screen at a time. 
6) Logit model​ - Utility estimation model used to calculate the share of preference (See more in 
How to set it up in QuestionPro​). 
 
You can view the MaxDiff analysis (best-worst scaling) in the following format: 
 



 
Hacker’s Guide to Advanced Research Methodologies 

 
 
Learn more about M
​ axDiff analysis in your surveys​.  
 
Additional asset: T​ he Hacker's Guide to MaxDiff 
 

Anchored MaxDiff scaling 
Anchored MaxDiff is a simple yet effective model to enhance a MaxDiff exercise from a relative model 
(where one option's utility is relative to the others) - to an absolute model. Anchored MaxDiff 
experiments supplement standard Max-Diff questions with additional questions designed to work out 
the absolute importance of the attributes. 
 
Traditional MaxDiff experiment identifies the relative importance of the attributes, an anchored MaxDiff 
experiment permits conclusions about whether specific attributes are actually important or not 
important at all. 
 
It may be that feature A is twice as important as feature B, which we would find out from MaxDiff. It may 
also be that neither A nor B is very important at all. We would not find this out from traditional MaxDiff. 
With anchoring - this allows researchers to add in an additional "Anchoring" question that determines if 
any of the features are fundamentally important. 
 


 
Hacker’s Guide to Advanced Research Methodologies 


Top reasons to use anchored MaxDiff scaling 
You can interpret a piece of data in ten different ways unless you have a rating scale. If I say there is a 
12% growth in the travel industry, that can mean 12% industry growth as well as 12% quarterly growth! 
But if I say the travel industry grew 12% this year as compared to the last year, you have a reference 
point to compare with. 
 
Making decisions off of insights about your respondent’s best/worst preferences can give you some 
insights, but adding an anchored question gives you a clear direction for the other choices as well. 
Instead of leaving the rest of the options in the dark, anchored MaxDiff question lets you get more out 
of the survey. 
 
Some of the questions that an Anchored Maxdiff question can answer are: 
 
1)

How do customers perceive product features or attributes? 

2) What do customers consider while making purchase decisions? 
3) How much does a product actually meet customers’ needs? 
4) How does the audience perceive different products or services? 
5) What products do customers view as ‘absolute essentials’? 
6) What features are customers willing to trade-off for the ‘absolute essentials’ features? 
7)

Does keeping a constant connection with the customers have an impact on their buying 
decisions? 

8) How does a business fare as compared to its close competitors? 
9) Is there a scope for a business to be “Most Preferred” by customers? 

10) Are there any early warning signs for a company? 
 
A single data point doesn’t make sense unless it has some reference or a benchmark. While a simple 
MaxDiff survey is apt for those who want to have a high-level understanding of audiences’ preferences, 
you can dig deeper with an Anchored MaxDiff survey. The latter lets you discriminate between various 
options and make effective decisions. 
 
Learn more about A
​ nchored MaxDiff analysis​ and how to set it up in your QuestionPro survey.  
 

Card sorting 
Card sorting in a survey is a very highly interactive question type that respondents use to sort answer 
options into specific buckets which helps to conduct user research. Answers rank from the least to the 


 
Hacker’s Guide to Advanced Research Methodologies 

most crucial answer options. These questions are highly engaging and require a more in-depth level 
interaction with the survey, which means they are preferred to question types that use radio buttons, 
single select, etc. 
 
This question type allows a high amount of control to the surveyor as well as the survey taker because 
the grouped answer options are either open-ended or close-ended. Due to different answer nesting 
types, there are various ways in the collection and interpretation of data. 

Types of card sorting in your surveys 
There are two types of card sorting in surveys. Both of them offer different kinds of insights for various 
reasons. They are: 

 
1)

Open card sorting​ - In this type of question and answer type, the respondents have the 
freedom to bucket answer options into any answer options that they would like. There's scope 
for collecting open-ended feedback or bucket options that the brand hadn’t even thought of at 
the outset. 
 
Learn more about o
​ pen card sorting​ and its analysis.  
 

 
 


 
Hacker’s Guide to Advanced Research Methodologies 

2) Closed card sorting​ - In this type, you define your categories for respondents to place answers. 
This method offers greater control over the data. It also helps to derive direct insights from the 
answers in a planned direction. 
 
Learn more about c​ losed card sorting​ and its analysis.  
 

 
 
Additional resource: W
​ ebinar on best practices of using card sorting in surveys 

 

Text highlighter 
A text highlighter question is used in surveys to get feedback on the text. The respondents can select 
the text and share their comments on the selection of words, grammar, semantic structure, context, and 
more. 
 
While document editors like Google Docs, Microsoft Word, OpenOffice Writer, and others allow you to 
comment on specific text in a document, they can't help when you need feedback from a large number 
of survey respondents. In such situations, a text highlighter question can help to collect opinions and 
gain insights into your audiences' choices. 

Text highlighter in surveys and analysis 
In QuestionPro surveys, text highlighter is an advanced question type with customizable multiple-choice 
options. By default, there are two options - 'Good' and 'Bad'. When the respondent selects the text, they 


 
Hacker’s Guide to Advanced Research Methodologies 

can also view an open-ended textbox along with these options. Respondents can choose one of them 
and submit their responses with or without entering the comments. They can highlight multiple words 
or sentences and offer their feedback. To improve the readability, survey creators can add line breaks 
within the text. 
 
You can view the distribution of responses of the text highlighter question on the report dashboard. To 
know which section of the text was selected for an answer option, click on the circles next to the 
options. 
 


 
 
Learn how to use the ​text highlighter survey question​ in your research.  
 

Van Westendorp 
The Van Westendorp pricing method asks a series of questions to respondents to identify critical 
psychological price points. It helps understand consumers' purchase power and how much they're 
willing to pay. Van Westerdorp's pricing question is an advanced question type used in surveys to create 
a better pricing strategy. 
 
Van Westendorp survey highlights the range of prices the customers are comfortable with and any 
change in their interest if the price falls out of it. It can be specifically useful if the marketer is planning 
a pricing change, or wants to learn more about consumers' perceptions of their products vs. 
competitors' products. 
 


 
Hacker’s Guide to Advanced Research Methodologies 

Analyzing responses of Van Westendorp question 
The Van Westendorp pricing model asks four questions and then throws an intersection point based 
reporting to derive the right pricing.  
 

 
 
The intersection of specific data points gives an idea about how respondents view price-value through 
the Optimal Price Point (OPP) and the Indifference Price Point (IDP). At OPP, an equal number of 

respondents consider the price as 'too expensive' or 'too cheap'. At IDP, an equal number of participants 
believe the price is either 'cheap' or 'expensive'. 
 
You can find the Point of marginal cheapness (PMC) - the price point where you will lose more sales due 
to poor quality than would be won from bargain hunters. The Point of Marginal Expensiveness (PME) is 
the price point above, which price is a significant concern. Here the respondent feels that the product is 
too expensive. 
 
Between PMC and PME is the Range of Acceptable Prices, which can be used by market researchers to 
decide the optimal price. This range of price also implies the price expected by consumers. 
 
Learn more about the ​Van Westendorp pricing sensitivity question​ and how to use it in your research. 


 
Hacker’s Guide to Advanced Research Methodologies 

Semantic differential scale 
The semantic differential scale in surveys asks the respondents to rate the product, service, events, 
brand, or company within the range of multi-point rating system. The answer option consists of 
opposite adjectives at each end. For example, pleasant/unpleasant, love/hate, happy/sad, etc. This 
feature is the most reliable way to measure people’s emotional attitudes towards a product or service. 

Types of semantic differential scales 
Depending on the need and response collection method, different types of semantic differential scales 
are used in surveys. The most common types are: 
 
1)

Slider rating scale​ - The slider rating scale question includes the text slider with textual answer 

options. Based on this, the respondents will leave the indicator at the point where the answer is 
to be selected. 

2) Non-slider rating scale​ - The non-slider rating scale question includes radio buttons, a more 
traditional survey look, and feels with a scale of 1 to 5. 
3) Smiley-rating scale​ - The question type with a 5 point rating scale, represents the sentiments 
from negative to positive. It communicates the emotions that are easy and engaging for the 
respondents. The intent of adding this question is to keep engaging the respondents and to get 
a better completion rate. 
4) Rank order question​ - The rank order question allows the respondents to order the product or 
brands as per their interest. In the below example, a person who loves to swim will rank the 
answer option as 5 and the person who hates skiing will rate the answer option as “1”. 
 

Heatmap analysis 
A heatmap is a visual method of representing which areas get the most attention in a color-coded chart. 
 
Heat maps give an instant idea of an area by grouping places into sections and displaying their density. 
The darker the color is, the higher is the density. 
 
The different types of heatmaps are: 


Scroll maps​ - These maps highlight which part of your page or image the visitors view most, 
how much area is visible without scroll and how far people scroll before leaving the page. 



Attention maps ​- Depending on the time spent by visitors, these maps show which part of the 
image or page gets the most attention. 




Click maps​ - These maps highlight the areas that get the most clicks. 


 
Hacker’s Guide to Advanced Research Methodologies 



Hover maps​ - These maps show which part of the page did people hover their cursors most on. 

Heatmap analysis methods 
Heatmaps can be used in primary research to get an overview of high-level information about user 
behavior or the choices of the respondents. You can perform heatmap research in surveys by uploading 
images as the answer options and asking respondents to click their preferred areas. 
 
1)

On-click heatmaps​ - The answer will be logged when the respondents click on the image. 
 

 
 
2) Drag and drop heatmaps -​ The answer will be logged when the respondents select an area by 
dragging and dropping the mouse cursor. The heatmap analytics tool uses color-coding to 
represent the area that was dragged most often by the respondents in dark red. The lesser 
number of times a section is pulled, the fainter red color it would be highlighted in. The dragged 
areas that do not overlap with the responses of other participants are highlighted in blue. 

 


 
Hacker’s Guide to Advanced Research Methodologies 

 
 
Learn more about how to use ​heatmap analysis​ in your surveys.  
 
Additional resource: W
​ ebinar on how to conduct heatmap testing 
 

Hotspot testing 
A hotspot question is used in the surveys to get feedback on images. It consists of an image, and 
respondents are asked whether they like or dislike a specific area of an image. There are no specific 
correct or incorrect answers. It is used just to know what part of the image is most popular among the 
respondents. 
 
This question in a survey shows one or more selected regions to respondents. On clicking that area, 
respondents will be able to share their views both graphically and textually. A pop-up box appears over 
the questionnaire with a 'thumbs up' and 'thumbs down' button for 'like' and 'dislike.' 
 
Respondents can also leave their comments in the text box. It will give researchers more insights about 
what they think of the image. Hotspot question type offers options to either force respondents to 
answer the question or request a response. You can also customize the border color of the area 
selected, which needs participants' feedback. 
 



 
Hacker’s Guide to Advanced Research Methodologies 

 
 
Learn how to use ​hotspot testing​ in your research.  
 
Additional resource: W
​ ebinar on how to conduct hotspot testing 
 

Advanced research logic 
QuestionPro's logic engine is the most advanced in the industry. We can do piping, extraction, 
branching, looping and logic based on criteria of questions and metadata like custom variables passed 
in from other sources. All this using a point-and-click interface - without scripting or coding. The 
research platform also allows you the ability to manage conditional block rotation & randomization, 
conditional looping, and conditional extraction. 
 
The logic in our survey research platform that makes us the choice for researchers all over the world, 
are: 
 

Advanced question & answer randomization 
A randomizer question type allows you to present a set of questions to your respondents randomly. 
Survey creators can use it to eliminate possible order bias from your respondent group. 
 


 

Hacker’s Guide to Advanced Research Methodologies 

Advanced randomization in surveys is a technique by which the answer options are presented to each 
respondent in a different order. It is used to overcome order bias - a behavior in which respondents 
tend to select the first option. To tackle this, survey creators present answer options such that they 
need to spend some time going through all the options and choose an honest answer. 
 
The platform allows you the flexibility to randomize questions and answers in multiple methods so that 
the survey data collection is as accurate as possible.  

Question randomization 
A randomizer question lets you display survey questions randomly to the respondents. You can 
configure question randomization at two levels and alter question order: 


Randomly display one question 



Randomize the sequence of a list of questions 

 
Learn how to use ​question randomization​ in your surveys. 

Answer randomization 
You can randomize the answer options in your surveys by the following methods:  


Simple ​- Randomize the order of all options. It is classified into three categories. 



Default​ - Answer items are displayed in the same sequence as set by the survey creator. 



Ascending​ - Answer options are sorted in the ascending order alphabetically. 



Descending ​- Answer options are sorted in the descending order alphabetically. 



Random​ - Randomize the order of few options out of all the options. 



Advanced randomization​ - Create subgroups of answer choices and display the groups 
randomly. 

 
Learn how to use ​answer randomization​ in your surveys. 
 

Block randomization 
The quality of data depends on various factors. Some of them are the type of questions asked, the 
depth and breadth, number, topics covered, research method, etc. Another factor that affects data 
quality is the order of questions. The sequence of items can impact the way a respondent thinks and 
makes choices. Hence, there is a high possibility that the respondents may feel inclined to answer in a 
particular direction. Thus, the survey creator unknowingly affects the way a respondent answers the 



 
Hacker’s Guide to Advanced Research Methodologies 

questions. This effect is known as order bias and affects the results of the research. Reports generated 
from the data inflicted with order bias generate inaccurate insights. 
 
A block randomizer lets you select a group of questions that must be asked to respondents in random 
order. You can keep one or more blocks fixed and randomize the order of others. Alternatively, you can 
also display all blocks of questions randomly. You can also randomize questions within a survey block. 
 

 
Learn more about how to set up b
​ lock randomization​ in your surveys.  
 

Distributed logic quotas 
The distributed logic quota is a combination of different types of quotas. Quota control helps you limit 
the number of responses to your survey. If the survey uses more than one quota control technique, it is 
considered as distributed quota control. Using this method, you can set up quota based on answers to 
multiple questions, custom variables, number of responses for each option, geo-location, and more. 
 
There are two approaches to quota control - pessimistic and optimistic. 
 
Pessimistic quota control does not count an individual as passing quota until they have completed a 
survey. So we can be sure that all quota cells are filled. If a user sends out many survey invitations, 
there is a strong chance that there will be an over quota situation due to too many completions. 



 
Hacker’s Guide to Advanced Research Methodologies 

QuestionPro implements the pessimistic quota system. You can apply a distributed logic quota on 
multiple questions. 
 
Optimistic quota control counts anyone who starts the survey but does not finish it. It has a lesser 
chance of being over quota, but a higher risk of being under quota. To avoid going over quota, the 
survey creator should manage the timing of survey distribution such that all respondents do not receive 
the invitation at the same time. It would also help researchers have better control over the responses. 
 

 

Types of advanced quota control in surveys 
Types of advanced quota control in surveys are: 
 
1)

Response quota​ - This type of quota control is used to limit the number of responses to your 
survey. Once the limit for the total number of responses is achieved, respondents will not be 
able to submit the survey. 

2) Complex quota control​ - This type of quota puts a limit on the number of responses to multiple 
questions and custom variables. It is also used as weighted quota control. You can set the 
percentage distribution of responses across the options of a specific question. 


 

Hacker’s Guide to Advanced Research Methodologies 

3) Custom variable quota control​ - You can set up quota based on the responses to multiple 
custom variables. When the custom variable is set with a specific value for a few times, 
respondents will not be able to answer the survey further. 
4) Advanced quota control​ - Use this method to limit the number of responses based on the 
selection of options, custom variable, geo-location, email list, and device type. 
5) Dynamic quota control -​ Using this method, you can program the survey to take one of the 
below actions once the quota limit is reached. 
 
Learn how to a
​ dvanced quota control​ in your surveys.  
 

Advanced research report management 
Some other features that we have developed keeping researchers and research oriented surveys in 
mind, are: 
 

Weighting & balancing 
Weighting and balancing is a survey feature that allows you to eliminate sample bias in your online 
surveys. You can adjust the captured data to represent the population accurately. This question helps 
researchers eliminate bias that occurs when the data derived from the survey does not represent the 
target population accurately to make sound decisions.  
 
The primary motive of weighting and balancing is to yield accurate data-backed decisions. This is 
achieved by eliminating data that does not add value to representing the population accurately. You 
can use weighting and balancing to eliminate demographic biases for the following and more: 
 



Age bias 



Gender bias 



Location bias 



Educational level 



Marital status 



And more. 

 
Learn how to set up ​weighting and balancing​ in your research.  
 


 
Hacker’s Guide to Advanced Research Methodologies 


Data quality 
Data quality is an important tool that enables researchers the ability to maintain a tab on the quality of 
survey and research data. This allows only high-quality data to be used for analysis and insights 
management. Using inaccurate data can skew up the results for researchers causing time and cost 
overruns. The data quality tool, when enabled before a study is deployed ensures that there is the 
ability to catch discrepancies in data at an early stage and this data doesn’t get considered for analysis.  
 
This also ensures that only high quality survey data is exported to external analysis tools or can be 
analyzed as is. The data quality tool works on qualitative as well as quantitative survey data.  
 

 
 
Learn how to set up ​data quality​ and analyze the most accurate data.  
 

Advanced research analysis 
Analysis of survey data is important because that drives you from data to insights. By leveraging 
advanced research analysis methods, you can derive insights that matter the most to your brand. In the 
QuestionPro Research suite, you can use basic survey analysis as well as advanced survey analysis to 


×