MARKETING STRATEGY

1
MARK2038 Data Base Marketing Strategies II
Week 11 Instructor Santo Ligotti Email
sligotti_at_gbrownc.on.ca
2
Testing, Metrics, and Post Analysis
3
This week

• Testing, metrics, and post analysis
• In-class assignment 5
• Structure/content of final test (July 18th, 2006)

4
Learning Objectives

• You just learned
• why testing of DBM programs is important
• 4 steps you can take to test DBM programs
• how to analyze the effectiveness of direct
  response campaigns including response rate, ROI
  and cost per response.

5
Campaign Management Process
Campaign Planning

• Planning
• List
• Budget
• Offer/call to action
• Fulfillment
• Creative format
• Messages and copy
• Response device
• Testing process
• Response tracking
• Financial success measures

List Compilation
Implementation
Measurement
6
Campaign Management Process
Campaign Planning

• List compilation
• Purchase response lists/compiled lists
• Ensure any last-minute field edits are complete
• Select list members
• Forward records to agency/suppliers
• Flag records for inclusion in CRM system

List Compilation
Implementation
Measurement
7
Campaign Management Process
Campaign Planning

• Implementation
• Campaign is activated
• Customer inquiries and orders are acted upon
• Information is received from selected media
  channels
• Measurement
• Monitor the results of the campaign for
  effectiveness
• Input recommendations to direct marketing planning

List Compilation
Implementation
Measurement
8
Time to Market

• Marketing campaigns require an average of 2.5
  months to implement.

9
Reducing Time to Market

• The longer the campaign lead time,
• The less likely the message will be relevant to
  its audience
• and the less likely it will be highly
  effective.

10
Getting the right mix, requires internal
partnerships

• A partnership between Marketing and Analytics
  will
• maximize campaign results
• Involve the data analytics team at the beginning
• of the campaign to establish key business
• objectives, pre-analysis, targeting and key
  metrics/tracking
• Continually integrate the data analytics teams
• tracking and key insights into future campaigns
  to
• maximize ROI of all marketing initiatives

11
The Business Challenge

• With increasing pressure from
  shareholders/analysts
• to continually improve financial results,
  marketers
• need to able to illustrate that their
  campaigns are
• delivering strong results
• In order to ensure marketing dollars are
  maximized,
• data analytics needs to become a key partner
  in the
• ongoing measurement tracking of campaigns
• A number of marketers are still struggling to
• demonstrate that their campaigns deliver
• quantifiable results

So how do we as marketers achieve this?
12
Data Analytics is key to CRM Process
13
Knowing Your Customer starts with Data Analytics
LISTEN
ACTION
DELIVER MESSAGE
TEST AND LEARN
KNOW THE CUSTOMER
CREATE APPROPRIATE MESSAGE
IDENTIFY POTENTIAL CUSTOMER ACTIONS
14
Utilizing Data Analytics allows you to Identify
Potential Customer Actions
LISTEN
ACTION
DELIVER MESSAGE
TEST AND LEARN
KNOW THE CUSTOMER
CREATE APPROPRIATE MESSAGE
IDENTIFY POTENTIAL CUSTOMER ACTIONS
15
Marketing and Data Analytics allows you to Create
Appropriate Message
LISTEN
ACTION
DELIVER MESSAGE
TEST AND LEARN
KNOW THE CUSTOMER
CREATE APPROPRIATE MESSAGE
IDENTIFY POTENTIAL CUSTOMER ACTIONS
16
Marketing Delivers the Message to the Customer
LISTEN
ACTION
DELIVER MESSAGE
TEST AND LEARN
KNOW THE CUSTOMER
CREATE APPROPRIATE MESSAGE
IDENTIFY POTENTIAL CUSTOMER ACTIONS
17
Data Analytics allows you to Listen to the
customers response
LISTEN
ACTION
DELIVER MESSAGE
TEST AND LEARN
KNOW THE CUSTOMER
CREATE APPROPRIATE MESSAGE
IDENTIFY POTENTIAL CUSTOMER ACTIONS
18
Data Analytics allows you to Track the Customer
Responses and gain Insights
LISTEN
ACTION

• Customer responds to the
• message
• Key Learnings are
• integrated into future
• programs by marketing

DELIVER MESSAGE
TEST AND LEARN
KNOW THE CUSTOMER
CREATE APPROPRIATE MESSAGE
IDENTIFY POTENTIAL CUSTOMER ACTIONS
19
Establishing a Test Learn Partnership between
marketing data analytics will maximize results
LISTEN
ACTION

• Conduct sophisticated tests,
• share learning widely, and
• implement fast read and
• re-launch capability

DELIVER MESSAGE
TEST AND LEARN
KNOW THE CUSTOMER
CREATE APPROPRIATE MESSAGE
IDENTIFY POTENTIAL CUSTOMER ACTIONS
20
The Concept of Testing

• Why Test?
• Good economics
• Use a sample to learn what works and what doesnt
  work before rolling to entire database
• Continuous Improvement
• Learn how to improve marketing programs to ensure
  theyre the most effective

21
Testing Multiple Variables

• Test all or some variables
• Why?
• Learning Loop Generates constant feedback on
  how to improve effectiveness of communications
• Commonly tested variables
• Lists
• Offers
• Creative execution
• Channel
• Content

22
Testing an Idea

• Four Steps
• Plan Test
• Define objectives
• Set up test and control groups
• Execute Test
• Track Results
• Analyze Results
• Response rate
• ROI
• Cost per response
• LTV

23
Example - Department Store

• Assumptions
• Store has a house credit card tied to customer
  database containing 400,000 men and women
• Store credit card allows capture of information
  about purchases
• Store has new line of designer clothes for women,
  being promoted through print ads
• Would like to increase sales of new clothing
  line
• Decide to test a direct mail program with a small
  group of women customers, before roll out to
  entire database
• Offer If buy new suit by May 30, will receive a
  free piece of costume jewelry worth 20 by
  presenting this offer

24
Step 1 Plan Test

• i) Define Marketing Objectives
• What are you trying to accomplish?
• Objectives should be measurable and time-bound.
• Department Store Example
• To increase sales to existing customers by 4
  within 1 year.
• To achieve sales of new clothing line of 4.2
  million.
• To increase LTV per customer from 80 to 125
  over next 12 months.

25
Step 1 Plan Test

• ii) Set up test and control groups

26
Why use a Control Group?

• Allows you to measure the effect of the promotion
  versus not running it
• No offer or promotional piece sent to the control
  group
• Can be larger/smaller than test group

27
Step 1 Plan Test

• Set up test and control groups
• Query the database to determine how many women
  have credit cards in their name
• Example - Department store
• 200,000 women with department store credit card
  in their name
• Must select 2 groups from this 200,000
• Women who get the direct mail offer (Test)
• Women who do not get the DM offer (Control)

28
Test Control Groups How large?

• Cost considerations make as small as possible
• Statistical validity make as large as possible
• Rule of Thumb
• Each group must be big enough so that you receive
  at least 500 responses from the promoted group
• Example
• If anticipate response rate of 2
• Test group needs to be (500/2) 25,000

29
Test Control Groups How large?

• Example Department Store
• Anticipate response rate of 2.5
• 200K women in database
• Test group size 500/.025 20,000

30
Step 1 Plan Test

• Set up test and control groups
• Construct Test Group using Nth method (per RFM)
• YOUR CONTROL WOULD BE THE SAME FOR THE ENTIRE
  MAILING UNIVERSE, REGARDLESS OF HOW MANY CELLS

Nth Total customers in database Test Group
Quantity

• Example Department store
• Test group of 20K
• Add another 20K for control group total 40K
• Nth 200,000/40,000 5
• Select every 5th customer from master database
• That is, select customer record 5, 10, 15 ...

31
Why use Nth select?

• Test and Control groups must be exact statistical
  replicas of the master database
• Must mirror the master database - will have the
  same percentage of people with similar
  characteristics
• Same postal code
• Same income
• Same of children
• Same lifestyle
• Same purchase behaviour etc.

32
Step 2 Execute Test

• Execute Program among test group, interacting
  normally with control group

33
Step 3 Track Results

• Assign a source code
• A source code is assigned to each test variable
  to facilitate measurement and analysis
• A source code is a series of letters or numbers
  used to identify a particular offer
• Rule different source code for each new variable
• Example
• Women who got offer OFFERMAY03
• Women who did not get offer NOOFFERMAY03

34
Step 4 Analyze Results
What is the key learning?
3,000 responses
2,000 responses
35
What is a response?

• A response can be ...
• Phoning a 1-800 number
• Providing information (e.g. survey answers)
• Entering a contest
• Purchasing a product
• Signing up for a service

Our example
36
Step 4 Analyze Results

• Evaluate success using a number of factors
• How did the program perform relative to
  objectives?
• Did the promotion come in on budget?
• Metrics used to analyze performance
• Response Rates Analysis (RR)
• Cost per Response (CPR)
• Return on Investment (ROI)
• LTV

37
Response Rate Analysis
38
First, calculate response rate for Test group
Response Rate Analysis

• Department Store Example
• Direct mail offer Get free piece of costume
  jewelry if buy suit by May 30
• 20,000 mailed, 3,000 responded

Test RR Responder Quantity x 10015
Test Quantity
39
Then calculate response rate for the Control group
Response Rate Analysis

• Department Store Example
• 20,000 in Control Group do not receive direct
  mail offer
• Still, 2,000 people respond to print advertising
  and buy a suit by May 30
• Control RR Responder Quantity x 100
• Control Quantity

40
Third, calculate Lift between groups
Response Rate Analysis

• Lift Test RR Control RR x 100
  Control RR
• Evaluation
• The higher the lift, the better
• Positive Lift Test performed better than
  Control
• Negative Lift Control performed better than
  Test

Based on the Department Store example, what is
the incremental lift percentage?
41
Cost per Response Analysis

• Campaign Costs / Budget Include
• Planning Campaign Development
• Agency Costs (e.g. Fees, Creative Development)
• List Development (e.g. data work)
• Campaign Execution
• Printing, Laser/Lettershop, Postage
• Response Costs
• The marketing cost associated with response to a
  database marketing campaign
• BRC postage, data entry, offer fulfillment, call
  centre

42
Cost per Response

• Cost per response Total cost of program
• responses
• Department Store Example
• Total program costs 210,000 (includes campaign
  development, execution, response costs)
• Cost/response 210,000/3,000 70

Evaluation the lower the cost, the better
43
Return on Investment (ROI) Analysis

• ROI what you earn on a campaign relative to
  what you spent on a campaign
• Evaluation the higher, the better
• Objective To determine if you made money from
  your database marketing investment

44
Return on Investment Analysis

• ROI Revenue Program Costs x 100
• Program Costs
• Department Store Example
• Total program costs 210,000
• Sales revenue 450/suit(4503000)
• 3,000 responses to program

What is the ROI ?
45
Short Term vs. Long Term Effects

• Lift, cost per response and ROI are effective
  in determining short term payout to help increase
  the effectiveness of marketing communications
• Good testing programs will follow the test and
  control groups for the next 12 months to
  determine the residual effects of the test - for
  example
• Those who responded likely moved to higher RFM
  cells
• They all became recent buyers, perhaps more
  frequent buyers and their monetary scores
  probably increased

46
Lifetime Value

• Next step
• Determine promotion effect on lifetime value
• Increased lifetime value, rather than immediate
  short-term payout, should be the real goal of
  database marketing
• Test effectiveness of alternative ways of
  increasing LTV

47
Testing an Idea

• Four Steps
• Plan Test
• Define objectives
• Set up test and control groups
• Execute Test
• Track Results
• Analyze Results
• Response rate
• ROI
• Cost per response
• LTV

48
Metrics ExampleCIBC Direct Mail Creative
Execution Test
49
Example CIBC Creative Test

• 3 different Direct Mail pieces created for launch
  of CIBC Adventura Gold Visa card
• Packages all the same except the outer envelope
• Cell A High-end envelope CIBC logo
• Cell B High-end envelope Adventura logo
• Cell C High-end envelope CIBC logo
  Aventura logo

50
Example CIBC Creative Test

• Calculate the lift, cost per response and ROI
  for each cell
• Which envelope creative would you roll out to the
  entire database of customers?

51
Example CIBC Creative Test
10,000
3,500
52
Example CIBC Creative Test
10,000
3,500
7
40
28.57
53
Example CIBC Creative Test
ROI Revenue Program Costs x 100 Program
Costs
100
40
200
Revenue
54
Example CIBC Creative Test
ROI Revenue Program Costs x 100 Program
Costs
200
100
40
Revenue
55
Based on the results, which envelope creative
would you roll out to all customers?
Example CIBC Creative Test

• Cell A High-end Envelope CIBC logo
• Cell B High-end envelope Adventura logo
• Cell C High-end envelope CIBC logo Adventura
  logo

56
In-class Exercise (Worth 10)-Part 1

• Read Luring em back to school, Strategy
  Magazine, November 2003
• Write 2 measurable, timebound objectives for the
  integrated marketing programs executed by CMC.
• What was the CMC strategy?
• What direct marketing tactics were used?
• How would you measure campaign success?

57
In-class Exercise-Part IILuring em back to
school

• Complete the following table comparing the
  differences between the direct mail and e-mail
  catalogue mailings. Which program appears to be
  more successful? Why?

58
Statistical Significance

• Statistical certainty is impossible
• We normally talk of level of confidence in
  statistical predictions
• In DM this is often 95 (19 out of 20 times) or
  90 (18 out of 20 times) confidence - results
  will be repeated within an acceptable margin of
  error
• The confidence level set normally depends on
  financial risk

59
Where Are the Other 95 - the Direct Marketers
Non-respondents

• Research evidence suggest that it is all due to
  poor timing!!
• Not ready or unable to transact because
• lack of funds
• dont know how the product or service will
  perform
• domestic upheaval (e.g. moving house)
• Is this the reason why repeat mailings and
  follow-ups are often successful?
• Also, is this the reason behind the possible
  discrepancy between test results and roll-up?

60
Selecting Response Channels

• How do you want them to respond?
• The 3 main channels are
• Mail
• Phone
• Internet
• Additional Channels include
• Mobile Devices

61
Response Channel Specific Metrics
Direct Mail Response Rate versus no mail group Creative Tests-different letter versions Offer Tests-different offer types Response Mechanisms (call/in-person) Telemarketing Response Rate versus no calll group Percentage Right Party Connect Wrap code analysis Cross and Up sells Creative Testing-Scripts
Internet Response Rate versus no contact View Rate Abandon Rate Accept Rate Click Through Rate Re-visit Rate Creative Tests-different content pages Push versus Pull tactics Channel Combinations Response Rate versus single channel

62
Measurement

• Its not enough to count responses.
• Response does not indicate the level of customer
  commitment.
• Measuring response doesnt tell us WHY consumers
  behave the way they do.
• Response builds only limited knowledge of
  customer behaviour.

63
Beyond Response

• What kind of people are responding?
• What other market segments are there?
• What offers trigger different groups to respond?
• How many ways can we present a message?
• Where are the overlaps in media used?
• What messages are appropriate for various media?

64
Performance Measurement

• Historical data can be useful in evaluating the
  performance of similar marketing campaigns.

65
Performance Measurement
MEASURE OPERATIONALIZATION
Response rate Percentage of prospects contacted who replied
Number of inquiries Number of fulfillments
Number of qualified leads Number of leads who expressed interest that were converted into sales or opportunities
New customers acquired Number of purchasers who had not purchased before
Customer lifetime value Net present value of customer over a specified period of time
Customer acquisition cost Total marketing costs divided by number of new customers
66
TEST
TEST
TEST
67
Testing Variables

• Products/Services
• Media e.g.. Lists, print, Internet
• The Offer
• Formats/Layouts
• Timing Schedules

68
Common Experimental Designs

• Split-run experiment
• Compare responses of campaign A to campaign B
  using the same list (split in two)
• Before-and-after experiment
• Compare the outcomes of campaign A recipients to
  a control group that did not receive it.

69
Good Pre-test Design

• A good experiment will measure the effect of ONE
  variable on another (response rate).
• Compare, on a limited audience
• (Offer A) vs. (Offer B) vs. (Offer C)
• (Creative A) vs. (Creative B)
• 3. (Segment A) vs. (Segment B)

70
Bad Pre-test Design

• Marketer attempts to
• alter more than 1 variable per test cell in the
  same experiment
• compare results in one medium to another
• test different response channels
• split the list into test cells that are too small
  (nlt30 responses)

71
Next Week Test Structure (25)

• Class Test July 18th, 2006
• 2 hours
• Final Exam
• Responsible for everything covered in class,
  including handouts
• Covers Materials from Week 1-Week 10
• Structure
• Multiple choice
• Short Answer
• Metrics Problem
• Case Study