When your measurement sources disagree

Disagreement can be a useful starting point

A shopper names a friend as the reason they discovered a store. An order record shows a visit from search before the purchase. Those accounts can describe different parts of the same journey. The immediate task is to understand the difference, not declare one source the winner.

Treat a mismatch as an investigation prompt. A survey response captures the answer someone gives to a particular question. A tracking system records the interactions available to it, then a report applies definitions and attribution rules. Neither description, on its own, proves what would have happened without the marketing activity.

Make a small source map

Before comparing totals, create one row for each source. Describe the event or answer it records, the unit it counts, and the period it includes. A row might describe a customer’s selected discovery answer; another might describe an order assigned to a channel under a particular attribution model.

Name what is missing as well as what is present. You may not have a survey answer for every order, or a usable tracking history for every respondent. “No recorded source” belongs in the source map. It should not be silently translated into “no marketing influence.”

• What is the unit: person, response, session, order, or credited conversion?
• Which dates, filters, channel definitions, and attribution rules apply?
• Which records cannot be matched or classified?

Read the rule before reading the ranking

Attribution is a rule for assigning credit to observed interactions. Shopify’s current marketing-report documentation describes several models, including first click, last click, and linear allocation. Changing the model can change which channel receives credit while the underlying order remains the same.

Record the actual model in the report you are using. Do not assume a default from a different dashboard or an older walkthrough. Also check the selected sales measure and how the report treats order status. A channel ranking is difficult to interpret when its label hides which business event and allocation rule produced it.

Compare the records you can actually connect

If you can connect a response to an order, inspect a small set of those matched records before building a larger story. Check that the survey question refers to discovery, the latest visit, or purchase motivation as intended. These are different questions, even when their answer options contain the same channel names.

Keep unmatched responses and orders visible as separate coverage information. A neat chart built from a subset can be useful, but its caption should say which subset. Avoid scaling the observed mismatch up to the whole business unless you have a defensible reason to expect the missing records behave similarly.

Write down more than one explanation

Imagine a hypothetical mismatch in which an answer names a creator but the recorded interaction is search. Possible explanations include remembering an earlier introduction, interpreting the question differently, or a missing trackable visit. These are possibilities to investigate, not findings established by the chart.

Choose a follow-up that distinguishes among them. You might inspect the wording of the question, review campaign-link conventions, or read optional customer explanations. Changing several tracking rules and the survey at once may make the next comparison harder to understand. Preserve enough of the original setup to tell what changed.

Turn the gap into a proportionate action

A useful note might say: “These matched responses describe a different discovery source from the channel credited in this report. We will review the question and tracking setup before changing spend.” That statement makes the evidence useful without turning it into a claim of incremental sales.

If a budget decision depends on whether a channel causes additional purchases, you need an appropriate evaluation design beyond a disagreement between reports. For everyday operations, the first win is simpler: shared definitions, visible coverage, and a specific investigation that another person can reproduce.