On a privacy survey, 90% of users said they “care deeply about personal privacy.”
Backend data showed that the same batch of users, without a moment’s hesitation, granted permissions for location, contacts, and microphone — all in exchange for a free sticker pack.
The data was not fabricated, and the response rate was perfectly fine. The problem is: the survey measured “what users are willing to say,” not “what users actually do.” When there is a gap between your measurement tool and what you truly want to measure, every number you get is wrong — even if it is technically correct.
This is measurement bias: the numbers are not miscalculated — your “ruler” was measuring the wrong thing from the very start.
7. Social Desirability Bias: Respondents Are Performing
Survey responses are often the respondent’s guess at what you expect to hear, not their actual thoughts and behaviors.
The privacy survey is the textbook example. “Do you regularly change your password?” Most people answer “yes” — backend logs show their password has not been changed in three years. This is not lying; it is the brain automatically choosing “socially desirable answer” over “actual situation.”
In user satisfaction surveys, this bias hits especially hard. Dissatisfied users may not bother to fill out the survey, and those who do may give inflated scores because they do not want to make the developers feel bad. Making decisions based on this kind of “self-reported” data is about as naive as trusting everyone who writes “proficient in Excel” on their resume.
The fix: replace self-reports with behavioral data. What users say they do is no match for what logs record them actually doing.
8. Observer Bias: The Measurer’s Subjectivity Contaminates the Result
Observer bias is when the measurer’s own expectations contaminate the measurement result. It is not the subject performing — it is the measurer seeing things wrong.
When engineers test their own code, they unconsciously avoid the operation paths “most likely to fail.” This is not deliberate cheating; it is the brain protecting self-esteem. When you expect a feature to succeed, you involuntarily choose the test cases most likely to confirm that success.
In code reviews, if you know the code was written by a senior engineer, your review standards shift compared to when you think it was written by an intern — even if the code is identical. This is why good code review processes sometimes hide authorship information.
9. Recall Bias: Memory Is a Screenwriter, Not a Camera
Ask a user “how many times did you use this feature in the past year?” and the data you get is essentially useless.
Human memory is not a recording device — it reconstructs based on emotion and narrative. We tend to remember peak experiences (Peak) and the most recent ending (End), while forgetting most of the moments in between. This is Kahneman’s Peak-End Rule.
A user might remember solving an urgent problem with a feature (a vivid impression) but forget the dozen times they tried it, found it useless, and gave up. What you receive is the version they remember, not their actual usage trajectory.
The fix: always prioritize system logs over user recall. Behavioral data is more honest than survey data.
10. Instrument & Measurement Error: The Ruler Itself Is Bent
Measuring API response times with a monitoring system that has latency issues gives you nothing but noise. This is not a problem with the analysis method — the measurement tool itself introduces error.
In the tech world, this bias more commonly shows up as “tool discrepancies”: the retention rate gap between iOS and Android may not be caused by functional differences between the two versions, but because your data collection SDK crashes on certain low-end Android devices, meaning those users’ data simply never makes it back. The “retention gap” you see is just an illusion created by missing data.
Inconsistent reporting logic across different app versions is another common form of instrument bias: the new version redefines a certain event, while the old version still uses the old definition — cross-version data comparisons become apples to oranges.
Before designing your data collection, the most important question is not “what do I want to measure” but “will my measurement method systematically measure something else entirely?”
11. Confirmation Bias in Collection: Only Recording What You Want to See
This is different from confirmation bias at the analysis stage. Here, we are talking about selectively recording only those observations that match your expectations during the data collection phase itself.
When you think a colleague is lazy, you pay extra attention every time they take a break and mentally log it; when they work overtime, you do not flag it the same way. Over time, your “mental database” contains only instances of them slacking off — a biased sample formed not because you fabricated anything, but because your attention was asymmetric from the start.
In user research, if you already lean toward a certain design direction, the questions you ask during interviews, the threads you pursue, and the points you record may all systematically tilt in that direction. The “user feedback” you end up with looks more like an echo of your own ideas than the users’ authentic voice.
12. Temporal & Seasonal Bias: You Measured the Right Number at the Wrong Moment
Measuring website traffic at 3 AM gives you a correct number (indeed, only 10 people are online), but that number is meaningless for daytime decisions. The value is not wrong — it simply does not represent the situation you are trying to understand.
Seasonal bias is another facet of temporal bias: a fitness app’s January data will look absurdly good — user growth surges, engagement spikes — thanks to the New Year’s resolution effect, not because your new feature is working. If you launched a new feature in January and attributed that month’s data growth to it, you would reach an entirely wrong conclusion.
E-commerce sales spike in December not because your redesign worked — it is Christmas. Mistaking seasonal fluctuations for your own accomplishments is the most common form of self-deception.
Temporal bias and the “time window bias” from the sampling bias article have a key distinction: time window bias is about selecting the wrong population by time period (different people show up at different times), while temporal and seasonal bias is about picking the wrong moment to measure a phenomenon that naturally fluctuates over time. The former is about “who enters your sample”; the latter is about “your measurement timing distorting the numbers.”
Measurement bias teaches us one thing: the numbers you can see are often proxies — stand-ins for the thing you actually want to measure. When the proxy itself is biased, every downstream analysis is a skyscraper built on a flawed foundation.
Before accepting any number, ask: “How was this number produced, and does it really measure what I think it measures?”
