Highlights §
- On the one hand, data and design feel like opposites:
Analysis versus aesthetics. Facts versus feelings. The cold hardness of numbers versus the warm fluidity of experience.
Product decisions are driven by metrics rather than people, complains one side.
Product decisions are made with ego rather than impact, charges the other. (View Highlight)
- In the deep heart of data and design burns the same flame: the search for the truth of a phenomenon. (View Highlight)
- Data is the contextual stage, the lessons learned, the structured hypotheses.
Design is the sword of action, the next experiment, the possible futures.
We diagnose with data. We treat with design. (View Highlight)
- The biggest misconception of data is that it provides certainty.
That somehow, a set of numbers is scientific, smart, and foolproof.
We think data is angular and rigorous, like a problem set or proof.
But data is not math; data is statistics—the messy world of aggregations, probabilities and percentages. Squishy as marshmallows. Leaky as cliches (View Highlight)
- The most decisive leaders think of data not as a diagnostic prop but as an instrument of operational hygiene, similar to regularly scheduled check-ups to the doctor or dentist. (View Highlight)
- You don’t look at data only when something is wrong; you look at it regularly—every day or at least every week.
This kind of operational hygiene helps you in a few ways:
- You pick up on issues early — for example, noticing slowing growth before it becomes a Stage 4 problem.
- You create more accountability for yourself and others — did that feature launch actually move the needle? Is our current strategy actually delivering what we hoped?
- You develop a better intuitive feel for your business — what’s normal, what’s not? Which further experiments are likely to resonate with customers, which aren’t?
- You start to notice patterns of questions asked over and over again, which pushes you to better discover the levers of your business. (View Highlight)
- Yet many of us who have tried to use data to inform decisions in organizations have experienced a different reality. One where we are constantly confused by how metrics are defined, bicker over how to interpret various analyses, and struggle to apply the insights into action. (View Highlight)
- It’s because building a data-informed culture is hard. Logging user actions, creating dashboards, running A/B tests, and shipping ML models — these are useful. But they are not the foundation of being data-informed. (View Highlight)
- We believe that being data-informed comes down to internalizing a set of values. These are simple, few, yet exceedingly powerful:
- Conviction around a purpose rather than searching for meaning in numbers
- Setting verifiable goals rather than vague aspirations
- Company-wide familiarity with metrics rather than outsourcing to “data people”
- Active testing of beliefs rather than seeking support for intuition
- Accepting probabilities rather than thinking in absolutes (View Highlight)
- The first step to being data-informed is understanding what data can’t do: give you a purpose. (View Highlight)
- Increasing Metric X” is not a purpose; a true purpose must relate in some way to creating value for other humans. If a data-informed team feels at any point like they are optimizing metrics in a way that compromises their mission, they scrap that work. (View Highlight)
- Before you can collect data to help you track what matters, you need to define what actually matters. Information itself is not an evaluation criteria. (View Highlight)
- How will you know if you are fulfilling your mission? You imagine outcomes that build towards the future you envision. Then you set goals to help you verify whether you are achieving those outcomes. (View Highlight)
- Data-informed teams push for quantitative goals to the greatest extent possible because they’re the best way for a team to focus on creating impact. They make progress transparent, force accountability, and rally your team around a shared outcome. (View Highlight)
- Goal-setting is more art than science. All metrics are proxies and every set of verifiable goals will have shortcomings in what they fail to capture. As you learn those shortcomings, you will iteratively refine your methods of measurement and your targets. Setting goals is a skill — you have to practice it to get better. (View Highlight)
- Everyone has to know the numbers. You cannot outsource a data-informed culture.
You can have the best team of analysts — but if you don’t know the numbers, then your decision-making will suffer. (View Highlight)
- Data-informed teams regularly meet to review key metrics. Why? Decisions are never made in these meetings, so why have them at all?
These meetings signal something important about the culture — that simply knowing and talking about the key metrics matters. Doing so builds a shared foundation among a team of what is going well or not well, and how they might best prioritize efforts (View Highlight)
- Good data-informed teams develop leaders who can weave data into clear narratives. These teams cultivate a skilled understanding of which data patterns are important and which ones are not. They point out conflicts in data and which interpretation is more likely. Most importantly, they are upfront about what the data can and cannot say. (View Highlight)
- We all have hypotheses about what our customers care about, what products will win, and what decisions are best. The more experienced we become, the more we trust — and even pride ourselves on — our intuition. (View Highlight)
- A well-honed intuition about which path to take is supremely valuable because we don’t have infinite time and resources to try every path. (View Highlight)
- The danger comes when your pride in your intuition leads you to close yourself off to evidence that you might be wrong. Instead of testing your intuition with data, you seek out data that confirms your intuition. (View Highlight)
- Openness to being proven wrong is insufficient. Data-informed teams actively seek out information that might disconfirm their assumptions. They value a high velocity of experimentation and setting hold-outs. Like scientists, they are constantly looking to test their hypotheses and validate their beliefs. They don’t view intuition versus data as a forced choice — they use data to refine their intuition over time. (View Highlight)
- Members of data-informed teams regularly ask each other: “What evidence would convince you that your intuition is wrong?” If the honest answer is “No piece of data would convince me,” then you have strayed off of the data-informed path. (View Highlight)
- Data will never give you certainty. Interpreting data means taking on many assumptions that are reasonable but not bulletproof. Even trusting a single piece of analysis means having faith that the events were logged correctly, that the metrics have been calculated without error, that everyone understands exactly what was logged and calculated, and that the interpretation is sound. You want to be 100% sure? Good luck. (View Highlight)
- People who are not truth-seekers constantly take advantage of this. The easiest way to discredit data is to demand perfection from it. (View Highlight)
- When data and intuition collide, some people always pick their intuition. They would rather be wrong betting on their intuition than wrong betting on data (this is rampant in the world of sports). That is a defeatist mindset. Your commitment to your mission requires that you make good decisions fast. Sometimes you will trust the data and you will be wrong. But if using data increases the likelihood of making the right call by 5–10%, those benefits will quickly compound. (View Highlight)