Data Observability: A Guide to Holistic Decision-Making

Gartner's study surveyed 500 organizations and found that 88% currently use data observability or plan to do so in the next 12 months. This number is expected to grow to 98% by 2025. Wow! We are almost at the end of the journey. Well, not quite. Saying doesn't equal doing.

Most organizations claiming that they follow data observability do not necessarily elevate above the troubleshooting data quality issues. The primary function of this significant term narrows down to identifying and fixing data quality issues, such as missing or corrupted data. We all know that this can't be done thoroughly. Let's leave the semantics for a moment and move to another category that companies claim today, like detecting data anomalies, such as unexpected changes in data patterns, or the fact that organizations claim to monitor data pipelines to ensure that data is flowing smoothly and accurately (I know this is an overstatement). Companies use other categories today driven by compliance, like GDPR, CCPA, and similar regulations worldwide. That is a fair point, but mostly to please the governments. Finally, we have data governance by providing visibility into data usage and access. That is taking multiple forms and shapes.

So why I'm so skeptical about these claims? The answer is - the future. Unfortunately, our data drives almost everything these days. You don't have to take it from me; I'll quote someone most of us know.

"Data really powers everything that we do." – Jeff Weiner, former CEO of LinkedIn.

If that's the case, and I agree with that statement, we are already creating large amounts of data beyond our ability to use them effectively. Data are still disconnected between the systems or, at best, connected fundamentally. I remember my first job in finance, which was challenging while trying to connect several systems before I could do my analysis. I needed to dance like this each month or more frequently to provide several simple answers (sure, I could be brighter, too). I spent 50-70% of my time connecting and clearing the data before something ended in a nice flat file. Don't get me wrong, I've learned a lot going through the pain, but I could be more effective in spending less time on connecting the legacy systems data with new systems that were implemented. Learning is not enough now because learning quicker and getting the answers to critical questions is all that matters. I'm sure that situation is much better these days, but we are still trailing behind the trend that is already in the future.

While Mr. Weiner might not have been talking about data observability per se, his point stands strong in today's business environment. These days, as executives, we have countless charts, graphs, and reports (still spreadsheets - nothing against it, but perhaps too many). So, we elevated the game from Excel to dashboards but needed help selecting the correct data to derive actionable insights.

Like with almost everything, if something is too much, it becomes overwhelming, and we lose the ability to see the clear objectives. That's the moment when a simple decision takes forever to make. We are spending more time understanding the processes, who does what, how the tables are created, what system, when data is pulled, charts relations, how they compare, or even the data compilation process (we could write about it forever). There are better ways to use your time than that.

Data observability isn't just about staring at numbers and graphs; it's about understanding those numbers' underlying stories that lead to conclusions and decision-making. Often, we rely on others to help us with that because the process is too painful or considered a waste of time. That's only a partial solution. In a startup or big company, data isn't just a by-product of operations; it's the lifeblood that fuels decision-making and strategy execution. The aim is not to get a dashboard but a number of critical decisions we can make based on the inputs. That is the whole point of the data observability.

Here are a few thoughts on why I think data observability is essential:

Identify and Troubleshoot Problems - Every executive's nightmare? Unforeseen problems are causing operational and financial hiccups. With data observability, these become less of a "Boo!" and more of an "I knew you were coming." Let's imagine your team is reckless about the spending while you know that dark clouds are gathering ahead of 2023 (I couldn't resist). Are you increasing your prices for your services in the bearish market? Are you investing in additional sales reps who could be more productive? Are you inflating your CAC while your LTV is declining? You see, we get these signals every day, but without connecting the dots, the picture is a piece of a jigsaw puzzle. You know there is an eye for something, but you don't know what. But it could be the eye of the cyclone.

Making Superior Decisions - Have you ever heard the saying, "A captain is only as good as his ship's radar"? Okay, I made that up, but the point stands. With precise data visibility, decision-making becomes less of a gamble and more of a calculated strategy. I've said this many times: If you don't know where you are going, you are going nowhere. I still believe that's the case. However, even if you know, do you know how to get there by picking the fastest route? That's what the data should tell you or hint about. Strategy is your 10% of a success story; the remaining 90% is how to execute the strategy. You can't rely on your judgment but can on data and assessment at the same time. I've said this many times before data alone is not a panacea but feeds your judgment capabilities that should evolve as you learn more about the facts.

Elevating Customer Experience - Know what your customers want before they do. That's the magic of data observability. It offers insights into customer behaviors, preferences, and pain points. Customer experience is often perceived as a reactive or cost-driven activity that should be optimized. Sure! Nobody wants to overpay for the resources not driving higher ROI, but let's start thinking about the customer experience from the first phone call your SDR is making. How about your website visitors? Are they served? Or are you trapped in a form that will make them pay for that visit with the 30-minute conversation with your SDR? Are they deceived or offered a real deal? Finally, how would you want to be treated in their shoes? Your data tells you everything you want to know about what they do but not how they want to be treated. But it could.

Risk Reduction - Life's a game of risk. But who says you can't tilt the odds in your favor? Executives can craft resilient and flexible strategies by understanding potential pitfalls and vulnerabilities. We need to be more often thinking about what could go wrong and what should be our contingency plan. The hurrah drives us, optimistic ice hockey sticks that make us "want something" instead of being realistic. Do you want to look like a naive dreamer that we have unicorns and rainbows everywhere? Although it is a rhetorical question, and I know the answer, we all fall into this trap, avoiding discussing the risks around us.

Conclusion

As we wrap up, remember that in today's business environment, data observability isn't just a tool; it's a culture, a mindset, a way of corporate life (yes, it is). The compass guides strategy mitigates risks, and propels businesses to new heights, allowing them to make timely decisions. Next time you look at a data report, remember: It's not just numbers; it's the very pulse of your company and impact on your ability to make decisions. Focus on a few metrics that are well defined and decisions it allows you to make.

Imagine a football game (American Football) where the coach has no strategies and decides to play without observing the opponent's movements or his team's performance. He sends his team out and hopes for the best. This game would probably be chaotic, with missed opportunities, errors, and a higher risk of losing.

Data observability as the coach's playbook and real-time strategy discussions. The coach (business executive) observes every player (data point), sees how they interact on the field, reacts to the opposing team's tactics, and adjusts the strategy based on real-time developments. Through data observability tools, like instant replays and stats, the coach gains insights into areas of strength and weakness, making informed decisions to increase the team's chances of scoring and winning. Only some metrics that the game can produce (remember Moneyball?)

Operating without data observability is like a football coach without a game plan, sending the team onto the field blindfolded. Just as every touchdown, interception, or fumble informs the coach's next move, every data insight should guide our strategic decisions. Do we want to risk fumbling our business operations, or do we want to score every touchdown opportunity that comes our way?

It's your call.

‍