What is Deep Analytics?

As data analysts, we waste too much time on making dashboards for other people and not enough time on answering deep questions about critical business issues. This is a waste of resources for the individual, and a waste of resources for the business.

The real meaty questions about the business such as: “what contributed to the sudden growth in revenues last month?” cannot be answered by a generic dashboard with revenue and traffic numbers. You need to do a deep dive into the problem. You need to unravel the question into specific queries.

  • Query revenue by month for the past twelve months
  • Check for seasonality patterns by cross-checking monthly revenue, spaced 12 months apart
  • Query marketing spend data
  • Query marketing traffic data
  • Query product usage data and cut by different actions taken by users
  • Look for differential product actions taken by all users
  • Look for differential product actions taken by cohorts of users

The conclusion may be that there was a seasonal effect (people buy more during certain times of the year), there was a successful marketing campaign that boosted traffic into the product, or a cohort of users materially affected the revenues. Whatever it is, there is a deep, explorative nature to finding out the answer to a seemingly simple question. There is also a second facet to this workflow: organizing and sharing these findings. I call this entire process “Deep Analytics” (as opposed to “shallow analytics” that is more focused on building generic metrics and sharing insights via dashboards — a more blunt, if not more all-purpose, tool).

Deep Workflows: The Forgotten Child

An example of a deep workflow. Image courtesy of Hyperquery

The problem is that the broader business community does not put weight into these deep workflows, even though this is precisely the type of work that drives the most amount of business outcomes. The really great, idiosyncratic insights don’t materialize from staring at dashboards. They come from looking at the business and its data in novel, non-obvious ways. And this by definition cannot be done in a one-size-fits-all fashion.

The truth is that even the analytics community does not put weight into these deep workflows. Lack of care result in lack of standards and tools. A clear example: the SQL IDE has not been redesigned in decades. Sharing and organizing analytical queries is an afterthought. There are no standards around discovering and choosing the different versions of analytical data to be used for analysis.

Call for Tooling: Tools Define The Workflow

Great tools for a workflow don’t just make the workflow more efficient. They define the workflow. Github defined version control and collaboration for code. Figma defined collaborative interface design. Ableton Live defined electronic music production. We need a tool that defines Deep Analytics.

Traditional BI tools just don’t cut it. They are built to help the analyst to build self-service data tools (e.g. dashboards) for the general business audience. SQL IDEs don’t cut it either — they were built for transactions, not for explorations (too many tabs!). Jupyter notebooks are great for data science workflows in Python, but not for analytics. Data Discovery tools are great for general observability, but are not integrated into the query-writing workflow. Wiki tools like Notion and Confluence are great general-purpose places to share information, but are not purpose built for sharing analytics insights. It’s a disjoint workflow with tooling that is not focused on getting the job done.

Call for Attention: Deep Analytics Matters

By giving a workflow a name, we bring life to a concept. We can refer to a nuanced concept in a concise way. We have a name for Data Science as a discipline. We have a name for Deep Learning as a transformative class of predictive models. We invented the term Business Intelligence to signify the importance of data-driven decision-making through self-service tools. More recently, we invented an entirely new profession: the Analytics Engineer. We need to give attention to the category of Deep Analytics. And we need tools to help us be more effective at it.

Original post: https://towardsdatascience.com/what-is-deep-analytics-e12d0c625044

Leave a Reply

Your email address will not be published. Required fields are marked *