The Future of Analytics: Cloud Data Warehouses, Data Lakehouses, and More

What’s ahead for data analytics during the next 12, 24, or even 36 months? This area of business technology has seen sweeping changes in recent times, largely thanks to the rise of accessible AI. And with it, there has been a concerted movement toward integrating the tools and platforms that deliver and support analytical applications—be they predictive, AI, or pattern matching.

Close to half of organizations in a recent DBTA study, 49%, place high priority on enabling AI and generative AI (GenAI) use cases. The same percentage are also focusing on improving data freshness and accessibility. Another 37% seek to enable self-service data analytics for decision making. In addition, 49% of enterprises have expectations for real-time streaming data that can take advantage of AI-enabling architecture and business cases, the survey also shows. As a result, this puts data lakehouses in strong strategic positions, as cited by 34%.

A data fabric and semantic layer can also be positioned to support these environments or architectures, now prevalent among 39% of the enterprises surveyed.

The challenge is that many enterprises have a spaghetti architecture with connections and integrations across applications, systems, and data environments. Typical enterprises have a crisscross of tools and platforms.

The difficulty is that there is no single source, repository, or system for data employed with analytical applications.

The data needed for the next generation of analytics is highly distributed across many systems and, unfortunately, siloes within today’s complex enterprises. Enterprise data environments are characterized as cloud data warehouses, data lakehouses, and real-time unified platforms.

There is, and will always be, a need to identify and connect with different data sources, whether they come through cloud applications, packaged applications, third-party applications, or homegrown applications. Plus, technology is evolving at such a rapid pace, data analytics systems—and their supporting pipelines—consistently need reassessment and revising. An analytics stack designed just 3 years ago, for example, is unprepared for the rise of GenAI that has happened during this time.

What is needed is a data analytics architecture that can adapt to an unknown future—which could be as few as 6 months away. The impetus is to be able to move data as close as possible to processing and analytics applications—so it is available to downstream applications such as production or marketing systems, where operators rely on GenAI insights.

A modern data architecture needs to be built through the following best practices to achieve an analytical-focused data environment:

• Engage the entire enterprise. The teams that will benefit the most from AI-based analytics will not be developers and data scientists, but business experts. It’s important, then, that business or nontechnical users are able to access and take advantage of data from all sources coming into the enterprise.
• Unify data resources and assets from across the enterprise. The concept of a unified data platform, of course, has been the ideal for decades, but has never reached fruition because technologies keep evolving at a blistering rate. Data warehouses arose in the 1990s with this goal in mind, but the movement to real-time analytics caused organizations to seek new approaches. However, this goal has never been so urgent as now, with companies seeking to compete in a fierce global economy with AI. This is where data warehouses, data lakes, and data lakehouses need to be implemented within a service layer that touches all corners of enterprises. Next-generation data architectures need to be able to handle structured, unstructured, and semi-structured data in a consistent way.
• Keep data clean and updated. Data quality has always been job number one for data teams, and the need for this now is urgent. It is the data that is employed to train and refresh GenAI-powered applications. Data needs to be deduped and as timely and accurate as possible.
• Empower data analytics democracy. Again, the concept of data democracy has been with us for some time—and companies are still working on opening data resources to all employees who need it—not just developers or software engineers. Businesses need to be able to quickly reposition applications and insights to the needs in front of them.
• Keep data and applications open. Data generated within and outside the organization needs to be accessible and ready for use by all applications, even if they have not been implemented yet. Data should be maintained infinitely in “raw” form, so it can be quickly deployed or transformed into appropriate applications. In addition, keep data storage separate from compute resources. While often an afterthought, data retention and storage are essential to the functioning of an AI- or data-driven enterprise.
• Evolve to a data-ready data architecture. Today’s and tomorrow’s data architectures need to be designed to be proactive, not reactive. Business priorities are evolving on an almost daily basis and are often upended by events or unexpected opportunities or problems. Decision makers need to be able to quickly assess the status of a customer, or to address problems that may arise. To evolve from a system built on established data warehouses or data lakes to a modern, data-ready architecture, seek to develop a data lakehouse, ensuring that a single copy of the data is accessible to all relevant applications—AI, analytics, business intelligence, and so on. This is an environment in which streaming and change data capture will function close to real-time needs than extract, transform, and load processes. Such architecture requires an integrated, AI-enabling stack that includes flexible storage, unified access, and intelligent processing capabilities.
• Focus on talent and culture. Data in the enterprise has become everyone’s business, and with the democratization of data analytics, there is a need for a supportive culture. A well-functioning culture relies on a commitment to such democratization, as well as a healthy blend of skills, from SQL to large language model development.
• Governance. A well-designed modern, highly trusted data architecture with advanced analytics requires policies, standards, and processes. Data foundations—data warehouses, data lakes, and, ultimately, data lakehouses—have an important role to play in the future of data analytics. This may require rethinking and redesigning data environments and architectures, ensuring information is always fresh and consistently available to the enterprise.