Whether it’s a financial services company looking to build a personalized virtual assistant or an insurance company in need of ML models that can identify potential fraud, artificial intelligence (AI) is poised to transform virtually any industry. In fact, a recent survey from Cloudera found that 88% of IT leaders said their organization is currently using AI in some way.
AI’s ability to automate repetitive tasks leads to significant time savings on processes related to content creation, data analysis and customer experience, freeing employees to work on more complex, creative issues. But adoption isn’t always easy. The road to achieving AI at scale is paved with numerous challenges: data quality and availability, implementation and integration with existing systems.
To overcome these challenges and successfully scale AI enterprise-wide, organizations must create a modern data architecture that leverages a mix of technologies, capabilities and approaches, including data lakehouses, data fabric and data mesh.
Barriers to AI at scale
Despite so many organizations investing in AI, the reality is that the value derived from these solutions is limited. The factors that influence this success vary and are not limited to purely technical limitations. There is also an element of employee involvement that can cause AI adoption to lag or even fail to materialize. Cloudera’s survey found that 39% of IT leaders who have already implemented AI in some way said that only some or almost none of their employees are currently using any form of AI tools. So even when projects are implemented on a large scale, in more than a third of cases employees simply don’t use them.
Another challenge here arises from the existing architecture within these organizations. While they are implementing AI, the data architecture they currently have is not equipped or able to scale with the massive amounts of data that power AI and analytics. This requires greater flexibility in systems to better manage data storage and ensure quality is maintained when data is fed into new AI models.
As data moves between environments, is fed into ML models, or is used in advanced analytics, considerations around issues like security and compliance are a top priority for many. In fact, of leaders surveyed, 74% identified security and compliance risks surrounding AI as one of the biggest barriers to adoption. These IT leaders face a simultaneous need for a data architecture that can support rapid AI scaling and prepare users for an evolving regulatory landscape.
This challenge is especially prevalent in the financial services industry with the advent of new regulations and policies, such as the Digital Operational Resilience Act (DORA), which imposes strict ICT risk management and security guidelines for companies in the European Union. Rapidly changing regulatory requirements require organizations to ensure they have full control and visibility of their data, which requires a modern approach to data architecture.
Building a strong, modern foundation
But what does a modern data architecture entail? While every platform is different, there are three key elements organizations should pay attention to when it comes to data lakehouses, data mesh, and data fabric. Each of these is responsible for a modern data architecture approach to data management that can help meet security requirements, break barriers such as data silos, and achieve stronger results from AI adoption across the enterprise.
Before we go any further, let’s quickly define what we mean by each of these terms. A data mesh is a set of best practices for managing data in a decentralized organization, allowing easy sharing of data products and a self-service approach to data management. A data fabric is a set of collaborating technologies that help create a unified view of data from disparate systems and services across the organization. Then there is the data lakehouse: an analytics system that allows data to be processed, analyzed and stored in both structured and unstructured forms.
Because AI models require vast amounts of structured and unstructured data for training, data lakehouses provide a highly flexible approach that is ideally suited to support them at scale. A data mesh provides greater ownership and governance to the IT team members who work closest to the data in question. Datafabric provides an effective means to unify data architecture, making data seamlessly connected and accessible using a single layer of abstraction.
These benefits are widely understood, with 67% of IT leaders surveyed by Cloudera noting that data lakehouses reduce the complexity of data pipelines. Likewise, both data mesh and data fabric have received a lot of attention among IT leaders in recent years, with 54% and 48% of respondents, respectively, stating they plan to have these components installed by the end of 2024.
Whatever the end goal of an organization’s AI adoption, its success can be traced back to the fundamental elements of the IT and data architecture that support it. And the results for those who embrace a modern data architecture speak for themselves.
For example, Cloudera customer OCBC Bank used Cloudera machine learning and a powerful data lakehouse to develop personalized recommendations and insights that can be pushed to customers via the bank’s mobile app. This was made possible by the hybrid data platform that OCBC Bank used, which allowed them to accelerate AI implementation and deliver a great return on their investment.
With a strong foundation of modern data architecture, IT leaders can advance AI initiatives, scale them over time, and drive greater value for their business.
To learn more about how enterprises can prepare their environments for AI, click here.
