Data and AI Transformation Efforts Progress Slowly for Many Leading Companies

This blog is an abridged version of Randy Bean’s Forbes article, first published on September 14. The blog is published here with the author’s permission.

On September 15, 2008, Lehman Brothers, the 4th largest investment bank in the United States, filed for bankruptcy. The bankruptcy triggered a 4.5% drop in the Dow Jones Industrial Average, the largest drop since the attacks of September 11, 2001. In the days and weeks ahead, the Federal Reserve assumed control of American International Group (AIG), and Merrill Lynch, then the 3rd largest U.S. investment bank, was acquired by Bank of America to avoid bankruptcy.

In the wake of the financial crisis of 2008-2009, banks and other large companies undertook measures to decrease their risk and ensure better financial controls. One resulting action was the establishment of the role of Chief Data Officer (CDO), or Chief Data and Analytics Officer (CDAO) as the role has come to be known at many organizations. Although some early innovators had dabbled with the role prior to 2008, the CDO/CDAO role in its present form was rooted in response to the need by financial services and other leading companies to better manage and understand their data, initially to comply with regulatory and risk management requirements. Today, 15 years after the financial crisis, 82.6% of leading companies have appointed a CDO/CDAO.

The CDO/CDAO role has evolved over time as companies have progressively transitioned from defensive functions related to risk and regulatory oversight to offense-driven efforts like business growth, customer acquisition and retention, cross-selling, and online servicing. Activities requiring advanced analytics, machine learning, and artificial intelligence (AI) have been incorporated into the CDO/CDAO mandate. The CDO/CDAO function has been transformed at many companies from being a technology role which reported to the Chief Information Officer (CIO) to a business function reporting to senior business leaders. Currently, 43.3% of CDO/CDAOs report to the COO or CEO.

Though the CDO/CDAO role is now ubiquitous, CDO/CDAOs continue to face challenges. Many CDO/CDAOs have been remarkably successful and happy during their tenures. However, turnover in the CDO/CDAO role is high, and progress has been slow to materialize for most companies – only 39.5% are managing data as a business asset, 23.9% have created a data-driven organization, and 20.6% have established a data culture. Just 35.5% believe that the CDO/CDAO role is successful and well-established, and 40.5% say the role is well understood in their company. Fifteen years after its establishment, the CDO/CDAO role is a work in progress.

Why haven’t data and AI transformation efforts progressed faster for many leading companies? Business transformation of any kind is never easy, and this is especially true for legacy companies, which constitute the core of the Fortune 500. In a recent article, The Economist presents a hypothesis, painting a legacy perspective of the mainstream business economy. It notes that only 52 Fortune 500 companies were founded after 1990, so that nearly 90% of the Fortune 500 is comprised of legacy firms. Since 1990, the average age of a Fortune 500 company increased from 75 to 90 years. The average age of Fortune 500 banks is 138 years old.

While innovation leaders Alphabet (Google), Amazon, Apple, Meta (Facebook), and Microsoft invested a combined $200B in R&D last year, this represents 30% of the total of R&D business investment, suggesting that investment in innovation remains heavily concentrated. The Economist concludes, “Inertia has slowed the pace of competitive upheaval in many industries, buying time for incumbents to adapt to digital technologies. The digital revolution has not been all that revolutionary in some parts of the economy.” For legacy companies, the gradual pace of transformation and change is reflected in the slow progress in building data-driven organizations and data culture.

Current economic uncertainty has renewed pressure on companies to deliver measurable business value from their data and AI investments. Some companies are engaged in reevaluating and reassessing the CDO/CDAO role – the need, its structure, its reporting relationship, and its mandate. These companies recognize that the CDO/CDAO role remains a challenge for both the company as well as for the incumbents who occupy the role — difficult to be successful in, stretched across multiple constituencies, often misunderstood, a periodic target of resentment from peer executives wary of the new kid on the block, new to the C-suite, and too often set up to fail with expectations nearly impossible to meet.

Where do we go from here? Are companies ready to name an analytics leader or an AI visionary to lead the CDO/CDAO function? Do companies want to see responsibility for AI to sit within the CDO/CDAO organization? Some companies are turning to accomplished and savvy business leaders to fill the CDO/CDAO role, seeking someone who is highly conversant in communications, storytelling, building alliances, muscling support, selling ideas, operating in the C-suite, and driving results. These companies are asking themselves whether they want to continue to elevate subject matter experts in disciplines such as data governance and data management, as has been the blueprint for many organizations in the past, or whether they are now looking for something different.

While the progress of data and AI adoption has been slow, the need for data and AI leadership is greater than ever. The volume and variety of data available for capture and analysis continues to grow. Opportunities to use data and AI to differentiate companies and personalize the customer experience are increasing. Generative AI poses opportunities and challenges. Will the gradual pace of business transformation and change continue to be a barrier to data and AI progress, as The Economist says?

History teaches us that change is rarely easy. Transformation plays out over a period of years and decades. While progress in data and AI has been slower than might have been hoped for, business understanding and technical proficiency in data and AI will only grow. Inevitably, the demand for data and AI leadership will increase. Progress will require fresh thinking, new perspectives, determined patience, and sustained commitment. Fifteen years after the 2008 financial crisis, we are still learning how to manage data and AI.