Organisations expand data management participation to support data-driven decision-making at scale
Have you thought about the status of your data? Do you know where it lives, who is using it and how often? Do people in your organisation know which data is appropriate to use for making decisions and how to access it?
Data-driven leaders are differentiating their organisations with new solutions for integrating their distributed data pipelines – the roles and processes for how data is prepared, curated and shared across the business are shifting alongside the evolution already happening within data technologies. IT should take a page from Ghostbusters: although it’s not advised to cross the streams, sometimes this can solve the biggest, scariest problems. In this case, blurring the lines between IT and business responsibilities around data management, organisations will no longer be limited by functional boundaries, enabling enterprise-wide data integration at scale and empowering people across the organisation with the right data at the right time.
Solving these data integration challenges is imperative for maintaining internal and external compliance, as well as enabling the organisation to get a complete picture of the business, understand customers and find new business opportunities. Many organisations are working to identify, prepare, govern and make widely available the data that most benefits the entire organisation. And where there’s success, data management is changing – beginning with technologies.
Core data management functionalities (such as data profiling, data cataloguing, metadata management and data integration) now appear in many individual data management applications and tools. The distinction among them is getting blurry, driving confusion in the market.
Solutions providers are increasingly incorporating data management capabilities with broader users than just IT in mind. And as functionality becomes more embedded in business users’ workflows – including in analytics platforms – employees will take a more active role in data management responsibilities that were traditionally owned by IT. This is the natural next step in the evolution of self-service in business intelligence: organisations first broadened data access, then enabled deeper exploration and new types of users to author analytics content. Now, some business users are able to get involved with the data itself. At each of these stages, IT learned how to balance governance and self-service so business users could take some of the load off. Crossing these streams will be critical to managing data and analytics as its adoption scales across the enterprise.
Self-service data prep demonstrates this evolution well. Various aspects of the traditional extract, transform and load processes can now be executed in a self-service manner using modern tools that integrate with the analytics workflow. This not only allows for greater ad-hoc discovery, but can serve as a starting point for new use cases to be tested before being scaled to the entire organisation. And it’s a win-win: the business is empowered to take on greater ownership in data management, thereby reducing the (traditionally) lengthy development lifecycle, and IT is freed up to take on the highly specialised work that they are in the best position to do.
Another example of this evolution is the data catalogue – an inventory of data assets that helps define and qualify data while tracking relationships between data sources, content and users. In organisations with distributed accountability for integrating and managing data, a catalogue is important as a central view of what is going on with the company’s data assets. Catalogues can help to more easily discover and promote data, understand its relevance and freshness, and monitor who is using certain assets.
Modern catalogues are surfacing this valuable information and adding business context right in the flow of users’ analyses. So as more data is integrated and becomes broadly available in the organisation, people learn to understand the quality of their data and how to use it while staying within policy guidelines. This is where data literacy is critical – at a minimum, users will learn to interpret data indicators and identify trusted, relevant data. When data users can be their own data stewards, this lessens the burden on IT and helps ensure responsible use when making decisions. Then, more sophisticated users with the right skills might go on to participate in self-service data prep, certify new data sources for the organisation to use or add business context as metadata in a curation process.
We need to help everyone understand the story that we're telling. And one of the ways that we do that is by using a data catalogue. We can start helping our residents and those who are consuming information to understand what we mean when we say those specific words. Words are really important. Definitions are really important.
IT and the business can reach new collaboration and harmony with the lines of functionality and responsibility blurred. With a tailored approach that includes business users and objectives, broader data management initiatives will succeed because IT and the business can share in the efforts to increase visibility, discoverability and trust in their data environment. This also means the organisation is empowered to identify and prioritise the data assets that are most broadly valuable, and better support governed data and analytics at scale.