DataOps – removing complexity that’s preventing innovation

For the NonStop community the message coming from HPE is focused on the IT transformation that ultimately leads us into an age of insight. However, this transformative journey is proving difficult for many enterprises. Not for the faint of heart is contemplating the cultural changes involved in ownership of processes and data or coming to terms with just how many data sources may exist across the enterprise. If there was ever a greater need to be agile then data brings such a requirement into focus. And yet, data is key to the story of IT transformations that have been the subject of so many articles, blog posts and commentaries.

In his May 4, 2021, post to the HPE Blog, It’s time for a new data vision – and a new architecture David Wang, HPE’s Director, Product Marketing, HPE Storage wrote:

“It comes as no surprise that data is the key to transformation. But what may be surprising is that close to 100% of organizations recently surveyed by ESG see the complexity of storage and data management as impeding transformation. This blog uncovers the challenges and unveils the new data experience enabled by HPE’s data vision and architecture.

“The impact of complexity is also far broader than IT. Data innovators — those who turn bits-and-bytes into new apps and insights — can’t get access to data fast enough. Manual processes inhibit data utilization and slow down time-to-value. Data managers are challenged to streamline data access, while at the same time protecting their crown jewels within an ever-increasing threat landscape.”

Wang then offers up a solution, DataOps:

“Unified DataOps is not just a desire, but a reality that’s possible with an original approach and new thinking. It embraces data, cloud, and AI to reimagine the data experience through data-centric policies and automation, cloud-native control and operations, and AI-driven insights and intelligence.”

DataOps has also been the subject of two recent posts to the Striim’s blog. As with the observations by HPE’s Wang, it is Striim’s understanding that in the drive to achieve greater business returns as a result of entering the age of insight it is a call for better ways to ensure data access isn’t limited by where the data is stored or how it arrived there. If complexity truly is standing in the way of innovation then it’s time to rethink how we deal with data. Leveraging much that has been learnt from the pursuit of DevOps, many of the same principles are now showing up within DataOps.

In the post of January 28, 2022, to the Striim blog, DataOps vs DevOps: An Overview of the Twins of Digital Transformation John Kutay and Mariana Park recoginze the more obvious overlapping principles between DataOps and DevOps:

“DevOps and DataOps share some underlying principles. They require a cultural shift from isolation to collaboration. They both depend heavily on tools and technologies for process automation, and they both employ agile methodologies that support incremental delivery. As such, both DevOps and DataOps represent a sweeping change to the core areas of culture, process, and technology.”

However, this is just the beginning of the conversation. Fueling the discussion is the almost universal lack of success among enterprises pursuing analytics. No discussion on data can stray far from the business need to subject data to analysis in order to derive meaningful and actionable insights. It’s simply the complexity of dealing with many different data types, structures and even the anomalies that arise through different interpretations of data that for all sakes and purposes look similar.

As Kutay and Park note all too well:

“According to a recent study by VentureBeat, lack of data access is one of the reasons why 87% of data science projects never make it to production. For instance, data consumers like data scientists and analysts responsible for utilizing data to generate insights depend on data operators such as database administrators and data engineers to provide data access and infrastructure.”

Just as DevOps brings together the formerly disparate organizations of development and operations, DataOps bridges the gulf that oftentimes opens up between data scientists and data engineers. Left siloed, these groups may be idle waiting on the outcomes of the other whereas at other times, what is needed may only be loosely defined leading to further ambiguity of purpose. Why would you need that data is just as much a question as to what do you want me to do with this?

In the later post of February 4, 2022, to the Striim blog, What Is DataOps and How Can It Add Value to Your Organization? Kutay brings clarity to the purpose of DataOps:

“DataOps is an umbrella term that can include processes (e.g., data ingestion), practices (e.g., automation of data processes), frameworks (e.g., enabling technologies like AI), and technologies (e.g., a data pipeline tool) that help organizations to plan, build, and manage distributed and complex data architectures.

“This includes management, communication, integration and development of data analytics solutions, such as dashboards, reports, machine learning models, and self-service analytics.

“DataOps aims to eliminate silos between data, software development, and DevOps teams. It encourages line-of-business stakeholders to coordinate with data analysts, data scientists, and data engineers.”

Returning to the application of DataOps among IT communities including the NonStop community, where managing data from multiple different sources is required, then “DataOps can use data analytics pipelines to consolidate data into a data warehouse or any other storage medium and perform complex data transformations to provide analytics via graphs and charts.” By implication and with experience gained to date, Striim has already met the needs of the NonStop community as it is the Striim real time data streaming platform that supports NonStop participation as a source within DataOps.

In case you were wondering about the openness of this approach perhaps it’s best to leave to last how both DataOps and DevOps rely on tools already available when it comes to execution and where many of them from open source projects. . Some of these are already familiar to the NonStop community given how DevOps for NonStop is now more widely used to bring new products to market on NonStop.  For instance, when it comes to DevOps, tools such as Jenkins and Ansible help automate the entire application lifecycle from development to deployment. And then, in the case of DataOps, platforms like Apache Airflow and DataKitchen help organizations control their data pipelines from data orchestration to deployment. This was highlighted in the post by Kutay and  Park who then posted, “data integration tools like Striim automate data integration from multiple sources, helping organizations quickly access their data.”

According to our Striim team and based on their own conversations to date, whenever developing your own DataOps architecture you need a reliable set of tools that can help you improve your data flows, especially when it comes to key aspects of DataOps, like data ingestion, data pipelines, data integration and the use of AI in analytics – you also need Striim. With Striim you have a unified real-time data integration and streaming platform that integrates with over 100 data sources and targets, including databases, message queues, log files, data lakes, and IoT. Striim ensures the continuous flow of data with intelligent data pipelines that span public and private clouds.

Should you have any questions about the Striim’s ability to bring value to the fresh data created on NonStop don’t hesitate to reach out to the Striim team. We would be only too happy to hear from you, anytime and all the time. To learn even more about how you can implement DataOps with Striim, get a free demo today.

Ferhat Hatay, Ph.D.
Sr. Director of Partnerships and Alliances, Striim, Inc.