What is DataOps ?

DataOps is a collaborative data management methodology aimed at improving communication, integration and automation of data flows between data managers and consumers within an organization.

DataOps shares the same principles and applies them to data processing to facilitate and accelerate the delivery of data analysis. In practice, it relies on the combination of teams, tools and unique processes to provide agility, orchestration and control throughout the projects. A concept that can be revolutionary if properly adopted within organizations.

In DataOps, several stages follow one another to automate the design, deployment and management of data delivery flows. This must be done with the appropriate levels of governance and metadata in order to improve the use of the data and the value it generates in a dynamic environment.

At the heart of this process is a data pipeline, which designates the succession of stages through which data passes during a data project. It begins with the extraction of data from a variety of sources, through to their exposure or visualization for business use.

DataOps orchestrates this pipeline and automates it to ensure it is scaled up to production, through various CI / CD practices. This complete process can be represented by the succession of three loops, where data models are promoted between environments at the end of each stage, increasing in volume as new data is introduced into the pipeline. 

Source : Dataops.rocks