After a decade of big data investment and hype, the term DataOps is gaining the fame and attention it deserves. This seemed like a perfect opportunity to take a step back to the basics. So we’re taking a look at what DataOps is, where it came from, and what it means to apply DataOps principles.
The origin of DataOps
The term originated in 2014, when Lenny Liebmanna first introduced it, and slowly got more and more popular as a framework to make big data projects more successful. Its name comes from ‘Data Operations’ and thus logically refers to the best practices of data-related operations. As a methodology, it drew heavy inspiration from DevOps, a framework that enables fast development cycles. It combines this framework with teachings from lean manufacturing methods and agile principles for good measure.
So what do we get from this methodological framework cocktail?
A framework that streamlines the data pipeline, incentivises agile behaviours, focuses on efficient collaboration, and provides automation to a smooth flow from raw data to insight.
The best way to define DataOps is as a collaborative data management practice. The focus lies on efficient communication, maintaining oversight of data complexity, and the automation of data flows. DataOps as a discipline, not only applies to data consumers and producers, but also data managers and leaders across an organisation. The goal of DataOps is to set a higher standard to how people view, use, and collaborate with data, and how it is used in the organisation. It does this by defining a set of practices that assist people in operationalising data in a timely, consistent and repeatable manner.
Does this still sound a bit vague to you? That’s because it’s an open discipline that’s still a work in progress. We’ve already come a long way since the term was first introduced, with DataOps even having its own manifesto. Yet what it means to apply its principles is heavily dependent on the use-case, and every general list should be taken with a grain of salt.
After all, DataOps is not a magic solution that solves all problems, and neither will it result in the same processes in every use case. It’s a collection of best practices that grew out to become a framework for efficient data operations.
What makes something DataOps?
Now that we’ve got a general idea of what DataOps tries to achieve, next blog in this series we’ll look at how you can apply the DataOps discipline, and even further in the line we’ll examine how DataOps inspired our data orchestration platform.
There’s are no ultimate DataOps guide, tools or platforms available, as DataOps is case-specific and the tools or platforms can only support you to apply DataOps principles. So you should best take every general list with a grain of salt.
Still, we hope you like and support this guide series, from defining DataOps to applying it with a DataOps supported platform and tools.