What is Meaning Data Engineering?
Hello Everyone,
I’m starting a new series. Today i want to talk about Data Enginnering. During the Data Engineering Botcamp, i would like to share my experiences with you. Let’s start it.
What is the meaning of Data Engineering?
“Data Engineering is the development, implementation and maintenance of systems and processes that take in raw data and produce high quality, consistent informations that supports downstream use cases, such as analysis and machine learning” say O’REILLY-Fundametals of Data Enginnering
Actually Data Engineering is the intersection of security, data management, DataOps, data architecture, orchestration and software engineering.
Maybe we can explain this with an example:
“We can think of Data Engineering as drilling crude oil out of underseas, cleaning up the extracted crude oil, and turning it into fuel oil for cars. Crude oil is cleaned for fuel oil, but it’s not just used for external customers, it’s used in the refinery, as in data engineering. Data Engineering also has both internal and external customers. Its internal customers are Data Scinetist, Data Analysts within the organization.”
Now let’s look at the Life Cycle of Data Engineering. Firstly we see the two way;
Data Engineering Lifecycle
Undercurrents.
What do they mean?
Data Engineering Lifecycle
Data Engineering Lifecycle include 5 main topic;
Generation/Sources
Ingestion
Storage
Transformation
Serving
Generation/ Source Systems
We will cover this in the generation section, how and where crude oil is found, which we call data source. First, a Data Engineer consumes data in a source system but does not control the source system itself. The Data Engineer needs to establish a working understanding of the way source systems work, the way they produce data, and the variety of data they produce.
Engineers also need to have open communication with system owners about changes that could disrupt the process and analytics. Resources used in Data Engineering;
Data source can be a file or a non-structural file (csv, json, avro, binary etc.).
An API can be your data source. It started to become very popular around this time.
Relational databases.
NoSQL
Database logs.
Streamings.
Storage
Of course, after you get the raw data from the source, you need a place to store the data. Choosing a right storage solution is one of the crucial and also complex stages of the data lifecycle for the rest of the data lifecycle. Data architectures in the cloud often leverage a variety of storage solutions.
While storage is one phase of the data engineering lifecycle, it often touches on other phases such as ingestion, transformation, and delivery.
Data Storage locations;
Object storage(S3, Azure, Google, Ozone, MinlO etc.)
Hadoop HDFS
Streaming
RDBMS
NoSQL
You can also use databases as Data Warehouse, Data Lake, Lake House.
Transformation
We’ve captured and stored the data, but what next? In the next stage, it is necessary to deal with data. Like the distillation process in an oil refinery. This means that the data must be transformed from its original form into something useful for subsequent use cases.
If we present raw data to customers without processing it, the data will remain dormant and not in a usable form for reports, analytics, or machine learning. Typically, the transformation phase is where the data begins to create value to the raw data for downstream user consumption. Basic transformations transform data into the correct data types immediately upon ingestion, placing records in optimal formats and removing bad ones. Later stages of the transformation can transform the data schema and apply what we call 3NF normalization.
Querying the data to be transformed is also an important point here. With what tools can we question this?
- Data Definition Language — DDL
- Data manipulating Language — DML
- Data Control Language — DCL
-Transaction Control Language — TCL
“Every data engineer needs to know SQL very well.” ~ Erkan Şirin
Serving Data
We have reached the final stage of the data engineering lifecycle. Now that the data has been captured in its raw form, stored, and transformed into consistent and useful structures, it is time to extract value from your data.
Get Fatma Nur Kurbanoğlu’s stories in your inbox
Join Medium for free to get updates from this writer.
Data has value when used for practical purposes. Data that is not consumed or queried is useless. It will be useless if companies use data in vanity projects. Many companies pursued vanity projects and collected huge data that was never consumed in data lakes. This meaningless storage of data means great loss for the company in terms of cost, time and labor.
Users to whom the data is presented;
Analytics, BI,Repoting
Data Science Team
Model Predictions
Where data is presented;
RDBMS
Streaming system
NoSQLDBs
APIs
Security
Security should be the first priority of both the company and the Data Engineers, and ignoring it actually means compromising very important information of the company in the long run. So security is the first undercurrent. Data Engineers must understand both data and access security by being careful about this and applying the principle of least privilege.
The principle of least privilege means granting a user or system access to only necessary data and resources rather than granting full privileges to perform an intended function. A common anti-pattern we see from data engineers with little security experience and knowledge is to give administrative access to all users. Awareness of this should be increased as much as possible.
Expected features of served data;
Trust
Accuracy
Right Format
Right shape, schema
Right time (SLA, Latency)
For example, your manager asked you to share data with themself, what would you do in this case?
Data Management
As formal as data management may sound, “Old school” data management practices are making their way into data and Machine Learning Engineering. What is old is new again, only transformation is required. Data management has been talked about all this time, but until recently it had not received much attention in Data Engineering. Data tools are becoming simpler to use and data engineers are making faster progress in managing the process. Data best practices once reserved for large companies can now be integrated into companies of all maturity levels.
Another important detail here is that the Data Engineer also gains competencies during the integration process. Just implementation is not enough, development is also important.
Other important concepts that Data Engineers should be interested in and familiar with are;
Data governance
Data observability and accountability
Data lineage
Data Modelling
Data integrations and interoperability
Data Catalog, Metadata
Discoverability
Data Architecture
Data architecture is necessary to reflect the current and future state of the data systems that support the organization’s long-term data needs and, accordingly, its strategy. Since an organization’s data needs will change rapidly and new tools will emerge almost every day, Data Engineers must design the data architecture specifically for the organization.
Firstly Data Engineer must understand the needs of the business and gather its requirements. A Data Engineer must then map these requirements to design new architectures to filter and present data to achieve optimality in cost and operational efficiency. This means mastering the trade-offs involved in source systems, design patterns, technologies, and tools in the data lifecycle.
Orchestration
Orchestration is as important as the other steps because we see it as truly the center of gravity of the software development lifecycle when it comes to data.
Orchestration is not only a centrally managed DataOps process, but also a critical part of the engineering and deployment flow for data businesses. So what is orchestration?
Orchestration is the coordination of data management so that it is carried out as quickly and efficiently as possible at a planned pace appropriate to the process. For example, orchestration is needed to manage interdependent jobs in the data flow. For example, people often use orchestration tools like Apache Airflow as timers in the data transformation process.
Software Engineering
Although data engineers do work such as extracting, cleaning and managing data, they actually go through the processes of the Software Engineer. In the early days of Data Engineering at the turn of the 21st century, data engineers worked at a lowly level.
With the popularity of big data after 2015, data engineering has actually gained importance. This importance actually emerges when Data Engineers use their software engineer skills. Data Engineers need to know many software programs well. Of course, these software have become easier to use over time. For example, some cloud data warehouses support powerful transformations using SQL semantics; Tools like Spark have become more user-friendly, moving away from low-level coding details to easy-to-use data frameworks.
We have covered Data Engineering under important headings. In fact, this process does not only consist of these headings. It contains many more important details. See you under new headings with future events and news.
Source
VBO - Data Engineering Bootcamp
O’Reilly- Fundamentals of Data Engineering
https://www.shiksha.com/online-courses/what-is-data-engineer-dg437
