Skip to content

Commit b183d2c

Browse files
korawicakorawica
committed
📄 docs: update index page.
1 parent ba553d8 commit b183d2c

1 file changed

Lines changed: 39 additions & 35 deletions

File tree

‎docs/index.md‎

Lines changed: 39 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -16,20 +16,21 @@ This project will deliver all Practice and Knowledge in **Data Developer and Eng
1616

1717
## :material-arrow-down-right: Getting Started
1818

19-
:material-page-last: First, ^^**Data Engineering** is a critical part of the
20-
==Data Lifecycle== that enables organizations to manage and process large volumes
21-
of data efficiently and reliably^^[^3].
19+
     :material-page-last: First, ^^**Data Engineering** is a
20+
crucial part of the ==Data Lifecycle==, enabling organizations to process and manage
21+
large volumes of data efficiently, reliably, and at scale.^^[^3]
2222

23-
By these concepts, **Data Engineer** should design and implement **Data Pipeline**
24-
and **Data Management Strategy** that meet the requirements and KPI of their
25-
organizations and ensure that your data was managed _Consistently_ and _Reliably_.
23+
In this context, a **Data Engineer** is responsible for designing and implementing
24+
**Data Pipelines** and **Data Management Strategies** that align with organizational
25+
requirements and KPIs, ensuring data is handled _consistently_ and _reliably_.
2626

27-
!!! quote "What is DE do?"
27+
!!! quote "What does a Data Engineer do?"
2828

29-
**Data Engineer** is who able to ==_**Develop**_, _**Implement**_, _**Operate**_,
30-
and _**Maintain**_== any tools on the current **Data Infrastructure** that
31-
your organization use, either On-premises or Cloud providers, comprising databases,
32-
storages, compute engines, and pipelines.[^1]
29+
A **Data Engineer** is someone who can ==_**Develop**_, _**Implement**_,
30+
_**Operate**_, and _**Maintain**_== the tools and systems that form the
31+
organization’s **Data Infrastructure**—whether on-premises or in the cloud.
32+
This infrastructure includes databases, storage systems, compute engines,
33+
and data pipelines that power data-driven operations.[^1]
3334

3435
<figure markdown="span">
3536
![Life Cycle of Data Engineering](img/life-cycle-of-data-engineering.png){ loading=lazy width="650" }
@@ -39,38 +40,41 @@ organizations and ensure that your data was managed _Consistently_ and _Reliably
3940
!!! quote "Fundamentals of Data Engineering"
4041

4142
**Data Engineering** is the development, implementation, and maintenance of
42-
systems and processes that take in raw data and produce high-quality, consistent
43-
information that supports downstream use cases, such as analysis and machine
44-
learning.
43+
systems and processes that transform raw data into high-quality, consistent
44+
information. This information then powers downstream use cases such as
45+
analytics and machine learning.
4546

46-
**Data engineering** is the intersection of security, data management, DataOps,
47-
data architecture, orchestration, and software engineering.
47+
At its core, **data engineering** sits at the intersection of security,
48+
data management, DataOps, data architecture, orchestration, and software
49+
engineering.
4850

49-
A **Data Engineer** manages the ^^Data Engineering Lifecycle^^, beginning with
50-
getting data from source systems and ending with serving data for use cases,
51-
such as analysis or machine learning.
51+
A **Data Engineer** manages the ^^Data Engineering Lifecycle^^, starting
52+
with ingesting data from source systems and ending with serving it for
53+
consumption—whether for reporting, analytics, or machine learning.
5254

53-
— Joe Reis and Matt Housley in [Fundamentals of Data Engineering](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/)
55+
— Joe Reis and Matt Housley, [*Fundamentals of Data Engineering*](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/)
5456

55-
You will see that stages of the cycle include _Data Ingestion_, _Data Transformation_,
56-
_Data Serving_, and _Data Storage_ components.
57+
The stages of this lifecycle typically include **Data Ingestion**,
58+
**Data Transformation**, **Data Storage**, and **Data Serving**.
5759

58-
| Best practice | Importance |
59-
|-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|
60-
| Proactive data monitoring | Regularly checks datasets for anomalies to maintain data integrity. This includes identifying missing, duplicate, or inconsistent data entries. |
61-
| Schema drift management | Detects and addresses changes in data structure, ensuring compatibility and reducing data pipeline breaks. |
62-
| Continuous documentation | Manages descriptive information about data, aiding in discoverability and comprehension. |
63-
| Data security measures | Controls and monitors access to data sources, enhancing security and compliance. |
64-
| Version control and backups | Tracks change to datasets over time, aiding in reproducibility and audit trails. |
60+
| Best practice | Why it matters |
61+
|-----------------------------|----------------------------------------------------------------------------------------------------------------|
62+
| Proactive data monitoring | Detects anomalies and ensures data integrity by flagging missing, duplicate, or inconsistent records. |
63+
| Schema drift management | Handles structural changes in data to prevent pipeline failures and maintain compatibility. |
64+
| Continuous documentation | Improves discoverability and understanding by capturing descriptive information about data assets. |
65+
| Data security measures | Protects sensitive information by controlling access and enforcing compliance standards. |
66+
| Version control and backups | Preserves historical states of data for reproducibility, auditing, and recovery in case of corruption or loss. |
6567

66-
:material-page-last: Since I started on this role, I got the idea about the future
67-
of my responsibilities. I know the Data Engineering tools shifts so fast because
68-
the last three year I started with the Map-Reduce processing on the **Hadoop HDFS**
69-
but nowadays, it changes to In-Memory processing like **Impala** or **Spark**.
70-
The knowledge I gained from Map-Reduce will be wasted :boom:.
68+
:material-page-last: As I’ve grown into this role, I’ve realized how quickly the
69+
**data engineering landscape evolves**. Just three years ago, I was working with
70+
MapReduce on **Hadoop HDFS**. Today, the focus has shifted toward in-memory
71+
processing engines like **Impala** and **Apache Spark**.
7172

72-
![The 2023 MAD (ML/AI/DATA) Landscape](img/mad-data-landscape.png){ loading=lazy width="370" align=right }
73+
The key takeaway? While tools may come and go, the fundamental skills and
74+
concepts—such as distributed processing, data modeling, and lifecycle management—
75+
remain invaluable :boom:.
7376

77+
![The 2023 MAD (ML/AI/DATA) Landscape](img/mad-data-landscape.png){ loading=lazy width="370" align=right }
7478
The right picture, the [**2023 MAD (ML/AI/Data) Landscape** :material-land-plots:](https://mad.firstmark.com/)[^2],
7579
that show about how many possibility tools that able to use on your project.
7680
It has many area that you should to choose which one that match with the current

0 commit comments

Comments
 (0)