66.data-analysis-overview.md

Overview of Data Analysis

In today's world, every industry depends more and more on information technology, and a huge amount of data is produced every day. We often feel that there is more and more data, but it becomes harder and harder to find valuable information from it. The information mentioned here can be understood as the result after processing a dataset. It is something extracted from the dataset that can be used to support and guide decision-making. The process of extracting valuable information from raw data is what we call data analysis, and it is an important part of data science.

Definition 1: Data analysis is the science of purposefully collecting, processing, organizing data, and using statistics, mining, and other techniques to explore, analyze, present, and explain data.

Definition 2: Data analysis is the activity of collecting, organizing, and analyzing data, extracting valuable information and insights from it, and using them to support decisions and optimize processes. (GPT-4o)

Definition 3: Data analysis is the process of systematically collecting, organizing, processing, examining, and explaining data, extracting valuable information from it, forming conclusions, and supporting decisions. Its core is to use statistics, algorithms, and logical methods to reveal the patterns, trends, or relationships behind the data. (DeepSeek)

For people who want to do data-analysis work, there are two parts of skills that need to be mastered. One is "data thinking", and the other is "analysis tools", as shown in the picture below.

In the picture above, the analysis-tools part is actually easier to master. Programming languages such as SQL or Python can be handled by most people as long as they go through systematic study and an appropriate amount of practice. Business-intelligence tools such as Power BI and Tableau make it even easier to start, because we can finish data visualization through drag-and-drop operations and then produce business insights on that basis. On the contrary, the data-thinking part is not easy for most beginners to master. For example, many people studied courses such as probability theory and statistics at school, but when they face real business scenarios, they still find it hard to map this knowledge to the business scenario and solve real problems. Also, if we have not mastered basic analysis methods, have not understood common analysis models, and do not have related business knowledge, then even if we get a lot of useful data, we still may feel that we do not know where to start, not to mention producing business insights and discovering business value. So for the data-thinking part, besides systematic study of related knowledge and skills, we also need to keep accumulating and practicing in real business scenarios.

Responsibilities of a Data Analyst

When HR publishes job requirements, positions such as data engineering, data analysis, and data mining are usually all called data-analysis positions. But according to differences in the nature of the work, they can still be divided into data governance, which is more engineering-oriented, business analysis, which is more business-oriented, data mining, which is more algorithm-oriented, data development, which is more application-oriented, and data product manager, which is more product-oriented. When we usually say data analyst, we mainly mean business data analyst. Many data analysts start their career from this role, and this role is also the one with the largest number of openings. Some companies place business data analysts inside specific business departments, such as marketing, operations, or product. Some companies have special data departments, such as a data-analysis team or data-science team. Some companies let data analysts directly serve top-level decision-making, so they belong to the corporate strategy department. Because of this, when you see job titles such as data operations, business analyst, or BI engineer on job websites, you should not feel surprised. Usually, the job description for a business data analyst that we see on job websites is shown below:

1. Be responsible for producing related reports.
2. Build and optimize the metric system.
3. Monitor data fluctuations and anomalies, and find problems.
4. Optimize and drive the business and promote digital operations.
5. Find the possible room for market and product growth.

From the description above, we can see that, as a business data analyst, our work is not to give a simple and shallow conclusion. It is to combine the company's business and complete work such as monitoring data, finding anomalies, finding reasons, and exploring trends. No matter whether you use Python, Excel, Tableau, SPSS, or some other business-intelligence tool, the tool is only a means to reach the goal. Data thinking is the core skill, and starting from real business problems and finally finding the business value in data is the ultimate goal. In many companies, data analyst is only a basic position. A data analyst who is very good at understanding the business can develop toward management positions such as data analysis manager or director of data operations. For a data analyst who is familiar with machine-learning algorithms, they can develop toward data-mining engineer or algorithm expert. Besides needing the matching mathematics and statistics knowledge, these positions also have higher requirements for programming ability than data analysts, and they may also require related experience in big-data storage and processing.

Let me also briefly talk about several other directions. A data-governance position mainly helps a company build a data warehouse or data lake, so that data can be moved from business systems, tracking systems, and logging systems to the data warehouse or data lake, providing infrastructure for later data analysis and mining. Data-governance positions have high requirements for SQL and HiveSQL, need skilled use of ETL tools, and also need a good understanding of the Hadoop ecosystem. As for a data product manager, besides the traditional product-manager skill stack, strong technical ability is also needed. For example, they need to understand common recommendation algorithms and machine-learning models, be able to provide a basis for improving algorithms, and be able to set standards and definitions for tracking points. Although they do not need to be experts in every algorithm, they still need to think from the product angle about the implementation of data models, metrics, algorithms, and so on.

Skill Stack of a Data Analyst

The skill stack of a data analyst also includes hard skills and soft skills. The following is only my understanding of this position, for reference only.

1. Computer science, such as data-analysis tools, programming languages, and databases
2. Mathematics and statistics, such as data thinking and statistical thinking
3. Artificial intelligence, such as machine-learning and deep-learning algorithms
4. Ability to understand the business, such as communication, expression, and experience
5. Ability to summarize and express, such as summary, reporting, and business PPT

Of course, for a beginner, it is impossible to master the content of the whole skill stack in a short time. But as this work goes deeper, you will more or less touch all the things mentioned above. Everyone can deepen one or some of these skills according to the actual needs of the work.

General Process of Data Analysis

When we mention data analysis, many times we may actually mean data analysis in a narrow sense. The main goal of this kind of data analysis is to generate visual reports and discover business problems through these reports. This kind of work is usually lagging. Data analysis in a broad sense also includes the data-mining part. It not only uses data to monitor and analyze the business, but also uses machine-learning algorithms to find the knowledge hidden behind the data, and then uses this knowledge to support future decisions, so it has some forward-looking nature.

Basic data-analysis work usually includes the following parts, although there will be slight differences because of different industries and job contents.

1. Determine the goal, input: understand the business and determine the definition of the metrics.
2. Get data: data warehouse, spreadsheets, third-party APIs, web crawlers, open datasets, and so on.
3. Clean data: handle missing values, duplicated values, outliers, and other preprocessing tasks such as formatting, discretization, and binarization.
4. Pivot data: sorting, statistics, group aggregation, crosstabs, pivot tables, and so on.
5. Present data, output: data visualization and publishing work results, such as a data-analysis report.
6. Analyze and gain insight, follow-up: explain changes in data and propose matching plans.

Deeper data-mining work usually includes the following parts, although there will also be slight differences because of different industries and job contents.

1. Determine the goal, input: understand the business and make the mining target clear.
2. Data preparation: data collection, data description, data exploration, quality judgment, and so on.
3. Data processing: extract data, clean data, transform data, special encoding, dimensionality reduction, feature selection, and so on.
4. Data modeling: model comparison, model selection, and algorithm application.
5. Model evaluation: cross-validation, parameter tuning, and result evaluation.
6. Model deployment, output: putting the model into use, improving the business, monitoring operations, and writing reports.

Libraries Related to Data Analysis

Using Python to do data-analysis-related work is a very good choice. First, Python is very easy to start with. Also, in the whole Python ecosystem, there are many mature software packages and tool libraries for data science. Different from some other programming languages for data science, such as Julia and R, Python can be used not only for data science, but also for many other things.

The classic three great tools

1. NumPy: supports common array and matrix operations. Through the ndarray class, it wraps multidimensional arrays and provides methods and functions to operate on these arrays. Because NumPy has built-in parallel-computing ability, when using a multi-core CPU, NumPy automatically does parallel computation.
2. Pandas: the core of pandas is its special data structures DataFrame and Series, and this makes pandas able to handle tables and time series that contain different kinds of data, something NumPy's ndarray cannot do. By using pandas, we can easily and smoothly load many forms of data, and then do operations such as slicing, chunking, reshaping, cleaning, aggregation, and presentation.
3. Matplotlib: matplotlib is a library that contains many plotting modules and can create high-quality charts according to the data we provide. In addition, matplotlib also provides the pylab module, and this module contains many plotting components like MATLAB.

Other related libraries

1. SciPy: improves NumPy and wraps a large number of scientific-computing algorithms, including linear algebra, statistical tests, sparse matrices, signal and image processing, optimization problems, fast Fourier transform, and so on.
2. Polars: a high-performance data-analysis library intended to provide faster data operations than pandas. It supports large-scale data processing and can use multi-core CPUs to speed up computation. It can be used to replace pandas when handling large-scale datasets.
3. Seaborn: Seaborn is a graph visualization tool based on matplotlib. Although beautiful statistical charts can be made directly with matplotlib, overall it is still not simple and convenient enough. Seaborn is equal to a wrapper over matplotlib, letting users make all kinds of attractive statistical charts in a more concise and effective way.
4. Scikit-learn: originally part of SciPy. It provides a large number of tools that may be used in machine learning, including data preprocessing, supervised learning, meaning classification and regression, unsupervised learning, meaning clustering, model selection, cross-validation, and so on.
5. Statsmodels: a library that contains classic statistics and econometrics algorithms, helping users complete tasks such as data exploration, regression analysis, and hypothesis testing.
6. PySpark: the Python version of Apache Spark, a big-data processing engine, used for large-scale data processing and distributed computing. It can efficiently clean, transform, and analyze data in a distributed environment.
7. TensorFlow: an open-source deep-learning framework developed by Google, mainly for deep-learning tasks, and often used to build and train machine-learning models, especially complex neural-network models.
8. Keras: a high-level neural-network API mainly used to build and train deep-learning models. Keras is suitable for beginners and researchers in deep learning, because it makes building and training neural networks much simpler.
9. PyTorch: an open-source deep-learning framework developed by Facebook and widely used in research and production environments. PyTorch is a popular framework in deep-learning research and is widely used in fields such as computer vision and natural-language processing.
10. NLTK / SpaCy: natural-language processing libraries.