90.machine-learning-in-practice.md

Machine Learning in Practice

In this lesson, we use the classic Titanic survival-prediction project to explain a complete machine-learning application. This project comes from Kaggle, a globally well-known data-science competition platform. This platform is one of the important communities in data science, machine learning, and artificial intelligence. It provides a place for data scientists and algorithm engineers to practice, share, and compete. Whether you are a beginner or an experienced expert, Kaggle can provide rich resources and challenges. Through the datasets, competitions, notebooks, and courses provided on the platform, users can improve related data-science skills according to their own needs and interact with data scientists around the world.

The Titanic survival-prediction project is one of the most famous beginner machine-learning projects on Kaggle. The task is to predict whether passengers survived the Titanic disaster from basic information such as age, sex, ticket class, and port of embarkation. This is a standard classification problem. On Kaggle, the competition is called Titanic - Machine Learning from Disaster. Inside it, you can download the data files train.csv and test.csv. The first is the training set, which contains both features and labels, while the second is the test set, which contains only features.

Data Exploration

We first load the training data. Suppose the data files are stored in a local data directory.

import numpy as np
import pandas as pd

df = pd.read_csv('data/train.csv', index_col='PassengerId')
df.head(5)

Output:

			Survived Pclass	Name			Sex		Age		SibSp	Parch	Ticket		Fare	Cabin	Embarked
PassengerId											
1			0		 3		Braund, ...	    male	22.0	1		0		A/5 ...		7.2500	NaN		S
2			1		 1		Cumings, ...	female	38.0	1		0		PC ...		71.2833	C85		C
3			1		 3		Heikkinen, ...  female	26.0	0		0		STON/O2 ...	7.9250	NaN		S
4			1		 1		Futrelle, ...   female	35.0	1		0		113803 ...	53.1000	C123	S
5			0		 3		Allen, ...	    male	35.0	0		0		373450 ...	8.0500	NaN		S

Note: When loading the data, we directly turn the PassengerId column into the row index.

The columns in the dataset have the following meanings:

Column	Meaning	Notes
Survived	Whether the passenger survived	Target label (1 for survived, 0 for died)
Pclass	Passenger class	Integer 1, 2, or 3
Name	Passenger name	String
Sex	Sex	male or female
Age	Age	Float
SibSp	Number of siblings / spouses aboard	Integer
Parch	Number of parents / children aboard	Integer
Ticket	Ticket number	String
Fare	Ticket fare	Float
Cabin	Cabin number	String
Embarked	Port of embarkation	C, Q, or S

Next, we can visualize the data and do some first exploration with charts.

import matplotlib.pyplot as plt

plt.figure(figsize=(16, 12), dpi=200)

plt.subplot(3, 4, 1)
ser = df.Survived.value_counts()
ser.plot(kind='bar', color=['#BE3144', '#3A7D44'])
plt.xticks(rotation=0)
plt.title('Figure 1. Survival distribution')

plt.subplot(3, 4, 2)
ser = df.Pclass.value_counts().sort_index()
ser.plot(kind='bar', color=['#FA4032', '#FA812F', '#FAB12F'])
plt.xticks(rotation=0)
plt.title('Figure 2. Passenger-class distribution')

plt.subplot(3, 4, 3)
ser = df.Sex.value_counts()
ser.plot(kind='bar', color=['#16404D', '#D84040'])
plt.xticks(rotation=0)
plt.title('Figure 3. Sex distribution')

plt.subplot(3, 4, 4)
ser = df.Embarked.value_counts()
ser.plot(kind='bar', color=['#FA4032', '#FA812F', '#FAB12F'])
plt.xticks(rotation=0)
plt.title('Figure 4. Port-of-embarkation distribution')

plt.show()

Output:

From these charts, we can already see some patterns. First-class passengers had a higher survival rate, female passengers survived at a higher rate than male passengers, and passengers who boarded at Cherbourg had a higher survival rate than those from the other two ports. If you want, you can also try to combine dimensions such as sex, class, and age and draw the corresponding charts. You can even explore whether titles in passenger names, such as Mr, Miss, Mrs, Dr, Master, and Major, have some relationship with survival.

We can also inspect the dataset with df.info():

df.info()

Output:

<class 'pandas.core.frame.DataFrame'>
Index: 891 entries, 1 to 891
Data columns (total 11 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   Survived  891 non-null    int64  
 1   Pclass    891 non-null    int64  
 2   Name      891 non-null    object 
 3   Sex       891 non-null    object 
 4   Age       714 non-null    float64
 5   SibSp     891 non-null    int64  
 6   Parch     891 non-null    int64  
 7   Ticket    891 non-null    object 
 8   Fare      891 non-null    float64
 9   Cabin     204 non-null    object 
 10  Embarked  889 non-null    object 
dtypes: float64(2), int64(4), object(5)
memory usage: 83.5+ KB

This tells us that the training set has 891 rows. Some columns such as Age, Cabin, and Embarked contain missing values, and some fields such as Name and Sex are strings, so they cannot be used directly for model training. This means we need to do some preparation work.

Feature Engineering

Feature engineering is an important part of machine learning. Through processing, transforming, selecting, and constructing the original data, we can extract features that help the model understand the data better and improve the prediction effect of the model.

The main steps usually include:

1. Data cleaning: handle missing values, duplicate values, and outliers.
2. Feature transformation: make raw data more suitable for model input, such as standardization, normalization, categorical encoding, and log transforms.
3. Feature selection: keep useful features and remove irrelevant or redundant ones.
4. Dimensionality reduction: map data from a high-dimensional space into a lower-dimensional one while trying to preserve the main information.
5. Feature construction: derive new features from the original ones.

For the Titanic dataset, we can:

• fill missing Age values with the median;
• fill missing Embarked values with the mode;
• turn Cabin into a binary feature indicating whether cabin information exists.

df['Age'] = df.Age.fillna(df.Age.median())
df['Embarked'] = df.Embarked.fillna(df.Embarked.mode()[0])
df['Cabin'] = df.Cabin.replace(r'.+', '1', regex=True).replace(np.nan, 0).astype('i8')

Then we scale Fare and Age with StandardScaler:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
df[['Fare', 'Age']] = scaler.fit_transform(df[['Fare', 'Age']])

Next we one-hot encode the categorical columns Sex and Embarked:

df = pd.get_dummies(df, columns=['Sex', 'Embarked'], drop_first=True)

The lesson also derives new features from passenger names and family information:

title_mapping = {
    'Mr': 0, 'Miss': 1, 'Mrs': 2, 'Master': 3, 'Dr': 4, 'Rev': 5, 'Col': 6, 'Major': 7,
    'Mlle': 8, 'Ms': 9, 'Lady': 10, 'Sir': 11, 'Jonkheer': 12, 'Don': 13, 'Dona': 14, 'Countess': 15
}
df['Title'] = df['Name'].map(
    lambda x: x.split(',')[1].split('.')[0].strip()
).map(title_mapping).fillna(-1)
df['FamilySize'] = df['SibSp'] + df['Parch'] + 1

Note: We map each title to a number. There is no need to care too much about the exact mapping between titles and numbers, and some titles do not appear in the training set at all. We handle unknown titles as -1.

After that, we can drop fields that are no longer needed:

df.drop(columns=['Name', 'SibSp', 'Parch', 'Ticket'], inplace=True)

Model Training

Now we split the dataset into a training set and a validation set. We are not making the final train/test split here, because Kaggle already gives us the real test set in test.csv. So we train on 90% of train.csv and validate on the remaining 10%.

from sklearn.model_selection import train_test_split

X, y = df.drop(columns='Survived'), df.Survived
X_train, X_valid, y_train, y_valid = train_test_split(X, y, train_size=0.9, random_state=3)

We can first try logistic regression:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

model = LogisticRegression(penalty='l1', tol=1e-6, solver='liblinear')
model.fit(X_train, y_train)
y_pred = model.predict(X_valid)
print(classification_report(y_valid, y_pred))

Output:

              precision    recall  f1-score   support

           0       0.88      0.80      0.84        56
           1       0.72      0.82      0.77        34

    accuracy                           0.81        90
   macro avg       0.80      0.81      0.80        90
weighted avg       0.82      0.81      0.81        90

Then we can try the stronger XGBoost model:

import xgboost as xgb

dm_train = xgb.DMatrix(X_train, y_train)
dm_valid = xgb.DMatrix(X_valid)

params = {
    'booster': 'gbtree',
    'objective': 'binary:logistic',
    'gamma': 0.1,
    'max_depth': 10,
    'lambda': 0.5,
    'subsample': 0.8,
    'colsample_bytree': 0.8,
    'eta': 0.05,
    'seed': 3,
    'nthread': 16,
}

model = xgb.train(params, dm_train, num_boost_round=200)
y_pred = model.predict(dm_valid)
y_pred_label = (y_pred > 0.5).astype('i8')
print(classification_report(y_valid, y_pred_label))

Output:

              precision    recall  f1-score   support

           0       0.89      0.91      0.90        56
           1       0.85      0.82      0.84        34

    accuracy                           0.88        90
   macro avg       0.87      0.87      0.87        90
weighted avg       0.88      0.88      0.88        90

The original lesson also strongly recommends using grid search and cross-validation to tune hyperparameters, and using learning curves to judge underfitting or overfitting.

Model Evaluation

Next, we load the real test data from test.csv, process it the same way, and use the trained model to make predictions. The result can be saved as a CSV file with two columns, PassengerId and Survived, then submitted to Kaggle for scoring.

test = pd.read_csv('data/test.csv', index_col='PassengerId')
test['Age'] = test.Age.fillna(test.Age.median())
test['Fare'] = test.Fare.fillna(test.Fare.median())
test['Embarked'] = test.Embarked.fillna(test.Embarked.mode()[0])
test['Cabin'] = test.Cabin.replace(r'.+', '1', regex=True).replace(np.nan, 0).astype('i8')
test[['Fare', 'Age']] = scaler.fit_transform(test[['Fare', 'Age']])
test = pd.get_dummies(test, columns=['Sex', 'Embarked'], drop_first=True)
test['Title'] = test['Name'].apply(lambda x: x.split(',')[1].split('.')[0].strip()).map(title_mapping).fillna(-1)
test['FamilySize'] = test['SibSp'] + test['Parch'] + 1
test.drop(columns=['Name', 'Ticket', 'SibSp', 'Parch'], inplace=True)

passenger_id, X_test = test.index, test
y_test_pred = model.predict(X_test)

result = pd.DataFrame({
    'PassengerId': passenger_id,
    'Survived': y_test_pred
})
result.to_csv('submission.csv', index=False)

The lesson also suggests several ways to keep improving feature engineering, such as:

1. discretizing Age into bins;
2. handling Cabin more carefully instead of simply binarizing it;
3. combining Pclass and Sex into interaction features;
4. deriving special indicators from titles like Mrs;
5. trying whether removing Embarked helps.

Model Deployment

Once the model is trained, it can be serialized for deployment:

import joblib

joblib.dump(model, 'model.pkl')

Later, it can be loaded again for prediction:

import joblib

model = joblib.load('model.pkl')
model.predict(X_test)

If we want to deploy it as a web service, we can build a simple API with Flask:

from flask import Flask
from flask import jsonify
from flask import request

import joblib
import pandas as pd
import xgboost as xgb

app = Flask(__name__)


@app.route('/predict', methods=['POST'])
def predict():
    query_df = pd.DataFrame(request.json)
    model = joblib.load('model.pkl')
    y_pred = (model.predict(xgb.DMatrix(query_df)) > 0.5).tolist()
    return jsonify({'message': 'OK', 'result': y_pred})


if __name__ == '__main__':
    app.run(debug=True)

Of course, in a real engineering project you would not wait until a request arrives to load the model, because that would seriously hurt web-service performance. The model should be loaded ahead of time at an appropriate moment after the service starts, and its resource usage should also be monitored according to real needs.

If you run the code above, a web server will run on local port 5000 by default. We can then use an API testing tool to send an HTTP request to the server and see whether the model can give a prediction result.

Besides deploying the model as a web service, we can also deploy it into scheduled tasks, or even into some edge devices in a special model format.

Summary

This lesson ties together data exploration, feature engineering, model training, model evaluation, and model deployment through a concrete Kaggle project. The details of the algorithm matter, but this example also shows a broader truth: in real projects, the workflow around the model is often just as important as the model itself.