Skip to content

Commit be31599

Browse files
carlomarxdkcarlomarxdk
committed
Add life2vec files
0 parents  commit be31599

111 files changed

Lines changed: 15935 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2023 Germans Savcisens
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
### Source Code
2+
3+
This repository contains scripts and several notebooks for the data processing, life2vec training, statistical analysis, and visualization. The model weights, experiment logs, and associated model outputs can be obtained in accordance with the rules of [Statistics Denmark's Research Scheme](https://www.dst.dk/en/TilSalg/Forskningsservice/Dataadgang).
4+
5+
Paths (e.g. to data, or model weights) were **redacted** before submitting scripts to the CodeOcean platform.
6+
7+
8+
### Overall Structure
9+
10+
We use [Hydra](https://hydra.cc/docs/intro/) to run the experiments. The `/conf` folder contain configs for the experiments:
11+
1. `/experiment` contains configuration `yaml` for pretraining and finetuning,
12+
2. `/tasks` contain the specification for data augmentation in MLM, SOP, CLS etc.,
13+
3. `/trainer` contains configuration for logging (not used) and multithread training (not used),
14+
4. `/data_new` contains configs for data loading and processing,
15+
5. `/datamodule` contains configs that specify how data should be loaded to PyTorch and PyTorch Lightning
16+
6. `callbacks.yaml` specifies configuration for the PyTorch Lightning Callbacks ,
17+
7. `prepare_data.yaml` can be used to run data preprocessing.
18+
19+
The `/analysis` folder contains `ipynb` notebooks for post-hoc evaluation:
20+
1. `/embedding` contains the analysis of the embedding spaces,
21+
2. `/metric` containt notebooks for the model evaluation
22+
3. `/visualisation` contains notebooks for the visualisation of spaces.
23+
24+
The source folder, `/src`, contains the codes related to the data loading and model training. Due to specifics of the `hydra` package, this folder includes have TCAV implementation (in `/src/analysis/tcav`) and hyperparameter tuning (in `/src/analysis/hyperparameter`). Here is the overview of the `/src` folder:
25+
1. The `/src/data_new` contains scripts to preprocess data as well as prepare data to load into the PyTorch or PyTorch Lightning,
26+
2. The `/src/models` contains the implementation of baseline models,
27+
3. The `/src/tasks` include code specific to the particular task, aka MLM, SOP, Mortality Prediction, Emigration Prediction etc.
28+
4. `/src/tranformer` contains the implementation of the life2vec model:
29+
1. In `performer.py`, we overwrite the functionality of the `performer-pytorch` package,
30+
2. In `cls_model.py`, we have implementation of the finetuning stage (aka life2vec for personality, mortality, emigration),
31+
3. `models.py` contains the code for the life2vec pretraining (aka the base life2vec model),
32+
4. The `transformer_utils.py` contains the implementation of custom modules, like losses, ac tivation functions etc.
33+
5. The `metrics.py` contains code for the custom metric,
34+
6. The `modules.py`, `attention.py`, `att_utils.py`, `embeddings.py` contain the implementation of modules used in the transformer-network (aka life2vec encoders).
35+
36+
Scripts such as `train.py`, `test.py`, `tune.py`, `val.py` used to run a particular stage of the training; while `prepare_data.py` used to run the data processing (see below the example).
37+
38+
39+
### Run the script
40+
To run the code you would use the following commands:
41+
42+
```
43+
# run the pretraining:
44+
HYDRA_FULL_ERROR=1 python -m src.train experiment=pretrain trainer.devices=[7]
45+
46+
# finetuning of the hyperparamaters (for the pretraining)
47+
HYDRA_FULL_ERROR=1 python -m src.train experiment=pretrain_optim
48+
49+
# assemble general dataset (GLOBAL_SET)
50+
HYDRA_FULL_ERROR=1 python -m src.prepare_data +data_new/corpus=global_set target=\${data_new.corpus}
51+
52+
# assemble dataset for the mortality prediction task (SURVIVAL_SET)
53+
HYDRA_FULL_ERROR=1 python -m src.prepare_data +data_new/population=survival_set target=\${data_new.population}
54+
55+
56+
# assemble labour source
57+
python -m src.prepare_data +data_new/sources=labour target=\${data_new.sources}
58+
59+
# run emigration finetuning
60+
HYDRA_FULL_ERROR=1 python -m src.train experiment=emm trainer.devices=[0] version=0.01
61+
```
62+
63+
## How to cite
64+
**Research Square Preprint**
65+
```bibtex
66+
@article{savcisens2023using,
67+
title={Using Sequences of Life-events to Predict Human Lives},
68+
author={Savcisens, Germans and Eliassi-Rad, Tina and Hansen, Lars Kai and Mortensen, Laust and Lilleholt, Lau and Rogers, Anna and Zettler, Ingo and Lehmann, Sune},
69+
year={2023}
70+
}
71+
```

‎analysis/embedding/emb_hubs.ipynb‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "code",
5+
"execution_count": null,
6+
"metadata": {},
7+
"outputs": [],
8+
"source": [
9+
"from sklearn.neighbors import kneighbors_graph\n",
10+
"from sklearn.metrics.pairwise import cosine_distances\n",
11+
"import seaborn as sns\n",
12+
"import networkx as nx\n",
13+
"import numpy as np\n",
14+
"import pandas as pd\n",
15+
"import matplotlib.pyplot as plt"
16+
]
17+
},
18+
{
19+
"attachments": {},
20+
"cell_type": "markdown",
21+
"metadata": {},
22+
"source": [
23+
"This notebook is for the Windows"
24+
]
25+
},
26+
{
27+
"cell_type": "code",
28+
"execution_count": null,
29+
"metadata": {},
30+
"outputs": [],
31+
"source": [
32+
"embedding_path = \"../token_embeddings/tensors.tsv\"\n",
33+
"vocab_path = \"../global_set/result.tsv\"\n",
34+
"emb_df = pd.read_csv(embedding_path, sep=\"\\t\", header=None)\n",
35+
"voc_df = pd.read_csv(vocab_path, sep=\"\\t\").set_index(\"ID\")\n",
36+
"vocab = voc_df[[\"TOKEN\"]]\n",
37+
"vocab = vocab.to_dict()[\"TOKEN\"]"
38+
]
39+
},
40+
{
41+
"cell_type": "code",
42+
"execution_count": null,
43+
"metadata": {},
44+
"outputs": [],
45+
"source": [
46+
"emb = emb_df.values - emb_df.sum(0).values / 2040\n",
47+
"emd_dist = cosine_distances(emb)\n",
48+
"A = kneighbors_graph(emb_df, n_neighbors=5, mode=\"connectivity\", metric=\"cosine\")\n",
49+
"G = nx.from_numpy_matrix(A.toarray(), create_using=nx.DiGraph)\n",
50+
"nx.set_node_attributes(G, vocab, \"token\")"
51+
]
52+
},
53+
{
54+
"cell_type": "code",
55+
"execution_count": null,
56+
"metadata": {},
57+
"outputs": [],
58+
"source": [
59+
"in_degree = G.in_degree()#nx.in_degree_centrality(G)\n",
60+
"in_degree_sorted = sorted(in_degree, key=lambda x: x[1], reverse=True)\n",
61+
"in_degree_dict = dict()\n",
62+
"for node_id, node_degree in in_degree_sorted:\n",
63+
" in_degree_dict[vocab[node_id]] = node_degree\n",
64+
"in_degree_vals = np.array([v for k,v in in_degree_sorted])\n",
65+
"k = 20\n",
66+
"print(\"Top %s nodes (based on in-degree)\" %k)\n",
67+
"for i in range(k):\n",
68+
" node_id, node_degree = in_degree_sorted[i]\n",
69+
" print(i, \"%s: %s\" %(vocab[i], node_degree))"
70+
]
71+
},
72+
{
73+
"cell_type": "code",
74+
"execution_count": null,
75+
"metadata": {},
76+
"outputs": [],
77+
"source": [
78+
"plt.figure(figsize=(17,5))\n",
79+
"plt.hist(in_degree_vals,100)\n",
80+
"\"Median of In-Degree distribution: %s\" % np.median(in_degree_vals)\n",
81+
"plt.title(\"In-degree distribution (Median: %s)\" %np.median(in_degree_vals))\n",
82+
"plt.xlabel(\"In-Degree\")\n",
83+
"plt.ylabel(\"Frequency\")\n",
84+
"sns.despine()\n",
85+
"plt.tight_layout()\n",
86+
"plt.show()"
87+
]
88+
},
89+
{
90+
"cell_type": "code",
91+
"execution_count": null,
92+
"metadata": {},
93+
"outputs": [],
94+
"source": [
95+
"voc_cats = voc_df[\"CATEGORY\"].unique()\n",
96+
"in_deg_cats = dict()\n",
97+
"print(\"Median for in_degrees\")\n",
98+
"for cat in voc_cats:\n",
99+
" if cat in [\"OTHER\", \"NON\"]:\n",
100+
" continue\n",
101+
" cat_tokens = voc_df[voc_df[\"CATEGORY\"] == cat][\"TOKEN\"].values\n",
102+
" cat_degrees = [in_degree_dict[token] for token in cat_tokens]\n",
103+
" print(\"\\t%s (n=%s): %s\" %(cat, len(cat_tokens), np.median(cat_degrees)))\n"
104+
]
105+
},
106+
{
107+
"cell_type": "code",
108+
"execution_count": null,
109+
"metadata": {},
110+
"outputs": [],
111+
"source": [
112+
"voc_cats = voc_df[\"CATEGORY\"].unique()\n",
113+
"in_deg_cats = dict()\n",
114+
"print(\"Number of nodes with 0 in-degrees\")\n",
115+
"for cat in voc_cats:\n",
116+
" if cat in [\"OTHER\", \"NON\"]:\n",
117+
" continue\n",
118+
" cat_tokens = voc_df[voc_df[\"CATEGORY\"] == cat][\"TOKEN\"].values\n",
119+
" cat_degrees = [in_degree_dict[token] == 0 for token in cat_tokens]\n",
120+
" print(\"\\t%s (n=%s): %s (%2.f %% )\" %(cat, len(cat_tokens), np.sum(cat_degrees), 100 * np.sum(cat_degrees)/len(cat_tokens)))"
121+
]
122+
},
123+
{
124+
"cell_type": "code",
125+
"execution_count": null,
126+
"metadata": {},
127+
"outputs": [],
128+
"source": []
129+
}
130+
],
131+
"metadata": {
132+
"interpreter": {
133+
"hash": "eebad428d04bd02616a0910f4c14d0e26b643c764bebaeff663e6d3a43f14d5a"
134+
},
135+
"kernelspec": {
136+
"display_name": "Python 3 (ipykernel)",
137+
"language": "python",
138+
"name": "python3"
139+
},
140+
"language_info": {
141+
"codemirror_mode": {
142+
"name": "ipython",
143+
"version": 3
144+
},
145+
"file_extension": ".py",
146+
"mimetype": "text/x-python",
147+
"name": "python",
148+
"nbconvert_exporter": "python",
149+
"pygments_lexer": "ipython3",
150+
"version": "3.9.13"
151+
}
152+
},
153+
"nbformat": 4,
154+
"nbformat_minor": 4
155+
}

0 commit comments

Comments
 (0)