-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathoutput.txt
More file actions
330 lines (308 loc) · 15.8 KB
/
output.txt
File metadata and controls
330 lines (308 loc) · 15.8 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
Lab 09 - Job Registry with PostgreSQL
Outputs captured during setup, execution, testing, verification, and troubleshooting.
================================================================================
1. Update system packages
================================================================================
Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble InRelease
Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:3 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:4 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:5 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [812 kB]
Get:6 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [621 kB]
Get:7 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [734 kB]
Get:8 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [512 kB]
Fetched 3,057 kB in 3s (1,070 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
27 packages can be upgraded. Run 'apt list --upgradable' to see them.
================================================================================
2. Install PostgreSQL
================================================================================
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
libcommon-sense-perl libjson-perl libjson-xs-perl libllvm18 libpopt0
libpq-dev libpq5 logrotate postgresql-16 postgresql-client-16
postgresql-client-common postgresql-common ssl-cert sysstat
Suggested packages:
postgresql-doc postgresql-doc-16
The following NEW packages will be installed:
libcommon-sense-perl libjson-perl libjson-xs-perl libllvm18 libpopt0
libpq-dev libpq5 logrotate postgresql postgresql-16 postgresql-client-16
postgresql-client-common postgresql-common postgresql-contrib ssl-cert sysstat
0 upgraded, 16 newly installed, 0 to remove and 27 not upgraded.
Need to get 45.8 MB of archives.
After this operation, 190 MB of additional disk space will be used.
Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 libpopt0 amd64 1.19+dfsg-1build1 [28.5 kB]
Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 logrotate amd64 3.21.0-2build1 [62.1 kB]
Get:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 libllvm18 amd64 1:18.1.3-1ubuntu1 [24.9 MB]
Get:4 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 libpq5 amd64 16.4-0ubuntu0.24.04.1 [141 kB]
Get:5 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 libpq-dev amd64 16.4-0ubuntu0.24.04.1 [151 kB]
Get:6 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 postgresql-client-common all 266ubuntu1 [35.1 kB]
Get:7 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 postgresql-common all 266ubuntu1 [258 kB]
Get:8 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 postgresql-client-16 amd64 16.4-0ubuntu0.24.04.1 [1,899 kB]
Get:9 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 postgresql-16 amd64 16.4-0ubuntu0.24.04.1 [16.7 MB]
Get:10 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 postgresql all 16+266ubuntu1 [10.2 kB]
Get:11 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 postgresql-contrib all 16+266ubuntu1 [10.2 kB]
Fetched 45.8 MB in 6s (7,633 kB/s)
Selecting previously unselected package libpopt0:amd64.
Selecting previously unselected package logrotate.
Selecting previously unselected package libllvm18:amd64.
Selecting previously unselected package libpq5:amd64.
Selecting previously unselected package libpq-dev.
Selecting previously unselected package postgresql-client-common.
Selecting previously unselected package postgresql-common.
Selecting previously unselected package postgresql-client-16.
Selecting previously unselected package postgresql-16.
Selecting previously unselected package postgresql.
Selecting previously unselected package postgresql-contrib.
Setting up libpopt0:amd64 (1.19+dfsg-1build1) ...
Setting up logrotate (3.21.0-2build1) ...
Created symlink /etc/systemd/system/timers.target.wants/logrotate.timer → /usr/lib/systemd/system/logrotate.timer.
Setting up libpq5:amd64 (16.4-0ubuntu0.24.04.1) ...
Setting up libpq-dev (16.4-0ubuntu0.24.04.1) ...
Setting up postgresql-client-common (266ubuntu1) ...
Setting up postgresql-common (266ubuntu1) ...
Creating config file /etc/postgresql-common/createcluster.conf with new version
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service → /usr/lib/systemd/system/postgresql.service.
Setting up postgresql-client-16 (16.4-0ubuntu0.24.04.1) ...
Setting up postgresql-16 (16.4-0ubuntu0.24.04.1) ...
Creating new PostgreSQL cluster 16/main ...
/usr/lib/postgresql/16/bin/initdb -D /var/lib/postgresql/16/main --auth-local peer --auth-host scram-sha-256 --no-instructions
The files belonging to this database system will be owned by user "postgres".
The database cluster will be initialized with locale "C.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/16/main ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Setting up postgresql (16+266ubuntu1) ...
Setting up postgresql-contrib (16+266ubuntu1) ...
Processing triggers for libc-bin (2.39-0ubuntu8.3) ...
================================================================================
3. Install Python and dependencies
================================================================================
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
python3 is already the newest version (3.12.3-0ubuntu2).
The following additional packages will be installed:
python3-dev python3-pkg-resources python3-setuptools python3-wheel
python3.12-venv
Suggested packages:
python-setuptools-doc
The following NEW packages will be installed:
python3-dev python3-pip python3-setuptools python3-venv python3-wheel
python3.12-venv
The following packages will be upgraded:
python3-pkg-resources
1 upgraded, 6 newly installed, 0 to remove and 26 not upgraded.
Need to get 9,412 kB of archives.
After this operation, 41.3 MB of additional disk space will be used.
Fetched 9,412 kB in 2s (4,809 kB/s)
Setting up python3-pkg-resources (68.1.2-2ubuntu1.1) ...
Setting up python3-setuptools (68.1.2-2ubuntu1.1) ...
Setting up python3-wheel (0.42.0-2) ...
Setting up python3-pip (24.0+dfsg-1ubuntu1) ...
Setting up python3.12-venv (3.12.3-1ubuntu0.3) ...
Setting up python3-venv (3.12.3-0ubuntu2) ...
Setting up python3-dev (3.12.3-0ubuntu2) ...
Processing triggers for man-db (2.12.0-4build2) ...
================================================================================
3. Install Python and dependencies
================================================================================
Collecting psycopg2-binary
Downloading psycopg2_binary-2.9.10-cp312-cp312-manylinux_2_17_x86_64.whl.metadata (4.9 kB)
Downloading psycopg2_binary-2.9.10-cp312-cp312-manylinux_2_17_x86_64.whl (3.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 18.7 MB/s eta 0:00:00
Installing collected packages: psycopg2-binary
Successfully installed psycopg2-binary-2.9.10
================================================================================
4. Start PostgreSQL service
================================================================================
Synchronizing state of postgresql.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable postgresql
================================================================================
5. Configure PostgreSQL access
================================================================================
ALTER ROLE
================================================================================
5. Configure PostgreSQL access
================================================================================
CREATE DATABASE
================================================================================
Step 2: Apply Schema
================================================================================
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
================================================================================
Step 3: Verify Schema Creation
================================================================================
Table "public.jobs"
Column | Type | Collation | Nullable | Default
---------------+-----------------------------+-----------+----------+---------------------------------------
job_id | integer | | not null | nextval('jobs_job_id_seq'::regclass)
job_name | character varying(255) | | not null |
status | character varying(20) | | not null |
created_at | timestamp without time zone | | not null | CURRENT_TIMESTAMP
started_at | timestamp without time zone | | |
completed_at | timestamp without time zone | | |
result_data | jsonb | | |
error_message | text | | |
Indexes:
"jobs_pkey" PRIMARY KEY, btree (job_id)
"idx_jobs_created_at" btree (created_at)
"idx_jobs_job_name" btree (job_name)
"idx_jobs_status" btree (status)
Check constraints:
"jobs_status_check" CHECK (status::text = ANY (ARRAY['pending'::character varying, 'running'::character varying, 'completed'::character varying, 'failed'::character varying]::text[]))
================================================================================
Step 3: Implement and Test
================================================================================
Testing Job Lifecycle...
Created job with ID: 1
Initial job status:
{
"job_id": 1,
"job_name": "data_processing_job",
"status": "pending",
"created_at": "2026-04-12 17:41:22.318547",
"started_at": null,
"completed_at": null,
"result_data": null,
"error_message": null
}
Job started: True
Job completed: True
Final job status:
{
"job_id": 1,
"job_name": "data_processing_job",
"status": "completed",
"created_at": "2026-04-12 17:41:22.318547",
"started_at": "2026-04-12 17:41:22.322901",
"completed_at": "2026-04-12 17:41:24.325208",
"result_data": {
"errors": 0,
"records_processed": 1500
},
"error_message": null
}
Job duration: 2.002307 seconds
Testing Failed Job...
Created failing job with ID: 2
Failing job started: True
Job failed: True
Failed job status:
{
"job_id": 2,
"job_name": "failing_job",
"status": "failed",
"created_at": "2026-04-12 17:41:24.333152",
"started_at": "2026-04-12 17:41:24.335401",
"completed_at": "2026-04-12 17:41:24.337982",
"result_data": null,
"error_message": "Simulated processing error: unable to connect to upstream service"
}
Testing Query by Status...
Completed jobs count: 1
{
"job_id": 1,
"job_name": "data_processing_job",
"status": "completed",
"created_at": "2026-04-12 17:41:22.318547",
"started_at": "2026-04-12 17:41:22.322901",
"completed_at": "2026-04-12 17:41:24.325208",
"result_data": {
"errors": 0,
"records_processed": 1500
},
"error_message": null
}
Failed jobs count: 1
{
"job_id": 2,
"job_name": "failing_job",
"status": "failed",
"created_at": "2026-04-12 17:41:24.333152",
"started_at": "2026-04-12 17:41:24.335401",
"completed_at": "2026-04-12 17:41:24.337982",
"result_data": null,
"error_message": "Simulated processing error: unable to connect to upstream service"
}
================================================================================
Verify Job Creation and Status Updates
================================================================================
job_id | job_name | status | created_at
--------+---------------------+-----------+----------------------------
1 | data_processing_job | completed | 2026-04-12 17:41:22.318547
2 | failing_job | failed | 2026-04-12 17:41:24.333152
(2 rows)
================================================================================
Verify Timestamps
================================================================================
job_id | job_name | started_at | completed_at | duration
--------+---------------------+----------------------------+----------------------------+----------------
1 | data_processing_job | 2026-04-12 17:41:22.322901 | 2026-04-12 17:41:24.325208 | 00:00:02.002307
2 | failing_job | 2026-04-12 17:41:24.335401 | 2026-04-12 17:41:24.337982 | 00:00:00.002581
(2 rows)
================================================================================
Verify Results Storage
================================================================================
job_id | job_name | result_data
--------+---------------------+------------------------------------------
1 | data_processing_job | {"errors": 0, "records_processed": 1500}
(1 row)
================================================================================
Verify Error Handling
================================================================================
job_id | job_name | error_message
--------+-------------+--------------------------------------------------------------
2 | failing_job | Simulated processing error: unable to connect to upstream service
(1 row)
================================================================================
Connection refused error
================================================================================
● postgresql.service - PostgreSQL RDBMS
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; preset: enabled)
Active: active (exited) since Sun 2026-04-12 17:39:58 UTC; 2min 14s ago
Main PID: 4121 (code=exited, status=0/SUCCESS)
CPU: 2ms
Apr 12 17:39:58 ip-172-31-10-184 systemd[1]: Starting postgresql.service - PostgreSQL RDBMS...
Apr 12 17:39:58 ip-172-31-10-184 systemd[1]: Finished postgresql.service - PostgreSQL RDBMS.
================================================================================
Module not found (`psycopg2`)
================================================================================
Requirement already satisfied: psycopg2-binary in ./venv/lib/python3.12/site-packages (2.9.10)
================================================================================
Module not found (`psycopg2`)
================================================================================
Collecting psycopg2-binary
Using cached psycopg2_binary-2.9.10-cp312-cp312-manylinux_2_17_x86_64.whl
Installing collected packages: psycopg2-binary
Successfully installed psycopg2-binary-2.9.10
================================================================================
Permission denied on database
================================================================================
GRANT
================================================================================
Check existing jobs
================================================================================
count
-------
2
(1 row)