Skip to content

Commit 207e08b

Browse files
MelissaGrahamMelissaGraham
authored
Merge pull request #92 from lsst/tickets/SP-2667
SP-2667: add column constraint cross match demo to 102.6
2 parents 2618d5b + 800f924 commit 207e08b

1 file changed

Lines changed: 207 additions & 8 deletions

File tree

‎DP1/100_How_to_Use_RSP_Tools/102_Catalog_access/102_6_Cross_match_to_uploaded_table.ipynb‎

Lines changed: 207 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,8 @@
2020
"Container Size: Large <br>\n",
2121
"LSST Science Pipelines version: r29.2.0 <br>\n",
2222
"Last verified to run: 2025-09-22<br>\n",
23-
"Repository: <a href=\"https://github.com/lsst/tutorial-notebooks\">github.com/lsst/tutorial-notebooks</a> <br>"
23+
"Repository: <a href=\"https://github.com/lsst/tutorial-notebooks\">github.com/lsst/tutorial-notebooks</a> <br>\n",
24+
"DOI: <a href=\"https://doi.org/10.11578/rubin/dc.20250909.20\">10.11578/rubin/dc.20250909.20</a> <br>"
2425
]
2526
},
2627
{
@@ -77,7 +78,9 @@
7778
"id": "834266cd-4d6e-4e2a-82b4-61e92dbe489b",
7879
"metadata": {},
7980
"source": [
80-
"Import general python package `astropy`. Import the `pyvo` module for accessing remote data. From the `lsst` package, import the module for accessing the Table Access Protocol (TAP) service."
81+
"Import general python packages `matplotlib`, `numpy`, and `astropy`.\n",
82+
"Import the `pyvo` module for accessing remote data.\n",
83+
"From the `lsst` package, import the module for accessing the Table Access Protocol (TAP) service."
8184
]
8285
},
8386
{
@@ -87,6 +90,8 @@
8790
"metadata": {},
8891
"outputs": [],
8992
"source": [
93+
"import matplotlib.pyplot as plt\n",
94+
"import numpy as np\n",
9095
"from astropy.table import Table\n",
9196
"import pyvo\n",
9297
"from lsst.rsp import get_tap_service"
@@ -227,7 +232,7 @@
227232
"id": "01f451cf-96b0-418a-a885-89bf6ce3231e",
228233
"metadata": {},
229234
"source": [
230-
"Run the query and upload the user-defined table. Retrieve the results."
235+
"Run the query and upload the user-defined table."
231236
]
232237
},
233238
{
@@ -242,7 +247,24 @@
242247
"job.wait(phases=['COMPLETED', 'ERROR'])\n",
243248
"print('Job phase is', job.phase)\n",
244249
"if job.phase == 'ERROR':\n",
245-
" job.raise_if_error()\n",
250+
" job.raise_if_error()"
251+
]
252+
},
253+
{
254+
"cell_type": "markdown",
255+
"id": "5cf99092-9f3c-475d-8157-936fd5f76d28",
256+
"metadata": {},
257+
"source": [
258+
"Retrieve the results."
259+
]
260+
},
261+
{
262+
"cell_type": "code",
263+
"execution_count": null,
264+
"id": "e6b1cb17-7c7e-47f3-b6b7-00d6d1f16837",
265+
"metadata": {},
266+
"outputs": [],
267+
"source": [
246268
"assert job.phase == 'COMPLETED'\n",
247269
"results = job.fetch_result().to_table()"
248270
]
@@ -265,14 +287,174 @@
265287
"results"
266288
]
267289
},
290+
{
291+
"cell_type": "markdown",
292+
"id": "7da39746-a3db-4f38-90e5-ae618a70200e",
293+
"metadata": {},
294+
"source": [
295+
"### 2.3. Cross-match with column constraints\n",
296+
"\n",
297+
"In addition to spatial constraints on the cross-match, constraints can also be placed on the values of other columns in the uploaded tables.\n",
298+
"\n",
299+
"The following query is similar to the one in Section 2.2, except that it:\n",
300+
"\n",
301+
"* returns the column `r` from `ut1` as `sdss_r` (the $r$-band SDSS magnitude)\n",
302+
"* returns the difference between DP1 and SDSS $r$-band magnitudes as `r_diffs`\n",
303+
"* constrains the results to rows of `ut1` with `r` > 17 mag\n",
304+
"* constrains the results to cross-matches for which the $r$-band magnitudes agree within 0.1 mag"
305+
]
306+
},
307+
{
308+
"cell_type": "code",
309+
"execution_count": null,
310+
"id": "8db4ced7-444a-484a-a77f-2d39e1bf2e94",
311+
"metadata": {},
312+
"outputs": [],
313+
"source": [
314+
"query_2 = \"SELECT ut1.objid AS ut1_objid, \"\\\n",
315+
" \"ut1.ra AS ut1_ra, ut1.dec AS ut1_dec, ut1.r AS sdss_r, \"\\\n",
316+
" \"objectId, coord_ra, coord_dec, \"\\\n",
317+
" \"u_cModelMag, g_cModelMag, r_cModelMag, \"\\\n",
318+
" \"i_cModelMag, z_cModelMag, y_cModelMag, \"\\\n",
319+
" \"ABS(ut1.r - r_cModelMag) AS r_diffs \"\\\n",
320+
" \"FROM dp1.Object \"\\\n",
321+
" \"JOIN TAP_UPLOAD.ut1 AS ut1 \"\\\n",
322+
" \"WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), \"\\\n",
323+
" \"CIRCLE('ICRS', ut1.ra, ut1.dec, 0.00027))=1 \"\\\n",
324+
" \"AND ut1.r > 17 AND ABS(ut1.r - r_cModelMag) < 0.1\""
325+
]
326+
},
327+
{
328+
"cell_type": "markdown",
329+
"id": "f955db63-c087-4d6b-a400-6ca397460abb",
330+
"metadata": {},
331+
"source": [
332+
"Submit the query to the TAP service."
333+
]
334+
},
335+
{
336+
"cell_type": "code",
337+
"execution_count": null,
338+
"id": "e0bc1ee3-122d-4ccb-8fa6-b6f1ca94f15e",
339+
"metadata": {},
340+
"outputs": [],
341+
"source": [
342+
"job = service.submit_job(query_2, uploads={\"ut1\": ut1})\n",
343+
"job.run()\n",
344+
"job.wait(phases=['COMPLETED', 'ERROR'])\n",
345+
"print('Job phase is', job.phase)\n",
346+
"if job.phase == 'ERROR':\n",
347+
" job.raise_if_error()"
348+
]
349+
},
350+
{
351+
"cell_type": "markdown",
352+
"id": "c785a166-3549-40df-8bbc-1714ed411390",
353+
"metadata": {},
354+
"source": [
355+
"Retrieve the results."
356+
]
357+
},
358+
{
359+
"cell_type": "code",
360+
"execution_count": null,
361+
"id": "a034d26d-dc9d-43b4-b655-ceef22c04fc3",
362+
"metadata": {},
363+
"outputs": [],
364+
"source": [
365+
"assert job.phase == 'COMPLETED'\n",
366+
"results_2 = job.fetch_result().to_table()"
367+
]
368+
},
369+
{
370+
"cell_type": "markdown",
371+
"id": "5089539c-0519-469c-80dd-e8a4ae3d09c1",
372+
"metadata": {},
373+
"source": [
374+
"Print the sizes of the original `results` table, and the `results_2` table which imposed constraints on $r$-band magnitude."
375+
]
376+
},
377+
{
378+
"cell_type": "code",
379+
"execution_count": null,
380+
"id": "8775aad8-632a-475b-85f1-1e6c283ea1df",
381+
"metadata": {},
382+
"outputs": [],
383+
"source": [
384+
"print(len(results), len(results_2))"
385+
]
386+
},
387+
{
388+
"cell_type": "markdown",
389+
"id": "b1b15313-2883-40b4-8c6a-47b467df3249",
390+
"metadata": {},
391+
"source": [
392+
"Option to display `results_2`."
393+
]
394+
},
395+
{
396+
"cell_type": "code",
397+
"execution_count": null,
398+
"id": "25878ace-508d-4dc2-83a8-1f6f08975853",
399+
"metadata": {},
400+
"outputs": [],
401+
"source": [
402+
"# results_2"
403+
]
404+
},
405+
{
406+
"cell_type": "markdown",
407+
"id": "a9ae9b18-958c-4233-a2b2-545ca10ea13b",
408+
"metadata": {},
409+
"source": [
410+
"As an example visualization, plot the distributions of $r$-band `Object` table magnitudes for both the `results` table (spatial cross-match only) and the `results_2` table (spatial cross-match with $r$-band magnitude constraints)."
411+
]
412+
},
413+
{
414+
"cell_type": "code",
415+
"execution_count": null,
416+
"id": "cabf8c22-fdc9-4f1c-9e4e-07ada4a2a807",
417+
"metadata": {},
418+
"outputs": [],
419+
"source": [
420+
"use_bins = np.arange(40, dtype='float')/2 + 7.5\n",
421+
"fig = plt.figure(figsize=(6, 4))\n",
422+
"plt.hist(ut1['r'], bins=use_bins, alpha=0.2, color='black', label='ut1, r')\n",
423+
"plt.hist(results['r_cModelMag'], bins=use_bins, histtype='step',\n",
424+
" ls='solid', lw=1, label='results, r_cModelMag')\n",
425+
"plt.hist(results_2['r_cModelMag'], bins=use_bins, histtype='step',\n",
426+
" ls='dashed', lw=2, label='results_2, r_cModelMag')\n",
427+
"plt.xlim([15, 24])\n",
428+
"plt.xlabel('r-band magnitude')\n",
429+
"plt.ylabel('number')\n",
430+
"plt.legend(loc='best')\n",
431+
"plt.show()"
432+
]
433+
},
434+
{
435+
"cell_type": "markdown",
436+
"id": "fc11748f-bc5b-4f39-94d3-b66a585aea2d",
437+
"metadata": {},
438+
"source": [
439+
"> **Figure 1:** Comparing the distribution of SDSS $r$-band magnitudes in the uploaded user table `ut1` (grey); the distribution of DP1 `r_cModelMag` magnitudes resulting from a spatial cross-match to SDSS (blue solid); and the distribution of DP1 `r_cModelMag` magnitudes resulting from a spatial *and magnitude-constrained* cross-match to SDSS (orange dashed)."
440+
]
441+
},
442+
{
443+
"cell_type": "markdown",
444+
"id": "f29d5e41-747f-47db-8fb7-f85446401260",
445+
"metadata": {},
446+
"source": [
447+
"Clean up."
448+
]
449+
},
268450
{
269451
"cell_type": "code",
270452
"execution_count": null,
271-
"id": "51a5125c-23c2-4604-b4e9-0f5475339f74",
453+
"id": "b215ff87-6589-40c0-8492-88ad1db44b3b",
272454
"metadata": {},
273455
"outputs": [],
274456
"source": [
275-
"del results, query, ut1"
457+
"del results, results_2, query, query_2, ut1"
276458
]
277459
},
278460
{
@@ -505,7 +687,7 @@
505687
"id": "d0d5c83b-f71e-462e-9930-ccb80eae6b8d",
506688
"metadata": {},
507689
"source": [
508-
"Run the query and upload the user-defined table. Retrieve the results."
690+
"Run the query and upload the user-defined table."
509691
]
510692
},
511693
{
@@ -520,7 +702,24 @@
520702
"job.wait(phases=['COMPLETED', 'ERROR'])\n",
521703
"print('Job phase is', job.phase)\n",
522704
"if job.phase == 'ERROR':\n",
523-
" job.raise_if_error()\n",
705+
" job.raise_if_error()"
706+
]
707+
},
708+
{
709+
"cell_type": "markdown",
710+
"id": "793a5a0a-52e2-4d77-8235-4c893441bdd8",
711+
"metadata": {},
712+
"source": [
713+
"Retrieve the results."
714+
]
715+
},
716+
{
717+
"cell_type": "code",
718+
"execution_count": null,
719+
"id": "20a9064a-932f-47e7-bd92-f51180a287e1",
720+
"metadata": {},
721+
"outputs": [],
722+
"source": [
524723
"assert job.phase == 'COMPLETED'\n",
525724
"results = job.fetch_result().to_table()"
526725
]

0 commit comments

Comments
 (0)