GraphNeuralNetworks.jl/docs/tutorials/intermediate_tutorials/graph_seg.jl at 5461ad572906d05d9f84b679b1840ed635813117 · JuliaGraphs/GraphNeuralNetworks.jl · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
### A Pluto.jl notebook ###
# v0.19.13

using Markdown
using InteractiveUtils

# ╔═╡ 33f0f180-59c4-468a-8654-2354184fe4ea
# ╠═╡ show_logs = false
begin
	using Pkg
    Pkg.activate(; temp=true)
    packages = [
        PackageSpec(; name="GraphNeuralNetworks", version="0.4"),
        PackageSpec(; name="Flux", version="0.13"),
        PackageSpec(; name="MLDatasets", version="0.7.5"),
        PackageSpec(; name="GeometryBasics"),
        PackageSpec(; name="PlutoUI"),
		PackageSpec(; name="NNlib"),
		PackageSpec(; name="MeshIO"),
		PackageSpec(; name="MLUtils"),
		PackageSpec(; name="JLD"),
    ]
    Pkg.add(packages)
end

# ╔═╡ 29c10c23-b7e5-402c-aa7a-667eef3b76b9
begin
	using GraphNeuralNetworks
	using MLDatasets
	using GeometryBasics
	using Flux
	using Flux: @functor
	using NNlib
	using MLUtils
	using FileIO
end

# ╔═╡ 47091c60-061d-425f-bb88-d4034a341d1b
md"""
This tutorial is a Julia port of tutorial [Deep Learning on 3D Meshes by Anya Fries](https://medium.com/stanford-cs224w/deep-learning-on-3d-meshes-9608a5b33c98) published under the Stanford CS224W GraphML Tutorials on medium.
"""

# ╔═╡ 8fdaf8fb-95fc-47eb-8c43-abe99def97ea
md"""
## Dataset: Faust
"""

# ╔═╡ b58c8ec3-cb35-4e6e-a4a9-21df8cabcc35
data = FAUST()

# ╔═╡ 76685428-22d9-4ee4-b0e5-084d95087e31
md"""
[](TODO: Write in your own words)

To solve the presented segmentation task, we leverage all data encoded in 3D meshes. A 3D mesh defines a surface via a collection of vertices and triangular faces. It is represented by two matrices:

A vertex matrix with dimensions $(n, 3)$, where each row specifies the spatial position, $[x, y, z]$ of a vertex in 3D-space.
A face integer matrix with dimensions $m, 3)$, where each row holds three indices of the vertex matrix that define a triangular face.

![](https://miro.medium.com/max/720/1*ZIzYXMQaQIydFkWYwMwkZQ.png)

Note that the vertex matrix captures node-level feature information and the face matrix describes the node connectivity. Formally, each mesh can be transformed into a graph $G = (X, A)$ with vertices $V$, where $X$ has dimension $(|V|, 3)$ and defines the spatial xyz-features for each node $u$ in $V$, and $A$, the adjacency matrix, has dimension $(|V|, |V|)$ and defines the connected neighborhood of each node. We work with this induced mesh graph.

"""

# ╔═╡ 04f3d007-fc7e-44ab-95ea-ef8d341ee4d8
meshes = data.registrations

# ╔═╡ cb40ca2e-a4d8-4c34-8426-976aca09f0ad
example_mesh = meshes[1]

# ╔═╡ d2e3a8d9-8fdf-425c-a909-473a487d7bf1
function get_coordinates(scan::GeometryBasics.Mesh)
	coords = getproperty.(coordinates(scan), :data) .|> collect
	coords_mat = hcat(coords...)
end

# ╔═╡ 6a4eecd0-3b68-4948-86bc-13e659f3e014
coords = get_coordinates.(meshes) |> batch

# ╔═╡ b9170176-2b72-45a4-b366-fe302c565e81
size(coords)

# ╔═╡ cbdccd33-5c1d-4e9a-a9f1-4a3e7bf80dc0
function get_faces(scan::GeometryBasics.Mesh)
	face_vertices = getproperty.(faces(scan), :data) .|> collect
	face_mat = hcat(face_vertices...) .|> GeometryBasics.value
end

# ╔═╡ 6f868ef6-7c9a-498f-8247-451406e55f8f
face_mats =  get_faces.(meshes) |> batch

# ╔═╡ 2a5fb8e7-9965-4812-bfad-79a15e063ef7
size(face_mats)

# ╔═╡ 4d474ec2-8f09-48a6-8106-dcece099d430
# ╠═╡ show_logs = false
begin
	# single grpah labels
	labels = load("labels.jld", "segmentation")
	# repeat same labels accross all graphs
	segmentation_labels = repeat(labels, 1, 100)
end

# ╔═╡ b44bb8bb-fc38-4906-b43b-b705f3a30678
# TODO: implement ndata for multiple graphs
g = GNNGraph(Int[], Int[], num_nodes=6890, graph_indicator=1:100, ndata=(coords[:, :, 1]), gdata=(;labels=segmentation_labels, faces=face_mats))

# ╔═╡ 7db44279-d879-427f-9e92-570f09014058
md"""
## Network
"""

# ╔═╡ 0ec2d22c-e8ab-4a1d-ac72-9018b90fab0b


# ╔═╡ Cell order:
# ╟─47091c60-061d-425f-bb88-d4034a341d1b
# ╟─33f0f180-59c4-468a-8654-2354184fe4ea
# ╠═29c10c23-b7e5-402c-aa7a-667eef3b76b9
# ╟─8fdaf8fb-95fc-47eb-8c43-abe99def97ea
# ╠═b58c8ec3-cb35-4e6e-a4a9-21df8cabcc35
# ╟─76685428-22d9-4ee4-b0e5-084d95087e31
# ╠═04f3d007-fc7e-44ab-95ea-ef8d341ee4d8
# ╠═cb40ca2e-a4d8-4c34-8426-976aca09f0ad
# ╠═d2e3a8d9-8fdf-425c-a909-473a487d7bf1
# ╠═6a4eecd0-3b68-4948-86bc-13e659f3e014
# ╠═b9170176-2b72-45a4-b366-fe302c565e81
# ╠═cbdccd33-5c1d-4e9a-a9f1-4a3e7bf80dc0
# ╠═6f868ef6-7c9a-498f-8247-451406e55f8f
# ╠═2a5fb8e7-9965-4812-bfad-79a15e063ef7
# ╠═4d474ec2-8f09-48a6-8106-dcece099d430
# ╠═b44bb8bb-fc38-4906-b43b-b705f3a30678
# ╟─7db44279-d879-427f-9e92-570f09014058
# ╠═0ec2d22c-e8ab-4a1d-ac72-9018b90fab0b