Skip to content

Commit aff3487

Browse files
dncrewsdncrews
authored
feat(rootly): add rootly integration plugin (#8877)
* feat(rootly): scaffold plugin skeleton Add the plugin entry point, impl shell with all required plugin interfaces except DataSourcePluginBlueprintV200 (wired in U2), connection model with Bearer auth, service + scope-config + placeholder incident/user/assignment models, and the initial migration script. API resources and subtasks are intentionally empty and will be populated by U2-U5. * feat(rootly): add API handlers and blueprint v200 Wire up connection test, remote scope list/search, scope CRUD, scope sync state, and blueprint v200 endpoints. Remote scope listing speaks JSON:API and page-based pagination to match the Rootly API shape. TestConnection validates the bearer token via GET /users/current. impl.go now wires api.Init and restores DataSourcePluginBlueprintV200 so the plugin can produce pipeline plans for selected Rootly services. * feat(rootly): collect and convert services to domain boards Add services_collector, services_extractor, and service_converter subtasks. Collector fetches GET /services/{id} for the scoped service, unwraps the JSON:API envelope, and populates _tool_rootly_services. Converter emits one ticket.Board per service using the standard domain-id generator, feeding into the existing TICKET pipeline. Also harden remote-scope pagination termination to key off the page we requested rather than meta.current_page so a response missing that field does not silently truncate the service list. * feat(rootly): collect and extract incidents with inline users Finalize the Incident and User models and add role-specific user-id fields (creator, started_by, mitigated_by, resolved_by, closed_by) directly on the Incident row. Add the incidents_collector and incidents_extractor subtasks. The collector is single-phase, filter[services]-scoped and filter[updated_at][gt]-incremental; the extractor unwraps the JSON:API envelope and pulls inline nested user objects from the incident attributes, emitting deduplicated User rows per incident. Drop the Assignment table entirely. Rootly's incident data model is role-per-lifecycle-event, not a list of assignees, so PagerDuty's n-assignees shape does not fit. The schema and GetTablesInfo are adjusted accordingly. * feat(rootly): convert incidents to domain issues Add incidents_converter. For each tool-layer incident, emit a ticket.Issue (Type=INCIDENT) with status, priority, lead time, and resolution date derived from the corresponding Rootly fields; emit one ticket.IssueAssignee per distinct role user (creator, started_by, mitigated_by, resolved_by, closed_by); and emit a ticket.BoardIssue linking the incident to its service board. This feeds DevLake's existing DORA pipeline so change-failure rate and MTTR compute correctly for teams running on Rootly. Unknown incident statuses fall back to IN_PROGRESS with a warning log rather than panicking (a deliberate divergence from the PagerDuty plugin, motivated by Rootly's more volatile status enum). Severity mapping accepts case-insensitive sev0-sev4; unknown severities are preserved as-is. Guard computeLeadTime against resolved-before-started timestamps (clock skew or backfill anomalies) by returning nil rather than the wraparound garbage a naive uint() cast would produce. Tighten test coverage on the dedup, key-fallback, and known-status-warning boundaries flagged during code review. * test(rootly): add e2e test for extract and convert pipeline Add an end-to-end test that drives the extract and convert subtasks over a crafted raw-incident fixture and verifies the tool-layer and domain-layer output against snapshot CSVs. Fixtures cover every branch of the status and severity mapping tables, the same-user-across-roles dedup path, the zero-user case, and the safety-net filter that drops incidents whose relationships point at a different service than the one the task was scoped to. * feat(rootly): register plugin in backend and config-ui Add rootly to the backend plugin startup test and the table-info test so CI exercises it alongside the other plugins. Register RootlyConfig in the config-ui plugin list so operators can create connections, browse scopes, and run blueprints through the UI. The cloud endpoint default includes the /v1/ API version prefix, which is what Rootly's REST API actually expects; without it requests land on the marketing site and return 404. The DOC_URL entries point at a Configuration/Rootly docs page that still needs to be written. Use the Rootly wordmark glyph as the plugin icon, rewritten as a fill-aware (currentColor) SVG so the config-ui can recolor it for selected/unselected states the same way it does every other plugin icon. * fix(rootly): align archived models with live model schemas The U1 init migration's archived Connection, Service, and ScopeConfig models were missing columns contributed by the live models' embedded helpers, so AutoMigrate produced tables the live models could not read or write. Each gap surfaced as an "Unknown column" error from MySQL the first time the table was touched: - Connection was missing endpoint, proxy, rate_limit_per_hour (contributed by helper.RestConnection on the live struct). - Service was missing scope_config_id (contributed by common.Scope). - ScopeConfig was missing connection_id and name (contributed by common.ScopeConfig, which the archived base type does not include). Fold the missing fields into the archived models so a single init migration produces the correct schema. Since rootly has not been deployed anywhere, keeping one init migration is cleaner than chaining follow-up ALTERs; a fresh migrate creates the correct tables in one pass. * fix(rootly): align API contract with real Rootly /v1 responses The original plugin shape was built from docs summaries rather than an actual response capture and diverged from the Rootly API in ways that silently produced zero incidents. Reconcile every code path with ground truth from the OpenAPI spec and a captured GET /v1/incidents response: - Connection test hits /users/me (Rootly's real "who am I" path); the original /users/current returns 404. - Incident list filter is filter[service_ids]=<uuid>, not filter[services]=<uuid>. The latter exists but accepts names and silently matches nothing for a UUID. - Role-bearing user fields (user, started_by, mitigated_by, resolved_by, closed_by) and severity are JSON:API response envelopes nested on attributes: {"data":{"id":...,"attributes":...}}. The previous flat NestedUser / SeverityAttrs shapes were reading the wrong paths, so those fields were always empty. - Service membership lives on the sibling relationships block as JSON:API id+type pointers, not on attributes. The safety-net scope-filter check now reads from the right place. - The incident resource does not have an urgency field. Drop the corresponding column from the model and archived schema. Also harden the collector: split the ResponseParser / next-page logic so pagination state is captured during parse (rather than re-reading the already-drained response body in GetNextPageCustomData), and add lightweight request/response diagnostics gated behind Debug logging. Verified end-to-end against a live Rootly tenant: 3 of 6 scoped services returned incidents, all 3 extracted and converted into ticket.Issue rows with creator assignees and board linkage. * style(rootly): trim narrative comments to match plugin conventions Match PagerDuty's comment density. Keep the few comments that flag non-obvious invariants (archived-base field overrides, 1-based pagination, deliberate divergence from PagerDuty's panic-on-unknown behavior, clock-skew guard). * refactor(rootly): address code review feedback Apply fixes from a multi-lens code review pass: - API: rename swagger {serviceId} to {scopeId} to match registered route; remove dead Proxy handler. - Models: add Sanitize() on RootlyConn; add RoleUserIds() helper on Incident; index ServiceId; drop unused Url field on User; remove dead RootlyResponse/ApiUserResponse types. - Migrations: mirror live schema in archived models (index on service_id; drop user.url). - Collector: switch pagination to reqData.Pager.Page (avoids divide-by-zero), cap at 10000 pages, extract buildIncidentsQuery as a pure helper, drop unreachable lastPageEmpty branch and unused TotalCount, remove diagnostic logs; add unit test pinning the filter[service_ids] param literal as a regression guard. - Services: preserve ScopeConfigId across re-collections; declare ProductTables on collector and extractor metas. - Extractor: skip emitting User rows with neither name nor email so sibling scope tasks can fill in fuller data; use generic resolve() for SequentialId; type ServiceRef as a named struct. - Converter: consolidate mapStatus to return (mapped, known); use Incident.RoleUserIds() instead of an inline slice. - impl.go: comment justifying services-before-incidents subtask ordering. - e2e: rewrite raw incident fixtures to JSON:API envelope shape; regenerate snapshots (drop urgency column). * feat(rootly): add rootly dashboard * chore: run gofmt * chore: add isBeta flag to rootly plugin
1 parent 420e494 commit aff3487

50 files changed

Lines changed: 4573 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎backend/plugins/rootly/api/blueprint_v200.go‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
/*
2+
Licensed to the Apache Software Foundation (ASF) under one or more
3+
contributor license agreements. See the NOTICE file distributed with
4+
this work for additional information regarding copyright ownership.
5+
The ASF licenses this file to You under the Apache License, Version 2.0
6+
(the "License"); you may not use this file except in compliance with
7+
the License. You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
*/
17+
18+
package api
19+
20+
import (
21+
"github.com/apache/incubator-devlake/core/errors"
22+
coreModels "github.com/apache/incubator-devlake/core/models"
23+
"github.com/apache/incubator-devlake/core/models/domainlayer/didgen"
24+
"github.com/apache/incubator-devlake/core/models/domainlayer/ticket"
25+
"github.com/apache/incubator-devlake/core/plugin"
26+
"github.com/apache/incubator-devlake/core/utils"
27+
"github.com/apache/incubator-devlake/helpers/pluginhelper/api"
28+
"github.com/apache/incubator-devlake/helpers/srvhelper"
29+
"github.com/apache/incubator-devlake/plugins/rootly/models"
30+
"github.com/apache/incubator-devlake/plugins/rootly/tasks"
31+
)
32+
33+
func MakeDataSourcePipelinePlanV200(
34+
subtaskMetas []plugin.SubTaskMeta,
35+
connectionId uint64,
36+
bpScopes []*coreModels.BlueprintScope,
37+
) (coreModels.PipelinePlan, []plugin.Scope, errors.Error) {
38+
connection, err := dsHelper.ConnSrv.FindByPk(connectionId)
39+
if err != nil {
40+
return nil, nil, err
41+
}
42+
scopeDetails, err := dsHelper.ScopeSrv.MapScopeDetails(connectionId, bpScopes)
43+
if err != nil {
44+
return nil, nil, err
45+
}
46+
plan, err := makePipelinePlanV200(subtaskMetas, scopeDetails, connection)
47+
if err != nil {
48+
return nil, nil, err
49+
}
50+
scopes, err := makeScopesV200(scopeDetails, connection)
51+
return plan, scopes, err
52+
}
53+
54+
func makePipelinePlanV200(
55+
subtaskMetas []plugin.SubTaskMeta,
56+
scopeDetails []*srvhelper.ScopeDetail[models.Service, models.RootlyScopeConfig],
57+
connection *models.RootlyConnection,
58+
) (coreModels.PipelinePlan, errors.Error) {
59+
plan := make(coreModels.PipelinePlan, len(scopeDetails))
60+
for i, scopeDetail := range scopeDetails {
61+
stage := plan[i]
62+
if stage == nil {
63+
stage = coreModels.PipelineStage{}
64+
}
65+
66+
scope, scopeConfig := scopeDetail.Scope, scopeDetail.ScopeConfig
67+
task, err := api.MakePipelinePlanTask(
68+
"rootly",
69+
subtaskMetas,
70+
scopeConfig.Entities,
71+
tasks.RootlyOptions{
72+
ConnectionId: connection.ID,
73+
ServiceId: scope.Id,
74+
},
75+
)
76+
if err != nil {
77+
return nil, err
78+
}
79+
stage = append(stage, task)
80+
plan[i] = stage
81+
}
82+
83+
return plan, nil
84+
}
85+
86+
func makeScopesV200(
87+
scopeDetails []*srvhelper.ScopeDetail[models.Service, models.RootlyScopeConfig],
88+
connection *models.RootlyConnection,
89+
) ([]plugin.Scope, errors.Error) {
90+
scopes := make([]plugin.Scope, 0, len(scopeDetails))
91+
92+
idgen := didgen.NewDomainIdGenerator(&models.Service{})
93+
for _, scopeDetail := range scopeDetails {
94+
scope, scopeConfig := scopeDetail.Scope, scopeDetail.ScopeConfig
95+
id := idgen.Generate(connection.ID, scope.Id)
96+
97+
if utils.StringsContains(scopeConfig.Entities, plugin.DOMAIN_TYPE_TICKET) {
98+
scopes = append(scopes, ticket.NewBoard(id, scope.Name))
99+
}
100+
}
101+
102+
return scopes, nil
103+
}

‎backend/plugins/rootly/api/connection_api.go‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
/*
2+
Licensed to the Apache Software Foundation (ASF) under one or more
3+
contributor license agreements. See the NOTICE file distributed with
4+
this work for additional information regarding copyright ownership.
5+
The ASF licenses this file to You under the Apache License, Version 2.0
6+
(the "License"); you may not use this file except in compliance with
7+
the License. You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
*/
17+
18+
package api
19+
20+
import (
21+
"context"
22+
"net/http"
23+
24+
"github.com/apache/incubator-devlake/core/errors"
25+
"github.com/apache/incubator-devlake/core/plugin"
26+
"github.com/apache/incubator-devlake/helpers/pluginhelper/api"
27+
"github.com/apache/incubator-devlake/plugins/rootly/models"
28+
)
29+
30+
func testConnection(ctx context.Context, connection models.RootlyConn) (*plugin.ApiResourceOutput, errors.Error) {
31+
if vld != nil {
32+
if err := vld.Struct(connection); err != nil {
33+
return nil, errors.Default.Wrap(err, "error validating target")
34+
}
35+
}
36+
apiClient, err := api.NewApiClientFromConnection(ctx, basicRes, &connection)
37+
if err != nil {
38+
return nil, err
39+
}
40+
response, err := apiClient.Get("users/me", nil, nil)
41+
if err != nil {
42+
return nil, err
43+
}
44+
if response.StatusCode == http.StatusUnauthorized {
45+
return nil, errors.HttpStatus(http.StatusBadRequest).New("StatusUnauthorized error while testing connection")
46+
}
47+
if response.StatusCode == http.StatusOK {
48+
return &plugin.ApiResourceOutput{Body: nil, Status: http.StatusOK}, nil
49+
}
50+
return &plugin.ApiResourceOutput{Body: nil, Status: response.StatusCode}, errors.HttpStatus(response.StatusCode).Wrap(err, "could not validate connection")
51+
}
52+
53+
// TestConnection test rootly connection
54+
// @Summary test rootly connection
55+
// @Description Test Rootly Connection
56+
// @Tags plugins/rootly
57+
// @Param body body models.RootlyConn true "json body"
58+
// @Success 200 {object} shared.ApiBody "Success"
59+
// @Failure 400 {string} errcode.Error "Bad Request"
60+
// @Failure 500 {string} errcode.Error "Internal Error"
61+
// @Router /plugins/rootly/test [POST]
62+
func TestConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
63+
var connection models.RootlyConn
64+
err := api.Decode(input.Body, &connection, vld)
65+
if err != nil {
66+
return nil, err
67+
}
68+
testConnectionResult, testConnectionErr := testConnection(context.TODO(), connection)
69+
if testConnectionErr != nil {
70+
return nil, plugin.WrapTestConnectionErrResp(basicRes, testConnectionErr)
71+
}
72+
return testConnectionResult, nil
73+
}
74+
75+
// TestExistingConnection test rootly connection
76+
// @Summary test rootly connection
77+
// @Description Test Rootly Connection
78+
// @Tags plugins/rootly
79+
// @Param connectionId path int true "connection ID"
80+
// @Success 200 {object} shared.ApiBody "Success"
81+
// @Failure 400 {string} errcode.Error "Bad Request"
82+
// @Failure 500 {string} errcode.Error "Internal Error"
83+
// @Router /plugins/rootly/connections/{connectionId}/test [POST]
84+
func TestExistingConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
85+
connection, err := dsHelper.ConnApi.GetMergedConnection(input)
86+
if err != nil {
87+
return nil, errors.BadInput.Wrap(err, "find connection from db")
88+
}
89+
if err := api.DecodeMapStruct(input.Body, connection, false); err != nil {
90+
return nil, err
91+
}
92+
testConnectionResult, testConnectionErr := testConnection(context.TODO(), connection.RootlyConn)
93+
if testConnectionErr != nil {
94+
return nil, plugin.WrapTestConnectionErrResp(basicRes, testConnectionErr)
95+
}
96+
return testConnectionResult, nil
97+
}
98+
99+
// @Summary create rootly connection
100+
// @Description Create Rootly connection
101+
// @Tags plugins/rootly
102+
// @Param body body models.RootlyConnection true "json body"
103+
// @Success 200 {object} models.RootlyConnection
104+
// @Failure 400 {string} errcode.Error "Bad Request"
105+
// @Failure 500 {string} errcode.Error "Internal Error"
106+
// @Router /plugins/rootly/connections [POST]
107+
func PostConnections(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
108+
return dsHelper.ConnApi.Post(input)
109+
}
110+
111+
// @Summary patch rootly connection
112+
// @Description Patch Rootly connection
113+
// @Tags plugins/rootly
114+
// @Param body body models.RootlyConnection true "json body"
115+
// @Success 200 {object} models.RootlyConnection
116+
// @Failure 400 {string} errcode.Error "Bad Request"
117+
// @Failure 500 {string} errcode.Error "Internal Error"
118+
// @Router /plugins/rootly/connections/{connectionId} [PATCH]
119+
func PatchConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
120+
return dsHelper.ConnApi.Patch(input)
121+
}
122+
123+
// @Summary delete rootly connection
124+
// @Description Delete Rootly connection
125+
// @Tags plugins/rootly
126+
// @Success 200 {object} models.RootlyConnection
127+
// @Failure 400 {string} errcode.Error "Bad Request"
128+
// @Failure 409 {object} services.BlueprintProjectPairs "References exist to this connection"
129+
// @Failure 500 {string} errcode.Error "Internal Error"
130+
// @Router /plugins/rootly/connections/{connectionId} [DELETE]
131+
func DeleteConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
132+
return dsHelper.ConnApi.Delete(input)
133+
}
134+
135+
// @Summary list rootly connections
136+
// @Description List Rootly connections
137+
// @Tags plugins/rootly
138+
// @Success 200 {object} models.RootlyConnection
139+
// @Failure 400 {string} errcode.Error "Bad Request"
140+
// @Failure 500 {string} errcode.Error "Internal Error"
141+
// @Router /plugins/rootly/connections [GET]
142+
func ListConnections(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
143+
return dsHelper.ConnApi.GetAll(input)
144+
}
145+
146+
// @Summary get rootly connection
147+
// @Description Get Rootly connection
148+
// @Tags plugins/rootly
149+
// @Success 200 {object} models.RootlyConnection
150+
// @Failure 400 {string} errcode.Error "Bad Request"
151+
// @Failure 500 {string} errcode.Error "Internal Error"
152+
// @Router /plugins/rootly/connections/{connectionId} [GET]
153+
func GetConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
154+
return dsHelper.ConnApi.GetDetail(input)
155+
}

‎backend/plugins/rootly/api/init.go‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
/*
2+
Licensed to the Apache Software Foundation (ASF) under one or more
3+
contributor license agreements. See the NOTICE file distributed with
4+
this work for additional information regarding copyright ownership.
5+
The ASF licenses this file to You under the Apache License, Version 2.0
6+
(the "License"); you may not use this file except in compliance with
7+
the License. You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
*/
17+
18+
package api
19+
20+
import (
21+
"github.com/apache/incubator-devlake/core/context"
22+
"github.com/apache/incubator-devlake/core/plugin"
23+
"github.com/apache/incubator-devlake/helpers/pluginhelper/api"
24+
"github.com/apache/incubator-devlake/plugins/rootly/models"
25+
"github.com/go-playground/validator/v10"
26+
)
27+
28+
var vld *validator.Validate
29+
var basicRes context.BasicRes
30+
31+
var dsHelper *api.DsHelper[models.RootlyConnection, models.Service, models.RootlyScopeConfig]
32+
var raProxy *api.DsRemoteApiProxyHelper[models.RootlyConnection]
33+
var raScopeList *api.DsRemoteApiScopeListHelper[models.RootlyConnection, models.Service, RootlyRemotePagination]
34+
35+
var raScopeSearch *api.DsRemoteApiScopeSearchHelper[models.RootlyConnection, models.Service]
36+
37+
func Init(br context.BasicRes, p plugin.PluginMeta) {
38+
vld = validator.New()
39+
basicRes = br
40+
dsHelper = api.NewDataSourceHelper[
41+
models.RootlyConnection, models.Service, models.RootlyScopeConfig,
42+
](
43+
br,
44+
p.Name(),
45+
[]string{"name"},
46+
func(c models.RootlyConnection) models.RootlyConnection {
47+
return c.Sanitize()
48+
},
49+
nil,
50+
nil,
51+
)
52+
raProxy = api.NewDsRemoteApiProxyHelper[models.RootlyConnection](dsHelper.ConnApi.ModelApiHelper)
53+
raScopeList = api.NewDsRemoteApiScopeListHelper[models.RootlyConnection, models.Service, RootlyRemotePagination](raProxy, listRootlyRemoteScopes)
54+
raScopeSearch = api.NewDsRemoteApiScopeSearchHelper[models.RootlyConnection, models.Service](raProxy, searchRootlyRemoteScopes)
55+
}

0 commit comments

Comments
 (0)