Skip to content

Commit bd28d8c

Browse files
test(e2e): add live end-to-end test suite for the jh CLI (#43)
* test(e2e): add live end-to-end test suite for the jh CLI Add an `e2e`-tagged Go test suite under e2e/ that builds the jh binary and runs its read/GET commands against a live JuliaHub instance (the platform nightly instance by default). One file per command category; commands that need a specific entity (dataset, registry, project) list first and drive the detail command off the first entry. Tests assert real behaviour and skip gracefully on backend gaps (missing perms, absent endpoints, disallowed queries, timeouts) while still failing on genuine CLI defects. Authentication is purely file-based: the harness materializes a throwaway ~/.juliahub in an isolated HOME from credentials supplied via env vars, so the CLI's auth path is exercised unchanged and a real login is never touched. The suite is build-tag gated so `go test ./...` is unaffected; CI compile-checks it. Also fix `admin group list`, which failed to parse the /app/config/groups response: the endpoint returns groups bucketed by category ({"juliahub":{"groups":[...]},"site":{"groups":[...]}}) but the code expected a top-level array. Parse the categorized object, flatten, and sort by id. Includes e2e/ci/ glue (composite action + example job) for wiring the suite into the JuliaHub platform nightly CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(e2e): cover the scan command Add e2e/scan_test.go for `jh scan`: a deterministic input-validation test (missing manifest) plus a submit→status→results lifecycle that uploads a sample Manifest.toml with --no-wait, then drives `scan status`/`scan results` off the server-assigned run_uuid. Skips when the static-analysis endpoint is absent on the instance (404). Adds an errorLine() harness helper for clearer skip diagnostics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(e2e): split harness, unify helpers, add Makefile, drop CI dupes Self-review cleanup of the e2e suite: - Split the 459-line harness_test.go: run machinery stays in harness_test.go; assertions, output parsers, and backend-gap/skip helpers move to helpers_test.go. - Unify the duplicated packageBackendGap/scanBackendGap into a single backendGap() helper; use the clearer errorLine() in their skip messages. - Fix a temp-dir leak: the freshly built binary's dir is now removed on exit. - Drop the write-only result.err field and the dead expectedName var. - Remove the duplicated e2e/ci/ CI snippets; the canonical wiring lives in the JuliaHub repo. Fold a concise "Running in CI" section into e2e/README.md. - Add a Makefile (make e2e / e2e-fast / e2e-compile / check) so developers don't have to remember the -tags e2e invocations; e2e-fast skips the slow project-list tests (~14s vs ~70s). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * readme fix * dded more tests + unit tests * dded more tests + unit tests * undo dev changes * undo dev changes --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 6b85033 commit bd28d8c

37 files changed

Lines changed: 2776 additions & 10 deletions

.claude/settings.local.json

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,7 @@
44
"Bash(go test:*)",
55
"Bash(go fmt:*)",
66
"Bash(go vet:*)",
7-
"Bash(go build:*)",
8-
"Bash(chmod:*)",
9-
"Bash(grep -oE \"[^/]*Manifest[^/]*\\\\.toml$\")",
10-
"Bash(rm -rf scantest)",
11-
"Bash(mkdir -p scantest/multi scantest/single)",
12-
"Bash(/home/ubuntu/jh/jh scan *)",
13-
"Bash(gofmt -l scan.go main.go)"
7+
"Bash(go build:*)"
148
],
159
"deny": [],
1610
"ask": []

.github/workflows/ci.yml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,14 @@ jobs:
2323
- name: Run go vet
2424
run: go vet ./...
2525

26+
- name: Compile-check e2e tests
27+
# The e2e suite (build tag `e2e`) runs against a live JuliaHub instance as
28+
# part of the platform nightly CI, not here. Vet it under the build tag so
29+
# it cannot bitrot. `-run '^$'` ensures nothing actually executes.
30+
run: |
31+
go vet -tags e2e ./e2e/...
32+
go test -tags e2e -run '^$' ./e2e/...
33+
2634
- name: Run go fmt
2735
run: |
2836
if [ "$(gofmt -s -l . | wc -l)" -gt 0 ]; then

.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,7 @@
11
jh
22
dist/
3+
4+
# coverage artifacts (make coverage)
5+
e2e.test
6+
cover.out
7+
cover.html

Makefile

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
.DEFAULT_GOAL := help
2+
3+
.PHONY: help build test fmt vet check e2e e2e-fast e2e-compile coverage
4+
5+
# Extra flags forwarded to the compiled e2e test binary (e.g. E2E_FLAGS=-test.v).
6+
E2E_FLAGS ?=
7+
8+
help: ## List available targets
9+
@grep -hE '^[a-zA-Z0-9_-]+:.*?## ' $(MAKEFILE_LIST) | \
10+
awk 'BEGIN{FS=":.*?## "}{printf " \033[36m%-12s\033[0m %s\n", $$1, $$2}'
11+
12+
build: ## Build the jh binary
13+
go build -o jh .
14+
15+
test: ## Run unit tests
16+
go test ./...
17+
18+
fmt: ## Format the code (gofmt -s)
19+
gofmt -s -w .
20+
21+
vet: ## Run go vet (incl. the e2e build tag)
22+
go vet ./...
23+
go vet -tags e2e ./e2e/...
24+
25+
check: fmt vet test e2e-compile ## Pre-commit checks (mirrors CI)
26+
27+
# --- end-to-end tests (run against a live JuliaHub instance) ---
28+
# These need a `jh auth login`, or JULIAHUB_SERVER + JULIAHUB_ID_TOKEN/JULIAHUB_TOKEN.
29+
# See e2e/README.md.
30+
31+
e2e: ## Run the live e2e suite (uses your ~/.juliahub login)
32+
go test -tags e2e -v ./e2e/...
33+
34+
e2e-fast: ## Run the e2e suite, skipping the slow project-list tests
35+
go test -tags e2e -v -skip 'TestProjectList' ./e2e/...
36+
37+
e2e-compile: ## Compile-check the e2e suite without running it (what CI does)
38+
go test -tags e2e -run '^$$' ./e2e/...
39+
40+
# --- coverage ---
41+
# Full coverage of package main: unit tests + the live e2e suite, merged.
42+
# Builds an instrumented `jh` (go build -cover) and runs the compiled e2e binary
43+
# so the subprocess emits runtime coverage via GOCOVERDIR — plain `go test`
44+
# swallows it. Needs a login for the e2e portion (see e2e/README.md); without one
45+
# the e2e tests skip and you still get unit coverage. Reports even if some live
46+
# tests fail/skip. Forward flags with E2E_FLAGS (e.g. E2E_FLAGS=-test.v).
47+
48+
coverage: ## Full coverage (unit + live e2e, merged) -> cover.out + cover.html
49+
go build -cover -o jh .
50+
go test -tags e2e -c -o e2e.test ./e2e/
51+
@u=$$(mktemp -d); e=$$(mktemp -d); m=$$(mktemp -d); \
52+
GOCOVERDIR=$$u go test -cover ./... -args -test.gocoverdir=$$u >/dev/null; \
53+
GOCOVERDIR=$$e JH_BIN=$(CURDIR)/jh ./e2e.test $(E2E_FLAGS) || true; \
54+
go tool covdata merge -i=$$u,$$e -o $$m; \
55+
go tool covdata textfmt -i=$$m -o cover.out; \
56+
go tool cover -func=cover.out | tail -1; \
57+
go tool cover -html=cover.out -o cover.html; \
58+
rm -rf $$u $$e $$m e2e.test; \
59+
echo "wrote cover.out, cover.html"

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ Invoke-WebRequest -Uri "https://raw.githubusercontent.com/JuliaComputing/jh/main
6464
6565
# Clean up
6666
Remove-Item install.ps1
67+
```
6768

6869
**Option 2: Command Prompt (CMD)**
6970

auth_test.go

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,13 @@
11
package main
22

33
import (
4+
"encoding/base64"
5+
"encoding/json"
6+
"os"
7+
"path/filepath"
8+
"strings"
49
"testing"
10+
"time"
511
)
612

713
func TestParseJWTClaims(t *testing.T) {
@@ -17,3 +23,112 @@ func TestParseJWTClaims(t *testing.T) {
1723
t.Error("decodeJWT should fail with empty token")
1824
}
1925
}
26+
27+
// makeJWT builds a syntactically valid (unsigned) JWT carrying the given claims.
28+
func makeJWT(t *testing.T, claims JWTClaims) string {
29+
t.Helper()
30+
enc := func(v any) string {
31+
b, err := json.Marshal(v)
32+
if err != nil {
33+
t.Fatalf("marshal: %v", err)
34+
}
35+
return base64.RawURLEncoding.EncodeToString(b)
36+
}
37+
header := enc(map[string]string{"alg": "none", "typ": "JWT"})
38+
return header + "." + enc(claims) + ".sig"
39+
}
40+
41+
func TestDecodeJWTValid(t *testing.T) {
42+
want := JWTClaims{Subject: "user-1", Email: "a@b.com", PreferredUsername: "ab", ExpiresAt: 1893456000}
43+
got, err := decodeJWT(makeJWT(t, want))
44+
if err != nil {
45+
t.Fatalf("decodeJWT: %v", err)
46+
}
47+
if got.Subject != want.Subject || got.Email != want.Email || got.ExpiresAt != want.ExpiresAt {
48+
t.Errorf("decodeJWT = %+v, want subject/email/exp from %+v", got, want)
49+
}
50+
}
51+
52+
func TestIsTokenExpired(t *testing.T) {
53+
past := time.Now().Add(-time.Hour).Unix()
54+
future := time.Now().Add(time.Hour).Unix()
55+
56+
expired, err := isTokenExpired(makeJWT(t, JWTClaims{ExpiresAt: past}), 0)
57+
if err != nil {
58+
t.Fatalf("isTokenExpired(past): %v", err)
59+
}
60+
if !expired {
61+
t.Error("token with past exp should be expired")
62+
}
63+
64+
expired, err = isTokenExpired(makeJWT(t, JWTClaims{ExpiresAt: future}), 0)
65+
if err != nil {
66+
t.Fatalf("isTokenExpired(future): %v", err)
67+
}
68+
if expired {
69+
t.Error("token with future exp should not be expired")
70+
}
71+
72+
// A malformed token is treated as expired (fail-safe) and returns an error.
73+
if expired, err := isTokenExpired("garbage", 0); err == nil || !expired {
74+
t.Errorf("isTokenExpired(garbage) = (%v, %v), want (true, error)", expired, err)
75+
}
76+
}
77+
78+
func TestFormatTokenInfo(t *testing.T) {
79+
tok := &StoredToken{
80+
AccessToken: makeJWT(t, JWTClaims{Subject: "sub-1", Issuer: "https://s/dex", Audience: "device", ExpiresAt: 1893456000}),
81+
TokenType: "Bearer",
82+
RefreshToken: "r",
83+
Server: "nightly.juliahub.dev",
84+
Name: "A B",
85+
Email: "a@b.com",
86+
}
87+
out := formatTokenInfo(tok)
88+
for _, want := range []string{
89+
"Server: nightly.juliahub.dev",
90+
"Token Status: Valid", // future exp
91+
"Subject: sub-1",
92+
"Issuer: https://s/dex",
93+
"Audience: device",
94+
"Token Type: Bearer",
95+
"Has Refresh Token: true",
96+
"Email: a@b.com",
97+
} {
98+
if !strings.Contains(out, want) {
99+
t.Errorf("formatTokenInfo missing %q\n---\n%s", want, out)
100+
}
101+
}
102+
103+
// A bad access token yields an error string rather than panicking.
104+
if got := formatTokenInfo(&StoredToken{AccessToken: "bad"}); !strings.Contains(got, "Error decoding token") {
105+
t.Errorf("expected decode-error string, got %q", got)
106+
}
107+
}
108+
109+
func TestReadStoredToken(t *testing.T) {
110+
home := t.TempDir()
111+
t.Setenv("HOME", home)
112+
config := strings.Join([]string{
113+
"server=nightly.juliahub.dev",
114+
"access_token=acc",
115+
"refresh_token=ref",
116+
"token_type=Bearer",
117+
"id_token=idt",
118+
"name=A B",
119+
"email=a@b.com",
120+
"",
121+
}, "\n")
122+
if err := os.WriteFile(filepath.Join(home, ".juliahub"), []byte(config), 0o600); err != nil {
123+
t.Fatalf("seed config: %v", err)
124+
}
125+
126+
tok, err := readStoredToken()
127+
if err != nil {
128+
t.Fatalf("readStoredToken: %v", err)
129+
}
130+
if tok.Server != "nightly.juliahub.dev" || tok.AccessToken != "acc" || tok.IDToken != "idt" ||
131+
tok.RefreshToken != "ref" || tok.Email != "a@b.com" || tok.Name != "A B" {
132+
t.Errorf("parsed token mismatch: %+v", tok)
133+
}
134+
}

credentials_test.go

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
package main
2+
3+
import (
4+
"encoding/base64"
5+
"os"
6+
"path/filepath"
7+
"strings"
8+
"testing"
9+
)
10+
11+
func TestToDataURL(t *testing.T) {
12+
got := toDataURL([]byte("hello"))
13+
const prefix = "data:application/octet-stream;base64,"
14+
if !strings.HasPrefix(got, prefix) {
15+
t.Fatalf("toDataURL missing prefix: %q", got)
16+
}
17+
decoded, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(got, prefix))
18+
if err != nil {
19+
t.Fatalf("payload is not valid base64: %v", err)
20+
}
21+
if string(decoded) != "hello" {
22+
t.Errorf("decoded payload = %q, want %q", decoded, "hello")
23+
}
24+
}
25+
26+
func TestResolvePrivateKey(t *testing.T) {
27+
// Inline key.
28+
inline, err := resolvePrivateKey("PEMDATA", "")
29+
if err != nil {
30+
t.Fatalf("inline: %v", err)
31+
}
32+
if inline != toDataURL([]byte("PEMDATA")) {
33+
t.Errorf("inline data URL mismatch: %q", inline)
34+
}
35+
36+
// File-based key.
37+
dir := t.TempDir()
38+
keyPath := filepath.Join(dir, "key.pem")
39+
if err := os.WriteFile(keyPath, []byte("FILEDATA"), 0o600); err != nil {
40+
t.Fatalf("write key: %v", err)
41+
}
42+
fromFile, err := resolvePrivateKey("", keyPath)
43+
if err != nil {
44+
t.Fatalf("file: %v", err)
45+
}
46+
if fromFile != toDataURL([]byte("FILEDATA")) {
47+
t.Errorf("file data URL mismatch: %q", fromFile)
48+
}
49+
50+
// Neither → empty, no error.
51+
none, err := resolvePrivateKey("", "")
52+
if err != nil || none != "" {
53+
t.Errorf("resolvePrivateKey(\"\",\"\") = (%q, %v), want (\"\", nil)", none, err)
54+
}
55+
56+
// Missing file → error.
57+
if _, err := resolvePrivateKey("", filepath.Join(dir, "missing.pem")); err == nil {
58+
t.Error("expected error for missing key file")
59+
}
60+
}
61+
62+
func TestSSHHostList(t *testing.T) {
63+
creds := &Credentials{SSHCreds: []CredSSH{
64+
{KnownHost: "github.com ssh-ed25519 AAAA", PrivateKey: "secret1"},
65+
{KnownHost: "gitlab.com ssh-rsa BBBB", PrivateKey: "secret2"},
66+
}}
67+
got := sshHostList(creds)
68+
if len(got) != 2 {
69+
t.Fatalf("got %d entries, want 2", len(got))
70+
}
71+
for i, s := range got {
72+
if s.PrivateKey != "" {
73+
t.Errorf("entry %d retained private key: %q", i, s.PrivateKey)
74+
}
75+
if s.KnownHost != creds.SSHCreds[i].KnownHost {
76+
t.Errorf("entry %d host = %q, want %q", i, s.KnownHost, creds.SSHCreds[i].KnownHost)
77+
}
78+
}
79+
}
80+
81+
func TestReadJSONInput(t *testing.T) {
82+
got, err := readJSONInput([]string{`{"a":1}`})
83+
if err != nil {
84+
t.Fatalf("readJSONInput: %v", err)
85+
}
86+
if string(got) != `{"a":1}` {
87+
t.Errorf("readJSONInput arg = %q, want %q", got, `{"a":1}`)
88+
}
89+
}
90+
91+
// Credential add/update validation runs before any network call, so these error
92+
// branches are exercised without a backend. A bogus server is supplied to prove
93+
// nothing reaches the wire.
94+
func TestCredentialValidation(t *testing.T) {
95+
const srv = "credential-validation.invalid"
96+
tests := []struct {
97+
name string
98+
fn func() error
99+
}{
100+
{"add token: invalid JSON", func() error { return addCredentialToken(srv, []byte("{")) }},
101+
{"add token: missing name", func() error { return addCredentialToken(srv, []byte(`{"url":"u","value":"v"}`)) }},
102+
{"add token: missing url", func() error { return addCredentialToken(srv, []byte(`{"name":"n","value":"v"}`)) }},
103+
{"add token: missing value", func() error { return addCredentialToken(srv, []byte(`{"name":"n","url":"u"}`)) }},
104+
{"add ssh: missing host_key", func() error { return addCredentialSSH(srv, []byte(`{}`)) }},
105+
{"add ssh: both keys", func() error {
106+
return addCredentialSSH(srv, []byte(`{"host_key":"h","private_key":"k","private_key_file":"f"}`))
107+
}},
108+
{"add github-app: missing app_id", func() error { return addCredentialGitHubApp(srv, []byte(`{"url":"u"}`)) }},
109+
{"add github-app: missing url", func() error { return addCredentialGitHubApp(srv, []byte(`{"app_id":"1"}`)) }},
110+
{"add github-app: both keys", func() error {
111+
return addCredentialGitHubApp(srv, []byte(`{"app_id":"1","url":"u","private_key":"k","private_key_file":"f"}`))
112+
}},
113+
{"update token: missing name", func() error { return updateCredentialToken(srv, []byte(`{"url":"u"}`)) }},
114+
{"update token: nothing to update", func() error { return updateCredentialToken(srv, []byte(`{"name":"n"}`)) }},
115+
{"update ssh: invalid index", func() error { return updateCredentialSSH(srv, []byte(`{"index":0,"host_key":"h"}`)) }},
116+
{"update ssh: nothing to update", func() error { return updateCredentialSSH(srv, []byte(`{"index":1}`)) }},
117+
{"update github-app: missing app_id", func() error { return updateCredentialGitHubApp(srv, []byte(`{"url":"u"}`)) }},
118+
{"update github-app: nothing to update", func() error { return updateCredentialGitHubApp(srv, []byte(`{"app_id":"1"}`)) }},
119+
}
120+
for _, tt := range tests {
121+
t.Run(tt.name, func(t *testing.T) {
122+
if err := tt.fn(); err == nil {
123+
t.Errorf("%s: expected validation error, got nil", tt.name)
124+
}
125+
})
126+
}
127+
}

0 commit comments

Comments
 (0)