KB per-container + Neo4j latency chart + reorder (v0.10.0)

Robbie1977 · Robbie1977 · commit 59fd7e93d088 · 2026-06-04T09:49:13.000+01:00
1. KB Neo4j (kb.virtualflybrain.org) gains per-container probing
   via service 1s56 (vfb-neo4j-kb-readonly, scale 2). Same
   pattern as PDB and ServerONLY: each member hit at
   http://&lt;ip&gt;:7474.

2. New "Query latency over time (cluster avg, ms)" sparkline on
   every Neo4j card. Plots the AVG(latency_ms) per ts already
   captured in neo4j_history.

3. Core user-facing group now renders at the very top of the
   page, above Rancher cluster / Neo4j / Apps / Caches. The
   remaining groups stay in their original order below.

Also:
- Auth-failure detection on Neo4j cards now applies cluster-wide
  (every container errors with 401/Unauthorized -&gt; auth? pill).
- Moved the all-auth-err / first-err computation out of Jinja
  (where the select('search', ...) test doesn't exist) into the
  card builder in Python.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,18 @@
 
 All notable changes to vfb-status are recorded here. The format is loosely based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.10.0] — 2026-06-04
+
+### Added
+
+- KB Neo4j (`kb.virtualflybrain.org`) now per-container: probes each `vfb-neo4j-kb-readonly` member (service `1s56`, scale 2) at `http://<primaryIpAddress>:7474`. Same SHOW DATABASES + count probe; same cluster summary + per-container table on the card.
+- Latency-over-time sparkline on each Neo4j card, below the node-count chart. Plots `AVG(latency_ms)` across cluster members per timestamp.
+- Auth-failure detection on Neo4j cards now applies to the whole cluster — if every container errors with a 401/Unauthorized, the card shows an amber `auth?` pill instead of red `down`. (Previously a single-container check.)
+
+### Changed
+
+- **`Core user-facing` group now renders at the very top of the page**, above the specialised sections (Rancher cluster, Neo4j databases, Application services, Caching services). The remaining groups stay in their original order below. Driven by a new `priority_groups` / `rest_groups` split passed to the template.
+
 ## [0.9.0] — 2026-06-04
 
 Per-container probing for `neo4j_services` + new ServerONLY PDB entry.
@@ -221,6 +233,7 @@ Initial release. Self-contained Docker uptime tracker for public-facing Virtual
 - Four subdomains (`nas0`, `iip3d`, `nblast`, `abd1-5.catmaid`) ship with `verify_tls: false` because the production cert SAN doesn't cover them. The servers are up; the cert provisioning is a separate problem.
 - Kubernetes nodes are intentionally not handled here — separate checks planned for a later release.
 
+[0.10.0]: https://github.com/VirtualFlyBrain/vfb-status/releases/tag/v0.10.0
 [0.9.0]: https://github.com/VirtualFlyBrain/vfb-status/releases/tag/v0.9.0
 [0.8.2]: https://github.com/VirtualFlyBrain/vfb-status/releases/tag/v0.8.2
 [0.8.1]: https://github.com/VirtualFlyBrain/vfb-status/releases/tag/v0.8.1
diff --git a/app/main.py b/app/main.py
@@ -2181,9 +2181,15 @@ def _neo4j_card(state: State, svc: Neo4jServiceSpec) -> dict[str, Any]:
     ok_containers = [c for c in containers if c.ok]
     counts = [c.node_count for c in ok_containers if c.node_count is not None]
     statuses = sorted({c.db_status for c in containers if c.db_status})
+    auth_markers = ("401", "Unauthorized", "auth")
+    def _is_auth_err(err: str | None) -> bool:
+        return bool(err) and any(m in err for m in auth_markers)
+    errored = [c for c in containers if not c.ok]
     summary = {
         "container_count": len(containers),
         "ok_count": len(ok_containers),
+        "is_all_auth_err": bool(errored) and len(errored) == len(containers) and all(_is_auth_err(c.error) for c in errored),
+        "first_err": next((c.error for c in errored if c.error), None),
         "node_count_max": max(counts) if counts else None,
         "node_count_min": min(counts) if counts else None,
         "node_count_mixed": len(set(counts)) > 1,
@@ -2204,12 +2210,14 @@ def _neo4j_card(state: State, svc: Neo4jServiceSpec) -> dict[str, Any]:
         svc.name, _cache_chart_window_seconds(), max_points=120
     )
     count_pts = [(r["ts"], r["node_count"]) for r in series if r["node_count"] is not None]
+    latency_pts = [(r["ts"], r["latency_ms"]) for r in series if r["latency_ms"] is not None]
     return {
         "spec": svc,
         "containers": containers,
         "summary": summary,
         "series": series,
         "count_pts": count_pts,
+        "latency_pts": latency_pts,
     }
 
 
@@ -2352,6 +2360,15 @@ async def index(request: Request) -> HTMLResponse:
     cache_cards = [_cache_card(state, s) for s in state.cache_services]
     app_cards = [_app_card(state, s) for s in state.app_services]
     neo4j_cards = [_neo4j_card(state, s) for s in state.neo4j_services]
+    # Split the groups dict so the template can render the priority group
+    # (Core user-facing) at the very top of the page, then the specialised
+    # sections (Rancher cluster, Neo4j, Apps, Caches), then the remaining
+    # groups in their original order.
+    priority_group = "Core user-facing"
+    priority_groups = {
+        priority_group: grouped[priority_group]
+    } if priority_group in grouped else {}
+    rest_groups = {k: v for k, v in grouped.items() if k != priority_group}
     # Cluster overview helpers
     cluster = state.cluster_result
     cluster_hosts_by_short = (
@@ -2392,6 +2409,8 @@ async def index(request: Request) -> HTMLResponse:
             "cache_cards": cache_cards,
             "app_cards": app_cards,
             "neo4j_cards": neo4j_cards,
+            "priority_groups": priority_groups,
+            "rest_groups": rest_groups,
             "cluster": cluster,
             "cluster_summary": cluster_summary,
             "cluster_hosts_by_short": cluster_hosts_by_short,
diff --git a/app/templates/index.html b/app/templates/index.html
@@ -254,6 +254,58 @@ <h1>VFB service status</h1>
       {%- if p is none -%}—{%- else -%}{{ "%.2f"|format(p) }}%{%- endif -%}
     {%- endmacro %}
 
+    {%- macro render_groups(group_dict) -%}
+    {% for group_name, rows in group_dict.items() %}
+      {% set is_rancher_group = group_name.startswith("Rancher servers") %}
+      <section>
+        <h2>{{ group_name }}{% if is_rancher_group and cluster_summary %} · LB on {{ cluster_summary.lb_coverage }}/{{ cluster_summary.hosts_total }} · {{ cluster_summary.dns_ingress_count }} on DNS{% endif %}</h2>
+        <div class="svc-list">
+          {% for row in rows %}
+            {% set r = row.result %}
+            {% set rh = none %}
+            {% if is_rancher_group %}
+              {%- set _short = r.name.split('.')[0].lower() -%}
+              {%- set rh = cluster_hosts_by_short.get(_short) -%}
+            {% endif %}
+            <div class="svc">
+              <div class="svc-status"><span class="pill {{ r.status }}">{{ r.status }}</span></div>
+              <div class="svc-name">
+                <a href="{{ r.url }}" rel="noopener noreferrer" target="_blank">{{ r.name }}</a>
+                {% if rh %}
+                  <span class="host-badges">
+                    <span class="badge {{ 'badge-up' if rh.state == 'active' else 'badge-down' }}" title="Rancher state: {{ rh.state }}{% if rh.agent_state %} / agent {{ rh.agent_state }}{% endif %}">rancher: {{ rh.state }}</span>
+                    {% if rh.has_lb %}<span class="badge badge-info" title="vfb-loadbalancer-main running on this host">LB</span>{% endif %}
+                    {% if rh.is_dns_ingress %}<span class="badge badge-dns" title="virtualflybrain.org DNS A-record points at this host's agentIpAddress ({{ rh.agent_ip }})">DNS</span>{% endif %}
+                  </span>
+                {% endif %}
+                {% if r.error %}<div class="svc-err">{{ r.error }}</div>{% endif %}
+              </div>
+              <div class="svc-meta">
+                {{ r.http_status if r.http_status is not none else "—" }}
+                {% if r.latency_ms is not none %} · {{ r.latency_ms }} ms{% endif %}
+              </div>
+              <div class="strip-cell">
+                <div class="strip" aria-label="last {{ history_buckets }} buckets, oldest left">
+                  {% for b in row.buckets %}
+                    <div class="bucket {{ b }}" title="bucket {{ loop.index0 }}: {{ b }}"></div>
+                  {% endfor %}
+                </div>
+              </div>
+              <div class="uptime">
+                <span class="pct {% if row.uptime_24h is not none and row.uptime_24h < 99 %}low{% endif %}">24h {{ fmt_pct(row.uptime_24h, row.n_24h) }}</span>
+                <span>7d {{ fmt_pct(row.uptime_7d, row.n_7d) }}</span>
+                <span>30d {{ fmt_pct(row.uptime_30d, row.n_30d) }}</span>
+              </div>
+            </div>
+          {% endfor %}
+        </div>
+      </section>
+    {% endfor %}
+    {%- endmacro %}
+
+    {# Core user-facing renders first, before the specialised sections #}
+    {{ render_groups(priority_groups) }}
+
     {% macro fmt_int(n) -%}
       {%- if n is none -%}—{%- else -%}{{ "{:,}".format(n) }}{%- endif -%}
     {%- endmacro %}
@@ -352,8 +404,6 @@ <h2>Neo4j databases — content checks</h2>
             {% set spec = c.spec %}
             {% set s = c.summary %}
             {% set containers = c.containers %}
-            {% set first_err = containers|selectattr('error')|map(attribute='error')|first %}
-            {% set is_all_auth_err = containers|length > 0 and (containers|selectattr('error')|map(attribute='error')|select('search', '(401|Unauthorized|auth)')|list|length) == (containers|length) %}
             <article class="cache-card">
               <header>
                 <div>
@@ -369,12 +419,12 @@ <h2>Neo4j databases — content checks</h2>
                   <span class="pill up">online</span>
                 {% elif s.container_count == 0 %}
                   <span class="pill unknown">no data</span>
-                {% elif is_all_auth_err %}
+                {% elif s.is_all_auth_err %}
                   <span class="pill auth" title="auth failed — check `password:` or NEO4J_PASSWORD">auth?</span>
                 {% elif s.ok_count > 0 %}
                   <span class="pill down" title="{{ s.container_count - s.ok_count }} container(s) failing">partial</span>
                 {% else %}
-                  <span class="pill down" title="{{ first_err or 'down' }}">down</span>
+                  <span class="pill down" title="{{ s.first_err or 'down' }}">down</span>
                 {% endif %}
               </header>
 
@@ -432,7 +482,14 @@ <h2>Neo4j databases — content checks</h2>
                   <div class="chart-label">Node count over time (cluster max)</div>
                   {{ sparkline(c.count_pts, width=320, height=42, color="#7d4cd0", fill="rgba(125,76,208,0.15)") }}
                 </div>
-              {% elif s.container_count == 0 %}
+              {% endif %}
+              {% if c.latency_pts %}
+                <div class="cache-chart">
+                  <div class="chart-label">Query latency over time (cluster avg, ms)</div>
+                  {{ sparkline(c.latency_pts, width=320, height=42, color="#3498db", fill="rgba(52,152,219,0.15)") }}
+                </div>
+              {% endif %}
+              {% if not c.count_pts and not c.latency_pts and s.container_count == 0 %}
                 <div class="cache-err">no data yet — first probe pending</div>
               {% endif %}
             </article>
@@ -637,52 +694,8 @@ <h2>Caching services — load over time</h2>
       </section>
     {% endif %}
 
-    {% for group_name, rows in groups.items() %}
-      {% set is_rancher_group = group_name.startswith("Rancher servers") %}
-      <section>
-        <h2>{{ group_name }}{% if is_rancher_group and cluster_summary %} · LB on {{ cluster_summary.lb_coverage }}/{{ cluster_summary.hosts_total }} · {{ cluster_summary.dns_ingress_count }} on DNS{% endif %}</h2>
-        <div class="svc-list">
-          {% for row in rows %}
-            {% set r = row.result %}
-            {% set rh = none %}
-            {% if is_rancher_group %}
-              {%- set _short = r.name.split('.')[0].lower() -%}
-              {%- set rh = cluster_hosts_by_short.get(_short) -%}
-            {% endif %}
-            <div class="svc">
-              <div class="svc-status"><span class="pill {{ r.status }}">{{ r.status }}</span></div>
-              <div class="svc-name">
-                <a href="{{ r.url }}" rel="noopener noreferrer" target="_blank">{{ r.name }}</a>
-                {% if rh %}
-                  <span class="host-badges">
-                    <span class="badge {{ 'badge-up' if rh.state == 'active' else 'badge-down' }}" title="Rancher state: {{ rh.state }}{% if rh.agent_state %} / agent {{ rh.agent_state }}{% endif %}">rancher: {{ rh.state }}</span>
-                    {% if rh.has_lb %}<span class="badge badge-info" title="vfb-loadbalancer-main running on this host">LB</span>{% endif %}
-                    {% if rh.is_dns_ingress %}<span class="badge badge-dns" title="virtualflybrain.org DNS A-record points at this host's agentIpAddress ({{ rh.agent_ip }})">DNS</span>{% endif %}
-                  </span>
-                {% endif %}
-                {% if r.error %}<div class="svc-err">{{ r.error }}</div>{% endif %}
-              </div>
-              <div class="svc-meta">
-                {{ r.http_status if r.http_status is not none else "—" }}
-                {% if r.latency_ms is not none %} · {{ r.latency_ms }} ms{% endif %}
-              </div>
-              <div class="strip-cell">
-                <div class="strip" aria-label="last {{ history_buckets }} buckets, oldest left">
-                  {% for b in row.buckets %}
-                    <div class="bucket {{ b }}" title="bucket {{ loop.index0 }}: {{ b }}"></div>
-                  {% endfor %}
-                </div>
-              </div>
-              <div class="uptime">
-                <span class="pct {% if row.uptime_24h is not none and row.uptime_24h < 99 %}low{% endif %}">24h {{ fmt_pct(row.uptime_24h, row.n_24h) }}</span>
-                <span>7d {{ fmt_pct(row.uptime_7d, row.n_7d) }}</span>
-                <span>30d {{ fmt_pct(row.uptime_30d, row.n_30d) }}</span>
-              </div>
-            </div>
-          {% endfor %}
-        </div>
-      </section>
-    {% endfor %}
+    {# Remaining groups (everything except Core user-facing which is rendered at the top) #}
+    {{ render_groups(rest_groups) }}
 
     <footer>
       <a href="/api/status">/api/status</a> ·
diff --git a/config/services.yml b/config/services.yml
@@ -230,7 +230,11 @@ neo4j_services:
     min_nodes: 1
     timeout: 20
     fronts: Knowledge Base (curation graph)
-    # No rancher: block — KB is a single-instance service via LB.
+    # vfb-neo4j-kb-readonly is the public KB read-only cluster (scale 2).
+    rancher:
+      service_url: https://herd.virtualflybrain.org/v2-beta/projects/1a5/services/1s56
+      container_port: 7474
+      container_scheme: http
 
 # Application services exposing their own /status JSON (shape: vfbquery, ...).
 # Probed on the same schedule as everything else; metrics stored in
