Fix outdated ingestion pipeline docs (#903)

* Fix outdated live ping tables description in pipeline overview

The Decoded topic is read by the Kubernetes Live Sink (part of the
ingestion-sink service on GKE), not a Dataflow job. Update the prose and
relabel the diagram node accordingly.

Also drop the inaccurate 'guaranteed to contain a few days of history'
caveat: live tables are for low-latency access to recent data, not a
supported analysis interface. Point readers to the stable tables instead.

* Fix outdated Landfill sink description in pipeline overview

The Raw topic is read by the Kubernetes Raw Sink (part of the ingestion-sink
service), not a Dataflow job, and writes to the payload_bytes_raw dataset. The
'Landfill' name is historical. Update the prose and relabel the diagram nodes;
keep a parenthetical for the old name. Match gcp-ingestion's own terminology
('Kubernetes Raw Sink job') rather than the stale 'custom Java application'
phrasing from a 2020 transition note.

* Fix outdated edge server behavior in HTTP edge spec

Verified against gcp-ingestion ingestion-edge (create_app.py, publish.py,
util.py, dockerflow.py):

- GET /status does not exist. Health is served via Dockerflow endpoints
  (/__lbheartbeat__, /__heartbeat__, /__version__). Rewrite the GET request
  and response-code sections accordingly.
- The ?v=4 query parameter is not validated or rejected at the edge; the edge
  routes on path only and forwards the body opaquely. Describe the actual
  behavior.
- POST/PUT response codes: add 431 (attribute exceeds PubSub's 1024-byte
  limit) and 507 (local retry queue full), which the edge actually returns;
  drop 411 and 414, which are not enforced.
- Define the previously dangling [decoder] and [schemas repository]
  reference-style links.

* Fix outdated transform location in schemas doc

Verified against gcp-ingestion: ingestion-beam's PubsubMessageToTableRow is
now a thin wrapper that adds Beam metrics and converts to a BigQuery TableRow;
the transformation logic (map types, nested arrays, tuples, coercion,
additional_properties) moved to ingestion-core's PubsubMessageToObjectNode.
Point the doc there and fix the 'Thee'/'telemtry' typos in the paragraph.

Author: akkomar

.spelling

@@ -117,6 +117,7 @@ DNS
 DNSSEC
 docs.telemetry.mozilla.org
 Dockerfile
+Dockerflow
 dropdown
 DSRE
 DTMO

src/concepts/pipeline/gcp_data_pipeline.md

@@ -16,13 +16,13 @@ graph TD
 
 f1(fa:fa-firefox Firefox) -->|HTTP Post| d0(fa:fa-filter Ingestion Edge)
 d0 --> p1(fa:fa-stream Raw Topic)
-p1 --> d1(fa:fa-exchange-alt Landfill Sink)
-d1 --> b1(fa:fa-database Landfill BQ)
+p1 --> d1(fa:fa-exchange-alt Raw Sink)
+d1 --> b1(fa:fa-database Raw BQ)
 p1 --> d2(fa:fa-exchange-alt Decoder)
 d2 -->|success| p2(fa:fa-stream Decoded Topic)
 d2 -.->|fail| p3(fa:fa-stream Errors Topic)
 p3 --> d4(fa:fa-exchange-alt Errors Sink)
-p2 --> d3(fa:fa-exchange-alt BigQuery Sink)
+p2 --> d3(fa:fa-exchange-alt Live Sink)
 d3 --> b2(fa:fa-database Live Tables BQ)
 d4 --> b3(fa:fa-database Error Tables BQ)
 
@@ -76,8 +76,10 @@ message body of optionally-gzipped JSON.
 These messages are forwarded to a PubSub message queue with minimal processing,
 and made available in a **Raw** topic.
 
-A [Dataflow] job reads this topic and writes the raw messages to a BigQuery **Landfill** sink.
-This Landfill data is not used for analysis, but is stored in its raw form for
+A Kubernetes `Raw Sink` job (part of the [`ingestion-sink`][ingestion-sink]
+service) reads this topic and writes the raw messages to a BigQuery
+`payload_bytes_raw` sink (historically called _Landfill_).
+This raw data is not used for analysis, but is stored in its raw form for
 recovery and backfill purposes.
 
 If there is a processing error or data-loss downstream in the pipeline, this is an important fail-safe.
@@ -129,11 +131,13 @@ and the [schema generator].
 The result are tables that contains SQL-friendly field names for all known
 measures, as implemented in the [probe scraper].
 
-A [Dataflow] job reads from the Decoded topic and writes out to
+A Kubernetes `Live Sink` job (part of the [`ingestion-sink`][ingestion-sink]
+service) reads from the Decoded topic and writes out to
 **[live ping tables][table layout]**.
 These tables are updated frequently, and typically reflect data within a few
-minutes of it being ingested. They are optimized for accessing recent data,
-but are only guaranteed to contain a few days of history.
+minutes of it being ingested. They are intended for low-latency access to
+recent data and are not the supported interface for analysis; for that, use the
+**[historical ping tables][table layout]** (stable tables) described below.
 
 Historical raw ping data is stored in **[historical ping tables][table layout]**,
 also known as **stable tables**.
@@ -234,4 +238,5 @@ build and work with aggregate datasets rather than individual-level data.
 [norm]: ../channels/channel_normalization.md
 [bqe]: https://github.com/mozilla/bigquery-etl
 [gcp-ingestion]: https://github.com/mozilla/gcp-ingestion
+[ingestion-sink]: https://github.com/mozilla/gcp-ingestion/tree/main/ingestion-sink
 [looker]: https://mozilla.cloud.looker.com

src/concepts/pipeline/http_edge_spec.md

@@ -23,14 +23,23 @@ documents described by a [JSONSchema].
 
 ### GET Request
 
-Accept GET on `/status`, returning `OK` if all is well. This can be used to
-check the health of web servers.
+The edge server does not serve application data over GET. It exposes
+[Dockerflow] health-check endpoints:
+
+- `/__lbheartbeat__` - returns 200 unconditionally; used by the load balancer
+  and Kubernetes probes to confirm the server is up.
+- `/__heartbeat__` - returns 200 when internal checks pass (enough free disk
+  for the local queue, and the queue is accessible) and 500 otherwise.
+- `/__version__` - returns build and version information.
+
+[dockerflow]: https://github.com/mozilla-services/Dockerflow
 
 ### GET Response codes
 
-- _200_ - OK. `/status` and all's well
-- _404_ - Any GET other than `/status`
-- _500_ - All is not well
+- _200_ - a health check passed (`/__lbheartbeat__`, or `/__heartbeat__` when
+  all checks pass), or version info was returned (`/__version__`)
+- _404_ - any other GET path
+- _500_ - a `/__heartbeat__` check failed
 
 ### POST/PUT Request
 
@@ -67,19 +76,20 @@ A specific example:
 Here the `namespace` is fixed as "telemetry", and there is no `docVersion` in the URL.
 This means that incoming JSON documents must be parsed to determine the schema version
 to apply for validation. This logic is part of the downstream [decoder] job.
-Also note the required query parameter suffix `?v=4`.
-Documents sent under `/submit/telemetry` without `v=4` will be rejected at the edge.
+Also note the query parameter suffix `?v=4`, a historical convention sent by
+Firefox Desktop. The edge server routes on the URL path only and forwards the
+request body opaquely; the query string is preserved as request metadata.
 
 ### POST/PUT Response codes
 
 - _200_ - OK. Request accepted into the pipeline.
 - _400_ - Bad request, for example an un-encoded space in the URL.
 - _404_ - not found - POST/PUT to an unknown namespace
 - _405_ - wrong request type (anything other than POST/PUT)
-- _411_ - missing content-length header
-- _413_ - request body too large (Note that if we have badly-behaved clients that retry on `4XX`, we may opt to send back 202 on body/path too long).
-- _414_ - request path too long (See above)
+- _413_ - request body too large
+- _431_ - a request header (or other captured metadata attribute) exceeds PubSub's 1024 byte attribute limit
 - _500_ - internal error
+- _507_ - insufficient storage - the edge's local retry queue is full (for example during an extended PubSub outage)
 
 ### Supported HTTP Headers
 
@@ -132,3 +142,5 @@ is restored and messages can be flushed to the queue.
 Based on [past outages], this is typically a few hours or less.
 
 [past outages]: https://status.cloud.google.com/incident/cloud-pubsub
+[schemas repository]: https://github.com/mozilla-services/mozilla-pipeline-schemas
+[decoder]: gcp_data_pipeline.md#decoding

src/concepts/pipeline/schemas.md

@@ -236,10 +236,11 @@ LIMIT 1
 
 #### Data Structure Normalization
 
-Thee decoder is also responsible for transforming the data to
-accommodate BigQuery limitations in data representation. All transformations are defined in
-[`ingestion-beam` under
-`com.mozilla.telemtry.transforms.PubsubMessageToTableRow`](https://github.com/mozilla/gcp-ingestion/blob/main/ingestion-beam/src/main/java/com/mozilla/telemetry/transforms/PubsubMessageToTableRow.java).
+The decoder is also responsible for transforming the data to
+accommodate BigQuery limitations in data representation. Most transformations are implemented in
+[`ingestion-core` under
+`com.mozilla.telemetry.ingestion.core.transform.PubsubMessageToObjectNode`](https://github.com/mozilla/gcp-ingestion/blob/main/ingestion-core/src/main/java/com/mozilla/telemetry/ingestion/core/transform/PubsubMessageToObjectNode.java);
+`ingestion-beam`'s `PubsubMessageToTableRow` is a thin wrapper that adds Beam metrics and converts the result to a BigQuery `TableRow`.
 
 The following transformations are currently applied: