nozzle.html.md.erb

---
title: Logs, Metrics, and Nozzles
owner: Services
---


This topic explains how to integrate services with Cloud Foundry's logging system, _Loggregator_, by writing to and reading from its _Firehose_ endpoint.

## <a id="overview"></a> Overview

Cloud Foundry's Loggregator logging system collects logs and metrics from apps and platform components and streams them to a single endpoint, Firehose. Your tile can integrate its service with Loggregator in two ways:

* By sending your service component logs and metrics to the Firehose, to be streamed along with core platform component logs and metrics

* By installing a _nozzle_ on Firehose that directs Firehose data to be consumed by external services or apps -- a built-in nozzle can enable a service to:
  - Drain metrics to an external dashboard product for system operators
  - Send HTTP request details to search or analysis tools
  - Drain app logs to an external system
  - Auto-scale itself based on Firehose metrics, as detailed in this [YouTube video](https://youtu.be/skJKvQfpKD4?t=1021)

For a real world production example of a nozzle see [Firehose-to-syslog](https://github.com/cloudfoundry-community/firehose-to-syslog) in GitHub.

## <a id="firehose"></a> Firehose Communication

<%= vars.app_runtime_full %> components publish logs and metrics to the Firehose through Loggregator agent processes that run locally on the component VMs. Loggregator agents input the data to the Loggregator system through a co-located Loggregator agent. To see how logs and metrics travel from <%= vars.app_runtime_abbr %> system components to the Firehose, see the [Cloud Foundry documentation](https://docs.cloudfoundry.org/loggregator/architecture.html).

Component VMs running <%= vars.app_runtime_abbr %> services can publish logs and metrics the same way, by including the same component, Loggregator Agent. Historically, components used Metron for this communication.

### <a id="https"></a> HTTPS Protocol

To enable a service component to supply logs and metrics to the Firehose through encrypted communications,
do the following:

1. Include a Loggregator agent in the service component's template definitions.
<br><br>
    For example:

    ```
    name: service
    label: Service
    templates:
      - name: service
        release: service
    manifest: |
      - name: bpm
        release: bpm
        properties: {}
      - name: loggregator_agent
        release: loggregator-agent
        consumes:
          doppler:
            deployment: cf-e8e79eaed2a50130f206
        properties:
          deployment: generator
          loggregator:
            tls:
              ca_cert: (( $ops_manager.ca_certificate ))
              agent:
                cert: ((CERTIFICATE))
                key: ((KEY))
    ```

    Where `CERTIFICATE` and `KEY` are the values used for mutual TLS communication. For example, `.properties.agent_certificate.cert_pem` and `.properties.agent_certificate.private_key_pem`.

1. Make the Ops Manager CA certificate generate and sign the certificate needed for mutual TLS communication.
Do so with the following properties:

    ```
    - name: agent_certificate
      type: rsa_cert_credentials
      label: Agent Security Certificate
      configurable: false
      default:
        domains:
        - agent.(( ..cf.cloud_controller.system_domain.value ))
      description: mTLS Certificate for Agent
    ```

## <a id="nozzle"></a> Nozzles

A nozzle is a component dedicated to reading and processing data that streams from Firehose.
A service tile can install a nozzle as either a managed service, with package type `bosh-release`, or as an app pushed to <%= vars.app_runtime_abbr %>, with the package type `app`.

### <a id="develop"></a> Develop a Nozzle

<%= vars.company_name %> recommends developing a nozzle in Go to leverage the
[NOAA library](https://github.com/cloudfoundry/noaa).
NOAA does the heavy lifting of establishing
an authenticated websocket connection to the logging system
as well as de-serializing the protocol buffers.

Draining the logs consists of:

1. Authenticating
1. Establishing a connection to the logging system
1. Forwarding events on to their ultimate destination

Authenticate against the API (https://github.com/cloudfoundry-community/go-cfclient)
with a user in the `doppler.firehose` group:

```go
import "github.com/cloudfoundry-community/go-cfclient"

...

config := &cfclient.Config{
  ApiAddress:        apiUrl,
  Username:          username,
  Password:          password,
  SkipSslValidation: sslSkipVerify,
}

client, err := cfclient.NewClient(config)
```

Using the client's token, create a consumer and connect to Firehose with a subscription ID.
The ID is important because Firehose looks for connections with the same ID and only
sends an event to one of those connections. A nozzle developer can run two or more
instances to prevent message loss during upgrades an other deployments.

```go
token, err := client.GetToken()

consumer := consumer.New(config.TrafficControllerURL, &tls.Config{
  InsecureSkipVerify: config.SkipSSL,
}, nil)
events, errors := consumer.Firehose(firehoseSubscriptionID, token)
```

`Firehose` gives back two channels, one for events and one for errors.

The events channel receives the following six types of events.

* _ValueMetric_ represents some platform metric at a point in time, emitted by platform components. For example, how many `2xx` responses the router has sent out.
* _CounterEvent_ represents an incrementing counter, emitted by platform components. For example, a Diego cell's remaining memory capacity.
* _Error_ represents an error in the originating process.
* _HttpStartStop_ represents HTTP request details, including both app and platform requests.
* _LogMessage_ represents a log message for an individual app.
* _ContainerMetric_ represents application container information. For example, memory used.

For the full details on events, see
[dropsonde protocol](https://github.com/cloudfoundry/dropsonde-protocol/tree/master/events)
in GitHub.

The above events show how this data targets two different personae:
platform operators and app developers. Keep this in mind when designing an integration.

The `doppler.firehose` scope gets nozzle data for *every* app as well as the platform.
Any filtering based on the event payload is the nozzle implementor's responsibility.
An advanced integration could combine a
[service broker](service-brokers.html) with a nozzle to:

* Let app developers opt in to logging (implementing filtering in the nozzle)
* Establish [SSO](https://docs.cloudfoundry.org/services/dashboard-sso.html) exchange for authentication so that developers only can access logs for their space's apps

For a full working example (suitable as an integration starting point),
see [firehose-nozzle](https://github.com/cf-platform-eng/firehose-nozzle).

### <a id="deploy"></a> Deploy a Nozzle

Once you have built a nozzle, you can deploy it as a managed service or as an app.

Visit [managed service](managed.html) for more details on what it means to be a
managed service. See also this [example nozzle BOSH release](https://github.com/cloudfoundry-incubator/example-nozzle-release).

You can also deploy the nozzle as an app on <%= vars.app_runtime_abbr %>. Visit the Tile Generator's
[section on pushed apps](tile-generator.html#pushed-applications)
for more details.

### <a id="examples"></a> Example Nozzles

There are several open source examples you could use
as a reference for building your nozzle.

* [firehose-nozzle](https://github.com/cf-platform-eng/firehose-nozzle)
simply writes to standard out. It is a useful starting point as scaffolding, tests,
and more are already in place.

* [example-nozzle](https://github.com/cloudfoundry-incubator/example-nozzle) in a
single file implementation with no tests.

* [gcp-tools-release](https://github.com/cloudfoundry-community/gcp-tools-release)
drains component syslogs and health data in addition to nozzle data. It shows how
to work with a BOSH add-on for additional data outside a nozzle. The nozzle is
managed through BOSH. Raw logs and metrics data take different paths in the source.

* [firehose-to-syslog](https://github.com/cloudfoundry-community/firehose-to-syslog)
includes implementation code that adds additional metadata, which might be needed
for the access control list (ACL) app name, space UUID and name, and org UUID and name.

* [logsearch-for-cloudfoundry](https://github.com/cloudfoundry-community/logsearch-for-cloudfoundry)
 packages this nozzle as a BOSH release.

* [splunk-firehose-nozzle](https://github.com/cf-platform-eng/splunk-firehose-nozzle)
has source code based on `firehose-to-syslog` and is packaged to run an app on <%= vars.app_runtime_abbr %>.

* [datadog-firehose-nozzle](https://github.com/DataDog/datadog-firehose-nozzle) is
another real world implementation.

## <a id="syslog-format-pcf"></a> Log Format for <%= vars.app_runtime_abbr %> Components

The standard log format for <%= vars.app_runtime_abbr %> adheres to the [RFC-5424 syslog protocol](https://tools.ietf.org/html/rfc5424), with log messages formatted as follows:

`<${PRI}>${VERSION} ${TIMESTAMP} ${HOST_IP} ${APP_NAME} ${PROD_ID} ${MSG_ID} ${SD-ELEMENT-instance} ${MESSAGE}`

The [Syslog Message Elements table](#syslog-elements) immediately below describes each element of the log, and the [Structured Instance Data Format](#sd-element) table describes the contents of the structured data element that carries Cloud Foundry VM instance information.

### <a id="syslog-elements"></a> Syslog Message Elements

This table describes each element of a standard <%= vars.app_runtime_abbr %> syslog message.

<table id='syslog-elements-table' border="1" class="nice" >
  <tr>
    <th>Syslog Message Element</th>
    <th>Meaning or Value</th>
  </tr><tr>
    <td><code>${PRI}</code></td>
    <td><p><a href="https://tools.ietf.org/html/rfc5424#section-6.2.1">Priority value (PRI)</a>, calculated as <code>8 × Facility Code + Severity Code</code></p>
    <p><%= vars.app_runtime_abbr %> uses a Facility Code value of <code>1</code>, indicating a user-level facility. This adds <code>8</code> to the RFC-5424 Severity Codes, resulting in the numbers listed in the <a href="#severity-codes">table below</a>.</p>
    <p>If in doubt, default to <code>13</code>, to indicate Notice-level severity.</p>
</td>
  </tr><tr>
    <td><code>${VERSION}</code></td>
    <td><a href="https://tools.ietf.org/html/rfc5424#section-9.1"><code>1</code></a></td>
  </tr><tr>
    <td><code>${TIMESTAMP}</code></td>
    <td><p>The <a href="https://tools.ietf.org/html/rfc5424#section-9.1">timestamp</a> of when the log message is forwarded; typically slightly after it was generated. Example: <code>2017-07-24T05:14:15.000003Z</code></p></td>
  </tr><tr>
    <td><code>${HOST_IP}</code></td>
    <td><a href="https://tools.ietf.org/html/rfc5424#section-6.2.4">Internal IP address</a> of origin server</td>
  </tr><tr>
    <td><code>${APP_NAME}</code></td>
    <td><p><a href="https://tools.ietf.org/html/rfc5424#section-6.2.5">Process name</a>  of the program the generated the message.  Prefixed with <code>vcap</code>. For example:</p>
    <ul><li><code>vcap.rep</code></li>
    <li><code>vcap.garden</code></li>
    <li><code>vcap.cloud_controller_ng</code></li></ul>
    <p>
    You can derive this process name from either the program name configured for the local Metron agent or the <code>:progname</code>that blackbox derives from the directory that syslog-release forwards logs into.</p></td>
  </tr><tr>
    <td><code>${PROD_ID}</code></td>
    <td>The <a href="https://tools.ietf.org/html/rfc5424#section-6.2.6">Process ID</a> of the syslog process doing the forwarding. If this is not easily available, default to <code>-</code> (hyphen) to indicate unknown.</td>
  </tr><tr>
    <td><code>${MSG_ID}</code></td>
    <td>The <a href="https://tools.ietf.org/html/rfc5424#section-6.2.7">type</a> of log message. If this is not easily available, default to <code>-</code> (hyphen) to indicate unknown.</td>
  </tr><tr>
    <td><code>${SD-ELEMENT-instance}</code></td>
    <td>Structured data (SD) relevant to <%= vars.app_runtime_abbr %> about the <a href="https://tools.ietf.org/html/rfc5424#section-6.3.1">source instance (VM)</a> that originates the log message. See the <a href="#sd-element">Structured Instance Data Format table</a> below for content and format.</td>
  </tr><tr>
    <td><code>${MESSAGE}</code></td>
    <td>The log message itself, ideally in JSON</td>
  </tr>
</table>

### <a id="severity"></a> RFC-5424 Severity Codes

<%= vars.app_runtime_abbr %> components generate log messages with the following severity levels. The most common severity level is `13`.

<table id='severity-table' border="1" class="nice" >
  <tr>
    <th>Severity Code</th>
    <th>Meaning</th>
  </tr><tr>
    <td><code>8</code></td>
    <td>Emergency: system is unusable</td>
  </tr><tr>
    <td><code>9</code></td>
    <td>Alert: action must be taken immediately</td>
  </tr><tr>
    <td><code>10</code></td>
    <td>Critical: critical conditions</td>
  </tr><tr>
    <td><code>11</code></td>
    <td>Error: error conditions</td>
  </tr><tr>
    <td><code>12</code></td>
    <td>Warning: warning conditions</td>
  </tr><tr>
    <td><code>13</code></td>
    <td>Notice: normal but significant condition</td>
  </tr><tr>
    <td><code>14</code></td>
    <td>Informational: informational messages</td>
  </tr><tr>
    <td><code>15</code></td>
    <td>Debug: debug-level messages</td>
 </tr>
</table>

### <a id="sd-element"></a> Structured Instance Data Format

The RFC-5424 syslog protocol includes a [structured data element](https://tools.ietf.org/html/rfc5424#section-6.3.1) that people can use as they see fit. <%= vars.app_runtime_abbr %> uses this element to carry VM instance information as follows:

<table id='sd-element-table' border="1" class="nice" >
  <tr>
    <th><code>SD-ELEMENT-instance</code> element</th>
    <th>Meaning</th>
  </tr><tr>
    <td><code>${ENTERPRISE_ID}</code></td>
    <td>Your Enterprise Number, as <a href="https://www.iana.org/assignments/enterprise-numbers/enterprise-numbers">listed</a> by the Internet Assigned Numbers Authority (IANA) </td>
  </tr><tr>
    <td><code>${DIRECTOR}</code></td>
    <td>The BOSH director managing the deployment. </td>
  </tr><tr>
    <td><code>${DEPLOYMENT}</code></td>
    <td>BOSH <code>spec.deployment</code> value</td>
  </tr><tr>
    <td><code>${INSTANCE_GROUP}</code></td>
    <td>BOSH <code>instance_group</code>, currently <code>spec.job.name</code></td>
  </tr><tr>
    <td><code>${AVAILABILITY_ZONE}</code></td>
    <td>BOSH <code>spec.az</code> value</td>
  </tr><tr>
    <td><code>${ID}</code></td>
    <td>BOSH <code>spec.id</code> value. This is a UUID, not an index. It is necessary because BOSH Availability Zone index values are not always unique or sequential.</td>
 </tr>
</table>

## <a id="metrics"></a> Making Sense of Metrics

[Monitoring <%= vars.platform_name %>](https://docs.pivotal.io/pivotalcf/monitoring/index.html) has a great rundown of the
various metrics and how to make them useful.

## <a id="resources"></a> Other Resources

* CF Summit Video [Monitoring Cloud Foundry: Learning about the Firehose](https://youtu.be/skJKvQfpKD4)

* [Loggregator GitHub repository](https://github.com/cloudfoundry/loggregator/)

* [Overview of the Loggregator System](https://docs.cloudfoundry.org/loggregator/architecture.html)

* [Loggregator's Slack Channel](https://cloudfoundry.slack.com/messages/loggregator/)