Skip to content

capturing-response.md

Latest commit

 

History

History
459 lines (363 loc) · 11.7 KB

capturing-response.md

File metadata and controls

459 lines (363 loc) · 11.7 KB
layout doc
title Capturing Response
section File Format

Capturing Response

Captures

Captures are optional values that are extracted from the HTTP response and stored in a named variable. These captures may be the response status code, part of or the entire the body, and response headers.

Captured variables can be accessed through a run session; each new value of a given variable overrides the last value.

Captures can be useful for using data from one request in another request, such as when working with CSRF tokens. Variables in a Hurl file can be created from captures or [injected into the session].

{% raw %}

# An example to show how to pass a CSRF token
# from one request to another:

# First GET request to get CSRF token value:
GET https://example.org
HTTP 200
# Capture the CSRF token value from html body.
[Captures]
csrf_token: xpath "normalize-space(//meta[@name='_csrf_token']/@content)"

# Do the login !
POST https://acmecorp.net/login?user=toto&password=1234
X-CSRF-TOKEN: {{csrf_token}}
HTTP 302

{% endraw %}

Structure of a capture:

my_varvariable : xpath "string(//h1)"query

A capture consists of a variable name, followed by : and a query. Captures section starts with [Captures].

Query

Queries are used to extract data from an HTTP response.

A query can extract data from

Extracted data can then be further refined using [filters].

Status capture

Capture the received HTTP response status code. Status capture consists of a variable name, followed by a :, and the keyword status.

GET https://example.org
HTTP 200
[Captures]
my_status: status

Version capture

Capture the received HTTP version. Version capture consists of a variable name, followed by a :, and the keyword version. The value captured is a string:

GET https://example.org
HTTP 200
[Captures]
http_version: version

Header capture

Capture a header from the received HTTP response headers. Header capture consists of a variable name, followed by a :, then the keyword header and a header name.

POST https://example.org/login
[Form]
user: toto
password: 12345678
HTTP 302
[Captures]
next_url: header "Location"

Cookie capture

Capture a [Set-Cookie] header from the received HTTP response headers. Cookie capture consists of a variable name, followed by a :, then the keyword cookie and a cookie name.

GET https://example.org/cookies/set
HTTP 200
[Captures]
session-id: cookie "LSID"

Cookie attributes value can also be captured by using the following format: <cookie-name>[cookie-attribute]. The following attributes are supported: Value, Expires, Max-Age, Domain, Path, Secure, HttpOnly and SameSite.

GET https://example.org/cookies/set
HTTP 200
[Captures]
value1: cookie "LSID"
value2: cookie "LSID[Value]"     # Equivalent to the previous capture
expires: cookie "LSID[Expires]"
max-age: cookie "LSID[Max-Age]"
domain: cookie "LSID[Domain]"
path: cookie "LSID[Path]"
secure: cookie "LSID[Secure]"
http-only: cookie "LSID[HttpOnly]"
same-site: cookie "LSID[SameSite]"

Body capture

Capture the entire body (decoded as text) from the received HTTP response. The encoding used to decode the body is based on the charset value in the Content-Type header response.

GET https://example.org/home
HTTP 200
[Captures]
my_body: body

If the Content-Type doesn't include any encoding hint, a [decode filter] can be used to explicitly decode the body response bytes.

# Our HTML response is encoded using GB 2312.
# But, the 'Content-Type' HTTP response header doesn't precise any charset,
# so we decode explicitly the bytes.
GET https://example.org/cn
HTTP 200
[Captures]
my_body: bytes decode "gb2312"

Bytes capture

Capture the entire body (as a raw bytestream) from the received HTTP response

GET https://example.org/data.bin
HTTP 200
[Captures]
my_data: bytes

XPath capture

Capture a [XPath] query from the received HTTP body decoded as a string. Currently, only XPath 1.0 expression can be used.

{% raw %}

GET https://example.org/home
# Capture the identifier from the dom node <div id="pet0">5646eaf23</div
HTTP 200
[Captures]
pet-id: xpath "normalize-space(//div[@id='pet0'])"

# Open the captured page.
GET https://example.org/home/pets/{{pet-id}}
HTTP 200

{% endraw %}

XPath captures are not limited to node values (like string, or boolean); any valid XPath can be captured and asserted with variable asserts.

# Test that the XML endpoint return 200 pets
GET https://example.org/api/pets
HTTP 200
[Captures]
pets: xpath "//pets"
[Asserts]
variable "pets" count == 200

XPath expression can also be evaluated against part of the body with a [xpath filter]:

GET https://example.org/home_cn
HTTP 200
[Captures]
pet-id: bytes decode "gb2312" xpath "normalize-space(//div[@id='pet0'])"

JSONPath capture

Capture a [JSONPath] query from the received HTTP body.

{% raw %}

POST https://example.org/api/contact
[Form]
token: {{token}}
email: toto@rookie.net
HTTP 200
[Captures]
contact-id: jsonpath "$['id']"

{% endraw %}

Explain that the value selected by the JSONPath is coerced to a string when only one node is selected.

As with [XPath captures], JSONPath captures can be anything from string, number, to object and collections. For instance, if we have a JSON endpoint that returns the following JSON:

{
  "a_null": null,
  "an_object": {
    "id": "123"
  },
  "a_list": [
    1,
    2,
    3
  ],
  "an_integer": 1,
  "a float": 1.1,
  "a_bool": true,
  "a_string": "hello"
}

We can capture the following paths:

GET https://example.org/captures-json
HTTP 200
[Captures]
an_object:  jsonpath "$['an_object']"
a_list:     jsonpath "$['a_list']"
a_null:     jsonpath "$['a_null']"
an_integer: jsonpath "$['an_integer']"
a_float:    jsonpath "$['a_float']"
a_bool:     jsonpath "$['a_bool']"
a_string:   jsonpath "$['a_string']"
all:        jsonpath "$"

Regex capture

Capture a regex pattern from the HTTP received body, decoded as text.

GET https://example.org/helloworld
HTTP 200
[Captures]
id_a: regex "id_a:([0-9]+)"
id_b: regex "id_b:(\\d+)"   # pattern using double quote 
id_c: regex /id_c:(\d+)/    # pattern using forward slash
name: regex "Hello ([a-zA-Z]+)"

The regex pattern must have at least one capture group, otherwise the capture will fail. When the pattern is a double-quoted string, metacharacters beginning with a backslash in the pattern (like \d, \s) must be escaped; literal pattern enclosed by / can also be used to avoid metacharacters escaping.

The regex syntax is documented at https://docs.rs/regex/latest/regex/#syntax. For instance, one can use flags to enable case-insensitive match:

GET https://example.org/hello
HTTP 200
[Captures]
word: regex /(?i)hello (\w+)!/

SHA-256 capture

Capture the [SHA-256] hash of the response body.

GET https://example.org/data.tar.gz
HTTP 200
[Captures]
my_hash: sha256

Like body assert, sha256 capture works after content encoding decompression (so the captured value is not affected by Content-Encoding response header).

MD5 capture

Capture the [MD5] hash of the response body.

GET https://example.org/data.tar.gz
HTTP 200
[Captures]
my_hash: md5

Like sha256 asserts, md5 assert works after content encoding decompression (so the predicates values are not affected by Content-Encoding response header)

URL capture

Capture the last fetched URL. This is most meaningful if you have told Hurl to follow redirection (see [[Options] section][options] or [--location option]). URL capture consists of a variable name, followed by a :, and the keyword url.

GET https://example.org/redirecting
[Options]
location: true
HTTP 200
[Captures]
landing_url: url

Redirects capture

Capture each step of redirection. This is most meaningful if you have told Hurl to follow redirection (see [[Options]section][options] or [--location option]). Redirects capture consists of a variable name, followed by a :, and the keyword redirects. Redirects query returns a collection so each step of the redirection can be capture.

GET https://example.org/redirecting/1
[Options]
location: true
HTTP 200
[Asserts]
redirects count == 3
[Captures]
step1: redirects nth 0 location
step2: redirects nth 1 location
step3: redirects nth 2 location

IP address capture

Capture the IP address of the last connection. The value of the ip query is a string.

GET https://example.org/hello
HTTP 200
[Captures]
server_ip: ip

Variable capture

Capture the value of a variable into another.

GET https://example.org/helloworld
HTTP 200
[Captures]
in: body
name: variable "in"

Duration capture

Capture the response time of the request in ms.

GET https://example.org/helloworld
HTTP 200
[Captures]
duration_in_ms: duration

SSL certificate capture

Capture the SSL certificate properties. Certificate capture consists of the keyword certificate, followed by the certificate attribute value.

The following attributes are supported: Subject, Issuer, Start-Date, Expire-Date and Serial-Number.

GET https://example.org
HTTP 200
[Captures]
cert_subject: certificate "Subject"
cert_issuer: certificate "Issuer"
cert_expire_date: certificate "Expire-Date"
cert_serial_number: certificate "Serial-Number"

Redacting Secrets

When capturing data, you may need to hide captured values from logs and report. To do this, captures can use secrets which are redacted from logs and reports, using [--secret option]:

$ hurl --secret pass=sesame-ouvre-toi file.hurl

If the secret value to be redacted is dynamic, or not known before execution, a capture can become a secret using redact at the end of the query's capture:

GET https://foo.com
HTTP 200
[Captures]
pass: header "token" redact

[injected into the session]: {% link _docs/templates.md %}#injecting-variables [Set-Cookie]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie [XPath]: https://en.wikipedia.org/wiki/XPath [JSONPath]: https://goessner.net/articles/JsonPath/ [XPath captures]: #xpath-capture [JavaScript-like Regular expression syntax]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions [options]: {% link _docs/request.md %}#options [--location option]: {% link _docs/manual.md %}#location [filters]: {% link _docs/filters.md %} [xpath filter]: {% link _docs/filters.md %}#xpath [decode filter]: {% link _docs/filters.md %}#decode [--secret option]: {% link _docs/templates.md %}#secrets [MD5]: https://en.wikipedia.org/wiki/MD5 [SHA-256]: https://en.wikipedia.org/wiki/SHA-2