Skip to content

Commit 060721f

Browse files
bneradtbneradt
authored
Propagate PROXY-Protocol src to outbound surfaces (#13120)
On listeners that receive PROXY-Protocol, the parsed source IP currently does not reach several origin-facing and operator-visible surfaces: the outbound `Client-ip` and `X-Forwarded-For` request headers always carry the immediate TCP peer (the CDN edge), and the outbound TPROXY local-bind address, the HostDB parent-selection affinity hash, and the Slow-Request error log all key off the TCP peer even on listeners that have opted into `:pp-clnt`. The `ssl_has_proxy_v1` debug line is also misleading: it fires for both PPv1 and PPv2 and only logs the destination. This aligns those outbound surfaces with the PROXY-Protocol source. `HttpTransact::add_client_ip_to_outgoing_request` is updated to prefer `pp_info.src_addr` whenever the user-agent connection has a parsed PROXY-Protocol header, mirroring `add_forwarded_field_to_request`, so the legacy headers agree with `Forwarded: for=` regardless of `:pp-clnt`. The outbound TPROXY local-bind, HostDB parent-selection affinity, and Slow-Request error log in `HttpSM.cc` are migrated to `t_state.effective_client_addr`, which is `:pp-clnt`-gated by construction so listeners without that flag are unaffected. The PP debug line is rewritten to print the actual PP version together with both `src` and `dst`. This extends the proxy_protocol autest with a PPv2-over-TLS session using a custom `src-addr` to lock in the new outbound-header behavior, and updates the admin-guide PROXY-Protocol page to clarify which surfaces are `:pp-clnt`-gated and which (`Client-ip`, `X-Forwarded-For`, `Forwarded: for=`) are unconditional.
1 parent dcec715 commit 060721f

12 files changed

Lines changed: 318 additions & 9 deletions

File tree

‎doc/admin-guide/configuration/proxy-protocol.en.rst‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@
1717
1818
.. include:: ../../common.defs
1919

20+
.. default-domain:: cpp
21+
2022
.. _proxy-protocol:
2123

2224
Proxy Protocol
@@ -59,6 +61,27 @@ enable PROXY protocol and want to apply ACL against the IP address delivered by
5961
If you specify the server_ports flag `pp-clnt` then the client IP address used for the
6062
transaction will be the one provided by proxy protocol.
6163

64+
The ``pp-clnt`` flag governs whether the operator-visible "client IP" is the
65+
PROXY-Protocol source address rather than the immediate TCP peer for the
66+
following surfaces:
67+
68+
* The squid log field ``%<chi>`` (and its derivatives ``%<chp>`` /
69+
``%<chh>``).
70+
* :func:`TSHttpTxnClientAddrGet`, :func:`TSHttpSsnClientAddrGet`, and
71+
:func:`TSNetVConnClientAddrGet` (the plugin-visible client address).
72+
* SNI ACL evaluation, the HTTP/2 ``PEER`` ACL, SSL diagnostics, and
73+
client-certificate validation.
74+
* Outbound transparency: when binding the proxy's outbound socket to the
75+
client's address (``addr_binding = FOREIGN_ADDR``), |TS| binds to the
76+
PROXY-Protocol source address only when ``pp-clnt`` is in effect.
77+
* HostDB parent-selection affinity: the consistent-hash key that groups
78+
requests onto the same upstream peer is the PROXY-Protocol source only
79+
when ``pp-clnt`` is in effect; otherwise it is the immediate TCP peer.
80+
81+
The new log field ``%<rchi>`` (and its derivatives ``%<rchp>`` /
82+
``%<rchh>``) always reports the immediate TCP peer regardless of
83+
``pp-clnt``.
84+
6285
1. HTTP Forwarded Header
6386

6487
The client IP address in the PROXY protocol header is passed to the origin server via an HTTP `Forwarded:
@@ -67,6 +90,13 @@ Detection of the PROXY protocol header is automatic. If the PROXY header
6790
precludes the request, it will automatically be parse and made available to the
6891
Forwarded: request header sent to the origin server.
6992

93+
The legacy outbound headers ``Client-ip:``
94+
(:ts:cv:`proxy.config.http.insert_client_ip`) and ``X-Forwarded-For:``
95+
(:ts:cv:`proxy.config.http.insert_squid_x_forwarded_for`) likewise carry
96+
the PROXY-Protocol source address whenever a PROXY-Protocol header is
97+
present, mirroring the ``Forwarded: for=`` parameter. This behavior is
98+
independent of the ``pp-clnt`` listener flag.
99+
70100
2. Outbound PROXY protocol
71101

72102
See :ts:cv:`proxy.config.http.proxy_protocol_out` for configuration information.

‎doc/admin-guide/files/records.yaml.en.rst‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@
1717
1818
.. include:: ../../common.defs
1919

20+
.. default-domain:: cpp
21+
2022
.. configfile:: records.yaml
2123

2224
records.yaml
@@ -844,6 +846,15 @@ pp
844846
:ref:`Proxy Protocol <proxy-protocol>` for more details on how to configure
845847
this option properly.
846848

849+
pp-clnt
850+
Use the source address from the Proxy Protocol header as the client IP
851+
address for the transaction. This affects which address is reported by the
852+
``%<chi>`` log field, by the plugin client-address APIs (such as
853+
:func:`TSHttpTxnClientAddrGet`), and by SNI / HTTP/2 peer ACLs and SSL
854+
client-certificate validation, among other surfaces. Only meaningful in
855+
combination with ``pp``. See :ref:`Proxy Protocol <proxy-protocol>` for the
856+
full enumeration of behaviors gated by this flag.
857+
847858
tr-full
848859
Fully transparent. This is a convenience option and is identical to specifying both ``tr-in`` and ``tr-out``.
849860

‎doc/developer-guide/api/functions/TSHttpSsnClientAddrGet.en.rst‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one or more
2+
contributor license agreements. See the NOTICE file distributed
3+
with this work for additional information regarding copyright
4+
ownership. The ASF licenses this file to you under the Apache
5+
License, Version 2.0 (the "License"); you may not use this file
6+
except in compliance with the License. You may obtain a copy of
7+
the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
14+
implied. See the License for the specific language governing
15+
permissions and limitations under the License.
16+
17+
18+
.. include:: ../../../common.defs
19+
20+
.. default-domain:: cpp
21+
22+
TSHttpSsnClientAddrGet
23+
**********************
24+
25+
Synopsis
26+
========
27+
28+
.. code-block:: cpp
29+
30+
#include <ts/ts.h>
31+
32+
.. function:: struct sockaddr const * TSHttpSsnClientAddrGet(TSHttpSsn ssnp)
33+
34+
Description
35+
===========
36+
37+
Return the socket address of the client for the HTTP session :arg:`ssnp`.
38+
The returned pointer references storage owned by |TS| and is only valid
39+
for the duration of the current callback; plugins that need to keep the
40+
value across callbacks must copy it into their own storage.
41+
42+
This is the session-level counterpart of :func:`TSHttpTxnClientAddrGet`
43+
and is appropriate when the caller has a session handle but no specific
44+
transaction (for example, in session-level hooks).
45+
46+
If the listener that accepted the connection has the ``pp-clnt`` flag set
47+
and a PROXY Protocol header was successfully parsed, the returned address
48+
is the PROXY-Protocol source address rather than the immediate TCP peer.
49+
Without ``pp-clnt`` the returned address is the immediate TCP peer even
50+
when PROXY Protocol is enabled. See :ref:`Proxy Protocol <proxy-protocol>`
51+
for the full enumeration of surfaces gated by ``pp-clnt``.
52+
53+
Return Value
54+
============
55+
56+
A pointer to the client address, or ``nullptr`` if :arg:`ssnp` is invalid
57+
or no client address is available.
58+
59+
See Also
60+
========
61+
62+
:manpage:`TSAPI(3ts)`,
63+
:func:`TSHttpTxnClientAddrGet`,
64+
:func:`TSNetVConnClientAddrGet`

‎doc/developer-guide/api/functions/TSHttpTxnClientAddrGet.en.rst‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one or more
2+
contributor license agreements. See the NOTICE file distributed
3+
with this work for additional information regarding copyright
4+
ownership. The ASF licenses this file to you under the Apache
5+
License, Version 2.0 (the "License"); you may not use this file
6+
except in compliance with the License. You may obtain a copy of
7+
the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
14+
implied. See the License for the specific language governing
15+
permissions and limitations under the License.
16+
17+
18+
.. include:: ../../../common.defs
19+
20+
.. default-domain:: cpp
21+
22+
TSHttpTxnClientAddrGet
23+
**********************
24+
25+
Synopsis
26+
========
27+
28+
.. code-block:: cpp
29+
30+
#include <ts/ts.h>
31+
32+
.. function:: struct sockaddr const * TSHttpTxnClientAddrGet(TSHttpTxn txnp)
33+
34+
Description
35+
===========
36+
37+
Return the socket address of the client that initiated the transaction
38+
:arg:`txnp`. The returned pointer references storage owned by |TS| and is
39+
only valid for the duration of the current callback; plugins that need to
40+
keep the value across callbacks must copy it into their own storage.
41+
42+
The returned ``struct sockaddr`` is address-family agnostic. Inspect the
43+
``sa_family`` field (or use the ``ats_ip_*`` helpers) to dispatch on IPv4
44+
versus IPv6.
45+
46+
If the listener that accepted the connection has the ``pp-clnt`` flag set
47+
and a PROXY Protocol header was successfully parsed, the returned address
48+
is the PROXY-Protocol source address rather than the immediate TCP peer.
49+
Without ``pp-clnt`` the returned address is the immediate TCP peer even
50+
when PROXY Protocol is enabled. See :ref:`Proxy Protocol <proxy-protocol>`
51+
for the full enumeration of surfaces gated by ``pp-clnt``.
52+
53+
Return Value
54+
============
55+
56+
A pointer to the client address, or ``nullptr`` if :arg:`txnp` is invalid
57+
or no client address is available.
58+
59+
See Also
60+
========
61+
62+
:manpage:`TSAPI(3ts)`,
63+
:func:`TSHttpSsnClientAddrGet`,
64+
:func:`TSNetVConnClientAddrGet`

‎doc/developer-guide/api/functions/TSNetVConnClientAddrGet.en.rst‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one or more
2+
contributor license agreements. See the NOTICE file distributed
3+
with this work for additional information regarding copyright
4+
ownership. The ASF licenses this file to you under the Apache
5+
License, Version 2.0 (the "License"); you may not use this file
6+
except in compliance with the License. You may obtain a copy of
7+
the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
14+
implied. See the License for the specific language governing
15+
permissions and limitations under the License.
16+
17+
18+
.. include:: ../../../common.defs
19+
20+
.. default-domain:: cpp
21+
22+
TSNetVConnClientAddrGet
23+
***********************
24+
25+
Synopsis
26+
========
27+
28+
.. code-block:: cpp
29+
30+
#include <ts/ts.h>
31+
32+
.. function:: struct sockaddr const * TSNetVConnClientAddrGet(TSVConn vc)
33+
34+
Description
35+
===========
36+
37+
Return the socket address of the client for the network virtual connection
38+
:arg:`vc`. The returned pointer references storage owned by |TS| and is
39+
only valid for the duration of the current callback; plugins that need to
40+
keep the value across callbacks must copy it into their own storage.
41+
42+
This is the VConn-level counterpart of :func:`TSHttpTxnClientAddrGet` and
43+
is appropriate in low-level hooks (for example, SSL or TCP hooks) where
44+
no HTTP session or transaction is yet associated with the connection.
45+
46+
If the listener that accepted the connection has the ``pp-clnt`` flag set
47+
and a PROXY Protocol header was successfully parsed, the returned address
48+
is the PROXY-Protocol source address rather than the immediate TCP peer.
49+
Without ``pp-clnt`` the returned address is the immediate TCP peer even
50+
when PROXY Protocol is enabled. Use ``TSNetVConnRemoteAddrGet()`` if the
51+
immediate TCP peer is always required regardless of ``pp-clnt``. See
52+
:ref:`Proxy Protocol <proxy-protocol>` for the full enumeration of
53+
surfaces gated by ``pp-clnt``.
54+
55+
Return Value
56+
============
57+
58+
A pointer to the client address, or ``nullptr`` if :arg:`vc` is invalid
59+
or no client address is available.
60+
61+
See Also
62+
========
63+
64+
:manpage:`TSAPI(3ts)`,
65+
:func:`TSHttpTxnClientAddrGet`,
66+
:func:`TSHttpSsnClientAddrGet`

‎src/iocore/net/SSLNetVConnection.cc‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -376,11 +376,15 @@ SSLNetVConnection::read_raw_data()
376376
if (this->has_proxy_protocol(buffer, &r)) {
377377
Dbg(dbg_ctl_proxyprotocol, "ssl has proxy protocol header");
378378
if (dbg_ctl_proxyprotocol.on()) {
379+
IpEndpoint src;
380+
src.sa = *(this->get_proxy_protocol_src_addr());
379381
IpEndpoint dst;
380382
dst.sa = *(this->get_proxy_protocol_dst_addr());
381-
ip_port_text_buffer ipb1;
382-
ats_ip_nptop(&dst, ipb1, sizeof(ipb1));
383-
DbgPrint(dbg_ctl_proxyprotocol, "ssl_has_proxy_v1, dest IP received [%s]", ipb1);
383+
ip_port_text_buffer src_ipb, dst_ipb;
384+
ats_ip_nptop(&src, src_ipb, sizeof(src_ipb));
385+
ats_ip_nptop(&dst, dst_ipb, sizeof(dst_ipb));
386+
DbgPrint(dbg_ctl_proxyprotocol, "ssl proxy protocol v%d header parsed: src=[%s] dst=[%s]",
387+
static_cast<int>(this->get_proxy_protocol_version()), src_ipb, dst_ipb);
384388
}
385389
} else {
386390
Dbg(dbg_ctl_proxyprotocol, "proxy protocol was enabled, but Proxy Protocol header was not present");

‎src/proxy/http/HttpSM.cc‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2411,7 +2411,10 @@ HttpSM::process_hostdb_info(HostDBRecord *record)
24112411
}
24122412

24132413
if (record && !record->is_failed()) {
2414-
t_state.dns_info.inbound_remote_addr = &t_state.client_info.src_addr.sa;
2414+
// HostDB parent-selection hashes by inbound_remote_addr; using
2415+
// effective_client_addr keeps sharding affinity tied to the real client
2416+
// when :pp-clnt is in effect, and is the TCP peer otherwise.
2417+
t_state.dns_info.inbound_remote_addr = &t_state.effective_client_addr.sa;
24152418
if (!use_client_addr) {
24162419
t_state.dns_info.set_active(
24172420
record->select_best_http(ts_clock::now(), t_state.txn_conf->down_server_timeout, t_state.dns_info.inbound_remote_addr));
@@ -5759,7 +5762,11 @@ HttpSM::do_http_server_open(bool raw, bool only_direct)
57595762
opt.local_ip = outbound_ip;
57605763
} else if (_ua.get_txn()->is_outbound_transparent()) {
57615764
opt.addr_binding = NetVCOptions::FOREIGN_ADDR;
5762-
opt.local_ip = t_state.client_info.src_addr;
5765+
// Use effective_client_addr so that listeners with :pp-clnt bind the
5766+
// outbound socket to the real client IP rather than the immediate TCP
5767+
// peer; without :pp-clnt, effective_client_addr is the TCP peer, so
5768+
// existing transparent deployments are unaffected.
5769+
opt.local_ip = t_state.effective_client_addr;
57635770
/* If the connection is server side transparent, we can bind to the
57645771
port that the client chose instead of randomly assigning one at
57655772
the proxy. This is controlled by the 'use_client_source_port'
@@ -7869,7 +7876,9 @@ HttpSM::update_stats()
78697876
}
78707877
int status = static_cast<int>(status_code);
78717878
char client_ip[INET6_ADDRSTRLEN];
7872-
ats_ip_ntop(&t_state.client_info.src_addr, client_ip, sizeof(client_ip));
7879+
// Log the operator-visible client IP (matches %<chi>) so the slow-request
7880+
// line is consistent with squid logs when :pp-clnt is in effect.
7881+
ats_ip_ntop(&t_state.effective_client_addr, client_ip, sizeof(client_ip));
78737882
Error("[%" PRId64 "] Slow Request: "
78747883
"client_ip: %s:%u "
78757884
"protocol: %s "
@@ -7902,7 +7911,7 @@ HttpSM::update_stats()
79027911
"sm_finish: %.3f "
79037912
"plugin_active: %.3f "
79047913
"plugin_total: %.3f",
7905-
sm_id, client_ip, t_state.client_info.src_addr.host_order_port(),
7914+
sm_id, client_ip, t_state.effective_client_addr.host_order_port(),
79067915
_ua.get_txn() ? _ua.get_txn()->get_protocol_string() : "-1", url_string, status, unique_id_string, redirection_tries,
79077916
client_response_body_bytes, fd, t_state.client_info.state, t_state.server_info.state,
79087917
milestones.difference_sec(TS_MILESTONE_TLS_HANDSHAKE_START, TS_MILESTONE_TLS_HANDSHAKE_END),

‎src/proxy/http/HttpTransact.cc‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5496,12 +5496,23 @@ HttpTransact::add_client_ip_to_outgoing_request(State *s, HTTPHdr *request)
54965496
char ip_string[INET6_ADDRSTRLEN + 1] = {'\0'};
54975497
size_t ip_string_size = 0;
54985498

5499-
if (!ats_is_ip(&s->client_info.src_addr.sa)) {
5499+
// Prefer the PROXY-Protocol source address when one is present, so that
5500+
// Client-ip and X-Forwarded-For agree with Forwarded: for= regardless of
5501+
// whether the listener carries the :pp-clnt flag.
5502+
IpEndpoint src_addr = s->client_info.src_addr;
5503+
if (s->state_machine->get_ua_txn() && s->state_machine->get_ua_txn()->get_netvc()) {
5504+
const ProxyProtocol &pp = s->state_machine->get_ua_txn()->get_netvc()->get_proxy_protocol_info();
5505+
if (pp.version != ProxyProtocolVersion::UNDEFINED) {
5506+
src_addr = pp.src_addr;
5507+
}
5508+
}
5509+
5510+
if (!ats_is_ip(&src_addr.sa)) {
55005511
return;
55015512
}
55025513

55035514
// Always prepare the IP string.
5504-
if (ats_ip_ntop(&s->client_info.src_addr.sa, ip_string, sizeof(ip_string)) != nullptr) {
5515+
if (ats_ip_ntop(&src_addr.sa, ip_string, sizeof(ip_string)) != nullptr) {
55055516
ip_string_size += strlen(ip_string);
55065517
} else {
55075518
// Failure, omg

‎tests/gold_tests/proxy_protocol/gold/access-cp.gold‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,4 @@
55
198.51.100.1 198.51.100.1 127.0.0.1
66
127.0.0.1 0 127.0.0.1
77
127.0.0.1 0 127.0.0.1
8+
198.51.100.1 198.51.100.1 127.0.0.1

‎tests/gold_tests/proxy_protocol/gold/access-nocp.gold‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,4 @@
55
127.0.0.1 198.51.100.1 127.0.0.1
66
127.0.0.1 0 127.0.0.1
77
127.0.0.1 0 127.0.0.1
8+
127.0.0.1 198.51.100.1 127.0.0.1

0 commit comments

Comments
 (0)