Skip to content

Commit e667053

Browse files
rkline0xrkline0x
committed
1 parent ac0c66c commit e667053

7 files changed

Lines changed: 328 additions & 58 deletions

File tree

‎changelog/index.html‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1409,6 +1409,17 @@
14091409
</label>
14101410
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
14111411

1412+
<li class="md-nav__item">
1413+
<a href="#unreleased" class="md-nav__link">
1414+
<span class="md-ellipsis">
1415+
1416+
Unreleased
1417+
1418+
</span>
1419+
</a>
1420+
1421+
</li>
1422+
14121423
<li class="md-nav__item">
14131424
<a href="#490" class="md-nav__link">
14141425
<span class="md-ellipsis">
@@ -1591,6 +1602,17 @@
15911602
</label>
15921603
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
15931604

1605+
<li class="md-nav__item">
1606+
<a href="#unreleased" class="md-nav__link">
1607+
<span class="md-ellipsis">
1608+
1609+
Unreleased
1610+
1611+
</span>
1612+
</a>
1613+
1614+
</li>
1615+
15941616
<li class="md-nav__item">
15951617
<a href="#490" class="md-nav__link">
15961618
<span class="md-ellipsis">
@@ -1733,6 +1755,15 @@
17331755

17341756

17351757
<h1 id="changelog">Changelog<a class="headerlink" href="#changelog" title="Permanent link">&para;</a></h1>
1758+
<h2 id="unreleased">Unreleased<a class="headerlink" href="#unreleased" title="Permanent link">&para;</a></h2>
1759+
<p>Graceful connection draining on secret removal (<a href="https://github.com/teleproxy/teleproxy/issues/45">#45</a>).</p>
1760+
<ul>
1761+
<li><strong>Zero-downtime secret rotation</strong> — removing a secret from <code>config.toml</code> and sending SIGHUP no longer drops the in-flight connections that were authenticated under it. The slot enters a "draining" state: new connections matching that secret are rejected, but existing ones keep working until they close on their own or <code>drain_timeout_secs</code> (default 300, <code>0</code> = infinite) elapses. Re-adding a draining secret revives the same slot — counters and IP tracking carry over.</li>
1762+
<li>New TOML option <code>drain_timeout_secs</code> (reloadable). Pinned <code>-S</code> CLI secrets are immutable across SIGHUP and never drain.</li>
1763+
<li>New stats: <code>secret_&lt;lbl&gt;_draining</code>, <code>secret_&lt;lbl&gt;_drain_age_seconds</code>, <code>secret_&lt;lbl&gt;_rejected_draining</code>, <code>secret_&lt;lbl&gt;_drain_forced</code>. Same fields exposed in Prometheus as <code>teleproxy_secret_draining</code>, <code>teleproxy_secret_drain_age_seconds</code>, <code>teleproxy_secret_rejected_draining_total</code>, <code>teleproxy_secret_drain_forced_total</code>.</li>
1764+
<li>Slot capacity expanded internally: 16 active secrets at a time as before, plus up to 16 additional draining slots.</li>
1765+
<li>Fixes a latent bug where the per-secret connection counter could go negative if a TLS connection was closed between the TLS handshake and the obfs2 init.</li>
1766+
</ul>
17361767
<h2 id="490">4.9.0<a class="headerlink" href="#490" title="Permanent link">&para;</a></h2>
17371768
<p>PROXY protocol v1/v2 listener support (<a href="https://github.com/teleproxy/teleproxy/issues/50">#50</a>) and per-IP top-N Prometheus metrics (<a href="https://github.com/teleproxy/teleproxy/issues/46">#46</a>).</p>
17381769
<ul>

‎fa/changelog/index.html‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1409,6 +1409,17 @@
14091409
</label>
14101410
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
14111411

1412+
<li class="md-nav__item">
1413+
<a href="#unreleased" class="md-nav__link">
1414+
<span class="md-ellipsis">
1415+
1416+
Unreleased
1417+
1418+
</span>
1419+
</a>
1420+
1421+
</li>
1422+
14121423
<li class="md-nav__item">
14131424
<a href="#490" class="md-nav__link">
14141425
<span class="md-ellipsis">
@@ -1591,6 +1602,17 @@
15911602
</label>
15921603
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
15931604

1605+
<li class="md-nav__item">
1606+
<a href="#unreleased" class="md-nav__link">
1607+
<span class="md-ellipsis">
1608+
1609+
Unreleased
1610+
1611+
</span>
1612+
</a>
1613+
1614+
</li>
1615+
15941616
<li class="md-nav__item">
15951617
<a href="#490" class="md-nav__link">
15961618
<span class="md-ellipsis">
@@ -1733,6 +1755,15 @@
17331755

17341756

17351757
<h1 id="changelog">Changelog<a class="headerlink" href="#changelog" title="Permanent link">&para;</a></h1>
1758+
<h2 id="unreleased">Unreleased<a class="headerlink" href="#unreleased" title="Permanent link">&para;</a></h2>
1759+
<p>Graceful connection draining on secret removal (<a href="https://github.com/teleproxy/teleproxy/issues/45">#45</a>).</p>
1760+
<ul>
1761+
<li><strong>Zero-downtime secret rotation</strong> — removing a secret from <code>config.toml</code> and sending SIGHUP no longer drops the in-flight connections that were authenticated under it. The slot enters a "draining" state: new connections matching that secret are rejected, but existing ones keep working until they close on their own or <code>drain_timeout_secs</code> (default 300, <code>0</code> = infinite) elapses. Re-adding a draining secret revives the same slot — counters and IP tracking carry over.</li>
1762+
<li>New TOML option <code>drain_timeout_secs</code> (reloadable). Pinned <code>-S</code> CLI secrets are immutable across SIGHUP and never drain.</li>
1763+
<li>New stats: <code>secret_&lt;lbl&gt;_draining</code>, <code>secret_&lt;lbl&gt;_drain_age_seconds</code>, <code>secret_&lt;lbl&gt;_rejected_draining</code>, <code>secret_&lt;lbl&gt;_drain_forced</code>. Same fields exposed in Prometheus as <code>teleproxy_secret_draining</code>, <code>teleproxy_secret_drain_age_seconds</code>, <code>teleproxy_secret_rejected_draining_total</code>, <code>teleproxy_secret_drain_forced_total</code>.</li>
1764+
<li>Slot capacity expanded internally: 16 active secrets at a time as before, plus up to 16 additional draining slots.</li>
1765+
<li>Fixes a latent bug where the per-secret connection counter could go negative if a TLS connection was closed between the TLS handshake and the obfs2 init.</li>
1766+
</ul>
17361767
<h2 id="490">4.9.0<a class="headerlink" href="#490" title="Permanent link">&para;</a></h2>
17371768
<p>PROXY protocol v1/v2 listener support (<a href="https://github.com/teleproxy/teleproxy/issues/50">#50</a>) and per-IP top-N Prometheus metrics (<a href="https://github.com/teleproxy/teleproxy/issues/46">#46</a>).</p>
17381769
<ul>

‎fa/features/secrets/index.html‎

Lines changed: 78 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1157,6 +1157,17 @@
11571157
</span>
11581158
</a>
11591159

1160+
</li>
1161+
1162+
<li class="md-nav__item">
1163+
<a href="#graceful-draining-on-removal" class="md-nav__link">
1164+
<span class="md-ellipsis">
1165+
1166+
Graceful Draining on Removal
1167+
1168+
</span>
1169+
</a>
1170+
11601171
</li>
11611172

11621173
<li class="md-nav__item">
@@ -1673,6 +1684,17 @@
16731684
</span>
16741685
</a>
16751686

1687+
</li>
1688+
1689+
<li class="md-nav__item">
1690+
<a href="#graceful-draining-on-removal" class="md-nav__link">
1691+
<span class="md-ellipsis">
1692+
1693+
Graceful Draining on Removal
1694+
1695+
</span>
1696+
</a>
1697+
16761698
</li>
16771699

16781700
<li class="md-nav__item">
@@ -1865,13 +1887,50 @@ <h2 id="secret-expiration">Secret Expiration<a class="headerlink" href="#secret-
18651887
<li><strong>Stats:</strong> <code>secret_temp_expires 1751327999</code>, <code>secret_temp_rejected_expired 12</code></li>
18661888
<li><strong>Prometheus:</strong> <code>teleproxy_secret_expires_timestamp{secret="temp"} 1751327999</code></li>
18671889
</ul>
1890+
<h2 id="graceful-draining-on-removal">Graceful Draining on Removal<a class="headerlink" href="#graceful-draining-on-removal" title="Permanent link">&para;</a></h2>
1891+
<p>When a secret is removed from the TOML and you <code>kill -HUP</code> the proxy, in-flight
1892+
connections that authenticated under that secret keep working — the slot
1893+
transitions to a draining state, new connections matching the removed secret
1894+
are rejected, and the slot is released only after all existing connections
1895+
have closed (or after <code>drain_timeout_secs</code> seconds, whichever comes first).
1896+
This enables zero-downtime secret rotation without dropping users mid-session.</p>
1897+
<p>TOML config:</p>
1898+
<div class="highlight"><pre><span></span><code><a id="__codelineno-17-1" name="__codelineno-17-1" href="#__codelineno-17-1"></a><span class="n">drain_timeout_secs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">300</span><span class="w"> </span><span class="c1"># 0 = infinite (never force-close), default 300</span>
1899+
</code></pre></div>
1900+
<p>Workflow for rotating a secret without dropping users:</p>
1901+
<ol>
1902+
<li>Add the new secret to <code>config.toml</code> alongside the old one.</li>
1903+
<li><code>kill -HUP $(pidof teleproxy)</code> — both secrets are now active.</li>
1904+
<li>Update clients to the new secret at your own pace.</li>
1905+
<li>Remove the old secret from <code>config.toml</code>.</li>
1906+
<li><code>kill -HUP</code> again — old secret enters draining state, new connections
1907+
matching it are rejected, in-flight ones keep going until they close.</li>
1908+
<li>After all clients have migrated (or <code>drain_timeout_secs</code> elapses), the
1909+
slot is released and metrics for it disappear.</li>
1910+
</ol>
1911+
<p>Re-adding a draining secret before the timeout revives it — the same slot,
1912+
counters, and IP tracking carry over. Pinned <code>-S</code> CLI secrets are immutable
1913+
across SIGHUP and never drain.</p>
1914+
<p>Metrics for a draining secret:</p>
1915+
<ul>
1916+
<li><strong>Stats:</strong> <code>secret_&lt;label&gt;_draining 1</code>, <code>secret_&lt;label&gt;_drain_age_seconds</code>,
1917+
<code>secret_&lt;label&gt;_rejected_draining</code>, <code>secret_&lt;label&gt;_drain_forced</code></li>
1918+
<li><strong>Prometheus:</strong> <code>teleproxy_secret_draining{secret="..."}</code>,
1919+
<code>teleproxy_secret_drain_age_seconds{secret="..."}</code>,
1920+
<code>teleproxy_secret_rejected_draining_total{secret="..."}</code>,
1921+
<code>teleproxy_secret_drain_forced_total{secret="..."}</code></li>
1922+
</ul>
1923+
<p>Capacity: at most 16 active secrets at any moment, plus up to 16 additional
1924+
draining slots. Rotating all 16 secrets twice in quick succession before the
1925+
first batch finishes draining will fail the second reload — wait for drain to
1926+
complete or set a smaller <code>drain_timeout_secs</code> to reclaim slots faster.</p>
18681927
<h2 id="per-ip-top-n-metrics">Per-IP Top-N Metrics<a class="headerlink" href="#per-ip-top-n-metrics" title="Permanent link">&para;</a></h2>
18691928
<p>Surface the heaviest client IPs per secret in <code>/metrics</code> to diagnose rogue clients. Disabled by default — opt in with a top-level config knob, not a per-secret one (cardinality is an operator concern).</p>
18701929
<p>TOML config:</p>
1871-
<div class="highlight"><pre><span></span><code><a id="__codelineno-17-1" name="__codelineno-17-1" href="#__codelineno-17-1"></a><span class="n">top_ips_per_secret</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="c1"># 0 disables; max 32</span>
1930+
<div class="highlight"><pre><span></span><code><a id="__codelineno-18-1" name="__codelineno-18-1" href="#__codelineno-18-1"></a><span class="n">top_ips_per_secret</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="c1"># 0 disables; max 32</span>
18721931
</code></pre></div>
18731932
<p>Docker:</p>
1874-
<div class="highlight"><pre><span></span><code><a id="__codelineno-18-1" name="__codelineno-18-1" href="#__codelineno-18-1"></a><span class="nv">TOP_IPS_PER_SECRET</span><span class="o">=</span><span class="m">10</span>
1933+
<div class="highlight"><pre><span></span><code><a id="__codelineno-19-1" name="__codelineno-19-1" href="#__codelineno-19-1"></a><span class="nv">TOP_IPS_PER_SECRET</span><span class="o">=</span><span class="m">10</span>
18751934
</code></pre></div>
18761935
<p>When enabled, three new Prometheus metric families are emitted, each labelled <code>{secret="...", ip="..."}</code> for the top-N heaviest IPs per secret (sorted by total bytes):</p>
18771936
<ul>
@@ -1882,23 +1941,23 @@ <h2 id="per-ip-top-n-metrics">Per-IP Top-N Metrics<a class="headerlink" href="#p
18821941
<p>Cardinality is bounded at <code>top_ips_per_secret × secret_count</code> labelled samples per family. Counters reset when an IP entry is recycled — Prometheus <code>rate()</code> handles this correctly.</p>
18831942
<h2 id="toml-config-example">TOML Config Example<a class="headerlink" href="#toml-config-example" title="Permanent link">&para;</a></h2>
18841943
<p>All per-secret features combined:</p>
1885-
<div class="highlight"><pre><span></span><code><a id="__codelineno-19-1" name="__codelineno-19-1" href="#__codelineno-19-1"></a><span class="k">[[secret]]</span>
1886-
<a id="__codelineno-19-2" name="__codelineno-19-2" href="#__codelineno-19-2"></a><span class="n">key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;cafe01234567890abcafe01234567890a&quot;</span>
1887-
<a id="__codelineno-19-3" name="__codelineno-19-3" href="#__codelineno-19-3"></a><span class="n">label</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;family&quot;</span>
1888-
<a id="__codelineno-19-4" name="__codelineno-19-4" href="#__codelineno-19-4"></a><span class="n">limit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">100</span>
1889-
<a id="__codelineno-19-5" name="__codelineno-19-5" href="#__codelineno-19-5"></a><span class="n">quota</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;50G&quot;</span>
1890-
<a id="__codelineno-19-6" name="__codelineno-19-6" href="#__codelineno-19-6"></a><span class="n">rate_limit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;10M&quot;</span>
1891-
<a id="__codelineno-19-7" name="__codelineno-19-7" href="#__codelineno-19-7"></a><span class="n">max_ips</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">10</span>
1892-
<a id="__codelineno-19-8" name="__codelineno-19-8" href="#__codelineno-19-8"></a><span class="n">expires</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ld">2026-12-31T23:59:59Z</span>
1893-
<a id="__codelineno-19-9" name="__codelineno-19-9" href="#__codelineno-19-9"></a>
1894-
<a id="__codelineno-19-10" name="__codelineno-19-10" href="#__codelineno-19-10"></a><span class="k">[[secret]]</span>
1895-
<a id="__codelineno-19-11" name="__codelineno-19-11" href="#__codelineno-19-11"></a><span class="n">key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;dead01234567890abcead01234567890a&quot;</span>
1896-
<a id="__codelineno-19-12" name="__codelineno-19-12" href="#__codelineno-19-12"></a><span class="n">label</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;guest&quot;</span>
1897-
<a id="__codelineno-19-13" name="__codelineno-19-13" href="#__codelineno-19-13"></a><span class="n">limit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">50</span>
1898-
<a id="__codelineno-19-14" name="__codelineno-19-14" href="#__codelineno-19-14"></a><span class="n">quota</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;5G&quot;</span>
1899-
<a id="__codelineno-19-15" name="__codelineno-19-15" href="#__codelineno-19-15"></a><span class="n">rate_limit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;2M&quot;</span>
1900-
<a id="__codelineno-19-16" name="__codelineno-19-16" href="#__codelineno-19-16"></a><span class="n">max_ips</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">3</span>
1901-
<a id="__codelineno-19-17" name="__codelineno-19-17" href="#__codelineno-19-17"></a><span class="n">expires</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ld">2025-06-30T00:00:00Z</span>
1944+
<div class="highlight"><pre><span></span><code><a id="__codelineno-20-1" name="__codelineno-20-1" href="#__codelineno-20-1"></a><span class="k">[[secret]]</span>
1945+
<a id="__codelineno-20-2" name="__codelineno-20-2" href="#__codelineno-20-2"></a><span class="n">key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;cafe01234567890abcafe01234567890a&quot;</span>
1946+
<a id="__codelineno-20-3" name="__codelineno-20-3" href="#__codelineno-20-3"></a><span class="n">label</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;family&quot;</span>
1947+
<a id="__codelineno-20-4" name="__codelineno-20-4" href="#__codelineno-20-4"></a><span class="n">limit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">100</span>
1948+
<a id="__codelineno-20-5" name="__codelineno-20-5" href="#__codelineno-20-5"></a><span class="n">quota</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;50G&quot;</span>
1949+
<a id="__codelineno-20-6" name="__codelineno-20-6" href="#__codelineno-20-6"></a><span class="n">rate_limit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;10M&quot;</span>
1950+
<a id="__codelineno-20-7" name="__codelineno-20-7" href="#__codelineno-20-7"></a><span class="n">max_ips</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">10</span>
1951+
<a id="__codelineno-20-8" name="__codelineno-20-8" href="#__codelineno-20-8"></a><span class="n">expires</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ld">2026-12-31T23:59:59Z</span>
1952+
<a id="__codelineno-20-9" name="__codelineno-20-9" href="#__codelineno-20-9"></a>
1953+
<a id="__codelineno-20-10" name="__codelineno-20-10" href="#__codelineno-20-10"></a><span class="k">[[secret]]</span>
1954+
<a id="__codelineno-20-11" name="__codelineno-20-11" href="#__codelineno-20-11"></a><span class="n">key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;dead01234567890abcead01234567890a&quot;</span>
1955+
<a id="__codelineno-20-12" name="__codelineno-20-12" href="#__codelineno-20-12"></a><span class="n">label</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;guest&quot;</span>
1956+
<a id="__codelineno-20-13" name="__codelineno-20-13" href="#__codelineno-20-13"></a><span class="n">limit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">50</span>
1957+
<a id="__codelineno-20-14" name="__codelineno-20-14" href="#__codelineno-20-14"></a><span class="n">quota</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;5G&quot;</span>
1958+
<a id="__codelineno-20-15" name="__codelineno-20-15" href="#__codelineno-20-15"></a><span class="n">rate_limit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;2M&quot;</span>
1959+
<a id="__codelineno-20-16" name="__codelineno-20-16" href="#__codelineno-20-16"></a><span class="n">max_ips</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">3</span>
1960+
<a id="__codelineno-20-17" name="__codelineno-20-17" href="#__codelineno-20-17"></a><span class="n">expires</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ld">2025-06-30T00:00:00Z</span>
19021961
</code></pre></div>
19031962

19041963

0 commit comments

Comments
 (0)