Skip to content

Commit 1b791e4

Browse files
authored
Varnish blocks crawler access
1 parent e7e6793 commit 1b791e4

2 files changed

Lines changed: 26 additions & 0 deletions

File tree

  • i18n/en/docusaurus-plugin-content-docs/version-2.0/apps
  • versioned_docs/version-2.0/apps

i18n/en/docusaurus-plugin-content-docs/version-2.0/apps/varnish.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,18 @@ import Meta from './_include/varnish.md';
2929

3030
4. Visiting the domain that Varnish is bound to, you can see that the access speed is greatly improved!
3131

32+
### Varnish blocks crawler access
33+
34+
1. Edit the relevant parameters in the Varnish application's `./src/default.vcl` file and add the following content, replacing `Sogou web spider` with the name of the crawler you want to block:
35+
```
36+
sub vcl_recv {
37+
if (req.http.user-agent ~ “Sogou web spider”) {
38+
return (synth(403, “Forbidden”));
39+
}
40+
}
41+
```
42+
2. After rebuilding the Varnish application, Varnish has been disabled for the crawler.
43+
3244
## Configuration options{#configs}
3345

3446
- Cache size: set via VARNISH_SIZE environment variable

versioned_docs/version-2.0/apps/varnish.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,20 @@ import Meta from './_include/varnish.md';
3030

3131
4. 访问 Varnish 所绑定的域名,便发现访问速度大大提升
3232

33+
### Varnish 禁止爬虫访问
34+
35+
1. 编辑 Varnish 应用的 `./src/default.vcl` 文件中相关参数,增加如下内容,其中 `Sogou web spider` 改成你想要禁用的爬虫名:
36+
```
37+
sub vcl_recv {
38+
if (req.http.user-agent ~ "Sogou web spider") {
39+
return (synth(403, "Forbidden"));
40+
}
41+
}
42+
43+
```
44+
45+
2. 重建 Varnish 应用后,Varnish 已经对爬虫禁用
46+
3347
## 配置选项{#configs}
3448

3549
- 缓存大小:通过 VARNISH_SIZE 环境变量设置

0 commit comments

Comments
 (0)