Skip to content

Commit 2afc265

Browse files
brendan-kellamclaude
brendan-kellam
and
claude
authored
feat(web): fetch /browse data client-side and disallow crawlers via robots.txt (#1426)
* feat(web): disallow crawlers via robots.txt, allowlisting link-preview bots Sourcebot exposes an unbounded URL space (every file x revision x commit) and crawler traffic (GPTBot, Applebot, meta-externalagent, AhrefsBot) generates heavy SSR load. Deny all crawling while keeping unfurl bots (Slack, X, LinkedIn, etc.) allowed so shared links keep their OpenGraph previews. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(web): fetch /browse panel data client-side instead of via RSC props Blob, tree, and commit-diff panels previously fetched git data (file source, folder contents, full commit diffs) in server components and passed it as props across the client boundary. React flight-encodes those props into the document at seconds of main-thread time for large payloads (a large commit page measured 38MB / ~8s of encoding), which starves the event loop and health probes under crawler load. The panels now render a shell and fetch via existing JSON API routes with react-query (per repo convention), plus a new /api/folder_contents route. Documents drop to the ~70KB shell regardless of content size, and JS-less crawlers never trigger the data fetch at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: add CHANGELOG entries for #1426 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(web): keep code-preview header/toolbar stable during load Render the path header, separator, and file toolbar (blame toggle + age legend) immediately from props, and scope the loading spinner / error to the body below. The line-count/size stat shows a skeleton while the file source is pending, then the real value once loaded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 9f42c66 commit 2afc265

12 files changed

Lines changed: 703 additions & 404 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1313
- Made the backend worker API address configurable via the `WORKER_API_URL` environment variable (default `http://localhost:3060`) instead of being hardcoded. [#1409](https://github.com/sourcebot-dev/sourcebot/pull/1409)
1414
- [EE] Disabled `DELETE /api/ee/user` while SCIM provisioning is enabled, and switched it to an org-scoped membership removal (with last-owner protection) instead of a global account delete. [#1425](https://github.com/sourcebot-dev/sourcebot/pull/1425)
1515
- [EE] Unified the `GET /api/ee/user` and `GET /api/ee/users` response shapes behind a shared mapper; the single-user endpoint is now scoped to org membership, and both include role, membership status, and last activity. [#1425](https://github.com/sourcebot-dev/sourcebot/pull/1425)
16+
- Browse blob, tree, and commit pages now fetch file sources, folder contents, and diffs client-side via API routes instead of embedding them in the server-rendered page, keeping documents small and event-loop pressure low for large files and commits. [#1426](https://github.com/sourcebot-dev/sourcebot/pull/1426)
1617

1718
### Added
1819
- Added per-step token cost tracking and estimated tool call token usage to Ask Sourcebot chat history. [#1353](https://github.com/sourcebot-dev/sourcebot/pull/1353)
@@ -23,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2324
- [EE] Added text file attachments to Ask Sourcebot, letting users attach text/code/config files to a chat message via the paperclip button, drag-and-drop, or paste, with large pastes auto-converted to attachments. [#1374](https://github.com/sourcebot-dev/sourcebot/pull/1374)
2425
- [EE] Added image attachments to Ask Sourcebot, letting users attach images to a chat message when the selected model supports image input. [#1375](https://github.com/sourcebot-dev/sourcebot/pull/1375)
2526
- Added deployment system resource stats (CPU cores + cgroup quota, host + container memory, disk, load average) to the service ping, so resource issues can be diagnosed more quickly. [#1424](https://github.com/sourcebot-dev/sourcebot/pull/1424)
27+
- Added a `robots.txt` that disallows crawlers, with an allowlist for link-preview bots so shared links keep their OpenGraph previews. [#1426](https://github.com/sourcebot-dev/sourcebot/pull/1426)
2628

2729
### Fixed
2830
- Send anonymous server-side PostHog events as personless so unauthenticated requests don't inflate person counts. [#1367](https://github.com/sourcebot-dev/sourcebot/pull/1367)

‎packages/web/src/app/(app)/browse/[...path]/components/codePreviewPanel/codePreviewPanel.tsx‎

Lines changed: 17 additions & 169 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,6 @@
11
import { getRepoInfoByName } from "@/actions";
2-
import { PathHeader } from "@/app/(app)/components/pathHeader";
3-
import { Button } from "@/components/ui/button";
4-
import { Separator } from "@/components/ui/separator";
5-
import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip";
6-
import { cn, getCodeHostInfoForRepo, isServiceError, truncateSha } from "@/lib/utils";
7-
import { X } from "lucide-react";
8-
import Image from "next/image";
9-
import Link from "next/link";
10-
import { getBrowsePath } from "../../../hooks/utils";
11-
import { BlameAgeLegend } from "./blameAgeLegend";
12-
import { BlameViewToggle } from "./blameViewToggle";
13-
import { PureCodePreviewPanel } from "./pureCodePreviewPanel";
14-
import { getFileBlame, getFileSource } from '@/features/git';
15-
16-
const formatFileSize = (bytes: number): string => {
17-
if (bytes < 1024) {
18-
return `${bytes} B`;
19-
}
20-
if (bytes < 1024 * 1024) {
21-
return `${(bytes / 1024).toFixed(1)} KB`;
22-
}
23-
return `${(bytes / 1024 / 1024).toFixed(1)} MB`;
24-
};
2+
import { isServiceError } from "@/lib/utils";
3+
import { CodePreviewPanelClient } from "./codePreviewPanelClient";
254

265
interface CodePreviewPanelProps {
276
path: string;
@@ -36,156 +15,25 @@ interface CodePreviewPanelProps {
3615
}
3716

3817
export const CodePreviewPanel = async ({ path, repoName, revisionName, previewRef, blame }: CodePreviewPanelProps) => {
39-
const contentRef = previewRef ?? revisionName;
40-
41-
const [fileSourceResponse, repoInfoResponse, blameResponse] = await Promise.all([
42-
getFileSource({
43-
path,
44-
repo: repoName,
45-
ref: contentRef,
46-
}, { source: 'sourcebot-web-client' }),
47-
getRepoInfoByName(repoName),
48-
blame
49-
? getFileBlame({
50-
path,
51-
repo: repoName,
52-
ref: contentRef,
53-
}, { source: 'sourcebot-web-client' })
54-
: Promise.resolve(undefined),
55-
]);
56-
57-
if (isServiceError(fileSourceResponse)) {
58-
return <div>Error loading file source: {fileSourceResponse.message}</div>
59-
}
18+
const repoInfoResponse = await getRepoInfoByName(repoName);
6019

6120
if (isServiceError(repoInfoResponse)) {
6221
return <div>Error loading repo info: {repoInfoResponse.message}</div>
6322
}
6423

65-
if (blameResponse !== undefined && isServiceError(blameResponse)) {
66-
return <div>Error loading blame: {blameResponse.message}</div>
67-
}
68-
69-
const source = fileSourceResponse.source;
70-
const lineCount = source.length === 0
71-
? 0
72-
: source.split('\n').length - (source.endsWith('\n') ? 1 : 0);
73-
const byteSize = Buffer.byteLength(source, 'utf-8');
74-
const fileSize = formatFileSize(byteSize);
75-
76-
const codeHostInfo = getCodeHostInfoForRepo({
77-
codeHostType: repoInfoResponse.codeHostType,
78-
name: repoInfoResponse.name,
79-
displayName: repoInfoResponse.displayName,
80-
externalWebUrl: repoInfoResponse.externalWebUrl,
81-
});
82-
83-
// @todo: this is a hack to support linking to files for ADO. ADO doesn't support web urls with HEAD so we replace it with main. THis
84-
// will break if the default branch is not main.
85-
const fileWebUrl = repoInfoResponse.codeHostType === "azuredevops" && fileSourceResponse.externalWebUrl ?
86-
fileSourceResponse.externalWebUrl.replace("version=GBHEAD", "version=GBmain") : fileSourceResponse.externalWebUrl;
87-
8824
return (
89-
<>
90-
<div className="flex flex-row py-1 px-2 items-center justify-between">
91-
<PathHeader
92-
path={path}
93-
repo={{
94-
name: repoName,
95-
codeHostType: repoInfoResponse.codeHostType,
96-
displayName: repoInfoResponse.displayName,
97-
externalWebUrl: repoInfoResponse.externalWebUrl,
98-
}}
99-
revisionName={contentRef}
100-
/>
101-
102-
{fileWebUrl && (
103-
104-
<a
105-
href={fileWebUrl}
106-
target="_blank"
107-
rel="noopener noreferrer"
108-
className="flex flex-row items-center gap-2 px-2 py-0.5 rounded-md flex-shrink-0"
109-
>
110-
<Image
111-
src={codeHostInfo.icon}
112-
alt={codeHostInfo.codeHostName}
113-
className={cn('w-4 h-4 flex-shrink-0', codeHostInfo.iconClassName)}
114-
/>
115-
<span className="text-sm font-medium">Open in {codeHostInfo.codeHostName}</span>
116-
</a>
117-
)}
118-
</div>
119-
<Separator />
120-
{!previewRef && (
121-
<div className="flex flex-row items-center gap-3 px-4 py-1 border-b shrink-0">
122-
<BlameViewToggle
123-
repoName={repoName}
124-
revisionName={revisionName}
125-
path={path}
126-
blame={blame ?? false}
127-
/>
128-
<span className="text-sm text-muted-foreground">
129-
{lineCount.toLocaleString()} lines · {fileSize}
130-
</span>
131-
{blame && (
132-
<>
133-
<Separator orientation="vertical" className="h-4" />
134-
<BlameAgeLegend />
135-
</>
136-
)}
137-
</div>
138-
)}
139-
{previewRef && (
140-
<div className="flex flex-row items-center justify-between gap-2 px-4 py-2 border-b shrink-0">
141-
<span className="text-sm">
142-
Previewing file at revision{" "}
143-
<Link
144-
href={getBrowsePath({
145-
repoName,
146-
revisionName,
147-
path: '',
148-
pathType: 'commit',
149-
commitSha: previewRef,
150-
})}
151-
className="font-mono text-link hover:underline"
152-
>
153-
{truncateSha(previewRef)}
154-
</Link>
155-
</span>
156-
<Tooltip key={previewRef}>
157-
<TooltipTrigger>
158-
<Button
159-
asChild
160-
variant="ghost"
161-
size="icon"
162-
className="h-6 w-6 text-muted-foreground"
163-
>
164-
<Link
165-
href={getBrowsePath({
166-
repoName,
167-
revisionName,
168-
path,
169-
pathType: 'blob',
170-
})}
171-
aria-label="Close preview"
172-
>
173-
<X className="h-4 w-4" />
174-
</Link>
175-
</Button>
176-
</TooltipTrigger>
177-
<TooltipContent>Close preview</TooltipContent>
178-
</Tooltip>
179-
</div>
180-
)}
181-
<PureCodePreviewPanel
182-
source={fileSourceResponse.source}
183-
language={fileSourceResponse.language}
184-
repoName={repoName}
185-
path={path}
186-
revisionName={contentRef ?? 'HEAD'}
187-
blame={blameResponse}
188-
/>
189-
</>
25+
<CodePreviewPanelClient
26+
path={path}
27+
repoName={repoName}
28+
revisionName={revisionName}
29+
previewRef={previewRef}
30+
blame={blame}
31+
repo={{
32+
name: repoInfoResponse.name,
33+
codeHostType: repoInfoResponse.codeHostType,
34+
displayName: repoInfoResponse.displayName,
35+
externalWebUrl: repoInfoResponse.externalWebUrl,
36+
}}
37+
/>
19038
)
191-
}
39+
}

0 commit comments

Comments
 (0)