Doesn't work in production

[Donald646]

This gets blocked in production

[emilthemaker]

Mind specifying what's happening?

[Donald646]

I'm using Next.js Supabase vercel

It seems like when I go to production, it's getting blocked. I get the error Transcript is Disabled, but I know that's not true, because it works locally.

I get the

Error: An error occurred in the Server Components render. The specific message is omitted in production builds to avoid leaking sensitive details. A digest property is included in this error instance which may provide additional details about the nature of the error.

I am using server components for it. Even when I switch to API routes it still gets blocked.

[Donald646]

import { YoutubeTranscript } from 'youtube-transcript';

export async function POST(request: Request) {
  const { videoUrl } = await request.json();

  try {
    const transcript = await YoutubeTranscript.fetchTranscript(videoUrl);
    let newTranscript = '';
    for (let i = 0; i < transcript.length; i++) {
      newTranscript += `Timestamp: ${transcript[i].offset}, Text: ${transcript[i].text}\n`;
    }
    return NextResponse.json({ transcript: newTranscript });
  } catch (error) {
    console.error('Error fetching transcript:', error);
    return NextResponse.json(
      { error: 'Failed to fetch video transcript. Please check the URL and try again.' },
      { status: 500 }
    );
  }
}

this is my code above

[Donald646]

I saw the other issue #11 and it seems like it's the issue I'm facing, but recommending it because there seems to be no good solution. If YouTube is blocking sites, then would this library be useless then?

[kellenmace]

I was encountering this issue on Vercel, too. I believe this issue exists because YouTube is returning different HTML for YouTube video pages depending on where the request is coming from. When I scrape a page like https://www.youtube.com/watch?v=rB9ql0L0cUQ on my local machine, the HTML page contains a script tag with a ytInitialPlayerResponse variable in it, and within that object is the URLs to the caption tracks (this is what the youtube-transcript library relies on to get the transcript URLs). When I scrape that same page in a Serverless Function on Vercel though, the ytInitialPlayerResponse object does not contain the caption track URLs, therefore the code that youtube-transcript runs errors out.

Option 1

If all you need is the lines of text from the transcript, then the method described in #11 where you use the youtubei.js NPM package to access the transcript may work just fine.

Option 2

If you need all of the transcript data, including the start time and duration for every line, the easiest solution is to run your youtube-transcript code somewhere else. You can create a Google Cloud Function that executes the code and call it from your app, for example. This isn't ideal, since you have to have this one bit of functionality live outside of your Vercel-hosted app, but it works.

[Donald646]

I see thank you. The durations are a pretty core part of my app. The commenter of the youtubei.js code said it was also possible to get the durations, so I will check that out first.

[terrytjw]

i am just experiencing this issue too in prod. works perfectly fine on local. @Donald646 did you manage to get it to work on prod?

[Donald646]

@terrytjw No I haven't, I don't think it works. I'm planning to switch to youtubei.js, but it seems like the issue is prevalent over there as well.

[terrytjw]

I was encountering this issue on Vercel, too. I believe this issue exists because YouTube is returning different HTML for YouTube video pages depending on where the request is coming from. When I scrape a page like https://www.youtube.com/watch?v=rB9ql0L0cUQ on my local machine, the HTML page contains a script tag with a ytInitialPlayerResponse variable in it, and within that object is the URLs to the caption tracks (this is what the youtube-transcript library relies on to get the transcript URLs). When I scrape that same page in a Serverless Function on Vercel though, the ytInitialPlayerResponse object does not contain the caption track URLs, therefore the code that youtube-transcript runs errors out.

Option 1

If all you need is the lines of text from the transcript, then the method described in #11 where you use the youtubei.js NPM package to access the transcript may work just fine.

Option 2

If you need all of the transcript data, including the start time and duration for every line, the easiest solution is to run your youtube-transcript code somewhere else. You can create a Google Cloud Function that executes the code and call it from your app, for example. This isn't ideal, since you have to have this one bit of functionality live outside of your Vercel-hosted app, but it works.

hey @kellenmace , other than Google Cloud Function, can aws lambda work too?

[terrytjw]

@terrytjw No I haven't, I don't think it works. I'm planning to switch to youtubei.js, but it seems like the issue is prevalent over there as well.

@Donald646 have you tried the Google Cloud Function approach?

[Donald646]

@terrytjw No I haven't, I don't think it works. I'm planning to switch to youtubei.js, but it seems like the issue is prevalent over there as well.

@Donald646 have you tried the Google Cloud Function approach?

No I haven't yet, but from another issue I was looking at on Youtubei.js they said It didn't work either, but they weren't getting transcripts so I don't know

SuspiciousLookingOwl/youtubei#113

[leandronorcio]

hey @kellenmace , other than Google Cloud Function, can aws lambda work too?

I can confirm it’s not working on AWS Lambda.

[Donald646]

Yea this library is cooked, it's kinda useless if it doesn't work in production

[emilthemaker]

@terrytjw No I haven't, I don't think it works. I'm planning to switch to youtubei.js, but it seems like the issue is prevalent over there as well.

@Donald646 have you tried the Google Cloud Function approach?

No I haven't yet, but from another issue I was looking at on Youtubei.js they said It didn't work either, but they weren't getting transcripts so I don't know

SuspiciousLookingOwl/youtubei#113

Working for me with youtubei.js but no clue why 😅

I'm on Supabase Edge Functions. Transcript works, but some other properties appear broken.

[metaloozee]

Because YouTube is always changing how it operates and because it enforces sign-in, it is making it impossible for data scrapers to obtain the data, this is a global problem.

You can find a similar issue that simply returns the YouTube message "Sign in to confirm you are not a bot" if you check at a few different libraries, including ytdl and youtubei.js.