Re-include older episodes when max_age is expanded

Increasing a feed's max_age now pulls in older episodes that were previously out of range, instead of silently
ignoring them. This required decoupling discovery depth from page_size and making episode re-inclusion work off
the database rather than the API window.

- Discovery depth is driven by max_age: a normal update still does a shallow scan (page_size most-recent
  episodes), but when max_age is expanded beyond what was previously scanned, the YouTube builder performs a
  one-time deep scan that pages back to the max_age cutoff. A per-feed ScannedThrough high-water mark gates this
  so it costs nothing in steady state, and it is bounded by a 20-page backstop. Brand-new feeds stay shallow.
- Previously cleaned (soft-deleted) episodes that match the filters again are re-queued for download off the
  database records directly, independent of the page_size API window. Only episodes that would survive the
  keep_last cleanup are re-queued, to avoid a re-download/re-clean loop.
- Cleanup no longer clears an episode's title/description when soft-deleting it, so a re-included episode keeps
  its metadata. For episodes cleaned by older versions (which wiped metadata), the title/description are
  recovered from the current feed query.
- The feed build skips a downloaded episode with an empty title instead of aborting the whole feed, so one
  malformed item can no longer take down the entire feed.
- A startup warning is logged when clean.keep_last exceeds page_size and no max_age is configured.

Currently implemented for YouTube.

Author: mm503

CLAUDE.md

@@ -38,9 +38,12 @@ Understanding how episodes flow through the system:
 
 ### Discovery Phase
 - Episodes are discovered during feed updates via platform APIs (YouTube Data API v3, Vimeo API, SoundCloud, Twitch)
-- `updateFeed()` in `services/update/updater.go:99-154` queries the platform API
+- `updateFeed()` in `services/update/updater.go:99-169` queries the platform API
 - New episodes are stored in BadgerDB with status `EpisodeNew`
 - Episodes matching feed URL are identified by provider-specific parsing in `pkg/builder/`
+- **Discovery depth (`max_age`-driven deep scan)**: normal updates do a shallow scan (the `page_size` most-recent episodes). To discover **never-before-seen** older episodes after `max_age` is increased, the updater triggers a one-time deep scan that pages the channel back to the `max_age` cutoff instead of stopping at `page_size`. The decision lives in `discoveryWindow()` (`services/update/updater.go`): each feed record stores `ScannedThrough`, a high-water mark of the oldest publish date discovery has paged to; a deep scan runs only when the `max_age` cutoff is older than `ScannedThrough` (i.e. `max_age` was actually expanded), so it costs nothing in steady state. The cutoff is passed to the builder via `feed.Config.DiscoverSince` (runtime-only field); the YouTube builder honors it in `queryItemsSince()`, bounded by `maxDeepScanPages` (20) as a quota backstop. `nextScannedThrough()` advances the mark to the cutoff only once every matching in-range episode is downloaded/cleaned/errored, so multi-cycle catch-up (bounded by `page_size` downloads per cycle) isn't truncated by episode pruning. Brand-new feeds stay shallow (no surprise back-catalog download). Currently implemented for YouTube only.
+- Previously cleaned episodes (`EpisodeCleaned`) that match the current filters again (e.g. after `max_age` is increased) are re-queued: status is reset to `EpisodeNew` (`resurrectEpisodes()` in `services/update/updater.go`). This works **off the database records directly, so it is independent of the `page_size` API window** — a cleaned episode no longer needs to be returned by the current feed query to be re-included. Cleaned records retain their metadata (see cleanup phase), so filters are evaluated against the stored title/description. Episodes cleaned by older podsync versions had their title/description wiped; for those the metadata is recovered from the current feed query (the deep-discovery scan brings in-range episodes back into the result with fresh metadata). An episode that still has no title — not in the DB and not in the feed query — is left cleaned until a later cycle surfaces it. When `keep_last` is configured, only episodes that would survive the next cleanup are re-queued (ranked by `PubDate` via `keepLastSurvivors()`), to avoid a re-download/re-clean loop.
+- Feed XML build (`feed.Build()` in `pkg/feed/xml.go`) skips any downloaded episode with an empty title (logging a warning) instead of aborting the entire feed build for one malformed item. (Original podsync aborted the whole feed on any empty title/description.)
 
 ### Download Phase
 - `fetchEpisodes()` iterates episodes with status `EpisodeNew` or `EpisodeError`
@@ -51,12 +54,13 @@ Understanding how episodes flow through the system:
 - On failure: status set to `EpisodeError`, retry attempted next cycle
 
 ### Cleanup Phase (Important!)
-- `cleanup()` in `updater.go:373-441` runs AFTER each successful update cycle
+- `cleanup()` in `updater.go:456-524` runs AFTER each successful update cycle
 - **Only triggered if `clean.keep_last` is configured** (global or per-feed)
 - Keeps most recent N episodes by PubDate (descending order)
-- Deleted episodes have status changed to `EpisodeCleaned` and title/description cleared
+- Deleted episodes have status changed to `EpisodeCleaned`; their title/description are **retained** so the record can be resurrected later (e.g. after `max_age` is increased) without losing metadata. Cleaned episodes are excluded from the feed by status, so keeping metadata has no effect on output.
 - **Files are deleted from storage but database records are retained**
 - This is a soft-delete: episodes remain in the database forever
+- **`keep_last` vs `page_size`**: a shallow update only sees `page_size` episodes, so on its own it can't fill a larger retention window. This is fine when `max_age` is set (the deep-discovery scan covers the look-back — see Discovery Phase) or when episodes were already discovered earlier. A startup warning is logged only when `clean.keep_last > page_size` **and** no `max_age` is configured (`applyDefaults()` in `cmd/podsync/config.go`).
 
 ### Episode Removal from Database
 - Episodes are removed from the database only if they:

cmd/podsync/config.go

@@ -167,7 +167,7 @@ func (c *Config) applyDefaults(configPath string) {
 		c.Database.Dir = filepath.Join(filepath.Dir(configPath), "db")
 	}
 
-	for _, _feed := range c.Feeds {
+	for id, _feed := range c.Feeds {
 		if _feed.UpdatePeriod == 0 {
 			_feed.UpdatePeriod = model.DefaultUpdatePeriod
 		}
@@ -196,6 +196,18 @@ func (c *Config) applyDefaults(configPath string) {
 		if _feed.Clean == nil && c.Cleanup != nil {
 			_feed.Clean = c.Cleanup
 		}
+
+		// keep_last greater than page_size means a normal (shallow) update only ever sees
+		// page_size episodes, so the retention window can never be filled. When max_age is set
+		// the updater performs a one-time deep discovery to cover it, so the mismatch is fine;
+		// only warn when there is no max_age to drive that deeper look-back.
+		if _feed.Clean != nil && _feed.Clean.KeepLast > _feed.PageSize && _feed.Filters.MaxAge <= 0 {
+			log.Warnf(
+				"feed %q: clean.keep_last (%d) is greater than page_size (%d) and no max_age is set; "+
+					"set page_size >= keep_last (or configure max_age) so the retention window can be filled",
+				id, _feed.Clean.KeepLast, _feed.PageSize,
+			)
+		}
 	}
 }
 

cmd/podsync/config_test.go

@@ -1,15 +1,17 @@
 package main
 
 import (
+	"bytes"
 	"os"
 	"testing"
 	"time"
 
-	"github.com/mxpv/podsync/services/web"
+	log "github.com/sirupsen/logrus"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
 	"github.com/mxpv/podsync/pkg/model"
+	"github.com/mxpv/podsync/services/web"
 )
 
 func TestLoadConfig(t *testing.T) {
@@ -215,6 +217,81 @@ data_dir = "/data"
 	assert.EqualValues(t, feed.Format, "video")
 }
 
+func TestKeepLastPageSizeWarning(t *testing.T) {
+	t.Run("warns when keep_last exceeds page_size and no max_age", func(t *testing.T) {
+		var buf bytes.Buffer
+		log.SetOutput(&buf)
+		t.Cleanup(func() { log.SetOutput(os.Stderr) })
+
+		const file = `
+[server]
+data_dir = "/data"
+
+[feeds]
+  [feeds.A]
+  url = "https://youtube.com/watch?v=ygIUF678y40"
+  page_size = 5
+  clean = { keep_last = 20 }
+`
+		path := setup(t, file)
+		defer os.Remove(path)
+
+		config, err := LoadConfig(path)
+		require.NoError(t, err)
+		require.NotNil(t, config)
+		assert.Contains(t, buf.String(), "clean.keep_last")
+	})
+
+	t.Run("does not warn when max_age is configured", func(t *testing.T) {
+		var buf bytes.Buffer
+		log.SetOutput(&buf)
+		t.Cleanup(func() { log.SetOutput(os.Stderr) })
+
+		const file = `
+[server]
+data_dir = "/data"
+
+[feeds]
+  [feeds.A]
+  url = "https://youtube.com/watch?v=ygIUF678y40"
+  page_size = 5
+  clean = { keep_last = 20 }
+  filters = { max_age = 30 }
+`
+		path := setup(t, file)
+		defer os.Remove(path)
+
+		config, err := LoadConfig(path)
+		require.NoError(t, err)
+		require.NotNil(t, config)
+		assert.NotContains(t, buf.String(), "clean.keep_last")
+	})
+
+	t.Run("does not warn when keep_last is within page_size", func(t *testing.T) {
+		var buf bytes.Buffer
+		log.SetOutput(&buf)
+		t.Cleanup(func() { log.SetOutput(os.Stderr) })
+
+		const file = `
+[server]
+data_dir = "/data"
+
+[feeds]
+  [feeds.A]
+  url = "https://youtube.com/watch?v=ygIUF678y40"
+  page_size = 50
+  clean = { keep_last = 20 }
+`
+		path := setup(t, file)
+		defer os.Remove(path)
+
+		config, err := LoadConfig(path)
+		require.NoError(t, err)
+		require.NotNil(t, config)
+		assert.NotContains(t, buf.String(), "clean.keep_last")
+	})
+}
+
 func TestHttpServerListenAddress(t *testing.T) {
 	const file = `
 [server]

pkg/builder/youtube.go

@@ -130,9 +130,10 @@ func (yt *YouTubeBuilder) listPlaylists(ctx context.Context, id, channelID strin
 
 // Cost: 3 units (call: 1, snippet: 2)
 // See https://developers.google.com/youtube/v3/docs/playlistItems/list#part
-func (yt *YouTubeBuilder) listPlaylistItems(ctx context.Context, feed *model.Feed, pageToken string) ([]*youtube.PlaylistItem, string, error) {
+func (yt *YouTubeBuilder) listPlaylistItems(ctx context.Context, feed *model.Feed, pageToken string, fullPage bool) ([]*youtube.PlaylistItem, string, error) {
 	count := maxYoutubeResults
-	if count > feed.PageSize {
+	// During a deep (max_age-driven) scan we page through full result sets regardless of PageSize.
+	if !fullPage && count > feed.PageSize {
 		// If we need less than 50
 		count = feed.PageSize
 	}
@@ -412,15 +413,19 @@ func (yt *YouTubeBuilder) queryVideoDescriptions(ctx context.Context, playlist m
 // Cost:
 // ASC mode = (3 units + 5 units) * X pages = 8 units per page
 // DESC mode = 3 units * (number of pages in the entire playlist) + 5 units
-func (yt *YouTubeBuilder) queryItems(ctx context.Context, feed *model.Feed) error {
+func (yt *YouTubeBuilder) queryItems(ctx context.Context, feed *model.Feed, since time.Time) error {
+	if !since.IsZero() {
+		return yt.queryItemsSince(ctx, feed, since)
+	}
+
 	var (
 		token       string
 		count       int
 		allSnippets []*youtube.PlaylistItemSnippet
 	)
 
 	for {
-		items, pageToken, err := yt.listPlaylistItems(ctx, feed, token)
+		items, pageToken, err := yt.listPlaylistItems(ctx, feed, token, false)
 		if err != nil {
 			return err
 		}
@@ -450,17 +455,69 @@ func (yt *YouTubeBuilder) queryItems(ctx context.Context, feed *model.Feed) erro
 		}
 	}
 
+	return yt.queryDescriptionsForSnippets(ctx, allSnippets, feed)
+}
+
+// maxDeepScanPages bounds a max_age-driven discovery pass as a quota backstop.
+const maxDeepScanPages = 20
+
+// queryItemsSince pages back through the channel until episodes are older than the cutoff
+// (a max_age-driven deep scan), instead of stopping at PageSize. It is bounded by
+// maxDeepScanPages to guard against a misconfigured max_age draining API quota.
+func (yt *YouTubeBuilder) queryItemsSince(ctx context.Context, feed *model.Feed, since time.Time) error {
+	var (
+		token       string
+		pages       int
+		allSnippets []*youtube.PlaylistItemSnippet
+	)
+
+	for {
+		items, pageToken, err := yt.listPlaylistItems(ctx, feed, token, true)
+		if err != nil {
+			return err
+		}
+
+		token = pageToken
+		pages++
+
+		if len(items) == 0 {
+			break
+		}
+
+		reachedCutoff := false
+		for _, item := range items {
+			// Keep only episodes within the requested window; the page that crosses the
+			// cutoff is where we stop.
+			if date, err := yt.parseDate(item.Snippet.PublishedAt); err == nil {
+				if date.Before(since) {
+					reachedCutoff = true
+					continue
+				}
+			}
+			allSnippets = append(allSnippets, item.Snippet)
+		}
+
+		if reachedCutoff || token == "" {
+			break
+		}
+
+		if pages >= maxDeepScanPages {
+			log.Warnf("deep scan for feed %q reached the %d-page cap before covering max_age; older episodes were not discovered (consider lowering max_age)", feed.ItemID, maxDeepScanPages)
+			break
+		}
+	}
+
+	return yt.queryDescriptionsForSnippets(ctx, allSnippets, feed)
+}
+
+func (yt *YouTubeBuilder) queryDescriptionsForSnippets(ctx context.Context, allSnippets []*youtube.PlaylistItemSnippet, feed *model.Feed) error {
 	snippets := map[string]*youtube.PlaylistItemSnippet{}
 	for _, snippet := range allSnippets {
 		snippets[snippet.ResourceId.VideoId] = snippet
 	}
 
 	// Query video descriptions from the list of ids
-	if err := yt.queryVideoDescriptions(ctx, snippets, feed); err != nil {
-		return err
-	}
-
-	return nil
+	return yt.queryVideoDescriptions(ctx, snippets, feed)
 }
 
 func (yt *YouTubeBuilder) Build(ctx context.Context, cfg *feed.Config) (*model.Feed, error) {
@@ -491,13 +548,14 @@ func (yt *YouTubeBuilder) Build(ctx context.Context, cfg *feed.Config) (*model.F
 		return nil, err
 	}
 
-	if err := yt.queryItems(ctx, _feed); err != nil {
+	if err := yt.queryItems(ctx, _feed, cfg.DiscoverSince); err != nil {
 		return nil, err
 	}
 
 	// YT API client gets 50 episodes per query.
-	// Round up to page size.
-	if len(_feed.Episodes) > _feed.PageSize {
+	// Round up to page size, unless a deep (max_age-driven) scan is requested, in which case
+	// we keep every episode within the window so the caller can catch up the back-catalog.
+	if cfg.DiscoverSince.IsZero() && len(_feed.Episodes) > _feed.PageSize {
 		_feed.Episodes = _feed.Episodes[:_feed.PageSize]
 	}
 

pkg/feed/config.go

@@ -62,6 +62,10 @@ type Config struct {
 	// FilenameTemplate controls output media filename (without extension)
 	// Supported tokens: {{id}}, {{title}}, {{pub_date}}, {{feed_id}}
 	FilenameTemplate string `toml:"filename_template"`
+	// DiscoverSince is a runtime-only hint (not configurable) set by the updater to request a
+	// deep discovery pass: the builder pages back through the channel until episodes are older
+	// than this date instead of stopping at PageSize. Zero means the default shallow discovery.
+	DiscoverSince time.Time `toml:"-"`
 }
 
 type CustomFormat struct {

pkg/feed/xml.go

@@ -11,6 +11,7 @@ import (
 
 	itunes "github.com/eduncan911/podcast"
 	"github.com/pkg/errors"
+	log "github.com/sirupsen/logrus"
 
 	"github.com/mxpv/podsync/pkg/model"
 )
@@ -167,6 +168,14 @@ func Build(_ctx context.Context, feed *model.Feed, cfg *Config, hostname string)
 
 		item.AddEnclosure(downloadURL, enclosureType, episode.Size)
 
+		// p.AddItem requires a non-empty title. An episode without one cannot be
+		// represented in the feed (e.g. metadata was never populated), so skip it
+		// instead of failing the whole feed build for every other episode.
+		if item.Title == "" {
+			log.Warnf("skipping episode %q in feed %q: missing title", episode.ID, cfg.ID)
+			continue
+		}
+
 		// p.AddItem requires description to be not empty, use workaround
 		if item.Description == "" {
 			item.Description = " "

pkg/feed/xml_test.go

@@ -48,6 +48,32 @@ func TestBuildXML(t *testing.T) {
 	assert.EqualValues(t, out.Items[0].Enclosure.Type, itunes.MP4)
 }
 
+func TestBuildXMLSkipsEpisodeWithoutTitle(t *testing.T) {
+	feed := model.Feed{
+		Episodes: []*model.Episode{
+			{
+				ID:          "no-title",
+				Status:      model.EpisodeDownloaded,
+				Description: "description",
+			},
+			{
+				ID:          "ok",
+				Status:      model.EpisodeDownloaded,
+				Title:       "title",
+				Description: "description",
+			},
+		},
+	}
+
+	cfg := Config{ID: "test"}
+
+	out, err := Build(context.Background(), &feed, &cfg, "http://localhost/")
+	// A single episode with a missing title must not fail the whole feed build
+	require.NoError(t, err)
+	require.Len(t, out.Items, 1)
+	assert.Equal(t, "ok", out.Items[0].GUID)
+}
+
 func TestBuildXMLWithFilenameTemplate(t *testing.T) {
 	feed := model.Feed{
 		Episodes: []*model.Episode{

pkg/model/feed.go

@@ -65,6 +65,10 @@ type Feed struct {
 	UpdatedAt       time.Time  `json:"updated_at"`
 	PlaylistSort    Sorting    `json:"playlist_sort"`
 	PrivateFeed     bool       `json:"private_feed"`
+	// ScannedThrough is the oldest publish date discovery has paged back to. It is a high-water
+	// mark used to decide when max_age has been expanded beyond what was ever scanned, so a deep
+	// (max_age-driven) discovery pass is only triggered once per expansion instead of every cycle.
+	ScannedThrough time.Time `json:"scanned_through"`
 }
 
 type EpisodeStatus string

services/update/matcher.go

@@ -42,6 +42,13 @@ func matchFilters(episode *model.Episode, filters *feed.Filters) bool {
 		return false
 	}
 
+	return matchDurationAndAge(episode, filters, logger)
+}
+
+// matchDurationAndAge evaluates the filters that depend only on metadata that is always
+// present on an episode record (duration and publish date). It is used as a cheap pre-filter
+// for cleaned episodes, whose title/description may have to be fetched separately.
+func matchDurationAndAge(episode *model.Episode, filters *feed.Filters, logger log.FieldLogger) bool {
 	if filters.MaxDuration > 0 && episode.Duration > filters.MaxDuration {
 		logger.WithField("filter", "max_duration").Infof("skipping due to duration filter (%ds)", episode.Duration)
 		return false