.authors attribute

[lemon24]

Like #320, but for authors.

Currently Entry, Feed and EntrySource expose .author, which looks like John Doe (john@example.com) (presumably formatted like this by feedparser).

The name and email are actually available separately, and there can be more than one author, so we should provide them like that.

[Anderson688]

Hi @lemon24 , I'd like to take a swing at implementing this! I've been looking through src/reader/types.py and the storage layers, and I have a few quick architectural questions before I put together a PR:

1. Should we introduce a new Author dataclass/NamedTuple? (containing name, email, href).

2. What is the plan for the existing .author string attribute? Should it be kept alongside .authors for backward compatibility, marked for deprecation, or removed entirely?

3. Since .authors will be a sequence of objects, should I add a new authors column to the feeds and entries SQLite tables to store them as JSON (similar to how enclosures and content are handled)?

Let me know your thoughts and I can get started!

[lemon24]

Hi @Anderson688, thank you for looking into this!

1. Yes, a frozen dataclass should be OK.
2. Can't remove .author yet due to backwards compatibility, but it can be made a property that returns name (email), ... and raises deprecation warning.
3. Technically yes, but I'd suggest to use the existing column for now, since SQLite got DROP COLUMN only in 2021 (I'll have to rename other columns at some point anyway). A migration will be needed to transform the current values into JSON (you can register a temporary SQLite function for this).

---

But, there's a complication – it turns out I was wrong in the issue description, feedparser exposes only one author in author_detail, kinda:

• Atom feeds can technically support more than one author, I guess we get the first one(?) (and author is set to name (email)). We can use author_detail as-is. (There's also contributors, but that is empty for all my feeds, so we can ignore it for now.)
• For RSS feeds, there's a single text <author> tag; feedparser does its best to extract the name and email, to some degree of success, which doesn't always work when there's more than one author. For RSS feeds, I'd suggest parsing author instead of using author_detail (and the same function can be used in the migration).

Here's a summary of the various author and author_detail values for my feeds (the multiple names and the weird stuff all come from RSS feeds):

# 'author' {...author_detail...}

# normal variants

'John Doe' {'name': 'John Doe'}
'John Doe (john@example.com)' 
    {'name': 'John Doe', 'email': 'john@example.com'}
'John Doe (john@example.com)' 
    {'name': 'John Doe', 'email': 'john@example.com', 'href': 'http://example.com'}
'John Doe'
    {'name': 'John Doe', 'href': 'http://example.com'}

# multiple names

# comma-separated, email not in name
# (but `author_detail[email]` exists, so we should use it)
'John Doe, Jane Smith, William Shakespeare' 
    {'name': 'John Doe, Jane Smith, William Shakespeare', 'email': 'blah@example.com'}

# "and" instead of comma (!)
'John Doe, Jane Smith and William Shakespeare'
    {'name': 'John Doe, Jane Smith and William Shakespeare'}

# but "and" is not always a separator 
# (how do we tell it apart from ^^^?; we don't – split only by comma)
'death and gravity' {'name': 'death and gravity'}

# each name has email (note author_detail[name] is mangled!)
# (if author has multiple names with emails, ignore author_detail) 
'John Doe (john@doe.com), Jane Smith (jane@smith.com)' 
    {'name': 'John Doe , Jane Smith (jane@smith.com', 'email': 'john@doe.com'}

# weird stuff

# not using @ in email (also it's "email (name)", not "name (email)")
# (feedparser makes sure it's "name (email)", but here it can't tell which is an email;
# it's fine to always assume "name (email)")
'me[at]example.com (John Doe)' {'name': 'me[at]example.com (John Doe)'}

# name in author, but not in author_detail
# (email comes from RSS <author>, name comes from <itunes:author>)
'John Doe' {'email': 'john@example.com'}

# empty fields (empty strings should be None; if all empty, skip the author)
'' {'name': ''}
'' {'name': '', 'href': 'http://example.com'}
'()' {'name': '()'}
'John Doe' {'name': 'John Doe', 'href': ''}

I used this script to extract the values above (summary done by hand):

import re, pathlib
import requests, feedparser
from reader import make_reader

out_dir = pathlib.Path('_feeds')

def save_feeds():
    out_dir.mkdir(exist_ok=True)
    reader = make_reader('db.sqlite')
    for feed in reader.get_feeds(updates_enabled=True):
        out_path = out_dir / re.sub(r'\W+', '-', feed.url)
        if out_path.exists():
            continue
        try:
            response = requests.get(feed.url)
            response.raise_for_status()
        except Exception:
            out_path.touch()
            continue
        out_path.write_bytes(response.content)
        print('done', feed.url)

save_feeds()

def print_authors(thing, version):
    if 'author' in thing:
        print('author:     ', repr(thing.author), thing.author_detail, version)
    print('authors:    ', thing.get('authors'))
    for contributor in thing.get('contributors', ()):
        print('contributor: ', contributor, version)

for feed_path in out_dir.glob('*'):
    feed = feedparser.parse(feed_path.read_bytes())
    version = feed.get('version')
    print_authors(feed.feed, version)
    for entry in feed.entries:
        print_authors(entry, version)

[lemon24]

BTW, if parsing the RSS author is too much, I can take care of it, please let me know.

[Anderson688]

Hi @lemon24, looking at the legacy web app, legacy/templates/entry.html currently just does {% if entry.author %} by {{ entry.author }}{% endif %}.

With the new .authors sequence containing name, email, and href, how would you like this rendered in the UI?

1. Just a comma-separated list of names? (e.g., by John Doe, Jane Smith)
2. Should we hyperlink the names if the href attribute is present?
3. Should we include the emails in the UI like the old string did, or keep it clean with just the names/links?

[lemon24]

I think a comma-separated list of names (no email) should be enough for now (you can make .author return that, then no change will be needed).

[lemon24]

Long-term, it may be useful to have some sort of link and/or tooltip (but decisions are needed – href? email? both?), and to show only one or two names, with the rest hidden behind some sort of collapsible, e.g. by John Doe, ....

But IMHO it seems overkill / risks cluttering the UI, so we can leave it for later (a separate PR, at least); anyway, even with lots of names, it doesn't look completely awful right now: