release: dev → main (email pipeline refactor + forwarding, placeholders, tracking, attachments) (#378)

* Feat: message sending-forwarding (#374)

* UserPersonalizer in CampaignProcessorMessageHandler

* HtmlToText

* MessageDataLoader

* TextParser

* RemotePageFetcher

* Use repo methods

* Use MessagePrecacheDto

* Refactor

* Todo

* SystemMailConstructor

* EmailBuilder

* InjectedByHeaderSubscriber

* TemplateImageManager

* ExternalImageCacher

* TemplateImageEmbedder

* Mailer

* RemotePageFetcherTest

* TextParserTest

* MessageDataLoaderTest

* MessageDataLoaderTest

* Test fix

* Fix: phpmd

* Fix: phpcs

* After review 0

* After review 1

* Add tests

* EmailBuilderTest

* update coderabbit.yaml

* Add tests

* MailSizeChecker

* Feat/email building with attachments (#375)

New Features
    PDF generation for messages, per-subscriber remote-content fetching, tracking-pixel user tracking, and richer attachment handling with downloadable copies.

Improvements
    Unified email builder flow with consistent composition and multi-format output (HTML/Text/PDF); expanded, context-aware placeholder personalization (many URL/list resolvers); improved remote-content precaching and output formatting; new configurable parameters and translations.

---------

Co-authored-by: Tatevik <tatevikg1@gmail.com>

* Feat: email forwarding (#377)

-  Message forwarding: send campaigns to friends (optional personal note), per-user limits, admin notifications on success/failure, and forwarding statistics; forwarded messages prefixed "Fwd".
-  Admin-copy emails: configurable toggle to send admin copies and select recipients.

---------

Co-authored-by: Tatevik <tatevikg1@gmail.com>

* Cutoff from forward_email_period config

* ForwardingResult

* Remove MessageFormat consts

* Testing bundle

* After review 3

* MessageDataLoader types

* Fix HTMLPurifier_Config

---------

Co-authored-by: Tatevik <tatevikg1@gmail.com>

* AttachmentDownloadService (#379)

New Features

    Attachment download service that validates access, resolves files, detects MIME types, and returns downloadable content.
    Lightweight downloadable attachment DTO and a new exception for missing attachment files.
    Public constant to mark forwarded attachments.

Bug Fixes

    Attachment download links now use a path-based format with encoded UID.

* Feat: user message open tracking (#380)

New Features

    Message view tracking: records when subscribers view messages and captures metadata (IP, User-Agent, referer).

Enhancements

    Quick actions to mark messages as viewed and to check viewed status.
    Message view counters now increment when viewed.

Refactor

    Repository lookup renamed for consistency and callers updated.

Chores

    Switched REST API parameter to a base URL and updated tracking image path.

* Remove 'to' from MessagePrecacheDto (#381)

    Refactor
        Recipient addresses are now passed explicitly throughout the email pipeline for clearer, more reliable message construction and handling.
        Subscriber creation now requires an email at instantiation, enforcing consistent initialization.
    Bug Fixes
        Admin notification filtering relaxed so system notifications are handled more consistently even when recipient info is missing.

* Upgrade guzzle

* Fix: manager configuration

* Subscriber getFilteredAfterId innerJoin => lefftJoin

* Add: SubscriberFilter options

* Fix: subscriber prop name in query builder condition

* PaginatedResult

* remove sorting

* Add getSubscriberDetails method

* Add: front-end url

* Fix: ExtraData

* Add:getSummaryStatistics

* Add: getCampaignPerformance, getRecentCampaigns

* Ref: FilterRequestInterface

* Ref: PaginatedFilter base class

* Fix: types

* Add: updateSubscriberList

* Add: fields to export

* Fix: subscriber attribute resolution

* Fix: subscribers getFilteredAfterId

* createSubscriptions with autoConfirm

* uniqueByMessageId

* todo

* Fix: metadata processed

* Campaign processing for handling specific lists

---------

Co-authored-by: Tatevik <tatevikg1@gmail.com>

Author: TatevikGr

.coderabbit.yaml

@@ -8,18 +8,21 @@ reviews:
   suggested_labels: false
   high_level_summary_in_walkthrough: false
   poem: false
+  finishing_touches:
+    docstrings:
+      enabled: false
   path_instructions:
     - path: "src/Domain/**"
       instructions: |
         You are reviewing PHP domain-layer code. Enforce domain purity, with a relaxed policy for DynamicListAttr:
 
-        - ❌ Do not allow persistence or transaction side effects here for *normal* domain models.
-        - Flag ANY usage of Doctrine persistence APIs on regular domain entities, especially:
+        - ❌ Do not allow, flag ANY DB write / finalization:
           - `$entityManager->flush(...)`, `$this->entityManager->flush(...)`
-          - `$em->persist(...)`, `$em->remove(...)`
-          - `$em->beginTransaction()`, `$em->commit()`, `$em->rollback()`
+          - `$em->beginTransaction()`, `$em->commit()`, `$em->rollback()`, `$em->transactional(...)`
+          - `$em->getConnection()->executeStatement(...)` for DML/DDL (INSERT/UPDATE/DELETE/ALTER/...)
         - ✅ Accessing Doctrine *metadata*, *schema manager*, or *read-only schema info* is acceptable
-          as long as it does not modify state or perform writes.
+          as long as it does not modify state or perform writes. Accessing Doctrine *persistence APIs* 
+          persist, remove, etc.) is acceptable, allow scheduling changes in the UnitOfWork (no DB writes)
 
         - ✅ **Relaxed rule for DynamicListAttr-related code**:
           - DynamicListAttr is a special case dealing with dynamic tables/attrs.

composer.json

@@ -1,7 +1,7 @@
 {
     "name": "phplist/core",
     "description": "The core module of phpList, the world's most popular open source newsletter manager",
-    "type": "phplist-module",
+    "type": "symfony-bundle",
     "keywords": [
         "phplist",
         "email",
@@ -46,6 +46,7 @@
     },
     "require": {
         "php": "^8.1",
+        "symfony/framework-bundle": "^6.4",
         "symfony/dependency-injection": "^6.4",
         "symfony/config": "^6.4",
         "symfony/yaml": "^6.4",
@@ -79,11 +80,17 @@
         "ext-imap": "*",
         "tatevikgr/rss-feed": "dev-main",
         "ext-pdo": "*",
-        "ezyang/htmlpurifier": "^4.19"
+        "ezyang/htmlpurifier": "^4.19",
+        "ext-libxml": "*",
+        "ext-gd": "*",
+        "ext-curl": "*",
+        "ext-fileinfo": "*",
+        "setasign/fpdf": "^1.8",
+        "phpdocumentor/reflection-docblock": "^5.2",
+        "guzzlehttp/guzzle": "^7.4.5"
     },
     "require-dev": {
         "phpunit/phpunit": "^9.5",
-        "guzzlehttp/guzzle": "^6.3.0",
         "squizlabs/php_codesniffer": "^3.2.0",
         "phpstan/phpstan": "^1.10",
         "nette/caching": "^3.0.0",
@@ -92,7 +99,6 @@
         "symfony/test-pack": "^1.1",
         "symfony/process": "^6.4",
         "composer/composer": "^2.7",
-        "symfony/framework-bundle": "^6.4",
         "symfony/http-kernel": "^6.4",
         "symfony/http-foundation": "^6.4",
         "symfony/routing": "^6.4",
@@ -152,8 +158,8 @@
                 "Doctrine\\Bundle\\DoctrineBundle\\DoctrineBundle",
                 "Doctrine\\Bundle\\MigrationsBundle\\DoctrineMigrationsBundle",
                 "PhpList\\Core\\EmptyStartPageBundle\\EmptyStartPageBundle",
-                "FOS\\RestBundle\\FOSRestBundle",
-                "TatevikGr\\RssFeedBundle\\RssFeedBundle"
+                "PhpList\\Core\\EmptyStartPageBundle\\PhpListCoreBundle",
+                "FOS\\RestBundle\\FOSRestBundle"
             ],
             "routes": {
                 "homepage": {

config/PHPMD/rules.xml

@@ -7,7 +7,7 @@
     <exclude-pattern>*/Migrations/*</exclude-pattern>
 
     <!-- rules from the "clean code" rule set -->
-    <rule ref="rulesets/cleancode.xml/BooleanArgumentFlag"/>
+<!--    <rule ref="rulesets/cleancode.xml/BooleanArgumentFlag"/>-->
     <rule ref="rulesets/codesize.xml/CyclomaticComplexity"/>
     <rule ref="rulesets/codesize.xml/NPathComplexity"/>
     <rule ref="rulesets/codesize.xml/ExcessiveMethodLength"/>
@@ -33,20 +33,20 @@
     <rule ref="rulesets/design.xml/DepthOfInheritance"/>
     <rule ref="rulesets/design.xml/CouplingBetweenObjects">
         <properties>
-            <property name="maximum" value="15"/>
+            <property name="maximum" value="17"/>
         </properties>
     </rule>
     <rule ref="rulesets/design.xml/DevelopmentCodeFragment"/>
 
     <!-- rules from the "naming" rule set -->
     <rule ref="rulesets/naming.xml/ShortVariable">
         <properties>
-            <property name="exceptions" value="id,ip,cc,io"/>
+            <property name="exceptions" value="id,ip,cc,io,to"/>
         </properties>
     </rule>
     <rule ref="rulesets/naming.xml/LongVariable">
         <properties>
-            <property name="maximum" value="25"/>
+            <property name="maximum" value="30"/>
         </properties>
     </rule>
     <rule ref="rulesets/naming.xml/ShortMethodName"/>

config/PhpCodeSniffer/ruleset.xml

@@ -103,6 +103,10 @@
     <rule ref="PEAR.WhiteSpace.ScopeClosingBrace"/>
     <rule ref="Squiz.WhiteSpace.CastSpacing"/>
     <rule ref="Squiz.WhiteSpace.LogicalOperatorSpacing"/>
-    <rule ref="Squiz.WhiteSpace.OperatorSpacing"/>
+    <rule ref="Squiz.WhiteSpace.OperatorSpacing">
+        <properties>
+            <property name="ignoreNewlines" value="true"/>
+        </properties>
+    </rule>
     <rule ref="Squiz.WhiteSpace.SemicolonSpacing"/>
 </ruleset>

config/parameters.yml.dist

@@ -25,16 +25,28 @@ parameters:
     env(DATABASE_PREFIX): 'phplist_'
     list_table_prefix: '%%env(LIST_TABLE_PREFIX)%%'
     env(LIST_TABLE_PREFIX): 'listattr_'
+    app.dev_version: '%%env(APP_DEV_VERSION)%%'
+    env(APP_DEV_VERSION): '0'
+    app.dev_email: '%%env(APP_DEV_EMAIL)%%'
+    env(APP_DEV_EMAIL): 'dev@dev.com'
+    app.powered_by_phplist: '%%env(APP_POWERED_BY_PHPLIST)%%'
+    env(APP_POWERED_BY_PHPLIST): '0'
+    app.preference_page_show_private_lists: '%%env(PREFERENCEPAGE_SHOW_PRIVATE_LISTS)%%'
+    env(PREFERENCEPAGE_SHOW_PRIVATE_LISTS): '0'
+    app.rest_api_base_url: '%%env(REST_API_BASE_URL)%%'
+    env(REST_API_BASE_URL): 'http://api.phplist.local/api/v2'
+    app.frontend_base_url: '%%env(FRONT_END_BASE_URL)%%'
+    env(FRONT_END_BASE_URL): 'http://frontend.phplist.local'
 
     # Email configuration
     app.mailer_from: '%%env(MAILER_FROM)%%'
     env(MAILER_FROM): 'noreply@phplist.com'
     app.mailer_dsn: '%%env(MAILER_DSN)%%'
-    env(MAILER_DSN): 'null://null'
+    env(MAILER_DSN): 'null://null' # set local_domain on transport
     app.confirmation_url: '%%env(CONFIRMATION_URL)%%'
-    env(CONFIRMATION_URL): 'https://example.com/subscriber/confirm/'
+    env(CONFIRMATION_URL): 'http://api.phplist.local/api/v2/subscriber/confirm/'
     app.subscription_confirmation_url: '%%env(SUBSCRIPTION_CONFIRMATION_URL)%%'
-    env(SUBSCRIPTION_CONFIRMATION_URL): 'https://example.com/subscription/confirm/'
+    env(SUBSCRIPTION_CONFIRMATION_URL): 'http://api.phplist.local/api/v2/subscription/confirm/'
     app.password_reset_url: '%%env(PASSWORD_RESET_URL)%%'
     env(PASSWORD_RESET_URL): 'https://example.com/reset/'
 
@@ -71,8 +83,10 @@ parameters:
     # A secret key that's used to generate certain security-related tokens
     secret: '%%env(PHPLIST_SECRET)%%'
     env(PHPLIST_SECRET): %1$s
+    phplist.verify_ssl: '%%env(VERIFY_SSL)%%'
+    env(VERIFY_SSL): '1'
 
-    graylog_host: 'graylog.example.com'
+    graylog_host: 'graylog.phplist.local'
     graylog_port: 12201
 
     app.phplist_isp_conf_path: '%%env(APP_PHPLIST_ISP_CONF_PATH)%%'
@@ -89,3 +103,57 @@ parameters:
     env(MESSAGING_MAX_PROCESS_TIME): '600'
     messaging.max_mail_size: '%%env(MAX_MAILSIZE)%%'
     env(MAX_MAILSIZE): '209715200'
+    messaging.default_message_age: '%%env(DEFAULT_MESSAGEAGE)%%'
+    env(DEFAULT_MESSAGEAGE): '691200'
+    messaging.use_manual_text_part: '%%env(USE_MANUAL_TEXT_PART)%%'
+    env(USE_MANUAL_TEXT_PART): '0'
+    messaging.blacklist_grace_time: '%%env(MESSAGING_BLACKLIST_GRACE_TIME)%%'
+    env(MESSAGING_BLACKLIST_GRACE_TIME): '600'
+    messaging.google_sender_id: '%%env(GOOGLE_SENDERID)%%'
+    env(GOOGLE_SENDERID): ''
+    messaging.use_amazon_ses: '%%env(USE_AMAZONSES)%%'
+    env(USE_AMAZONSES): '0'
+    messaging.use_precedence_header: '%%env(USE_PRECEDENCE_HEADER)%%'
+    env(USE_PRECEDENCE_HEADER): '0'
+    messaging.embed_external_images: '%%env(EMBEDEXTERNALIMAGES)%%'
+    env(EMBEDEXTERNALIMAGES): '0'
+    messaging.embed_uploaded_images: '%%env(EMBEDUPLOADIMAGES)%%'
+    env(EMBEDUPLOADIMAGES): '0'
+    messaging.external_image_max_age: '%%env(EXTERNALIMAGE_MAXAGE)%%'
+    env(EXTERNALIMAGE_MAXAGE): '0'
+    messaging.external_image_timeout: '%%env(EXTERNALIMAGE_TIMEOUT)%%'
+    env(EXTERNALIMAGE_TIMEOUT): '30'
+    messaging.external_image_max_size: '%%env(EXTERNALIMAGE_MAXSIZE)%%'
+    env(EXTERNALIMAGE_MAXSIZE): '204800'
+    messaging.forward_alternative_content: '%%env(FORWARD_ALTERNATIVE_CONTENT)%%'
+    env(FORWARD_ALTERNATIVE_CONTENT): '0'
+    messaging.email_text_credits: '%%env(EMAILTEXTCREDITS)%%'
+    env(EMAILTEXTCREDITS): '0'
+    messaging.always_add_user_track: '%%env(ALWAYS_ADD_USERTRACK)%%'
+    env(ALWAYS_ADD_USERTRACK): '1'
+    messaging.send_list_admin_copy: '%%env(SEND_LISTADMIN_COPY)%%'
+    env(SEND_LISTADMIN_COPY): '0'
+
+    phplist.forward_email_period: '%%env(FORWARD_EMAIL_PERIOD)%%'
+    env(FORWARD_EMAIL_PERIOD): '1 minute'
+    phplist.forward_email_count: '%%env(FORWARD_EMAIL_COUNT)%%'
+    env(FORWARD_EMAIL_COUNT): '1'
+    phplist.forward_personal_note_size: '%%env(FORWARD_PERSONAL_NOTE_SIZE)%%'
+    env(FORWARD_PERSONAL_NOTE_SIZE): '0'
+    phplist.forward_friend_count_attribute: '%%env(FORWARD_FRIEND_COUNT_ATTRIBUTE)%%'
+    env(FORWARD_FRIEND_COUNT_ATTRIBUTE): ''
+    phplist.keep_forwarded_attributes: '%%env(KEEPFORWARDERATTRIBUTES)%%'
+    env(KEEPFORWARDERATTRIBUTES): '0'
+
+    phplist.upload_images_dir: '%%env(PHPLIST_UPLOADIMAGES_DIR)%%'
+    env(PHPLIST_UPLOADIMAGES_DIR): 'images'
+    phplist.editor_images_dir: '%%env(FCKIMAGES_DIR)%%'
+    env(FCKIMAGES_DIR): 'uploadimages'
+    phplist.public_schema: '%%env(PUBLIC_SCHEMA)%%'
+    env(PUBLIC_SCHEMA): 'https'
+    phplist.attachment_download_url: '%%env(PHPLIST_ATTACHMENT_DOWNLOAD_URL)%%'
+    env(PHPLIST_ATTACHMENT_DOWNLOAD_URL): 'https://example.com/download/'
+    phplist.attachment_repository_path: '%%env(PHPLIST_ATTACHMENT_REPOSITORY_PATH)%%'
+    env(PHPLIST_ATTACHMENT_REPOSITORY_PATH): '/tmp'
+    phplist.max_avatar_size: '%%env(MAX_AVATAR_SIZE)%%'
+    env(MAX_AVATAR_SIZE): '100000'

config/services.yml

@@ -57,10 +57,9 @@ services:
     calls:
       - [ set, [ 'Cache.SerializerPath', '%kernel.cache_dir%/htmlpurifier' ] ]
       - [ set, [ 'HTML.ForbiddenElements', [ 'script', 'style' ] ] ]
-      - [ set, [ 'CSS.Disable', true ] ]
-      - [ set, [ 'URI.DisableJavaScript', true ] ]
-      - [ set, [ 'URI.DisableDataURI', true ] ]
-      - [ set, [ 'HTML.Doctype', 'HTML5' ] ]
+      - [ set, [ 'CSS.AllowedProperties', [] ] ]
+      - [ set, [ 'URI.AllowedSchemes', { http: true, https: true, mailto: true } ] ]
+      - [ set, [ 'HTML.Doctype', 'XHTML 1.0 Transitional' ] ]
       - [ set, [ 'HTML.Allowed', 'p,br,b,strong,i,em,u,a[href|title],ul,ol,li,blockquote,img[src|alt|title],span,div'] ]
 
   HTMLPurifier:

config/services/builders.yml

@@ -4,22 +4,34 @@ services:
     autoconfigure: true
     public: false
 
-  PhpList\Core\Domain\Messaging\Service\Builder\MessageBuilder:
-    autowire: true
-    autoconfigure: true
+  PhpList\Core\Domain\:
+    resource: '../../src/Domain/*/Service/Builder/*'
 
-  PhpList\Core\Domain\Messaging\Service\Builder\MessageFormatBuilder:
-    autowire: true
-    autoconfigure: true
+  # Concrete mail constructors
+  PhpList\Core\Domain\Messaging\Service\Constructor\SystemMailContentBuilder: ~
+  PhpList\Core\Domain\Messaging\Service\Constructor\CampaignMailContentBuilder: ~
 
-  PhpList\Core\Domain\Messaging\Service\Builder\MessageScheduleBuilder:
-    autowire: true
-    autoconfigure: true
+  # Two EmailBuilder services with different constructors injected
+  PhpList\Core\Domain\Messaging\Service\Builder\SystemEmailBuilder:
+    arguments:
+      $googleSenderId: '%messaging.google_sender_id%'
+      $useAmazonSes: '%messaging.use_amazon_ses%'
+      $usePrecedenceHeader: '%messaging.use_precedence_header%'
+      $devVersion: '%app.dev_version%'
+      $devEmail: '%app.dev_email%'
 
-  PhpList\Core\Domain\Messaging\Service\Builder\MessageContentBuilder:
-    autowire: true
-    autoconfigure: true
+  PhpList\Core\Domain\Messaging\Service\Builder\EmailBuilder:
+    arguments:
+      $googleSenderId: '%messaging.google_sender_id%'
+      $useAmazonSes: '%messaging.use_amazon_ses%'
+      $usePrecedenceHeader: '%messaging.use_precedence_header%'
+      $devVersion: '%app.dev_version%'
+      $devEmail: '%app.dev_email%'
 
-  PhpList\Core\Domain\Messaging\Service\Builder\MessageOptionsBuilder:
-    autowire: true
-    autoconfigure: true
+  PhpList\Core\Domain\Messaging\Service\Builder\ForwardEmailBuilder:
+    arguments:
+      $googleSenderId: '%messaging.google_sender_id%'
+      $useAmazonSes: '%messaging.use_amazon_ses%'
+      $usePrecedenceHeader: '%messaging.use_precedence_header%'
+      $devVersion: '%app.dev_version%'
+      $devEmail: '%app.dev_email%'