Task/page content extractor api

CHANGELOG.md

@@ -2,6 +2,18 @@
 
 ## Unreleased
 
+- **Task "PageContentExtractor is public API"**: Extensions can use `PageContentExtractor` to get the text
+a visitor sees on a page, see "Page content" in `Documentation/Usage.md`. `FrontendPageRenderer` is
+internal.
+
+- **Bugfix "Page cache after a frontend render"**: Rendering a page for content extraction
+(`FrontendPageRenderer`, used by `PageContentExtractor`) stored its output in the regular page cache (#43),
+which then went to every visitor. From a backend request that output has file URLs prefixed twice
+(`//fileadmin/...`), because the backend and the frontend `PublicUrlPrefixer` are both registered,
+so visitors got pages with every image broken; from CLI the URLs came out relative. The render now
+runs with the page cache disabled. Pages already cached with such URLs stay broken until the page
+cache is flushed.
+
 - **Bugfix "Capabilities of static catalogs"**: A Symfony AI bridge with a static model catalog
 advertises only the capabilities its models have (#41). Every bridge was granted all of them, so a
 bridge without a single image model was listed with image generation. Catalog models that map to no

Classes/Service/FrontendPageRenderer.php

@@ -22,6 +22,7 @@
 use TYPO3\CMS\Core\Page\PageRenderer;
 use TYPO3\CMS\Core\Site\SiteFinder;
 use TYPO3\CMS\Core\Utility\GeneralUtility;
+use TYPO3\CMS\Frontend\Cache\CacheInstruction;
 use TYPO3\CMS\Frontend\Http\Application;
 
 /**
@@ -31,6 +32,8 @@
  * should treat as "just fall back to something else". No site configured
  * for the page (a common state for a brand-new page), a redirect/error
  * response, or any exception during rendering.
+ *
+ * @internal Use PageContentExtractor.
  */
 final class FrontendPageRenderer
 {
@@ -91,12 +94,26 @@ private function buildServerRequest(int $pageUid): ServerRequestInterface
             'HTTPS' => $uri->getScheme() === 'https' ? 'on' : 'off',
         ];
 
-        $request = new ServerRequest($uri, 'GET', 'php://input', ['Host' => [$httpHost]], $serverParams);
-
-        return $request
+        $request = (new ServerRequest($uri, 'GET', 'php://input', ['Host' => [$httpHost]], $serverParams))
             ->withAttribute('site', $site)
             ->withAttribute('language', $site->getDefaultLanguage())
             ->withAttribute('normalizedParams', NormalizedParams::createFromServerParams($serverParams));
+
+        return $this->withPageCacheDisabled($request);
+    }
+
+    /**
+     * In-process output differs from what visitors get and must neither reach nor come from the page cache.
+     */
+    private function withPageCacheDisabled(ServerRequestInterface $request): ServerRequestInterface
+    {
+        if (!class_exists(CacheInstruction::class)) {
+            // TYPO3 12.4: the frontend middlewares read the "noCache" attribute
+            return $request->withAttribute('noCache', true);
+        }
+        $cacheInstruction = new CacheInstruction();
+        $cacheInstruction->disableCache('EXT:aim: In-process render for content extraction');
+        return $request->withAttribute('frontend.cache.instruction', $cacheInstruction);
     }
 
     /**

Classes/Service/PageContentExtractor.php

@@ -22,9 +22,8 @@
 use TYPO3\CMS\Core\Utility\GeneralUtility;
 
 /**
- * Gets a page's real body copy for feeding into VoiceCalibrationService as
- * an example-text source an editor doesn't have to hand-copy from the
- * frontend.
+ * Public API: a page's visible text, safe to call from backend requests and CLI.
+ * Renders the live default-language page, uncached.
  *
  * Tries an actual frontend render first (FrontendPageRenderer, the only
  * way to see content a plugin renders dynamically, e.g. news teasers, form

Documentation/Usage.md

@@ -327,6 +327,22 @@ $response = $this->pipeline->dispatch($followUp, $resolvedProvider);
 
 Keep looping (execute tool calls → send `toolResults` → check `requiresToolExecution()` again) until the model returns plain content. Always cap the number of rounds: nothing in AiM stops a model from calling tools indefinitely.
 
+## Page content
+
+`PageContentExtractor` returns the text a visitor sees on a page, for use as prompt context:
+
+```php
+public function __construct(
+    private readonly \B13\Aim\Service\PageContentExtractor $pageContentExtractor,
+) {}
+
+$result = $this->pageContentExtractor->extractTextWithSources([$pageId]);
+$result['text'];             // "--- Page title ---\n…", one section per page
+$result['sources'][$pageId]; // 'rendered', 'fallback' or 'none'
+```
+
+It renders the page in-process, uncached, so it is safe to call from backend requests and CLI. Without a renderable page (no site, error response) it falls back to the text fields of the page's content elements (`fallback`), or to nothing (`none`). The render always shows the live page in the default language. `extractText()` returns the text alone.
+
 ## Capabilities
 
 Each provider implements one or more capability interfaces:

Tests/Functional/Fixtures/Extensions/test_internal_request_context/Configuration/RequestMiddlewares.php

@@ -0,0 +1,18 @@
+<?php
+
+declare(strict_types=1);
+
+return [
+    'frontend' => [
+        'b13/aim-test/internal-request-context' => [
+            'target' => \B13\Aim\Tests\Functional\Fixtures\Frontend\InternalRequestContextMiddleware::class,
+            'after' => [
+                'typo3/cms-frontend/backend-user-authentication',
+            ],
+            'before' => [
+                'typo3/json-response/frontend-user-authentication',
+                'typo3/json-response/backend-user-authentication',
+            ],
+        ],
+    ],
+];

Tests/Functional/Fixtures/Extensions/test_internal_request_context/composer.json

@@ -0,0 +1,11 @@
+{
+    "name": "b13/aim-test-internal-request-context",
+    "type": "typo3-cms-extension",
+    "description": "Fixture: adds the testing context to frontend requests not built by the testing framework.",
+    "license": "GPL-2.0-or-later",
+    "extra": {
+        "typo3/cms": {
+            "extension-key": "test_internal_request_context"
+        }
+    }
+}

Tests/Functional/Fixtures/Frontend/InternalRequestContextMiddleware.php

@@ -0,0 +1,35 @@
+<?php
+
+declare(strict_types=1);
+
+/*
+ * This file is part of TYPO3 CMS-based extension "aim" by b13.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ */
+
+namespace B13\Aim\Tests\Functional\Fixtures\Frontend;
+
+use Psr\Http\Message\ResponseInterface;
+use Psr\Http\Message\ServerRequestInterface;
+use Psr\Http\Server\MiddlewareInterface;
+use Psr\Http\Server\RequestHandlerInterface;
+use TYPO3\TestingFramework\Core\Functional\Framework\Frontend\InternalRequestContext;
+
+/**
+ * The testing framework's json_response middlewares expect every frontend
+ * request to carry a "typo3.testing.context" attribute, which only requests
+ * sent through executeFrontendSubRequest() have.
+ */
+final class InternalRequestContextMiddleware implements MiddlewareInterface
+{
+    public function process(ServerRequestInterface $request, RequestHandlerInterface $handler): ResponseInterface
+    {
+        if ($request->getAttribute('typo3.testing.context') === null) {
+            $request = $request->withAttribute('typo3.testing.context', new InternalRequestContext());
+        }
+        return $handler->handle($request);
+    }
+}

Tests/Functional/Fixtures/Frontend/Page.typoscript

@@ -0,0 +1,4 @@
+page = PAGE
+page.10 = TEXT
+page.10.field = title
+page.10.noTrimWrap = |Rendered by the frontend: ||

Tests/Functional/Service/FrontendPageRendererTest.php

@@ -0,0 +1,120 @@
+<?php
+
+declare(strict_types=1);
+
+/*
+ * This file is part of TYPO3 CMS-based extension "aim" by b13.
+ *
+ * It is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License, either version 2
+ * of the License, or any later version.
+ */
+
+namespace B13\Aim\Tests\Functional\Service;
+
+use B13\Aim\Service\FrontendPageRenderer;
+use PHPUnit\Framework\Attributes\Test;
+use Psr\Log\NullLogger;
+use TYPO3\CMS\Core\Cache\Backend\TaggableBackendInterface;
+use TYPO3\CMS\Core\Cache\Backend\Typo3DatabaseBackend;
+use TYPO3\CMS\Core\Cache\CacheManager;
+use TYPO3\CMS\Core\Core\Environment;
+use TYPO3\CMS\Core\Site\SiteFinder;
+use TYPO3\CMS\Core\Utility\GeneralUtility;
+use TYPO3\CMS\Frontend\Http\Application;
+use TYPO3\TestingFramework\Core\Functional\Framework\Frontend\InternalRequest;
+use TYPO3\TestingFramework\Core\Functional\FunctionalTestCase;
+
+final class FrontendPageRendererTest extends FunctionalTestCase
+{
+    protected array $testExtensionsToLoad = [
+        'b13/aim',
+        '../Tests/Functional/Fixtures/Extensions/test_internal_request_context',
+    ];
+
+    /**
+     * The testing framework puts the page cache on the NullBackend, which
+     * would let this test pass whatever the renderer does.
+     */
+    protected array $configurationToUseInTestInstance = [
+        'SYS' => [
+            'caching' => [
+                'cacheConfigurations' => [
+                    'pages' => [
+                        'backend' => Typo3DatabaseBackend::class,
+                    ],
+                ],
+            ],
+        ],
+    ];
+
+    protected function setUp(): void
+    {
+        parent::setUp();
+
+        $this->getConnectionPool()->getConnectionForTable('pages')->insert('pages', [
+            'uid' => 1,
+            'pid' => 0,
+            'title' => 'Home',
+            'slug' => '/',
+            'doktype' => 1,
+            'is_siteroot' => 1,
+        ]);
+        $this->setUpFrontendRootPage(1, ['EXT:aim/Tests/Functional/Fixtures/Frontend/Page.typoscript']);
+
+        $siteDirectory = Environment::getConfigPath() . '/sites/test';
+        GeneralUtility::mkdir_deep($siteDirectory);
+        file_put_contents($siteDirectory . '/config.yaml', implode("\n", [
+            'rootPageId: 1',
+            'base: \'https://example.com/\'',
+            'languages:',
+            '  - languageId: 0',
+            '    title: English',
+            '    locale: en_US.UTF-8',
+            '    base: /',
+            '',
+        ]));
+    }
+
+    #[Test]
+    public function rendersThePageWithoutStoringItInThePageCache(): void
+    {
+        $html = $this->createRenderer()->renderPageHtml(1);
+
+        self::assertNotNull($html);
+        self::assertStringContainsString('Rendered by the frontend: Home', $html);
+        self::assertSame([], $this->getPageCacheBackend()->findIdentifiersByTag('pageId_1'));
+    }
+
+    #[Test]
+    public function rendersThePageWithoutReadingItFromThePageCache(): void
+    {
+        $this->executeFrontendSubRequest(new InternalRequest('https://example.com/'));
+        self::assertNotSame([], $this->getPageCacheBackend()->findIdentifiersByTag('pageId_1'));
+        // Changing the record directly leaves the cached page stale
+        $this->getConnectionPool()->getConnectionForTable('pages')->update('pages', ['title' => 'Renamed'], ['uid' => 1]);
+        // The frontend uses its own CacheManager instance, whose runtime cache still holds the record
+        GeneralUtility::makeInstance(CacheManager::class)->getCache('runtime')->flush();
+
+        $html = $this->createRenderer()->renderPageHtml(1);
+
+        self::assertNotNull($html);
+        self::assertStringContainsString('Rendered by the frontend: Renamed', $html);
+    }
+
+    private function createRenderer(): FrontendPageRenderer
+    {
+        return new FrontendPageRenderer(
+            $this->get(Application::class),
+            $this->get(SiteFinder::class),
+            new NullLogger(),
+        );
+    }
+
+    private function getPageCacheBackend(): TaggableBackendInterface
+    {
+        $pageCacheBackend = $this->get(CacheManager::class)->getCache('pages')->getBackend();
+        self::assertInstanceOf(TaggableBackendInterface::class, $pageCacheBackend);
+        return $pageCacheBackend;
+    }
+}