diff --git a/index.bs b/index.bs index de706bb..46f74b6 100644 --- a/index.bs +++ b/index.bs @@ -45,6 +45,9 @@ Complain about:accidental-2119 yes, missing-example-ids yes } } +
+spec:storage; type:dfn; text:storage key
+

Introduction

@@ -136,6 +139,110 @@ This does not preclude adding support for this as a future API enhancement, and The above recommendations are intended to reduce this risk of such attacks. +
+ +

On-Device Model Privacy Considerations

+ +

This section is non-normative.

+ +

Language Pack Availability

+ +For on-device speech recognition, the exact download status of language packs can present a fingerprinting vector. How many bits this vector provides depends on the options provided to {{SpeechRecognition/available()}} or {{SpeechRecognition/install()}}, and how they influence the download (e.g., if different language packs have different availability statuses). + +
Download Masking
+ +To prevent cross-origin fingerprinting and site collusion, the specification requires user agents to partition language pack availability by [=storage key=] and mask the current download status by returning {{"downloadable"}} from {{SpeechRecognition/available()}} by default, even if the underlying language pack is {{"available"}} or {{"downloading"}} on the user's device (see the download masking requirements for the [=download masking state=] under {{SpeechRecognition/available()}} and this step of the [=availability algorithm=]). + +Because the [=download masking state=] for a given (options, [=storage key=]) tuple is only unmasked after a web page within that [=storage key=] explicitly calls {{SpeechRecognition/install()}} (see this step of {{SpeechRecognition/install()}})—and because {{SpeechRecognition/start()}} also treats masked language packs as unavailable when {{SpeechRecognition/processLocally}} is `true` (see this step of the [=start session algorithm=])—web pages only gain access to the true download status after taking a more costly action, such as consuming a user activation or triggering a user permission prompt (see Installation-Time Friction). + +The requirement to clear the download masking state whenever other storage for that [=storage key=] is cleared prevents the unmasked state from persisting across storage resets. + +
Installation-Time Friction
+ +The mitigation described in Download Masking works against attempts to silently fingerprint using {{SpeechRecognition/available()}}. The specification also contains requirements to prevent {{SpeechRecognition/install()}} from being easily used for fingerprinting, by introducing friction: + +* The {{SpeechRecognition/install()}} method both requires and consumes [=transient activation=] when, as far as the calling web page can observe, it would initiate a download (see this step of the {{SpeechRecognition/install()}} algorithm). +* The {{SpeechRecognition/install()}} method allows the user agent to prompt the user for permission, or to implicitly reject download attempts based on previous signals (such as an observed pattern of abuse; see this step of the {{SpeechRecognition/install()}} algorithm). +* Access to {{SpeechRecognition/install()}} and {{SpeechRecognition/available()}} is gated behind the [=policy-controlled feature=] "on-device-speech-recognition", which means that only top-level origins and their delegates can use these methods. + +Additionally, initiating the download process via {{SpeechRecognition/install()}} is more or less a one-time operation for a given language. The availability status will only transition from {{"downloadable"}} to {{"downloading"}} to {{"available"}} via these guarded installation operations. That is, while {{SpeechRecognition/install()}} can be used to read some of these fingerprinting bits (by observing the resolution of its promise and subsequent calls to {{SpeechRecognition/available()}}), doing so will effectively "destroy" those bits by changing the state. + +(For details on cases where downloading might happen more than once, and how privacy and security are preserved in those cases, see Download Cancelation, Download Eviction, and Disk Space for Language Packs.) + +
Download Cancelation
+ +Because the availability of a language pack is strictly partitioned by [=storage key=] (see Download Masking), a web page can only observe the download state within its own partition. + +Therefore, if a user agent cancels an ongoing download due to page-controlled actions (e.g., navigation, page unload, or an iframe being destroyed) as permitted by the cancelation rules for {{SpeechRecognition/install()}}, the resulting change in state (e.g., reverting from {{"downloading"}} to {{"downloadable"}}) is only visible to that specific [=storage key=], as long as canceling does not alter the availability status observable by other [=storage keys=]. This prevents the cancelation behavior from being used as a cross-origin fingerprinting vector. + +
Download Eviction
+ +Another ingredient in ensuring that websites cannot toggle the availability state back and forth is to ensure that user agents don't use a quota-based eviction system for downloaded language packs that web pages can indirectly control. For example, if a user agent evicted less-recently-used language packs when new ones are installed, a web page could trigger such evictions to toggle the state of a target language pack. + +To avoid this, the download eviction rules for {{SpeechRecognition/install()}} discourage systems that allow web pages to control the eviction of downloaded language packs, including via indirect triggers such as further subsequent downloads. One way to fulfill this is to never evict downloaded material in response to web page-initiated storage pressure, instead refusing to download new material (e.g., {{SpeechRecognition/install()}} resolving to `false`) if doing so would cause storage pressure. + +Evicting downloads in response to user-controlled actions (e.g., via a browser settings UI) is not problematic. + +

Sensitive Language Information

+ +Even if fingerprinting risks from availability status are mitigated, knowing a user has downloaded a specific language pack (e.g., for a minority language) can be sensitive. + +For this reason, on top of installation-time friction, this step of the {{SpeechRecognition/install()}} algorithm allows user agents to artificially fake a download (e.g., by adding a delay to the resolution of the {{SpeechRecognition/install()}} promise while reporting {{"downloading"}} for that [=storage key=]) if they believe it would be helpful for privacy reasons, instead of having {{SpeechRecognition/install()}} resolve instantly if the language pack is already present. This provides plausible deniability: if {{SpeechRecognition/install()}} takes a few seconds to resolve `true`, it could be a fake delay or a quick real download. + +Such fake delays are not foolproof but offer some privacy benefit, especially when combined with other mitigations like prompts. + +

Model Version

+ +The specific version or behavior of an on-device speech recognition model can also be a fingerprinting vector. This API does not expose model versions directly. + +The best way to prevent the model version from becoming a fingerprinting vector is to tie it to the user agent's version, such that the model's version (and behavior) only updates alongside already-exposed information (like the User-Agent string). Accordingly, the model version rules under {{SpeechRecognition/available()}} encourage limiting the number of possible model versions that a single user agent version can be paired with when determining if a language pack is {{"available"}} via {{SpeechRecognition/available()}}. This might involve not providing model updates to older user agent versions or ignoring already-downloaded models below a minimum version threshold after a user agent update (instead, {{SpeechRecognition/available()}} might report {{"downloadable"}} for a newer version). + +There is a tradeoff: aggressively locking new user agent versions to new model versions can increase transitions between {{"available"}} and {{"downloadable"}}. This can be mitigated by allowing older models with newer user agent versions while a new model downloads, keeping the status {{"available"}} but briefly allowing identification of users with older-model/newer-user-agent combinations. + +

User Input and Speech Data

+ +Speech data is inherently sensitive. The user input retention rules under {{SpeechRecognition/processLocally}} prohibit implementations from training or fine-tuning on-device speech recognition models on user speech input obtained through this API, or otherwise storing user speech input in a way that models can consult in the future (e.g., for personalization beyond the current session or across origins). + +Using user speech input in such a way would be a significant privacy leak, potentially exposing user information or information derived from interactions with one site to another. + +This reinforces the existing requirement in Security and privacy considerations against personalizing speech recognition when performing speech recognition on a {{MediaStreamTrack}}, and applies broadly to any speech processed by on-device models via this API. + +

Cloud-Based vs. On-Device Implementations

+ +The Web Speech API can support both server-based (cloud) and client-based/embedded (on-device) recognition and synthesis. The {{SpeechRecognition/processLocally}} attribute allows developers to indicate a requirement for on-device processing. + +When {{SpeechRecognition/processLocally}} is `false` (the default), user speech data might be sent to a remote server for processing. Web developers need to be aware of this possibility and the associated privacy implications if they do not explicitly request local processing, and user agents are encouraged to be transparent with users about where speech processing occurs. + +When {{SpeechRecognition/processLocally}} is `true`, the considerations in this "On-Device Model Privacy Considerations" section are paramount. + +

On-Device Model Security Considerations

+ +

This section is non-normative.

+ +

Disk Space for Language Packs

+ +Downloading language packs for on-device speech recognition via {{SpeechRecognition/install()}} could use significant amounts of the user's disk space. + +As specified in the storage pressure rules for {{SpeechRecognition/install()}}, in the event of storage pressure, user agents are expected to balance the utility of on-device speech recognition with the disk space its language packs take up—possibly by having {{SpeechRecognition/install()}} resolve to `false` for new downloads or by freeing up disk space in other ways, while remaining mindful of the privacy impacts discussed in Download Eviction and potentially involving the user via download-time prompts or a browser UI for managing downloaded language packs. + +If a previously installed language pack is evicted (e.g., by the user or due to extreme storage pressure) while it might be in use or expected to be available, subsequent attempts to use it (e.g., via {{SpeechRecognition/start()}} with {{SpeechRecognition/lang}} set to that language and {{SpeechRecognition/processLocally}} set to `true`) are expected to fail gracefully. This might involve {{SpeechRecognition/available()}} returning {{"downloadable"}} (or {{"unavailable"}} if no longer supported), and {{SpeechRecognition/start()}} firing a {{SpeechRecognitionErrorEvent}} with an appropriate error code like {{SpeechRecognitionErrorCode/service-not-allowed}} or {{SpeechRecognitionErrorCode/language-not-supported}}. + +

Runtime Shared Resources

+ +On-device speech recognition can consume significant runtime resources like CPU, memory, and potentially specialized hardware accelerators. + +As described in the runtime resource requirements of the [=start session algorithm=], user agents are expected to ensure that one web page's use of on-device speech recognition does not overly interfere with another web page's use of the API, another web page's general operation, or overall system stability. For example, a background tab ought not to be able to monopolize speech processing resources and prevent a foreground tab from using them. + +This specification does not mandate any particular mitigation strategy, but possible approaches include queuing requests, rate limiting, prioritizing foreground tabs, or detecting abusive behavior, and failing speech recognition operations (e.g., by firing a {{SpeechRecognitionErrorEvent}} with {{SpeechRecognitionErrorCode/service-not-allowed}}) when necessary to prevent resource exhaustion or instability. + +

OS-Provided Models

+ +One implementation strategy for on-device speech recognition is to delegate to models or capabilities provided by the underlying operating system. This can offer benefits like a consistent user experience and efficient resource usage. + +However, this approach comes with the usual considerations of exposing OS capabilities to the web. As specified in the requirements for OS-provided models under {{SpeechRecognition/processLocally}}, user agents still need to ensure that all privacy and security requirements of this specification are met when using OS-provided models, even if the user agent has less control over the model's behavior. Particularly notable requirements to watch out for are not training on user speech input (see User Input and Speech Data) and fair and stable resource sharing (see Runtime Shared Resources). + +
+

API Description

This section is normative.

@@ -345,6 +452,15 @@ interface SpeechRecognitionPhrase {
This attribute, when set to true, indicates a requirement that the speech recognition process MUST be performed locally on the user's device. If set to false, the user agent can choose between local and remote processing. The default value is false. +

+ When {{SpeechRecognition/processLocally}} is `false`, user speech data might be sent to a remote server for processing; user agents SHOULD be transparent with users about where speech processing occurs (see Cloud-Based vs. On-Device Implementations). +

+

+ Regardless of the value of {{SpeechRecognition/processLocally}}, implementations MUST NOT train or fine-tune on-device speech recognition models on user speech input obtained through this API, or otherwise store user speech input in a way that models can consult in the future (e.g., for personalization beyond the current session or across origins; see User Input and Speech Data). +

+

+ When delegating on-device speech recognition to models or capabilities provided by the underlying operating system, user agents MUST ensure that all privacy and security requirements of this specification are met, even if the user agent has less control over the model's behavior (see OS-Provided Models). +

phrases attribute
@@ -404,6 +520,16 @@ See [=default allowlist/'self'=]. +

+ To prevent cross-origin fingerprinting and site collusion, user agents MUST partition on-device language pack availability by [=storage key=]. User agents MUST mask the current download status by returning {{"downloadable"}} from {{SpeechRecognition/available()}} by default when {{SpeechRecognitionOptions/processLocally}} is `true`, even if the actual underlying language pack is {{"available"}} or {{"downloading"}} on the user's device (see this step of the [=availability algorithm=] and Download Masking). + This download masking state is established for each (options, [=storage key=]) tuple (specifically, per language in {{SpeechRecognitionOptions/langs}} and {{SpeechRecognitionOptions/quality}} of options for a given [=storage key=], ignoring {{SpeechRecognitionOptions/processLocally}}). The user agent MUST only set the state to "unmasked" for a specific (options, [=storage key=]) tuple after a web page within that key explicitly calls {{SpeechRecognition/install()}} with a given set of options, and that call either successfully initiates (or joins an ongoing) download or resolves to `true` (indicating the language pack is ready; see this step of {{SpeechRecognition/install()}}). + Implementations MUST ensure that this [=storage key=]-based [=download masking state=] is cleared whenever other storage for that [=storage key=] is cleared. +

+ +

+ User agents SHOULD limit the number of possible on-device model versions that a single user agent version can be paired with when determining if a language pack is {{"available"}} via {{SpeechRecognition/available()}} (for example, by tying model updates to user agent updates or ignoring already-downloaded models below a minimum version threshold after a user agent update and reporting {{"downloadable"}} for a newer version instead; see Model Version). +

+ When invoked, run these steps: 1. Let promise be a new promise. 1. Run the availability algorithm with options and promise. @@ -416,6 +542,7 @@ See [=default allowlist/'self'=]. When invoked, run these steps: @@ -424,12 +551,18 @@ See If {{SpeechRecognition/available()}} would resolve to {{"downloadable"}} for options (evaluated as if {{SpeechRecognitionOptions/processLocally}} of options were true), that is, if, as far as the calling web page can observe, this call would initiate a download, then: + 1. If the [=current settings object=]'s [=relevant global object=] does not have [=transient activation=], reject promise with a {{NotAllowedError}}, abort these steps, returning promise. + 1. [=Consume user activation=] given the [=current settings object=]'s [=relevant global object=]. + +

+ Note: Because {{SpeechRecognition/available()}} is subject to [=download masking state=] rules, this check is based on the masked availability status rather than on the true download status. As a result, whether {{SpeechRecognition/install()}} requires [=transient activation=] does not reveal the true download status to a web page for which it is still masked. +

1. [=In parallel=], run the following steps: - 1. For each lang in {{SpeechRecognitionOptions/langs}} of options, initiate the download of the on-device speech recognition language pack for lang matching the requested {{SpeechRecognitionOptions/quality}} level floor. -

- Note: The user agent can prompt the user for explicit permission to download the on-device speech recognition language pack. -

- 1. Wait for the download of all languages specified by {{SpeechRecognitionOptions/langs}} of options to either succeed or fail. + 1. The user agent MAY prompt the user for explicit permission to download the on-device speech recognition language pack(s), or MAY decline to initiate the download based on implicit signals (such as an observed pattern of abuse; see Installation-Time Friction) or storage pressure. In the event of storage pressure, user agents SHOULD balance the utility of on-device speech recognition with the disk space its language packs take up (possibly by declining new downloads or freeing up disk space in other ways, subject to the eviction requirements below; see Disk Space for Language Packs), and MAY involve the user in these decisions (e.g., via the download-time permission prompt or a browser UI for managing downloaded language packs). If the download is not permitted, [=queue a task=] on the [=relevant global object=]'s [=task queue=] to resolve promise with false, and abort these steps. + 1. For each lang in {{SpeechRecognitionOptions/langs}} of options, initiate the download of the on-device speech recognition language pack for lang matching the requested {{SpeechRecognitionOptions/quality}} level floor (if not already installed or downloading). + 1. If the download was successfully initiated (or the language pack is already installed or downloading) for every lang in {{SpeechRecognitionOptions/langs}} of options, set the [=download masking state=] for (options, [=storage key=]) to "unmasked". + 1. Wait for the download of all languages specified by {{SpeechRecognitionOptions/langs}} of options to either succeed or fail. User agents MAY artificially fake a download (e.g., by adding a delay before resolving promise) if they believe it would be helpful for privacy reasons, instead of resolving instantly if the language pack is already present (see Sensitive Language Information). During such a fake download, the user agent MUST report the availability status for that [=storage key=] as {{"downloading"}} (and treat the language pack as not yet installed for other purposes observable by that [=storage key=], including in the [=start session algorithm=] and the {{SpeechRecognitionOptions/processLocally}} `false` branch of the [=availability algorithm=]) so that the web page cannot distinguish a fake delay from a real download. 1. If the download of all those languages succeeded, let result be true, otherwise let result be false. 1. [=Queue a task=] on the [=relevant global object=]'s [=task queue=] to resolve promise with result.

@@ -437,8 +570,16 @@ See + User agents MAY cancel ongoing language pack downloads when the requesting document is destroyed or navigated away from, in accordance with their standard resource management policies, provided that doing so does not change the availability status observable by any other [=storage key=] (e.g., not canceling an underlying download that another [=storage key=] is also waiting on; see Download Cancelation). +

+ +

+ User agents SHOULD NOT implement systems which allow web pages to control the eviction of downloaded language packs, including via indirect triggers such as further subsequent downloads (for example, quota-based or least-recently-used eviction in response to web page-initiated storage pressure; see Download Eviction). Evicting downloads in response to user-controlled actions (e.g., via a browser settings UI) is permitted. +

@@ -482,7 +623,7 @@ See language in langs, let status be {{AvailabilityStatus/available}}. 1. Else, let status be {{AvailabilityStatus/unavailable}}. 1. If {{SpeechRecognitionOptions/processLocally}} of options is `true`:
    @@ -493,10 +634,11 @@ See model version limits) and no fake download is in progress for the caller's [=storage key=] per the `install()` algorithm, set currentLanguageStatus to {{AvailabilityStatus/available}}. +
  1. Else if on-device speech recognition for language matching the requested {{SpeechRecognitionOptions/quality}} of options is currently being downloaded (including a fake download for the caller's [=storage key=] per the `install()` algorithm), set currentLanguageStatus to {{AvailabilityStatus/downloading}}.
    2. +
  2. Else if on-device speech recognition for language matching the requested {{SpeechRecognitionOptions/quality}} of options is supported by the user agent but not yet installed, set currentLanguageStatus to {{AvailabilityStatus/downloadable}}.
  3. Else (on-device speech recognition for language is not supported), set currentLanguageStatus to {{AvailabilityStatus/unavailable}}.
    4. +
  4. If currentLanguageStatus is {{AvailabilityStatus/available}} or {{AvailabilityStatus/downloading}}, and the [=download masking state=] for (options, [=storage key=]) is not "unmasked" for language, set currentLanguageStatus to {{AvailabilityStatus/downloadable}}.
  5. If currentLanguageStatus comes after finalStatus in the ordered list `[{{AvailabilityStatus/available}}, {{AvailabilityStatus/downloading}}, {{AvailabilityStatus/downloadable}}, {{AvailabilityStatus/unavailable}}]`, set finalStatus to currentLanguageStatus.
@@ -521,8 +663,8 @@ following steps: 1. [=Queue a task=] to [=fire an event=] named error at [=this=] using {{SpeechRecognitionErrorEvent}} with its {{SpeechRecognitionErrorEvent/error}} attribute initialized to `phrases-not-supported` and its {{SpeechRecognitionErrorEvent/message}} attribute set to an implementation-defined string detailing the reason. 1. Abort these steps. 1. If this.{{SpeechRecognition/[[processLocally]]}} is `true`: - 1. If the user agent determines that local speech recognition is not available for this.{{SpeechRecognition/lang}}, or if it cannot fulfill the local processing requirement for other reasons: - 1. [=Queue a task=] to [=fire an event=] named error at [=this=] using {{SpeechRecognitionErrorEvent}} with its {{SpeechRecognitionErrorEvent/error}} attribute initialized to {{SpeechRecognitionErrorCode/service-not-allowed}} and its {{SpeechRecognitionErrorEvent/message}} attribute set to an implementation-defined string detailing the reason. + 1. The user agent MUST treat a language pack whose [=download masking state=] is not "unmasked" for the caller's [=storage key=] as not available for local speech recognition. If the user agent determines that local speech recognition is not available for this.{{SpeechRecognition/lang}} (including when its [=download masking state=] is not "unmasked" for the caller's [=storage key=] or when a previously installed language pack was evicted; see the download masking requirements and the eviction failure requirements below), or if it cannot fulfill the local processing requirement for other reasons (such as preventing resource exhaustion): + 1. [=Queue a task=] to [=fire an event=] named error at [=this=] using {{SpeechRecognitionErrorEvent}} with its {{SpeechRecognitionErrorEvent/error}} attribute initialized to {{SpeechRecognitionErrorCode/service-not-allowed}} and its {{SpeechRecognitionErrorEvent/message}} attribute set to an implementation-defined string detailing the reason, which MUST NOT reveal whether a language pack whose [=download masking state=] is not "unmasked" for the caller's [=storage key=] is installed or being downloaded. 1. Abort these steps. 1. Set {{[[started]]}} to `true`. 1. If |requestMicrophonePermission| is `true` and [=request @@ -532,6 +674,13 @@ following steps: 1. Once the system is successfully listening to the recognition, queue a task to [=fire an event=] named start at [=this=]. +

+ If a previously installed language pack is evicted (e.g., by the user or due to extreme storage pressure) while it might be in use or expected to be available, subsequent attempts to use it (e.g., via {{SpeechRecognition/start()}} with {{SpeechRecognition/lang}} set to that language and {{SpeechRecognition/processLocally}} set to `true`) SHOULD fail gracefully (such as {{SpeechRecognition/available()}} returning {{"downloadable"}} (or {{"unavailable"}} if no longer supported), and {{SpeechRecognition/start()}} firing a {{SpeechRecognitionErrorEvent}} with an appropriate error code like {{SpeechRecognitionErrorCode/service-not-allowed}} (or {{SpeechRecognitionErrorCode/language-not-supported}} if no longer supported); see Disk Space for Language Packs). +

+

+ User agents SHOULD ensure that one web page's use of on-device speech recognition does not overly interfere with another web page's use of the API, or another web page's general operation, or the overall system stability (for example, by queuing requests, rate limiting, prioritizing foreground tabs, or detecting abusive behavior; see Runtime Shared Resources). If necessary to prevent resource exhaustion or instability, the user agent MAY cause speech recognition operations to fail (e.g., by firing a {{SpeechRecognitionErrorEvent}} with {{SpeechRecognitionErrorCode/service-not-allowed}}). On-device speech recognition sessions are also subject to the user input retention rules and the OS-provided model rules under {{SpeechRecognition/processLocally}}. +

+

SpeechRecognition Events

The DOM Level 2 Event Model is used for speech recognition events.