Should we add slide presentation specific actions? about mediasession

[youennf]

• Raise an issue to follow up relaxing the Media Session processing rules

I filed #285 to cover this.

[beaufortfrancois]

Thanks for raising this question @youennf

FYI Chrome is going to experiment with Document Picture-in-Picture to allow websites to open an always-on-top window that can be populated with arbitrary HTMLElements (not just HTMLVideoElements).

See explainer and Intent to prototype

[youennf]

nextslide/previousslide

capture handle actions also mention first and last.

See explainer and Intent to prototype

Thanks for the links.
The explainer does not have a lot of use cases, is slide show applications in scope of Document PiP use cases?

[beaufortfrancois]

@steimelchrome will know more about use cases for Document PiP

[chrisn]

From discussion in the 23 May 2022 Media WG / WebRTC joint meeting, could the use cases for Document PiP motivate adding next slide / previous slide actions to MediaSession? Or are there other developer requests for such actions in MediaSession that mean we should consider adding them?

[liberato-at-chromium]

steimelchrome@ and I had a discussion after yesterday's media wg meeting about how we would have built MediaSession if these use cases had been known in advance.

We concluded that we might well have created a registry of actions that have well-defined semantics on the browser context, e.g., an ActionRegistry gotten from navigator.getActionRegistry() that supports actions very much like MediaSession.setActionHandler() does in today's spec. There would be a list of action names (like MediaSessionAction in the current world), but these could include actions that are not specific to media. The only requirement would be that the semantics of the action would be part of the spec, just like play is now for MediaSession. The UA would be free to trigger those actions.

A site would register action callbacks like 'play' or 'pause' in this registry instead of via MediaSession, to indicate the intent to respond to those actions. MediaSession would look through those actions to find ones that are relevant to media. In other words, MediaSession would stop being both a registrar of and trigger for these actions, to simply being a trigger for them in response to hw keys, pip, etc.

That's the world we might have built. However, we can still move towards that today, just having MediaSession.setActionHandler() register on ActionRegistry behind the scenes. If we do, then it might address some of the capture use-cases. (Of course, the ActionRegistry for the captured page would replace the need for setSupportedCaptureActions.)

If we do adopt that model, then it might help out capture actions. Specifically:

• "Registry of UA-understood Action" is now disjoint from "things that trigger actions" -- mediasession triggers them today, capturers triggers others tomorrow, who-knows-what the day after.
• Whether UA chooses to expose any new actions via existing mediasession-style paths (e.g., pip or hardware keys etc.) is now orthogonal to this discussion. The notion of 'Active media session' is also; it's an internal detail of routing media keys to ActionRegistry actions. Maybe the UA would want to put a "next slide" button somewhere, who knows.
• Backwards compatibility -- capturers can "see" mediasession actions on existing sites for free, assuming that there are not additional security-related restrictions that we wish to impose. This was a concern that came up in the wg meeting, if I remember.
• Adding new types actions has a lower bar; they don't need to strictly fit into what mediasession chooses to do with them, since mediasession can simply ignore new actions. They simply have to be well-defined in the sense that it's clear what triggering the action means, so that the site knows what it's signing up for.

Sorry for the long post, but stemelchrome@ and I wanted to see if this is a direction that might be helpful for the capture use case.

[jan-ivar]

Yesterday I learned Chrome is sending some PiP actions to the right place at its own discretion. This makes sense, and suggests perhaps a more lax model than I got just from reading the spec. Actions can have different routing and different defaults.

In such a model, it seems OK to add "nextslide" and "previousslide" actions by simply specifying specific rules that apply only to them. A concrete proposal:

navigator.mediaSession.setActionHandler("previousslide", () => updateMyDoc(--mySlideNumber));
navigator.mediaSession.setActionHandler("nextslide", () => updateMyDoc(++mySlideNumber));

We specify that sources for these two actions MAY include:

1. browser buttons (Firefox ships a PiP feature without any JS API, so this may be of interest to Mozilla)
2. hardware buttons (UAs could decide to route e.g. a ⏭️ keyboard button to nextslide if nexttrack is unavailable)
3. other capturing sites through the transient-activated sendCaptureAction API which is behind user permission.

I suggest we punt on "firstslide" and "lastslide" which are derivative. Thoughts?

[liberato-at-chromium]

I think we're saying very similar things. The only bit i'd add is that those new actions don't have much to do with MediaSession, so it might be a good time to move the registry of actions somewhere else.

[jan-ivar]

those new actions don't have much to do with MediaSession, so it might be a good time to move the registry

I think they're related. The WebRTC WG has been asking the Media WG to get more involved with media capture.

togglemicrophone, togglecamera and hangup already expand "media session" to include media capture sessions effectively, which seems defensible to me in today's world of media interaction, even if it's not a perfect weld. Screen capture is a media capture session with a "nextslide" advancement action rooted in old slide projector devices.

The more I think about it, I don't think we should solve "actions that are not specific to media". That's venturing into semantic web territory, and seems out of scope both for this WG and WebRTC.

I don't think we need discovery, and we have #228 on feature detection. I'd rather not have a low bar, and think success in this space requires pushing back on some of the grander ideas.

[liberato-at-chromium]

I don't think we should solve "actions that are not specific to media".

We're looking at things of the form "page can do X" and "a capturer of that page might reasonably want to tell the captured page to do X". That, to me, doesn't make X more related (or less) to media. "next slide" or "next page", for example, fall into this. It's problematic since the captured page would register these actions just in case it happens to be captured, which it likely won't be. Granted, the UA could be pretty helpful too with hardware media key bindings and similar. So good things in there too.

The existing toggle* and hangup actions already stretch it slightly in my opinion. They're controlling audio / video directly, though, so I guess it's as good as anywhere to have them.

That said, I prefer having it all under mediasession than having a separate, mostly parallel, system to register "actions that could be triggered by capture". Keeping action registration all in one place seems much better to me, even if we all don't quite yet share the same view of how "media" those actions are.

There are some changes to the MediaSession API that could go along with that, to keep things neater. Haven't thought too much about this yet.

Please note that, unlike my comment above, this post isn't a summary of a discussion with steimelchrome@; I'm speaking just for myself now.

[chrisn]

@youennf, @jan-ivar, @eric-carlson, @steimelchrome, @liberato-at-chromium and I met today to continue this discussion.

We looked at two approaches: (1) adding next/previous slide actions to Media Session, and (2) introducing a new, more general purpose, action registry API that could be used by both Media Session and Capture Handle Actions.

Although option (2) is a more generic and flexible design, there was concern expressed about designing for the future, whereas option (1) would be enough to allow for the current use cases. We also considered that the next/previous slide actions are media-related enough to be included in Media Session.

We concluded that we should:

• Add next slide and previous slide actions to Media Session, with processing rules that allow them to be invoked from Capture Handle Actions. These actions could also be invoked from UA provided buttons, e.g., in a Picture in Picture window
• Specify Capture Handle Actions to reference Media Session, invoking the algorithms defined in Media Session. Media Session is otherwise independent of Capture Handle Actions
• Add dedicated next slide and previous slide actions, preferable to overloading MediaSession nexttrack and previoustrack actions
• Consider of the security and privacy model around the next slide and previous slide actions as part of Capture Handle Actions
• Raise an issue to follow up relaxing the Media Session processing rules

[beaufortfrancois]

• Add next slide and previous slide actions to Media Session, with processing rules that allow them to be invoked from Capture Handle Actions. These actions could also be invoked from UA provided buttons, e.g., in a Picture in Picture window

For the record, Chrome shipped Capture Handle Identity in Chrome 102, not Capture Handle Actions. Capture Handle Actions have not shipped yet in any browser.

[beaufortfrancois]

FYI I've just sent #284 to start adding presenting slides actions to the spec.

[youennf]

Closing as fixed by #284