Integrate media features and experiments with Gutenberg's experimental Media Editor

[jeffpaul]

What problem does this address?

As we continue to add media-related AI features/experiments, we're currently targeting the media experience within the post editor and within the Media Library. We may want to consider also integrating with the experimental Gutenberg media editor as a forward-leaning media editing experience within WordPress.

What is your proposed solution?

If Gutenberg is enabled and the Media Editor experiment is activated, then perhaps we consider interjecting our media-related AI features/experiments into that experience?

[TylerB24890]

@jeffpaul I've been playing around with this and the new Media Editor and have something working for the Alt Text field, however there are currently no filters or Slots/Fills in the new Media Editor, which means we have to go through DataViews with registerEntityField() to get there. This also means we can't hook into the existing field output.

I played around with a version of this where we just added a new "Generate Alt Text" button to the end of the field list, but it felt disconnected not being inline with or right next to the Alt Text input field.

So then I looked into replacing the core alt_text input with our own, wrapped in the AI Experiment. This works (see screenshot below) but I couldn't just replace the alt text field because because registerEntityField runs asynchronously from usePostFields() (which is what the Media Editor uses), and whatever field is registered last wins. Because they run async from each other we don't know which field (core or ours) would be registered first.

To get around this I'm setting a timeout of 1ms on the registerEntityField() method. This seems to be working to override the core field, but I'm not sure how stable it is given the unknown time it might take for usePostFields() to complete. In the event that it takes longer than 1ms to complete, the core field might override our field instead. The other downside is we can't retain the current field ordering, because they are registered independently from each other. Our alt_text field is being added to the end of the current Media fields because we can't hook into the current field rendering.

Here is a screenshot of what I have implemented:

All this said, I think we have a couple paths forward:

1. I can open the PR for what I have for testing and we can proceed with that for now.
2. We can open an issue (or comment on an existing one, open a PR, etc..) in the Gutenberg repository to add some filters to the Media Editor field output. Either allowing us to extend each individual field or replace them.

Option 2 is probably the ideal long-term solution, but I'm happy to push up what I have now for testing or an interim solution if we want. Just let me know how you want to proceed here. (cc @dkotter )

[jeffpaul]

@andrewserong any advice on forthcoming filters/slotfills for the AI plugin to be able to extend the experimental media editor?

[TylerB24890]

Here's the MVP/PoC PR for the Alt Text Generation: #446 -- altered the custom Field registration a little bit to subscribe to the postType state change to attachment. This ensures the re-register of the Field only happens once.

[andrewserong]

@andrewserong any advice on forthcoming filters/slotfills for the AI plugin to be able to extend the experimental media editor?

From my side of things it's a little too early at this stage to start to build in filters and slots as I'm not 100% sure of the current media editor experiment's direction. I'm working with @ramonjd to explore cropping as the next step, where we'll get a feel for how the whole thing is hooked up, and it'd be good to get that in place before adding extension points.

One thing we're going to be looking into is the idea of putting media editing into a modal instead of a separate route/state, as I think that's going to make it easier to work with updating ids for blocks, etc. But I'm planning to hack on this over the next couple of weeks and see how it all might look and feel.

Here's the MVP/PoC PR for the Alt Text Generation: #446 -- altered the custom Field registration a little bit to subscribe to the postType state change to attachment. This ensures the re-register of the Field only happens once.

For now re-registering fields is probably the pragmatic way to go about it 👍

One thing I would mention when interacting with the experiments in Gutenberg is that it's highly likely that we'll be changing things a fair bit and/or removing or updating experiments with little notice, so please don't treat anything behind the experiment as a stable API. What I meant to say is, I wouldn't want to accidentally undo any work y'all are doing with our explorations at this stage.

For the next bit of work on the experimental media editor stuff, I'm planning to use these issues to keep track:

• Media Library: Create a media editor, that can be used from within the post and site editors
• Media Editor task tracking

[jeffpaul]

One thing I would mention when interacting with the experiments in Gutenberg is that it's highly likely that we'll be changing things a fair bit and/or removing or updating experiments with little notice, so please don't treat anything behind the experiment as a stable API. What I meant to say is, I wouldn't want to accidentally undo any work y'all are doing with our explorations at this stage.

Totally, we're keeping any integration we build into the AI plugin as an Experiment and not a Feature for that very reason. Hopefully we can at least help drive some testing to the media experiment in Gutenberg and help with real world user feedback. 🤞🏼

[andrewserong]

Totally, we're keeping any integration we build into the AI plugin as an Experiment and not a Feature for that very reason. Hopefully we can at least help drive some testing to the media experiment in Gutenberg and help with real world user feedback. 🤞🏼

Excellent, sounds good! 🙂