Media: Add UltraHDR (ISO 21496-1) gain map support

[adamsilverstein]

Summary

• Adds UltraHDR/gain map detection and resize support to the media upload workflow
• Uploads original UltraHDR JPEGs unmodified (already backwards-compatible with SDR displays)
• Sub-sized images preserve the ISO 21496-1 gain map so every size retains HDR capability
• All UltraHDR processing flows through the existing @wordpress/vips worker — no extra WASM module is bundled

Closes #74874

Similar to adamsilverstein/client-side-media-experiments#15

Approach

UltraHDR support comes from wasm-vips (libvips compiled to WebAssembly), which gained native UltraHDR (uhdrload / uhdrsave) in 0.0.17. This was unblocked by google/libultrahdr#386, which dual-licensed libultrahdr under Apache-2.0 OR MIT and made it compatible with WordPress's GPLv2-or-later codebase.

@wordpress/vips is bumped to wasm-vips@^0.0.17 and gains:

• a getUltraHdrInfo(buffer) helper that probes a JPEG for an embedded gain map and reports HDR headroom
• an isUltraHdr flag on resizeImage that routes through uhdrloadBuffer + uhdrsaveBuffer so the gain map is downsized in lockstep with the base image and re-embedded on save

@wordpress/upload-media uses these helpers in its existing DetectUltraHdr operation and tags sub-size resizes with isUltraHdr so the resize step transparently preserves the gain map. Thanks to @kleisauke for landing native UltraHDR support in wasm-vips.

Removed

• The standalone open-ultrahdr and open-ultrahdr-wasm dependencies are no longer needed.
• The EncodeUltraHdr queue operation type is gone — there's no separate re-encode step; resize handles it inline.
• The OffscreenCanvas-based ISO 21496-1 gain map reconstruction (encodeUltraHdrItem) is removed.

Follow-up

wasm-vips@0.0.18 will land a docs-only update to THIRD-PARTY-NOTICES.md to make the dual-licensing explicit. We can bump to ^0.0.18 when it's published.

Test plan

• Upload an UltraHDR JPEG image to the media library
• Verify the upload completes successfully and the original is uploaded unmodified
• Verify sub-sized images also retain the HDR gain map (preserved through resize)
• Verify the uploaded image displays correctly on non-HDR displays (SDR fallback)
• Upload a regular JPEG and verify it still works normally

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @kleisauke, @gregbenz.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Unlinked contributors: kleisauke, gregbenz.

Co-authored-by: adamsilverstein <adamsilverstein@git.wordpress.org>
Co-authored-by: andrewserong <andrewserong@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[github-actions]

Flaky tests detected in e847253.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/27102056948
📝 Reported issues:

• [Flaky Test] shouldn't conflict with server-side autosave #78706 in /test/e2e/specs/editor/various/autosave.spec.js

[github-actions]

Size Change: +228 kB (+2.77%)

Total Size: 8.44 MB

📦 View Changed

Filename	Size	Change
build/modules/vips/worker.min.js	4.79 MB	+228 kB (+4.99%)	🔍
build/scripts/upload-media/index.min.js	12.7 kB	+128 B (+1.02%)

_{compressed-size-action}

[andrewserong]

Nice, this is testing well for me! I was going to ask how heavy the upstream library is, and then I noticed that it's from your own repo, so I assume you're quite comfortable with it 😄

The main question I have is to do with the idea that if most uploaded JPEGs are not UltraHDR, how expensive is the operation to detect it? If it's very cheap, then this seems like a great addition. I've left a comment, but before I looked at the code, my assumption is that this kind of operation would flag if it succeeds / does something, but wouldn't warn if it can't find an UltraHDR data.

What do you think?

This is all still behind an experiment of course, so don't let my questions here stop you from progressing! (Also I'll be AFK early next week, so apologies if my reply is delayed until mid-week)

[adamsilverstein]

Nice, this is testing well for me! I was going to ask how heavy the upstream library is, and then I noticed that it's from your own repo, so I assume you're quite comfortable with it 😄

I am comfortable with it working based on the tests, although I also plan to ask for help doing manual testing to verify it properly handles real world images.

The main question I have is to do with the idea that if most uploaded JPEGs are not UltraHDR, how expensive is the operation to detect it? If it's very cheap, then this seems like a great addition. I've left a comment, but before I looked at the code, my assumption is that this kind of operation would flag if it succeeds / does something, but wouldn't warn if it can't find an UltraHDR data.

That is a great question and worth investigating more. I will review the detection logic to see if it can be optimized. Also, I'm still not certain about what we should output. My assumption is that UltraHDR JPEGs are decoder compatible with existing JPEGs so we would want to output UltraHDR for uploaded UltraHDR. Then again, some users may prefer to use the SDR version for their website exclusively because they want smaller file sizes and don't care about HDR. We might need a new way for developers to implement that since the mime type is still image/jpeg and out=r image_editor_output_format filter is based on transforming mime types. I plan to ask photography expert and UltraHDR fan @gregbenz for advice in this regard once I have a completely testable solution.

This is all still behind an experiment of course, so don't let my questions here stop you from progressing! (Also I'll be AFK early next week, so apologies if my reply is delayed until mid-week)

Hopefully for something fun!

[gregbenz]

@adamsilverstein libultrahdr includes a probe() method to validate that required components are present without decoding.
google/libultrahdr#368

I haven't tested it for performance, but it may serve as inspiration.

Gain maps require an auxiliary image. You could check the header and reject if that's missing.

If it exists, you could then further check for a supported gain map (ie a more robust check, but you've already quickly skipped most images that aren't relevant). There are two kinds of map encoding supported by libultra: (a) ISO standard gain maps and (b) gain maps encoded with the Android XMP spec.

The ISO encoding is widely used for new images now and has been for a while in most encoders. Many now dual encode (as the auxiliary image is the same and you're just writing redundant data - binary in the aux image for ISO and standard XMP for the Android spec). The only thing you'd miss by skipping XMP would be older images captured on Android or exported with old versions of Adobe software (Apple has a proprietary method they used pre-ISO and it is not supported by libultra). I believe skipping XMP would generally be ok to do, as it is a very limited edge case (and the impact would be that transcoding falls back to SDR rather than total failure).

[gregbenz]

@adamsilverstein If the source includes a supported gain map, the output should preserve it by default. SharpJS is working towards a keepGainMap() method which should do this, however, I'm not sure how much guidance libvips needs. Coordinating with SharpJS may make sense here, as both efforts are likely chasing the same goals for default transcoding (ie ability to do basic crop/resize/compress while retaining an output which shows high fidelity to the original).

Questions for transcoding will inevitably come up around tuning (compression of the base image / map). Testing to make sure the final result is ok based on the compression applied to both the base and map will be an important step to validate quality vs size objectives. Existing approaches for the base image probably translate well, and this is probably mostly a question of how to compress the map.

Aside from that, most of the options chosen for encoding should remain as they were in the source. The metadata should be unchanged (HDR capacity, offsets, ranges, etc). The map scaling (1:1 vs 1:2 or 1:4) and number of channels (1 for luminosity only map vs 3 for full color) should remain the same. It should not alter sub-sampling (ie keep 444 if that's how it was encoded as the map is not a color image and the assumptions behind sub-sampling are not applicable in this domain, and can cause artifacts if altered).

An affordance for aggressive compression of the image (including loss of HDR) may be useful for some users / workflows, but is not required and could get complicated. There are also other ways beyond stripping the map which can compress the image without stripping HDR, but they involve quality tradeoffs that would need careful assessment. For example, you could downgrade the map from 1:1 resolution in full color to a 1:2 map with luminosity only. That will cause loss of high frequency detail in the HDR rendition, as well as color error (which may be a small or significant issue depending on the relationship of the SDR to the HDR rendition).

Due to the effort, complexity, and risk of confusion here, it may be best that initial support just preserve the gain map and not use more complex options to compress the image further. That can be managed now via the encoding of the file uploaded to WordPress (upload SDR if you only want SDR, phone captures are already low resolution luminosity maps, etc). Advanced transcoding for compression might best be left for 3rd-party plugins to address.

[adamsilverstein]

@adamsilverstein libultrahdr includes a probe() method to validate that required components are present without decoding.

great, thanks for the tip - i will take a look at that.

Gain maps require an auxiliary image. You could check the header and reject if that's missing.

👍🏼

Due to the effort, complexity, and risk of confusion here, it may be best that initial support just preserve the gain map and not use more complex options to compress the image further.

Right, be default we will always try to use the uploaded format for the output (so uploaded UltraHDR should output UltraHDR). My pondering was more about how would enable developers to choose to extract only the SDR image for output/front end if thats what they wanted. The current output mapping won't work, so we may need a more explicit filter to choose HDR or SDR encoding when an UltraHDR is uploaded.

[adamsilverstein]

@adamsilverstein libultrahdr includes a probe() method to validate that required components are present without decoding.

great, thanks for the tip - i will take a look at that.

I created an issue to add a probe mode to lib-open-ultrahdr:
adamsilverstein/lib-open-ultrahdr#5

[adamsilverstein]

@adamsilverstein libultrahdr includes a probe() method to validate that required components are present without decoding.

Thanks again for the tip @gregbenz - I added the probe feature in adamsilverstein/lib-open-ultrahdr#6 based on the libultrahdr approach

[adamsilverstein]

The main question I have is to do with the idea that if most uploaded JPEGs are not UltraHDR, how expensive is the operation to detect it? If it's very cheap, then this seems like a great addition. I've left a comment, but before I looked at the code, my assumption is that this kind of operation would flag if it succeeds / does something, but wouldn't warn if it can't find an UltraHDR data.

Thanks again for the question @andrewserong - it turns out libultrahdr has a probe feature explicitly to determine this without the expensive decoding phase. I added a similar feature to lib-open-ultrahdr in adamsilverstein/lib-open-ultrahdr#6 and will update the PR to leverage that for detection early in the flow.

[gregbenz]

@adamsilverstein See gainmapQuality in gregbenzphotography.com/test/wasm-vips-hdr-demo/wasm-vips-hdr-transcode.js for how I'm controlling it independently of the base quality (I set same as default but my demo page allows independent control).

The "scope:" in the image URL shows up with standard JPGs, I'm assuming this is the nature of the playground environment.

[adamsilverstein]

hmmm. i thought we already returned the filtered wp_editor_set_quality value from the original upload. we may need to implement that, similar to #75784.

with that in place, we can:

• sideload the original gif
• the rest api response will contain the correct image quality to use for the image. in the rest callback the filter is applied with the mime type
• if the client side is going to generate multiple sub sizes, the endpoint runs the filter against each size so it can include the size context (see https://developer.wordpress.org/reference/hooks/wp_editor_set_quality/). the quality setting is returned for each size where it does not match the default.

[adamsilverstein]

hmmm. i thought we already returned the filtered wp_editor_set_quality value from the original upload. we may need to implement that, similar to #75784.

Working on a fix in #78419

[adamsilverstein]

Thanks for the careful reviews, @kleisauke and @andrewserong! Just rebased on trunk (resolving the 2.0.0 batchResizeImage removal). Here's where things stand on the open threads — all of them are addressed in the current revision:

@kleisauke — packages/vips/src/index.ts

1. Drop isUltraHdr routing — confirmed. resizeImage now uses plain vips.Image.newFromBuffer() / vips.Image.thumbnailBuffer() and lets libvips's uhdrload*() priority handle detection. No more isUltraHdr flag on the API. (packages/vips/src/index.ts L417, L432)
2. keep: 'icc|gainmap' — applied in buildSaveOptions, with the comment you suggested. (L341-L361)
3. Crop the gainmap on positional crop — applyResizeAndCrop now scales the crop rect to the gainmap's resolution and re-attaches it via setImage('gainmap', ...) on a copy() of the cropped main image (matching the à-la-carte processing example). (L308-L331)

The only remaining UltraHDR-specific helper in @wordpress/vips is getUltraHdrInfo() — it's still useful in @wordpress/upload-media to (a) record HDR metadata on the attachment (ultrahdr: true, hdr_capacity) and (b) gate format transcoding so we don't accidentally convert UltraHDR JPEGs to AVIF/WebP and strip the gain map.

@andrewserong — packages/upload-media/src/store/

1. CreateUltraHdr usage — gone entirely (grep returns no matches anywhere in packages/). The current code only uses OperationType.DetectUltraHdr, which is queued once in prepareItem and dispatched in the operation loop.
2. detectUltraHdr re-queuing operations — no longer does that. The current thunk just probes the file, marks the item in an ultraHdrItems Set, writes ultrahdr/hdr_capacity to attachment meta, and calls finishOperation. Operation queueing is centralized in prepareItem. (private-actions.ts L813-L852)
3. try/catch scope — narrowed to just the probe (arrayBuffer() + vipsGetUltraHdrInfo()). The dispatch happens outside the try. (L821-L826)
4. Debug warning for non-UltraHDR JPEGs — removed. The catch block is now silent (with a comment) since most JPEGs aren't UltraHDR and that's the expected path.

Re-requesting review when you both have a moment 🙏

[kleisauke]

I'm not a WordPress expert, but the wasm-vips part and its usage w.r.t. UltraHDR processing looks good to me.

[adamsilverstein]

@gregbenz this is ready for additional testing

[andrewserong]

Thanks for the updates @adamsilverstein! Just double-checking but is the intention that generated sub-size images retain their gain maps? In manual testing (with one Greg's lovely photos: https://gregbenzphotography.com/hdr-gain-map-gallery/) it appears that the uploaded file has the gain map, but if I go to view the generated subsize jpegs on my phone (the only HDR display I have unfortunately!), they appear to be rendering in SDR and the highlights are no longer super bright.

Is it working for you? I tried running npm install again to keep my environment up to date and rebuilt my local env to double check, but it's possible I might have missed something.

(Code-wise this is looking nice, good work simplifying things!)

[adamsilverstein]

Thanks for the updates @adamsilverstein! Just double-checking but is the intention that generated sub-size images retain their gain maps? In manual testing (with one Greg's lovely photos: https://gregbenzphotography.com/hdr-gain-map-gallery/) it appears that the uploaded file has the gain map, but if I go to view the generated subsize jpegs on my phone (the only HDR display I have unfortunately!), they appear to be rendering in SDR and the highlights are no longer super bright.

Thanks for testing! Yes - the intention is for output files to also include a gain map. I haven't had a chance to manually test that though so appreciate you doing so. I'll dig in here to see if I can figure out why its failing and ideally add some tests that validates gain maps are present in sub-sized images.

[adamsilverstein]

Sub-size gain map output: investigated and confirmed ✅

The headline question here was whether generated sub-sizes actually carry the ISO 21496-1 gain map. I traced the full pipeline and confirmed they do, for every size:

1. The original UltraHDR JPEG uploads unmodified, so the full-size image keeps its gain map.
2. detectUltraHdr flags the parent item in a module-level ultraHdrItems set.
3. generateThumbnails routes every sub-size and the -scaled copy through ResizeCrop and deliberately skips format transcoding for tracked items — transcoding to AVIF/WebP would strip the gain map.
4. resizeImage preserves the gain map across all three resize paths:
   • Simple downscale and boolean (centre) crop → libvips thumbnail updates the gain map automatically. (Per the libvips docs: "Two high-level libvips operations will automatically update the gainmap for you during processing: vips_dzsave() and vips_thumbnail().")
   • Positional crop → the gain map is cropped manually with the hscale/vscale scaling the docs prescribe, since the gain map may be lower-resolution than the base image.
   • Save uses keep: 'icc|gainmap', so jpegsave delegates to uhdrsave.

Tests added to confirm it

• upload-media unit (generateThumbnails): proves every sub-size and the -scaled copy is dispatched through resize without a transcode op for UltraHDR sources, and that a non-tracked item transcodes normally (so the guard is demonstrably what preserves the gain map).
• vips unit (resizeImage): adds coverage for the simple-downscale and boolean-crop paths preserving the gain map without manual cropping, alongside the existing positional-crop test.
• e2e: the existing test now asserts both the hard-cropped thumbnail and the proportional medium sub-size carry the gain map and match their registered dimensions — covering both the manual-crop and auto-resize paths end to end with real bytes.

Other feedback

@kleisauke — all three of your points are in:

• newFromBuffer/thumbnailBuffer are used directly now (relying on uhdrload's higher priority) instead of explicit uhdr* calls — thanks for the gist 🙏
• keep: 'icc|gainmap' on save.
• Positional crop now also crops the attached gain map.

@andrewserong:

• Debug warning on non-UltraHDR JPEGs: removed (detection is silent now).
• CreateUltraHdr/EncodeUltraHdr: removed (no longer any separate encode step).
• detectUltraHdr no longer replaces the operations list — it just tags the item; prepareItem owns the operation queue.
• The try/catch in detectUltraHdr now wraps only the probe.
• The set growing on cancel: ultraHdrItems cleanup moved to removeItem, which runs on both completion and cancellation.
• Changelog: dropped the note about the never-landed open-ultrahdr deps.
• check-system-requirements.js: reverted to trunk.
• Webpack changes: already removed when the approach switched to native wasm-vips — the remaining diff is just trunk being slightly ahead.

[andrewserong]

Thanks for the update @adamsilverstein. How did you go with the manual testing?

The workflow I've been using has been to download one of @gregbenz' photos from https://gregbenzphotography.com/hdr-gain-map-gallery/ then upload it to my test site, and then download the generated JPEGs onto my phone (the only HDR display I have). The original image displays ultra bright areas as expected, but the generated sub-sizes don't. (I retested and the generated ones still appear to be rendering in SDR).

I'm slightly wary of trying to review this PR or approve it without being able to verify that it works on one of my devices. So I might need to defer to you and @gregbenz on this one!

Edit: one other question: are there some testing steps I can use to inspect the generated JPEG that it really does include the gain map data?

[gregbenz]

@andrewserong The free Adobe gain map demo app will tell you (in text on the right column) if the image is still a valid gain map. You can hover over the image to read decoded RGB pixel values, which should be consistent with the source (at least not near edges where resampling will affect results). And you can click the radio button to see the gain map itself. The demo app has a quirk where opening similar images at launch may not show all, you can right click the filmstrip at the bottom and add there via file picker to work around that.

You can test with this image, which is built to catch a range of transcoding / quality issues. Feel free to share a result if you need 3rd party review.
ISO JPG - P3 - transcoding test - Greg Benz.jpg.zip

Chromium browsers on a MacBook Pro are an ideal way to review. Set brightness to more than 50% and you'll have the full 4 stops of headroom.

Viewing on a phone is helpful, but Apple decodes in a non-standard way and Android is limited to 2.3 stops (even though the hardware could do much more). So it's great for consumption and basic checks, but you won't see the full HDR rendering. Any failure of quality is likely still visible for this image on both iOS and Android, so the full and proper 4 stops decoding is just nice to have - but not required for visual testing.

[andrewserong]

Wonderful, thanks so much for the testing instructions @gregbenz and that test image! I currently use a Macbook Air and retired my old Macbook Pro as I wanted something lighter for travel. Sadly this means I am currently without a proper HDR screen of any kind other than my iPhone 😆

But that testing app and your testing image worked perfectly, and I can now confirm in my local env that the sub-sizes contain the gain maps and they're resized as expected:

And thanks again for all the back and forth @adamsilverstein, this LGTM now. I left a couple of nit comments, but nothing blocking 🙂