Che deployment blocked on OpenShift for IBM Z(s390x) — quay.io/openshift/origin-oauth-proxy has no s390x manifest

[swapnilsingh-ibm]

Describe the bug

When attempting to Deploy Eclipse Che on an OpenShift cluster running on s390x worker nodes, the deployment fails before Che becomes operational.

The che-operator pod itself starts successfully on the s390x node, and most component images (che-dashboard, che-server, configbump, kube-rbac-proxy) correctly carry s390x manifests and pull without issue. The deployment breaks at the che-gateway pod, which runs four sidecar containers. Three start successfully, the fourth — the OAuth proxy authentication sidecar — enters ImagePullBackOff because its image has no s390x manifest.

The image in question is hardcoded in eclipse-che/che-operator at config/manager/manager.yaml:

 - name: RELATED_IMAGE_gateway_authentication_sidecar
   value: quay.io/openshift/origin-oauth-proxy:4.22

The quay.io/openshift/origin-oauth-proxy community images are built for linux/amd64 only. We verified the published tags and they don't carry an s390x manifest. As a result, Che cannot complete deployment on s390x OpenShift regardless of which version of the operator image is used, because this dependency is missing and making Che undeployable on s390x OpenShift.

Che version

next (development version)

Steps to reproduce

1. Provision an OpenShift cluster running on s390x worker nodes
2. Install DevWorkspace Operator
3. Apply deploy/deployment/openshift/combined.yaml --server-side
4. Create a CheCluster CR
5. Observe che-gateway pod — oauth-proxy container enters ImagePullBackOff

Expected behavior

All four gateway containers start successfully and the Che deployment reaches a healthy state on s390x.

Runtime

OpenShift

Screenshots

Installation method

other (please specify in additional context)

Environment

other (please specify in additional context)

Eclipse Che Logs

oc describe pod -n eclipse-che che-gateway-cd77699ff-x9w76 | grep -A3 "Failed\|ErrImage\|image:"
  Warning  Failed          2m29s (x3 over 3m9s)   kubelet            Failed to pull image "quay.io/openshift/origin-oauth-proxy:4.22": unable to pull image or OCI artifact: pull image err: choosing an image from manifest list docker://quay.io/openshift/origin-oauth-proxy:4.22: no image found in manifest list for architecture "s390x", variant "", OS "linux"; artifact err: image reference: choose manifest instance: no image found in manifest list for architecture "s390x", variant "", OS "linux"
  Warning  Failed          2m29s (x3 over 3m9s)   kubelet            Error: ErrImagePull
  Normal   BackOff         110s (x7 over 3m4s)    kubelet            Back-off pulling image "quay.io/openshift/origin-oauth-proxy:4.22"
  Warning  Failed          110s (x7 over 3m4s)    kubelet            Error: ImagePullBackOff

Additional context

Installation method detail:
Direct manifest apply via oc apply --server-side -f deploy/deployment/openshift/combined.yaml from eclipse-che/che-operator

Environment detail:
OpenShift cluster running on IBM Z (s390x) hardware. Worker node architecture confirmed as s390x via oc get node.

Suggested fix — long-term:

Rather than hardcoding a community image reference for the OpenShift oauth-proxy sidecar, the operator could resolve this image dynamically from the OpenShift cluster's own release payload during CheCluster reconciliation: oc adm release info --image-for=oauth-proxy

Every OpenShift cluster carries an architecture-native oauth-proxy image in its release payload. Using this reference ensures the correct image is used on any architecture the cluster runs on — s390x, amd64, arm64, ppc64le — without maintaining per-architecture image references or depending on any external community image. This approach is consistent with how Red Hat's downstream Dev Spaces product handles this dependency.

Why this fix:
The quay.io/openshift/origin-oauth-proxy images are built by OpenShift's community CI pipeline for linux/amd64 only. This source repo (openshift/oauth-proxy) is maintained primarily through automated bot PRs with minimal human review activity is what we saw and is a forked repository too, so we are not sure if it is a practical path to add s390x build support for openshift/oauth-proxy. The fix therefore needs to live in eclipse-che/che-operator as suggested above.

Please guide us on how to proceed.

[tolusha]

Hello, @swapnilsingh-ibm

There is the workaroud:
#23412 (comment)

[swapnilsingh-ibm]

Thank you @tolusha for the quick response.

We looked at the workaround in #23412, but unfortunately it does not apply to s390x for two specific reasons:

1. The workaround patches an OLM Subscription. On s390x, the OLM installation path is not available—quay.io/eclipse/eclipse-che-olm-catalog:next carries only linux/amd64 and linux/arm64 manifests. On a pure s390x cluster, the CatalogSource pod cannot run, so there is no Subscription to patch.

2. The suggested image quay.io/openshift/origin-oauth-proxy:4.9 is also amd64-only. We verified that tag 4.22 (currently hardcoded in che-operator) has no s390x manifest. The full tag series is built by OpenShift's community CI pipeline, which produces amd64-only images.

During further investigation, we found that Red Hat Dev Spaces (the downstream of Eclipse Che) is already running successfully on the same s390x OpenShift cluster with che-gateway 4/4. It uses the following image for oauth-proxy: registry.redhat.io/openshift4/ose-oauth-proxy.

This confirms that a working s390x oauth-proxy solution exists. However, upstream Che cannot use registry.redhat.io images directly, as they require Red Hat subscription authentication, which is not appropriate for an open-source project.

The same cluster's release payload provides an equivalent image that is already accessible to the cluster via its existing pull secret—no additional authentication configuration is needed:

oc adm release info --image-for=oauth-proxy

This returns a digest-pinned, architecture-native image for whatever architecture the cluster runs on — s390x, amd64, arm64 — without any hardcoded image references.

Resolving this image dynamically at CheCluster reconciliation time would be the cleanest upstream fix and would benefit all non-amd64 architectures.

We would be happy to contribute this fix to eclipse-che/che-operator if the maintainers agree on this approach.

[tolusha]

Hello, @swapnilsingh-ibm

Sorry for the late reply.
A contribution would be very welcome. To make the catalog image available for all platforms, you only need to update the list of architectures [1]

Resolving this image dynamically at CheCluster reconciliation time would be the cleanest upstream fix and would benefit all non-amd64 architectures.

sounds good for me

[1] https://github.com/eclipse-che/che-operator/blob/main/build/scripts/olm/release-catalog.sh

[tolusha]

Hello @swapnilsingh-ibm
Do you have any progress or do you need any help with this issue?

[swapnilsingh-ibm]

Hi @tolusha
Due to some urgent deliverables, I had to pause this work. Resumed it now and working on end-to-end testing of the changes, before I submit the PR.

[tolusha]

Thank you for update.
If you need any help, pls let me know