Troubleshooting confidential containers deployments

Check if nodes have finished rebooting after MachineConfig updates:

Wait for all MachineConfigPools to show UPDATED=True and DEGRADED=False.

Verify the KataConfig is ready:

The status should show InProgress: False and the RuntimeClasses should be created (kata-remote for Azure, kata-cc for bare metal).

If the KataConfig is stuck, check the operator logs:

Note: ArgoCD applications are deployed in per-clusterGroup namespaces, not in openshift-gitops. Use oc get applications -A to locate them.

View application health across all namespaces:

For stuck applications, check sync status and errors:

Common dependency order issues:

The pattern defaults to Standard_DCas_v5 VMs, but you can configure other Azure confidential VM families in values-global.yaml by changing the VM size parameters.

Check that your Azure region supports your chosen confidential VM family (default: Standard_DCas_v5):

Visit https://azure.microsoft.com/en-us/explore/global-infrastructure/products-by-region/ and search for your VM family in your target region.

Verify quota for confidential VM sizes in your subscription:

Navigate to Azure Portal > Subscriptions > Usage + quotas and filter for "DC" or "EC" families depending on your chosen VM type. Request a quota increase if needed.

Check sandboxed containers operator logs for Azure API errors:

Verify Azure service principal credentials in Vault:

Ensure values-global.yaml has correct Azure networking values (clusterSubnet, clusterNSG, clusterResGroup).

Check the pod’s initdata annotation:

Pods using initdata ConfigMap have strict policies that deny exec. Pods using debug-initdata allow exec.

The strict policy is a security feature, not a bug. To test CDH functionality interactively, use a pod with coco.io/initdata-configmap: debug-initdata (like the insecure-policy pod in hello-openshift).

Check that initdata ConfigMaps exist and contain the KBS TLS certificate:

If the ConfigMap is missing, check if Kyverno propagated it:

The source ConfigMap should exist in the imperative namespace. If missing, the init-data-gzipper job may have failed:

Check KBS logs for attestation errors:

Look for messages like Attestation verification failed or PCR mismatch.

Check Kyverno pods are healthy:

All Kyverno pods should be Running. If not, check logs:

Verify the pod has the coco.io/initdata-configmap annotation:

If missing, the deployment/pod template must include this annotation. Kyverno only injects cc_init_data if this annotation is present.

Check if the Kyverno policy exists:

Review policy status and events:

View validation policy:

Required fields in initdata ConfigMaps:

Rollout restart the deployment to pick up new initdata:

Verify the new CoCo pods have the updated cc_init_data annotation:

The annotation value should be a long base64-encoded string. If it matches the old value, the ConfigMap may not have been updated. Check the source ConfigMap in the imperative namespace.

For Intel TDX:

Check if TDX is enabled in BIOS. Consult your hardware vendor’s documentation for TEE enablement.

Verify the TDX kernel module is loaded:

Expected output should include kvm_intel with TDX support.

Check NFD worker logs:

For AMD SEV-SNP:

Check if SEV-SNP is enabled in BIOS.

Verify SEV capabilities:

Expected output: Y (enabled)

Check PCCS pod logs:

Look for authentication errors or missing API key messages.

Verify the PCCS secret exists and contains the API key:

The secret should have PCCS_API_KEY field (base64 encoded).

If the secret is missing or incorrect, update ~/values-secret-coco-pattern.yaml with your Intel PCS API key and re-run:

Check node labels:

Nodes with TDX should have intel.feature.node.kubernetes.io/tdx=true.

If labels are missing, NFD may not have detected TDX. See "NFD not detecting TDX" troubleshooting above.

Check QGS DaemonSet status:

If DESIRED is 0, no nodes match the nodeSelector. If DESIRED > 0 but READY is 0, check pod events:

Verify the KataConfig CR exists:

Check the KataConfig status and conditions:

Check sandboxed containers operator logs for errors:

If the RuntimeClass is missing after 10+ minutes, manually trigger reconciliation by adding an annotation:

List pending install plans:

Approve the install plan:

Check if the reconcile-kataconfig-gpu job has run:

Check job logs:

If the job hasn’t run, it may be waiting for GPU nodes to be labeled. Verify GPU Operator labeled the nodes:

Nodes with GPUs should have nvidia.com/gpu.present=true.

Manually trigger KataConfig reconciliation:

Check pod events:

Common issues:

IOMMU not enabled: Check kernel parameters:

Expected: intel_iommu=on (Intel) or amd_iommu=on (AMD)

If missing, verify the MachineConfig applied:

Nodes must reboot for IOMMU kernel parameters to take effect.

GPU not bound to VFIO: Check GPU driver binding:

GPUs should show Kernel driver in use: vfio-pci. If not, check VFIO manager logs:

Check GPU CC Manager logs:

Verify GPU supports CC mode:

Expected output: CC Mode: Enabled

If CC mode is not supported, the GPU firmware may not have confidential computing capabilities. NVIDIA confidential GPUs (H100, H200, B100, B200) with specific firmware versions support CC mode. Consult NVIDIA confidential computing documentation for supported GPU models and firmware requirements.

Check KBS logs for the specific PCR that failed:

Re-extract PCR measurements from the current peer-pod image:

For Azure:

For bare metal: Follow the manual PCR collection procedure for your hardware. See tested environments for guidance.

Update ~/values-secret-coco-pattern.yaml with the new measurements and refresh Vault:

Check if Trustee can reach PCCS:

Expected: JSON response with PCCS version information.

If connection fails, verify PCCS is running:

Check PCCS logs for errors fetching collateral from Intel PCS API:

Verify the Trustee KBS configuration points to the correct PCCS service:

Expected: pccs-service.intel-dcap.svc.cluster.local:8042

Check if SEV-SNP is enabled at the kernel level:

Expected: Y (enabled)

Verify SEV-SNP is enabled in BIOS. Consult AMD SEV developer documentation and your hardware vendor’s BIOS documentation for SEV-SNP enablement procedures.

Check KBS logs for certificate chain verification errors:

AMD SEV-SNP uses a certificate chain-based attestation model, so no external collateral service (like PCCS) is required. The certificate chain is embedded in the attestation evidence.

Wait for all nodes to finish rebooting:

All MachineConfigPools should show UPDATED=True and DEGRADED=False.

Re-trigger secret loading:

Verify Vault is unsealed and healthy:

If Vault is sealed, follow the Vault unsealing procedure documented in the Validated Patterns framework.

Delete the pod to trigger recreation with correct annotations:

The deployment will recreate the pod, and Kyverno will inject the cc_init_data annotation during admission.

Verify the new pod has the annotation:

Before rebuilding a TDX cluster, perform an SGX factory reset in BIOS. The exact procedure varies by hardware vendor. Consult your server vendor’s BIOS documentation or the Intel TDX BIOS setup guide for reset procedures.

Common BIOS settings to check:

For Intel TDX:

Check BIOS settings according to the Intel TDX BIOS setup guide.

Verify TDX is detected by the kernel:

Expected: Messages indicating TDX initialization succeeded.

For AMD SEV-SNP:

Check BIOS settings according to the AMD SEV developer documentation and your hardware vendor’s TEE enablement guide.

Verify SEV-SNP is detected by the kernel:

Expected: Messages indicating SEV-SNP initialization succeeded.

If TEE capabilities are not detected at the kernel level, Node Feature Discovery (NFD) will not label nodes, and confidential runtime classes will not be schedulable. Fix the BIOS configuration before proceeding with pattern deployment.