Validated Patterns

Overview of secrets management

Secrets management is a critical aspect of software development, especially when dealing with sensitive information such as passwords, API keys, and tokens. In the context of GitOps, where the state of an application is stored in a Git repository, managing secrets requires careful handling to avoid exposing sensitive information. Managing secrets in values files involves securely handling this sensitive information within configuration files, ensuring that these secrets are safely stored and managed, thereby reducing the risk of exposure or unauthorized access.

Recommended Practices

  • Avoid committing secrets to Git repositories: Never store secrets directly in Git repositories, including private ones, as they can be accidentally exposed or accessed by unauthorized users. If secrets are committed, they should be considered compromised and rotated immediately.

  • Use secret references: Instead of storing actual secret values in configuration files, store references to those secrets. This method is supported by major secret management solutions like HashiCorp Vault, AWS Secrets Manager, and Azure Key Vault. Applications can fetch the secrets securely at runtime using these references.

  • Encrypt secrets at rest: Always encrypt secrets wherever they are stored. This includes encrypting files on local machines, using encrypted storage solutions, and leveraging built-in encryption mechanisms provided by secret management tools. Ansible, for example, supports encryption of files at rest with tools like ansible-vault.

  • Use external secret management tools: Use external tools like the External Secrets Operator (ESO) to manage secrets outside of the application’s core configuration. This reduces the risk of accidental exposure and ensures a more secure secret management process.

Basic secrets configuration in a ClusterGroup

The secrets required for a pattern are not directly represented in the ClusterGroup configuration. Instead, the values-secret.yaml.template file located in the root of validated patterns specifies the necessary secrets, detailing the required data types and keys for each pattern.

In this template file, the secrets key contains an array of secrets. Each secret includes an array of fields, where each field has a name and value that describe the structure of the secret as used by the pattern.

The pattern framework uses the configured SecretStore backend to inject the secrets into the configured backend. Additionally, values can be retrieved in other ways than specifying a string value. This is convenient when the secret material, such as SSH private and/or public keys, already exists as a file elsewhere on the machine from which the pattern is being installed. The framework also supports encoding the field using base64, which is required for binary data files, such as Red Hat Satellite-style manifests.

Secret file processing in the framework

The secrets loader in the framework, triggered by make load-secrets (also run as part of make install), searches for files in the following order and uses the first one found:

  • ~/.config/hybrid-cloud-patterns/values-secret-{{ pattern_name }}.yaml

  • ~/.config/validated-patterns/values-secret-{{ pattern_name }}.yaml

  • ~/values-secret-{{ pattern_name }}.yaml

  • ~/values-secret.yaml

  • {{ pattern_dir }}/values-secret.yaml.template

Here, “pattern_name” refers to the pattern name (for example, “ansible-edge-gitops”, “industrial-edge”), and “pattern_dir” is the root directory of the locally checked-out pattern. This scheme allows for separate secret files for different patterns or a single file for all patterns. If the secrets file is encrypted, the load-secrets process prompts for the password. When entered, the process will continue and the secrets will be loaded.

Values secret file structure

The values secret file comprises top-level keys and sub-parameters, facilitating the management of secrets within the backend.

Top-Level keys:

  • version: (Optional): Must be “2.0” if specified. Defaults to a deprecated 1.0 schema if not specified.

  • backingStore: (Optional): Defaults to the backingStore configured in values-global. If specified, it must match values-global. It is best not to include it.

  • vaultPolicies: (Optional): Allows injecting vault policies for automatic password generation; effective only when the secrets backend is vault. It’s a dictionary, allowing multiple policies.

  • secrets: The actual secrets to inject into the backend, with each secret having various sub-fields as described below.

Sub-parameters

  • name: Indicates the name of the secret within the backend.

  • fields: Defines the fields within the secret object. Each field has a name and a value. In the template file, the value is used as a placeholder to define the use of the field. This is used by the framework in error messages related to the field.

  • targetNamespaces: (Optional, effective with “Kubernetes” or “none” backends): Secret objects are created in all specified namespaces. Multiple namespaces are allowed as the same secret material may need to be injected into multiple secret objects in different namespaces.

  • labels: (Optional, effective with “Kubernetes” and “none” backends): Kubernetes labels to add to secret objects created through the framework.

  • annotations: (Optional, effective with “Kubernetes” and “none” backends): Kubernetes annotations to add to secret objects created through the framework.

  • type: (Optional, effective with “Kubernetes” and “none” backends): Kubernetes “type” of secret object. Defaults to “Opaque” if unspecified.

  • vaultPrefixes: (Optional, only effective with vault backend): Keys in vault belong to prefixes, allowing creation of the same key with multiple prefixes.

Fields sub-parameters

  • name: Each field in a secret must have a unique, descriptive name (for example, username, password, manifest_content).

  • value: (Optional, technically): Actual secret material. Can be provided via equivalent convenience functions. Value designates content directly, but the framework can also use file contents or parse an ini-file for data. In the template yaml, we always specify value with a description of the name/value pair.

  • path: (Optional): Local file to use as secret content. Framework understands ~ (user home directory) expansion under Linux, Mac, and WSL for convenience.

  • ini_file: (Optional) : Ini-file to retrieve secret material from. Must also include ini_key.

  • ini_key: (Optional): Ini_key within the ini file to retrieve secret material from.

  • ini_section: (Optional): Ini_section within an ini_file to retrieve the key from. Needed only for ini files with multiple sections.

  • base64: (Optional): Boolean value (true/false, default is false) to base64-encode secret material before sending it to the secrets backend. This is independent of the other options, so you can specify plaintext file path and not specify base64 for encoding, or specify a binary file along with base64: true for encoding.

  • onMissingValue: (Optional): (error/prompt/generate) Specifies behavior when the values secret file does not contain a value for the field. By default it throws an error. The “prompt” option asks user to input a secret on the keyboard. The “generate” option uses vault’s secret generation feature to create a suitable secret.

  • override: (Optional, only effective with vault backend): When onMissingValue is set to “generate” and a secret has already been generated, ensures a new one is generated each time the secrets loader is run.

  • vaultPolicy: (Optional; only effective with vault backend): Defines the vault policy (which defines allowable character types and overall length) to use when generating a secret.

Examples:

Here are examples demonstrating the template file from AnsibleEdge GitOps and reasonable ways to provide the secret values using the convenience mechanisms provided by the framework. Alternatively, values can be directly filled in using the value field with the literal contents of each secret.

Template File:

version: "2.0"
secrets:
  - name: aws-creds
    fields:
    - name: aws_access_key_id
      value: "An aws access key that can provision VMs and manage IAM (if using portworx)"

    - name: aws_secret_access_key
      value: "An aws access secret key that can provision VMs and manage IAM (if using portworx)"

  - name: kiosk-ssh
    fields:
    - name: username
      value: 'Username of user to attach privatekey and publickey to - cloud-user is a typical value'

    - name: privatekey
      value: 'Private ssh key of the user who will be able to elevate to root to provision kiosks'

    - name: publickey
      value: 'Public ssh key of the user who will be able to elevate to root to provision kiosks'

  - name: rhsm
    fields:
    - name: username
      value: 'username of user to register RHEL VMs'
    - name: password
      value: 'password of rhsm user in plaintext'

  - name: kiosk-extra
    fields:
    # Default: '--privileged -e GATEWAY_ADMIN_PASSWORD=redhat'
    - name: container_extra_params
      value: "Optional extra params to pass to kiosk ignition container, including admin password"

  - name: cloud-init
    fields:
    - name: userData
      value: |-
        #cloud-config
        user: 'username of user for console, probably cloud-user'
        password: 'a suitable password to use on the console'
        chpasswd: { expire: False }

  - name: aap-manifest
    fields:
    - name: b64content
      path: 'full pathname of file containing Satellite Manifest for entitling Ansible Automation Platform'
      base64: true

Providing values in a “real” values-secret.yaml:

version: "2.0"
secrets:
  - name: aws-creds
    fields:
    - name: aws_access_key_id
      ini_file: ~/.aws/credentials
	ini_key: aws_access_key_id

    - name: aws_secret_access_key
      ini_file: ~/.aws/credentials
	ini_key: aws_secret_access_key

  - name: kiosk-ssh
    fields:
    - name: username
      value: 'cloud-user'

    - name: privatekey
      path: ~/.ssh/id_cloud_user

    - name: publickey
      Path: ~/.ssh/id_cloud_user.pub

  - name: rhsm
    fields:
    - name: username
      value: 'rhsm-user'
    - name: password
      value: 'rhsm-password'

  - name: kiosk-extra
    fields:
    # Default: '--privileged -e GATEWAY_ADMIN_PASSWORD=redhat'
    - name: container_extra_params
      value: "--privileged -e GATEWAY_ADMIN_PASSWORD=B3tt3RP8ssw@rd"

  - name: cloud-init
    fields:
    - name: userData
      value: |-
        #cloud-config
        user: 'cloud-user'
        password: 'aaaa-bbbbb-ccccc-ddddd'
        chpasswd: { expire: False }

  - name: aap-manifest
    fields:
    - name: b64content
      path: '~/manifest_Pattern_20240605T170000Z.zip'
      base64: true

Secret store configuration in a ClusterGroup

The secretStore parameter defines the configuration for connecting to the secret management backend. This parameter allows the ClusterGroup to interact with the chosen secret storage solution. It is normally configured in the values-global.yaml configuration file. The framework provides tools for modifying the secret store configuration.

Many historical patterns have a separate secretStore key. However, the new recommendation is to use the global.secretStore key instead. In general using the tools provided to configure a pattern secrets backend is recommended, as there are implications for both the hashicorp-vault and golang-external-secrets charts, as well as the clustergroup namespace lists.

Sub-parameters

  • backend: Specifies the secret store backend, currently supporting “vault”, “Kubernetes”, or “none”. The default is “vault.”

  • kind: This may be “SecretStore” or “ClusterSecretStore”. In most cases, patterns opt for ClusterSecretStore as it enables access from multiple namespaces. “ClusterSecretStore” is the default.

Examples:

In values-global.yaml:

global:
  secretStore:
    backend: kubernetes

Secrets backends in the Validated Patterns framework

The validated patterns framework currently supports three secrets storage schemes:

  • secrets-backend-vault: This is the default scheme where the community edition of HashiCorp Vault and the External Secrets Operator are installed. Secrets are stored in vault and then projected into the pattern via External Secrets objects. This approach ensures versionable secrets without exposing them in a Git repository.

  • secrets-backend-Kubernetes: The Kubernetes secrets backend also uses the community External Secrets Operator, but rather than injecting secrets into Vault (which is not installed when this backend is selected), secrets are injected into a separate Kubernetes namespace. ESO still brokers the projection of these secrets into the pattern.

  • secrets-backend-none: In this scheme, secret objects are directly injected into the pattern, based on the values-secret configuration. Neither Vault nor the External Secrets Operator are installed. This method is provided for users who require full Red Hat Supportability for their pattern installations.

Switching between secrets backends

The framework provides Makefile-based mechanisms for changing secrets storage schemes. It is recommended to decide on a secrets storage mechanism before installing the pattern. To configure your pattern with a different secrets storage scheme, run the appropriate make target. This is only necessary if you want to switch from the vault scheme, as all patterns come configured with the vault backend by default. The Makefile targets modifies the values file in your pattern repository, which you must then commit and push to take effect. The changes made by the Makefile targets will be displayed in "diff" format.

Example:

Setting backend to Kubernetes for the Ansible Edge GitOps pattern:

% make secrets-backend-kubernetes
make -f common/Makefile secrets-backend-kubernetes
make[1]: Entering directory '/home/martjack/gitwork/edge/ansible-edge-gitops'
common/scripts/set-secret-backend.sh kubernetes
common/scripts/manage-secret-namespace.sh validated-patterns-secrets present
Namespace validated-patterns-secrets not found, adding
common/scripts/manage-secret-app.sh vault absent
Removing namespace vault
Removing application wth chart location common/hashicorp-vault
common/scripts/manage-secret-app.sh golang-external-secrets present
diff --git a/values-global.yaml b/values-global.yaml
index e594a84..adbd0fa 100644
--- a/values-global.yaml
+++ b/values-global.yaml
@@ -1,15 +1,14 @@
 ---
 global:
   pattern: ansible-edge-gitops
-
   options:
     useCSV: false
     syncPolicy: Automatic
     installPlanApproval: Automatic
-
   hub:
     provider: aws
     storageClassName: gp2
-
+  secretStore:
+    backend: kubernetes
 main:
   clusterGroupName: hub
diff --git a/values-hub.yaml b/values-hub.yaml
index ccf662d..463f24b 100644
--- a/values-hub.yaml
+++ b/values-hub.yaml
@@ -2,31 +2,25 @@
 clusterGroup:
   name: hub
   isHubCluster: true
-
   namespaces:
-    - vault
     - golang-external-secrets
     - ansible-automation-platform
     - openshift-cnv
     - openshift-storage
     - edge-gitops-vms
-
+    - validated-patterns-secrets
   subscriptions:
     aap-operator:
       name: ansible-automation-platform-operator
       namespace: ansible-automation-platform
-
     openshift-virtualization:
       name: kubevirt-hyperconverged
       namespace: openshift-cnv
-
     openshift-data-foundation:
       name: odf-operator
       namespace: openshift-storage
-
   projects:
     - hub
-
   imperative:
     jobs:
       - name: deploy-kubevirt-worker
@@ -52,36 +46,25 @@ clusterGroup:
           - get
           - list
           - watch
-
   applications:
     aap:
       name: ansible-automation-platform
       project: hub
       path: charts/hub/ansible-automation-platform
-
     aap-config:
       name: aap-config
       project: hub
       path: charts/hub/aap-config
-
-    vault:
-      name: vault
-      namespace: vault
-      project: hub
-      path: common/hashicorp-vault
-
     golang-external-secrets:
       name: golang-external-secrets
       namespace: golang-external-secrets
       project: hub
       path: common/golang-external-secrets
-
     openshit-cnv:
       name: openshift-cnv
       namespace: openshift-cnv
       project: hub
       path: charts/hub/cnv
-
     odf:
       name: odf
       namespace: openshift-storage
@@ -89,12 +72,10 @@ clusterGroup:
       path: charts/hub/openshift-data-foundations
       extraValueFiles:
         - '/overrides/values-odf-{{ $.Values.global.clusterPlatform }}-{{ $.Values.global.clusterVersion }}.yaml'
-
     edge-gitops-vms:
       name: edge-gitops-vms
       namespace: edge-gitops-vms
       project: hub
       path: charts/hub/edge-gitops-vms
-
   # Only the hub cluster here - managed entities are edge nodes
   managedClusterGroups: []
Secrets backend set to kubernetes, please review changes, commit, and push to activate in the pattern
make[1]: Leaving directory '/home/martjack/gitwork/edge/ansible-edge-gitops'

Setting backend to none in Ansible Edge GitOps:

% make secrets-backend-none
make -f common/Makefile secrets-backend-none
make[1]: Entering directory '/home/martjack/gitwork/edge/ansible-edge-gitops'
common/scripts/set-secret-backend.sh none
common/scripts/manage-secret-app.sh vault absent
Removing namespace vault
Removing application wth chart location common/hashicorp-vault
common/scripts/manage-secret-app.sh golang-external-secrets absent
Removing namespace golang-external-secrets
Removing application wth chart location common/golang-external-secrets
common/scripts/manage-secret-namespace.sh validated-patterns-secrets absent
Removing namespace validated-patterns-secrets
diff --git a/values-global.yaml b/values-global.yaml
index e594a84..62d43c8 100644
--- a/values-global.yaml
+++ b/values-global.yaml
@@ -1,15 +1,14 @@
 ---
 global:
   pattern: ansible-edge-gitops
-
   options:
     useCSV: false
     syncPolicy: Automatic
     installPlanApproval: Automatic
-
   hub:
     provider: aws
     storageClassName: gp2
-
+  secretStore:
+    backend: none
 main:
   clusterGroupName: hub
diff --git a/values-hub.yaml b/values-hub.yaml
index ccf662d..b9e168e 100644
--- a/values-hub.yaml
+++ b/values-hub.yaml
@@ -2,31 +2,23 @@
 clusterGroup:
   name: hub
   isHubCluster: true
-
   namespaces:
-    - vault
-    - golang-external-secrets
     - ansible-automation-platform
     - openshift-cnv
     - openshift-storage
     - edge-gitops-vms
-
   subscriptions:
     aap-operator:
       name: ansible-automation-platform-operator
       namespace: ansible-automation-platform
-
     openshift-virtualization:
       name: kubevirt-hyperconverged
       namespace: openshift-cnv
-
     openshift-data-foundation:
       name: odf-operator
       namespace: openshift-storage
-
   projects:
     - hub
-
   imperative:
     jobs:
       - name: deploy-kubevirt-worker
@@ -52,36 +44,20 @@ clusterGroup:
           - get
           - list
           - watch
-
   applications:
     aap:
       name: ansible-automation-platform
       project: hub
       path: charts/hub/ansible-automation-platform
-
     aap-config:
       name: aap-config
       project: hub
       path: charts/hub/aap-config
-
-    vault:
-      name: vault
-      namespace: vault
-      project: hub
-      path: common/hashicorp-vault
-
-    golang-external-secrets:
-      name: golang-external-secrets
-      namespace: golang-external-secrets
-      project: hub
-      path: common/golang-external-secrets
-
     openshit-cnv:
       name: openshift-cnv
       namespace: openshift-cnv
       project: hub
       path: charts/hub/cnv
-
     odf:
       name: odf
       namespace: openshift-storage
@@ -89,12 +65,10 @@ clusterGroup:
       path: charts/hub/openshift-data-foundations
       extraValueFiles:
         - '/overrides/values-odf-{{ $.Values.global.clusterPlatform }}-{{ $.Values.global.clusterVersion }}.yaml'
-
     edge-gitops-vms:
       name: edge-gitops-vms
       namespace: edge-gitops-vms
       project: hub
       path: charts/hub/edge-gitops-vms
-
   # Only the hub cluster here - managed entities are edge nodes
   managedClusterGroups: []
Secrets backend set to none, please review changes, commit, and push to activate in the pattern
make[1]: Leaving directory '/home/martjack/gitwork/edge/ansible-edge-gitops'

Letsencrypt secrets configuration in a ClusterGroup

The letsencrypt parameter configures the use of Letsencrypt for obtaining and managing SSL certificates. This ensures OpenShift clusters have valid SSL certificates, replacing the default self-signed certificates. In regular production environments, clusters are expected to be deployed using an organization’s own certificates, instead of default self-signed ones. Letsencrypt, a popular service providing trusted certificates for web services, eliminates browser security warnings and errors caused by self-signed certificates.

The certificates requested by letsencrypt come from the global.localClusterDomain setting.

Support Level

  • AWS Only: Letsencrypt integration is currently supported only on AWS environments within the validated patterns framework.

Example:

Enabling letsencrypt support

To enable Let’s Encrypt support, add the following entries to values-AWS.yaml in the pattern root directory. Be sure to adjust the region: and email: settings as needed:

letsencrypt:
      region: eu-central-1
      server: https://acme-v02.api.letsencrypt.org/directory
      email: foo@bar.it

    clusterGroup:
      applications:
        letsencrypt:
          name: letsencrypt
          namespace: letsencrypt
          project: default
          path: common/letsencrypt

Sub-parameters

  • region: The AWS region where the cluster is running.

  • server: The letsencrypt server endpoint for authentication.

  • email: The e-mail address to use as a DNS contact.

QUICKLINKS

CONTRIBUTE

STAY IN TOUCH

Validated Patterns

Validated Patterns are an evolution of how you deploy applications in a hybrid cloud. With a pattern, you can automatically deploy a full application stack through a GitOps-based framework. With this framework, you can create business-centric solutions while maintaining a level of Continuous Integration (CI) over your application.