The write-up describes how to install API Connect with the secondary GatewayCluster. When you install API Connect using the top-level custom resource (CR) called APIConnectCluster, all the subsystems will be installed — Management cluster, Analytics cluster, Portal cluster and gateway cluster.

In many cases, you will need a second Gateway cluster for external APIs. You can deploy using the custom resource (CR) called GatewayCluster.

We should assume that we have these components installed

1. Red Hat OpenShift Container Platform v4.12
2. Cloud Pak for Integration (common services) v2022.4.1
3. API Connect Operator 3.3

Steps

1. Create a project/namespace

% oc new-project apic

2. Create an IBM entitlement key

% oc create secret docker-registry ibm-entitlement-key \
    --docker-username=cp \
    --docker-password=[REPLACE_ME]
    --docker-server=cp.icr.io \
    --namespace=apic

3. Create APIConnectCluster by applying the following YAML file.

% oc apply -f apiconnectcluster.yaml -n apic

## apiconnectcluster.yaml
apiVersion: apiconnect.ibm.com/v1beta1
kind: APIConnectCluster
metadata:
  name: myapic
  labels:
    app.kubernetes.io/instance: apiconnect
    app.kubernetes.io/managed-by: ibm-apiconnect
    app.kubernetes.io/name: myapic
  namespace: apic
spec:
  license:
    accept: true
    license: L-VQYA-YNM22H
    metric: VIRTUAL_PROCESSOR_CORE
    use: nonproduction
  gateway:
    datapowerLogLevel: 3
    replicaCount: 1
    webGUIManagementEnabled: true
    webGUIManagementPort: 9090
  analytics:
    storage:
      enabled: true
      type: shared
    mtlsValidateClient: true
  portal:
    mtlsValidateClient: true
  profile: n1xc7.m48
  version: 10.0.5.3

4. Create Certificates for external gateway.

% oc apply -f ext-gw-certificate.yaml -n apic

## ext-gw-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myapic-ext-gw-svc
  labels: {
    app.kubernetes.io/instance: "myapic",
    app.kubernetes.io/managed-by: "ibm-apiconnect",
    app.kubernetes.io/name: "myapic-ext-gw-svc"
  }
spec:
  commonName: myapic-ext-gw-svc
  secretName: myapic-ext-gw-svc
  issuerRef:
    name: myapic-ingress-issuer
  usages:
  - "client auth"
  - "signing"
  - "key encipherment"
  duration: 17520h # 2 years
  renewBefore: 48h
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myapic-ext-gw-peer
  labels: {
    app.kubernetes.io/instance: "myapic",
    app.kubernetes.io/managed-by: "ibm-apiconnect",
    app.kubernetes.io/name: "myapic-ext-gw-peer"
  }
spec:
  commonName: myapic-ext-gw-peer
  secretName: myapic-ext-gw-peer
  issuerRef:
    name: myapic-ingress-issuer
  usages:
  - "server auth"
  - "client auth"
  - "signing"
  - "key encipherment"
  duration: 17520h # 2 years
  renewBefore: 48h

5. Create admin secret for external gateway.

% oc create secret generic myapic-ext-gw-admin --from-literal=password=[REPLACE_ME]

6. Create GatewayCluster by applying the following YAML file.

% oc apply -f ext-gatewaycluster.yaml -n apic

## ext-gatewaycluster.yaml
apiVersion: gateway.apiconnect.ibm.com/v1beta1
kind: GatewayCluster
metadata:
  name: myapic-ext-gw
  labels:
    app.kubernetes.io/part-of: myapic
spec:
  version: 10.0.5.3
  profile: n1xc4.m8
  apicGatewayServiceV5CompatibilityMode: false
  gatewayEndpoint:
    annotations:
      cert-manager.io/issuer: myapic-ingress-issuer
    hosts:
    - name: [NAME]-gateway-[NAMESPACE].[DOMAIN_NAME]
      secretName: myapic-ext-gw
  gatewayManagerEndpoint:
    annotations:
      cert-manager.io/issuer: myapic-ingress-issuer
    hosts:
    - name: [NAME]-gateway-manager-[NAMESPACE].[DOMAIN_NAME]
      secretName: [NAME]-gateway-manager
  apicGatewayServiceTLS:
    secretName: myapic-ext-gw-svc
  apicGatewayPeeringTLS:
    secretName: myapic-ext-gw-peer
  certManagerIssuer:
    kind: Issuer
    name: myapic-self-signed
  datapowerLogLevel: 3
  defaultLogFormat: text
  replicaCount: 1
  webGUIManagementEnabled: true
  webGUIManagementPort: 9090
  configSequenceInterval: 3000
  mtlsValidateClient: false
  apiDebugProbeMaxRecords: 1000
  apiDebugProbe: true
  apiDebugProbeExpirationMinutes: 60
  license:
    accept: true
    use: nonproduction
    license: L-VQYA-YNM22H
    metric: VIRTUAL_PROCESSOR_CORE
  adminUser:
    secretName: myapic-ext-gw-admin

7. Check the status of the APIConnectCluster deployment.

% oc get apiconnectcluster -n apic
NAME     READY   STATUS   VERSION    RECONCILED VERSION   MESSAGE                        AGE
myapic   7/7     Ready    10.0.5.3   10.0.5.3-4934        API Connect cluster is ready   2d12h

8. Check the status of the GatewayCluster deployment.

NAME            READY   STATUS    VERSION    RECONCILED VERSION   AGE
myapic-ext-gw   2/2     Running   10.0.5.3   10.0.5.3-4934        2d12h
myapic-gw       2/2     Running   10.0.5.3   10.0.5.3-4934        2d12h

9. After installation is complete, you will need to Register the Gateway.

Tools to be installed:

1. Kubernetes CLI (kubectl)
2. Helm CLI (helm)
3. Operator SDK (olm)
4. Curl

Install certificate manager

(1) Run the following command to install the cert-manager resources from a YAML manifest file on GitHub:

% kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml

(2) To verify the installation

% kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-655bf9748f-4t5fd              1/1     Running   0          41s
cert-manager-cainjector-7985fb445b-7hfpb   1/1     Running   0          42s
cert-manager-webhook-6dc9656f89-qcf5n      1/1     Running   0          40s

(3) Run the following command to patch your deployment. Do not replace ${POD_NAMESPACE} in this command; this value is referenced within the cert-manager.yaml file. When the command completes, it adds a flag to ensure that any auto-generated secrets that store certificates are automatically removed when necessary.

% kubectl patch deployment \
  cert-manager \
  --namespace cert-manager  \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": [
  "--v=2",
  "--cluster-resource-namespace=$(POD_NAMESPACE)",
  "--leader-election-namespace=kube-system",
  "--enable-certificate-owner-ref"
]}]'

Install Operator Lifecycle Manager

(1) Before this, you need to install Operator SDK on MacOS.

% brew install operator-sdk

(2) Install Operator Lifecycle Manager in your cluster. This command creates the relevant custom resources definition.

% operator-sdk olm install

The output is:

INFO[0003] Fetching CRDs for version "latest"           
INFO[0003] Fetching resources for resolved version "latest" 
INFO[0018] Creating CRDs and resources                  
INFO[0018]   Creating CustomResourceDefinition "catalogsources.operators.coreos.com" 
INFO[0019]   Creating CustomResourceDefinition "clusterserviceversions.operators.coreos.com" 
INFO[0020]   Creating CustomResourceDefinition "installplans.operators.coreos.com" 
INFO[0021]   Creating CustomResourceDefinition "olmconfigs.operators.coreos.com" 
INFO[0021]   Creating CustomResourceDefinition "operatorconditions.operators.coreos.com" 
INFO[0021]   Creating CustomResourceDefinition "operatorgroups.operators.coreos.com" 
INFO[0022]   Creating CustomResourceDefinition "operators.operators.coreos.com" 
INFO[0022]   Creating CustomResourceDefinition "subscriptions.operators.coreos.com" 
INFO[0022]   Creating Namespace "olm"                   
INFO[0023]   Creating Namespace "operators"             
INFO[0023]   Creating ServiceAccount "olm/olm-operator-serviceaccount" 
INFO[0023]   Creating ClusterRole "system:controller:operator-lifecycle-manager" 
INFO[0023]   Creating ClusterRoleBinding "olm-operator-binding-olm" 
INFO[0024]   Creating OLMConfig "cluster"               
INFO[0025]   Creating Deployment "olm/olm-operator"     
INFO[0025]   Creating Deployment "olm/catalog-operator" 
INFO[0025]   Creating ClusterRole "aggregate-olm-edit"  
INFO[0026]   Creating ClusterRole "aggregate-olm-view"  
INFO[0026]   Creating OperatorGroup "operators/global-operators" 
INFO[0026]   Creating OperatorGroup "olm/olm-operators" 
INFO[0027]   Creating ClusterServiceVersion "olm/packageserver" 
INFO[0027]   Creating CatalogSource "olm/operatorhubio-catalog" 
INFO[0027] Waiting for deployment/olm-operator rollout to complete 
INFO[0028]   Waiting for Deployment "olm/olm-operator" to rollout: 0 of 1 updated replicas are available 
INFO[0030]   Deployment "olm/olm-operator" successfully rolled out 
INFO[0030] Waiting for deployment/catalog-operator rollout to complete 
INFO[0030]   Deployment "olm/catalog-operator" successfully rolled out 
INFO[0030] Waiting for deployment/packageserver rollout to complete 
INFO[0031]   Waiting for Deployment "olm/packageserver" to rollout: 0 of 2 updated replicas are available 
INFO[0036]   Deployment "olm/packageserver" successfully rolled out 
INFO[0042] Successfully installed OLM version "latest"                          olm          CatalogSource               Installed

(3) Verify the CRDs installed.

% kubectl get crds

The output is:

NAME                                            NAMESPACE    KIND                        STATUS
catalogsources.operators.coreos.com                          CustomResourceDefinition    Installed
clusterserviceversions.operators.coreos.com                  CustomResourceDefinition    Installed
installplans.operators.coreos.com                            CustomResourceDefinition    Installed
olmconfigs.operators.coreos.com                              CustomResourceDefinition    Installed
operatorconditions.operators.coreos.com                      CustomResourceDefinition    Installed
operatorgroups.operators.coreos.com                          CustomResourceDefinition    Installed
operators.operators.coreos.com                               CustomResourceDefinition    Installed
subscriptions.operators.coreos.com                           CustomResourceDefinition    Installed
olm                                                          Namespace                   Installed
operators                                                    Namespace                   Installed
olm-operator-serviceaccount                     olm          ServiceAccount              Installed
system:controller:operator-lifecycle-manager                 ClusterRole                 Installed
olm-operator-binding-olm                                     ClusterRoleBinding          Installed
cluster                                                      OLMConfig                   Installed
olm-operator                                    olm          Deployment                  Installed
catalog-operator                                olm          Deployment                  Installed
aggregate-olm-edit                                           ClusterRole                 Installed
aggregate-olm-view                                           ClusterRole                 Installed
global-operators                                operators    OperatorGroup               Installed
olm-operators                                   olm          OperatorGroup               Installed
packageserver                                   olm          ClusterServiceVersion       Installed
operatorhubio-catalog                           olm          CatalogSource               Installed

Create Operator Group for IBM App Connect Operator

(1) Create a namespace ace of App Connect Operator

% kubectl create namespace ace

(2) Create OperatorGroup (ace as the namespace) using the yaml file below

## 
## appconn-operator-group.yaml
## 
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: ace-operator-group
  namespace: ace
spec:
  targetNamespaces:
  - ace

% kubectl apply -f appconn-operator-group.yaml

Create the IBM App Connect Operator catalog

(1) Create a CatalogSource object for the IBM App Connect Operator using the yaml file below.

## 
## appconn-catalog-source.yaml
## 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-appconnect-catalog
  namespace: olm
spec:
  displayName: "IBM App Connect Operator Catalog k8S"
  publisher: IBM
  sourceType: grpc
  image: icr.io/cpopen/appconnect-operator-catalog-k8s
  updateStrategy:
    registryPoll:
      interval: 45m
  grpcPodConfig:
    securityContextConfig: restricted

% kubectl apply -f appconn-catalog-source.yaml -n olm

(2) Verify the installation.

% kubectl get CatalogSources ibm-appconnect-catalog -n olm
NAME                     DISPLAY                                TYPE   PUBLISHER   AGE
ibm-appconnect-catalog   IBM App Connect Operator Catalog k8S   grpc   IBM         2m21s

(3) If there is problem, check the pods and CatalogSource resources

% kubectl get catalogsource,pods -n olm
NAME                                                        DISPLAY                                TYPE   PUBLISHER        AGE
catalogsource.operators.coreos.com/ibm-appconnect-catalog   IBM App Connect Operator Catalog k8S   grpc   IBM              4m6s
catalogsource.operators.coreos.com/operatorhubio-catalog    Community Operators                    grpc   OperatorHub.io   5h32m

NAME                                    READY   STATUS    RESTARTS   AGE
pod/catalog-operator-655fb46cd4-gzfvn   1/1     Running   0          5h32m
pod/ibm-appconnect-catalog-vjdsj        1/1     Running   0          4m6s
pod/olm-operator-67fdb4b99d-sdklb       1/1     Running   0          5h32m
pod/operatorhubio-catalog-jbmkg         1/1     Running   0          5h32m
pod/packageserver-586d76ddcf-9rhbp      1/1     Running   0          5h32m
pod/packageserver-586d76ddcf-cc4mx      1/1     Running   0          5h32m

Create the IBM App Connect Operator subscription

(1) Create Subscription object for IBM App Connect Operator.

##
## appconn-sub.yaml
##
kind: Subscription
metadata:
  name: ibm-appconnect
  namespace: ace
spec:
  channel: v7.2
  name: ibm-appconnect
  source: ibm-appconnect-catalog
  sourceNamespace: olm

% kubectl apply -f appconn-sub.yaml

(2) Check progress of installation — view the logs for the Catalog Operator pod in the olm namespace.

% kubectl logs catalog-operator-655fb46cd4-gzfvn -n olm

Add entitlement key to namespace

(1) Obtain the entitlement key from this IBM Container software library.

(2) Create a secret that holds the entitlement key.

% oc create secret docker-registry ibm-entitlement-key \
    --docker-username=cp \
    --docker-password=myEntitlementKey \
    --docker-server=cp.icr.io \
    --namespace=ace

Creating ingress definitions for external access to your IBM App Connect instances

(1) Create namespace for ingress

% kubectl create namespace ingress

(2) Add the NGINX ingress controller chart repository and then update all Helm repositories in your system.

% helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
% helm repo update

(3) Install the Helm chart in the ingress namespace.

% helm install nginx-ingress ingress-nginx/ingress-nginx \
     --namespace ingress \
     --set controller.replicaCount=2 \
     --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
     --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
     --set controller.admissionWebhooks.patch.nodeSelector."beta\.kubernetes\.io/os"=linux

Result output:

NAME: nginx-ingress
LAST DEPLOYED: Fri Mar 24 22:31:05 2023
NAMESPACE: ingress
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress get services -o wide -w nginx-ingress-ingress-nginx-controller'

(4) To check the status for the Load Balancer IP.

% kubectl --namespace ingress get services -o wide nginx-ingress-ingress-nginx-controller
NAME                                     TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE    SELECTOR
nginx-ingress-ingress-nginx-controller   LoadBalancer   10.0.169.121   xxx.xxx.xxx.xxx   80:30461/TCP,443:30688/TCP   4m6s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx

(5) Note down the external IP address. We will used this to configure our Ingress objects later.

Deploy App Connect Dashboard

(1) Create Storage Class for writing purpose.

##
## my-azurefile-csi-premium.yaml
##
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: my-azurefile-csi-premium
provisioner: kubernetes.io/azure-file
mountOptions:
- mfsymlinks
- actimeo=30
- dir_mode=0777
- file_mode=0777
- uid=1000
- cache=strict
parameters:
  skuName: Premium_LRS 
reclaimPolicy: Delete
volumeBindingMode: Immediate

% oc apply -f my-azurefile-csi-premium.yaml

(2) Create App Connect Dashboard. See https://ibm.biz/acelicense for license and use fields.

##
## appconn-ace-dashboard.yaml
## 
apiVersion: appconnect.ibm.com/v1beta1
kind: Dashboard
metadata:
  name: ace-dashboard
  namespace: ace
spec:
  license:
    accept: true
    license: L-APEH-CJUCNR
    use: AppConnectEnterpriseNonProductionFREE
  pod:
    containers:
      content-server:
        resources:
          limits:
            cpu: 250m
            memory: 512Mi
          requests:
            cpu: 50m
            memory: 50Mi
      control-ui:
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 50m
            memory: 125Mi
  replicas: 1
  storage:
    size: 5Gi
    type: persistent-claim
    class: my-azurefile-csi-premium
  useCommonServices: false
  version: '12.0.7-lts'

% kubectl apply -f appconn-ace-dashboard.yaml  -n ace

(3) Create ingress route for App Connect Dashboard. Since you do not have DNS, we will use nip.io to configure the host name. Use the External IP you have noted earlier.

##
## appconn-ingress-dashboard.yaml
## 
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: ace-dashboard-ingress
  namespace: ace
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
spec:
  tls:
    - hosts:
      - ace-dashboard.XXX.XXX.XXX.XXX.nip.io
  rules:
    - host: ace-dashboard.XXX.XXX.XXX.XXX.nip.io
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: ace-dashboard-dash
              port:
                number: 8300

% kubectl apply -f appconn-ingress-dashboard.yaml -n ace

(4) Check the status of the Dashboard.

% kubectl get dashboard -n ace
NAME            RESOLVEDVERSION   REPLICAS   CUSTOMIMAGES   STATUS    URL        AGE
ace-dashboard   12.0.7.0-r4       1          false          Ready     https://   11h

(5) Finally, you can access the Dashboard using the URL

https://ace-dashboard.XXX.XXX.XXX.XXX.nip.io

Deploy App Connect Integration Server

(1) Before we start, we are going to use this sample BAR file.

https://github.com/khongks/cp4i-ace-books/raw/main/BARfiles/Books.bar

(2) Create a Configuration object that contains authentication information to git repository.

##
## github-barauth.yaml
##
apiVersion: appconnect.ibm.com/v1beta1
kind: Configuration
metadata:
  name: github-barauth
  namespace: ace
spec:
  description: Stores bar auth
  data: eyJhdXRoVHlwZSI6IkJBU0lDX0FVVEgiLCJjcmVkZW50aWFscyI6eyJ1c2VybmFtZSI6IiIsInBhc3N3b3JkIjoiIn19Cg==
  type: baraut

% oc apply -f github-barauth.yaml -n ace

Note that the data field contains a base64 encoding of the authentication information.

{"authType":"BASIC_AUTH","credentials":{"username":"","password":""}}

(3) Create App Connect Integration Server. See https://ibm.biz/acelicense for license and use fields.

##
## appconn-ingress-intserverhttp.yaml
##
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ace-books-ingress
  namespace: ace
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
spec:
  tls:
    - hosts:
      - ace-books.xxx.xxx.xxx.xxx.nip.io
  rules:
    - host: ace-books.xxx.xxx.xxx.xxx..nip.io
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: ace-books-is
              port:
                number: 7800

(4) Check the status of the Integration Server.

% kubectl get integrationserver -n ace
NAME        RESOLVEDVERSION   REPLICAS   AVAILABLEREPLICAS   CUSTOMIMAGES   STATUS   AGE
ace-books   12.0.7.0-r4       1          1                   false          Ready    11h

(5) Finally, you can test the REST API running on the Integration Server.

curl -kv https://ace-books.xxx.xxx.xxx.xxx..nip.io/api/v1/books

Reference Links:

Installing IBM App Connect in a Kubernetes environment

https://community.ibm.com/community/user/integration/blogs/joel-gomez1/2022/02/15/installing-the-ibm-app-connect-operator-on-amazon

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/connection-issues-application-hosted-aks-cluster

I have drawn up a simple chart the illustrate the available deployment and platform options.

If you look at the product in Cloud Pak for Integration (v2022.1), we could map out the deployment options in the following table.

Grid view - Airtable

In the traditional world, where DataPower runs on physical or virtual appliances, the developer will define the services and configuration within a DataPower domain using the DataPower WebUI. The changes are applied and saved; and because service becomes active immediately. In the container world, the development process will be slightly different. The following diagram illustrates this process:

Development Process

1. The DataPower developer could use a DataPower running on a Docker container to develop the DataPower services. There is a couple of good articles on how to do this https://github.com/fxnaranjo/datapower-operator or https://github.com/ibm-datapower/datapower-tutorials/blob/master/getting-started/start-with-docker.md.
2. Once, the development work is completed and unit tested, the developer can export the configuration to an export file.
3. The developer can use the migrate-backup.sh script to extract the configuration from the export file into YAML files (in a folder). These generated YAML files can be applied directly to create config maps.
4. The developer will then check-in the export folder into a Git repository. The check-in event can be configured to trigger a Jenkins Pipeline.

The detailed steps are found in this IBM documentation.

Jenkins Pipeline

1. Pre-deploy stage: creates the Namespace, pull Secret for IBM entitlement key, admin credential, for TLS keys and certificates (not shown in the article), and ConfigMaps for configurations.
2. Deploy stage: creates DataPowerService (which in turn creates DataPowerMonitor), Service that sends requests to the defined DataPower service endpoint, and Route that sends requests to the Service.
3. Test service stage: Runs a curl command to test the endpoint (via the Route).

Step by Step:

Before you begin, you must have the following setup

• Jenkins — see my previous article on installing Jenkins on the same OpenShift cluster of the CP4I.
• Red Hat OpenShift cluster installed (my version is 4.8.36), Also tested on v4.10.17.
• Cloud Pak for Integration is installed on the OpenShift cluster (my version is 2021.4.1). Also tested on v2022.2.1.
• Github account. Updated to v2022.2.1 (using DataPower v10.5.0.0).
• Clone/fork this repository.

Create a credential in Jenkins to store the IBM entitlement key and DataPower admin password (from Jenkins Console)

1. Create Global credentials (kind: Secret text) to store the IBM entitlement key and DataPower admin password. See the detailed steps in the previous article.
2. These two credentials will be referenced in the Jenkinsfile to be used by the pipeline.

Configure a pipeline to deploy DataPower service (from Jenkins Console)

1. Click on Dashboard > New Item > Pipeline. Provide a name (e.g. mq-pipeline) and click Ok.
2. Go to Pipeline section, specify the following and click Save.

• Definition: Pipeline script from SCM
• SCM: Git
• Repository URL: https://github.com/khongks/dp-jenkins.git
• Branch Specifier: */main
• Script Path: Jenkinsfile

Perform test build (from Jenkins Console)

1. Click Build Now.
2. Click Build History > #1 > Console Output
3. When the pipeline is completed, you should see the following output, with the test results of the curl command.

Pre Deploy Output

Pre-Deploy ~ setup configuration before deploy
[Pipeline] sh
+ ./scripts/01-pre-deploy.sh **** hello dp ****
----------------------------------------------------------------------
 INFO: Pre deploy
----------------------------------------------------------------------

Namespace dp found
create_pull_secret ibm-entitlement-key, dp, cp.icr.io, cp, ****, khongks@gmail.com

Secret ibm-entitlement-key already created
Apply all YAMLs from the backup
export-output/default-cfg.yaml
oc apply -f export-output/default-cfg.yaml -n dp

configmap/default-cfg configured
export-output/default-local.yaml
oc apply -f export-output/default-local.yaml -n dp

configmap/default-local configured
export-output/hello-cfg.yaml
oc apply -f export-output/hello-cfg.yaml -n dp

configmap/hello-cfg configured

Delete and Create admin-credentials secret

secret "admin-credentials" deleted

secret/admin-credentials created

Deploy Output

Deploy ~ deploy DataPowerService
[Pipeline] sh
+ ./scripts/02-deploy.sh hello dp L-RJON-C5SF54 production 10.0.4.0
----------------------------------------------------------------------
 INFO: Deploy
----------------------------------------------------------------------

datapowerservice.datapower.ibm.com/hello unchanged
service/hello-service unchanged
route.route.openshift.io/hello-route unchanged
Waiting for [hello] of type [DatapowerService] in namespace [dp] to be in [Running] status

Installation status: Failed
=== Installation has failed ===
=== Retry again ===
Sleeping 30 seconds...

Installation status: Running

Test Service Output

Test Service
[Pipeline] sh
+ ./scripts/03-test-service.sh hello

----------------------------------------------------------------------
 INFO: Test endpoint https://hello.itzroks-3100015379-wxzo6e-6ccd7f378ae819553d37d5f2ee142bd6-0000.au-syd.containers.appdomain.cloud/api/v1/users
----------------------------------------------------------------------

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

...

{"createdAt":"2022-05-27T09:40:32.058Z","name":"Anthony Gorczany","avatar":"https://cloudflare-ipfs.com/ipfs/Qmd3W5DuhgHirLHGVixi6V76LhCkZUz6pnFt5AJBiyvHye/avatar/409.jpg","id":"45"},{"createdAt":"202
100  8562    0  8562    0     0    325      0 --:--:--  0:00:26 --:--:--  1756
100  9084    0  9084    0     0    345      0 --:--:--  0:00:26 --:--:--  2346
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }

[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node

[Pipeline] End of Pipeline
Finished: SUCCESS

References

https://github.com/fxnaranjo/datapower-operator

https://github.com/ibm-datapower/datapower-tutorials/blob/master/getting-started/start-with-docker.md

https://www.ibm.com/docs/en/cloud-paks/cp-integration/2021.4?topic=integration-datapower-gateway-transformation-guide

https://www.ibm.com/docs/en/cloud-paks/cp-integration/2022.2?topic=integration-datapower-gateway-transformation-guide

IBM DataPower Operator

Disclaimer:

All opinions expressed here are very much my own and not of IBM. All codes/scripts/artifacts are provided as-is with no support unless otherwise stated.

The following diagram show ONE strategy on how you can do that for API Connect deployed on OpenShift (via the Cloud Pak for Integration). Note that this strategy is not the only way.

Disclaimer:

All opinions expressed here are very much my own and not of IBM. All codes/scripts/artifacts are provided as-is with no support unless otherwise stated.

This is a continuation of my previous story (link)on how to deploy MQ on CP4I using Jenkins. Deploying ACE is more complex than MQ because to me, ACE is really an integration application where you need to compile into a deployable unit called Broker Archive (BAR) file before you can deploy to an integration server.

Therefore, a typical CI/CD pipeline of deploying ACE applications would look very similar to any other applications pipeline. The diagram shows the System Context diagram of the pipeline for the purpose of this article.

Our pipeline above has the following stages:

• Git Checkout — This stage check out the source code for App Connect Enterprise (ACE) from a Git repository
• Build Bar File — Package the ACE application artifacts into a BAR file. To do so, we need to run this using a Docker image that is installed with ACE Toolkit.
• Upload Bar File — Upload the BAR file into a file repository, e.g. JFrog Artifactory.
• Deploy Integration Server — Deploy an integration server that is configured to point to the file location of the previous BAR file.
• Unit Test — Use a curl command to test the API endpoint of the integration application.

Required components/systems

The following software is needed to implement this pipeline. I will not write in detail how to install the various software; I will just provide references on how to do this.

• Jenkins — see my previous article on installing Jenkins on the same OpenShift cluster of the CP4I.
• JFrog Artifactory — We use this for artifact management. I installed it in the OpenShift cluster using the IBM Cloud Native Toolkit.
• Docker Registry — I used the default OpenShift image registry.
• GitHub — I used the public GitHub.com. I have two repositories, one contains the source code of the integration application and another contains the utility artifacts like scripts and templates.
• OpenShift cluster — installed with CP4I and ACE dashboard. OCP is v4.8.26. CP4I is v2021.4.1.

Create build and deploy images

I used 3 images for the pipeline.

• jnlp-slave image which is located here
• oc-deploy image that has these tools — openshift-client, kubectl, curl, jq, git, sed, wget. This image is used in the stages-Upload Bar File, Deploy Integration Server, and Unit Test. The Dockerfile is located here.
• ace-buildbar image that has the ACE toolkit and configured with libraries to run mqsicreatebar (e.g. libgtk2.0–0 libxtst6 xvfb libswt-gtk-4-java libswt-gtk-4-jni). The Dockerfile is located here.

You need to build the image, tag, and push to the OpenShift image registry. The following are the instructions for building the ace-buildbar image. You can follow the same instructions for the oc-deploy image.

// Build the image
docker build -t ace-buildbar:12.0.4.0-ubuntu -f Docker.ace-12.0.4-buildbar .

// Create an external route for the image registry
oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge

// Get the hostname of the image registry
oc get route -n openshift-image-registry default-route -ojson | jq -r .spec.host

// Get the hostname into variable HOST
HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')

// Login to OpenShift image registry
docker login -u $(oc whoami) -p $(oc whoami -t) $HOST

// Tag the image
docker tag ace-buildbar:12.0.4.0-ubuntu $HOST/jenkins/ace-buildbar:12.0.4.0-ubuntu

// Push the image into the OpenShift image registry
docker push $HOST/jenkins/ace-buildbar:12.0.4.0-ubuntu

The image can now be pulled by containers within the OpenShift cluster via the following tag.

image-registry.openshift-image-registry.svc:5000/jenkins/ace-buildbar:12.0.4.0-ubuntu

Set up Artifactory credentials in Jenkins

1. Create Global credentials (kind: Username with password) to store the username and password for JFrog Artifactory. I used the name artifactory_credentials in the Jenkinsfile to reference the username/password securely.

Set up Artifactory credentials (BarAuth) for ACE

We use the spec.barURL in the integration server to retrieve the Bar file from JFrog Artifactory. To do that we need to set up the BarAuth configuration that contains both the credentials and CA certificate.

Here is the IBM ACE documentation on Configuration and BarAuth.

Create a secret for the certificate

1. Get the certificate for the JFrog Artifactory endpoint.

openssl s_client -connect artifactory-tools.itzroks-3100015379-x94hbr-6ccd7f378ae819553d37d5f2ee142bd6-0000.au-syd.containers.appdomain.cloud:443

2. Create a secret using the following YAML (artifactory-cacert-secret.yaml).

kind: Secret
apiVersion: v1
metadata:
    name: artifactory-cacert-secret
    namespace: ace
data:
    ca.crt: '{{ base64 encoded of the certificate }}'
    tls.crt: ''
    tls.key: ''
type: kubernetes.io/tls

3. Note: Use the following instruction to get the ca.crt value

cat artifactory-ca.crt | base64

4. Finally, run the following command to create the secret.

oc apply -f artifactory-cacert-secret.yaml

Create a BarAuth configuration

1. Create a BarAuth JSON file, see here. Populate the values as follows.

{
    "authType":"BASIC_AUTH",
    "credentials": {
        "username":"{{username}}",
        "password":"{{password}}",
        "caCertSecret":"artifactory-cacert-secret"
    }
}

2. Create a BarAuth secret using the JSON file in the namespace you want to deploy the ACE integration server.

oc create secret generic bar-auth-secret --from-file=configuration=bar-auth.json --namespace=ace

3. Finally, create the ACE Configuration object referencing the secret name, the YAML file here.

oc create -n ace -f bar-auth-config.yaml

// bar-auth-config.yaml
apiVersion: appconnect.ibm.com/v1beta1
kind: Configuration
metadata:  
    name: bar-auth-config  
    namespace: ace
spec:  
    description: Stores bar auth
    secretName: bar-auth-secret
    type: barauth

Set up Jenkins pipeline

1. Create a pipeline that points to our source code Git repository. The Git repository contains a Jenkinsfile that defines the entire Pipeline script.

Upload and retrieve files from JFrog Artifactory

1. The easiest way to know how to use upload and retrieve is to go to the JFrog Artifactory UI and click on the Set Me Up button (top right).

2. I have written two scripts to do this — in this folder. To upload a file, we need to calculate the checksum and include it in the HTTP header.

CHECKSUM=`shasum -a 1 $ARTIFACTORY_BAR_FILE | awk '{ print $1 }'`

-H 'X-Checksum-Sha1:${CHECKSUM}'

Anatomy of the Jenkinfiles

The following pipeline is implemented in the Jenkinsfile that is conveniently located together with the source code Git repository.

Git Checkout stage: This stage check out the source code and utility files (scripts and templates) from their respective Git repositories. The script files and template file (integration server YAML file) are copied into the source code folder.

Build Bar File stage: This stage uses the ace-buildbar container to create the Bar file. The runs the following commands:

• It runs the X virtual frame buffer to enable you to run graphical applications without a display. It is needed for the mqsicreatebar command because it uses some SWT libraries.
• Run the mqsiprofile to configure the environment for ACE toolkit.
• Append the build number to the Bar file name.
• Run mqsicreatebar to generate the Bar file.

Xvfb -ac :99 &
export DISPLAY=:99

export LICENSE=accept

pwd
source /opt/ibm/ace-12/server/bin/mqsiprofile
cd $PROJECT_DIR

BAR_FILE="${BAR_NAME}_${BUILD_NUMBER}.bar"
mqsicreatebar -data . -b $BAR_FILE -a $APP_NAME -cleanBuild -trace -configuration .

ls -lha

Build Bar File stage: This stage uses the ace-buildbar container to create the Bar file. The runs the following commands:

• Run the script upload-barfile-to-artifactory.sh that does precisely that.

set -e

cd $PROJECT_DIR

ls -lha

echo "Calling upload-barfile-to-artifactory.sh ${ARTIFACTORY_HOST} ${ARTIFACTORY_REPO} ${ARTIFACTORY_BASE_PATH} "${BAR_NAME}_${BUILD_NUMBER}.bar" ${ARTIFACTORY_USER} ${ARTIFACTORY_PASSWORD}"

./upload-barfile-to-artifactory.sh ${ARTIFACTORY_HOST} ${ARTIFACTORY_REPO} ${ARTIFACTORY_BASE_PATH} "${BAR_NAME}_${BUILD_NUMBER}.bar" ${ARTIFACTORY_USER} ${ARTIFACTORY_PASSWORD}

Deploy Integration Server stage: This stage deploys the ACE integration server by fetching the Bar file from JFrog Artifactory. The commands are as follows:

• Create the integration-server.yaml file from the template file, but substitute the variables.
• Run an oc command to deploy the integration server, using the YAML file.

set -e
cd $PROJECT_DIR
BAR_FILE="${BAR_NAME}_${BUILD_NUMBER}.bar"
cat integration-server.yaml.tmpl
sed -e "s/{{NAME}}/$SERVER_NAME/g" \
    -e "s/{{NAMESPACE}}/$NAMESPACE/g" \
    -e "s/{{ARTIFACTORY_HOST}}/$ARTIFACTORY_HOST/g" \
    -e "s/{{ARTIFACTORY_PORT}}/$ARTIFACTORY_PORT/g" \
    -e "s/{{ARTIFACTORY_REPO}}/$ARTIFACTORY_REPO/g" \
    -e "s/{{ARTIFACTORY_BASE_PATH}}/$ARTIFACTORY_BASE_PATH/g" \
    -e "s/{{BAR_FILE}}/$BAR_FILE/g" \
    -e "s/{{CONFIGURATION_LIST}}/$CONFIGURATION_LIST/g" \
    -e "s/{{ACE_VERSION}}/$ACE_VERSION/g" \
    -e "s/{{ACE_LICENSE}}/$ACE_LICENSE/g" \
    -e "s/{{REPLICAS}}/$REPLICAS/g" \
    integration-server.yaml.tmpl > integration-server.yaml
cat integration-server.yaml
oc apply -f integration-server.yaml

The sample copy of the integration-server.yaml file is as follows. It configures the following files

• metadata.name — with the integration server name.
• spec.barURL — with artifactory host, artifactory port, artifactory repository, bar file name.
• spec.configurations — with configurationList variable.

apiVersion: appconnect.ibm.com/v1beta1
kind: IntegrationServer
metadata:
  name: books
  namespace: ace
spec:
  license:
    accept: true
    license: L-APEH-C79J9U
    use: CloudPakForIntegrationNonProduction
  pod:
    containers:
      runtime:
        resources:
          limits:
            cpu: 300m
            memory: 350Mi
          requests:
            cpu: 300m
            memory: 300Mi
  adminServerSecure: true
  enableMetrics: true
  createDashboardUsers: true
  barURL: https://artifactory-tools.itzroks-3100015379-x94hbr-6ccd7f378ae819553d37d5f2ee142bd6-0000.au-syd.containers.appdomain.cloud:443/artifactory/generic-local/cp4i/Books_14.bar
  configurations: [ bar-auth-config  ]
  router:
    timeout: 120s
  designerFlowsOperationMode: disabled
  service:
    endpointType: http
  version: '12.0'
  replicas: 1
  logFormat: basic

Unit Test stage: This stage performs a simple unit test using a curl command.

HOSTNAME=$(oc get route -n ace books-http -ogo-template --template='{{.spec.host}}')
curl -k http://${HOSTNAME}/api/v1/books | jq -r .

Sample output:

[
  {
    "Id": "1",
    "Title": "Steve Jobs",
    "ISBN": "1451648537",
    "Author": "Walter Isaacson",
    "Published": "Nov 24, 2011",
    "Language": "English",
    "Formats": [
      "Hardcover",
      "Paperback",
      "Audiobook CD",
      "Audible"
    ]
  },
  {
    "Id": "2",
    "Title": "Beautiful Whale",
    "ISBN": "1419703846",
    "Author": "Bryant Austin, Sylvia Earle",
    "Published": "May 02, 2013",
    "Language": "English",
    "Formats": [
      "Hardcover"
    ]
  },
  {
    "Id": "3",
    "Title": "A Game of Thrones (A Song of Ice and Fire, Book 1)",
    "ISBN": "9780553103547",
    "Author": "George R. R. Martin",
    "Published": "Sep 01, 1996",
    "Language": "English",
    "Formats": [
      "Kindle",
      "Hardcover",
      "Paperback",
      "Audiobook CD",
      "Audible"
    ]
  }
]

Conclusion

Finally, a video showing how everything hangs together.

Disclaimer:

All opinions expressed here are very much my own and not of IBM. All codes/scripts/artifacts are provided as-is with no support unless otherwise stated.

Before you begin, you must have the following setup

• Red Hat OpenShift cluster installed (my version is 4.8.21)
• Cloud Pak for Integration installed on the OpenShift cluster (my version is 2021.3.1)
• Github account
• Clone this repository (https://github.com/khongks/mq-pipeline.git)

Install Jenkins on Red Hat OpenShift (from the Openshift Console)

1. Create a project called jenkins.
2. From the Operator Hub, install jenkins-operator provided by Red Hat in the jenkins namespace. Wait for installation to complete.

3. From the jenkins-operator, create a Jenkins instance. Give a name to your instance (e.g. jenkins-kks). Wait for installation to complete.

4. To access the Jenkins console, get the route using the following command in a terminal.

oc get route -n jenkins jenkins-jenkins-kks -ojson | jq -r .spec.host

You should have the URL for the Jenkins console that looks like this:

jenkins-jenkins-kks-jenkins.<domain-name>

Create a credential in Jenkins to store IBM entitlement key (from Jenkins Console)

1. Click on Manage Jenkins > Manage Credentials

2. Click on Jenkins (global) > Global credentials (unrestricted)

3. Click on Add Credentials, with the following setup.

• kind: Secret text
• Scope: Global
• Secret: (obtained from https://myibm.ibm.com/products-services/containerlibrary)
• ID: ibm_entitlement_key

This ID is reference by the Jenkins pipeline to retrieve the value of the Secret.

Enable Jenkins service account to create objects in OpenShift (from terminal)

1. Run this command to find the service account in the namespace

oc get sa -n jenkins

The response should be:

NAME                SECRETS AGE
builder             2       73m
default             2       73m
deployer            2       73m
jenkins-jenkins-kks 2       54m
jenkins-operator    2       56m

2. Run this command to assign cluster-admin role to the service account. I assigned cluster-admin role, to simplify the setup; but you can create a custom role with the appropriates to do this.

oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:jenkins:jenkins-jenkins-kks

Configure a pipeline to build MQ queue manager (from Jenkins Console)

1. Click on Dashboard > New Item > Pipeline. Provide a name (e.g. mq-pipeline) and click Ok.
2. Go to Pipeline section, specify the following and click Save.

• Definition: Pipeline script from SCM
• SCM: Git
• Repository URL: https://github.com/khongks/mq-pipeline.git
• Branch Specifier: */main
• Script Path: Jenkinsfile

Perform test build (from Jenkins Console)

1. Click Build Now.
2. Click Build History > #1 > Console Output
3. When the pipeline is completed, you should see the following output, with the MQ Console URL, with login user and password. It also generates a ccdt.json to access the queue manager.

...
Installation status: Running
MQ console URL: <<MQ_CONSOLE_URL>>
User: admin
Pass: <<PASSWORD>>
{
  "channel": [
    {
      "name": "QM1CHL",
      "clientConnection": {
        "connection": [
          {
            "host": "<<HOST>>",
            "port": 443
          }
        ],
        "queueManager": "QM1"
      },
      "transmissionSecurity": {
        "cipherSpecification": "ANY_TLS12_OR_HIGHER"
      },
      "type": "clientConnection"
    }
  ]
}
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

4. You should see the status of the completed job (#3 as follows).

Test the deployed queue manager (from terminal)

1. Clone the git repository

git clone https://github.com/khongks/mq-pipeline.git

2. Go to the folder mq-pipeline/test and update the file ccdt.json.

3. Put a message.

./mq-put.sh
Sample AMQSPUT0 start
target queue is APPQ
TESTING

4. Get the message

./mq-get.sh
Sample AMQSGET0 start
message <TESTING>

Configure Jenkins pipeline to be triggered from GitHub events

(From the Github web)

1. Go to this Github URL to add a webhook

https://github.com/{GIT_ORG}/mq-pipeline/settings/hooks/new

2. Configure Payload URL and Content type

• Payload URL: https://<jenkins-route-hostname>/github-webhook. You can obtain this value from previous step
• Content type: application/json

3. Select event types that triggers this weekhook.

• Pushes
• Pull requests

4. Finally, click on Add webhook to create the webhook.

(From Jenkins console)

5. Click Configure > Build Triggers and check GitHub hook trigger for GITScm polling. Click Save.

6. To test, make a change in the config.mqsc.tmpl file and push the changes into Github repository.

7. Note that build job is triggered in Github.

and in Jenkins.

Hopeful you enjoy this tutorial. Merry Christmas and Happy New Year!

Disclaimer:

All opinions expressed here are very much my own and not of IBM. All codes/scripts/artifacts are provided as-is with no support unless otherwise stated.

Before I continue, let us understand vanity endpoints in API Connect.

• What is vanity API endpoint?
• Why is it needed?
• How it is implemented?

From the API Connect documentation, an API has two endpoints

• The gateway endpoint at which the API is invoked.
• The endpoint that is visible to the consumer in the Developer Portal.

By default, these two endpoints are the same. E.g. if your provider organization is acme, your catalog is public, your API base path is /v1/app and the resource path is /customers; then the application will consume the API using this endpoint

https://apigateway/acme/public/v1/app/customers

However, in most cases, company would like their public APIs to be known differently. Using the example above, they would like to “hide” the provider organization and catalog in API endpoint.

https://apigateway/v1/app/customer

So, the way to do this in API Connect is using vanity API endpoint, which can be configured in the Catalog. In doing so, this vanity API endpoint will be published to the Developer Portal and is used by the application to invoke the API.

However, what is not clear in the documentation is you have to handle the traffic routing so that when the application invokes the published vanity API endpoint, it gets routed to the actual API invocation endpoint of the gateway services.

There are various ways to implement this — and this article describe how this can be done using ingress in K8S. The following is an example of the ingress object.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/proxy-read-timeout: "240"
    ingress.kubernetes.io/proxy-send-timeout: "240"
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/rewrite-target: /${porg_name}/${catalog_name}/\$1
    ingress.kubernetes.io/ssl-prefer-server-ciphers: "true"
    ingress.kubernetes.io/ssl-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256"
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: ${release_name}
    meta.helm.sh/release-namespace: ${namespace}
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/proxy-read-timeout: "240"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "240"
    nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /${porg_name}/${catalog_name}/\$1
    nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers: "true"
    nginx.ingress.kubernetes.io/ssl-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256"
  name: ${name}
  namespace: ${namespace}
spec:
  rules:
  - host: ${endpoint_name}
    http:
      paths:
      - backend:
          serviceName: ${release_name}-dynamic-gateway-service-ingress
          servicePort: 9443
        path: /(.*)
  tls:
  - hosts:
    - ${endpoint_name}
    secretName: ${secret_name}
EOF

The above ingress does the following

• Terminates the TLS using the server certificate and key defined a secret. In this case, the secret contains the tls.crt and tls.key.
• Defines a list of preferred ciphers
• Rewrite any incoming URI path to /${porg_name}/${catalog_name}/${incoming_uri_path}

Another variation of this configuration is to terminate MTLS traffic. This ingress object shows how this can be done.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/proxy-read-timeout: "240"
    ingress.kubernetes.io/proxy-send-timeout: "240"
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/rewrite-target: /${porg_name}/${catalog_name}/\$1
    ingress.kubernetes.io/ssl-prefer-server-ciphers: "true"
    ingress.kubernetes.io/ssl-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256"
    ingress.kubernetes.io/auth-tls-verify-client: "on"
    ingress.kubernetes.io/auth-tls-secret: "${namespace}/${secret_name}"
    ingress.kubernetes.io/auth-tls-verify-depth: "2"
    ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: ${release_name}
    meta.helm.sh/release-namespace: ${namespace}
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/proxy-read-timeout: "240"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "240"
    nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /${porg_name}/${catalog_name}/\$1
    nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers: "true"
    nginx.ingress.kubernetes.io/ssl-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-secret: "${namespace}/${secret_name}"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "2"
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
  name: ${name}
  namespace: ${namespace}
spec:
  rules:
  - host: ${endpoint_name}
    http:
      paths:
      - backend:
          serviceName: ${release_name}-dynamic-gateway-service-ingress
          servicePort: 9443
        path: /(.*)
  tls:
  - hosts:
    - ${endpoint_name}
    secretName: ${secret_name}

The above ingress does the following

• Terminates the MTLS using the server certificate and key defined a secret, and contains the signer certificate in truststore. In this case, the secret contains the tls.crt, tls.key and ca.crt.
• Defines a list of preferred ciphers
• Pass on the client certificate to downstream application as header ssl-client-cert in URL encoded format. So, if downstream application wants to use the certification, it needs to do URL decode.
• Rewrite any incoming URI path to /${porg_name}/${catalog_name}/${incoming_uri_path}
• Finally, it is important to tune the verify depth parameter you can use the signer certificate to do client certificate verification.

One benefit of this approach is you can isolate TLS-only APIs from the MTLS-only APIs while sharing the same gateway service. You can publish TLS-only APIs to a TLS-only catalog and MTLS-only APIs to a MTLS-only catalog.

Let me know what you think in the comment section.

References

https://www.ibm.com/docs/en/api-connect/10.0.1.x?topic=catalogs-creating-configuring

Vanity URLs in APIConnect v2018 and onwards

Disclaimer:

All opinions expressed here are very much my own and not of IBM. All codes/scripts/artifacts are provided as-is with no support unless otherwise stated.

This is the next installment of the series — to enable TLS communications for the AMQP channel we setup in the previous article. See Part 1 here. In this installment, I am going to show you to scenarios:

• Server authentication where the AMQP client checks the server certificate from the MQ queue manger
• Mutual authentication where in addition to server authentication above; there is client authentication where the MQ queue manager checks the client certificate from the AMQP client.

Begin I begin

1. To simplify this exercise, I have written a script that generate the keys, certificates, keystores and truststores in the format that we need. In order for the script to run, you need to have the following tools:

• openssl — which is found in most Linux OS
• ikeycmd — which is packaged in the JRE of the MQ installation
• keytool — which is packaged in the JRE of the MQ installation.

2. Before you run the script, you should configure the variables.

• ROOT_CN — the label of the root certificate authority cert.
• SERVER_CN — the label of the server certificate.
• CLIENT_CN — the label of the client certificate.
• C — the name of the country, e.g. AU
• ST — the name of the state, e.g. VIC (for Victoria)
• L — the name of the locality or city, e.g. Melbourne
• O— the name of the organization, e.g. KKS
• OU — the name of the organization or business unit, e.g. Cloud
• DNS_1 — the DNS name of the MQ server
• IP_1 — the IP address of the MQ server
• PASS — the password for both the server and client keystore (for simplicity)
• IKEYCMD_PATH — the path of ikeyman, e.g. `/opt/mqm/java/jre64/jre/bin`
• KEYTOOL_PATH — the path of keytool, e.g. `/opt/mqm/java/jre64/jre/bin`

3. The script generates the following that we are going to use. See scripts here (https://github.com/khongks/amqp-on-mq/blob/master/cert/gen_certs.sh)

• mqclient.crt — client certificate
• mqclient.key — client key
• mqclient.p12 — keystore/truststore that contains the client certificate, client key and the root CA certificate in PKCS12 format; this is used by the AMQP client application.
• mqserver.crt — server certificate
• mqserver.key — server key
• mqserver.p12 — keystore/truststore that contains the server certificate, server key and the root CA certificate in PKCS12 format; this is not used.
• mqserver.kdb —key database that stores personal certificates, personal certificate requests, and signer certificates.
• mqserver.rdb — “request” database file that contains certificate requests that are outstanding and have not yet been received from the CA.
• mqserver.sth — “stash” file that stores an obfuscated version of the key database password; also stores private keys, if there are any.
• rootca.crt — the root CA signer certificate
• rootca.key — the root CA key

Step 1: Configure the certificates in the queue manager

1. Inspect the queue manager to find where we should put the key database files. From here, it shows that the key database files should located in the folder /var/mqm/qmgrs/AMQP_SAMPLE_QM/ssl and the file name should be key.kdb.

$ runmqsc AMQP_SAMPLE_QM

: DIS QMGR

SSLKEYR(/var/mqm/qmgrs/AMQP_SAMPLE_QM/ssl/key)

2. Login as user of mqm group and copy the following files into the folder; and changing the file names. Change the file ownership if you need to

cp -p mqserver.kdb /var/mqm/qmgrs/AMQP_SAMPLE_QM/ssl/key.kdb
cp -p mqserver.rdb /var/mqm/qmgrs/AMQP_SAMPLE_QM/ssl/key.rdb
cp -p mqserver.sth /var/mqm/qmgrs/AMQP_SAMPLE_QM/ssl/key.sth

chown mqm:mqm /var/mqm/qmgrs/AMQP_SAMPLE_QM/ssl/key.kdb
chown mqm:mqm /var/mqm/qmgrs/AMQP_SAMPLE_QM/ssl/key.rdb
chown mqm:mqm /var/mqm/qmgrs/AMQP_SAMPLE_QM/ssl/key.sth

Step 2: Server authentication

1. Instead of using the same channel, we are creating a new AMQP channel for TLS communications. Create the following MQSC file AMQP.CHL.TLS.mqsc.

• Defines are channel of type AMQP, with the label mqserver that matches the label we used during the creation of the certificate; that listens on port 5671 and specify client authentication SSLCAUTH as OPTIONAL.
• Reload the SSL configuration into the queue manager memory.
• Start the channel.

DEF CHL(SAMPLE.AMQP.CHL.TLS) CHLTYPE(AMQP) CERTLABL('mqserver') SSLCAUTH(OPTIONAL) PORT(5671) MCAUSER(nobody) SSLCIPH(TLS_RSA_WITH_AES_256_GCM_SHA384)

REFRESH SECURITY TYPE(SSL)
START CHL(SAMPLE.AMQP.CHL.TLS)

2. Run the MQSC file and make sure there is no error.

$ runmqsc AMQP_SAMPLE_QM < SAMPLE.AMQP.CHL.TLS.mqsc

3. Copy the root CA certificate into the ./cert folder.

4. Run the AMQP applications, specifying the following:

• -s URL, --service=URL: amqps://mqserver:5671
• -c FILE, --trust-certificate=FILE: rootca.crt

a. Start Receive application.

java -classpath ./lib/proton-j-0.12.1.jar:./lib/gson-2.2.4.jar:./lib/logback-classic-1.1.2.jar:./lib/logback-core-1.1.2.jar:./lib/netty-all-4.0.21.Final.jar:./lib/slf4j-api-1.7.5.jar:./lib/stateless4j-2.5.0.jar:./mqlight/target/mqlight-api-1.0-SNAPSHOT.jar:./mqlight-samples/target/mqlight-api-samples-1.0-SNAPSHOT.jar com.ibm.mqlight.api.samples.Receive -s amqps://mqserver:5671 -c ./cert/rootca.crt

b. Start Send application to send 10 messages.

java -classpath ./lib/proton-j-0.12.1.jar:./lib/gson-2.2.4.jar:./lib/logback-classic-1.1.2.jar:./lib/logback-core-1.1.2.jar:./lib/netty-all-4.0.21.Final.jar:./lib/slf4j-api-1.7.5.jar:./lib/stateless4j-2.5.0.jar:./mqlight/target/mqlight-api-1.0-SNAPSHOT.jar:./mqlight-samples/target/mqlight-api-samples-1.0-SNAPSHOT.jar com.ibm.mqlight.api.samples.Send -r 10 -s amqps://mqserver:5671 -c ./cert/rootca.crt

Step 3: Mutual authentication

1. Update the AMQP channel to perform client authentication. Update the previous MQSC file AMQP.CHL.TLS.mqsc.

• Update the AMQP channel with client authentication — SSLCAUTH as REQUIRED.
• Reload the SSL configuration into the queue manager memory.
• Stop and start the channel.

ALTER CHL(SAMPLE.AMQP.CHL.TLS) CHLTYPE(AMQP) SSLCAUTH(REQUIRED) 

REFRESH SECURITY TYPE(SSL)
STOP CHL(SAMPLE.AMQP.CHL.TLS)
START CHL(SAMPLE.AMQP.CHL.TLS)

2. Run the MQSC file and make sure there is no error.

$ runmqsc AMQP_SAMPLE_QM < SAMPLE.AMQP.CHL.TLS.MUTUAL.mqsc

3. Copy the client keystore/truststore PKCS12 file into the ./cert folder.

4. Run the AMQP applications, specifying the following:

• -s URL, --service=URL: amqps://mqserver:5671
• -k FILE, --keystore=FILE: mqclient.p12
• -p PASSPHRASE, --keystore-passphrase=PASSPHRASE: Passw0rd!

a. Start Receive application.

java -classpath ./lib/proton-j-0.12.1.jar:./lib/gson-2.2.4.jar:./lib/logback-classic-1.1.2.jar:./lib/logback-core-1.1.2.jar:./lib/netty-all-4.0.21.Final.jar:./lib/slf4j-api-1.7.5.jar:./lib/stateless4j-2.5.0.jar:./mqlight/target/mqlight-api-1.0-SNAPSHOT.jar:./mqlight-samples/target/mqlight-api-samples-1.0-SNAPSHOT.jar com.ibm.mqlight.api.samples.Receive -s amqps://mqserver:5671 -k ./cert/mqclient.p12 -p Passw0rd!

b. Start Send application to send 10 messages.

java -classpath ./lib/proton-j-0.12.1.jar:./lib/gson-2.2.4.jar:./lib/logback-classic-1.1.2.jar:./lib/logback-core-1.1.2.jar:./lib/netty-all-4.0.21.Final.jar:./lib/slf4j-api-1.7.5.jar:./lib/stateless4j-2.5.0.jar:./mqlight/target/mqlight-api-1.0-SNAPSHOT.jar:./mqlight-samples/target/mqlight-api-samples-1.0-SNAPSHOT.jar com.ibm.mqlight.api.samples.Send -r 10 -s amqps://mqserver:5671 -k ./cert/mqclient.p12 -p Passw0rd!

Alternative, instead of the PKCS12 file, you can run the AMQP applications; specify the following:

• -s URL, --service=URL: amqps://mqserver:5671
• --client-certificate=FILE: mqclient.crt
• --client-key=FILE: mqclient.key
• --client-key-passphrase=PASSPHRASE: Passw0rd!
• -c FILE, --trust-certificate=FILE: rootca.crt

a. Start Receive application.

java -classpath ./lib/proton-j-0.12.1.jar:./lib/gson-2.2.4.jar:./lib/logback-classic-1.1.2.jar:./lib/logback-core-1.1.2.jar:./lib/netty-all-4.0.21.Final.jar:./lib/slf4j-api-1.7.5.jar:./lib/stateless4j-2.5.0.jar:./mqlight/target/mqlight-api-1.0-SNAPSHOT.jar:./mqlight-samples/target/mqlight-api-samples-1.0-SNAPSHOT.jar com.ibm.mqlight.api.samples.Receive -s amqps://mqserver:5671 --client-certificate=./cert/mqclient.crt --client-key=./cert/mqclient.key --client-key-passphrase=Passw0rd! --trust-certificate=./cert/rootca.crt

b. Start Send application to send 10 messages.

java -classpath ./lib/proton-j-0.12.1.jar:./lib/gson-2.2.4.jar:./lib/logback-classic-1.1.2.jar:./lib/logback-core-1.1.2.jar:./lib/netty-all-4.0.21.Final.jar:./lib/slf4j-api-1.7.5.jar:./lib/stateless4j-2.5.0.jar:./mqlight/target/mqlight-api-1.0-SNAPSHOT.jar:./mqlight-samples/target/mqlight-api-samples-1.0-SNAPSHOT.jar com.ibm.mqlight.api.samples.Send -r 10 -s amqps://mqserver:5671 --client-certificate=./cert/mqclient.crt --client-key=./cert/mqclient.key --client-key-passphrase=Passw0rd! --trust-certificate=./cert/rootca.crt

Finally

As with any security type of configuration, it tends to be tedious, hard to debug if you encountered issues. If you have a good understanding of Public Key Infrastructure (PKI) and how TLS/SSL works; you can guess the problems. The most challenging hurdle of this task, for me was really to get the keys, certificates, keystores and truststores generated correctly, and in the right format that is required by the queue manager and the AMQP Java application — trying to figure out how to specify the input paramaters. As you noted, the MQ specific configurations are pretty straight forward. How you find this article useful.

You can find my script here.

https://github.com/khongks/amqp-on-mq/blob/master/cert/gen_certs.sh

A couple of days ago, a customer requested for a Proof of Concept to show how to configure IBM MQ to provide messaging service via AMQP. This feature is not new, but I never gotten around to play with it. I spent a couple of hours fiddling and found that it wasn’t rocket science. I wrote this article so that others can benefit from my journey.

As it is middle of a lock down due to COVID-19, I do have some spare time; therefore I decided this to be a 4-part series.

• Part 1 — Getting started on AMQP using IBM MQ
• Part 2 — Enable TLS communications via AMQP
• Part 3 — Create a custom Docker image with AMQP configurations
• Part 4 — Deploy Docker image on Red Hat OpenShift

Before I begin

First of all, let me start by laying out the environment.

• IBM MQ v9.1.0.4
• RedHat v7.7
• OpenJDK v11 to compile and run the sample Java code

Step 1: Prepare the AMQP configuration

1. Ensure that you have included the AMQP bundle when installing IBM MQ. To check if you have installed this bundle, run the following command.

$ rpm -qa --qf "%{NAME}-%{VERSION}-%{RELEASE}\t%{INSTPREFIXES}\n" | grep MQSeriesAMQP

MQSeriesAMQP-9.1.0-4 /opt/mqm

2. Run sample script to create a sample configuration for AMQP. Note that you must run this command using a user that is in the mqmgroup.

$ cd /opt/mqm/amqp/samples
$ ./SampleMQM.sh

This script does the following:

• Creates a queue manager AMQP_SAMPLE_QM.
• Sets authorizations to allow everybody to connect to the queue manager and to publish and subscribe to the base topic SYSTEM.BASE.TOPIC.
• Start the queue manager and the AMQP service SYSTEM.AMQP.SERVICE.
• Defines a AMQP channel with port ranging 1–5672
• Finally starts the channel

# This script creates an AMQP-enabled Queue manager to allow quick testing.

# Get the name of the Queue Manager
QMGR=AMQP_SAMPLE_QM

# Get the Port number
PORT=${1-5672}

# Check the queue manager doesn't exist already
dspmq -m $QMGR >/dev/null 2>&1
rc=$?
if [ $rc -eq 0 ]; then
  echo "Queue manager $QMGR already exists."
  exit 1
fi

crtmqm $QMGR
strmqm $QMGR

# Allow user nobody to publish and subscribe on any topic.
# Note: Allowing the user nobody to do something in MQ, allows all users to do it!
setmqaut -m $QMGR -t qmgr -p nobody -all +connect
setmqaut -m $QMGR -t topic -n SYSTEM.BASE.TOPIC -p nobody -all +pub +sub

# Start the service
echo "START SERVICE(SYSTEM.AMQP.SERVICE)" | runmqsc $QMGR

# Create and start a channel
echo "DEFINE CHANNEL(SAMPLE.AMQP.CHANNEL) CHLTYPE(AMQP) PORT($PORT) MCAUSER('nobody')" | runmqsc $QMGR

sleep 2
echo "START CHANNEL(SAMPLE.AMQP.CHANNEL)" | runmqsc $QMGR

Step 2: Compile and run a sample Java code

1. Clone the java-mqlight repository from Github

$ git clone https://github.com/mqlight/java-mqlight.git

2. Before you try to compile and generate the JAR file, let’s make a couple of modifications in the pom.xml to save you many heartaches.

• Remove the version here — so that we can use the later version. There is some problems with version 3.0.0.

<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<!-- version>3.0.0</version -->

• Remove the plugin maven-javadoc-plugin. This is some issues generating Java documentation

<!--
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-javadoc-plugin</artifactId>
          <version>2.10.1</version>
          <configuration>
            <excludePackageNames>com.ibm.mqlight.api.impl</excludePackageNames>
          </configuration>
          <executions>
            <execution>
              <id>attach-javadocs</id>
              <goals>
                <goal>jar</goal>
              </goals>
            </execution>
          </executions>
        </plugin>
-->

3. Now, you can start compile and generate the target JAR files; but remember to skip the test because there are some issues. All we need is the JAR executable is generated.

$ cd java-mqlight
$ mvn install -Dmaven.test.skip=true

5. On build success, you will see this.

[INFO] Reactor Summary for com.ibm.mqlight:mqlight-project 1.0-SNAPSHOT:
[INFO]
[INFO] com.ibm.mqlight:mqlight-project .................... SUCCESS [  0.347 s]
[INFO] com.ibm.mqlight:mqlight-api ........................ SUCCESS [  8.752 s]
[INFO] com.ibm.mqlight:mqlight-api-samples ................ SUCCESS [  2.572 s]
[INFO] com.ibm.mqlight:mqlight-distribution ............... SUCCESS [  0.487 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  12.726 s
[INFO] Finished at: 2020-04-18T17:47:51+10:00
[INFO] ------------------------------------------------------------------------

6. You will find the following executable JARs generated that you are interested in.

• ./mqlight/target/mqlight-api-1.0-SNAPSHOT.jar
• ./mqlight-samples/target/mqlight-api-samples-1.0-SNAPSHOT.jar

7. There is a error in the packaging; a missing dependencies JAR file — proton-j-0.12.1.jar. Google and download to the lib folder.

8. Now, try to run the Receive application.

java -classpath ./lib/proton-j-0.12.1.jar:./lib/gson-2.2.4.jar:./lib/logback-classic-1.1.2.jar:./lib/logback-core-1.1.2.jar:./lib/netty-all-4.0.21.Final.jar:./lib/slf4j-api-1.7.5.jar:./lib/stateless4j-2.5.0.jar:./mqlight/target/mqlight-api-1.0-SNAPSHOT.jar:./mqlight-samples/target/mqlight-api-samples-1.0-SNAPSHOT.jar com.ibm.mqlight.api.samples.Receive -s amqp://ibmmqserver:5672

It shows it is connected and it is subscribed to the public topic.

Connected to amqp://ibmmqserver:5672 using client-id AUTO_7e2839f
Subscribed to pattern: public

9. Open another terminal window, and run the Send application to send 10 messages.

java -classpath ./lib/proton-j-0.12.1.jar:./lib/gson-2.2.4.jar:./lib/logback-classic-1.1.2.jar:./lib/logback-core-1.1.2.jar:./lib/netty-all-4.0.21.Final.jar:./lib/slf4j-api-1.7.5.jar:./lib/stateless4j-2.5.0.jar:./mqlight/target/mqlight-api-1.0-SNAPSHOT.jar:./mqlight-samples/target/mqlight-api-samples-1.0-SNAPSHOT.jar com.ibm.mqlight.api.samples.Send -s amqp://[ibmmqserver]:5672 -r 10

It shows it is connected and sending 10 messages before exiting

Connected to amqp://ibmmqserver:5672 using client-id AUTO_714422f
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Exiting

10. Now switch back to the terminal window that is running the Receive application. You will see the messages received.

Subscribed to pattern: public
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!

Finally

OK, there you have it, it was not difficult. The most challenging aspect was trying to get the Java sample code to compile. There were Java incompatibility issues and bundle incompatibility issues. And then, there was a packaging issue (maybe should feedback to the author). Nevertheless, I did some workaround and managed to make it work. It was not perfect, but it works!! Maybe, I should try the Python sample code next round?

Disclaimer:

All opinions expressed here are very much my own and not of IBM. All codes/scripts/artifacts are provided as-is with no support unless otherwise stated.