Update SSL Certificate

The process involves updating certificates in Azure Key Vault, accessing ArgoCD for deployment management, and ensuring proper synchronization of secrets across the cluster.

The Elqano application uses:

Azure Key Vault for secure storage of certificates and secrets
Kubernetes Secret Store CSI Driver for syncing secrets from Key Vault
ArgoCD for GitOps-based deployment management
Azure Application Gateway for ingress and SSL termination

Prerequisites

Before starting the SSL certificate refresh process, ensure you have:

Required Tools:

az (Azure CLI) - version 2.0 or later
kubectl - configured to access the Elqano cluster
Web browser for ArgoCD access
SSH access or port forwarding capabilities

Required Permissions:

Azure Key Vault:
- Certificate import permissions (certificates/import)
- Secret read permissions (secrets/get, secrets/list)
Kubernetes Cluster:
- Access to the elqano namespace
- Pod execution permissions for ArgoCD access
ArgoCD:
- Application sync permissions
- Repository access (if applicable)

Account Access:

Azure subscription with access to the Elqano resource group
Kubernetes cluster credentials configured locally
ArgoCD admin credentials (stored in argocd-initial-admin-secret)

Certificate Requirements:

Valid SSL certificate (not self-signed) in PFX/PEM format
Certificate must match the application’s FQDN
Certificate should include the full certificate chain

Accessing ArgoCD Web Interface

Ensure Kubectl Context

Verify you’re connected to the correct Kubernetes cluster:

# Check current context
kubectl config current-context

# List available contexts
kubectl config get-contexts

# Switch to Elqano cluster if needed
kubectl config use-context <elqano-cluster-context>

Verify ArgoCD Installation

Check that ArgoCD is running in the cluster:

# Check ArgoCD namespace
kubectl get pods -n argocd

# Verify ArgoCD server is running
kubectl get svc -n argocd argocd-server

Expected output should show ArgoCD pods in Running status.

Set Up Port Forwarding

Create a secure tunnel to access the ArgoCD web interface:

# Forward ArgoCD server port to localhost
kubectl port-forward svc/argocd-server -n argocd 8080:443

Keep this terminal session open. The ArgoCD interface will be accessible at https://localhost:8080.

Retrieve ArgoCD Admin Password

Get the initial admin password:

# Get the admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d && echo

Important: Save this password securely and consider deleting the secret after logging in:

kubectl -n argocd delete secret argocd-initial-admin-secret

Access ArgoCD Web Interface

Open your web browser and navigate to https://localhost:8080
Accept the self-signed certificate warning (this is normal for port-forwarded connections)
Login with:
- Username: admin
- Password: The password retrieved in the previous step

SSL Certificate Update Process

Prepare the New SSL Certificate

Ensure your new SSL certificate is in the correct format:

# If you have separate files, combine them (if needed)
cat your-certificate.crt intermediate.crt root.crt > fullchain.crt

# Convert to PFX format if needed (optional)
openssl pkcs12 -export -out certificate.pfx -inkey private.key -in fullchain.crt

Upload Certificate to Azure Key Vault

Upload the new certificate to your Azure Key Vault:

# Set variables
KEYVAULT_NAME="<your-keyvault-name>"
CERT_NAME="<your-cert-name>"  # Same name as defined in values.yml
CERT_FILE="certificate.pfx"   # or certificate.pem

# Upload the certificate
az keyvault certificate import \
  --vault-name $KEYVAULT_NAME \
  --name $CERT_NAME \
  --file $CERT_FILE

Verify Certificate Upload

Confirm the certificate was uploaded successfully:

# List certificates in Key Vault
az keyvault certificate list --vault-name $KEYVAULT_NAME

# Get specific certificate details
az keyvault certificate show --vault-name $KEYVAULT_NAME --name $CERT_NAME

Refreshing Application Secrets

The Elqano application automatically syncs secrets from Azure Key Vault using the Secret Store CSI Driver. However, you may need to trigger a refresh manually.

Understanding the Secret Sync Process

The application uses a SecretProviderClass that defines:

TLS Certificate: Synced as secret-tls (type: kubernetes.io/tls)
Application Secrets: Synced as app-secret (type: Opaque)

Key secrets include:

ELASTIC_SEARCH_KEY
POSTGRES_URL
SECRET_KEY_BASE
SECURE_ATTRIBUTE_KEY
AZURE_STORAGE_ACCESS_KEY
Docker registry credentials

Check Current Secret Status

Verify the current state of secrets in the cluster:

# Switch to elqano namespace
kubectl config set-context --current --namespace=elqano

# Check secret provider class
kubectl get secretproviderclass

# Check synced secrets
kubectl get secrets

# Examine the TLS secret
kubectl describe secret secret-tls

# Check application secrets
kubectl describe secret app-secret

Force Secret Refresh

If secrets are not automatically updating, you can force a refresh:

Method 1: Restart Secret Store CSI Driver Pods

# Restart the CSI driver pods to force re-sync
kubectl delete pods -n kube-system -l app=secrets-store-csi-driver

# Wait for pods to restart
kubectl get pods -n kube-system -l app=secrets-store-csi-driver

Method 2: Restart Application Pods

# Restart application deployments to remount secrets
kubectl rollout restart deployment elqanoqa
kubectl rollout restart deployment elqanoqa-clock
kubectl rollout restart deployment elqanoqa-worker
kubectl rollout restart deployment nlp

# Check rollout status
kubectl rollout status deployment elqanoqa

Method 3: Delete and Recreate Secrets

# Delete existing secrets (they will be recreated automatically)
kubectl delete secret secret-tls app-secret

# Wait a few moments for automatic recreation
sleep 30

# Verify secrets are recreated
kubectl get secrets

ArgoCD Synchronization

Access Your Application in ArgoCD

In the ArgoCD web interface, locate your Elqano application
Click on the application tile to view details
Check the current sync status and health

Manual Synchronization

If automatic sync is disabled or you want to force an immediate sync:

Via Web Interface:
- Click the “SYNC” button
- Select sync options:
  - ☑️ PRUNE: Remove resources not in Git
  - ☑️ DRY RUN: Preview changes (optional)
  - ☐ FORCE: Force sync even if no changes detected
- Click “SYNCHRONIZE”

Via CLI (if ArgoCD CLI is installed):

# Login to ArgoCD CLI
argocd login localhost:8080 --username admin --password <admin-password> --insecure

# Sync application
argocd app sync elqano

# Force sync with prune
argocd app sync elqano --prune --force

Monitor Sync Progress

Watch the sync progress in the ArgoCD interface

Monitor pod status during the update:

# Watch pod status
kubectl get pods -w

# Check events for any issues
kubectl get events --sort-by=.metadata.creationTimestamp

Handle Sync Conflicts

If you encounter sync conflicts:

Out of Sync Resources:
- Review the differences shown in ArgoCD
- Use “HARD REFRESH” to reload the desired state
- Apply manual sync with appropriate options
Failed Health Checks:
- Check pod logs: kubectl logs <pod-name>
- Verify secret mounting: kubectl describe pod <pod-name>
- Check ingress status: kubectl describe ingress elqanoqa

Verification Steps

Verify Certificate Deployment

# Check TLS secret content
kubectl get secret secret-tls -o yaml

# Verify certificate expiration
kubectl get secret secret-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate

# Check ingress TLS configuration
kubectl describe ingress elqanoqa

Verify Application Health

# Check pod status
kubectl get pods

# Verify deployments are ready
kubectl get deployments

# Check application logs
kubectl logs -l app=elqanoqa --tail=50

# Test internal connectivity
kubectl exec -it deployment/elqanoqa -- curl -k https://localhost:3000/health

Verify External Access

# Get ingress external IP
kubectl get ingress elqanoqa

# Test external HTTPS access
curl -I https://<your-fqdn>

# Verify certificate via browser
# Navigate to https://<your-fqdn> and check certificate details

Verify Secret Synchronization

# Check secret provider class status
kubectl describe secretproviderclass secret-provider

# Verify all application secrets are present
kubectl get secret app-secret -o yaml | grep -E "ELASTIC_SEARCH_KEY|POSTGRES_URL|SECRET_KEY_BASE"

# Test database connectivity (if applicable)
kubectl exec -it deployment/elqanoqa -- rails runner "puts ActiveRecord::Base.connection.execute('SELECT 1').first"

Troubleshooting

Common Issues and Solutions

1. ArgoCD Access Issues

Problem: Cannot access ArgoCD web interface Solutions:

# Check ArgoCD pods status
kubectl get pods -n argocd

# Restart ArgoCD server if needed
kubectl rollout restart deployment argocd-server -n argocd

# Verify port forwarding
lsof -i :8080

2. Certificate Not Updating

Problem: New certificate not reflected in Kubernetes Solutions:

# Verify Key Vault access permissions
az keyvault show --name $KEYVAULT_NAME --query "properties.accessPolicies"

# Check CSI driver logs
kubectl logs -n kube-system -l app=secrets-store-csi-driver

# Force secret recreation
kubectl delete secret secret-tls
kubectl rollout restart deployment elqanoqa

3. Application Gateway SSL Issues

Problem: Application Gateway not using new certificate Solutions:

# Check ingress annotations
kubectl describe ingress elqanoqa

# Patch ingress to force refresh
kubectl patch ingress elqanoqa -p '{"metadata":{"labels":{"refresh":"'$(date +%s)'"}}}'

# Verify Application Gateway backend health
az network application-gateway show-backend-health --name <APP_GW_NAME> --resource-group <RESOURCE_GROUP>

4. Secret Sync Failures

Problem: Secrets not syncing from Key Vault Solutions:

# Check secret provider class events
kubectl describe secretproviderclass secret-provider

# Verify managed identity permissions
az role assignment list --assignee <MANAGED_IDENTITY_CLIENT_ID>

# Check Key Vault access policies
az keyvault show --name $KEYVAULT_NAME --query "properties.accessPolicies[?objectId=='<OBJECT_ID>']"

5. ArgoCD Sync Failures

Problem: ArgoCD cannot sync the application Solutions:

# Check application status in ArgoCD
argocd app get elqano

# Refresh application configuration
argocd app hard-refresh elqano

# Check for resource conflicts
kubectl get events --field-selector type=Warning

6. Pod Startup Issues

Problem: Pods failing to start after secret refresh Solutions:

# Check pod logs
kubectl logs <pod-name> --previous

# Verify secret mounting
kubectl describe pod <pod-name>

# Check resource constraints
kubectl describe node

# Verify image pull secrets
kubectl get secrets regcred -o yaml

Emergency Procedures

Rollback Certificate

If the new certificate causes issues:

# Upload previous certificate to Key Vault
az keyvault certificate import --vault-name $KEYVAULT_NAME --name $CERT_NAME --file old-certificate.pfx

# Force immediate sync
kubectl delete secret secret-tls
kubectl rollout restart deployment elqanoqa

Rollback Application

If ArgoCD sync causes application issues:

# Rollback to previous deployment
kubectl rollout undo deployment elqanoqa

# Check rollout status
kubectl rollout status deployment elqanoqa

# Or use ArgoCD history rollback
argocd app rollback elqano <revision-id>

Quick Reference

Essential Commands

# Access ArgoCD
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Get ArgoCD password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

# Force secret refresh
kubectl delete secret secret-tls app-secret
kubectl rollout restart deployment elqanoqa

# Check certificate expiration
kubectl get secret secret-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate

# Monitor deployment
kubectl get pods -w
kubectl logs -f deployment/elqanoqa

Kubernetes Cluster