Skip to content

USHIFT-6748: [QE] Migrate OTP tests into RF: operators #6502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
agullon wants to merge 1 commit into
base: main
Choose a base branch
from
Open

USHIFT-6748: [QE] Migrate OTP tests into RF: operators #6502

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions test/assets/olm/nginx-ok-catalog-source-all.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: catalog-all
namespace: openshift-marketplace
spec:
sourceType: grpc
image: quay.io/olmqe/nginx-ok-index:v1399-fbc-multi
displayName: Test Nginx Operators
publisher: OLM QE
grpcPodConfig:
securityContextConfig: restricted
12 changes: 12 additions & 0 deletions test/assets/olm/nginx-ok-catalog-source-single.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: nginx-ok-catalog
namespace: olm-microshift-single
spec:
sourceType: grpc
image: quay.io/olmqe/nginx-ok-index:v1399-fbc-multi
displayName: Test Nginx Operators
publisher: OLM QE
grpcPodConfig:
securityContextConfig: restricted
11 changes: 11 additions & 0 deletions test/assets/olm/nginx-ok1-subscription.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: nginx-ok1-1399
namespace: olm-microshift-single
spec:
channel: alpha
installPlanApproval: Automatic
name: nginx-ok1-1399
source: nginx-ok-catalog
sourceNamespace: olm-microshift-single
11 changes: 11 additions & 0 deletions test/assets/olm/nginx-ok2-subscription.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: nginx-ok2-1399
namespace: openshift-operators
spec:
channel: alpha
installPlanApproval: Automatic
name: nginx-ok2-1399
source: catalog-all
sourceNamespace: openshift-marketplace
5 changes: 5 additions & 0 deletions test/assets/olm/og-all.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: og-all
namespace: openshift-operators
8 changes: 8 additions & 0 deletions test/assets/olm/og-single.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: og-singlenamespace
namespace: olm-microshift-single
spec:
targetNamespaces:
- olm-microshift-single
246 changes: 239 additions & 7 deletions test/suites/optional/olm.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,24 @@ ${SUBSCRIPTION} ./assets/olm/subscription.yaml
${SUBSCRIPTION_NAME} amq-broker
${MARKETPLACE_NAMESPACE} openshift-marketplace
${OPERATORS_NAMESPACE} openshift-operators
${OLM_NAMESPACE} openshift-operator-lifecycle-manager

# Single namespace install mode
${SINGLE_NS} olm-microshift-single
${SINGLE_OG} ./assets/olm/og-single.yaml
${SINGLE_CATALOG} ./assets/olm/nginx-ok-catalog-source-single.yaml
${SINGLE_CATALOG_NAME} nginx-ok-catalog
${SINGLE_SUB} ./assets/olm/nginx-ok1-subscription.yaml
${SINGLE_SUB_NAME} nginx-ok1-1399
${SINGLE_PKG} nginx-ok1-1399

# All namespaces install mode with OperatorGroup conflict
${ALL_OG} ./assets/olm/og-all.yaml
${ALL_OG_NAME} og-all
${ALL_CATALOG} ./assets/olm/nginx-ok-catalog-source-all.yaml
${ALL_CATALOG_NAME} catalog-all
${ALL_SUB} ./assets/olm/nginx-ok2-subscription.yaml
${ALL_SUB_NAME} nginx-ok2-1399


*** Test Cases ***
Expand All @@ -36,7 +54,85 @@ Deploy AmqBroker From Red Hat Operators catalog
... AND
... Delete CSV ${OPERATORS_NAMESPACE} ${csv}
... AND
... Wait For Deployments Deletion @{deployments}
... Wait For Deployments Deletion ${OPERATORS_NAMESPACE} @{deployments}

Install Operator In Single Namespace Mode
[Documentation] Creates a dedicated namespace with a SingleNamespace OperatorGroup
... targeting the dedicated namespace (${SINGLE_NS}), installs nginx-ok1-1399 operator,
... verifying successful CSV installation and expected operator resources.
... Migrated from openshift-tests-private 69867.
[Setup] OLM Should Be Ready

Create Namespace ${SINGLE_NS}
Oc Create -f ${SINGLE_OG}
Oc Create -f ${SINGLE_CATALOG}
Wait Until Keyword Succeeds 10m 15s
... CatalogSource Should Be Ready ${SINGLE_NS} ${SINGLE_CATALOG_NAME}
Oc Create -f ${SINGLE_SUB}
Wait Until Keyword Succeeds 10m 15s
... Subscription Should Be AtLatestKnown ${SINGLE_NS} ${SINGLE_SUB_NAME}
${csv}= Get CSV Name From Subscription ${SINGLE_NS} ${SINGLE_SUB_NAME}
Wait For CSV ${SINGLE_NS} ${csv}
Operator Should Have Expected Resources ${SINGLE_PKG} ${SINGLE_NS}

[Teardown] Single Namespace Test Teardown

Install Operator In All Namespaces Mode With OperatorGroup Conflict
[Documentation] Creates a second AllNamespaces OperatorGroup in openshift-operators
... alongside the existing global-operators OG, installs nginx-ok2-1399 and verifies
... the MultipleOperatorGroupsFound error blocks installation. Resolves the conflict by
... deleting the extra OG and verifies the CSV installs successfully and is copied into
... the default namespace, confirming AllNamespaces mode propagation.
... Migrated from openshift-tests-private 69868.
[Setup] OLM Should Be Ready

VAR ${csv}= ${EMPTY}
Comment thread
agullon marked this conversation as resolved.
Oc Get operatorgroup ${OPERATORS_NAMESPACE} global-operators
Oc Create -f ${ALL_OG}
Oc Create -f ${ALL_CATALOG}
Wait Until Keyword Succeeds 10m 15s
... CatalogSource Should Be Ready ${MARKETPLACE_NAMESPACE} ${ALL_CATALOG_NAME}
Oc Create -f ${ALL_SUB}
Wait Until Keyword Succeeds 10m 15s
... OperatorGroup Should Have MultipleOperatorGroupsFound ${OPERATORS_NAMESPACE} ${ALL_OG_NAME}
Wait Until Keyword Succeeds 2m 10s
... Subscription Should Have Empty Installed CSV ${OPERATORS_NAMESPACE} ${ALL_SUB_NAME}
Oc Delete operatorgroup ${ALL_OG_NAME} -n ${OPERATORS_NAMESPACE}
${csv}= Wait For Installed CSV ${OPERATORS_NAMESPACE} ${ALL_SUB_NAME}
Wait Until Keyword Succeeds 2m 10s
... CSV Should Exist In Namespace ${csv} default
Comment thread
agullon marked this conversation as resolved.

[Teardown] All Namespaces Test Teardown ${csv}

OLM Network Policies Are Correctly Configured
[Documentation] Verifies that OLM-managed NetworkPolicies exist with correct pod
... selectors, policy types, and key ingress/egress port rules.
... Migrated from openshift-tests-private 83581.
[Setup] OLM Should Be Ready

# catalog-operator: metrics ingress; API server, DNS, and gRPC (50051) egress; pod-scoped
Verify NetworkPolicy Pod Selector Label catalog-operator ${OLM_NAMESPACE} app catalog-operator
Verify NetworkPolicy Policy Types catalog-operator ${OLM_NAMESPACE}
Verify NetworkPolicy Spec Field catalog-operator ${OLM_NAMESPACE} ingress metrics
Verify NetworkPolicy Spec Field catalog-operator ${OLM_NAMESPACE} egress 50051

# default-deny-all-traffic: no ingress/egress rules, applies to all pods in OLM namespace
Verify NetworkPolicy Has Empty Pod Selector default-deny-all-traffic ${OLM_NAMESPACE}
Verify NetworkPolicy Policy Types default-deny-all-traffic ${OLM_NAMESPACE}
Verify NetworkPolicy Spec Field default-deny-all-traffic ${OLM_NAMESPACE} ingress ${EMPTY}
Verify NetworkPolicy Spec Field default-deny-all-traffic ${OLM_NAMESPACE} egress ${EMPTY}

# olm-operator: metrics ingress; API server and DNS egress; pod-scoped
Verify NetworkPolicy Pod Selector Label olm-operator ${OLM_NAMESPACE} app olm-operator
Verify NetworkPolicy Policy Types olm-operator ${OLM_NAMESPACE}
Verify NetworkPolicy Spec Field olm-operator ${OLM_NAMESPACE} ingress metrics
Verify NetworkPolicy Spec Field olm-operator ${OLM_NAMESPACE} egress 53

# default-allow-all: both Ingress and Egress defined with no port restrictions in openshift-operators
Verify NetworkPolicy Has Empty Pod Selector default-allow-all ${OPERATORS_NAMESPACE}
Verify NetworkPolicy Policy Types default-allow-all ${OPERATORS_NAMESPACE}
Verify NetworkPolicy Spec Field default-allow-all ${OPERATORS_NAMESPACE} ingress
Verify NetworkPolicy Spec Field default-allow-all ${OPERATORS_NAMESPACE} egress


*** Keywords ***
Expand Down Expand Up @@ -95,7 +191,7 @@ CatalogSource Should Be Ready
TRY
Should Be Equal As Strings READY ${catalog.status.connectionState.lastObservedState}
EXCEPT
Run With Kubeconfig oc get events -n openshift-marketplace --sort-by='.lastTimestamp'
Run With Kubeconfig oc get events -n ${namespace} --sort-by='.lastTimestamp'
Fail Catalog Source Is Not Ready
END

Expand All @@ -106,15 +202,17 @@ Create Subscription
... Subscription Should Be AtLatestKnown ${OPERATORS_NAMESPACE} ${SUBSCRIPTION_NAME}

Subscription Should Be AtLatestKnown
[Documentation] Checks if subscription has state "AtLeastKnown"
[Documentation] Checks if subscription has state "AtLatestKnown"
[Arguments] ${namespace} ${name}
${sub}= Oc Get subscriptions.operators.coreos.com ${namespace} ${name}
Should Be Equal As Strings AtLatestKnown ${sub.status.state}

Get CSV Name From Subscription
[Documentation] Obtains Subscription's CSV name.
[Arguments] ${namespace} ${name} # robocop: off=unused-argument
${sub}= Oc Get subscriptions.operators.coreos.com ${OPERATORS_NAMESPACE} ${SUBSCRIPTION_NAME}
[Arguments] ${namespace} ${name}
${sub}= Oc Get subscriptions.operators.coreos.com ${namespace} ${name}
Should Not Be Empty ${sub.status.currentCSV}
... msg=Subscription ${name} in ${namespace} has no currentCSV set yet
RETURN ${sub.status.currentCSV}

Wait For CSV
Expand Down Expand Up @@ -165,6 +263,140 @@ Wait For Deployments Deletion
[Documentation] Wait for Deployments to be deleted.
[Arguments] ${namespace} @{deployments}
FOR ${deploy} IN @{deployments}
Run With Kubeconfig
... oc wait deployment --for=delete -n ${namespace} ${deploy} --timeout=${DEFAULT_WAIT_TIMEOUT}
Oc Wait -n ${namespace} deployment/${deploy} --for=delete --timeout=${DEFAULT_WAIT_TIMEOUT}
END

Operator Should Have Expected Resources
[Documentation] Verifies that the operators.operators.coreos.com resource for a given
... package and namespace contains expected resource type references in its status.
... The Operator resource name follows the convention <package>.<namespace>.
[Arguments] ${package} ${namespace}
${status}= Oc Get JsonPath
... operators.operators.coreos.com
... ${EMPTY}
... ${package}.${namespace}
... .status
Should Contain ${status} ClusterRole
Should Contain ${status} ClusterRoleBinding
Should Contain ${status} ClusterServiceVersion
Should Contain ${status} CustomResourceDefinition
Should Contain ${status} Deployment
Should Contain ${status} OperatorCondition
Should Contain ${status} Subscription

Single Namespace Test Teardown
[Documentation] Cleanup resources created by the single namespace install test.
... Uses Run Keyword And Continue On Failure so all steps run even if one errors.
Run Keyword And Continue On Failure
... Oc Delete subscription ${SINGLE_SUB_NAME} -n ${SINGLE_NS} --ignore-not-found
Run Keyword And Continue On Failure
... Oc Delete csv --all -n ${SINGLE_NS} --ignore-not-found
Run Keyword And Continue On Failure
... Oc Delete catalogsource ${SINGLE_CATALOG_NAME} -n ${SINGLE_NS} --ignore-not-found
Run Keyword And Continue On Failure
... Oc Delete namespace ${SINGLE_NS} --ignore-not-found

OperatorGroup Should Have MultipleOperatorGroupsFound
[Documentation] Checks that the OperatorGroup status contains the MultipleOperatorGroupsFound condition.
[Arguments] ${namespace} ${name}
${status}= Oc Get JsonPath operatorgroup ${namespace} ${name} .status
Should Contain ${status} MultipleOperatorGroupsFound

Subscription Should Have Empty Installed CSV
[Documentation] Verifies that the subscription's installedCSV is empty, indicating no CSV
... has been installed (e.g. due to OperatorGroup conflict). Call this only after confirming
... the OperatorGroup has MultipleOperatorGroupsFound, which guarantees OLM has reconciled
... the conflict before the subscription state is checked.
[Arguments] ${namespace} ${name}
${installed_csv}= Oc Get JsonPath
... subscriptions.operators.coreos.com
... ${namespace}
... ${name}
... .status.installedCSV
Should Be Empty ${installed_csv}

Wait For Installed CSV
[Documentation] Polls the subscription until installedCSV is set, then returns it.
[Arguments] ${namespace} ${name} ${timeout}=10m
${csv}= Wait Until Keyword Succeeds ${timeout} 15s
... Subscription Should Have Installed CSV ${namespace} ${name}
RETURN ${csv}

Subscription Should Have Installed CSV
[Documentation] Fails if the subscription's installedCSV field is empty, returns the CSV name.
[Arguments] ${namespace} ${name}
${csv}= Oc Get JsonPath
... subscriptions.operators.coreos.com
... ${namespace}
... ${name}
... .status.installedCSV
Should Not Be Empty ${csv}
RETURN ${csv}

CSV Should Exist In Namespace
[Documentation] Verifies that a CSV with the given name exists in the namespace.
[Arguments] ${csv} ${namespace}
Oc Get clusterserviceversion.operators.coreos.com ${namespace} ${csv}

All Namespaces Test Teardown
[Documentation] Cleanup resources created by the all namespaces install test.
... When ${csv} is empty (test failed before CSV was recorded), falls back to
... bulk CSV cleanup to avoid orphaned resources. Uses Run Keyword And Continue On Failure
... so all steps run even if one errors.
[Arguments] ${csv}=${EMPTY}
Run Keyword And Continue On Failure
... Oc Delete operatorgroup ${ALL_OG_NAME} -n ${OPERATORS_NAMESPACE} --ignore-not-found
Run Keyword And Continue On Failure
... Oc Delete subscription ${ALL_SUB_NAME} -n ${OPERATORS_NAMESPACE} --ignore-not-found
IF "${csv}" != "${EMPTY}"
Run Keyword And Continue On Failure
... Oc Delete csv ${csv} -n ${OPERATORS_NAMESPACE} --ignore-not-found
ELSE
Log csv not recorded; attempting bulk CSV cleanup to avoid orphaned resources WARN
Run Keyword And Continue On Failure
... Oc Delete csv --all -n ${OPERATORS_NAMESPACE} --ignore-not-found
Comment thread
agullon marked this conversation as resolved.
END
Run Keyword And Continue On Failure
... Oc Delete catalogsource ${ALL_CATALOG_NAME} -n ${MARKETPLACE_NAMESPACE} --ignore-not-found

Verify NetworkPolicy Pod Selector Label
[Documentation] Verifies that a NetworkPolicy's podSelector has the expected label key=value.
[Arguments] ${name} ${namespace} ${label_key} ${expected_value}
${actual}= Oc Get JsonPath
... networkpolicy
... ${namespace}
... ${name}
... .spec.podSelector.matchLabels.${label_key}
Should Be Equal ${actual} ${expected_value}

Verify NetworkPolicy Has Empty Pod Selector
[Documentation] Verifies that a NetworkPolicy's podSelector has no matchLabels
... (i.e. applies to all pods in the namespace).
[Arguments] ${name} ${namespace}
${labels}= Oc Get JsonPath
... networkpolicy
... ${namespace}
... ${name}
... .spec.podSelector.matchLabels
Should Be Empty ${labels}

Verify NetworkPolicy Policy Types
[Documentation] Verifies that a NetworkPolicy has both Ingress and Egress policy types.
[Arguments] ${name} ${namespace}
Verify NetworkPolicy Spec Field ${name} ${namespace} policyTypes Ingress
Verify NetworkPolicy Spec Field ${name} ${namespace} policyTypes Egress

Verify NetworkPolicy Spec Field
[Documentation] Gets .spec.${field} from a NetworkPolicy and asserts based on ${expected}:
... - not provided (defaults to ${NONE}): asserts the field value is not empty
... - ${EMPTY}: asserts the field value is empty (deny-all / no rules check)
... - any other string: asserts the field value contains that string (port or type check)
[Arguments] ${name} ${namespace} ${field} ${expected}=${NONE}
${value}= Oc Get JsonPath networkpolicy ${namespace} ${name} .spec.${field}
IF $expected is None
Should Not Be Empty ${value}
ELSE IF $expected == ''
Should Be Empty ${value}
ELSE
Should Contain ${value} ${expected}
END