Can't create a connection string secret in a different namespace

[limwa]

What did you do to encounter the bug?

Steps to reproduce the behavior:

1. Create a kind cluster
2. Install MongoDB Kubernetes Operator on the mongodb namespace (set to watch all namespaces in the cluster)
   • helm repo add mongodb https://mongodb.github.io/helm-charts
   • helm install community-operator mongodb/community-operator --namespace mongodb --create-namespace --set operator.watchNamespace="*"
3. Apply this manifest. Version with syntax highlighting available here.
4. Watch the testing namespace for new secrets being created.
   • kubectl get secret -n testing -w --output-watch-events
5. No connection string secrets are created in the testing namespace, even after the MDBC resource reaches the "Running" phase.
6. Read the operator logs. There is a message saying the secret "mongodb-apps-testinguser-credentials" could not be found.

What did you expect?

I expected a connection string secret with the name "testinguser-mongodb-secret" in the "testing" namespace to be created, after the MDBC resource reached the "Running" phase.

What happened instead?

No connection string secret was created in the "testing" namespace, even after the MDBC resource reached the "Running" phase. This only happened for the "testinguser". The "adminuser", which doesn't specify the "connectionStringSecretNamespace" parameter, had its connection string secret created in the mongodb namespace.

Operator Information

• Operator Version: 0.10.0 from quay.io
• MongoDB Image used: 7.0.12 from docker.io

Kubernetes Cluster Information

• Distribution: Kind
• Version: v1.27.3
• Image Registry location: not sure - according to the helm chart, mongodb comes from docker.io, and agent, versionUpgradeHook, readinessProbe and operator come from quay.io

Additional context

If I add the mongodb-apps-testinguser-credentials secret to both the mongodb and the testing namespaces, the testinguser-mongodb-secret secret is created in the testing namespace, however, it is instantly deleted.

If I only create the mongodb-apps-testinguser-credentials secret in the testing namespace, the MDBC resource never reaches the "Running" phase.

Operator Logs: https://gist.github.com/limwa/f5c507000ba48edcae238134b49b0af3/raw/934bcc1016e6a2f935c576a0a7557a5452d7beaf/operator-logs.txt

kubectl get sts -n mongodb -oyaml: https://gist.githubusercontent.com/limwa/f5c507000ba48edcae238134b49b0af3/raw/e712f778aa27e37daa361284e633301853099629/sts-out.yaml

kubectl get pods -n mongodb -oyaml: https://gist.githubusercontent.com/limwa/f5c507000ba48edcae238134b49b0af3/raw/e712f778aa27e37daa361284e633301853099629/pods-out.yaml

Regarding the mongodb-apps-0 pod (all pods seem fine, I just chose this one because it was the first one):

• Logs: https://gist.githubusercontent.com/limwa/f5c507000ba48edcae238134b49b0af3/raw/e712f778aa27e37daa361284e633301853099629/pod-logs.txt
• The agent clusterconfig: https://gist.githubusercontent.com/limwa/f5c507000ba48edcae238134b49b0af3/raw/e712f778aa27e37daa361284e633301853099629/cluster-config.json
• The agent health status: https://gist.githubusercontent.com/limwa/f5c507000ba48edcae238134b49b0af3/raw/e712f778aa27e37daa361284e633301853099629/agent-health-status.json
• The verbose agent logs of the faulty members: https://gist.githubusercontent.com/limwa/f5c507000ba48edcae238134b49b0af3/raw/e712f778aa27e37daa361284e633301853099629/automation-agent-verbose.log

[limwa]

From what I've seen, it appears this could be caused by the following code

mongodb-kubernetes-operator/controllers/mongodb_users.go

Lines 42 to 74 in 3108a94

	func (r ReplicaSetReconciler) updateConnectionStringSecrets(ctx context.Context, mdb mdbv1.MongoDBCommunity, clusterDomain string) error {
	for _, user := range mdb.GetAuthUsers() {
	secretName := user.ConnectionStringSecretName
	secretNamespace := mdb.Namespace
	if user.ConnectionStringSecretNamespace != "" {
	secretNamespace = user.ConnectionStringSecretNamespace
	}
	existingSecret, err := r.client.GetSecret(ctx, types.NamespacedName{
	Name: secretName,
	Namespace: secretNamespace,
	})
	if err != nil && !apiErrors.IsNotFound(err) {
	return err
	}
	if err == nil && !secret.HasOwnerReferences(existingSecret, mdb.GetOwnerReferences()) {
	return fmt.Errorf("connection string secret %s already exists and is not managed by the operator", secretName)
	}
	pwd := ""
	if user.Database != constants.ExternalDB {
	secretNamespacedName := types.NamespacedName{Name: user.PasswordSecretName, Namespace: secretNamespace}
	pwd, err = secret.ReadKey(ctx, r.client, user.PasswordSecretKey, secretNamespacedName)
	if err != nil {
	return err
	}
	}
	connectionStringSecret := secret.Builder().
	SetName(secretName).
	SetNamespace(secretNamespace).

In it:

1. In lines 46-49, secretNamespace is set to the connection string secret namespace, if it is defined
2. In lines 65-69, the password secret is read on the secretNamespace
3. In line 74, the connection string secret is set to be created on the secretNamespace.

Therefore, the password secret and connection string secret need to be in the same namespace.

However, when ensuring user resources, in the following code, it is expected that the password secret is in the namespace of the MDBC, and not in the one specified for the connection string secret.

mongodb-kubernetes-operator/controllers/mongodb_users.go

Lines 16 to 20 in 3108a94

	func (r ReplicaSetReconciler) ensureUserResources(ctx context.Context, mdb mdbv1.MongoDBCommunity) error {
	for _, user := range mdb.GetAuthUsers() {
	if user.Database != constants.ExternalDB {
	secretNamespacedName := types.NamespacedName{Name: user.PasswordSecretName, Namespace: mdb.Namespace}
	if _, err := secret.ReadKey(ctx, r.client, user.PasswordSecretKey, secretNamespacedName); err != nil {

Ensuring consistency in the namespace used in both functions should fix the problem of needing to have the secret in both namespaces for the connection string secret to be created.

However, I'm not sure what is causing the connection string secret to be deleted immediately after it is created (could be related, or not).

[limwa]

An update on this: ensuring consistency on the namespaces used did solve the problem of needing to have the password secret in both namespaces (I can open a PR for this, it's a single line change).

However, the problem with the connection string secret being immediately deleted was not solved. After some investigation, I came to the conclusion that the secret was being garbage collected by kubernetes. https://kubernetes.io/docs/concepts/architecture/garbage-collection/#owners-dependents states that cross-namespace owner references are disallowed by design, which is causing the secret to be immediately deleted.

[limwa]

A possible solution to the connection string secret problem is to follow an approach similar to cert-manager (https://cert-manager.io/docs/devops-tips/syncing-secrets-across-namespaces) and allow for a "connectionStringSecretTemplate" as well. This would allow users to set annotations that can be used by reflector or kubernetes-replicator to sync the secret across namespaces. The connectionStringSecretNamespace parameter doesn't need to be removed because it is a simpler solution for cluster-wide deployments of the mongodb-kubernetes-operator.

[limwa]

Since creating the "connectionStringSecretTemplate" proved to be too complex for me (I haven't ever programmed in Go), I've gone ahead and implemented a "connectionStringSecretAnnotations" property. It is implemented in my fork. Let me know if you'd like me to open a PR for the namespace consistency fix and/or the connectionStringSecretAnnotations property.