Error during user and/or database creation

[jsenzier-oxeva]

Hi, I need to open an issue to understand what is going wrong during a user / database creation on my couchdb cluster

Description

I have an issue on a production cluster (version 3.3.3) of 3 nodes (with per_user activated) deployed on a Kubernetes Cluster

We have ~ 19000 users databases :

Sometimes we have this error which appears during the user creation, I got this error :

Steps to Reproduce

We are using this repository :

- name: couchdb url: https://apache.github.io/couchdb-helm/

Here's my couchdb YAML configuration : https://gist.github.com/jsenzier-oxeva/d286c892e41a76be8846f4945a5dd272

The encountered error is in previous Description section
And the php code which is executed : https://gist.github.com/jsenzier-oxeva/dc9a05cca44ac2610b743cb97b9bb69e

The error appears during this code execution :

User is created, then we execute a loop 5 times with a sleep 2 to wait database creation, but it never happens as we encountered the "security" error
As discussed on Slack (thanks for your help) : it appears the security object is not identical across all shards of your database

But I'm asking why and how can I resolve those errors ?

Expected Behaviour

Not having 500 error, so have a user and database creation working without any error in couchdb logs

• CouchDB version used: 3.3.3
• Operating system and version: Kubernetes 1.28.15 with CouchDB deployment (chart version 4.6.0) from official repository : https://apache.github.io/couchdb-helm/

[janl]

I traced this to an odd assertion in the PUT _security code in peruser, but I had not yet have time to investigate why this exists. I’ll stay on it.

[jsenzier-oxeva]

Thank you Janl for your investigations ;)

[jsenzier-oxeva]

Hi Janl, hope you're good, did you investigate on this security issue ?