RabbitMQ users and its permission are deleted after resource restart

Issue

Manually created rabbitmq users and user's permission are deleted after Pacemaker restart

1. login to a controller, list users, and list cluster status.

   [stack@716084-lab4-director ~]$ ssh heat-admin@172.24.216.31
   Last login: Thu Jan 28 18:40:35 2016 from 172.24.216.198
   [heat-admin@overcloud-controller-0 ~]$ sudo su -
   Last login: Thu Jan 28 18:41:14 EST 2016 on pts/0
   [root@overcloud-controller-0 ~]# rabbitmqctl list_users
   Listing users ...
   guest   [administrator]
   ...done.
   
   [root@overcloud-controller-0 ~]# rabbitmqctl cluster_status
   Cluster status of node 'rabbit@overcloud-controller-0' ...
   [{nodes,[{disc,['rabbit@overcloud-controller-0',
                 'rabbit@overcloud-controller-1',
                 'rabbit@overcloud-controller-2']}]},
   {running_nodes,['rabbit@overcloud-controller-2',
                  'rabbit@overcloud-controller-1',
                  'rabbit@overcloud-controller-0']},
   {cluster_name,<<"rabbit@overcloud-controller-0.localdomain">>},
   {partitions,[]}]
   ...done.
   

2. Create user, add tag and permissions.

   [root@overcloud-controller-0 ~]# rabbitmqctl add_user maas_user maas_password
   Creating user "maas_user" ...
   ...done.
   [root@overcloud-controller-0 ~]# rabbitmqctl set_user_tags maas_user monitoring
   Setting tags for user "maas_user" to [monitoring] ...
   ...done.
   [root@overcloud-controller-0 ~]# rabbitmqctl set_permissions -p "/" maas_user ".*" ".*" ".*"
   Setting permissions for user "maas_user" in vhost "/" ...
   ...done.
   [root@overcloud-controller-0 ~]# rabbitmqctl list_users
   Listing users ...
   guest   [administrator]
   maas_user       [monitoring]
   ...done.
   

3. User is listed as well on the other overcloud nodes:

   [stack@716084-lab4-director ~]$ ssh heat-admin@172.24.216.33
   [heat-admin@overcloud-controller-1 ~]$ sudo su -
   [root@overcloud-controller-1 ~]# rabbitmqctl list_users
   Listing users ...
   guest   [administrator]
   maas_user       [monitoring]
   ...done.
   

4. We noticed these files were updated after the creation of the user

   [root@overcloud-controller-0 ~]# ls -l /var/lib/rabbitmq/mnesia/rabbit\@overcloud-controller-0/ | grep user
   -rw-r-----. 1 rabbitmq rabbitmq   273 Jan 29 11:17 rabbit_user.DCD
   -rw-r-----. 1 rabbitmq rabbitmq   300 Jan 29 11:17 rabbit_user_permission.DCD
   [root@overcloud-controller-0 ~]# date
   Fri Jan 29 11:22:55 EST 2016
   

5. Restarting rabbitmq pacemaker resource

   [root@overcloud-controller-0 ~]# pcs resource restart rabbitmq-clone
   rabbitmq-clone successfully restarted
   [root@overcloud-controller-0 ~]# pcs status | grep -A1 rabbitmq
   Clone Set: rabbitmq-clone [rabbitmq]
      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
   

6. Listing rabbitmq users. this time they the one created is no longer there and the user database files were updated.

   [root@overcloud-controller-0 ~]# rabbitmqctl list_users
   Listing users ...
   guest   [administrator]
   ...done.
   [root@overcloud-controller-0 ~]# ls -l /var/lib/rabbitmq/mnesia/rabbit\@overcloud-controller-0/ | grep user
   -rw-r-----. 1 rabbitmq rabbitmq     8 Jan 29 11:25 rabbit_user.DCD
   -rw-r-----. 1 rabbitmq rabbitmq   323 Jan 29 11:25 rabbit_user.DCL
   -rw-r-----. 1 rabbitmq rabbitmq     8 Jan 29 11:25 rabbit_user_permission.DCD
   -rw-r-----. 1 rabbitmq rabbitmq   268 Jan 29 11:25 rabbit_user_permission.DCL
   

7. Same things happen from the other nodes, user is not listed

   [root@overcloud-controller-1 ~]# rabbitmqctl list_users
   Listing users ...
   guest   [administrator]
   ...done.