Ceilometer notification and ceilometer collector consuming large amounts of memory in Red Hat OpenStack Platform

Issue

Ceilometer notification and ceilometer collector consuming large amounts of memory in Red Hat OpenStack Platform. This happens after a service restart as suggested in https://access.redhat.com/solutions/3067161. Both ceilometer collector and ceilometer agent notification consume high amounts of memory.

cat ./controller-0/free
              total        used        free      shared  buff/cache   available
Mem:       65806784    49588960    10734920      406236     5482904    15538416
Swap:       2097148      316332     1780816

cat ./controller-1/free
              total        used        free      shared  buff/cache   available
Mem:       65806784    64267496      377072      410608     1162216      870640
Swap:       2097148     1655300      441848

cat ./controller-2/free
              total        used        free      shared  buff/cache   available
Mem:       65806784    60928356     3972232      398676      906196     4225040
Swap:       2097148     1673556      423592

cat ./controller-0/sos_commands/process/ps_auxwww | sort -k 4 | tail -n5
postfix   2169  0.0  0.0  89192    48 ?        S    Mar27   0:13 qmgr -l -t unix -u
root       613  0.0  0.1 188460 68112 ?        Ss   Mar27   7:25 /usr/lib/systemd/systemd-journald
root     14896 10.1  0.1 404168 70904 pts/0    S+   17:05   0:09 /usr/bin/python /sbin/sosreport --tmp-dir /var/log/ceilometer/
ceilome+ 19688 12.5 74.0 48983536 48720716 ?   Ss   Jun02 509:20 /usr/bin/python2 /usr/bin/ceilometer-collector --logfile /var/log/ceilometer/collector.log
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

cat ./controller-1/sos_commands/process/ps_auxwww | sort -k 4 | tail -n5
root       615  0.0  0.1 200624 71752 ?        Ss   Mar27   8:04 /usr/lib/systemd/systemd-journald
root      6702  8.8  1.2 1207932 821252 pts/0  S+   17:24   0:07 /usr/bin/python /sbin/sosreport --tmp-dir /var/log/ceilometer/
ceilome+ 24246  8.0 41.9 29179400 27582376 ?   Ss   Jun02 363:27 /usr/bin/python2 /usr/bin/ceilometer-agent-notification --logfile /var/log/ceilometer/agent-notification.log
ceilome+ 28509 12.4 53.0 35154884 34892052 ?   Ss   Jun02 509:53 /usr/bin/python2 /usr/bin/ceilometer-collector --logfile /var/log/ceilometer/collector.log
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

cat ./controller-2/sos_commands/process/ps_auxwww | sort -k 4 | tail -n5
root       608  0.0  0.1 160312 80468 ?        Ss   Mar27   7:45 /usr/lib/systemd/systemd-journald
root      3141  7.8  0.1 469704 83136 pts/0    S+   17:29   0:09 /usr/bin/python /sbin/sosreport --tmp-dir /var/log/ceilometer/
ceilome+ 17453  6.6 40.0 27950060 26331416 ?   Ss   Jun02 300:18 /usr/bin/python2 /usr/bin/ceilometer-agent-notification --logfile /var/log/ceilometer/agent-notification.log
ceilome+ 21610 13.3 50.4 33482820 33219952 ?   Ss   Jun02 544:26 /usr/bin/python2 /usr/bin/ceilometer-collector --logfile /var/log/ceilometer/collector.log
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

Meaning that ceilometer processes account for nearly all used memory.

Ceilometer versions:

find . -name '*installed*rpm*' | xargs -I {} grep ceilo {}
openstack-ceilometer-alarm-5.0.5-1.el7ost.noarch            Tue Apr 25 13:35:54 2017
openstack-ceilometer-api-5.0.5-1.el7ost.noarch              Tue Apr 25 13:35:53 2017
openstack-ceilometer-central-5.0.5-1.el7ost.noarch          Tue Apr 25 13:35:50 2017
openstack-ceilometer-collector-5.0.5-1.el7ost.noarch        Tue Apr 25 13:35:53 2017
openstack-ceilometer-common-5.0.5-1.el7ost.noarch           Tue Apr 25 13:34:56 2017
openstack-ceilometer-notification-5.0.5-1.el7ost.noarch     Tue Apr 25 13:34:56 2017
openstack-ceilometer-polling-5.0.5-1.el7ost.noarch          Tue Apr 25 13:35:21 2017
python-ceilometer-5.0.5-1.el7ost.noarch                     Tue Apr 25 13:34:56 2017
python-ceilometerclient-1.5.2-1.el7ost.noarch               Tue Apr 25 13:34:51 2017

Oslo messaging version:

cat installed-rpms | grep oslo-messaging
python-oslo-messaging-2.5.0-11.el7ost.noarch                Tue Apr 25 13:35:01 2017