How can detect memory leak for services under systemd service?
Environment
- Red Hat Enterprise Linux 7.x
Issue
How can detect memory leak for services under systemd service?
Resolution
Side effects : Use of valgrind tool for debugging can be cause addition system load or unexpected service stop to the system.
Use "systemctl edit --full foobar.service" and prefix it's ExecStart= parameter with an invocation of valgrind.
The memory leak for service daemon is determined from the logs file written by the valgrind by adding the valgrind command to the existing service run command on systemd.
Root Cause
If memory leak occurs for services operated by systemd, debug is possible using the debug tool valgrind.
Diagnostic Steps
For example, If you want to check memory leak for irqbalance service with valgrind, please refer to the step as below,
1.To check the irqblance service on the running service from systemd
# systemctl status
● localhost.localdomain
State: running
Jobs: 0 queued
Failed: 0 units
Since: Tue 2017-07-18 04:34:51 EDT; 18min ago
CGroup: /
└─system.slice
...
├─irqbalance.service
│ └─729 /usr/sbin/irqbalance --foreground
....
2.Check whether valgrind is installed on system.
# rpm -qa | grep valgrind
3.If not, use the YUM command to install valgrind package on system.
NOTE: valgrind package include rhel-7-server-rpms repository, you must to register subscription in your system.
# yum install valgrind
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
rhel-7-server-extras-rpms | 3.4 kB 00:00:00
rhel-7-server-rpms | 3.5 kB 00:00:00
...
Resolving Dependencies
--> Running transaction check
---> Package valgrind.x86_64 1:3.13.0-10.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=====================================================================================
Package Arch Version Repository Size
=====================================================================================
Installing:
valgrind x86_64 1:3.13.0-10.el7 rhel-7-server-rpms 7.6 M
Transaction Summary
======================================================================================
Install 1 Package
Total download size: 7.6 M
Installed size: 27 M
Is this ok [y/d/N]: y
Downloading packages:
valgrind-3.13.0-10.el7.x86_64.rpm | 7.6 MB 00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 1:valgrind-3.13.0-10.el7.x86_64 1/1
rhel-7-server-rpms/7Server/x86_64/productid | 2.1 kB 00:00:00
Verifying : 1:valgrind-3.13.0-10.el7.x86_64 1/1
Installed:
valgrind.x86_64 1:3.13.0-10.el7
4.When the irqbalance service is running, use the ExecStart parameter to run the debugging tool together.
ExecStart parameter will be executed when this service starts. For more information, see the link page,
systemd.service — Service unit configuration
5.Add line /usr/bin/valgrind --tool=memcheck --leak-check=yes --log-file=/var/log/valgrind/irqbalance.log to the ExecStart parameter.
Confirm that it is set as below,
# systemctl edit --full irqbalance.service
[Unit]
Description=irqbalance daemon
After=syslog.target
[Service]
EnvironmentFile=/etc/sysconfig/irqbalance
ExecStart=/usr/bin/valgrind --tool=memcheck --leak-check=yes --log-file=/var/log/valgrind/irqbalance.log /usr/sbin/irqbalance --foreground $IRQBALANCE_ARGS
[Install]
WantedBy=multi-user.target
~
~
6.Create directories and files for logging
# mkdir /var/log/valgrind/
# touch /var/log/valgrind/irqbalance.log
7.Restart irqbalance service
# systemctl restart irqbalance
8.Check the status of irqbalance service in running
# systemctl status irqbalance
● irqbalance.service - irqbalance daemon
Loaded: loaded (/etc/systemd/system/irqbalance.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2018-05-13 21:55:09 EDT; 2s ago
Main PID: 2855 (memcheck-amd64-)
CGroup: /system.slice/irqbalance.service
└─2855 /usr/bin/valgrind --tool=memcheck --leak-check=yes --log-file=/var/log/valgrind/irqbalance.log /usr/sbin/irqbalance --foreground
May 13 21:55:09 rhel73 systemd[1]: Started irqbalance daemon.
May 13 21:55:09 rhel73 systemd[1]: Starting irqbalance daemon...
9.Check the log file for valgrind on system
# tail -f /var/log/valgrind/irqbalance.log
==2855== Memcheck, a memory error detector
==2855== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2855== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==2855== Command: /usr/sbin/irqbalance --foreground
==2855== Parent PID: 1
==2855==
For more information, please check the links for valgrind
4. Memcheck: a memory error detector
Additionally, for services other than irqblance service, you can use the valgrind tool for debugging.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments