How can detect memory leak for services under systemd service?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7.x

Issue

How can detect memory leak for services under systemd service?

Resolution

Side effects : Use of valgrind tool for debugging can be cause addition system load or unexpected service stop to the system.

Use "systemctl edit --full foobar.service" and prefix it's ExecStart= parameter with an invocation of valgrind.
The memory leak for service daemon is determined from the logs file written by the valgrind by adding the valgrind command to the existing service run command on systemd.

Root Cause

If memory leak occurs for services operated by systemd, debug is possible using the debug tool valgrind.

Diagnostic Steps

For example, If you want to check memory leak for irqbalance service with valgrind, please refer to the step as below,
1.To check the irqblance service on the running service from systemd

# systemctl status
● localhost.localdomain
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Tue 2017-07-18 04:34:51 EDT; 18min ago
   CGroup: /
           └─system.slice
...
             ├─irqbalance.service
             │ └─729 /usr/sbin/irqbalance --foreground
....

2.Check whether valgrind is installed on system.

# rpm -qa | grep valgrind

3.If not, use the YUM command to install valgrind package on system.
NOTE: valgrind package include rhel-7-server-rpms repository, you must to register subscription in your system.

# yum install valgrind
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast    
rhel-7-server-extras-rpms                                                                                                                                                                                             | 3.4 kB  00:00:00     
rhel-7-server-rpms                                                                                                                                                                                                    | 3.5 kB  00:00:00     
...
Resolving Dependencies
--> Running transaction check
---> Package valgrind.x86_64 1:3.13.0-10.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved
=====================================================================================
 Package     Arch           Version         Repository              Size
=====================================================================================
Installing:
 valgrind    x86_64     1:3.13.0-10.el7   rhel-7-server-rpms        7.6 M

Transaction Summary
======================================================================================
Install  1 Package

Total download size: 7.6 M
Installed size: 27 M
Is this ok [y/d/N]: y
Downloading packages:
valgrind-3.13.0-10.el7.x86_64.rpm    | 7.6 MB  00:00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:valgrind-3.13.0-10.el7.x86_64  1/1 
    rhel-7-server-rpms/7Server/x86_64/productid  | 2.1 kB  00:00:00     
  Verifying  : 1:valgrind-3.13.0-10.el7.x86_64  1/1 

Installed:
  valgrind.x86_64 1:3.13.0-10.el7

4.When the irqbalance service is running, use the ExecStart parameter to run the debugging tool together.
ExecStart parameter will be executed when this service starts. For more information, see the link page,
systemd.service — Service unit configuration

5.Add line /usr/bin/valgrind --tool=memcheck --leak-check=yes --log-file=/var/log/valgrind/irqbalance.log to the ExecStart parameter.
Confirm that it is set as below,

# systemctl edit --full irqbalance.service

[Unit]
Description=irqbalance daemon
After=syslog.target

[Service]
EnvironmentFile=/etc/sysconfig/irqbalance
ExecStart=/usr/bin/valgrind --tool=memcheck --leak-check=yes --log-file=/var/log/valgrind/irqbalance.log /usr/sbin/irqbalance --foreground $IRQBALANCE_ARGS

[Install]
WantedBy=multi-user.target
~                                                                                                                                                                                                                                           
~

6.Create directories and files for logging

# mkdir /var/log/valgrind/
# touch /var/log/valgrind/irqbalance.log

7.Restart irqbalance service

# systemctl restart irqbalance

8.Check the status of irqbalance service in running

# systemctl status irqbalance
● irqbalance.service - irqbalance daemon
   Loaded: loaded (/etc/systemd/system/irqbalance.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2018-05-13 21:55:09 EDT; 2s ago
 Main PID: 2855 (memcheck-amd64-)
   CGroup: /system.slice/irqbalance.service
           └─2855 /usr/bin/valgrind --tool=memcheck --leak-check=yes --log-file=/var/log/valgrind/irqbalance.log /usr/sbin/irqbalance --foreground

May 13 21:55:09 rhel73 systemd[1]: Started irqbalance daemon.
May 13 21:55:09 rhel73 systemd[1]: Starting irqbalance daemon...

9.Check the log file for valgrind on system

# tail -f /var/log/valgrind/irqbalance.log
==2855== Memcheck, a memory error detector
==2855== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2855== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==2855== Command: /usr/sbin/irqbalance --foreground
==2855== Parent PID: 1
==2855==

For more information, please check the links for valgrind
4. Memcheck: a memory error detector

Additionally, for services other than irqblance service, you can use the valgrind tool for debugging.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments