Chapter 5. Troubleshooting problems related to SELinux
If you plan to enable SELinux on systems where it has been previously disabled or if you run a service in a non-standard configuration, you might need to troubleshoot situations potentially blocked by SELinux. Note that in most cases, SELinux denials are signs of misconfiguration.
5.1. Identifying SELinux denials
Follow only the necessary steps from this procedure; in most cases, you need to perform just step 1.
Procedure
When your scenario is blocked by SELinux, the
/var/log/audit/audit.logfile is the first place to check for more information about a denial. To query Audit logs, use theausearchtool. Because the SELinux decisions, such as allowing or disallowing access, are cached and this cache is known as the Access Vector Cache (AVC), use theAVCandUSER_AVCvalues for the message type parameter, for example:# ausearch -m AVC,USER_AVC,SELINUX_ERR,USER_SELINUX_ERR -ts recentIf there are no matches, check if the Audit daemon is running. If it does not, repeat the denied scenario after you start
auditdand check the Audit log again.In case
auditdis running, but there are no matches in the output ofausearch, check messages provided by thesystemdJournal:# journalctl -t setroubleshootIf SELinux is active and the Audit daemon is not running on your system, then search for certain SELinux messages in the output of the
dmesgcommand:# dmesg | grep -i -e type=1300 -e type=1400Even after the previous three checks, it is still possible that you have not found anything. In this case, AVC denials can be silenced because of
dontauditrules.To temporarily disable
dontauditrules, allowing all denials to be logged:# semodule -DBAfter re-running your denied scenario and finding denial messages using the previous steps, the following command enables
dontauditrules in the policy again:# semodule -BIf you apply all four previous steps, and the problem still remains unidentified, consider if SELinux really blocks your scenario:
Switch to permissive mode:
# setenforce 0 $ getenforce Permissive
- Repeat your scenario.
If the problem still occurs, something different than SELinux is blocking your scenario.
5.2. Analyzing SELinux denial messages
After identifying that SELinux is blocking your scenario, you might need to analyze the root cause before you choose a fix.
Prerequisites
-
The
policycoreutils-python-utilsandsetroubleshoot-serverpackages are installed on your system.
Procedure
List more details about a logged denial using the
sealertcommand, for example:$ sealert -l "*" SELinux is preventing /usr/bin/passwd from write access on the file /root/test. ***** Plugin leaks (86.2 confidence) suggests ***************************** If you want to ignore passwd trying to write access the test file, because you believe it should not need this access. Then you should report this as a bug. You can generate a local policy module to dontaudit this access. Do # ausearch -x /usr/bin/passwd --raw | audit2allow -D -M my-passwd # semodule -X 300 -i my-passwd.pp ***** Plugin catchall (14.7 confidence) suggests ************************** ... Raw Audit Messages type=AVC msg=audit(1553609555.619:127): avc: denied { write } for pid=4097 comm="passwd" path="/root/test" dev="dm-0" ino=17142697 scontext=unconfined_u:unconfined_r:passwd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=0 ... Hash: passwd,passwd_t,admin_home_t,file,writeIf the output obtained in the previous step does not contain clear suggestions:
Enable full-path auditing to see full paths to accessed objects and to make additional Linux Audit event fields visible:
# auditctl -w /etc/shadow -p w -k shadow-writeClear the
setroubleshootcache:# rm -f /var/lib/setroubleshoot/setroubleshoot.xml- Reproduce the problem.
Repeat step 1.
After you finish the process, disable full-path auditing:
# auditctl -W /etc/shadow -p w -k shadow-write
-
If
sealertreturns onlycatchallsuggestions or suggests adding a new rule using theaudit2allowtool, match your problem with examples listed and explained in SELinux denials in the Audit log.
Additional resources
-
sealert(8)man page
5.3. Fixing analyzed SELinux denials
In most cases, suggestions provided by the sealert tool give you the right guidance about how to fix problems related to the SELinux policy. See Analyzing SELinux denial messages for information how to use sealert to analyze SELinux denials.
Be careful when the tool suggests using the audit2allow tool for configuration changes. You should not use audit2allow to generate a local policy module as your first option when you see an SELinux denial. Troubleshooting should start with a check if there is a labeling problem. The second most often case is that you have changed a process configuration, and you forgot to tell SELinux about it.
Labeling problems
A common cause of labeling problems is when a non-standard directory is used for a service. For example, instead of using /var/www/html/ for a website, an administrator might want to use /srv/myweb/. On Red Hat Enterprise Linux, the /srv directory is labeled with the var_t type. Files and directories created in /srv inherit this type. Also, newly-created objects in top-level directories, such as /myserver, can be labeled with the default_t type. SELinux prevents the Apache HTTP Server (httpd) from accessing both of these types. To allow access, SELinux must know that the files in /srv/myweb/ are to be accessible by httpd:
# semanage fcontext -a -t httpd_sys_content_t "/srv/myweb(/.*)?"
This semanage command adds the context for the /srv/myweb/ directory and all files and directories under it to the SELinux file-context configuration. The semanage utility does not change the context. As root, use the restorecon utility to apply the changes:
# restorecon -R -v /srv/mywebIncorrect context
The matchpathcon utility checks the context of a file path and compares it to the default label for that path. The following example demonstrates the use of matchpathcon on a directory that contains incorrectly labeled files:
$ matchpathcon -V /var/www/html/*
/var/www/html/index.html has context unconfined_u:object_r:user_home_t:s0, should be system_u:object_r:httpd_sys_content_t:s0
/var/www/html/page1.html has context unconfined_u:object_r:user_home_t:s0, should be system_u:object_r:httpd_sys_content_t:s0
In this example, the index.html and page1.html files are labeled with the user_home_t type. This type is used for files in user home directories. Using the mv command to move files from your home directory may result in files being labeled with the user_home_t type. This type should not exist outside of home directories. Use the restorecon utility to restore such files to their correct type:
# restorecon -v /var/www/html/index.html
restorecon reset /var/www/html/index.html context unconfined_u:object_r:user_home_t:s0->system_u:object_r:httpd_sys_content_t:s0
To restore the context for all files under a directory, use the -R option:
# restorecon -R -v /var/www/html/
restorecon reset /var/www/html/page1.html context unconfined_u:object_r:samba_share_t:s0->system_u:object_r:httpd_sys_content_t:s0
restorecon reset /var/www/html/index.html context unconfined_u:object_r:samba_share_t:s0->system_u:object_r:httpd_sys_content_t:s0Confined applications configured in non-standard ways
Services can be run in a variety of ways. To account for that, you need to specify how you run your services. You can achieve this through SELinux booleans that allow parts of SELinux policy to be changed at runtime. This enables changes, such as allowing services access to NFS volumes, without reloading or recompiling SELinux policy. Also, running services on non-default port numbers requires policy configuration to be updated using the semanage command.
For example, to allow the Apache HTTP Server to communicate with MariaDB, enable the httpd_can_network_connect_db boolean:
# setsebool -P httpd_can_network_connect_db on
Note that the -P option makes the setting persistent across reboots of the system.
If access is denied for a particular service, use the getsebool and grep utilities to see if any booleans are available to allow access. For example, use the getsebool -a | grep ftp command to search for FTP related booleans:
$ getsebool -a | grep ftp
ftpd_anon_write --> off
ftpd_full_access --> off
ftpd_use_cifs --> off
ftpd_use_nfs --> off
ftpd_connect_db --> off
httpd_enable_ftp_server --> off
tftp_anon_write --> off
To get a list of booleans and to find out if they are enabled or disabled, use the getsebool -a command. To get a list of booleans including their meaning, and to find out if they are enabled or disabled, install the selinux-policy-devel package and use the semanage boolean -l command as root.
Port numbers
Depending on policy configuration, services can only be allowed to run on certain port numbers. Attempting to change the port a service runs on without changing policy may result in the service failing to start. For example, run the semanage port -l | grep http command as root to list http related ports:
# semanage port -l | grep http
http_cache_port_t tcp 3128, 8080, 8118
http_cache_port_t udp 3130
http_port_t tcp 80, 443, 488, 8008, 8009, 8443
pegasus_http_port_t tcp 5988
pegasus_https_port_t tcp 5989
The http_port_t port type defines the ports Apache HTTP Server can listen on, which in this case, are TCP ports 80, 443, 488, 8008, 8009, and 8443. If an administrator configures httpd.conf so that httpd listens on port 9876 (Listen 9876), but policy is not updated to reflect this, the following command fails:
# systemctl start httpd.service Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details. # systemctl status httpd.service httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled) Active: failed (Result: exit-code) since Thu 2013-08-15 09:57:05 CEST; 59s ago Process: 16874 ExecStop=/usr/sbin/httpd $OPTIONS -k graceful-stop (code=exited, status=0/SUCCESS) Process: 16870 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
An SELinux denial message similar to the following is logged to /var/log/audit/audit.log:
type=AVC msg=audit(1225948455.061:294): avc: denied { name_bind } for pid=4997 comm="httpd" src=9876 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket
To allow httpd to listen on a port that is not listed for the http_port_t port type, use the semanage port command to assign a different label to the port:
# semanage port -a -t http_port_t -p tcp 9876
The -a option adds a new record; the -t option defines a type; and the -p option defines a protocol. The last argument is the port number to add.
Corner cases, evolving or broken applications, and compromised systems
Applications may contain bugs, causing SELinux to deny access. Also, SELinux rules are evolving – SELinux may not have seen an application running in a certain way, possibly causing it to deny access, even though the application is working as expected. For example, if a new version of PostgreSQL is released, it may perform actions the current policy does not account for, causing access to be denied, even though access should be allowed.
For these situations, after access is denied, use the audit2allow utility to create a custom policy module to allow access. You can report missing rules in the SELinux policy in Red Hat Bugzilla. For Red Hat Enterprise Linux 9, create bugs against the Red Hat Enterprise Linux 9 product, and select the selinux-policy component. Include the output of the audit2allow -w -a and audit2allow -a commands in such bug reports.
If an application asks for major security privileges, it could be a signal that the application is compromised. Use intrusion detection tools to inspect such suspicious behavior.
The Solution Engine on the Red Hat Customer Portal can also provide guidance in the form of an article containing a possible solution for the same or very similar problem you have. Select the relevant product and version and use SELinux-related keywords, such as selinux or avc, together with the name of your blocked service or application, for example: selinux samba.
5.4. Creating a local SELinux policy module
Adding specific SELinux policy modules to an active SELinux policy can fix certain problems with the SELinux policy. You can use this procedure to fix a specific Known Issue described in Red Hat release notes, or to implement a specific Red Hat Solution.
Use only rules provided by Red Hat. Red Hat does not support creating SELinux policy modules with custom rules, because this falls outside of the Production Support Scope of Coverage. If you are not an expert, contact your Red Hat sales representative and request consulting services.
Prerequisites
-
The
setools-consoleandauditpackages for verification.
Procedure
Open a new
.cilfile with a text editor, for example:# vim <local_module>.cilTo keep your local modules better organized, use the
local_prefix in the names of local SELinux policy modules.Insert the custom rules from a Known Issue or a Red Hat Solution.
ImportantDo not write your own rules. Use only the rules provided in a specific Known Issue or Red Hat Solution.
For example, to implement the SELinux denies cups-lpd read access to cups.sock in RHEL solution, insert the following rule:
NoteThe example solution has been fixed permanently for RHEL in RHBA-2021:4420. Therefore, the parts of this procedure specific to this solution have no effect on updated RHEL 8 and 9 systems, and are included only as examples of syntax.
(allow cupsd_lpd_t cupsd_var_run_t (sock_file (read)))
Note that you can use either of the two SELinux rule syntaxes, Common Intermediate Language (CIL) and m4. For example,
(allow cupsd_lpd_t cupsd_var_run_t (sock_file (read)))in CIL is equivalent to the following in m4:module local_cupslpd-read-cupssock 1.0; require { type cupsd_var_run_t; type cupsd_lpd_t; class sock_file read; } #============= cupsd_lpd_t ============== allow cupsd_lpd_t cupsd_var_run_t:sock_file read;- Save and close the file.
Install the policy module:
# semodule -i <local_module>.cilNoteWhen you want to remove a local policy module which you created by using
semodule -i, refer to the module name without the.cilsuffix. To remove a local policy module, usesemodule -r <local_module>.Restart any services related to the rules:
# systemctl restart <service-name>
Verification
List the local modules installed in your SELinux policy:
# semodule -lfull | grep "local_" 400 local_module cilNoteBecause local modules have priority
400, you can filter them from the list also by using that value, for example, by using thesemodule -lfull | grep -v ^100command.Search the SELinux policy for the relevant allow rules:
# sesearch -A --source=<SOURCENAME> --target=<TARGETNAME> --class=<CLASSNAME> --perm=<P1>,<P2>Where
<SOURCENAME>is the source SELinux type,<TARGETNAME>is the target SELinux type,<CLASSNAME>is the security class or object class name, and<P1>and<P2>are the specific permissions of the rule.For example, for the SELinux denies cups-lpd read access to cups.sock in RHEL solution:
# sesearch -A --source=cupsd_lpd_t --target=cupsd_var_run_t --class=sock_file --perm=read allow cupsd_lpd_t cupsd_var_run_t:sock_file { append getattr open read write };The last line should now include the
readoperation.Verify that the relevant service runs confined by SELinux:
Identify the process related to the relevant service:
$ systemctl status <service-name>Check the SELinux context of the process listed in the output of the previous command:
$ ps -efZ | grep <process-name>
Verify that the service does not cause any SELinux denials:
# ausearch -m AVC -i -ts recent <no matches>The
-ioption interprets the numeric values into human-readable text.
Additional resources
- How to create custom SELinux policy module wisely Knowledgebase article
5.5. SELinux denials in the Audit log
The Linux Audit system stores log entries in the /var/log/audit/audit.log file by default.
To list only SELinux-related records, use the ausearch command with the message type parameter set to AVC and AVC_USER at a minimum, for example:
# ausearch -m AVC,USER_AVC,SELINUX_ERR,USER_SELINUX_ERRAn SELinux denial entry in the Audit log file can look as follows:
type=AVC msg=audit(1395177286.929:1638): avc: denied { read } for pid=6591 comm="httpd" name="webpages" dev="0:37" ino=2112 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:nfs_t:s0 tclass=dirThe most important parts of this entry are:
-
avc: denied- the action performed by SELinux and recorded in Access Vector Cache (AVC) -
{ read }- the denied action -
pid=6591- the process identifier of the subject that tried to perform the denied action -
comm="httpd"- the name of the command that was used to invoke the analyzed process -
httpd_t- the SELinux type of the process -
nfs_t- the SELinux type of the object affected by the process action -
tclass=dir- the target object class
The previous log entry can be translated to:
SELinux denied the httpd process with PID 6591 and the httpd_t type to read from a directory with the nfs_t type.
The following SELinux denial message occurs when the Apache HTTP Server attempts to access a directory labeled with a type for the Samba suite:
type=AVC msg=audit(1226874073.147:96): avc: denied { getattr } for pid=2465 comm="httpd" path="/var/www/html/file1" dev=dm-0 ino=284133 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:samba_share_t:s0 tclass=file-
{ getattr }- thegetattrentry indicates the source process was trying to read the target file’s status information. This occurs before reading files. SELinux denies this action because the process accesses the file and it does not have an appropriate label. Commonly seen permissions includegetattr,read, andwrite. -
path="/var/www/html/file1"- the path to the object (target) the process attempted to access. -
scontext="unconfined_u:system_r:httpd_t:s0"- the SELinux context of the process (source) that attempted the denied action. In this case, it is the SELinux context of the Apache HTTP Server, which is running with thehttpd_ttype. -
tcontext="unconfined_u:object_r:samba_share_t:s0"- the SELinux context of the object (target) the process attempted to access. In this case, it is the SELinux context offile1.
This SELinux denial can be translated to:
SELinux denied the httpd process with PID 2465 to access the /var/www/html/file1 file with the samba_share_t type, which is not accessible to processes running in the httpd_t domain unless configured otherwise.
Additional resources
-
auditd(8)andausearch(8)man pages
