Red Hat Training

A Red Hat training course is available for Red Hat Enterprise Linux

11.3. Fixing Problems

The following sections help troubleshoot issues. They go over: checking Linux permissions, which are checked before SELinux rules; possible causes of SELinux denying access, but no denials being logged; manual pages for services, which contain information about labeling and Booleans; permissive domains, for allowing one process to run permissive, rather than the whole system; how to search for and view denial messages; analyzing denials; and creating custom policy modules with audit2allow.

11.3.1. Linux Permissions

When access is denied, check standard Linux permissions. As mentioned in Chapter 1, Introduction, most operating systems use a Discretionary Access Control (DAC) system to control access, allowing users to control the permissions of files that they own. SELinux policy rules are checked after DAC rules. SELinux policy rules are not used if DAC rules deny access first.
If access is denied and no SELinux denials are logged, use the following command to view the standard Linux permissions:
~]$ ls -l /var/www/html/index.html
-rw-r----- 1 root root 0 2009-05-07 11:06 index.html
In this example, index.html is owned by the root user and group. The root user has read and write permissions (-rw), and members of the root group have read permissions (-r-). Everyone else has no access (---). By default, such permissions do not allow httpd to read this file. To resolve this issue, use the chown command to change the owner and group. This command must be run as root:
~]# chown apache:apache /var/www/html/index.html
This assumes the default configuration, in which httpd runs as the Linux Apache user. If you run httpd with a different user, replace apache:apache with that user.
See the Fedora Documentation Project "Permissions" draft for information about managing Linux permissions.

11.3.2. Possible Causes of Silent Denials

In certain situations, AVC denial messages may not be logged when SELinux denies access. Applications and system library functions often probe for more access than required to perform their tasks. To maintain the least privilege without filling audit logs with AVC denials for harmless application probing, the policy can silence AVC denials without allowing permission by using dontaudit rules. These rules are common in standard policy. The downside of dontaudit is that, although SELinux denies access, denial messages are not logged, making troubleshooting more difficult.
To temporarily disable dontaudit rules, allowing all denials to be logged, enter the following command as root:
~]# semodule -DB
The -D option disables dontaudit rules; the -B option rebuilds policy. After running semodule -DB, try exercising the application that was encountering permission problems, and see if SELinux denials — relevant to the application — are now being logged. Take care in deciding which denials should be allowed, as some should be ignored and handled by dontaudit rules. If in doubt, or in search of guidance, contact other SELinux users and developers on an SELinux list, such as fedora-selinux-list.
To rebuild policy and enable dontaudit rules, enter the following command as root:
~]# semodule -B
This restores the policy to its original state. For a full list of dontaudit rules, run the sesearch --dontaudit command. Narrow down searches using the -s domain option and the grep command. For example:
~]$ sesearch --dontaudit -s smbd_t | grep squid
dontaudit smbd_t squid_port_t : tcp_socket name_bind ;
dontaudit smbd_t squid_port_t : udp_socket name_bind ;

11.3.3. Manual Pages for Services

Manual pages for services contain valuable information, such as what file type to use for a given situation, and Booleans to change the access a service has (such as httpd accessing NFS volumes). This information may be in the standard manual page or in the manual page that can be automatically generated from the SELinux policy for every service domain using the sepolicy manpage utility. Such manual pages are named in the service-name_selinux format. Such manual pages are also shipped with the selinux-policy-doc package.
For example, the httpd_selinux(8) manual page has information about what file type to use for a given situation, as well as Booleans to allow scripts, sharing files, accessing directories inside user home directories, and so on. Other manual pages with SELinux information for services include:
  • Samba: the samba_selinux(8) manual page for example describes that enabling the samba_enable_home_dirs Boolean allows Samba to share users home directories.
  • NFS: the nfsd_selinux(8) manual page describes SELinux nfsd policy that allows users to setup their nfsd processes in as secure a method as possible.
The information in manual pages helps you configure the correct file types and Booleans, helping to prevent SELinux from denying access.
See Section 5.4, “Generating Manual Pages: sepolicy manpage for further information about sepolicy manpage.

11.3.4. Permissive Domains

When SELinux is running in permissive mode, SELinux does not deny access, but denials are logged for actions that would have been denied if running in enforcing mode. Previously, it was not possible to make a single domain permissive (remember: processes run in domains). In certain situations, this led to making the whole system permissive to troubleshoot issues.
Permissive domains allow an administrator to configure a single process (domain) to run permissive, rather than making the whole system permissive. SELinux checks are still performed for permissive domains; however, the kernel allows access and reports an AVC denial for situations where SELinux would have denied access.
Permissive domains have the following uses:
  • They can be used for making a single process (domain) run permissive to troubleshoot an issue without putting the entire system at risk by making it permissive.
  • They allow an administrator to create policies for new applications. Previously, it was recommended that a minimal policy be created, and then the entire machine put into permissive mode, so that the application could run, but SELinux denials still logged. The audit2allow could then be used to help write the policy. This put the whole system at risk. With permissive domains, only the domain in the new policy can be marked permissive, without putting the whole system at risk.

11.3.4.1. Making a Domain Permissive

To make a domain permissive, run the semanage permissive -a domain command, where domain is the domain you want to make permissive. For example, enter the following command as root to make the httpd_t domain (the domain the Apache HTTP Server runs in) permissive:
~]# semanage permissive -a httpd_t
To view a list of domains you have made permissive, run the semodule -l | grep permissive command as root. For example:
~]# semodule -l | grep permissive
permissive_httpd_t    (null)
permissivedomains     (null)
If you no longer want a domain to be permissive, run the semanage permissive -d domain command as root. For example:
~]# semanage permissive -d httpd_t

11.3.4.2. Disabling Permissive Domains

The permissivedomains.pp module contains all of the permissive domain declarations that are presented on the system. To disable all permissive domains, enter the following command as root:
~]# semodule -d permissivedomains

Note

Once a policy module is disabled through the semodule -d command, it is no longer showed in the output of the semodule -l command. To see all policy modules including disabled, enter the following command as root:
~]# semodule --list-modules=full

11.3.4.3. Denials for Permissive Domains

The SYSCALL message is different for permissive domains. The following is an example AVC denial (and the associated system call) from the Apache HTTP Server:
type=AVC msg=audit(1226882736.442:86): avc:  denied  { getattr } for  pid=2427 comm="httpd" path="/var/www/html/file1" dev=dm-0 ino=284133 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:samba_share_t:s0 tclass=file

type=SYSCALL msg=audit(1226882736.442:86): arch=40000003 syscall=196 success=no exit=-13 a0=b9a1e198 a1=bfc2921c a2=54dff4 a3=2008171 items=0 ppid=2425 pid=2427 auid=502 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
By default, the httpd_t domain is not permissive, and as such, the action is denied, and the SYSCALL message contains success=no. The following is an example AVC denial for the same situation, except the semanage permissive -a httpd_t command has been run to make the httpd_t domain permissive:
type=AVC msg=audit(1226882925.714:136): avc:  denied  { read } for  pid=2512 comm="httpd" name="file1" dev=dm-0 ino=284133 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:samba_share_t:s0 tclass=file

type=SYSCALL msg=audit(1226882925.714:136): arch=40000003 syscall=5 success=yes exit=11 a0=b962a1e8 a1=8000 a2=0 a3=8000 items=0 ppid=2511 pid=2512 auid=502 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
In this case, although an AVC denial was logged, access was not denied, as shown by success=yes in the SYSCALL message.
See Dan Walsh's "Permissive Domains" blog entry for further information about permissive domains.

11.3.5. Searching For and Viewing Denials

This section assumes the setroubleshoot, setroubleshoot-server, dbus and audit packages are installed, and that the auditd, rsyslogd, and setroubleshootd daemons are running. See Section 4.2, “Which Log File is Used” for information about starting these daemons. A number of utilites are available for searching for and viewing SELinux AVC messages, such as ausearch, aureport, and sealert.

ausearch

The audit package provides the ausearch utility that can query the audit daemon logs for events based on different search criteria.[10] The ausearch utility accesses /var/log/audit/audit.log, and as such, must be run as the root user:
Searching ForCommand
all denialsausearch -m avc,user_avc,selinux_err,user_selinux_err
denials for that todayausearch -m avc -ts today
denials from the last 10 minutesausearch -m avc -ts recent
To search for SELinux AVC messages for a particular service, use the -c comm-name option, where comm-name is the executable’s name, for example, httpd for the Apache HTTP Server, and smbd for Samba:
~]# ausearch -m avc -c httpd
~]# ausearch -m avc -c smbd
With each ausearch command, it is advised to use either the --interpret (-i) option for easier readability, or the --raw (-r) option for script processing. See the ausearch(8) manual page for further ausearch options.

aureport

The audit package provides the aureport utility, which produces summary reports of the audit system logs. [11] The aureport utility accesses /var/log/audit/audit.log, and as such, must be run as the root user. To view a list of SELinux denial messages and how often each one occurred, run the aureport -a command. The following is example output that includes two denials:
~]# aureport -a

AVC Report
========================================================
# date time comm subj syscall class permission obj event
========================================================
1. 05/01/2009 21:41:39 httpd unconfined_u:system_r:httpd_t:s0 195 file getattr system_u:object_r:samba_share_t:s0 denied 2
2. 05/03/2009 22:00:25 vsftpd unconfined_u:system_r:ftpd_t:s0 5 file read unconfined_u:object_r:cifs_t:s0 denied 4

sealert

The setroubleshoot-server package provides the sealert utility, which reads denial messages translated by setroubleshoot-server.[12] Denials are assigned IDs, as seen in /var/log/messages. The following is an example denial from messages:
setroubleshoot: SELinux is preventing /usr/sbin/httpd from name_bind access on the tcp_socket. For complete SELinux messages. run sealert -l 8c123656-5dda-4e5d-8791-9e3bd03786b7
In this example, the denial ID is 8c123656-5dda-4e5d-8791-9e3bd03786b7. The -l option takes an ID as an argument. Running the sealert -l 8c123656-5dda-4e5d-8791-9e3bd03786b7 command presents a detailed analysis of why SELinux denied access, and a possible solution for allowing access.
If you are running the X Window System, have the setroubleshoot and setroubleshoot-server packages installed, and the setroubleshootd, dbus and auditd daemons are running, a warning is displayed when access is denied by SELinux:
An AVC denial message
Clicking on Show launches the sealert GUI, which allows you to troubleshoot the problem:
Alternatively, run the sealert -b command to launch the sealert GUI. To view a detailed analysis of all denial messages, run the sealert -l \* command.

11.3.6. Raw Audit Messages

Raw audit messages are logged to /var/log/audit/audit.log. The following is an example AVC denial message (and the associated system call) that occurred when the Apache HTTP Server (running in the httpd_t domain) attempted to access the /var/www/html/file1 file (labeled with the samba_share_t type):
type=AVC msg=audit(1226874073.147:96): avc:  denied  { getattr } for  pid=2465 comm="httpd" path="/var/www/html/file1" dev=dm-0 ino=284133 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:samba_share_t:s0 tclass=file

type=SYSCALL msg=audit(1226874073.147:96): arch=40000003 syscall=196 success=no exit=-13 a0=b98df198 a1=bfec85dc a2=54dff4 a3=2008171 items=0 ppid=2463 pid=2465 auid=502 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=6 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
{ getattr }
The item in the curly brackets indicates the permission that was denied. The getattr entry indicates the source process was trying to read the target file's status information. This occurs before reading files. This action is denied due to the file being accessed having a wrong label. Commonly seen permissions include getattr, read, and write.
comm="httpd"
The executable that launched the process. The full path of the executable is found in the exe= section of the system call (SYSCALL) message, which in this case, is exe="/usr/sbin/httpd".
path="/var/www/html/file1"
The path to the object (target) the process attempted to access.
scontext="unconfined_u:system_r:httpd_t:s0"
The SELinux context of the process that attempted the denied action. In this case, it is the SELinux context of the Apache HTTP Server, which is running in the httpd_t domain.
tcontext="unconfined_u:object_r:samba_share_t:s0"
The SELinux context of the object (target) the process attempted to access. In this case, it is the SELinux context of file1. Note that the samba_share_t type is not accessible to processes running in the httpd_t domain.
In certain situations, the tcontext may match the scontext, for example, when a process attempts to execute a system service that will change characteristics of that running process, such as the user ID. Also, the tcontext may match the scontext when a process tries to use more resources (such as memory) than normal limits allow, resulting in a security check to see if that process is allowed to break those limits.
From the system call (SYSCALL) message, two items are of interest:
  • success=no: indicates whether the denial (AVC) was enforced or not. success=no indicates the system call was not successful (SELinux denied access). success=yes indicates the system call was successful. This can be seen for permissive domains or unconfined domains, such as unconfined_service_t and kernel_t.
  • exe="/usr/sbin/httpd": the full path to the executable that launched the process, which in this case, is exe="/usr/sbin/httpd".
An incorrect file type is a common cause for SELinux denying access. To start troubleshooting, compare the source context (scontext) with the target context (tcontext). Should the process (scontext) be accessing such an object (tcontext)? For example, the Apache HTTP Server (httpd_t) should only be accessing types specified in the httpd_selinux(8) manual page, such as httpd_sys_content_t, public_content_t, and so on, unless configured otherwise.

11.3.7. sealert Messages

Denials are assigned IDs, as seen in /var/log/messages. The following is an example AVC denial (logged to messages) that occurred when the Apache HTTP Server (running in the httpd_t domain) attempted to access the /var/www/html/file1 file (labeled with the samba_share_t type):
hostname setroubleshoot: SELinux is preventing httpd (httpd_t) "getattr" to /var/www/html/file1 (samba_share_t). For complete SELinux messages. run sealert -l 32eee32b-21ca-4846-a22f-0ba050206786
As suggested, run the sealert -l 32eee32b-21ca-4846-a22f-0ba050206786 command to view the complete message. This command only works on the local machine, and presents the same information as the sealert GUI:
~]$ sealert -l 32eee32b-21ca-4846-a22f-0ba050206786
SELinux is preventing httpd from getattr access on the file /var/www/html/file1.

*****  Plugin restorecon (92.2 confidence) suggests   ************************

If you want to fix the label. 
/var/www/html/file1 default label should be httpd_sys_content_t.
Then you can run restorecon.
Do
# /sbin/restorecon -v /var/www/html/file1

*****  Plugin public_content (7.83 confidence) suggests   ********************

If you want to treat file1 as public content
Then you need to change the label on file1 to public_content_t or public_content_rw_t.
Do
# semanage fcontext -a -t public_content_t '/var/www/html/file1'
# restorecon -v '/var/www/html/file1'

*****  Plugin catchall (1.41 confidence) suggests   **************************

If you believe that httpd should be allowed getattr access on the file1 file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'httpd' --raw | audit2allow -M my-httpd
# semodule -i my-httpd.pp


Additional Information:
Source Context                system_u:system_r:httpd_t:s0
Target Context                unconfined_u:object_r:samba_share_t:s0
Target Objects                /var/www/html/file1 [ file ]
Source                        httpd
Source Path                   httpd
Port                          <Unknown>
Host                          hostname.redhat.com
Source RPM Packages           
Target RPM Packages           
Policy RPM                    selinux-policy-3.13.1-166.el7.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     hostname.redhat.com
Platform                      Linux hostname.redhat.com
                              3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57
                              EDT 2017 x86_64 x86_64
Alert Count                   2
First Seen                    2017-07-20 02:52:11 EDT
Last Seen                     2017-07-20 02:52:11 EDT
Local ID                      32eee32b-21ca-4846-a22f-0ba050206786

Raw Audit Messages
type=AVC msg=audit(1500533531.140:295): avc:  denied  { getattr } for  pid=24934 comm="httpd" path="/var/www/html/file1" dev="vda1" ino=31457414 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:samba_share_t:s0 tclass=file


Hash: httpd,httpd_t,samba_share_t,file,getattr
Summary
A brief summary of the denied action. This is the same as the denial in /var/log/messages. In this example, the httpd process was denied access to a file (file1), which is labeled with the samba_share_t type.
Detailed Description
A more verbose description. In this example, file1 is labeled with the samba_share_t type. This type is used for files and directories that you want to export using Samba. The description suggests changing the type to a type that can be accessed by the Apache HTTP Server and Samba, if such access is required.
Allowing Access
A suggestion for how to allow access. This may be relabeling files, enabling a Boolean, or making a local policy module. In this case, the suggestion is to label the file with a type accessible to both the Apache HTTP Server and Samba.
Fix Command
A suggested command to allow access and resolve the denial. In this example, it gives the command to change the file1 type to public_content_t, which is accessible to the Apache HTTP Server and Samba.
Additional Information
Information that is useful in bug reports, such as the policy package name and version (selinux-policy-3.13.1-166.el7.noarch), but may not help towards solving why the denial occurred.
Raw Audit Messages
The raw audit messages from /var/log/audit/audit.log that are associated with the denial. See Section 11.3.6, “Raw Audit Messages” for information about each item in the AVC denial.

11.3.8. Allowing Access: audit2allow

Warning

Do not use the example in this section in production. It is used only to demonstrate the use of the audit2allow utility.
The audit2allow utility gathers information from logs of denied operations and then generates SELinux policy allow rules.[13] After analyzing denial messages as per Section 11.3.7, “sealert Messages”, and if no label changes or Booleans allowed access, use audit2allow to create a local policy module. When access is denied by SELinux, running audit2allow generates Type Enforcement rules that allow the previously denied access.
You should not use audit2allow to generate a local policy module as your first option when you see an SELinux denial. Troubleshooting should start with a check if there is a labeling problem. The second most often case is that you have changed a process configuration, and you forgot to tell SELinux about it. For more information, see the Four Key Causes of SELinux Errors white paper.
The following example demonstrates using audit2allow to create a policy module:
  1. A denial message and the associated system call are logged to the /var/log/audit/audit.log file:
    type=AVC msg=audit(1226270358.848:238): avc:  denied  { write } for  pid=13349 comm="certwatch" name="cache" dev=dm-0 ino=218171 scontext=system_u:system_r:certwatch_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir
    
    type=SYSCALL msg=audit(1226270358.848:238): arch=40000003 syscall=39 success=no exit=-13 a0=39a2bf a1=3ff a2=3a0354 a3=94703c8 items=0 ppid=13344 pid=13349 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="certwatch" exe="/usr/bin/certwatch" subj=system_u:system_r:certwatch_t:s0 key=(null)
    
    In this example, certwatch was denied the write access to a directory labeled with the var_t type. Analyze the denial message as per Section 11.3.7, “sealert Messages”. If no label changes or Booleans allowed access, use audit2allow to create a local policy module.
  2. Enter the following command to produce a human-readable description of why the access was denied. The audit2allow utility reads /var/log/audit/audit.log, and as such, must be run as the root user:
    ~]# audit2allow -w -a
    type=AVC msg=audit(1226270358.848:238): avc:  denied  { write } for  pid=13349 comm="certwatch" name="cache" dev=dm-0 ino=218171 scontext=system_u:system_r:certwatch_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir
    	Was caused by:
    		Missing type enforcement (TE) allow rule.
    
    	You can use audit2allow to generate a loadable module to allow this access.
    
    The -a command-line option causes all audit logs to be read. The -w option produces the human-readable description. As shown, access was denied due to a missing Type Enforcement rule.
  3. Enter the following command to view the Type Enforcement rule that allows the denied access:
    ~]# audit2allow -a
    
    
    #============= certwatch_t ==============
    allow certwatch_t var_t:dir write;
    

    Important

    Missing Type Enforcement rules are usually caused by bugs in the SELinux policy, and should be reported in Red Hat Bugzilla. For Red Hat Enterprise Linux, create bugs against the Red Hat Enterprise Linux product, and select the selinux-policy component. Include the output of the audit2allow -w -a and audit2allow -a commands in such bug reports.
  4. To use the rule displayed by audit2allow -a, enter the following command as root to create a custom module. The -M option creates a Type Enforcement file (.te) with the name specified with -M, in your current working directory:
    ~]# audit2allow -a -M mycertwatch
    ******************** IMPORTANT ***********************
    To make this policy package active, execute:
    
    semodule -i mycertwatch.pp
    
  5. Also, audit2allow compiles the Type Enforcement rule into a policy package (.pp):
    ~]# ls
    mycertwatch.pp  mycertwatch.te
    
    To install the module, enter the following command as the root:
    ~]# semodule -i mycertwatch.pp

    Important

    Modules created with audit2allow may allow more access than required. It is recommended that policy created with audit2allow be posted to the upstream SELinux list for review. If you believe there is a bug in the policy, create a bug in Red Hat Bugzilla.
If you have multiple denial messages from multiple processes, but only want to create a custom policy for a single process, use the grep utility to narrow down the input for audit2allow. The following example demonstrates using grep to only send denial messages related to certwatch through audit2allow:
~]# grep certwatch /var/log/audit/audit.log | audit2allow -R -M mycertwatch2
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i mycertwatch2.pp


[10] See the ausearch(8) manual page for further information about ausearch.
[11] See the aureport(8) manual page for further information about aureport.
[12] See the sealert(8) manual page for further information about sealert.
[13] See the audit2allow(1) manual page for more information about audit2allow.