How to troubleshoot subscription-manager and yum issues

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7.x
  • Red Hat Enterprise Linux 6.x
  • Red Hat Enterprise Linux 5.8 and later
  • Red Hat Subscription Manager (RHSM)

Issue

  • The following steps can be used for basic troubleshooting on subscription-manager and yum issues. In general, this can help solve many errors.

  • Let's troubleshoot this error for example:

    # yum check-update
Loaded plugins: downloadonly, product-id, subscription-manager
Updating certificate-based repositories.
https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/cf-tools/1.0/os/repodata/repomd.xml: [Errno 14] problem making ssl connection
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: rhel-6-server-cf-tools-1-rpms. Please verify its path and try again

Resolution

  • Check if a proxy (or anything between the server and Red Hat) is not changing the SSL certificate. Sometimes proxies are configured to resign SSL communication (similar to man-in-the-middle attack).

  • Crosscheck if UUID of system matches on the Customer Portal:

    1. Get the UUID from the system:

      # openssl x509 -in /etc/pki/consumer/cert.pem -noout -text| grep 'Subject: CN'
    2. Check the system's UUID on the Customer Portal: Go to access.redhat.com --> Subscriptions --> Systems (under the subheading Subscriber Inventory) or open: https://access.redhat.com/management/consumers?type=system. Then enter the system's UUID in the filter box to see if it matches.

    3. If the UUID does not match, refresh the subscription data from system:

      # subscription-manager refresh
# subscription-manager attach --auto

      -or-

      # subscription-manager attach --pool=POOLID

  • When using a http proxy with RHSM, the proxy settings need to be put in /etc/rhsm/rhsm.conf (and not /etc/yum.conf).

  • Check if a system is using standard curl package from Red Hat.

Additional information: RHSM Subscription Issues Troubleshooting Do's and Don'ts

Root Cause

The communication is not properly established with the correct certificates:

# curl -v --proxy-user user:password --proxy proxy.example.com:8080 https://subscription.rhn.redhat.com --cacert /etc/rhsm/ca/redhat-uep.pem
* About to connect() to proxy proxy.example.com port 8080 (#0)
*   Trying 10.0.0.1... connected
* Connected to proxy.example.com (10.0.0.1) port 8080 (#0)
* Establish HTTP proxy tunnel to subscription.rhn.redhat.com:443
* Proxy auth using Basic with user 'username'
> CONNECT subscription.rhn.redhat.com:443 HTTP/1.1
> Host: subscription.rhn.redhat.com:443
> Proxy-Authorization: Basic XXEEAA =
> User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
< Date: Tue, 03 Jul 2012 13:03:51 GMT
< Age: 2
< Proxy-Connection: Keep-Alive
< Via: 1.0 localhost.localdomain
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: /etc/pki/nssdb
*   CAfile: /etc/rhsm/ca/redhat-uep.pem
  CApath: none
* Peer's certificate issuer is not recognized: 'CN=proxy.example.com,O=My,L=RedHat,ST=South Carolina,C=US'
* NSS error -8179
* Closing connection #0
* Peer certificate cannot be authenticated with known CA certificates
curl: (60) Peer certificate cannot be authenticated with known CA certificates
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Diagnostic Steps

Find the troubleshooting steps which can help you to find the cause of the issue:

  • Make sure that the system's time and date is correct to the time and date of the geographical location.
  • Next thing to do is to unregister and register again the subscription manager - to be sure the server is properly registered using RHSM:

Unregister the server:

# date && subscription-manager --proxy=proxy.example.com:3128 unsubscribe --all
This machine has been unsubscribed from all subscriptions

# date && subscription-manager --proxy=proxy.example.com:3128 unregister
Consumer 11111111-2222-3333-4444-555555555556 has been deleted

Get the information about the server:

# date && subscription-manager --proxy=proxy.example.com:3128 facts --list

Register the server:

# date && subscription-manager --proxy=proxy.example.com:3128 register
The system has been registered with id: 11111111-2222-3333-4444-555555555555

# date && subscription-manager --proxy=proxy.example.com:3128 identity
Current identity is: 11111111-2222-3333-4444-555555555555
name: gss01.example.com
org name: 1111111
org id: 11111112222222333333334444444555

date && subscription-manager --proxy=proxy.example.com:3128 subscribe --pool=<PoolId_from_the_previous_command>

#Or use simply:
# date && subscription-manager --proxy=proxy.example.com:3128 list --available
+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+

ProductName:            Red Hat Enterprise Linux Server, Self-support (1-2 sockets)
                        (Up to 1 guest)
ProductId:              RH0197181                
PoolId:                 11111111111111111111111111111111
Quantity:               10                       
Multi-Entitlement:      No                       
Expires:                01/01/2013               
MachineType:            physical
.
.

# date && subscription-manager --proxy=proxy.example.com:3128 subscribe --auto
Installed Product Current Status:
Product Name:           Red Hat Enterprise Linux Server
Status:                 Subscribed


# date && subscription-manager --proxy=proxy.example.com:3128 list --consumed
+-------------------------------------------+
    Consumed Product Subscriptions
+-------------------------------------------+

Product Name:           Red Hat Enterprise Linux Server
Contract Number:        4444444
Account Number:         333333
Serial Number:          2222222222222222222
Active:                 True                     
Quantity Used:          1                        
Service Level:          None                     
Service Type :          None                     
Begins:                 22/09/11                 
Expires:                01/01/22
.
.
.


# date && subscription-manager --proxy=proxy.example.com:3128 repos --list
+----------------------------------------------------------+
    Entitled Repositories in /etc/yum.repos.d/redhat.repo
+----------------------------------------------------------+
.
.
.
Repo Name:              Red Hat Enterprise Linux 6 Server (RPMs)
Repo Id:                rhel-6-server-rpms       
Repo Url:               https://cdn.redhat.com/content/dist/rhel/server/6/$releasever/$basearch/os
Enabled:                1                        

Repo Name:              Red Hat CloudForms Tools for RHEL 6 (RPMs)
Repo Id:                rhel-6-server-cf-tools-1-rpms
Repo Url:               https://cdn.redhat.com/content/dist/rhel/server/6/$releasever/$basearch/cf-tools/1.0/os
Enabled:                1
.
.
  • Generate and check the strace:
yum clean all
export URLGRABBER_DEBUG=1,- 
sed -i.orig 's/debuglevel = 0/debuglevel = 1/g' /usr/lib64/python2.6/httplib.py 
strace -ttT -s1024 -v -o /tmp/yum.update.strace yum -d10 check-update
mv /usr/lib64/python2.6/httplib.py.orig /usr/lib64/python2.6/httplib.py

There should be something like this in the strace:

  • The system was able to connect to subscription.rhn.redhat.com:443 through the proxy server:
2012-03-12 10:04:36,002 [DEBUG]  @connection.py:194 - Using proxy: proxy.server.com:3128
2012-03-12 10:04:36,002 [DEBUG]  @connection.py:209 - Making request: GET https://subscription.rhn.redhat.com:443/subscription/consumers/8c1cb059-xxxx-xxxx-xxxx-46025d2b9d35/certificates?serials=50
27624092264466693
2012-03-12 10:04:37,952 [DEBUG]  @connection.py:218 - Response status: 200
  • The strace of the client's yum attempt shows an initially successful connection to cdn.redhat.com:443 through the proxy proxy.server.com:3128:
14:59:05.914373 connect(7, {sa_family=AF_INET, sin_port=htons(3128), sin_addr=inet_addr("192.168.0.2")}, 16) = -1 EINPROGRESS (Operation now in progress) <0.000022>
14:59:05.914423 poll([{fd=7, events=POLLOUT|POLLWRNORM}], 1, 29998) = 1 ([{fd=7, revents=POLLOUT|POLLWRNORM}]) <0.000227>
14:59:05.914693 getsockopt(7, SOL_SOCKET, SO_ERROR, [0], [4]) = 0 <0.000017>
14:59:05.914749 sendto(7, "CONNECT cdn.redhat.com:443 HTTP/1.1\r\nHost: cdn.redhat.com:443\r\nUser-Agent: urlgrabber/3.9.1 yum/3.2.29\r\nProxy-Connection: Keep-Alive\r\n\r\n", 136, MSG_NOSIGNAL, NU
LL, 0) = 136 <0.000012>
14:59:05.914794 poll([{fd=7, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 1 ([{fd=7, revents=POLLIN|POLLRDNORM}]) <0.231993>
14:59:06.146848 recvfrom(7, "HTTP/1.0 200 Connection established\r\n\r\n", 16384, 0, NULL, NULL) = 39 <0.000024>

curl --head --key /etc/pki/entitlement/4790939584130415916-key.pem -E /etc/pki/entitlement/4790939584130415916.pem -k https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/cf-tools/1.0/os/repodata/repomd.xml -x proxy.example.com:3128 --verbose -H "Cache-control: no-cache" -H "Pragma: no-cache" --cacert /etc/rhsm/ca/redhat-uep.pem
* About to connect() to proxy proxy.example.com port 3128 (#0)                                                                                                                                              
*   Trying 10.0.0.1... connected                                                                                                                                                                            
* Connected to proxy.example.com (10.0.0.1) port 3128 (#0)                                                                                                                                               
* Establish HTTP proxy tunnel to cdn.redhat.com:443                                                                                                                                                            
> CONNECT cdn.redhat.com:443 HTTP/1.1                                                                                                                                                                          
> Host: cdn.redhat.com:443                                                                                                                                                                                     
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2                                                                                           
> Proxy-Connection: Keep-Alive                                                                                                                                                                                 
>                                                                                                                                                                                                              
< HTTP/1.0 200 Connection established                                                                                                                                                                          
<                                                                                                                                                                                                              
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: /etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* NSS: client certificate: PEM Token #1:5027624092264466693.pem
*       subject: CN=8a85f98435edb09d01360626e0924712
*       start date: Jan 01 05:00:00 2012 GMT
*       expire date: Jan 01 04:59:59 2013 GMT
*       common name: 8a85f98435edb09d01360626e0924712
*       issuer: E=ca-support@redhat.com,CN=Red Hat Candlepin Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: CN=cdn.redhat.com,OU=IT,O="Red Hat, Inc.",L=Raleigh,ST=NORTH CAROLINA,C=US
*       start date: Jul 19 16:16:14 2010 GMT
*       expire date: Jul 16 16:16:14 2020 GMT
*       common name: cdn.redhat.com
*       issuer: E=ca-support@redhat.com,CN=Red Hat Entitlement Operations Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
> GET /content/dist/rhel/server/5/5Server/x86_64/cf-tools/1.0/os/repodata/repomd.xml HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Host: cdn.redhat.com
> Accept: */*
> 
< HTTP/1.1 200 OK
  • Without proxy using openssl:
openssl s_client -port 443 -CApath /etc/pki/entitlement/ -host cdn.redhat.com -prexit -cert 6666666666666666666.pem -key 6666666666666666666-key.pem
CONNECTED(00000003)
depth=1 C = US, ST = North Carolina, O = "Red Hat, Inc.", OU = Red Hat Network, CN = Red Hat Entitlement Operations Authority, emailAddress = ca-support@redhat.com
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=NORTH CAROLINA/L=Raleigh/O=Red Hat, Inc./OU=IT/CN=cdn.redhat.com
   i:/C=US/ST=North Carolina/O=Red Hat, Inc./OU=Red Hat Network/CN=Red Hat Entitlement Operations Authority/emailAddress=ca-support@redhat.com
 1 s:/C=US/ST=North Carolina/O=Red Hat, Inc./OU=Red Hat Network/CN=Red Hat Entitlement Operations Authority/emailAddress=ca-support@redhat.com
   i:/C=US/ST=North Carolina/L=Raleigh/O=Red Hat, Inc./OU=Red Hat Network/CN=Entitlement Master CA/emailAddress=ca-support@redhat.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIENjCCAh6gAwIBAgIBJzANBgkqhkiG9w0BAQUFADCBsTELMAkGA1UEBhMCVVMx
FzAVBgNVBAgMDk5vcnRoIENhcm9saW5hMRYwFAYDVQQKDA1SZWQgSGF0LCBJbmMu
MRgwFgYDVQQLDA9SZWQgSGF0IE5ldHdvcmsxMTAvBgNVBAMMKFJlZCBIYXQgRW50
aXRsZW1lbnQgT3BlcmF0aW9ucyBBdXRob3JpdHkxJDAiBgkqhkiG9w0BCQEWFWNh
LXN1cHBvcnRAcmVkaGF0LmNvbTAeFw0xMDA3MTkxNjE2MTRaFw0yMDA3MTYxNjE2
MTRaMHYxCzAJBgNVBAYTAlVTMRcwFQYDVQQIEw5OT1JUSCBDQVJPTElOQTEQMA4G
A1UEBxMHUmFsZWlnaDEWMBQGA1UEChMNUmVkIEhhdCwgSW5jLjELMAkGA1UECxMC
SVQxFzAVBgNVBAMTDmNkbi5yZWRoYXQuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQDcVN7OoMTenrvBx2GEdW+t5GYaWd/rg2dyFooyItr+snRgyTkeBQOL
ZmvGH/VbwQxrUmDCWnaxWX84hcZMn0xIEEOu/OdsRzrDdhBmsWTRRkabucJVg7BW
Zu89vATSpjZ+hCp/J+67ZNscuMqI7GzFG6A7KOgMRehg4pbgIR2sFwIDAQABoxcw
FTATBgNVHSUEDDAKBggrBgEFBQcDATANBgkqhkiG9w0BAQUFAAOCAgEAZxzgeGbE
YHpzqu0g+IAaTRIQ53xrRWy0PFqaeD9aRmJoqnjHJiYXguNsHcIY7el082RgxONf
hP2k4iE2Rcxo7MA3OiBEG7CvM73oLo94BhLKy/Ytbcut1K0WH1Q8hY3jIdk76sfM
yX2bFgFAHAbBcZFOAhvdaNrzxSFcX+WGvTMhvuXq4u3dRQJlG9QFG/d8Xx247G5e
rnRlBy23RHAqYVDs1bx2SA/SLx0WCUkuBgh7O8aFGam73ynVW5RosTtX5tw2L68I
JCknBfSEtJIMwJVv+OwMhV6EBydorEZkPZO0Q/3c/wNmp9VfjnfMybb+hyS8NP1T
zP2ewQvVwFv8yZYfNIXQCH3YGFXwtG0x1uKcgHQNpf53bvOqQy2DL76/QzEfya63
KVmomyyGX74v3fNytYOTVSRRCS4NoIzb+9oAIuHgLkgQyADfUDsIsk/DwM8oduq1
4n832s2DUJEZA1czt2ZdJXSfas3syP5BtsjaFPdUInqVLRAvC7hmj61GtnehTKx0
SYxs4ijC7SeXckE9vaO5hr6GcAYNzv/pvZ5DB6FBdzmeyX1lpUz/xn8Gc4xhM9lA
L5eQ180iMiLjbX9MAUayJLecwd49W3fQ1gbBkR6xTwy+gveRXJ+M1fxot+FUhV5v
Lt0DgnVbXOn0l4R5oZVv7k3iVBSPaDe4Dlc=
-----END CERTIFICATE-----
subject=/C=US/ST=NORTH CAROLINA/L=Raleigh/O=Red Hat, Inc./OU=IT/CN=cdn.redhat.com
issuer=/C=US/ST=North Carolina/O=Red Hat, Inc./OU=Red Hat Network/CN=Red Hat Entitlement Operations Authority/emailAddress=ca-support@redhat.com
---
Acceptable client certificate CA names
/C=US/ST=North Carolina/O=Red Hat, Inc./OU=Red Hat Network/CN=Red Hat Entitlement Operations Authority/emailAddress=ca-support@redhat.com
/C=US/ST=North Carolina/L=Raleigh/O=Red Hat, Inc./OU=Red Hat Network/CN=Entitlement Master CA/emailAddress=ca-support@redhat.com
/C=US/ST=North Carolina/O=Red Hat, Inc./OU=Red Hat Network/CN=Red Hat Candlepin Authority/emailAddress=ca-support@redhat.com
---
SSL handshake has read 3693 bytes and written 100396 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: 4EF45E40519DD86B56C5D62938E42B2ACB85F421F1B4853E607BE42738047A31
    Session-ID-ctx: 
    Master-Key: 7E3E5B2F9E2C257D76EB23E3477C67423761D7B843068EA95A43D3749D5464A52040F85635E3D8113F3D01A60E1AFEB4
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1345283601
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

Testing connectivity to subscription.rhn.redhat.com

  • Verifying the machine can connect to subscription.rhn.redhat.com with curl:
$ curl -v -k https://subscription.rhn.redhat.com/subscription/ --cacert /etc/rhsm/ca/redhat-uep.pem
* About to connect() to subscription.rhn.redhat.com port 443 (#0)
*   Trying 10.4.127.49... connected
* Connected to subscription.rhn.redhat.com (10.4.127.49) port 443 (#0)
...
* Closing connection #0
[{"rel":"consumers","href":"/consumers"},{"rel":"distributor_versions","href":"/distributor_versions"},{"rel":"entitlements","href":"/entitlements"},{"rel":"hypervisors","href":"/hypervisors"},{"rel":"jobs","href":"/jobs"},{"rel":"owners","href":"/owners"},{"rel":"pools","href":"/pools"},{"rel":"products","href":"/products"},{"rel":"subscriptions","href":"/subscriptions"},{"rel":"users","href":"/users"}]
  • Or with openssl s_client:
$ openssl s_client -port 443 -CApath /etc/rhsm/ca/redhat-uep.pem -host subscription.rhn.redhat.com
CONNECTED(00000003)
depth=2 C = US, ST = North Carolina, L = Raleigh, O = "Red Hat, Inc.", OU = Red Hat Network, CN = Entitlement Master CA, emailAddress = ca-support@redhat.com
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
...

    Start Time: 1397793816
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---

Then type in this:

GET /subscription/

... to get this output:

[{"rel":"consumers","href":"/consumers"},{"rel":"distributor_versions","href":"/distributor_versions"},{"rel":"entitlements","href":"/entitlements"},{"rel":"hypervisors","href":"/hypervisors"},{"rel":"jobs","href":"/jobs"},{"rel":"owners","href":"/owners"},{"rel":"pools","href":"/pools"},{"rel":"products","href":"/products"},{"rel":"subscriptions","href":"/subscriptions"},{"rel":"users","href":"/users"}]closed
  • Or testing basic connectivity when you have a consumer cert and key, and where RHNUSERNAME is your customer portal login:
# curl --key /etc/pki/consumer/key.pem --cert /etc/pki/consumer/cert.pem --cacert redhat-uep.pem https://subscription.rhn.redhat.com/subscription/users/RHNUSERNAME/owners

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments