Receiving message 'failed with 500 - error' when trying to update rhui certificates

Solution Verified - Updated -

Environment

  • Red Hat Update Infrastructure 3
  • Red Hat Enterprise Linux

Issue

When running the rhui-subscription-sync command the following errors are seen:

2022-01-25 11:12:30,882 - Connecting to RHUA [rhua]...
2022-01-25 11:12:30,987 - Uncaught exception: RequestException: POST request on /pulp/api/v2/actions/login/ failed with 500 - error signing cert request: Signature ok
subject=/CN=admin:admin:5c88a67c872da6e6ecf5f85d
Error opening CA Certificate /etc/pki/rhui/certs/entitlement-ca.crt
139764180613008:error:0200100D:system library:fopen:Permission denied:bss_file.c:402:fopen('/etc/pki/rhui/certs/entitlement-ca.crt','r')
139764180613008:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
--------
Getting CA Private Key
Error opening CA Private Key /etc/pki/rhui/private/entitlement-ca.key
140228115896208:error:0200100D:system library:fopen:Permission denied:bss_file.c:402:fopen('/etc/pki/rhui/private/entitlement-ca.key','r')
140228115896208:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
unable to load CA Private Key

Resolution

Ensure that /etc/pki/rhui/certs/entitlement-ca.crt has 755 permission on the RHUA server:

# chmod 755 /etc/pki/rhui/private/entitlement-ca.key
# restorecon -RFv /etc/pki/rhui/private/entitlement-ca.key

If /etc/pki/rhui/private/entitlement-ca.key no longer matches modulus for /etc/pki/rhui/certs/entitlement-ca.crt and the key needs to be replaced copy key from /etc/pki/rhui/private/rhui-default-ca.key;

# cp /etc/pki/rhui/private/entitlement-ca.key /etc/pki/rhui/private/entitlement-ca.key.backup
# cp  /etc/pki/rhui/private/rhui-default-ca.key /etc/pki/rhui/private/entitlement-ca.key

Root Cause

Using stat we could see the entitlement-ca.key file was altered at a time different from when the entitlement-ca.crt was created. As well, the modulus of the key did not match the crt. Since the key file had been altered, replace it with the copied key file from rhui-default-ca.key.

Diagnostic Steps

[root@rhua ~]# su - apache -s /bin/bash -c "stat /etc/pki/rhui/private/entitlement-ca.key"
  File: ‘/etc/pki/rhui/private/entitlement-ca.key’
  Size: 1679        Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 807190      Links: 1
Access: (0440/-r--r-----)  Uid: (    0/    root)   Gid: (   48/  apache)
Access: 2022-01-20 14:56:24.787546397 -0500
Modify: 2021-05-13 11:51:48.356692392 -0400
Change: 2021-05-13 11:51:48.357692478 -0400
 Birth: -

[root@rhua ~]# ls -ldZ /etc/pki/rhui/private/entitlement-ca.key
-r--r-----. root apache system_u:object_r:cert_t:s0      /etc/pki/rhui/private/entitlement-ca.key

[root@rhua ~]# namei -lom /etc/pki/rhui/private/entitlement-ca.key
f: /etc/pki/rhui/private/entitlement-ca.key
dr-xr-xr-x root root   /
drwxr-xr-x root root   etc
drwxr-xr-x root root   pki
drwxr-xr-x root apache rhui
drwxr-x--- root apache private
-r--r----- root apache entitlement-ca.key

[root@rhua ~]# ls -ldZ /etc/pki/rhui/private/entitlement-ca.key /etc/pki/rhui/private/ /etc/pki/rhui/
drwxr-xr-x. root apache system_u:object_r:cert_t:s0      /etc/pki/rhui/
drwxr-x---. root apache system_u:object_r:cert_t:s0      /etc/pki/rhui/private/
-r--r-----. root apache system_u:object_r:cert_t:s0      /etc/pki/rhui/private/entitlement-ca.key
[root@rhua ~]# openssl rsa -modulus -noout -in /etc/pki/rhui/private/entitlement-ca.key
Modulus=C22C292F16F12A5B3396CF4DE85206B576AC3DEF274BB1A4494746F57D188809C2D8F5A1EA575DFFD9501B9E8C93CEA6B4A6D75615ADF56A6B6B7F345EBB8EB59E2D70F91D2FD43470B3C0B642EEF49C3990367A1E5BD1061DD0591C30180DB7EBE3499C9DD1FFF03C6BAD8E04BA53F7E281D3674A47D4D74708ECB417C18AAB9BA7E6BFA9ADD37C60632BD946F161C5326FBE7CB76A8469970D6D6F3EFC0DB7B287678EEA5E00ADCCC02642ECCE5723B59A1966BD33EF7C9E0FD7610D0EA59D68830450701F3F9C014DFB0D62606A263FA98F14B43A3C832C759A66EC072B213A637529DCEADC43E57A58111D0790AEA0E82FAED34561519CB5DC1609587C4F8616E28E10030C26FA162BF0B781C71CE92C34CFB363B43A79B296EE76B0A884FE8A38B4DA91E77DAE35165E5023A6DBDE5E0BE9035E90DDDCCC78BD4FB2900994604D38739B13B86FB2BC5F82A0C60185D322EDEA20E77354DBAE87045A9D735AD96C568EAEDA6F72F2CD105784A1E2762F1A1DD3E1D537FE42F32670CF494D7176C0D93EF7195D253DAF44C8922DAD5CE1A0827EC13122E6B6B1F7D6C48E7B1ACFC3E251F8A55982543D3E789A75C6F26AF29ACDE85E301CD5805567E7B40CA9AEE50491BF4901B4CB6750FAF0054BA89F2DF9D37E79A013B677A081E6066C7EEA2BA756065E42F64508BB0E8A93BE10239897C1DDE731A0C975F06E1A08C9
--------
[root@rhua ~]# openssl x509 -modulus -noout -in /etc/pki/rhui/certs/entitlement-ca.crt
Modulus=93D9E52B48A4B27EA220BEC41CBE62B77206813DB506B08D2E7BB6485A815695D30D67BFA2EF067939A6DC3E6BA085EE4B4065BBA0189DB7C3429FD5591D49F599A0A940354D40622C12640D931CE5A11168D9AE40B386493365FC261126280E75C70C966807F1C90277EC1B79FBFADDC5227F194BF3CEF157AAF90C810ACF6CA7AAFC61AA94F168CE06230ED9E7CA962A2120BB684F657409B22AF0A48E8EEE4EA6990175420A965E705974839352D90AC97F4EBB4546BD173625A9AFB3E515BF48FC8CD849B423A61AAE8CBAE34A4E9E36BB999956589B1CDBDA3C7913476CEFE1C1D862002A4DD55E88DFB1E5C31CC9B6EAD4A6CE15FB28215C812F5D7F39

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.