Red Hat Satellite (Disconnected) 5.6 drops channel content

Latest response

This has now happened to me twice and Red Hat GSS can't explain whys, so I'm posting here to see what others know.

Within the last few weeks, I'm not able to sync or get the latest content after importing in update isos for RHEL5 x32 and x86_64 channels. Since I operate a disconnected RH Satellite, its going to take forever to download everything and import it back in.

This happened earlier this year with RHEL6 x86_64 content too.

Red Has GSS has attributed this to either ISO file corruption and/or Intermittent sync issue. How do I know that this is happening or is there a way to prevent this from happening?

thanks

Responses

Hey Christopher, Interesting. I run two Satellite servers (both disconnected) one version 5.7 and the second 6.1 in testing. I haven't had any issues with the last batch of isos for 5.7 sync'ing (only running RHEL 6 x86_64), but the batch from 8 April for sat 6.1 (RHEL 7 Server) will not sync at all. The directory structure appears to be missing for that batch. All the checksums were good on the 9 isos. I've got a support case open with them, but haven't gotten anything resolved yet. I take it the checksums were good on the isos you downloaded?

Ryan Holt

Correct, checksusm are ok. Not sure why this keeps happening.

It takes forever to get the content downloaded again.

[re-written/edited]

Chris, I imagine you've unpacked && ingested your iso files in some fashion as I mentioned to you in this thread), correct?

I see you've checked the checksums...

What happens when you unpack the rpms, then make the permissions proper and do a satellite-sync -m /path/to/the/unpacked/specific/channel/ --list-channels

Also, what happens when you run satellite-sync -m /path/to/the/unpacked/specific/channel/

I have an nfs server that only serves the unpacked files to my satellite server. I made sure the permissions to the unpacked rpms on the nfs server was proper. I mounted that nfs mount to the satellite server and then did satellite sync with the --list-channels option for sanity. We did the unpacking on the nfs server, and one directory each for each separate set of channels (segregate out base channels, incremental channels and architectures to separate directories). So, rhel6 64-bit gets it's own directory, same with rhel5 (and for you of course a separate one for 32 & 64 bit), and base channels and incremental channels are segregated. Always (always) do base channels before any incremental (sorry, you probably have already done this, but I write it because others will read this).

Verify in your web gui that the "channel-repodate-bunch" task shows "FINISHED" before ingesting a new set of channels!!! 1) Go to the web gui as an Red Hat Satellite Admin (special role), and 2) look at the top menu, 3) click on "Admin" 4) then on the left, click on "Task Schedules" then 5) look for "Channel-repo-data-bunch" (right side, top link, generally) and 6) make sure it has a status of "FINISHED" before ingesting another set of ISOs

R. Hilton, I've been following what was passed down from that thread where I asked about using a disconnected RH Satellite.

  • Download the ISOs
  • Check MD5
  • Unpack to a certain directory, in my case, /tmp/satdump
  • chmod 07555 /tmp/satdump/*
  • then run satellite-sync -m /tmp/satdump --list-channels

Once that comes back ok, then run

satellite-sync -m /tmp/satdump

The last two commands will depend on for what version of RHEL (either RHEL5 x32, RHEL5 x86_64 and RHEL6 x86_64)

And that is correct, I always do the base channels first and then go onto the child channels. However sometimes I don't realize that there are updated base channels that need to be downloaded, so this might be the smoking gun.

I didn't know about checking the channel-repodata-bunch either, so I can check that the next time I try to re-install isos.

thanks

Hi Chris

I'd avoid the recent incremental channel of [2015-10-10 spanning to 2016-05-11 rhel 6 64-bit], and just take the base channels.

But I see the consternation you're facing is with the 32-bit channels. Did you see that as of 2016-05-11 there's a new set of Base Channels for [RHEL 6 (i386) + EUS + AUS + RHN Tools + Supplementary (Base 2016-05-11)] ? (there are 19 DVDs in that set)

If you are using VMware for your satellite server, make sure to delete any snapshots (to the Red Hat Satellite in VMware) you have, particularly before ingesting them or could easily fill up your ESX storage pool that your Red Hat Satellite is parked on. (This would cause consternation for any server parked on that storage LUN if it fills up).

When you unpack the 19 DVDs, have a place that can accommodate 19 DVDs, and the DVDs unpacked (I see you are using /tmp/). If using NFS, (and if you have enough storage on nfs), you can unpack them on an nfs storage and present it as an NFS mount (I've done this in a pinch on a workstation with a huge local hard drive and then we activate the nfs service that only presents (via /etc/exports) the files to the satellite server and the nfs service is typically off with chkconfig off).

When you do the ingest of the Base Channels, you'll have to use "-c" with every channel you wish to import. You don't necessarily have to import all the channels, some are older channels like 6.0, 6.1 etc. I have at times ingested them all, but I have enough storage on my satellite server to take it all. (I mentioned this a bit in that guide I wrote for you in that other thread I cited earlier)

R.Hilton, as expected nothing new package-wise has shown up for RHEL5 x32. I'm focusing on this one right now. Once I can get that to work and/or fix the issue, then I will move onto the other two (RHEL5 x86_64 and RHEL6 x86_64).

We have a policy, where we don't allow VMWare snapshots to linger more then 48 hrs because they go corrupt and cause problems. So no snapshots lingering here.

I unpack to /tmp/satdump and it is allocated 370 GB of space, so no space issue there.

So once I unpack, I run to scripts against that directory where the unpacked contents live:

satellite-sync -m /tmp/satdump --list-channels

Then I run the following, based on the output from above.

satellite-sync  -c rhel-x86_64-server-5 -c rhel-x86_64-server-hts-5 -c rhel-x86_64-server-productivity-5 -c rhel-x86_64-server-supplementary-5
 -c rhel-x86_64-server-vt-5 -c rhn-tools-rhel-i386-server-5

Right now I'm running both from a script to automate it, however since I'm having so many issues with this, it might be worth it to type it all out to see if that is the issue.

Also it might be worth it to add:

--debug-level 6

To figure out where this is having issues.

I see there are a few other options under the man page for satellite-sync

--steps=[channels|rpms|packages|errate|kickstarts]

However I'm not sure what these really do...

I'm looking thru /var/log/rhn/taskomatic logs and don't see any red flags...Do you have any recommends of other logs to look at or watch to look for red flags?

thanks for all of your help

Chris, just in case this applies (I don't know if it does/does not for your case) ... RHEL 6.8 base channels were released 5/11/2016. make sure to download/install the base channels first before applying any incrementals. If you have applied incremental ISO channel dumps that followed after the milestone of 5/11/2016, it will certainly bugger your channels, as base channels ==must== be ingested first...

I remember reading thru those threads when they came out originally.

We didn't have any issues at that time, and had issues with our RHEL6 x86_64 around January and now having issues with the RHEL56 x32 channels.

Quick update.

I've just imported in all of the Base/Child Channels for RHEL5 x32 for the disconnected Satellite.

The only issue that I ran into is that the taskomatic daemon wasn't running as we had to do some maintenance earlier in the morning, so I had to restart that that. However there were no error messages when importing in all of the content.

I'm running yum commands manually from the CLI to see if I can see if there are any new packages to be installed and so far don't see anything. However this could take time...

Chris,

If (and only if) you really wanted to take in all of any given RHEL Base channel (and you might not), this would make the script: (this is in reply to something you posted a bit ago, and you said you were scripting the ingest...

echo -e 'satellite-sync -m /path/to/unpacked/rpms' `satellite-sync -m /path/to/unpacked/rpms --list-channels | grep 'full support' | awk '{print "-c   " $3}'` >/tmp/verify_this_script_and_change_to_700_perms_if_good

Also, in reply to 20 May 2016 1:28 PM - Regarding what you mentioned on:

--steps=[channels|rpms|packages|errate|kickstarts]

I've never used that option.

Did the ingest of the channels return "Completed"? Did you get "FINISHED" in the "channel-repodate-bunch" 1) Go to the web gui as an Red Hat Satellite Admin (special role), and 2) look at the top menu, 3) click on "Admin" 4) then on the left, click on "Task Schedules" then 5) look for "Channel-repo-data-bunch" (right side, top link, generally) and 6) make sure it has a status of "FINISHED". Then do a yum clean all against a client, and echo n | yum update or yum check-update to see if more packages are available.

R. Hinton,

As an example, I just brought in incremental for 5-20-2016.

[root@ameda4aisrx0232 scripts]# ./satellite_channel_sync_rhel5_x32.sh
14:03:47 Red Hat Satellite - file-system synchronization
14:03:47    mp:  /tmp/satdump
14:03:47    db:  (removed for security reasons)
14:03:47
14:03:47 Retrieving / parsing channel-families data
14:03:47 channel-families data complete
14:03:47
14:03:47 Retrieving / parsing product names data
14:03:48
14:03:48 Retrieving / parsing arches data
14:03:48 arches data complete
14:03:48
14:03:48 Retrieving / parsing additional arches data
14:03:48 additional arches data complete
14:03:48
14:03:48 Retrieving / parsing channel data
14:03:48    p = previously imported/synced channel
14:03:48    . = channel not yet imported/synced
14:03:48    base-channels:
14:03:48         NONE RELEVANT
14:03:48    rhel-i386-server-5:
14:03:48       p rhn-tools-rhel-i386-server-5               53 incremental import from Sun Sep 28 00:00:00 2014 - Sat May 21 00:00:00 2016
14:03:48
14:03:49 Channel data complete
14:03:49
14:03:49 Retrieving short package metadata (used for indexing)
14:03:49    Retrieving / parsing short package metadata: rhn-tools-rhel-i386-server-5 (53)
14:03:49 Diffing package metadata (what's missing locally?): rhn-tools-rhel-i386-server-5
            ________________________________________
Diffing:    ######################################## - complete
14:03:53
14:03:53 Downloading package metadata
14:03:53    Retrieving / parsing *relevant* package metadata: rhn-tools-rhel-i386-server-5 (NONE RELEVANT)
14:03:53
14:03:53 Downloading rpm packages
14:03:53    Fetching any missing RPMs: rhn-tools-rhel-i386-server-5 (NONE MISSING)
14:03:53 Processing rpm packages complete
14:03:53
14:03:53 Importing package metadata
14:03:53    Importing *relevant* package metadata: rhn-tools-rhel-i386-server-5 (NONE RELEVANT)
14:03:53
14:03:53 Linking packages to channels
14:03:55
14:03:55 Downloading errata data
14:03:55    Retrieving / parsing errata data: rhn-tools-rhel-i386-server-5 (NONE RELEVANT)
14:03:55 Downloading errata data complete
14:03:55
14:03:55 Downloading kickstartable trees metadata
14:03:55    Retrieving / parsing kickstart data: rhn-tools-rhel-i386-server-5 (NONE RELEVANT)
14:03:55
14:03:55 Downloading kickstartable trees files
14:03:55    Retrieving / parsing kickstart tree files: rhn-tools-rhel-i386-server-5 (NONE RELEVANT)
14:03:55
14:03:55 Importing channel errata
14:03:55    Importing *relevant* errata: rhn-tools-rhel-i386-server-5 (NONE RELEVANT)
14:03:55    No new kickstartable tree to import
    Import complete:
        Begin time: Tue May 24 14:03:47 2016
        End time:   Tue May 24 14:03:55 2016
        Elapsed:    0 hours, 0 minutes, 8 seconds

It completes successfully with no errors. Once doing the yum clean and yum check-update, no new packages show up.

As for the "channel-repodata-bunch" daemon, I don't see it. Admin > Task Schedules, I only see the following:

channel-repodata-default cleanup-data-default clear-taskologs-default cobbler-sync-default compare-configs-default daily-status-default errata-cache-default errata-queue-default kickstart-cleanup-default kickstart-sync-default package-cleanup-default sandbox-cleanup-default satcert-check-default session-cleanup-default sync-probe-default

What is the content of "./satellite_channel_sync_rhel5_x32.sh" script?

Also looks like that sync did not brought any new packages (see "Fetching any missing RPMs: rhn-tools-rhel-i386-server-5 (NONE MISSING)") and/or erratas (see "etrieving / parsing errata data: rhn-tools-rhel-i386-server-5 (NONE RELEVANT)"), so if your clients are up2date, there should not be any new update available for them.

If you suspect issue with Taskomatic daemon and its channel repodata bunch, check Admin -> Task Engine Status -> Channel Repodata or for more details go to Admin -> Task Schedules -> channel-repodata-bunch and check status column for newest record in the table (should be "FINISHED" and start time should be recent - by default that bunch gets started every minute).

Here is the script that I use to import the ISOs. I have one for RHEL 5 x86_64 and RHEL 6 x86_64.

#!/bin/bash
#
# satellite_channel_sync_rhel5
# last modified 2/24/2016
# chris miller <redacted>
# script to sycn isos unpacks with the RH Satellite
# this is only for RHEL5 x32

mode1=" -c rhel-x86_64-server-5 -c rhel-x86_64-server-hts-5 -c rhel-x86_64-server-productivity-5 -c rhel-x86_64-server-supplementary-5"

mode2=" -c rhel-x86_64-server-vt-5 -c rhn-tools-rhel-i386-server-5"

/usr/bin/satellite-sync -m /tmp/satdump ""${mode1}""${mode2}""

I haven't had time to do more troubleshooting on this issue, however looking at Admin > Task Engine Status shows the status of Finished. I would have to bring in again the RHEL base and child channels and then check this.

However I would prefer to monitor the logs in real time to see if that would give me more information on this issue.

That's odd... On my Satellite 5.6 and 5.7 servers, I have a link in that location named "Channel-repo-data-bunch" with a status of "FINISHED".

Chris, do you get this error at this Red Hat Solution --> https://access.redhat.com/solutions/319983 ? Please check to see if that solution is a fit.

This could be a contributing problem, maybe. Look in the tomcat logs within that area in the satellite web interface, there's a link for tomcat logs. I'd also recommend calling support with this at this point. Cite this discussion area in the ticket.

ADDED #1:

Chris, also see this link https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/5.6/html/User_Guide/chap-Red_Hat_Satellite-User_Guide-Scheduling_Red_Hat_Satellite_Administrative_Tasks.html

ADDED #2:

Chris, In the Satellite web Gui - go to "Admin" (top bar), then on the left, go to "Task Schedules", then:

++ Look at the (probably) top link named "channel-repodate-default" and is there anything in the "Frequency" column? ++ Click on "channel-repodate-default" link and you should see a page with:

Schedule Name:   channel-repodata-default
Bunch: channel-repodata-bunch  # this is a hyperlink
Frequency: Select a schedule:
<various "radio buttons" & drop down menus to schedule something>

Do you see anything like what's in that code block (on the web page in the satellite gui)? If you do, what do you get when you click on "channel-repodata-bunch within that specific page?

ADDED #3:

Chris, Is there anything ... bad ... in /var/log/rhn/rhn_taskomatic_daemon.log ?

ADDED #4

Chris, also see this https://www.redhat.com/archives/spacewalk-list/2015-February/msg00038.html what happens when you do a rhn_satellite restart ? what does the /var/log/tomcat6/catalina.out file give too?

We run the disconnected Satellite. We had an issue with the imports earlier this year. They definitely posted a corrupt incremental. I rebuilt the server from scratch, updated to the latest Satellite packages and then RH delivered a newer Base Channel content. They have also increased the frequency of the incremental updates sometimes releasing an incremental that only contains updates between the last incremental and the current one and and incremental that goes way back to the Current date. I believe they did that with the RHEL 6-x86_64 date 20160521. Do you have the base channel content dowloaded again? You could always unsubscribe all your systems from the base channel and delete the channels and then re-import.

Daniel, the latest base channels for RHEL 6 are 2016-05-11 so if that incremental you cited [(Incremental 2016-05-11:2016-05-21)] was taken before it's parent base channel, that would certainly cause consternation... in case that was the case.

I too have had to rebuild a satellite server when it went south, they're not terribly difficult to rebuild, it's the ingesting of the channels, and rejoining of the client systems that's no fun.

I suspect there may be a way to redo the channels without rebuilding the server if it comes to that... Chris check my last posts

I can see the content under /var/cache/repodata/channel-labels, so I'm thinking that is ok.

I've just noticed under the tomcat logs under /var/log/tomcat5:

cat catalina.out | less

There are two Oracle errors: ORA-12514 and ORA-12528. We have an external Oracle database and I'm currently having the Oracle DBA look into the issues and see if he can resolve them. Perhaps this is the smoking gun that we are looking for.

If I'm checking out the content for channel-repodata-default , this is what I'm seeing:

Schedule Name - channel-repodata-default
Bunch - channel-repodata-bunch
Frequence
Disable Schedule (radio button)
Daily (radio button)
Weekly - Every Sunday at 0:00 
Monthly - Day 1 at 0:00 
Custom Quartz format 0 * * * * ? 

Looking under /var/log/rhn/rhn_taskomatic_daemon.log.1, there was an issue with Oracle.

ORA-28001:  the password has expired 

However I resolved this with the DBA. This was back on May 11th 2016 and there hasn't been any new log entries since then.

If I view the following:

[root@ameda4aisrx0232 ~]# rhn-satellite status
router (pid 4775) is running...
sm (pid 4795) is running...
c2s (pid 4803) is running...
s2s (pid 4812) is running...
/etc/init.d/tomcat5 is already running (4729)
httpd (pid  4761) is running...
osa-dispatcher (pid  4843) is running...
rhn-search is running (4911).
cobblerd (pid 6640) is running...
RHN Taskomatic is running (25144).
[root@ameda4aisrx0232 ~]# rhn-satellite restart
Shutting down spacewalk services...
Stopping RHN Taskomatic...
Stopped RHN Taskomatic.
Stopping cobbler daemon:                                   [  OK  ]
Stopping rhn-search...
Stopped rhn-search.
Stopping MonitoringScout ...
[ OK ]
Stopping Monitoring ...
[ OK ]
Shutting down osa-dispatcher:                              [  OK  ]
Stopping httpd:                                            [  OK  ]
Stopping tomcat5:                                          [  OK  ]
Terminating jabberd processes ...
Stopping s2s:                                              [  OK  ]
Stopping c2s:                                              [  OK  ]
Stopping sm:                                               [  OK  ]
Stopping router:                                           [  OK  ]
Done.
Starting spacewalk services...
Initializing jabberd processes ...
Starting router:                                           [  OK  ]
Starting sm:                                               [  OK  ]
Starting c2s:                                              [  OK  ]
Starting s2s:                                              [  OK  ]
Starting tomcat5:                                          [  OK  ]
Waiting for tomcat to be ready ...
Starting httpd:                                            [  OK  ]
Starting osa-dispatcher:                                   [  OK  ]
Starting Monitoring ...
[ OK ]
Starting MonitoringScout ...
[ OK ]
Starting rhn-search...
Starting cobbler daemon:                                   [  OK  ]
Starting RHN Taskomatic...
Done.
[root@ameda4aisrx0232 ~]#

Wanted to add that I watched catalina.out with tail -f when I restarted the rhn-satellite command. I don't have a complete understanding of what is going on here...I don't see any errors, however I keep seeing busy or failed.

[root@ameda4aisrx0232 tomcat5]# tail -f catalina.out
                com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:964)
                com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:54                                                                                           7)


Jun 1, 2016 8:29:24 AM com.mchange.v2.async.ThreadPoolAsynchronousRunner processReplacedThreads
WARNING: Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@a640a64 (in deadlocked PoolThread) f                                                                                           ailed to complete in maximum time 60000ms. Trying interrupt().
Jun 1, 2016 8:29:24 AM com.mchange.v2.async.ThreadPoolAsynchronousRunner processReplacedThreads
WARNING: Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@a7f0a7f (in deadlocked PoolThread) f                                                                                           ailed to complete in maximum time 60000ms. Trying interrupt().
Jun 1, 2016 8:29:24 AM com.mchange.v2.async.ThreadPoolAsynchronousRunner processReplacedThreads
WARNING: Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@a9a0a9a (in deadlocked PoolThread) f                                                                                           ailed to complete in maximum time 60000ms. Trying interrupt().
Jun 3, 2016 12:51:27 PM org.apache.coyote.http11.Http11BaseProtocol pause
INFO: Pausing Coyote HTTP/1.1 on http-127.0.0.1-8080
Jun 3, 2016 12:51:27 PM org.apache.catalina.connector.Connector pause
SEVERE: Protocol handler pause failed
Throwable occurred: java.net.SocketException: Protocol family unavailable
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:412)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:271)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:258)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:376)
        at java.net.Socket.connect(Socket.java:546)
        at java.net.Socket.connect(Socket.java:495)
        at java.net.Socket.<init>(Socket.java:392)
        at java.net.Socket.<init>(Socket.java:232)
        at org.apache.jk.common.ChannelSocket.unLockSocket(ChannelSocket.java:493)
        at org.apache.jk.common.ChannelSocket.pause(ChannelSocket.java:290)
        at org.apache.jk.server.JkMain.pause(JkMain.java:681)
        at org.apache.jk.server.JkCoyoteHandler.pause(JkCoyoteHandler.java:163)
        at org.apache.jk.server.JkMain.pause(JkMain.java:681)
        at org.apache.jk.server.JkCoyoteHandler.pause(JkCoyoteHandler.java:163)
        at org.apache.catalina.connector.Connector.pause(Connector.java:1032)
        at org.apache.catalina.core.StandardService.stop(StandardService.java:489)
        at org.apache.catalina.core.StandardServer.stop(StandardServer.java:734)
        at org.apache.catalina.startup.Catalina.stop(Catalina.java:602)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:577)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)
Jun 3, 2016 12:51:28 PM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Jun 3, 2016 12:51:29 PM org.apache.coyote.http11.Http11BaseProtocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-127.0.0.1-8080
Jun 3, 2016 12:51:29 PM org.apache.jk.common.ChannelSocket destroy
INFO: Error shutting down the channel 8009 java.lang.NullPointerException
Jun 3, 2016 12:51:29 PM org.apache.jk.common.ChannelSocket destroy
INFO: Error shutting down the channel 8009 java.lang.NullPointerException
Jun 3, 2016 12:51:29 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: Failed shutdown of Apache Portable Runtime
Jun 3, 2016 12:51:43 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on t                                                                                           he java.library.path: /usr/lib/jvm/java-1.6.0-ibm-1.6.0.16.7.x86_64/jre/lib/amd64/default:/usr/lib/jvm/java-1.6.0-ibm                                                                                           -1.6.0.16.7.x86_64/jre/lib/amd64:/usr/lib/oracle/10.2.0.4/client64/lib:/usr/lib
Jun 3, 2016 12:51:43 PM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-127.0.0.1-8080
Jun 3, 2016 12:51:43 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 668 ms
Jun 3, 2016 12:51:43 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jun 3, 2016 12:51:43 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.5.23
Jun 3, 2016 12:51:43 PM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
Jun 3, 2016 12:51:45 PM com.mchange.v2.log.MLog <clinit>
INFO: MLog clients using java 1.4+ standard logging.
Jun 3, 2016 12:51:45 PM com.mchange.v2.c3p0.C3P0Registry banner
INFO: Initializing c3p0-0.9.1.2 [built 17-January-2013 15:41:51; debug? false; trace: 5]
Jun 3, 2016 12:51:45 PM com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager
INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@8620de19 [ connectionPoolDataSource -> com.m                                                                                           change.v2.c3p0.WrapperConnectionPoolDataSource@95201791 [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireR                                                                                           etryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkou                                                                                           tTimeout -> 0, connectionCustomizerClassName -> com.redhat.rhn.common.db.RhnConnectionCustomizer, connectionTesterCla                                                                                           ssName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, factoryCla                                                                                           ssLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 13rnqi09h6hnpuyxmpomx|74517451, idle                                                                                           ConnectionTestPeriod -> 300, initialPoolSize -> 5, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime                                                                                            -> 300, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection -> 0, m                                                                                           inPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@59881e47 [ description -> null, driv                                                                                           erClass -> null, factoryClassLocation -> null, identityToken -> 13rnqi09h6hnpuyxmpomx|7dd57dd5, jdbcUrl -> jdbc:oracl                                                                                           e:oci:@//[address redacted]/RHNSatDB, properties -> {user=******, password=******, driver_proto=jdbc:or                                                                                           acle:oci} ], preferredTestQuery -> select 'c3p0 ping' from dual, propertyCycle -> 0, testConnectionOnCheckin -> false                                                                                           , testConnectionOnCheckout -> true, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; user                                                                                           Overrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 13rnqi09h6hnpuyxmpomx|23a523a                                                                                           5, numHelperThreads -> 3 ]
Jun 3, 2016 12:51:47 PM org.apache.coyote.http11.Http11BaseProtocol start
INFO: Starting Coyote HTTP/1.1 on http-127.0.0.1-8080
Jun 3, 2016 12:51:47 PM org.apache.catalina.connector.MapperListener init
INFO: Registering Hosts
Jun 3, 2016 12:51:47 PM org.apache.catalina.connector.MapperListener init
INFO: Registering WebModule Contexts
Jun 3, 2016 12:51:47 PM org.apache.catalina.connector.MapperListener init
INFO: Registering Servlets
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /127.0.0.1:8009
Jun 3, 2016 12:51:47 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/18  config=null
Jun 3, 2016 12:51:47 PM org.apache.catalina.connector.MapperListener init
INFO: Registering Hosts
Jun 3, 2016 12:51:47 PM org.apache.catalina.connector.MapperListener init
INFO: Registering WebModule Contexts
Jun 3, 2016 12:51:47 PM org.apache.catalina.connector.MapperListener init
INFO: Registering Servlets
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8009 java.net.SocketException: Protocol family unavailable
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8010 java.net.SocketException: Protocol family unavailable
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8011 java.net.SocketException: Protocol family unavailable
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8012 java.net.SocketException: Protocol family unavailable
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8013 java.net.SocketException: Protocol family unavailable
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8014 java.net.SocketException: Protocol family unavailable
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8015 java.net.SocketException: Protocol family unavailable
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8016 java.net.SocketException: Protocol family unavailable
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8017 java.net.SocketException: Protocol family unavailable
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8018 java.net.SocketException: Protocol family unavailable
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8019 java.net.SocketException: Protocol family unavailable
Jun 3, 2016 12:51:47 PM org.apache.jk.common.ChannelSocket init
SEVERE: Can't find free port 8009 8019
Jun 3, 2016 12:51:47 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/7  config=null
Jun 3, 2016 12:51:47 PM org.apache.catalina.connector.MapperListener init
INFO: Registering Hosts
Jun 3, 2016 12:51:47 PM org.apache.catalina.connector.MapperListener init
INFO: Registering WebModule Contexts
Jun 3, 2016 12:51:47 PM org.apache.catalina.connector.MapperListener init
INFO: Registering Servlets
Jun 3, 2016 12:51:47 PM org.apache.catalina.storeconfig.StoreLoader load
INFO: Find registry server-registry.xml at classpath resource
Jun 3, 2016 12:51:47 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 4520 ms

Chris,

sorry, I've been rather busy lately and am pretty busy this week - I think you easily have enough to open a case with Red Hat and point them to the direction of this discussion in your case.

By the way, are you running SELinux on your satellite server?

So I've brought back in all of the RHEL5 x32 ISOs after checking the MD5SUMs and they were all ok.

There were no error messages when importing them back into the RH Satellite.

However, still not seeing any new updates from the RH Satellite.

For troubleshooting, trying to remove a server and bring it back in and having an error with it. Never have seen this before:

Here are the commands that I type: rhn_register > Yes, Continue > Next > User ID (rhnsadm) and Passwd > Next > Next > Next >

Then I get, "Problem sending hardware profile: Internal Server Error

Gets to around 60% before the error message.

I've even tried to reboot to see what would happen and still the same error.

Chris, have you ever used bootstrap, the bootstrap script to bring in a system? 1) make an activation key 2) assign groups, software channels, software packages to the activation key 3) go to the web gui, there's a simple method to generate it from the web gui. Once it's generated, you can go to http://yoursatellitehostname.something.xyz/pub/bootstrap/bootstrap.sh and then within that script, add your activation key (type it in the generated script), and also decide if you want to have the bootstrap script fully update the system upon registration or not. I can give you more details next week, but this week I'm at the RH Summit.

Chris, is there any chance you're having this issue OutOfMemoryError https://access.redhat.com/solutions/43122?

Is ipv6 blacklisted within the various /etc/modprobe.d/xyz.conf files? SIDE ISSUE: I found nfs mounts failed on certain systems where I went out of my way to blacklist IPV6, a while back and I mitigated that.

Also see https://www.redhat.com/archives/spacewalk-list/2012-July/msg00080.html

I looked at the URL for the OutOfMemoryError. I'm not even sure what JVM I'm running on that RHEL Server. I looked online for a way to figure this out and didn't find anything.

I don't have any NFS mounts.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.