rhnpush fails with Internal error 500

Latest response

Getting "Internal Server Error 500" while trying to Upload a package using rhnpush in Redhat Satellite server.

Internal server error 500 Internal Server Error

Error pushing /customsw/ilmt-1-0.x86_64.rpm: !DOCTYPE HTML PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xml:lang="en" xmlns="http://www.w3.org/1999/xhtml" lang="en"><head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
    <link rel="shortcut icon" href="/img/favicon.ico">
    <link rel="stylesheet" href="/css/rhn-base.css" type="text/css">
  <script src="/javascript/prototype-1.6.0.js"></script>

I have checked below link

/var/satellite has enough space, selinux is in permissive mode and also tried using --force to upload. It doesnt help.

OS version - Red Hat Enterprise Linux Server release 6.6


I have a few questions (guesses):
* Are you sure no other satellite sync activity was going on at that time (I'm not actually sure this would cause an issue, but I wonder)
* Can you provide the actual rhnpush command you have used?
* what are the permissions for /customsw/ilmt-1-0.x86_64.rpm?
* have you tried "rhnpush --verbose" for better information?
* have you reviewed the Satellite log?

cd /var/log/rhn
tail -f rhn_taskomatic_daemon.log

I would recommend opening a case. The only time I have had an issue with rhnpush was my own fault, but I still needed Red Hat's assistance to determine what I was doing wrong.

Here's another guess (or five, see the last bit especially) to go along with James' above.

What version of spacewalk-java are you currently running? Now compare that rpm to this list of current spacewalk-java rpms. I ask this because spacewalk-java does -not- get updated with disconnected satellite servers for both rhel5/rhel6, and satelllite versions 5.5 & 5.6.

I was having some very serious issues with my satellite server (not all documented at that discussion), until I upgraded all associated spacewalk-java rpms, and increaased java/tomcat memory usage. ADDED - spacewalk-java rpms are NOT updated automatically with disconnected satellites, not sure about connected satellites

The only other thing I can think of is rhncfg, rhncfg-client, rhncfg-actions recently took a dive and I had to do a yum downgrade on several of my satellite servers AND clients to make configuration channel file ingest work. While this is not the same obviously as an rhnpush, it makes me wonder if it is somehow related. -- ( had to do a yum downgrade because all of my satellite servers are disconnected, and we get channel dumps downloaded, so we get to wait for a while until the new rhncfg rpms are published in iso dumps.)*

Another thought is this discussion http://www.redhat.com/archives/spacewalk-list/2013-February/msg00078.html that if you are running selinux on your satellite, to check permissions of some specific files, and any AVC denial messages: "You might also check /var/log/audit/audit.log for selinux AVC denial messages."

sealert -a /var/log/audit/audit.log 

  • If you are using satellite 5.5, did the table space for oracle fill up?
# su - oracle
$ db-control report

One last thought...

IF you are using VMware, is there any chance you have a snapshot that is killing the storage pool you are on???? (even from another server using the same pool)
- We had that occur once to us, and we deleted the snapshot, and then storage was no longer behaving as though it were full.
- When this occurred to us, the file system on the server did not show as 100%, but we received such errors, and until we deleted the offending snapshots causing consternation, the file system behaved as full until we deleted the snapshot.
- This can be common if a snapshot is taken, and then base channels (rhel 5.11 released recently) or incrementals are ingested.

But James is probably right... you may just need to open a case with Red Hat.

Hope it goes smoother...

Thanks James for you update!

  • I dont think any sync process was running
  • Here is the command I used "rhnpush -c custom-channel-x86_64-server-6 --server localhost /customsw/ilmt-1-0.x86_64.rpm --nosig --verbose"

  • Permission of /customsw/ilmt-1-0.x86_64.rpm is 755

Be sure to review Remmele's response as well - he has a lot of experience with Satellite in some fairly interesting circumstances.

One thing I just considered (based on your response) have you tried using the actual hostname of the Satellite, instead of localhost? I'm not sure how confident I am that it would cause an issue - but, your Satellite listens on a few ports and based on the connection does different things (i.e port 80 is just the html/pub and 443 is the application, etc...) I wonder if the server is not listening (correctly) using localhost. Again .. this is complete guess and it should work fine using localhost, but I would try.

Also - a few of the examples I am discovering show that the "server" parameter is http://localhost/APP - something you may want to try as well.

I tried looking through all of my scripts and I realize I no longer use "rhnpush" but instead do:

spacewalk-repo-sync --channel $CHANNEL_LABEL --url file://$PKGDIR

which is why I am probably not that helpful with rhnpush.

lastly: I would validate that custom-channel-x86_64-server-6 exists. I use spacecmd (which is available from EPEL and is very useful)

spacecmd softwarechannel_listbasechannels | grep "custom-channel-x86_64-server-6"
spacecmd softwarechannel_listchildchannels | grep "custom-channel-x86_64-server-6"