Error 500 raised while cloning a channel with database error located in /var/log/tomcat*/catalina.out on satellite 5.5 (unique constraint RHNSAT.RHN_CNP_CID_NID_UQ violated)

Solution Verified - Updated -

Environment

  • satellite 5.5
  • rhn-schema-version 5.5.0.17-1.el6sat

Issue

  • Error 500 raised while cloning a channel with database error located in /var/log/tomcat*/catalina.out on satellite 5.5
  • database error while cloning channels on satellite 5.5 :
2013-07-05 10:57:19,706 [TP-Processor2] ERROR com.redhat.rhn.frontend.servlets.SessionFilter - Error during transaction. Rolling back
javax.servlet.ServletException: com.redhat.rhn.common.db.ConstraintViolationException: ORA-00001: unique constraint (RHNSAT.RHN_CNP_CID_NID_UQ) violated
ORA-06512: at "RHNSAT.RHN_CHANNEL", line 1096

Resolution

Solution (unverified)

To solve this issue, proceed as follow :

1- Update your satellite 5.5 to the latest version using these steps : :
#yum install yum-downloadonly
#yum upgrade --downloadonly

when you are ready to update :

# rhn-satellite stop

Backup the database now

# sudo -u oracle db-control backup /path/to/backup/location

run the update :

# yum upgrade -y && service oracle start && spacewalk-schema-upgrade

Make sure you have done the backup as the spacewalk-schema-upgrade assisstant will remind you ; after the upgrade stop oracle and restart satellite :

# service oracle stop && sleep 1m &&  rhn-satellite start
2- Clean up the existing RHEL6 channels :
# spacewalk-remove-channel -c rhn-tools-rhel-x86_64-server-6 -c rhel-x86_64-server-supplementary-6 -c rhel-x86_64-server-optional-6  -c rhel-x86_64-server-6 -u -v -p
# rm -rf /var/cache/rhn/repodata/*-6

*NOTE : this command will also unsubscribe systems from the channels being removed. You can create groups to make sure you can easily re-associate the machines to that channel later. The list of channels here is arbitrary. You can use spacewalk-remove-channel -l to list all channels present on your satellite. *

3- re-sync the RHEL6 content just removed :
# rm -rf /var/cache/rhn/satsync/*
# satellite-sync  -c rhn-tools-rhel-x86_64-server-6 -c rhel-x86_64-server-supplementary-6 -c rhel-x86_64-server-optional-6  -c rhel-x86_64-server-6 --debug-level=6 --force-all-errata --force-all-packages
4- After the syncronization is finished

wait until the repodata has been generated for the channel you wish to clone ; you can use

channels > software channels > red hat channels > click_on_the_name 

and look for

Repo Cache Status:  Completed 

to know when the repodata generation is completed.

Workaround

  • Create the clone channel with no errata then clone the erratas one by one (this will also allow you to identify which errata cause the problem, providing you select them one by one) using
channels > manage software channnel > click_on_label_of_clone > erratas > clone erratas

Root Cause

the cloning of erratas attempts to add packages that already are in the channel

Diagnostic Steps

according to the `/var/log/httpd/ssl_access_log̀ the error 500 was raised at these times :

10.57.135.132 - - [05/Jul/2013:10:55:06 +0200] "GET /rhn/channels/manage/errata/PublishErrata.do?cid=353 HTTP/1.1" 500 6862
[...]
10.57.135.132 - - [05/Jul/2013:10:58:28 +0200] "POST /rhn/channels/manage/Edit.do HTTP/1.1" 500 3088
10.57.135.132 - - [05/Jul/2013:10:59:23 +0200] "GET /rhn/channels/manage/errata/Errata.do?cid=353 HTTP/1.1" 500 3088
[...]
10.57.135.132 - - [05/Jul/2013:11:03:22 +0200] "GET /rhn/channels/manage/errata/Errata.do?cid=353 HTTP/1.1" 500 3088

the errors in the /var/log/httpd/ssl_error_log were:

[Fri Jul 05 11:01:58 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Jul 05 11:01:58 2013] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (*)
[Fri Jul 05 11:02:53 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Jul 05 11:02:53 2013] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (*)
[Fri Jul 05 11:02:53 2013] [error] [client 10.57.135.132] File does not exist: /var/www/html/img/logo.png, referer: https://satellite.example.com/rhn/channels/manage/errata/Errata.do?cid=353
[Fri Jul 05 11:03:36 2013] [warn] [client 10.57.135.132] incomplete redirection target of '/rhn/Login.do' for URI '/index.html' modified to 'https:///satellite.example.com/rhn/Login.do'
[Fri Jul 05 11:03:48 2013] [warn] [client 10.57.135.132] incomplete redirection target of '/rhn/Login.do' for URI '/index.html' modified to 'https:///satellite.example.com/rhn/Login.do'
[Fri Jul 05 11:04:07 2013] [warn] [client 10.57.135.132] incomplete redirection target of '/rhn/Login.do' for URI '/index.html' modified to 'https:///satellite.example.com/rhn/Login.do', referer: https:///satellite.example.com/rhn/channels/manage/errata/Errata.do?cid=353
[Fri Jul 05 11:06:52 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Jul 05 11:06:52 2013] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (*)
[Fri Jul 05 11:07:06 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Jul 05 11:07:06 2013] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (*)
[Fri Jul 05 11:07:19 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Jul 05 11:07:19 2013] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (*)
[Fri Jul 05 11:07:37 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Jul 05 11:07:37 2013] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (*)
[Fri Jul 05 11:08:24 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Jul 05 11:08:24 2013] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (*)
[Fri Jul 05 11:12:13 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Jul 05 11:12:13 2013] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (*)
[Fri Jul 05 11:16:31 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Jul 05 11:16:31 2013] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (*)

the problem is, according to /var/log/tomcat6/catalina.out

2013-07-05 10:57:19,706 [TP-Processor2] ERROR com.redhat.rhn.frontend.servlets.SessionFilter - Error during transaction. Rolling back
javax.servlet.ServletException: com.redhat.rhn.common.db.ConstraintViolationException: ORA-00001: unique constraint (RHNSAT.RHN_CNP_CID_NID_UQ) violated
ORA-06512: at "RHNSAT.RHN_CHANNEL", line 1096
ORA-06512: at line 1

        at org.apache.struts.action.RequestProcessor.processException(RequestProcessor.java:520)
        at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:427)
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
        at com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:99)
        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
        at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.redhat.rhn.frontend.servlets.AuthFilter.doFilter(AuthFilter.java:120)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
        at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:108)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:55)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:736)
Caused by: 
com.redhat.rhn.common.db.ConstraintViolationException: ORA-00001: unique constraint (RHNSAT.RHN_CNP_CID_NID_UQ) violated
ORA-06512: at "RHNSAT.RHN_CHANNEL", line 1096
ORA-06512: at line 1

        at com.redhat.rhn.common.translation.SqlExceptionTranslator.oracleSQLException(SqlExceptionTranslator.java:75)
        at com.redhat.rhn.common.translation.SqlExceptionTranslator.sqlException(SqlExceptionTranslator.java:42)
        at com.redhat.rhn.common.db.NamedPreparedStatement.execute(NamedPreparedStatement.java:118)
        at com.redhat.rhn.common.db.datasource.CachedStatement.executeCallable(CachedStatement.java:513)
        at com.redhat.rhn.common.db.datasource.CallableMode.execute(CallableMode.java:34)
        at com.redhat.rhn.domain.channel.ChannelFactory.refreshNewestPackageCache(ChannelFactory.java:581)
        at com.redhat.rhn.manager.channel.ChannelManager.refreshWithNewestPackages(ChannelManager.java:178)
        at com.redhat.rhn.manager.channel.ChannelManager.refreshWithNewestPackages(ChannelManager.java:165)
        at com.redhat.rhn.frontend.action.channel.manage.PublishErrataAction.execute(PublishErrataAction.java:101)
        at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
        ... 40 more
Caused by: 
java.sql.SQLException: ORA-00001: unique constraint (RHNSAT.RHN_CNP_CID_NID_UQ) violated
ORA-06512: at "RHNSAT.RHN_CHANNEL", line 1096
ORA-06512: at line 1

        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
        at oracle.jdbc.driver.T2CConnection.checkError(T2CConnection.java:676)
        at oracle.jdbc.driver.T2CConnection.checkError(T2CConnection.java:602)
        at oracle.jdbc.driver.T2CCallableStatement.executeForDescribe(T2CCallableStatement.java:572)
        at oracle.jdbc.driver.T2CCallableStatement.executeForRows(T2CCallableStatement.java:765)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1190)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3370)
        at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3476)
        at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4400)
        at com.mchange.v2.c3p0.impl.NewProxyCallableStatement.execute(NewProxyCallableStatement.java:2706)
        at com.redhat.rhn.common.db.NamedPreparedStatement.execute(NamedPreparedStatement.java:115)
        ... 47 more
2013-07-05 10:58:28,591 [TP-Processor7] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.channel.ClonedChannel is setting empty string description
2013-07-05 10:58:28,592 [TP-Processor7] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.channel.ClonedChannel is setting empty string GPGKeyId
2013-07-05 10:58:28,592 [TP-Processor7] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.channel.ClonedChannel is setting empty string GPGKeyFp
2013-07-05 10:58:28,592 [TP-Processor7] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.channel.ClonedChannel is setting empty string maintainerName
2013-07-05 10:58:28,592 [TP-Processor7] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.channel.ClonedChannel is setting empty string maintainerEmail
2013-07-05 10:58:28,592 [TP-Processor7] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.channel.ClonedChannel is setting empty string maintainerPhone
2013-07-05 10:58:28,592 [TP-Processor7] WARN  com.redhat.rhn.common.hibernate.EmptyVarcharInterceptor - Object com.redhat.rhn.domain.channel.ClonedChannel is setting empty string supportPolicy
Jul 5, 2013 11:16:55 AM org.apache.jk.common.ChannelSocket processConnection

Following the analysis of the rhn_cnp_cid_nid_uq constraint, which regulates that the association "package channel architecture" is unique, it appears there was an attempt to push two times the same package in the same architecture in the channel.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments