Tomcat 7 and LDAP binding broke after upgrading to RHEL7.3

Posted on

We are using RHEL7.3 and Tomcat 7.0.69 and we lose the LDAP search-binding after upgrading from Redhat Enterprise 7.2 to 7.3 When looking inside the tomcat log, new java severe errors event just appear, something like; The web application [/birt-viewer] appears to have started a thread named [Thread-4] but has failed to stop it. This is very likely to create a memory leak. We are using Tomcat essentially for BIRT. Tomcat authentication integrated with Apapche still work fine, but the java .jsp request.getRemoteUser() return a null user instead of the uid.

Here the full log when tomcat is restarted:

Dec 05, 2016 1:53:02 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8080"]
Dec 05, 2016 1:53:02 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8444"]
Dec 05, 2016 1:53:02 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-8009"]
Dec 05, 2016 1:53:02 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
SEVERE: The web application [/birt-viewer] appears to have started a thread named [Thread-4] but has failed to stop it. This is very likely to create a memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [org.apache.axis.utils.XMLUtils$ThreadLocalDocumentBuilder@2d11f387]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [org.apache.xerces.jaxp.DocumentBuilderImpl@83dcc5b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7bee60ea]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [org.eclipse.birt.report.context.BirtContext@3559ddb2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@17f68d55]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [com.ibm.icu.impl.ResourceBundleWrapper@714fd4e4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@79ae0317]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7bee60ea]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [org.eclipse.birt.report.context.BirtContext@261f5dd]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [org.apache.axis.utils.XMLUtils$ThreadLocalDocumentBuilder@2d11f387]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [org.apache.xerces.jaxp.DocumentBuilderImpl@21de4ce1]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@17f68d55]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [com.ibm.icu.impl.ResourceBundleWrapper@714fd4e4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@79ae0317]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7bee60ea]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [org.eclipse.birt.report.context.BirtContext@2e3954ca]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [org.apache.axis.utils.XMLUtils$ThreadLocalDocumentBuilder@2d11f387]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [org.apache.xerces.jaxp.DocumentBuilderImpl@27865ca1]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@17f68d55]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [com.ibm.icu.impl.ResourceBundleWrapper@714fd4e4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@79ae0317]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@7bee60ea]) and a value of type [org.eclipse.birt.report.context.BirtContext] (value [org.eclipse.birt.report.context.BirtContext@2a448a91]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [org.apache.axis.utils.XMLUtils$ThreadLocalDocumentBuilder@2d11f387]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [org.apache.xerces.jaxp.DocumentBuilderImpl@792e727c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@17f68d55]) and a value of type [com.ibm.icu.impl.ResourceBundleWrapper] (value [com.ibm.icu.impl.ResourceBundleWrapper@714fd4e4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/birt-viewer] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@79ae0317]) and a value of type [com.ibm.icu.util.ULocale] (value [en_US]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 05, 2016 1:53:02 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8080"]
Dec 05, 2016 1:53:02 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8444"]
Dec 05, 2016 1:53:02 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8009"]
Dec 05, 2016 1:53:02 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-8080"]
Dec 05, 2016 1:53:02 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-8444"]
Dec 05, 2016 1:53:02 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-8009"]
Dec 05, 2016 1:53:03 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Realm/Realm} Setting property 'debug' to '10' did not find a matching property.
Dec 05, 2016 1:53:03 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Dec 05, 2016 1:53:03 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Dec 05, 2016 1:53:03 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8444"]
Dec 05, 2016 1:53:04 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Dec 05, 2016 1:53:04 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 857 ms
Dec 05, 2016 1:53:04 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Dec 05, 2016 1:53:04 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.69
Dec 05, 2016 1:53:04 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /var/lib/tomcat/webapps/birt-viewer
Dec 05, 2016 1:53:10 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://www.eclipse.org/birt/taglibs/birt.tld is already defined
Dec 05, 2016 1:53:10 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Dec 05, 2016 1:53:13 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat/webapps/birt-viewer has finished in 9,524 ms
Dec 05, 2016 1:53:13 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /var/lib/tomcat/webapps/host-manager
Dec 05, 2016 1:53:13 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Dec 05, 2016 1:53:13 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat/webapps/host-manager has finished in 193 ms
Dec 05, 2016 1:53:13 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /var/lib/tomcat/webapps/manager
Dec 05, 2016 1:53:14 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Dec 05, 2016 1:53:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat/webapps/manager has finished in 183 ms
Dec 05, 2016 1:53:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /var/lib/tomcat/webapps/demo-java
Dec 05, 2016 1:53:14 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Dec 05, 2016 1:53:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat/webapps/demo-java has finished in 171 ms
Dec 05, 2016 1:53:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /var/lib/tomcat/webapps/ReportEngine
Dec 05, 2016 1:53:14 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Dec 05, 2016 1:53:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat/webapps/ReportEngine has finished in 167 ms
Dec 05, 2016 1:53:14 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Dec 05, 2016 1:53:14 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8444"]
Dec 05, 2016 1:53:14 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Dec 05, 2016 1:53:14 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 10322 ms

Also here a sample of our JNDIRealm config inside server.xml (all was working perfectly before the upgrade):

<Realm className="org.apache.catalina.realm.JNDIRealm" debug="10"
        connectionURL="ldap://localhost:10389"
        connectionName="uid=admin,ou=system"
        connectionPassword="*******"
        userBase="ou=users,dc=Mycompany,dc=com"
        roleBase="ou=groups,dc=Mycompany,dc=com"
        roleSearch="(uniqueMember={0})"
        userSearch="(uid={0})"
        roleName="cn"/>

If someone can help us out with this issue, it would be greatly appreciated!

Thanks!

Finnaly downgraded tomcat to version 7.0.54 temporary fix the problem. Still need to identify what have changed in the configuration files. Most likely in /etc/tomcat/catalina.policy and /etc/tomcat/catalina.properties

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.