Why does my login module get unexpectedly triggered for Timer beans and local EJB calls on JBoss EAP 6?

Issue

We have a client-server setup, with multiple instances of our application deployed onto one server.

We have a custom login module defined for the security domain we have applied to each EAR, intended to be used to authenticate (by username and password) remote EJB calls to the server from our client application. We do not make any use @RolesAllowed, for authorization, so do not specify any on our EJBs.

Normally, this works as expected, with remote calls requiring authentication from the Login Module, and local EJB calls (and scheduled timers, etc) being made as "anonymous", not requiring further authorization, and not triggering the module.

The problem we have is that intermittently we are seeing the container becoming corrupted somehow and the login module is being triggered unexpectedly, causing the timers and certain local calls to require authentication where they don't usually.

These undesirable authentication calls appear to use a leaked or cached username/password combination, so if that arbitrary combination is no longer valid (password expired, for example), all of our timers then fail, and local calls can't be made, instead failing with an EJBAccessException. Each time this happens from then on the authentication attempt is made using the same invalid credentials, across ALL deployed applications on the server, even though the credentials originated from only one of the EARs. This continues until the server is restarted, whereupon it again works as expected.

Specifically, under these circumstances we are seeing authentication being triggered by:

• Timeout of scheduled EJB timers, created within a class annotated as @Singleton and @Startup
• Making a call through the @Local interface of an @EJB injected @Stateless bean

As this is intermittent, and we so far do not know what is causing it, we haven't been able to reproduce it outside of our production environment.

Here is the error that is logged:

12:20:00,036 ERROR [org.jboss.as.ejb3] (EJB default - 10) JBAS014122: Error during retrying timeout for timer: [id=0a8a3ecf-29c8-4706-b51e-8bb55ba1167f timedObjectId=testear.org.jboss.example.SimpleTimer auto-timer?:false persistent?:false timerService=org.jboss.as.ejb3.timerservice.TimerServiceImpl@957b3a3 initialExpiration=null intervalDuration(in milli sec)=0 nextExpiration=Fri Jan 19 12:21:00 CST 2018 timerState=RETRY_TIMEOUT info=MyTimer]: javax.ejb.EJBAccessException: JBAS013323: Invalid User
        at org.jboss.as.ejb3.security.SecurityContextInterceptor$1.run(SecurityContextInterceptor.java:60)
        at org.jboss.as.ejb3.security.SecurityContextInterceptor$1.run(SecurityContextInterceptor.java:48)
        at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:86)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
        at org.jboss.as.ejb3.timerservice.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:101)
        at org.jboss.as.ejb3.timerservice.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:111)
        at org.jboss.as.ejb3.timerservice.task.CalendarTimerTask.callTimeout(CalendarTimerTask.java:63)
        at org.jboss.as.ejb3.timerservice.task.TimerTask.retryTimeout(TimerTask.java:210)
        at org.jboss.as.ejb3.timerservice.task.TimerTask.run(TimerTask.java:167)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_111]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1]

Interestingly, the container does not have permission to call the timeout method on the timer bean.