[StackRox] Troubleshooting why the RHACS (StackRox) Collectors are in a CrashLoopBackOff state after upgrading a AKS Cluster to v1.18.14
Issue
- Upon upgrading an AKS cluster to 1.18.14, the following components are reporting error messages similar to the ones demonstrated below;
Collector
- Version 3.0.13 Collectors are in a
CrashLoopBackOff
state and the log output is reporting the following error message;
2021-03-03T14:31:32.372915296Z curl: (7) Failed to connect to sensor.stackrox port 443: Connection refused
2021-03-03T14:31:34.388864035Z curl: (7) Failed to connect to sensor.stackrox port 443: Connection refused
2021-03-03T14:31:36.404901575Z curl: (7) Failed to connect to sensor.stackrox port 443: Connection refused
2021-03-03T14:31:38.420885414Z curl: (7) Failed to connect to sensor.stackrox port 443: Connection refused
2021-03-03T14:31:40.436909253Z curl: (7) Failed to connect to sensor.stackrox port 443: Connection refused
2021-03-03T14:31:40.441229256Z Failed to download kernel module.
2021-03-03T14:31:40.441463156Z All attempts to download the kernel module have failed.
2021-03-03T14:31:40.441582256Z The kernel module may not have been compiled for version 5.4.0-1039-azure.
2021-03-03T14:31:40.441717156Z Error: Failed to find kernel module for kernel version 5.4.0-1039-azure.
2021-03-03T14:31:40.441842956Z
2021-03-03T14:31:40.441933756Z Please provide this complete error message to StackRox support.
2021-03-03T14:31:40.442036156Z This program will now exit and retry when it is next restarted.
2021-03-03T14:31:40.442180256Z
2021-03-03T14:31:40.539128611Z HTTP Status Code 000
Sensor
- Version 3.0.50.1 Sensor log output reports the following error message;
common/sensor: 2021/03/03 16:55:51.640781 sensor.go:270: Error: Sensor reported an error: receiving initial cluster config: rpc error: code = PermissionDenied desc = not authorized: no identity in context
main: 2021/03/03 16:55:51.640839 main.go:68: Fatal: Sensor exited with error: receiving initial cluster config: rpc error: code = PermissionDenied desc = not authorized: no identity in context
Central
- Version 3.0.50.1 Central log output reports the following error message;
pkg/grpc/authn: 2021/03/03 16:39:27.346977 interceptor.go:22: Error: Error extracting identity: could not parse service cert token: time difference 1m1.692312192s > 1m0s detected
pkg/grpc/authn: 2021/03/03 16:39:27.500219 interceptor.go:22: Error: Error extracting identity: could not parse service cert token: time difference 1m1.744584045s > 1m0s detected
pkg/grpc/authn: 2021/03/03 16:44:32.386261 interceptor.go:22: Error: Error extracting identity: could not parse service cert token: time difference 1m1.689826684s > 1m0s detected
pkg/grpc/authn: 2021/03/03 16:44:32.617708 interceptor.go:22: Error: Error extracting identity: could not parse service cert token: time difference 1m1.742961089s > 1m0s detected
pkg/grpc/authn: 2021/03/03 16:49:36.244335 interceptor.go:22: Error: Error extracting identity: could not parse service cert token: time difference 1m1.72795163s > 1m0s detected
pkg/grpc/authn: 2021/03/03 16:49:36.458158 interceptor.go:22: Error: Error extracting identity: could not parse service cert token: time difference 1m1.767501771s > 1m0s detected
pkg/grpc/authn: 2021/03/03 16:54:49.466061 interceptor.go:22: Error: Error extracting identity: could not parse service cert token: time difference 1m1.714695046s > 1m0s detected
pkg/grpc/authn: 2021/03/03 16:54:49.695463 interceptor.go:22: Error: Error extracting identity: could not parse service cert token: time difference 1m1.757255829s > 1m0s detected
Environment
- StackRox Version - 3.0.50.1 or earlier.
- Orchestrator - Azure Kubernetes Service (AKS) - v1.18.14.
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.