CSI provisioner pods crash with invalid memory address or nil pointer dereference

Issue

• CSI provisioner pods are often restarted. Example of affected cluster:

csi-cephfsplugin-provisioner-8995b9bfd-dlth8       6/6     Running   0          24d
csi-cephfsplugin-provisioner-8995b9bfd-g2jx8       6/6     Running   54         25d
csi-rbdplugin-provisioner-5784795949-6mdnk         6/6     Running   84         38d
csi-rbdplugin-provisioner-5784795949-zvvmt         6/6     Running   0          34d

• The logs of the affected pods show the following error messages:

I0613 19:07:07.406975       1 csi-provisioner.go:121] Version: v2.0.4-0-g0b9348770
I0613 19:07:07.407098       1 csi-provisioner.go:135] Building kube configs for running in cluster...
I0613 19:07:07.422046       1 connection.go:153] Connecting to unix:///csi/csi-provisioner.sock
I0613 19:07:07.422704       1 common.go:111] Probing CSI driver for readiness
W0613 19:07:07.426181       1 metrics.go:333] metrics endpoint will not be started because `metrics-address` was not specified.
I0613 19:07:07.427758       1 csi-provisioner.go:210] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I0613 19:07:07.429658       1 leaderelection.go:243] attempting to acquire leader lease  rook-ceph/rook-ceph-cephfs-csi-ceph-com...
I0613 19:07:07.440916       1 leaderelection.go:253] successfully acquired lease rook-ceph/rook-ceph-cephfs-csi-ceph-com
I0613 19:07:07.741463       1 controller.go:820] Starting provisioner controller rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-5c77db96f5-88l8f_e3d38e6f-d515-41e6-ac8c-9b71eded2132!
I0613 19:07:07.741521       1 clone_controller.go:66] Starting CloningProtection controller
I0613 19:07:07.741530       1 volume_store.go:97] Starting save volume queue
I0613 19:07:07.750406       1 clone_controller.go:84] Started CloningProtection controller
I0613 19:07:08.542567       1 controller.go:869] Started provisioner controller rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-5c77db96f5-88l8f_e3d38e6f-d515-41e6-ac8c-9b71eded2132!
E0613 20:22:07.723078       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 20:22:07.723099       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 20:37:08.033996       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 20:37:08.044631       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 20:52:08.036934       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 20:52:08.043814       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 20:52:08.045168       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 21:22:07.767930       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 21:22:08.063227       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 21:37:08.076996       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 21:37:08.088475       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 21:52:08.074251       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 22:07:08.093536       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 22:22:08.095421       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 22:52:07.852336       1 shared_informer.go:781] unrecognized notification: <nil>
E0613 23:37:07.884800       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 60 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x16ad960, 0x24bace0)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89
panic(0x16ad960, 0x24bace0)
    /home/travis/.gimme/versions/go1.15.linux.amd64/src/runtime/panic.go:969 +0x175
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/client-go/tools/cache/shared_informer.go:774 +0x194
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000574760)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000514f60, 0x1a7ee60, 0xc0004a0000, 0x167a201, 0xc00049e000)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000574760, 0x3b9aca00, 0x0, 0x1, 0xc00049e000)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000550900)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0002b4a70, 0xc00056cea0)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x136ce54]

goroutine 60 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x16ad960, 0x24bace0)
    /home/travis/.gimme/versions/go1.15.linux.amd64/src/runtime/panic.go:969 +0x175
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/client-go/tools/cache/shared_informer.go:774 +0x194
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000574760)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000514f60, 0x1a7ee60, 0xc0004a0000, 0x167a201, 0xc00049e000)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000574760, 0x3b9aca00, 0x0, 0x1, 0xc00049e000)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000550900)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0002b4a70, 0xc00056cea0)
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
    /home/travis/gopath/src/github.com/kubernetes-csi/external-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65