Request to gateway throws 403 errors only for specific application

Issue

• When backend-redis database was restored from a backup, APICast gateway was throwing 403 http errors for the only application in one of the services

• Following is the error message I see on APICast logs

  2020/13/32 00:00:00 [info] 19#19: *465 backend_client.lua:133: call_backend_transaction(): backend client uri: http://backend-listener:3000/transactions/authrep.xml?service_token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&service_id=xxxxx&usage%5Bhits.10080%5D=1&user_key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ok: true status: 404 body:  error: nil, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, client: 10.10.10.10, server: _, request: "GET / HTTP/1.1", host: "api-apicast-staging.example.com:443"
  2020/13/32 00:00:00 [debug] 19#19: *465 proxy.lua:374: handle_backend_response(): [backend] response status: 404 body: , requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  2020/13/32 00:00:00 [notice] 19#19: *465 cache_handler.lua:47: cache_handler(): apicast cache delete key: 10380:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:usage%5Bhits.10080%5D=1 cause status 404, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, client: 10.10.10.10, server: _, request: "GET / HTTP/1.1", host: "api-apicast-staging.example.com:443"
  2020/13/32 00:00:00 [info] 19#19: *465 errors.lua:17: authorization failed for service 10380, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, client: 10.10.10.10, server: _, request: "GET /mapping/mapping/id/xxxxxxxx HTTP/1.1", host: "api-apicast-staging.example.com:443"
  .....
  2020/13/32 00:00:00 [info] 19#19: *465 proxy.lua:328: [async] skipping after action, no cached key, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, client: 10.10.10.10, server: _, request: "GET / HTTP/1.1", host: "api-apicast-staging.example.com:443"
  2020/13/32 00:00:00 [debug] 19#19: *465 policy_chain.lua:199: post_action(): policy chain execute phase: post_action, policy: Metrics, i: 4, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  2020/13/32 00:00:00 [debug] 19#19: *465 executor.lua:26: log(): executor phase: log, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  2020/13/32 00:00:00 [debug] 19#19: *465 policy_chain.lua:199: log(): policy chain execute phase: log, policy: Load Configuration, i: 1, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  2020/13/32 00:00:00 [debug] 19#19: *465 policy_chain.lua:199: log(): policy chain execute phase: log, policy: Find Service Policy, i: 2, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  2020/13/32 00:00:00 [debug] 19#19: *465 policy_chain.lua:199: log(): policy chain execute phase: log, policy: Local Policy Chain, i: 3, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  2020/13/32 00:00:00 [debug] 19#19: *465 policy_chain.lua:199: policy chain execute phase: log, policy: Routing policy, i: 1, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  2020/13/32 00:00:00 [debug] 19#19: *465 policy_chain.lua:199: policy chain execute phase: log, policy: CORS Policy, i: 2, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  2020/13/32 00:00:00 [debug] 19#19: *465 policy_chain.lua:199: policy chain execute phase: log, policy: APIcast, i: 3, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  2020/13/32 00:00:00 [debug] 19#19: *465 policy_chain.lua:199: log(): policy chain execute phase: log, policy: Metrics, i: 4, requestID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  [32/Xxx/2020:00:00:00 +0000] api-apicast-staging.example.com:8080 10.10.10.10:42942 "GET / HTTP/1.1" 403 32 (0.147) 0