How to quantify and tune route cache?

Issue

• How to quantify and tune route cache?
• We are trying to get to over 95% route cache efficiency (i.e. less than 5% route cache lookup misses)
• Need to know the route cache garbage collections settings needed to avoid or minimize cache misses (slow lookups).
• Traffic load is running non-stop (from a dedicated load generator) at a steady rate of 123k packets/sec. The src/dst IPs are drawn randomly from predefined pools. In particular, source IPs are drawn from a /24 subnet and destinations from another /17 subnet. Thus there are 2^(8+15) possible combinations (8M) of src/dst flows. All the traffic is UDP passing/routed through the system (not destined to it) in one direction only and rp_filter is set to 0. On average, there is one packet from each flow (per a unique combination of IP src/dst values) passing thru the system every 68 seconds.
• The peak number of entries I saw was around 7,000,000 and there was still around 10% misses (ie. slow lookups). I’m also noticing that if I extend the gc_interval (double it or make is 4x the default) to reduce the route cache frequency of entries removal (that is undesired in my case), it does run less frequently (as expected) but seems to actually clean up more entries thus reducing some of the benefits of gc interval increase. I need to know the route cache garbage collections settings needed to avoid or minimize cache misses (slow lookups).
• I use one of the following commands to monitor the number of entries in the route cache and the lookup hits vs. misses:
  lnstat -s1 -i10 -c-1 -f rt_cache
lnstat -s1 -i1 -c-1 -f rt_cache
• Can you comment on why the route cache GC seems to be taking so much time?
• Can we see from vmcores the state of the route cache and any of the rt_cache stats?
• Do you think some tuning of the route cache garbage collector would be appropriate here, there are tunables like gc_elasticity, gc_thresh and gc_timeout