chronyd keeps changing "Selected source"

Issue

• After updating chrony package to version 4.1, we keep seeing below messages in the log:

Apr  8 01:45:48 server1 chronyd[1463839]: Selected source 120.132.170.41
Apr  8 01:52:54 server1 chronyd[1463839]: Selected source 10.65.129.12
Apr  8 02:24:32 server1 chronyd[1463839]: Selected source 120.132.170.41
Apr  8 02:44:48 server1 chronyd[1463839]: Selected source 10.65.129.12
Apr  8 03:42:18 server1 chronyd[1463839]: Selected source 120.132.170.41
Apr  8 04:03:49 server1 chronyd[1463839]: Selected source 10.65.129.12
Apr  8 04:21:02 server1 chronyd[1463839]: Selected source 120.132.170.41
Apr  8 08:55:14 server1 chronyd[1463839]: Selected source 10.65.129.12
Apr  8 09:22:48 server1 chronyd[1463839]: Selected source 120.132.170.41
Apr  8 09:57:14 server1 chronyd[1463839]: Selected source 120.132.170.41
Apr  8 10:04:07 server1 chronyd[1463839]: Selected source 10.65.129.12
Apr  8 10:10:07 server1 chronyd[1463839]: Selected source 120.132.170.41

• source ntp servers in chrony.conf file. Top 3 servers are external, and bottom two are internal sources:

server 120.132.170.41 iburst prefer 
server 205.32.88.241 iburst prefer
server 205.33.243.2 iburst prefer

server 10.65.129.12 key 12 iburst 
server 10.62.137.11 key 12 iburst 

• We are expecting external ntp servers selected first (using "prefer" in chrony.conf) , and if no external one is available, one of the internal ones to be selected. With chrony version 3.5, it was always picking an external ntp server as expected. After moving to 4.1, it was picking internal or keep changing the selected source.