GFS2 files grow a large discrepancy for reported "df" used space compared to other tools such as "du" or "ls"

Issue

The GFS2 filesystem will grow a large discrepancy in what's reported for df usage, compared to what's actually observable on the files using du This discrepancy is not immediate and appears to manifest over time:

Thu May  2 00:40:02 EST 2024
2539596 /mnt/gfs2-mount <--- 2.25 Gb of files on gfs2 according to "du"
Filesystem               1K-blocks     Used Available Use% Mounted on
/dev/mapper/gfs2_vg1-gfs2_lv1 125814336 94107448  31706888  75% /mnt/gfs2-mount <--- ~89.7 Gb reported as used by "df"

Thu May  2 00:50:02 EST 2024
2539596 /mnt/gfs2-mount
Filesystem               1K-blocks     Used Available Use% Mounted on
/dev/mapper/gfs2_vg1-gfs2_lv1 125814336 94107448  31706888  75% /mnt/gfs2-mount

Thu May  2 01:00:02 EST 2024
2539596 /mnt/gfs2-mount
Filesystem               1K-blocks     Used Available Use% Mounted on
/dev/mapper/gfs2_vg1-gfs2_lv1 125814336 94107448  31706888  75% /mnt/gfs2-mount
-----------------------------------------8<----------------------------------------- 
Thu May  2 06:50:01 EST 2024

2539596 /mnt/gfs2-mount <--- 2.5 Gb worth of files
Filesystem               1K-blocks     Used Available Use% Mounted on
/dev/mapper/gfs2_vg1-gfs2_lv1 125814336 94107448  31706888  75% /mnt/gfs2-mount <--- ~89.7 Gb reported used

Thu May  2 07:00:01 EST 2024
2539596 /mnt/gfs2-mount
Filesystem               1K-blocks     Used Available Use% Mounted on
/dev/mapper/gfs2_vg1-gfs2_lv1 125814336 94107448  31706888  75% /mnt/gfs2-mount