How to force xfs filesystem to shutdown?

Issue

• Unmounting an xfs filesystem failed with EBUSY.

# umount /mount_point
umount: /mount_path: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

• Could not identify any user process accessing the filesystem:

# fuser -muv /mount_point
                     USER        PID ACCESS COMMAND
/mount_path:               root     kernel mount (root)/mount_point

# lsof | grep "/mount_point"
# 

• Unmounted the filesystem with --lazy option to detach that from the filesystem hierarchy.

# umount -l /mount_point
# 

• After that, deactivating the volume group still failed with below error:

# vgchange -an vg01
  Logical volume vg01/lv01 contains a filesystem in use.
  Can't deactivate volume group "vg01" with 1 open logical volume(s)

• Looks like some unknown process stil have an access on the filesystem after that filesystem was detached from the filesystem hierarchy with --lazy umount option. How do I force an xfs filesystem to shutdown when unmounting that filesystem failed?