IPv4 Link-Local Addresses 169.254.0.0/16

Issue

I have been investigating about Network 169.254.0.0/16.
These are my findings which may explain the behavior you are experiencing:

1) Network 169.254.0.0/16 is a "reserved network" named IPv4 Link-Local Addresses:

RFC 3927 [1] describes how a host may
automatically configure an interface with an IPv4 address within the
169.254/16 prefix that is valid for communication with other devices
connected to the same physical (or logical) link.

IPv4 Link-Local addresses are not suitable for communication with
devices not directly connected to the same physical (or logical)
link, and are only used where stable, routable addresses are not
available (such as on ad hoc or isolated networks). This document
does not recommend that IPv4 Link-Local addresses and routable
addresses are configured simultaneously on the same interface.

2) You set enable_isolated_metadata = True, so metadata service is enabled:

# The DHCP server can assist with providing metadata support on isolated
# networks. Setting this value to True will cause the DHCP server to append
# specific host routes to the DHCP request. The metadata service will only
# be activated when the subnet does not contain any router port. The guest
# instance must be configured to request host routes via DHCP (Option 121).
# This option doesn't have any effect when force_metadata is set to True.
enable_isolated_metadata = True

Cloud-init uses 169.254.0.0/16 network --> 169.254.169.254 (metadata agent)

3) You are using internal network with CIDR: 169.254.100.0/24 and cloud-init which triggers 169.254.0.0/16 as specified in the RFC.
When you set enable_isolated_metadata = False, which deactivate cloud-init and /16, you are able to configure /24.

I hope this helps to understand what is the behaviour you experienced.

[1] https://tools.ietf.org/html/rfc3927