How to troubleshoot Prometheus remote write on RHOCP 4

Issue

• Prometheus remote_write is falling behind and metrics are arriving late at the remote endpoint.
• The lag between the newest sample ingested and the newest sample successfully sent continues to grow.
• Desired shard count does not increase despite visible lag.
• Samples are being dropped, retried, or failing to send to the remote endpoint.