Did some troubleshooting on why this wasn’t working.
3 of the databases have a date range without times in the “Dates” field
But the Google Calendar integration uses Datetimes for the “Dates” field in one of the databases (the one for not-all-day events).
I’ve added a feature request that can fix this Expose Default Time Values in Date Ranges for Sorting