Time does describe a physical system, in the sense that we can use it to measure change, and if it is involved in measurement, it has a unit. However, most of the disagreements aren't really about time, but about calendar and dates. Dates make sense because from year to year, every local region can find similarity from the date of one day on one year, to the following date of another year, in terms of climate, responsibilities, tasks, etc. It is easy to compare May vs. December, it is not so easy to compare 365.47 vs. 1048.63, in those terms.
Everything we do here is dictated by the year cycle. It makes sense to catalogue the passage of time using that cycle to maintain consistency.
Once we are free of that cycle, it obviously becomes a less important thing to consider.