I agree with you. Personally, it would be great to just get a kind of minimal installation of macOS similar to ubuntu’s ros-*-ros-base. In fact, installing just the base tools compiles much much quicker and doesn’t require SIP to be disabled. Here is how I used to do it. We could probably just create a vcs file that just contains the base packages.
I’m currently waiting for them to release a 16" MacBook Pro with an Apple Silicon chip to attempt building on that.