4

The first order of business is to make sure to avoid potential parallelization.

However to address this more directly the first thing that can be considered is to make find ways that compile objects can be made smaller. As well there is a significant improvements available from doing things like auditing include orders and leveraging forward declarations instead of requiring full definitions and linking of objects. This is often overlooked in template heavy libraries but is just as important there as in classic libraries.

The PCL project (another heavily templated library) ran into this problem back about a decade ago, and with a close look at their project structure was able to reduce compile time resources by an order of magnitude without changing the API at all.