If you want to avoid building “everything” at once (including all recursive dependencies) you will need packaging to provide the binary packages for the dependencies.
Dirk is exactly right. Even if you only want to use travis for CI (which is easy enough to set up), once you have enough dependencies, your build will timeout if you don’t install them from pre-compiled binaries (bottles in the homebrew case).