Wow - thanks @Luxonis-Brandon - I never expected such a detailed reply!
I only have experience with the Sony IMX219 sensor used in the cheaper wide-angle cameras for the Jetson Nano (e.g. this 160° one from Waveshare).
One cool option might have been to make it possible to swap out the sensor and optics. Despite its low price, this is possible with the current (V2) standard Raspberry Pi camera like so:
So companies like Waveshare sell alternative modules with different optics and sensors (e.g. this 160° module) that you can snap on and off the main camera PCB as needed.
Something for the OAK-D mark 2 