Hi Jack,
I feel that is similar to the “duckiebot” developed by duckietown.
Also, I think that image processing with a raspberry is a little complex because you would lost many frames (works like 10 fps if you apply some filters with decent quality (640x480)).
It could be interesting use a odroid ux4 or some main board with more power and a lower cost.