[install] Remove numpy dependency
Can the numpy dependency be removed? It needs to be added to the requirements.txt file, which is confusing because we don't depend on it (adding to install_requires and setup_requires doesn't work). I have no idea why the setup.py needs to be so complicated. Is this all for doc building?