This paper proposes a novel panoramic stereo hyperspectral imaging system with theoretical analysis and experiments. The system has two line scan hyperspectral cameras and is set above a rotary stage. This provides the panoramic property of the system, and stereo hyperspectral panoramas are created at each 360 degree rotation. Furthermore, a novel algorithm is proposed, in which multi-band Census transform is implemented to estimate the disparity map information from the acquired stereo hyperspectral images. The system is utilizable for change detection, target detection and classification applications because it provides both disparity map and spectral property for any pixel or area in the image.