Scene structure recovery is a crucial process for assisting scene reconstruction and understanding by extracting vital scene structure information and has been widely used in smart city, VR/AR and intelligent robot navigation. Omnidirectional image with a 180° or 360° field of view (FoV) provides greater visual information, making them a significant research topic in computer vision and computational photography. However, indoor omnidirectional scene structure recovery faces challenges like severe occlusion of critical local regions caused by cluttered objects and large nonlinear distortion. To address these limitations, we propose a geometric-driven indoor structure recovery method based on planar depth map learning, aiming to mitigate the interference caused by occlusions in critical local regions. Our approach involves designing an OmniPDMNet, a planar depth map learning network for omnidirectional image, which uses upsampling and a feature-based objective loss function to accurately estimate high-precision planar depth map. Furthermore, we leverage prior knowledge from the omnidirectional depth map and introduce it into the structure recovery network (OmniSRNet) to extract global structural features and enhance the overall quality of structure recovery. We also introduce a distortion-aware module for feature extraction from omnidirectional image, allowing adaptability to omnidirectional geometric distortion and enhancing the performance of both OmniPDMNet and OmniSRNet. Finally, we conduct extensive experiments on omnidirectional dataset focusing on planar depth and structure recovery demonstrate that our proposed method achieves state-of-the-art performance.