Microstructural classification is typically done manually by human experts, which gives rise to uncertainties due to subjectivity and reduces the overall efficiency. A high-throughput characterization is proposed based on deep learning, rapid acquisition technology, and mathematical statistics for the recognition, segmentation, and quantification of microstructure in weathering steel. The segmentation results showed that this method was accurate and efficient, and the segmentation of inclusions and pearlite phase achieved accuracy of 89.95% and 90.86%, respectively. The time required for batch processing by MIPAR software involving thresholding segmentation, morphological processing, and small area deletion was 1.05 s for a single image. By comparison, our system required only 0.102 s, which is ten times faster than the commercial software. The quantification results were extracted from large volumes of sequential image data (150 mm², 62,216 images, 1024 × 1024 pixels), which ensure comprehensive statistics. Microstructure information, such as three-dimensional density distribution and the frequency of the minimum spatial distance of inclusions on the sample surface of 150 mm², were quantified by extracting the coordinates and sizes of individual features. A refined characterization method for two-dimensional structures and spatial information that is unattainable when performing manually or with software is provided. That will be useful for understanding properties or behaviors of weathering steel, and reducing the resort to physical testing.