This paper presents an augmented deep factor model that generates latent factors for cross-sectional asset pricing. The conventional security sorting on firm characteristics for constructing long-short factor portfolio weights is nonlinear modeling, while factors are treated as inputs in linear models. We provide a structural deep learning framework to generalize the complete mechanism for fitting cross-sectional returns by firm characteristics through generating risk factors -- hidden layers. Our model has an economic-guided objective function that minimizes aggregated realized pricing errors. Empirical results on high-dimensional characteristics demonstrate robust asset pricing performance and strong investment improvements by identifying important raw characteristic sources.