The focus of this thesis is on automation in road asset inspection using deep neural networks. Even though some progress has been made in automation of data collection and condition assessment, the amount of manual operation and the cost of road inspection is still considerable. Although, adding automation to the inspection process has been investigated by many studies, most of the works either are focused on traditional computer vision and classical machine learning methods with a low scalability or they have explored novel deep learning methods, but they cover a few number of road assets. Road assets are valuable components of the road infrastructure such as guardrail and pavement. In this thesis a deep learning based framework for classification of a wide range of road assets from guardrail to pavement and slope is introduced as the primary step to design a scalable system for automated road asset inspection. A Convolutional Neural Network (CNN) is used as the computational core of the model for visual analytics. Since the available dataset is challenging due to limited amount of data with high variety of visual scenes under each class, transfer learning is used to obtain the knowledge of a large scale dataset with images including similar basic features and improve the discriminative capability of the model. Accuracy is measured as the rate of correct class prediction. Results are reported for training a model with 2 to 14 classes showing the scalability of the model and 80% test accuracy for the final model with 12 classes. A comprehensive confusion and misclassification analysis is accomplished on the model outputs. Moreover, the proposed model is utilized in a hierarchical structure for designing a multi-level classification model which is able to generate two levels of class predictions to explore the possibility of road asset classification and assessment both in one integrated model.