Causal modeling enables robust counterfactual reasoning and interventional mechanisms to make predictions across different hypothetical scenarios. Nevertheless, uncovering causal relationships from observational data presents a considerable challenge, as unobserved confounders, limited sample sizes, and variations in distributions can give rise to misleading cause-effect associations. Models relying on these relationships may perform poorly when spurious correlations do not hold in test cases. To address these challenges, researchers augment causal learning with known causal relations. This dissertation first investigates the incorporation of domain knowledge in structure learning by introducing additional constraints that convey qualitative knowledge about causal relationships. The experimental designs are specifically equipped to evaluate the role of domain knowledge. Secondly, a concept-driven approach is implemented to determine the advantages of incorporating concept-level prior knowledge. Given the invariant nature of causal relationships, the study then showcases the broader applicability of incorporating domain knowledge by employing a machine learning method for learning adsorption energies, illustrating the advantages of harnessing domain knowledge to obtain invariant molecular representations in catalyst screening. Finally, a novel approach is introduced to enhance robustness and out-of-distribution generalization by leveraging gradient agreement across different environments to identify reliable features. Collectively, these experimental designs advance causal discovery and robust machine learning by utilizing prior knowledge and relational invariances, paving the way for future research on integrating domain knowledge and invariance principles into the learning process.