In this study, I examine the Fannie Mae (FNMA) single-family mortgage loan modification process over the period from 1999 to 2021. Loan modification rates are higher during the period leading up to the bust periods, while first-time home buyers have lower modification rates than the other homeowners. In addition to loan and borrower characteristics, the results suggest that whether the homebuyer is a first-time home buyer, the unemployment rate, interest rate level, and interest rate slope are important factors of loan modifications. The set of determinants of loan modification rates remain the same between the expansionary and contraction periods, but the impacts of determinants are stronger in the contraction periods. I also examine how the 2008-2009 financial crisis and the 2020-2021 pandemic affects the relation between loan modification rates and the determinants. The findings indicate that the determinants are consistent across various sub-periods, however, the loan modification rates exhibit the strongest sensitivity to changes in the factors during the crisis period or the pandemic. The study provides important contributions to the development of preventive mechanisms against mortgage defaults of homeowners and financial distress of lending institutions.