Modeling of linear and nonlinear coupling in multiple-mode fiber optic transmission with MIMO signal processing