Nonlinear coupling and instability in the forced dynamics of a non-shallow arch: Theory and experiments