Resilience of long modes in cosmological observables