Simultaneous Decoherence and Mode Filtering in Quantum Channels: Theory and Experiment