Theoretical-computational modeling of quantum vibrational states of polyatomic systems in complex environment