My best guess is that speaking about these inner worlds is "emo" or something of that sort. So even if one were to understand where you are coming from, one might not want to talk about them or deny them. You sort of have to "be there" to really understand what it means? Not to sound hipster-ish or anything.
Socially, it is just awkward unless the people you talk to also want to delve deeper into music. It is sort of like talking about the Mona Lisa/The Last Judgement/Oath of the Horatii/Pilgrimage to Cythera with an art person; interesting if the other person is really interested in everything about it, painful if another person just want to look at the art piece. You can talk about the colors, the textures, and about how the Mona Lisa was drawn multiple times and that the actual picture is really tiny. It would be all for naught if the other person doesn't care.
An architect could marvel at how the Golden Gate Bridge/Petronas Towers/Taipei 101/Burj Khalifa was built, while another would just want to see.
Unless another wants to get deeper into these meanings, it is a hassle communicating about it.