As a person who understands the language, it's hard to separate the language sounds from the meaning of the words being said. So the second you try to just hear to the sounds, it's easy to get distracted and start listening.
Maybe these videos will help shed some light on this mystery?
In 1972, an Italian singer, Adriano Celentano, wrote a song full of gibberish (in any language), but has enough American-sounding syllables to pass as an "English" song. It's called "Prisencolinensinainciusol":
I've only listened to it once and it's already in my head.
Similarly, a pair of London filmmakers, brian & karl, made a short film called "Skwerl" in 2011, where a couple has some pretty convincing English conversations... all while not really saying anything:
Feeling like your ears can relate more to our international friends'? Eh illy cope to dew!