Because musicians and students must analyze music, their perspective on the difficulty of a work is often different from the perspective of someone, like me, who listens for the mere pleasure of sound. For example, decades ago, I read that Webern's work was notoriously difficult, but when I sat down to hear it, I found the music accessible, compelling, haunting in its beauty. Of course, that ease of hearing was the result of hard playing by the musicians.
In a similar way, after having read of how difficult Elliott Carter's quartets can be to perform, I heard last night the Arditti Quartet play Carter's Quartet No. 1, and found the music accessible, enjoyable, beautiful. Right away, I heard the piece again, and again found it no more difficult than, say, the quartets of Bartok or Ligeti.
For those of us freed from the task of mastering a piece of music, hearing it can be an uncomplicated pleasure, and one that we owe to dedicated performers. I wish this idea were common, because I often suspect that many casual listeners avoid modern music (the music of a century ago!) because they have heard of how challenging it can be. And it often is a challenge -- to performers. To the rest of us, the task is often easier; all we need to do is to listen.