One of the ways that lazy critics get around admitting that they don't understand someone's work is to accuse them of being "difficult just to be difficult." This is a version, of course, of "you're just being awkward," or "you're just being perverse." As if there are all these poets out there who, instead of following whatever line of inquiry feels most neccessary to them, wake up in the morning and think, "How can I write something that no-one will understand?"
I don't buy that. There are lots of poets I don't understand and don't in any way "get", even emotionally. JH Prynne is an example - probably the most "notoriously" difficult poet in England - although John Wilkinson and Drew Milne could probably join him in the "don't understand" box for me. Well, are they just being awkward? I don't believe they are; they have agendas and interests that make them write the poems they write in much the same way as, say, UA Fanthorpe or Carol Rumens have agendas that they pursue. I don't have to like what they do; but I can at least accept the possibility that they are pursuing their own goals in their own way without accusing them of bad faith.
That's what accusations of being "difficult for difficulty's sake" amount to: accusing another poet of just playing games, of not being "serious", of not being interested in "communicating." Well, someone who writes a sonnet as opposed to a prose poem is also playing games, of course; the interesting thing about sonnets for the writer is trying to fit your thought into that little space. The writer is always at least partly writing for him or herself, to work out an emotion, an idea, a tricky formal question. This is true for "mainstream poets" and "non-mainstream poets"; it's just that they go about things in different ways.
Some of the difficulty of "non-mainstream poetry", especially in England where it's less visible, is unfamiliarity with the forms involved. Some of it may well have to do with the fact that the poet perhaps is so intent with following an idea to ground, they forget that there might be people watching. But that's not "difficulty for difficulty's sake", and shouldn't be seen as such. There are also some reasons usually behind the awkwardness of some poets: they may be wanting us to wake up to some new possibility, some new and maybe interesting route for feelings and language to go down.
Difficult poems are sometimes difficult because, well, life is difficult.