In 2000 I went back again. We were spending a holiday nearby and after a few weeks it occurred to me that I could actually go and see how Cadaquès had changed in the meantime. To Figueres by train and then through the mountains by bus. Towards midday we arrived at the one-time fishing village. The anarchists’ hideout. It looked fuller than it had done. The place had grown a little, out to the back, towards the mountains which the bus had just struggled through, down the rather wild serpentine bends. It’s these same mountains that Cadaquès has to thank for its relative seclusion – given that it’s on the Mediterranean coast of Spain. But along the front, by the sea, it still looked just as it had almost 25 years ago.

I spent two legendary, intense summers in Cadaquès – 1976 and 1977 – each time from June until the autumn. It was like a mixture of Joni Mitchell’s Carey and Luis Buñuel’s Exterminating Angel: you just couldn’t escape. Your day would barely have begun and already you’d find yourself back in one of the four or five bars, which you now wouldn’t leave until the next morning. Maybe a quick swim in-between times. English punks were camping in our favourite bay; they didn’t emerge from their tents until darkness had fallen, to be sure not to get any sun. Patti Smith’s Horses was our favourite album in 1976.

Carlos practically only ever talked in Dylan quotes. A good-looking Colombian, he must have been about ten years older than me. And, like so many of his generation, he always had the right catch phrase, the perfect Dylan sound bite at the ready. Most of his always amusing conversational style consisted of verbal flotsam of that sort. And he relied on it when he wanted to impress the young people who used to pour into the famous village in those days, where Duchamp used to love to spend his holidays and where Dalí had built his Egg House.

I had completely forgotten Carlos for years. But on the journey to Cadaquès, little by little I recounted the whole story of those two summers that finally came to an end in autumn 1977, when everything in Germany changed, when Baader, Ensslin and Raspe died in prison in Stuttgart and the entire population was hunting down terrorists. When 1968 was finally over and punk began.

And somehow, in the midst of this story with its many bottles of absinthe, Carlos had reappeared. I saw him before me again, steering me into his flat at the harbour with Dylan quips and inspired fabrications. Bragging about his famous friends, although always with a wry smile and his thoughts somewhere else entirely. Wanting to take us to see Dalí one evening. I hadn’t quite believed him until the eggs really did loom into view in the tiny cove known as Port Lligat.

We were led into a courtyard. Dalí was sitting at the head of a small table for six. I was placed next to Gala. Opposite us were Carlos and my girlfriend; at the other end of the table was an Austrian bodybuilder. Looking back, I wouldn’t be surprised if it was Schwarzenegger. No, it can’t have been. This one was blond. A Wagner overture was playing through a scratchy loudspeaker with the same sound quality as an old gramophone.

This time, in 2000, when we arrived in Cadaquès I made straight for the kiosk next to the L’Hostal, a bar that also hadn’t changed since the 1970s. That’s where I first heard about Schleyer being kidnapped in 1977 by the Red Army Faction – who used to be the Baader-Meinhof Group – and the state crisis that ensued in Germany. All the international newspapers were still to be had at the kiosk. But beside the till there was a book, in English, which took me by surprise: Sex, Surrealism, Dalí, and me by Carlos Lozano. On the cover was a much more beautiful young Colombian than the one I remembered. It was Carlos in the late 1960s, a member of the Paris cast of Hair, whom Dalí had met at the time.

I expressed my surprise. ‘Did you know him?’ asked the woman who owned the kiosk.’We all knew him.’ Everyone in the tiny shop had a story to tell. But why the past tense? ‘He died six months ago, a few months after the book came out. At the end of the year it’s coming out in Catalan and Spanish.’ I learned that in the last years of his life Carlos had had a gallery where he had shown art by various people, including Buñuel’s son, Juan Luis Buñuel.

In those days, in 1977, we were taken aback by the simple fact that Carlos – who never stopped talking about celebs and connections – really was a welcome guest at Dalí’s table. And not just that. As he sat whittling away at a small figure, it became clear from what he said that Carlos was his contact man in Cadaquès; his contact with the outside world. He asked Carlos if there were any interesting people in the town; Carlos pointed to us and the bodybuilder. This was asking a bit much of me. I was nineteen years old; how could I be interesting by Dalí’s standards? Chicken, brushed with red food colouring, was brought to the table. With it, matching strawberry champagne was served. Now they were singing on the Wagner recording. Gala whispered something to me, which I couldn’t quite understand. Dalí turned to my girlfriend.

I was in my third semester, studying Spanish. My special subject was Spanish film, Buñuel in particular. I did actually have some questions for Dalí, who had been involved in a notorious authorship dispute with Buñuel about Un Chien Andalou. But the conversation was far from ordered or linear. Gala was soon tugging at my shoulder again, whispering something. Meanwhile, Dalí was muttering to my girlfriend: ‘Say butterfly.’

After 1977 other interests took my attention, including the visual arts. Dalí didn’t figure any longer for me. In the political history of art in the twentieth century he had been on the wrong side, and somehow it was the same as far as his position in the aesthetic-formal history of art went. Although he had close connections with Warhol and Lacan – two figures who are of such importance today – Greenberg’s anti-Surrealism and his own collected flippancies and political misconceptions had relegated him to a well-lit cul-de-sac in history, perched on a luxurious rubbish tip.

It was only shortly before my return to Cadaquès that he had re-entered my field of vision. Together with other curators and cultural historians – Matthias Haase, Juliane Rebentisch, Martin Saar and Ruth Sonderegger – I had been part of the team that devised a theme for the Graz avant-garde festival Steirischer Herbst: Cross Gender, Cross Genre.This involved performances (with Terre Thaemlitz among others), film screenings (a complete Jack Smith retrospective), an exhibition (with an installation and a documentary video by Mike Kelley) and a symposium. Our premise was that 1960s queer art – questioning or attacking gender boundaries – often attacked those genre boundaries not just in parallel, but in the same breath, in the same action. In Warhol’s Factory, in the Theatre of the Ridiculous, in early glam rock, in Jack Smith’s studios, the distinction between theatre, film, easel painting, performance art and so on was at stake, in just the same way as – and often in step with – the binary nature of the sexes. And Dalí’s name would often come up.

One of our most important collaborators was Kelley. He had long been interested in the path taken by individuals through different, opposing cultural environments. He had a copy of a film by photographer and director Steven Arnold called Luminous Procuresses, which featured the Cockettes, a theatre troupe from San Francisco. It was a stunning movie that meticulously combined the troupe’s hippie-queer sensibility with psychedelic cinematography. Warner Jepson created a highly artificial electronic sound-track, using the effects that were available in the early 1970s. On screen the largely naked Cockettes dance, happily letting their dicks dangle, their bodies covered with glitter from top to toe, wearing chains and cloths, and generally wobbling about. But rather than the music or rhythm they are dancing to, you just hear the shrill, electronic whistling sound. The camera’s crazy, brightly-coloured filters bathe the scene in a sickly psychedelic light.

According to the Cockettes’ manager, the film’s director was ‘Dalí’s darling’. Arnold’s executor, the photographer Stephanie Farago, told us that Dalí called him ‘my prince’. Apparently Dalí co-produced this movie, or at least put up money for it. Arnold must have lived with the artist for some time, as he helped him with the fitting out of the museum that opened in Figueres in 1976. Holly Woodlawn, who was also interviewed by Mike Kelley for our Graz project, could talk for hours about the circle around Arnold in Los Angeles. If the ‘Factory was typical New York’, then ‘that was Versailles’. He must have maintained a salon where things were much the way they were in Luminous Procuresses. Woodlawn, who had been a transvestite performer, also had fond memories of Dalí as a promoter of and host to the transgender culture of the 1970s, even to the extent of paying for Amanda Lear’s sex change operation. He loved it when there was someone running around who he could almost describe as his work. So I arrive in Cadaquès in 2000 with very different interests from a quarter of a century earlier. Although not as easily impressed as in 1977. Once again, Dalí says to my companion: ‘Say butterfly.’ She’s irritated. What’s with the bullshit? What does this guy want? But of course she has no alternative, and sooner or later says tersely: ‘Butterfly.’ Dalí raises his index finger and corrects her. It’s not ‘butterfly’, it’s ‘butterfly’, his intonation rising upwards on the last syllable. Great joke. Gala nudges me in the side and whispers: ‘A genius.’ There are a few other jokes of that calibre and the conversation doesn’t really take off. The bodybuilder is silent. Carlos runs the show, increasingly listlessly. Gala repeats that her husband is a genius. Wagner repeats his leitmotifs. More bottles of strawberry champagne don’t help. We’re ill at ease, and I don’t dare to say: ‘Señor Dalí, you knew Luis Buñuel very well. You used to hang around with him and Lorca in that famous students’ residence in Madrid.’ He doesn’t look as though he would feel like talking about that. We decide to take our leave.

Mike Kelley asked Holly Woodlawn about Dalí’s sexual orientation, and she told him that it was shrouded in mystery: the subject never really came up – pretty much interested in lots of things, you know? In his book Carlos says that although Dalí was 100 per cent gay, he wasn’t at all interested in close sexual contact. In his biography, Ian Gibson confirms that the artist hated being touched. Carlos claims that Dalí had sexual intercourse once with a woman and once with a man. The woman was Gala, the man was García Lorca. In Dalí’s view, sex with women was over-rated and sex with men was painful.

I also discovered from Carlos’s book why young people – like ourselves at the time – were invited to the Egg House. It was Carlos’s job to take them to the old Surrealist. He would then watch them having sex. Gala used to like that, too, and everyone would be happy. Remuneration would take the form of a work by the maestro. Dalí obviously had no inhibitions in his dealings with the art market. We must just not have seemed suitable on that evening. The music must have been wrong.

Along with numerous Dylan quotes, the butterfly story is also in Carlos’s book. He must have told it again and again. So what’s wrong with sticking to your favourite joke?