Paul Auster occupies a unique position in contemporary American fiction, his reputation resting somewhere between cult writer and literary lion. Although he has a robust readership and plenty of critical champions, he nonetheless lacks the heavyweight status of Don DeLillo or Philip Roth. He has won significant prizes, though they tend to come from overseas, such as the Prince of Asturias award (Spain) or the Prix Médicis Etranger (France). And although he has had four scripts produced, he refuses to sell his books to Hollywood for adaptation.

“I think people in the United States can get a little irritated by my work,” Auster admits, when I ask about his problematical status. “Frustrated, even. They think I’m a clever, clever boy. But I’ve found it’s okay to be in a state of conflict and opposition. You don’t want to be too embraced.”

We have met at Auster’s beautiful Brooklyn brownstone on a hot afternoon to discuss his new novel, Man in the Dark, which just might nudge him nearer the top rung of American literary life. The 61-year-old lives in Park Slope, perhaps the most commodious of New York’s neighbourhoods, a true melting pot of races, classes and lifestyles. It seems a fitting location for a writer whose books are crammed with multiple story lines and shifting identities. (His last novel, Travels in the Scriptorium, features an amnesiac prisoner visited by characters from a manuscript he discovers in his cell.) After a warm greeting, Auster leads me into a lounge decorated with pictures and figurines of typewriters, works whose true significance becomes clear only later in our chat.

“Well, it’s home,” he says, after I compliment him on the house. “At least, as long as I can get up and down the stairs.”

We launch right into a discussion of his new book, and it becomes immediately clear that Man in the Dark, Auster’s 12th novel, is very special to him. Although it contains the cunning narrative strategies for which he is well known — in books such as Mr Vertigo and his breakthrough, 1987’s New York Trilogy — it is his most explicitly political novel. It also turns out to be among his most personal. Set in the present day, it details one long night in the life of a 72-year-old book critic named August Brill, who is mourning the recent death of his beloved wife. Sleepless in the Vermont home he shares with daughter Miriam and granddaughter Katya, he decides to tell himself a bedtime story, to keep from “thinking about the things I would prefer to forget”.

In this story, a small-time Brooklyn magician named Owen Brick wakes to find himself in the middle of a war that rages in a country almost exactly like contemporary America. Brick soon figures out he is embroiled in a second American civil war, which began after the Supreme Court settled the 2000 election in George Bush’s favour. The magician also learns the reason for his presence in this war-torn alternate universe when a group of rebel operatives order him to return to the “real” world to assassinate the man responsible for dreaming up the civil war: August Brill.

Before Brill’s dystopic bedtime fable can reach a conclusion, however, he is overwhelmed by the “ghosts” it was meant to keep away: a series of war stories culled from Brill’s actual life. These involve three harrowing tales set in Europe, as well as a story featuring a lawyer who found himself at the centre of the 1967 Newark riots.

“Those stories are all true,” Auster explains, after dragging on an omnipresent cheroot. “The three set in Europe were told to me by other writers and my publisher, while the Newark story happened to my stepfather. He was actually there when the mayor broke down and cried in the middle of the riot.” But he makes it clear that the book’s origins are much more than a simple desire to retell stories he has heard. On its deepest level, Man in the Dark was born of two profound shocks. The first was the Bush-Gore election. “It was a travesty of justice,” he says, “a coup by legal and political means. The reality is that Al Gore was elected. Which means we’re living in an alternate universe, an alternate reality. Like Owen Brick. In the real world, there would be no war in Iraq, no torture at Abu Ghraib, and possibly — just possibly — no 9/11.”

Another unsettling inspiration for the book is the death of Uri Grossman, the son of the Israeli novelist David Grossman, a good friend Auster calls “a man of extraordinary depth and humanity”. In the summer of 2006, the 20-year-old Uri, a sergeant in the Israeli army, was killed during the invasion of Lebanon. In a bitterly ironic twist, his death came just two days after his father gave a press conference to call for an immediate ceasefire. “His death was devastating to David,” Auster explains, clearly still saddened by the incident. “The idea that this great man should have to suffer the loss of such a fine son is appalling.”

Uri’s story provides the basis of the book’s darkest moment — the slaughter of an idealistic young man in Iraq. It is a moment that will shock readers and should put to rest, at least for now, the charge that Auster is a coolly detached artificer unconcerned with the workings of the world. “I wrote Man in the Dark during a very short period of time,” Auster explains. “Just four months. It kept writing itself. It was the most intense experience I’ve ever had as a writer.”

This turns out to be a departure for Auster, whose customary creative tempo is tortoise-like. “I write everything by hand. Paragraph by paragraph. And then, at the end of the day, I type up what I’ve written — usually 1,000 words — on a typewriter. When the book is done, I’ll do handwritten corrections, then retype it all on the typewriter. I wrote a screenplay on a computer once and hated it so much that I vowed I would never do it again. I got carpal tunnel and wound up having to wear a brace.”

Now all those typewriters surrounding us make sense. It turns out Auster has no dealings whatsoever with the internet. When I express surprise that a man considered to be a master of postmodernism spends no time in cyberspace, Auster concedes this makes him a “strange duck in the big pond of digital life”. “I just don’t trust the internet,” he says. “Most of what’s on it is false. I’ll give readings where the person introducing me will provide information that is simply wrong. I’ll ask them where they got it, and invariably they’ll say ‘the internet’.”

Auster also has an idiosyncratic relationship with the film industry. On the one hand, he categorically rejects offers from film-makers to adapt his novels. “Novels are about inner life, about the experience of being in the world from inside,” he claims. “Films are about life from the outside. Rarely do they touch the depths of novels.” On the other hand, Auster is an avid film-maker himself, co-directing his second script, Blue in the Face, with the indie maestro Wayne Wang, while serving as the sole director of Lulu on the Bridge and The Inner Life of Martin Frost. One gets the sense, however, that the experience of auteur has taken its toll on the author, who currently has nothing in production.

“I really went through the wringer with Martin Frost,” he says wistfully. “Once we were done, we had a screening for the distributor. After saying how much he liked it, he added, almost as an afterthought, that it had ‘no commercial potential at all’. That’s just how it is these days, especially in America. There’s no life left for marginal films.”

Wondering if this reluctance to take on the Herculean task of directing another film might have something to do with his being on the far side of 60, I ask Auster about the trend in his most recent novels to have protagonists who are either infirm or aged — or both. “I don’t know why I’m gravitating toward stories about death,” he says, as if the idea had never occurred to him. “It’s probably nothing more than a function of getting older.” He quickly adds that his next novel has a young, healthy protagonist who comes of age in 1967. “It’s good to inhabit a young person again. Especially in that era.”

We are interrupted by the arrival of Auster’s wife, Siri Hustvedt, a highly acclaimed novelist in her own right. The conversation soon turns to Vermont, where the couple spent many summers with their daughter, Sophie. Suddenly animated, Auster launches into a funny story about a Vermont neighbour, an old codger who lived in a shack at the end of the lane.

“His mailbox read: The Bill Smith. Not Bill Smith, but The Bill Smith.” “Two Es,” Hustvedt adds. “T-h-e-e. Thee Bill Smith.” “As opposed to the other Bill Smith,” I joke.

With this, Auster grows silent, his dark eyes dancing as he ponders the notion of two men struggling over a single identity in a cramped Vermont shack — a Paul Auster story if ever there was one. I don’t know if Thee Bill Smith is still out there, but if he is, he’d better watch out, because there’s a good chance he’s about to get some unexpected company.