For nearly 50 years, Richard Yates has been the best-kept secret in American fiction. Largely ignored by the general public — even that dwindling percentage of it that reads books — he has been praised by writers such as Richard Ford and Kurt Vonnegut as being every bit the equal of household names such as Updike, Mailer and Bellow. Some have even gone so far as to call him the true inheritor of F Scott Fitzgerald’s mantle as America’s premier stylist. The hushed reverence used to describe his work is usually accompanied by baffled speculation as to why this writer is not more famous.

The upcoming release of a big-budget film version of Yates’s greatest novel, Revolutionary Road, could finally put an end to this obscurity. Although the book has often been optioned for production since its publication in 1961, it took the high-octane producer Scott Rudin — who reportedly had wanted to film the novel since he was a teenager — and the star power of Leonardo DiCaprio and Kate Winslet, under the A-list direction of Sam Mendes, to finally bring it to the screen. Not that any of this matters to Yates, who died impoverished and alone while living in a dingy apartment in Alabama 17 years ago, an end more fitting to one of his woebegone characters than to one of the finest prose writers of his era.

So why has a writer held in such high regard by peers, critics and a small band of devoted readers failed to find a permanent place in the reading public’s imagination? One clue might be the man himself, whose immense talent was often matched by an equally prodigious appetite for personal and professional self-destruction. Richard Walden Yates was born in 1926 in suburban New York City, and from the very first it seemed as if he was destined for a life of misery. His father, a sales manager for General Electric, was a largely absent figure in the boy’s life, leaving him to be raised by an overbearing, alcoholic, artistically pretentious mother, who suffocated young Richard with a toxic brew of affection and neuroses. After being packed off to boarding school, Yates was drafted into the army and served in the second world war. He spent the post-war years working in hated white-collar jobs, including writing promotional copy and speeches for a business-machine corporation.

Meanwhile, the budding writer suffered a string of rejections from magazines and publishing houses. Stints as an expatriate in Paris and London failed to jump-start his career; a bout of tuberculosis deepened his melancholy and sense of isolation. As early success eluded him, he increasingly sought relief in alcohol, though this only exacerbated his struggles with ill health and depression, as well as contributing to the ruin of two marriages.

Yates fervently hoped his breakthrough would come with the publication of his first book, Revolutionary Road, which tells the story of a restless couple leading lives of quiet desperation in the 1950s Connecticut suburbs. As with most of Yates’s fiction, it was deeply autobiographical. The novel’s hero, Frank Wheeler, was in many ways a mirror image of the author, a man with powerful artistic yearnings who felt smothered by the bourgeois blandness of Eisenhower’s America. Yet despite high expectations on the part of both author and publisher, the book garnered mixed reviews and mediocre sales, and Yates’s bitter disappointment was deepened by its failure to win the National Book Award after making the shortlist.

Following this setback, Yates’s mental problems became more acute, as did his drinking. Although his financial situation was eased somewhat by a university teaching job in New York City and work as a speechwriter for Bobby Kennedy, it took him eight years to complete his next novel, A Special Providence. That, too, failed to elevate Yates into the literary Valhalla, where royalties flow freely and awards are there for the picking. A 1965 screenwriting pilgrimage to Hollywood to work with the independent film-maker Roger Corman ended in disaster, with Yates

suffering a breakdown on Sunset Boulevard that landed him in a psychiatric ward. A residency at the celebrated Bread Loaf Writers’ Conference, in Vermont, also concluded early after the binge-drinking author climbed onto a dormitory roof and started screaming that he was Christ.

The next two decades saw Yates enduring increasingly less glamorous faculty positions in places such as Kansas and Alabama, as well as spending time in a variety of mental institutions for alcohol-related ailments. Friends and colleagues recall him as a man capable of considerable grace and charm who became monstrous when afflicted by the twin demons of alcohol and depression. He got into the habit of getting drunk and placing threatening late-night phone calls to editors who had rejected his work — not the best way to rescue a faltering career. As Blake Bailey memorably describes in his excellent 2004 biography, A Tragic Honesty, Yates in his later years was often indistinguishable from a street person, living in filthy, “crepuscular” flats, smoking 100 cigarettes a day and sporting a beard often “matted with drool and snot”. Time and again, he was rescued from the abyss by friends, colleagues and ex-students. A somewhat less harrowing portrait of the Yates of this era is provided by the Seinfeld creator Larry David, who briefly dated Yates’s daughter Monica and used the gruff, bearded author as the basis for the character of Elaine Benes’s novelist father. Though Yates continued to write through these painful times, creating several unforgettable short stories in the process, his work never “broke out” from a small circle of followers. He died in obscurity in 1992.

Although this is hardly the sort of life that would lend itself to success in our career-savvy times (it’s hard to picture Yates on Oprah’s sofa, at least for very long), there is more to his inability to reach a wider readership than just this deep self-destructive streak. Even more limiting has been his reputation for being unremittingly bleak; possibly the finest, Yates is no doubt his generation’s saddest fiction writer. Though his novels and stories are set in a time when most of America seemed to be on the rise — economically, socially, sexually — his focus remained unwaveringly on those mired in the depths of failure and depression.

And this emphasis was never more piercing than when his subject was his own broken soul. Writers often use their misfortune and pain to fuel their work, but Yates perfected this process. The philosopher Kierkegaard defined a poet as an “unhappy person who conceals profound anguish in his heart but whose lips are so formed that as sighs and cries pass over them they sound like beautiful music”. If this is true, Yates could have been one of the Three Tenors. The critic Robert Towers suggested that it was “as if Yates were under some enchantment that compelled him to keep circling the same half-acre of pain”. Yates himself was acutely aware of this tendency, addressing it head-on in his remarkable short story Saying Goodbye to Sally, as a struggling writer contemplates his crippling gloominess. “What saved him. . . was his knowledge that any number of sanctimonious people had agreed to hang that bleak and terrible label on F Scott Fitzgerald too.” Even if the comparison to Fitzgerald is apt, it takes a sturdy reader to endure the blasted lives that populate a novel such as The Easter Parade, the story of two sisters whose lives seem to be nothing more than attempts to outdo one another in misery.

Another reason Yates never broke through to the front rank of American literature was that he was often at odds with the prevailing taste of his era. The 1960s and 1970s saw a radical shift in literary fashion from the plain-spoken realism in which Yates traded to the “experimental” forms of writers such as Robert Coover and John Barth. Unwilling to adapt his prose and subject matter to changing times, Yates stubbornly adhered to the spare, simple style he’d honed as a chain-smoking, hard-drinking young man. This cost him dearly, especially in the review pages of New York magazines and newspapers, where he was pigeonholed as a relic of a bygone era. Even today, some speak of Yates in the same breath as writers such as John O’Hara, as nothing more than a correspondent from the era of suburban conformity, nuclear families and the “man in the grey flannel suit”. At his best, however, Yates easily transcends this stereotype. His depictions of the Wheelers, or the aptly named Grimes sisters in The Easter Parade, are no more restricted to their time than Fitzgerald’s rendering of Gatsby and the Buchanans is to the 1920s. After all, literary fashions come and go — who reads Coover and Barth now?

It remains to be seen if the film of Revolutionary Road finally lifts Yates into the forefront of contemporary American fiction, though it would be ironic if Hollywood, a world Yates saw as fatally sunny and dishonest, were to prove his saviour. The true measure of a breakthrough, however, will come after the final credits, when readers seduced by Kate and Leo’s dazzling full-colour beauty turn to the book and are confronted by Yates’s world rendered in the grim, unforgiving black and white of the printed page.

Best of Yates

Revolutionary Road (1961)

Yates’s best book. Picture-perfect in its rendering of the malaise at the heart of the American Dream, it is also a timeless portrait of youthful disillusionment.

The Easter Parade (1976)

In many ways the equal of Revolutionary Road, though probably too bleak for some people’s tastes. Reading Yates’s story of two promising sisters whose lives devolve into heartache and missed opportunities is like listening to a scratchy Edith Piaf record on a rainy day.

Liars in Love (1981)

Yates’s best story collection. The title story memorably depicts a young American writer’s affair with a working-class prostitute in a post-war London gripped by an “evil-smelling sulphurous fog that stained everything yellow, that seeped through closed windows and door to hang in your rooms and afflict your wincing, weeping eyes”.

Eleven Kinds of Loneliness (1962)

His first collection of stories from the 1950s. Has ever a writer titled a book more aptly?

Revolutionary Road opens on January 30