A Stranger In the House of Ayer

The offices of Essex Street Associates are tucked deep into a stand of pine woods in Beverly, in a compact building hidden from the view of motorists speeding along nearby Route 128. It’s a humble headquarters for a company with a rarefied purpose: managing the estate of the late Frederick Ayer, who made his name a century ago as Boston’s most flamboyant multimillionaire.

When Ayer died in 1918, at the age of 95, the Boston Globe described him as New England’s richest man, and the New York Times sketched his rise from a poor shopkeeper to a titan of industry. Neither paper mentioned what Ayer, a fastidious planner (late in life, he took to stowing a bronze coffin in the boxcar of his train, lest his family be troubled by what to do with his remains should he expire en route), had left behind for his family: namely, a fortune of nearly $20 million—the equivalent of $300 million today—and the expectation that it be protected. With that dictate in mind, two of Ayer’s sons placed their inheritance into an investment trust, hoping that, untouched, it would secure the financial future of generations of Ayers.

By last year, the value of that trust had grown to $600 million, and an outsider named Jack Doorly stood watch over all of it. A tall man, thick through the middle and balding on top, Doorly, 56, had worked for the Ayers for most of his professional life, climbing from a position as an entry-level computer operator to president of Essex Street. He oversaw a 10-person team of investment professionals, tax experts, and support staff managing some 300 trust accounts for more than 100 family members scattered around the country. Some had tens of thousands of dollars in Doorly’s care; others, tens of millions.

The details of the family’s investments in government bonds, home mortgages, and real estate crossed Doorly’s desk every day. He tracked debits and credits to the trust accounts on his computer. But he also kept records of his own. A handwritten ledger listed money he had given to his friends and family. There were also documents detailing purchases and projects that the Ayers say they never knew about—things like a Back Bay condo and a Gulfstream jet.

On March 21, 2006, Doorly was at his desk by 7 a.m., the sort of early start that had long been his habit. But this morning felt different. His boss, family member Caleb Loring III, had scheduled a meeting with him and explained that Jamie Totten, a great-grandson of Frederick Ayer and a member of the company’s board, would be joining them. This struck Doorly as unusual. As he walked into a conference room at 9:30, he got another surprise. An attorney had driven up from Boston to sit in on the session.

Unbeknown to Doorly, the lawyer had spent the previous day in Suffolk Superior Court filing a civil suit that alleged Doorly had looted $7 million from the Ayer fortune. In the weeks and months to come, that estimate would swell, as would the acrimony. The Ayers’ voluminous suit—which drags in more than a dozen people Doorly gave or loaned money to—claims that he spent more than a decade spinning elaborate schemes to defraud the family. Doorly declined comment for this piece, but maintains through his attorney that he had been using the money to make loans and other investments that would benefit the Ayers, transactions that were within his authority and never hidden from them. Though criminal charges have not been filed, the Ayers now claim Doorly robbed them of $58.2 million—a number that dwarfs the state’s largest embezzlement case by more than $50 million.

Doorly sat down in the conference room and Loring began peppering him with questions. After about 15 minutes, Loring fired Doorly, and a copy of the legal complaint was thrust into Doorly’s hands. He was given a minute to gather his coat and his briefcase before a large man he’d never seen before escorted him to his car.

His head reeling, Doorly pulled his Cadillac Escalade out of Essex Street’s driveway and started dialing his cell phone, trying to manage the pieces of a secret life that was starting to emerge. His first call was to his mistress, Sarah Hunt (a former Ayer family employee herself), who had been named in the suit as a beneficiary of his stolen money. She couldn’t believe that Doorly had been fired. Neither could Peter Broom, the guy who serviced his jet down in Florida. When Doorly reached him, it was to tell Broom not to contact him through his Essex Street e-mail address. Broom was puzzled by the instructions: He’d always assumed that Doorly was Essex Street, that the vast fortune was his own. It was a common misconception among those in Doorly’s circle, one that he worked hard to cultivate—one that he himself might even have come to believe.

Go on to the next page...