A Stranger In the House of Ayer

Equal to the mystery of how Jack Doorly could allegedly help himself to $58 million is the question of how the Ayer family could fail to notice for so long. After the scandal broke, Caleb Loring III, the family liaison in Essex Street’s office, drew the difficult task of explaining what had gone wrong to his relatives. It was a rare and uncomfortable digression on a path that had been essentially laid out for him at birth. Raised in Belmont and Beverly, Loring earned his college degree and a master’s in business administration from Dartmouth. During Vietnam, he worked for the CIA. Later, he went to work at Bank of Boston, rising over 10 years to become a vice president of commercial lending. When he joined Essex Street in 1982, he followed two generations of family members handpicked to manage the clan’s fortune.

Jack Doorly’s climb to Essex Street was decidedly steeper. He was raised in Lynn by blue-collar parents who struggled to afford his tuition at St. Mary’s, a small Catholic school for boys. Captain of the hockey team, he was effortlessly popular and impressed those around him with his ambition. In 1967, his senior year, his classmates elected him their president.

Doorly enrolled at Salem State College, but was impatient to start working and dropped out before graduation. He found a job in the accounting department at Tucker Anthony Management Company in Boston. A partner there, Gilbert Steward, also happened to be an Ayer family member. In 1973, two months after his wedding, the 24-year-old Doorly joined the Ayers’ company, then located in Winthrop Square in Boston. His duties were modest. One involved converting the office’s accounting system to computer, painstakingly entering figures from countless checkbooks and ledgers. Though grinding, the chore left him with an intimate understanding of every aspect of the family’s holdings.

In 1984, Loring and the family decided to relocate Essex Street to Beverly, closer to where some of the Ayers who managed the operation lived. Doorly claims a family member was supposed to lead the project, but no one showed much interest. So he stepped in, hired an architect, oversaw construction of the new office building, and coordinated the move. The task satisfied his considerable drive, but Doorly couldn’t shake the feeling that his contributions weren’t sufficiently appreciated. “Thus began my work for the company that far exceeded my responsibilities to it,” he’d write later.

Though the seeds of resentment had been planted, Doorly nonetheless continued to impress the family. He and Loring were appointed partners in the trust company on the same day in 1989. And as Doorly’s standing in the company grew over the next decade, so did his eagerness to see it make more money.

For decades, the family had settled for modest returns on nearly risk-free investments like government bonds. Doorly convinced them to branch out. They bought a New Hampshire car dealership in 1994 and a Leominster strip mall. Among their most successful investments was the $20.5 million they spent developing three assisted-living facilities. When construction troubles threatened the project, Doorly fired the contractor. The move showed his ability to take charge, but it also gave him an opening to launch his own company, Orbit Construction, to take over the work, installing his nephew as manager. Doorly took the name from another investment he’d made with family money: a stake in Orbit Sports Marketing, a Florida company that managed charity golf tournaments for clients like Michael Jordan and Fidelity.

As Doorly’s successes mounted, the Ayer family’s presence in the office diminished. Neil Ayer, a notable North Shore equestrian, retired, and another relative, David Ayer, left to run a northern Michigan timber firm established by Frederick Ayer himself. Loring, Doorly claims, was often out of the office focusing on his work for corporate boards, such as Fidelity’s, and his philanthropic interests, including the Episcopal Church and the Beverly Historical Society.

Doorly picked up the slack, overseeing a new staff built from outside the family. An amendment to the trust documents further consolidated Doorly’s power and brought him deeper into the family’s inner circle than anyone who wasn’t blood had ever gone. It gave him the ability to write checks and start businesses for the family without having to seek their permission—a sweeping degree of latitude that put at risk everything Frederick Ayer had sought to protect.

Go on to the next page...