A Stranger In the House of Ayer

To grow and keep watch over their father’s fortune, Frederick Ayer’s sons made use of a distinctly Bostonian institution: the family trust. Used to safeguard assets, a trust creates a sort of wall between beneficiaries and their money. Post-colonial ship captains often left control of their estates in the hands of faithful advisers, a local practice that foreshadowed the first American trust company setting up shop in Boston in 1893. Over the next half-century, trusts were woven into the fabric of the city, a fact that Time magazine noted in 1936, shortly after the Ayers established theirs: “A Bostonian who is not either a beneficiary or a trustee of at least one personal trust fund is liable to find himself at a distinct social disadvantage.”

Social disadvantage was long a bugbear for Frederick Ayer. His father died in 1829, when Frederick was seven, and money was tight throughout his childhood. As a young man, he was more focused on getting to work than getting an education. After some time as a shopkeeper, he joined his brother’s patent medicine business in 1855. Medicinal elixirs purported to be cure-alls—and like any miracle in a bottle, they were big sellers. Out of a factory in Lowell, the brothers eventually turned out some 630,000 doses a day in varieties that claimed to fight everything from baldness to whooping cough (part of their popularity no doubt owed to the cocaine in some recipes). Before long, they diversified into textile mills. In 1878, Ayer’s wife and brother both died. He threw himself headlong into his work, eventually growing his operation into the largest textile business in the world.

With his staggering fortune secure, Ayer spent the second half of his life trying to scrub away any sign of his modest beginnings. But he found New England’s Brahmin circle tough to penetrate. In his sixties, he wed a socialite some 30 years his junior and moved from Lowell to Boston, where he hired Louis Comfort Tiffany to design their new Back Bay home. Intended to announce the Ayers’ entry into polite society, the Commonwealth Avenue mansion was not well received. Its pink granite exterior and oily art glass looked garish to the Yankee neighbors, who inside would have found a foyer decorated with gold leaf, marble, and a 7-foot-long stuffed jaguar.

In 1905, the Ayers purchased a sprawling summer estate in Pride’s Crossing, naming it Avalon. Initially, they had trouble fitting in on the North Shore, as well. Their property neighbored the estate of the Lorings, who had helped establish the area as an aristocratic enclave 50 years earlier. Loring family members included a Civil War general who became director of the Museum of Fine Arts, as well as the judge who replaced Oliver Wendell Holmes on the Massachusetts Supreme Court. Though the Ayer progeny married well—a young Army lieutenant named George Patton proposed to an Ayer daughter in the crushed-velour library of the Boston mansion in 1910—the family’s social standing wasn’t cemented until 1943, when an Ayer granddaughter married Caleb Loring Jr., the father of Doorly’s boss.

The Ayers might also have thought the union between the two families could help safeguard their fortune. The Lorings, after all, had been wealth management experts since the advent of trust law in this country. Augustus P. Loring published one of the earliest textbooks on the subject in 1898, and the family’s financial acumen kept them entrenched among the area’s business elite for decades. The elder Caleb Loring served as a top executive and loyal confidant for Fidelity’s billionaire owners, the Johnsons, and remains to this day one of the city’s most active philanthropists. (It was at a charity event for the USS Constitution that he tasted his first slice of pizza. “So,” he said, holding it up for inspection. “This is pizza.”)

The wealth that kept generations of Ayer descendants insulated from the workaday world also bred in some a measure of shame about the déclassé provenance of their fortune. That is, if they knew of its origins at all. A young Ayer once asked her Victorian aunt where their wealth came from, only to be rebuffed. “We will not speak of it,” she was told. That the Ayers had money, and always would, was all that mattered. “Americans who were rich enough to live in Pride’s Crossing liked to have money,” says a family member today. “But they didn’t like to contemplate where it came from.”

Go on to the next page...