When Story assumed his spot on the bench, his boss was legendary Chief Justice John Marshall. Marshall's chief concern as Chief Justice was court unity. He was the first to recommend that the Court pass down one single opinion rather than each justice assembling his own opinion. He also felt that all of the justices should become as familiar as possible with each other's personalities and habits. To that end, he introduced the idea that all of the justices should live and dine together in the same boardinghouse during the six weeks a year that the court was in session. (William Rehnquist recounted an amusing anecdote about the court's dinners together in this 2001 speech.) According to Story, "the Judges here live with perfect harmony, and as agreeable as absence from friends and from families could make our residence." By the way, I think it would be incredibly awesome if they still did this. (And made a reality show about it.)
But those weeks in Washington weren't entirely about discussing the Constitution. Even Supreme Court justices need a little recreation. So they didn't just work together, live together, and eat together -- they also partied together. In a letter to his wife during his rookie year on the Court, Joseph Story recounts the following:
Two of the Judges are widowers, and of course objects of considerable attraction among the ladies of the city. We have fine sport at their expense, and amuse our leisure with some touches at match-making. We have already ensnared one of the Judges, and he is now (at the age of forty-seven) violently afflicted with the tender passion. Being myself a veteran in the service, I take great pleasure in administering to his relief, and I feel no small pride in remarking that the wisdom of years does not add any thing of discretion to the impatience, jealousies, or doubts of a lover.
I'm not sure which justice he's talking about here - the one who's 47 at the time of the letter is Thomas Todd, who was happily married at the time to James Madison's sister-in-law. If I had to guess, I'd say Bushrod Washington, but I really haven't got much to go on there, except that he is near that age. Wikipedia doesn't seem to have much about most of these guys' marital status. I guess they figure if we wanted to date them, that ship sailed about 200 years ago.
But a special note to ladies of the 21st century: if you're ever out drinking in Washington, I hear David Souter is single. Just don't offer to buy him a Diet Coke. You will only confuse him.