A special Thanksgiving service was certainly held on April 30, 1789 in honor of Washington’s inauguration. Following the ceremony at Federal Hall, in which nobody remembered to bring a Bible, the new President and both houses of Congress walked to St. Paul’s. President George Washington, in the two years in which New York was the national capital, regularly attended St. Paul’s and his pew in the north aisle is preserved as a tourist attraction.What's more fascinating to me is that the Tetragrammaton, the Four Letters, the Hashem, the Name, "YHWH", is part of the decor of this place. Here are two links to two different pictures (I'm not posting them as I don't have the rights to them, thus I'm merely linking to them).
In this first link, note the two tablets containing the Ten Commandments near the center of the picture, at the bottom of the center pane of the window; straight up from that, at the top of the window's outer arch, is the decoration containing the Tetragrammaton. If you let your browser show the image at full-size, you can actually see a blurry image of the Tetragrammaton.
And here is a black-and-white close-up of that decoration: