I took the family to Door County about a dozen years ago. We went to Washington Island, but only briefly as a waystation on our way to Rock Island. Rock Island is a small island all of which is a park. You can only take a passenger boat there. We took a picnic lunch with us. Upon reaching the Island, we hiked across it (not a long hike, the kids were all under 10), found a stretch of rocky beach with no one else in sight, and hung out. I loved it, but we're an outdoorsy family.
As for Washington Island, my memory is that it is quite a bit more peaceful than the Bay side of Door County, which can be quite crowded in the Summer. It seemed quite nice, but I didn't see enough of it to judge.
If you decide on Door County, my favorite park was Newport, a large and not very crowded park on the more peaceful ocean side.
Jonah