The Olbrich Botanical Gardens are some of the nicest public gardens I have seen! It is completely free, although you can pay $2 to enter the enclosed portion of the garden that contains more exotic plants (that wouldn't survive in Madison's climate). It has tons of cool flowers and great landscape design. It is a very popular spot for photo shoots, weddings, and more. They also have a very nice Thai pavilion-- one of three outside of Thailand. When you're done here, head across the street to the Olbrich beer gardens and enjoy a beer and corn hole toss by the lake!