My wife and I have been to Leu Gardens a couple times, and we always enjoy it, even though were not gardening nuts.
The garden is very peaceful, and there're many places to sit and enjoy beautiful views. Every time we visit, there's something new to maintain our interest.
If you go, don't miss the tour of the Leu House (every 1/2 hour.) The best time to go is in Jan-Feb when the Camellias are blooming.
I you want to have lunch nearby, try the Panera Bread at Fashion Square Mall, or Athenian Garden Cafe on Orange Avenue.
Pros: Cheap, No crowds, Location