This is probably the easiest and most convenient place to paddle near Baton Rouge (other than the city lakes). The launches are concrete, the parking lot is large, and the river is really nice, with lots of wildlife (I counted over 30 alligators one trip). This is actually inside the Maurepas Swamp National Wildlife Refuge.
From the launch, go south under the US 61 bridge and then continue under the railroad bridge as well. The low height of the railroad bridge will keep most of the motor boat traffic out, which is nice. Continue down the river as far as you want to go. There are a couple of side bayous that can be explored as well. There are some really nice old cypress trees along the river. I've seen all manner of wading birds back here, including numerous egrets and Roseate Spoonbills. Even came across a Swallow-tailed Kite one day. It can be very quiet if you get far enough away from the highway and train tracks.
If you decide to go north, watch out for speed boats -- I'd advise staying very close to the bank. Check out Tchackachou Bayou, which is the first bayou on the right. It's really pretty, although it can get choked with plants in the summer. If you look on the map, it looks like you can take this bayou all the way around in a loop to the canal next to the highway to return to the launch, and you can, but it's a pretty tough paddle. The water has to be pretty high and you'll have to push over weeds and obstructions.