Using a Kayak Rudder
Let me tell you right off the bat: if your kayak doesn't have a rudder, don't worry about it. Rudders are far from essential pieces of gear. If you really decide you want one, you can usually just add one later.
Rudders flip down from their stored position on top of the deck by use of hull lines found alongside the cockpit. You'll then control the rudder with foot pedals.
Believe it or not, the main purpose of a rudder isn't to turn a kayak. It's to keep it running straight when you are paddling in wind, because a kayak will naturally want to turn into wind. That's something that's called weather cocking. If you don't have a rudder, what you'll end up doing is taking extra hard strokes on one side to keep your boat running straight.
Physics of Weathercocking
The reason a kayak weathercocks is because when you are traveling forward, water is hitting the sides of your kayak in the bow, and this water that is hitting your kayak is bouncing off and creating wake. That water that's hitting the sides of your kayak is effectively pinning the bow of your kayak in place.
Now past the center point, or the widest point of your kayak, you won't have water hitting the sides of your kayak. It's more of a low pressure zone back there. So your stern isn't being pinned in place like your bow is. What that means is when you have a side wind, your bow is being pinned, your stern isn't, so the stern will get blown down wind more than the bow, and that's why your kayak turns into the wind. This is where the rudder comes in. It drops in at the stern to hold the stern in place.
Being able to control your rudder with your foot pegs lets you adjust for different strengths and directions of wind, which will cause your kayak to weather cock to different degrees. Of course, it has the added value of helping you to turn your kayak.
So if you're going to be paddling in windy and open areas, you probably will want a kayak with a rudder.