Some thieves use a tactic called ATM skimming to steal your card number and PIN. While this tactic has been used for several years in large cities and tourist areas, there is some evidence that the tactic is now being used more in smaller areas.
ATM skimming depends on a device that clips onto the outside of the ATM and sits on top of the cardslot. The device is camouflaged to look like a normal part of the ATM. When you insert your card, the skimmer reads the magnetic strip as it goes past into the regular ATM card reader. The skimmer does not interfere with the operation of the ATM.
The thief will often also put up a small camera positioned to watch your hand as you enter your PIN. The camera might be camouflaged as a brochure holder. These devices often have built-in antennae to send the numbers wirelessly to the thief's computer in a nearby car.
See this Snopes.com article for pictures of an ATM skimmer in place.
Banks regularly check their ATMs for these devices but if you see something suspicious, you should call the Bank immediately. If your bank offers a "smart card" (a card with an embedded chip), those are currently secure from most skimmer attacks. Regardless, you should always monitor your bank statement carefully to make sure that all the transactions on it are really yours.