Nancy, is this really thin aluminum? I've had better luck punching holes in really thin stuff. Especially tiny pieces that are hard to hold. You need to be careful. If one gets away from you and the drill bit grabs it, you can get cut badly. Ask me how I know...
What size hole? Since the first went well, you might have plugged up the flutes of the drill bit with melted aluminum. Look at the bit carefully and see if the flutes are clear.
As Kent said, using a punch to mark the hole keeps the bit from wandering and breaking.