I planted potatoes in my garden and...

You can harvest new potatoes when they are about the size of a quarter or maybe slightly larger. Or you can wait a while and let the potatoes get bigger. However, you should harvest all of your potatoes once the vines die, or the potatoes may rot.