I frequently refer to non-knitters when writing blog posts. I've decided we need a new word for that group, something that defines them separately from our sub-culture. I'm thinking of something similar to "muggles", the name J. K. Rowling uses, that refers to a person who lacks any sort of magical ability and was not born into the magical world.
I think knitting is also a form of magic, especially when observed by non-knitters.
I used a word generator and came up with a few suggestions nontes, knitnos, noniters, knitnions and noles.
If you want to play too, the generator is here. Let me know in the comments if you come up with something good. The idea of a special name for "them" tickles my fancy!