The substance you are seeing is the coffee chaff, which is a natural by-product after roasting. The chaff is a parchment-like layer in the center of a coffee bean that will look like a light tan color flake. When the beans are ground, the chaff will look like lighter specks scattered amongst the coffee. They are totally safe to brew.