Question about glasses?

I got glasses like a few days ago and my eye doctor told me I don't have to wear them all the time, however I have been wearing them all the time. Today, I noticed that when I took them off everything wasn't like it used to be. Like, I don't remember everything being so blurry. Is it possible that by wearing my glasses all the time, my eyesight could've gotten worse than before (meaning , my eyesight without glasses), or am I just worrying?? Please help, thanks!