I quit caffeine almost a year ago and I haven't looked back. For me caffeine caused anxiety - similar to what you have described as apprehensions and jittery feeling. To be able to still enjoy the flavour of coffee I have switched to decaf - if you get a chance, do try it - you'd be able to enjoy the smell of coffee without any side-effects of caffeine.