Argo,it has been a while since you made your post. hopefully you might revisit your post. First, you are under feeding your tree. Your tree's leaves have lost their luster, and now how a dull green. At the very minimum your tree will need fertilizing one a month, once every two weeks during the growing
season is much better for growth. Fertilizer spikes are not a good fertilizer to use on a citrus tree. Use a water soluble fertilizer with a high nitrogen content. Try to find a fertilizer with a formula having a 5-1-3 ratio. Also your tree is infected with an insect called scale, and needs to be sprayed with a good horticultural oil. Don't worry about some of the little fruit being discarded by the tree. This is very common. Only about 1 to 2 percent of blooms will produce fruit, if every flower
produced a fruit the tree would be crushed under it own weight. It is very common for many of the original small fruits to drop from the tree. A citrus tree only retains the amount of fruit it is able to bring to maturity. good luck with your tree.