dwarf orange trees and problems

I have a dwarf orange tree and it is growing quite fine but has a few issues. One thing we notice it that is blooms very well, starts producing small green oranges that look healthy and grow and then they all drop off. Not one little orange stays on to a normal colour yet the tree looks great. We put it outside in the summer and bring it in the winter. Any ideas from those who had dwarf orange trees of what we can do. We live in the northern parts.

we encounter the same problem with our dwarf lemon tree

we put it outside and the rain and wind knock off a lot of the young fruit

we normally get 5 or so lemons per harvest (indoors or outdoors) and so when we see 20 or so tiny lemons reduced to a fourth of that number we don't get concerned

whatever you can do to cut down on wind exposure helps