The generally accepted rule is that if it is hanging over the sidewalk it's fair game. I used to get lots of guavas, figs, and lemons that way. Of course I always left the trees full of fruit when I did this and the owners were always putting extras on their wall anyway. Leaving you with only 3 lemons is pretty messed up.