References Goes Wrong After Sorting

I have a problem with my references when sorting in excel. I have been searching for the problem, but did not found a useful solution.
I'm using excel 2003.

Let me try to explain my problem with a simple exampel.

I have a column with numbers and one with formulas pointing at these.
A B
2 =A1
4 =A2
3 =A3

Now I want to sort column A, and the column B should keep pointing at the original number. So I want this:
A B
2 =A1
3 =A3
4 =A2

But I get this:
A B
2 =A1
3 =A2
4 =A3

This output is what I would expect if I used absolut reference($).

Using offset(B1;0;-1) will work, but I cannot use this, since I will add/delete rows and the reference is not neccessarily next to it. Plus it can be a rather large sheet.

I cannot use dynamically name definition either, because it is intended for other users, and this will be too much work for them. What I basically need is a simple formula they can enter.

My sorting is done via a VBA macro, so if there is anything I can do through macro/VBA that will be okay as well, since I can do this without the user seeing it.