Where ever you go, there you are...

... still the center of your own universe, still a captive in the Matrix, the Ring of
Doom remains intact.

Different cultures and authors have chosen to associate the elements with different points of the compass. The earliest reference to this I have seen is in the way the tribes of Israel were directed to camp around the temple while Moses led them through the wilderness. On the east side was Judah the Lion. Ephraim was on the west was the ox or bull. Reuben was on the South was Aquarius and Dan was on the north as an eagle(a well known alternative symbol for Scorpio. From where they were geographically these associations made perfect sense.

East/LEO/FIRE - Dawn
WEST/TAURUS/EARTH - Africa
South/Aquarius/Air - Sahara, and the disease carrying winds of Set
North/Scorpio/Water - Mediterranean

Frank Baum has his imaginary world set up thus:
WEST/YELLOW/TIN/AIR/Aquarius - Winkie Country. It is distinguished by the color yellow which is worn by most of the local inhabitants as well as the color of their surroundings. This was the country ruled by the Wicked Witch of the West before Dorothy melted her with a bucket of water in The Wonderful Wizard of Oz; since then, the Tin Woodman has ruled the Winkies as their Emperor.

EAST/BLUE/SILVER/WATER/SCORPIO - Munchkin Country. They are described as being somewhat short of stature, and wear only blue. The Wicked Witch of the East is the ruler of Munchkin Country. The young girl's house lands on top of The Witch. The Witch, who had been ruling over the Munchkins, is killed instantly with only her feet sticking out from under the house. The Witch was wearing magical shoes (made of silver in the book).

NORTH/Purple/GOLD/EARTH/Taurus - Gillikin Country is the northern division of L. Frank Baum's land of Oz. It is distinguished by the color purple worn by most of the local inhabitants as well as the color of their surroundings.
* The forest of the Winged Monkeys, creatures that obey whoever wears the enchanted Golden Cap
* The palace of the Good Witch of the North (never actually mentioned, but assumed to exist because the Good Witch of the North rules this particular province)
* The ruby palace of Gayelette

SOUTH/RED/COPPER/FIRE/LEO - Quadling Country is the southern division of L. Frank Baum's Land of Oz. It is distinguished by the color red, worn by most of the local inhabitants as well as the color of their surroundings.
*Glinda, the Good Witch of the South
*The Cowardly Lion kills a giant spider, who is terrorizing the animals in a forest, and he agrees to return there to rule them.
In the books, Glinda is depicted as a beautiful young woman with long, rich red hair and blue eyes, wearing a pure white dress. Glinda lives in a palace near the southern border of the Quadling Country, attended by one hundred beautiful maidens (twenty-five from each country of Oz). She also employs a large army of female soldiers, with which she placed Ozma on the throne of Oz at the end of The Marvelous Land of Oz. Men are not prominent in Glinda's society.

CENTER/GREEN/Emerald Tablet of Hermes _ The Oz books describe the Emerald City as being built of green glass, emeralds, and other jewels. The buildings were decorated with gold as well.

JKRowling seems to place them thus:
Air/West - Ravenclaw house.

Water/East - Slytherin

Fire/North - Gryffindor

Earth/South - Hufflepuff

Modern Pagan(and I am going to lump Wiccan in with the whole) practice seems divided on whether air or earth should go in the north.

Geographically it would make sense if the elemental/directional correspondences changed to reflect your place on the earth. In that view a person practicing on the east coast of the US would put water in the east(the Atlantic), fire in the south, earth in the west(the continent), and air to the north. Or if you lived far enough north to see the aurora you might even put fire in the north.

Fortunately, for me at least, we are talking about a deeper, more universal understanding. This is my case for Earth being in the north/spiritual realm -
1) The qualities that make "earth"/solid, different from the other elements are its density, and resistance to flow. The further north you go the more solid, the earth is, due to frost. The hardest form of earth is the diamond(as far as I know), and in the north we have diamond-like snow.
2) The alchemists say the highest form of matter is gold, and the highest form of gold is the philosopher's stone, which is not a stone at all, but a fine white powder - like snow.
3) While we are the God at the axis mundi of our own known, microcosm, the one who stands as axis of the world is God of the Earth, and all the world is known to him. He sees all that passes beneath and above. He is the Observer.

He may have two "wives", one representing the point of contact with the earth(the earthen mound usually being an under statement for the nearest holy mountain), and the other representing the point of contact with the heavens(the pole star or circumpolar constellation). We see this in the two forms of Arianrod's name. Arianrhod - the Welsh arian, "silver," and rhod, "wheel". Alternately, the earliest form of the name may have been Aranrot "Round Hill, or Mound".

The Wise amongst ancient people surely knew that the axis mundi was as far North (speaking only of the Northern hemisphere) as north existed on this earth - the shortest distance between the earth and the celestial pole.

Symbolically it may have been seen as the nearest holy mountain, or oracle, because at these places the unknown could be seen more clearly. But in truth it was in the distant north, behind the north wind, so to speak.

If you have ever stepped back and just watched yourself, perhaps observing your body instincts wanting to react to some stressor, then you have "become the observer". That is your spiritual place of power. In that state you can know the unknown, the occult, the hidden, heal yourself and others, remote view, all that. That is embodying your God. And where is that God found? At the North Pole. He IS the North Pole, but he is also Santa, and he has gifts of occult knowledge in his "sack". And since that God is also within, you don't even have to travel.

These things, to me, equate the element of earth with north and Spirit.

There was also an understanding I read once that explained how ancient thinkers believed the immaterial became manifest. They thought it descended through the different elements, from finest to solid. So it would travel from ether to fire, to air, then water then solid. My understanding of this is that they may have actually thought it traveled through the actual elements, or they may have been using the elements as symbolic of the actual process. In the symbolic scenario it would go something like this. There is a spark of inspiration (fire) received from the All(ether) that is then contemplated(thought/air) which produces emotions(water - the chemicals of emotion in the blood stream) and the chemicals of emotion stimulate the cells to make proteins, matter. In this scenario we literally become what we have given focus to. The next step would be that we draw to ourselves what we are, then the think contemplated on should show up in our explicit reality. Since this is how I have been taught manifestation actually works, I am sort of reconstructing what the old teaching seems to have really meant. To me this is why - fire is associated with innocence, beginnings, and the passion that comes with epiphany; air with thought and intellect; and water with emotion. Earth or solid then becomes a thing we can actually experience and gain wisdom from - making known the unknown - and wisdom is the diamond, or fruit of the earth.