The Olives was designed by Edmund William Wright for his brother Edward. The avenue from High Street to the house was bordered by tall kaffir apple trees.
In the early 20th century the house was owned by Mr and Mrs Alfred Roberts. They gave parties for interstate and international tennis players. Mr Roberts O.B.E. was Mayor of Glenelg from 1899 to 1902 and again from 1915 to 1917.