It has been labelled as Hogwarts for hippies and was once owned by a legendary American newspaper baron