Lansdowne House – which, with its grounds, occupies the whole of the south side of the square - was started in the middle of the eighteenth century by the Earl of Bute, from designs by Robert Adam.
It is known, as Wraxall phrased it, that this magnificent residence "exposed him to very malignant comments respecting the means by which he had reared so expensive a pile," while political scandal at the time asserted that Lansdowne House was built by one Peace (Lord Bute's in 1762) and paid for by another (Lord Shelburne's in 1783).
It was not, however, until 1765 that Lord Bute sold the place in an unfinished condition to the Earl of Shelburne for £22,500, although as it then stood it is thought to have cost him some thousands more.
Lord Shelburne, the "Malagrida" of Junius and the "Jesuit of Berkeley Square" of George III, would appear to have been contemplating the purchase of a house, or a site on which to build one, some years previously, as C. J. Fox wrote to him on 29th June 1761, talking of retiring from public life:
"I see you have ordered Adam to look out for space to build a hotel upon. The late Lord Leicester and the late Lord Digby were about a fine piece of ground for that purpose, still to be had, the garden of which, or the court before which, may extend all along the bottom of Devonshire garden, though no house must be built there; the house must be where some old paltry stables stand at the lower end of Bolton Row."
However, this site was secured by Lord Bute before Lord Shelburne could make up his mind to take Fox's advice. The current position of Lansdowne House - not facing Berkeley Square as might be expected - is therefore accounted for by the restrictions mentioned by Fox — restrictions undoubtedly made when Lady Berkeley of Stratton originally developed the estate.