The sloping frontage that forms the Piazza de' Pitti is closed on three sides by the circling wings of the Palace, which the Florentines have always been reluctant to call the Royal Palace because of their innate tendency for understatement, in spite of the fact that the building has hosted all the ruling dynasties (the Medicis, the Lorraines, the Bourbons, the Bonapartes, the Savoys) ever since Cosimo I's wife, the beautiful and rich Duchess Eleonora of Toledo, bought it from Buonaccorso Pitti in 1550 to turn it into the new home of the Medici family.

The merchant Luca Pitti, first the friend and later the rival of Cosimo "the Elder", commissioned Filippo Brunelleschi to carry out the design of the building in around 1440. Cosimo "the Elder" had meanwhile commissioned Michelozzo carry out Palazzo Medici in Via Larga, turning down an earlier project by Brunelleschi, which he thought was too obstentious and likely to arouse the jealousy of his fellow citizens . This thought apparently did not worry Pitti at all, and he decided to employ the most brilliant architect of the times, whom he apparently, or so the legend goes, ordered to make the windows as big as the doors of the Medici residence and create an internal courtyard that was large enough to contain the whole of the palace in Via Larga.

Brunelleschi's structure reflects his idea of a what a Renaissance palace should look like: a cube equal in both height and depth, and covered with the typical Florentine "rustic stone-work" (faceted stones), quarried directly from the Boboli hill itself. The palace was built on three floors, with three entrance doors on the ground floor and seven windows on each side of the two upper floors. A balcony crossed the entire facade, linking up the windows, while a loggia was built beneath the roof to give it a finishing touch.

We can see what the palace looked like after the first stage in its construction in the well known map of Florence known as "the Catena", carried out in 1470. When Eleonora of Toledo bought the estate in 1550, construction work on the monument was intensified and can be divided into two historical stages: the Pitti Palace and the Medici family (until 1737) and the Pitti Palace after the Medici family.