The small Palace was built in 1317. It is in the Northern part of the square, opposite the Papal residence, and so named because of its modest size and value, compared with the Palace of the popes.

The story of his appearance vague: according to one version – it was built for a nephew of Pope John XXII, Arnaud de via, on the other - it was built by cardinal Berengere F. Senior. After his death, the Palace and the surrounding area was purchased by cardinal Arnaud de via. In 1335 the via De died and the Palace did not belong to anyone until the arrival of Pope Benedict XII, who again bought it and turned it into the official residence of the Archbishop of Avignon.

The building was badly damaged in the period from 1396 in 1411. since it was considered a fortified citadel of the Avignon popes. After the end of hostilities, the Palace gradually fell into decay. In the second half of the XV century Bishop Alan de Cotui and the follower Giuliano della Rovere (who later would become Pope Julius II decided to restore the building and to 1503, gave him almost original form. Della Rovere arrived in Avignon in 1474, as he was appointed by his uncle, Pope Sixtus IV, Bishop of Avignon. He constructed the southern and Western facades in the style of the Italian Renaissance and in 1487, was erected a tower (later, in 1767 it was destroyed).

During the French revolution, the Palace became the property of the state, in the nineteenth century it was home to a Catholic school, and later vocational school. Only in the end of XX century on the territory of the Palace Museum was opened.

Since 1958 there is the Museum of the middle Ages. In 1976 there was an art gallery of works by masters of the Renaissance. In the 19 halls of the Museum, in particular the work of Sandro Botticelli, Taddeo Gaddi, Taddeo di Bartolo, Lorenzo Monaco, including here is the famous "Madonna" by Botticelli.