Of the ancient Olivetan convent, which used to reside next door to the church of Our Lady delle Grazie, only the refectory remains, decorated with gorgeous 13th century frescos painted by Nicolo Corso. Over the years, part of the convent has been turned into residential flats.
The monastery was built using money donated by Giovanni da Alessandria di Santa Maria di Lucedio, of the Vercelli diocese, who left the Olivetans an oratory and a small house near Portovenere. The monastery was not actually built until the 15th century, when Pope Eugene IV gave the Olivetani di Quarto the money to do so.
The convent was originally built on two floors, and was later extended. In the 17th century, the monastery comprised of nearly 30 rooms, not including the vestry and the choir. During the French Revolution, part of the monastery was turned into a public school, and later the Napolonic War saw the monks thrown out of the monastery altogether.