The castle was built in the middle of the 16th century by the craftsmen from Sweden at the expenses of the Duke Matas Giedraitis and his son Martynas Giedraitis. The castle, known as the Brick Videniškiai, belonged to the defensive system of the middle part of Lithuania and protected the old road Vilnius-Ryga. At the beginning of the 17th century, an ornate palace with a Renaissance garden was built. The house had a so called golden hall and a treasury. Baltadvaris Castle is a unique example of military wooden and stone architecture connecting two styles and technologies - Lithuanian and Swedish.