The building was constructed in 1631 for the clergyman Christen Rasmussen from Nørre Lyndelse. His widow, Johanne Nielsdatter, lived in the house until her death in 1660.
The facade is decorated with carved ornaments in Renaissance style with characteristic carved rosettes. Two emblems are found over the gate. The first: CR above a star and a heart, the second: IND above a snake on a cross. The initials stand for Christen Rasmussen and his wife Johanne Niels Datter.
The building has had many owners, but it is the owner from 1857, dyer Østerbye that the building is named after.
The building still contains the original walls and ceilings. Since 1945, it has been part of the museum complex. Until 1982, it was located in Møntestræde, after which it was moved to its present location at Sortebrødre Torv. The building is used for special exhibitions, but is normally not open to the public.