Section 5. The management and repair of such infirmary shall be vested in a joint board of directors, who shall be chosen annually by the several towns interested.