The ruling came Wednesday afternoon, less than 24 hours before Shumlin was scheduled to leave office.
Shumlin was unveiling his official portrait at the Statehouse in Montpelier at the time of the announcement. He later told reporters he disagreed with the high court's interpretation of the law but would abide by the decision.
The case concerned the retirement of Associate Justice John Dooley, who first was appointed to the bench in 1987 and has announced plans to retire when his term ends March 31.
Shumlin, a Democrat, had been "prepared" to name a replacement for Dooley from a list of six names submitted by the state Judicial Nominating Board, according to court documents. The Governor's Office had argued Shumlin simply was doing his job.
Two Republican lawmakers, Rep. Don Turner of Milton and Sen. Joe Benning of Caledonia County, filed a petition at the high court late last month to challenge Shumlin's authority to make the nomination.
The five-member court ruled unanimously that Benning had the right to challenge the governor's action and agreed with Turner and Benning on the merits of the case. The case sets a new precedent about the executive power of outgoing governors.