Sir Charles was chosen as the tenth Governor of the colony of New South Wales by Lord Stanley in 1845.
He replaced Sir George Gipps as governor who had been a strong ruler but had provoked the animosity of many in the colony.
FitzRoy, his wife and his son George arrived in the colony on board HMS Carysfort on 2 August 1846.
Soon after his arrival he was asked to use his influence to procure the disallowance of an act of the Tasmanian legislature imposing a duty of 15% on products imported from New South Wales.
The need for some type of federation between the various colonies was recognized, and as a step towards this Fitzroy was given a commission in 1850 appointing him governor-general of the Australian colonies.
During his governorship great steps were made in the development of New South Wales.
Transportation of convicts ceased, the Sydney University was founded, a branch of the royal mint was established and responsible government was granted.
In 1847, Fitzroy served briefly as Governor of the Colony of North Australia, although his lieutenant-governor, George Barney had the main responsibility for establishing the new colony under FitzRoy's direction.