Collier was born circa 1848 as a slave in Mississippi and was the third generation to serve the Hinds family on Plum Ridge Plantation, built by General Thomas Hinds, who was a veteran of the Battle of New Orleans in 1815. At the request of General Andrew Jackson, Hinds had surveyed central Mississippi and chose the site for the state capital, Jackson, before settling nearby in the area which is now Hinds County.
Collier killed his first bear at the age of ten; thereafter, his job was to supply meat for the table of the Hinds family and the field hands. With the outbreak of the American Civil War, Collier’s master Howell Hinds and his seventeen-year-old son Tom, who was Collier’s childhood companion, left for the war. Although told by his master that he was too young to fight, Collier stowed away on a riverboat and joined Howell and his son in Memphis, Tennessee.
At the 1862 Battle of Shiloh in western Tennessee, Collier witnessed the death of Confederate General Albert Sidney Johnston. Collier’s biographer says that although there was a prohibition against blacks serving in uniform, Confederates made an exception for Collier because of his demonstrable skills. Collier stayed with the Hinds men until later being given the opportunity to ride with the 9th Texas Cavalry Regiment. He served in Company I through the rest of the war and fought in Mississippi, Alabama, and Tennessee.