The musician, who is credited with influencing many hip hop artists, died in a Las Vegas hospital, family members confirmed.
His hits Just Begun and Troglodyte were sampled by many artists including Ice Cube and Usher.
Kanye West used Castor's I Just Wanna Stop on his track We Don't Care, which featured on his debut album.
Castor's son Jimmy Castor Jr said the first few chords of It's Just Begun are instantly recognisable.
"No matter what country you're in, no matter what language you speak, everyone knows it," he said.
A sample of the song was famously used in the breakdance battle scene in the 1983 movie Flashdance.
Castor's work has been sampled by other artists more than 3,000 times, the star's son added.
Dancer Richard Colon, professionally known as Crazy Legs, paid tribute to the star, saying "people have been impacted by him and don't even know it".
The musician began his career for filling in for rock and roll singer Frankie Lymon when he was unable to perform with his band the Teenagers.
The star soon went on to form his own group, Jimmy and the Juniors, who wrote and recorded the song I Promise to Remember, which was later also recorded by Lymon and the Teenagers in 1956.
In 1972 Castor started another band, the Jimmy Castor Bunch, which concentrated on funk music.
They had hits with Troglodyte and The Bertha Butt Boogie.
Castor continued to work until last summer, and in November suffered a heart attack and underwent quadruple heart bypass surgery.
SOURCE:http://www.bbc.co.uk/news/entertainment-arts-16607855