Caroline Morabe, a spokeswoman for his record label, told CNN; said: "I can confirm that Oliver Mtukudzi has passed on. The family will release an official statement with more details later today."
The cause of death was not disclosed although he had long struggled with diabetes.
"He has rested. He had been in and out of the hospital since last year when he developed a heart problem," Clive Mukundu, a music producer who worked with Mtukudzi, told CNN outside the Harare clinic.
"It is difficult to accept, I have no words," said musician and poet Albert Nyathi, who joined several other mourners at the hospital. "What is left is to celebrate his life."
Tuku breathed his last on the same day as his friend and fellow continental great Hugh Masekela of South Africa who died on January 23 last year.
"Today we said goodbye to a true patriot. Oliver Mtukudzi, your voice has given us comfort during difficult times, and will remain with us for posterity," President Emmerson Mnangagwa said.
Temba Mliswa, the member of parliament for Norton Constituency – Tuku's home town – extended his condolences to the Mtukudzi family adding that he is applying to the President's office for the musician to be granted national hero status for his immense contribution to the arts industry.
Obert Gutu, vice president of the opposition MDC party added: "It's absolutely devastating and extremely sad! An icon has fallen! This is not only a loss to the Mtukudzi family, it also a loss to Zimbabwe, to Africa and indeed, to the whole of humanity.
"Through his music, Tuku touched millions of hearts around the globe. He was an immensely talented composer, with a husky and inimitable voice.
"He sang about love, about social justice and about the trials and tribulations of human existence.
"Tuku is a national hero. Anyway, heroes don't die! Legends don't die! They simply go home to rest! Rest in Power, Samanyanga!"
Fans mourned "a great loss" across the world, sharing Tuku's music and pictures on social media pages.
Born 22 September in 1952, Mtukudzi began his music career in 1977, singing alongside Thomas Mapfumo in the Wagon Wheels band.
With a music career that spanned over thirty years, Tuku has serenaded fans across all five continents and attracting adoring fans worldwide.
The 'Neria' hitmaker has over 60 albums to his name and won several national and international prestigious awards.
He was also an actor known for his appearance in the films; Jit (1990), Neria (1993) and Zimbabwe-Respect for Africa (1994).
He is survived by his wife, Daisy, four children and two grandchildren.