Shakhtar were without 10 first-team players and nine members of staff due to coronavirus infections but recorded a remarkable win at the Alfredo di Stefano stadium, after scoring three times in 13 first-half minutes.
Strikes from Tete and Manor Solomon, either side of a Raphael Varane own-goal, gave the Ukrainian champions a shock lead at the interval and while Luka Modric and Vinicius Junior pulled goals back, Madrid fell short.
Fede Valverde thought he had grabbed an equaliser in injury time but his deflected shot was ruled out after VAR showed Vinicius was offside.
A huge upset puts Madrid on the back foot in Group B, especially as Shakhtar were in theory their weakest opponent, the others being Inter Milan and Borussia Moenchengladbach.
But it also cranks up the pressure, on the Real team ahead of the Clasico in La Liga on Saturday, and on their coach Zinedine Zidane, who had just overseen a dreadful display in last weekend's loss to Cadiz.
Zidane paid the price for rotating his starting line-up for that game and shuffled his team again, presumably with the upcoming clash at the Camp Nou in mind.
Toni Kroos and Karim Benzema both started on the bench and had to come on in the second half while Eder Militao, in for the injured Sergio Ramos, Rodrygo and Marco Asensio were given chances to impress.
Shakhtar's three-goal flurry before half-time began with Viktor Korniyenko beating Militao too easily and as Marcelo was pulled across to cover, he left Tete free behind him to finish Korniyenko's poked pass.
Luka Jovic should have equalised with a free header but instead Shakhtar scored again, Tete's rasping effort from range palmed out by Thibaut Courtois and prodded in for an own-goal by Varane has he tried to stop Dentinho reaching the rebound.
Madrid looked shell-shocked but it got worse. Tete raced in behind and this time a brilliant back-heel allowed Solomon to take a touch and fire in number three.
Benzema replaced Rodrygo at half-time but it was Modric who kick-started the comeback attempt, shifting right past two Shakhtar defenders before unleashing a superb shot into the top-right corner.
Tete then missed a sitter for 4-1 and Vinicius capitalised. The Brazilian came on as a substitute, stole the ball back and found the net, seemingly all in one movement.
Madrid had 31 minutes to rescue a point but their momentum stalled. Benzema's finish was blocked by Valerii Bondar and Shakhtar thought they had a fourth but it was ruled out for offside.
Valverde celebrated an equaliser in injury time when his deflected shot from the edge of the area crept in but there was a final twist in the tale as VAR showed the offside Vinicius was obstructing the goalkeeper's view.