Kramer, whose only experience as a Bundesliga coach is a handful of games coaching Greuther Fuerth in 2012/13, has signed a contract until June 2023.
He took his first training session on Tuesday, the day after Neuhaus, 61, was dismissed with Bielefeld winless in their last five league and 16th in Germany's top flight.
Despite poor results this season, Neuhaus remains a fans favourite after steering Bielefeld back to the Bundesliga for the first time in 11 years by winning promotion in 2019/20.
Kramer spent his first press conference on Tuesday addressing the angry reaction of club fans on social media who are fuming at the dismissal of Neuhaus.
"I respect these reactions. I'm trying to get the mood back on track through hard work. It can only be done together," Kramer said.
The 48-year-old, who most recently coached Germany's junior teams, has 12 Bundesliga games left to get Bielefeld out of the bottom three to guarantee their survival.
His most recent job was head of youth development at Red Bull Salzburg.
He has a chance to make his mark with back-to-back home games in quick succession against Union Berlin on Sunday, then Werder Bremen three days later.