Another 28 injuries have been reported in Santa Barbara County following the deluge that overwhelmed an area scorched by wildfires last month.
More than 100 homes were destroyed, and another 300 have been damaged.
One rescue worker tearfully described plucking a “muddy doll” of a toddler from under several feet of debris.
Berkeley Johnson said the two-year-old girl was taken to hospital and suffered an injury to her hip.
“I don’t know how the baby survived,” Mr Berkeley told the Santa Barbara Independent newspaper.
He said of his own home in the community of Montecito: “This was an hour of rain, and the house was gone.”
Mr Johnson said he and his wife, Karen, heard a baby crying after the flooding had subsided and they managed to climb down from the roof of their swamped home.
The pair joined a fireman to dig the toddler out, scooping mud from her mouth before she was taken to hospital.
“Had we not gone over there, I don’t think that kid would have [survived],” he added.
Police say 17 people remain missing. The death toll rose on Wednesday afternoon after two more bodies were found.
More than 50 people have been rescued already but many places were still inaccessible. Several roads were closed, including the major Highway 101 which authorities say will not be reopened until Monday.
“We are still very much in active search-and-rescue mode,” said Chris Elms, a spokesman for Cal Fire, warning that the death toll may still rise.
“That’s a fear. We are still very hopeful that we will locate people alive,” he added.
Santa Barbara County spokesman Amber Anderson said: “We have no idea where they’re at. We think somewhere in the debris field.”
The upmarket neighbourhood of Montecito is home to celebrities such as actor Rob Lowe, chat show host Ellen DeGeneres and media mogul Oprah Winfrey.
On Instagram, Ms Winfrey said her property had escaped the worst as she toured the scene, but described the house next door as “devastated”.
Some Montecito residents returned to their homes on Wednesday to see what they could salvage from the ruins.
“The house being gone, it’s just a house, it’s just some clothes and a house,” Sam Johnson told Reuters.
“But in a neighbourhood this small, every name that turns up is someone’s dad, someone’s cousin, someone’s teacher and that’s got to be the worst part of it all, I think. We’re just happy for everyone that makes it.”
The first rain in months caused mudslides when it hit ground that had been burned by December’s huge wildfires.
After a wildfire, burned vegetation and charred soil create a water repellent layer which blocks water absorption. Together with the loss of vegetation, this leads to an increased risk of mudslides and floods.
The Federal Emergency Management Agency says the risk of flooding stays “significantly higher” for up to five years after a wildfire.
“Recent burn areas will be especially vulnerable where dangerous mud and debris flows are possible,” said the National Weather Service in a statement.
In some places mud was waist-deep, officials said.
Thousands had to leave their homes, many for the second time in two months. The emergency service declared an exclusion zone, saying anyone moving around the area would be in the way of rescuers and would be subject to arrest.
Those who have stayed in the area have been warned to boil their tap water before drinking it.
Heavy rain run-off caused a mudflow in Montecito, where some homes were knocked off their foundations, said Santa Barbara County Fire Department spokesman Mike Eliason.
County Fire Captain Dave Zaniboni said five people were found dead on Tuesday in Montecito, possibly as a result of the storm.
The US Coast Guard has sent “multiple airships to support rescue operations” and warned the public not to fly drones, otherwise the flights would be grounded.
This comes after a record-setting year of $306bn (£226bn) of weather and climate related disaster costs in the United States, with 2017 the third warmest year on record, according to the US National Oceanic and Atmospheric Administration.
California has suffered severe drought in five out of six of the past years.
In December, California Governor Jerry Brown said the state was “facing a new reality” as climate change meant wildfires could erupt “every year or every few years”.