It's culture. Put very simply, the culture that dominates the party makes it very difficult for the party to reach--or perhaps even to understand that it must reach--the voters it needs to succeed politically. Begin with this piece by Sheri Berman on party "representation gaps".
"A disconnect exists between the preferences of voters and the stances of the Democratic Party on social and cultural issues, while an equivalent chasm exists for Republicans on economic matters. Both are facing what political scientists call a “representation gap.”
A majority of voters, including many Republicans, support Democratic, even progressive, positions on economic issues including tax policy, healthcare, education, the minimum wage and more. But on crucial non-economic issues, voters are moderate, and the Democratic Party’s stances—in particular those of its progressive activists regarding illegal immigration and border security, police reform, “political correctness,” gender identity, sexual harassment and affirmative action—are unpopular, even among many Democratic voters."
Berman details four possible responses by parties to such representation gaps: avoidance, intransigence, concession and persuasion. She recommends persuasion:
"Here, the party attempts to close the representation gap by trying to move voters closer to its policies. Changing people’s preferences requires openly grappling with the unpopularity of particular policies, and engaging with those who disagree, rather than disparaging or ignoring them.
Successful engagement means avoiding language and behavior that can be easily misunderstood or trigger fears in voters, and can be exploited by opponents. Progressive activists erred by ignoring this, with some describing themselves as “socialists,” embracing the slogan “defund the police,” and failing to denounce looting and violence that accompanied some protests—behavior that scared some voters away from the Democratic Party. Above all, the persuasion strategy requires patience since shifting voters’ preferences and priorities takes time.
One disadvantage of the persuasion strategy is obvious: It is the most difficult. But its advantages are great. It is the only strategy that reconciles idealism with realism, joining a commitment to currently unpopular but prized policies with a recognition that realizing such policies first requires winning elections."
And winning elections--enough of the right type of elections--is very difficult for Democrats in the actually-existing electoral system of the United States (not the dream system frequently invoked by progressives). Derek Thompson in the Atlantic:
"As Democrats like to point out, they have won the national popular vote seven times in the past eight presidential elections, along with a majority of ballots cast nationwide in six of the past eight Senate elections. But national vote tallies don’t count for anything under our constitutional rules. Liberals can (and do) complain about this all day, but saying “If the Constitution didn’t exist, Capitol Hill would be mostly Democrats” is like saying “If the weak nuclear force didn’t exist, the galaxy would be mostly photons.” Neat, but so what?...
“We have an election system that makes it basically impossible for Democrats’ current coalition to ever wield legislative power,” the polling analyst David Shor told Politico....If Democrats don’t find a way to broaden their coalition into less populous states with smaller metro areas, it may be impossible to pass liberal laws for the next generation."
This is above all a cultural problem.
"Earlier this month, the Los Angeles Times published an analysis of California ballot measures that found that “the state’s two major population centers have grown more and more different” from the rest of the state. Residents of Los Angeles and the Bay Area were at least 30 percentage points more likely than other Californians to support various propositions, such as reinstating affirmative action and allowing parolees to vote....
This phenomenon is not specific to California; it is evident across the country. America’s richest and most progressive cities—from San Francisco to New York and Washington, D.C.—have filled with young, unmarried, “extremely online” graduates of elite colleges, who have collectively birthed a novel philosophy you could call “Instagram socialism.” Instagram socialists are highly educated, but not necessarily high-earning, urbanites who shop like capitalists and post like Marxists and frequently do so in adjacent tabs. Many of their causes are virtuous, such as universal health care and higher pay for low-income service workers. But given the dynamics of online communication, which prizes extremity, Instagram socialism usually functions as a crowd-sourcing exercise to brand widely appealing ideas in their most emotional and viral—and, therefore, most radical—fashion. Thus, major police reform (a popular idea) is branded “Abolish the Police” (an unpopular idea); a welcoming disposition toward immigrants (a popular idea) is blurred with calls for open borders (an unpopular idea); and universal health care (a popular idea) is folded into socialism (an unpopular idea)....
The urban-online core of the Democratic Party is arguably its most fecund source of new ideas, but it has evolved to become a kind of third party whose ideology and tone are not a good fit for the typical swing voter toggling between the two major parties. The “culture problem” of liberals’ urban dominance also has important implications for the media. As national politics has polarized according to youth, education, and density, the news industry has become particularly young, educated, and densely packed into a handful of cities. Thus people who work in national journalism (like me, a resident of Washington, D.C.) tend to have a set of strongly held views about the world that are artifacts of their zip code. America’s most progressive metros are becoming politically unusual at the same time that national journalists are less capable of seeing how unusual they have become."
In other words, party activists are increasingly ill-equipped to successfully reach out to the rest of the country that doesn't share their views--or indeed to even realize how far they need to reach to do so.
"In the past 100 years, Democrats have transformed from a largely rural party to a coalition of density and diplomas—a political handshake between the cities and highly educated suburbs. To win not only the presidency but also legislative majorities, Democrats will have to turn back the clock and refamiliarize themselves with Americans who live outside denser zip codes where Democrats currently dominate. They will have to rediscover the right balance between their progressive urban core and the moderate exurban frontier—between metro and retro."
In some ways it is very easy to explain how this unfortunate state of affairs arose: it is all about the increasing domination of college-educated whites in Democratic politics. This has several strands to it:
1. There are simply more of them in the party's support base, due to their rise in numbers and significant shift into the Democratic party. Their weight in the Democratic coalition is now much larger than it was 30 years ago.
2. They are heavily over-represented in the donor base of the party.
3. They are heavily over-represented in media that cover the activities of the party and give it publicity.
4. They literally run the party on a day to day basis, providing the professionals who staff the infrastructure and run for office.
Thus, it should come as no surprise that the culture of this demographic group increasingly dominates the party. But that it is understandable does not make it desirable. The party cannot achieve the success it desires without competing more successfully in rural, small town and less densely-populated parts of the country, which have disproportionate political influence and where noncollege whites are especially numerous. And it cannot compete successfully in these areas without moving out of its current cultural comfort zone. Indeed, we are now getting to the point where even Democrats' hold on noncollege nonwhites may be threatened by the cultural dominance of educated whites.
I'll give the final word to data scientist David Shor:
"The big-picture problem is that the Democratic Party is increasingly reflecting the cultural values and political preferences of educated white people...Culturally, working-class nonwhite people have more in common with working-class white people.”
That's it in a nutshell. A serious course correction is in order.