Americans consider that amount of money to be the “perfect salary,” according to a recent survey of more than 2,000 U.S. adults by Talker Research. This is the average amount respondents said they would need in order to be happy, and half of the respondents said the current amount of money they make isn’t enough to support their lifestyle, even beyond housing.

While the average amount was $74,000, that’s not nearly enough to afford to buy a home in all but two U.S. states: West Virginia and Louisiana, according to Realtor.com—and even doubling that “perfect salary” to $148,000 won’t get you a house in every state.

“Earning the ‘perfect salary’ may still fall short of affording a median-priced home in most states,” Hannah Jones, senior economic research analyst at Realtor.com, said in a statement.

The median-priced new home in the U.S. costs more than $410,000 and an existing home will set you back more than $422,000, U.S. Census Bureau and National Association of Realtors data shows. And in states like California, Hawaii, Massachusetts, Colorado, and Washington, buyers can expect to shell out well over $600,000 to buy just a median-priced home.

Assuming you purchase a home for $422,000, put down a conventional 20%, and your mortgage rate is about 6.5%, that means you’d end up spending nearly $2,500 on your monthly mortgage payment. That would be well over one-third of a monthly gross salary, which is generally discouraged. Most real estate experts warn against spending more than one-third of your salary on housing. 

But assuming a $148,000 salary, that $2,500 payment wouldn’t feel as overbearing—that is, if you have the ability to shell out on the down payment and can even find a home that meets your needs within that median price range. 

The biggest hurdles for U.S. homebuyers

While much of the housing-market conversation has been focused on mortgage rates—which continue to hover in the mid-6% range—a sticky problem is home prices remain historically high. 

“It’s really the home prices that are the bigger hurdle,” Michelle Griffith, a luxury real-estate broker with Douglas Elliman in New York City, told Fortune. “Even if mortgage rates dropped to zero, the reality is that buying into the market…still requires a significant amount of cash upfront. Inventory is tight and competition is high, so the cost of the property itself is what keeps most buyers on the sidelines.”

Still, mortgage rates are a barrier for some buyers—especially those who recall the sub-3% mortgage rates of the pandemic era. It’s also the reason many current homeowners are staying in place and refusing to sell. 

“Many homeowners are reluctant [to] put their homes on the market and give up the low mortgage rates they already have,” according to Warren Buffett’s Berkshire Hathaway HomeServices. “To them, high price gains won’t mitigate their ability to pay more for another home at significantly higher interest rates.”

Torsten Sløk, chief economist for Apollo, wrote in a Thursday note that housing supply is holding steady because current homeowners don’t want to sell and take on higher mortgage rates. Meanwhile, demand is slowing because home prices and mortgage rates remain relatively high. That could be somewhat good news about home prices.

“The bottom line is that there is downward pressure on home prices coming from falling demand and rising supply,” Sløk wrote.

While not by much, mortgage rates are also trending slightly lower during the past few months and home-price growth is mostly flat or slightly declining. 

Improving housing affordability “will take time, likely years, [but] the balance of power is no longer as one-sided as it was during the pandemic frenzy,” wrote Mark Fleming, chief economist for Fortune 500 financial services firm First American. “For those prospective buyers who have been waiting on the sidelines, the housing market is finally starting to listen.”