Originally Published: 02 JUN 22 10:33 ET

By Anna Bahney, CNN Business

(CNN) -- Mortgage rates fell slightly this week, marking the third consecutive week of declines. But with rates hovering above 5% and home prices well above where they were this time last year, prospective buyers are finding it increasingly difficult to afford a home.

The 30-year, fixed-rate mortgage averaged 5.09% in the week ending June 2, down from 5.10% the week before, according to Freddie Mac. It is still well above the 2.99% average from this time last year.

"Mortgage rates continued to inch downward this week but are still significantly higher than last year, affecting affordability and purchase demand," said Sam Khater, Freddie Mac's chief economist. "Heading into the summer, the potential homebuyer pool has shrunk, supply is on the rise and the housing market is normalizing. This is welcome news following unprecedented market tightness over the last couple years."

Rising prices had already been pushing many prospective buyers to the sidelines. But as rates continued to rise this year, even more home shoppers have put their search on hold.

"The relentless rise in home prices and recent increase in interest rates slowed down buyer activity in April as shown by a month-over-month drop in existing home sales, new home sales, and pending home sales, as some buyers opted out of the market altogether," said Hannah Jones, economic data analyst at Realtor.com. "Still, home buyers continued to contend with record-high home prices in May."

But, she said, there may be good news ahead with more homes coming onto the market.

"While inventory is still low by historical standards, it is starting to tilt in a more buyer-friendly direction," said Jones. "This is likely to lead to slower price growth in the not-so-distant future as sellers compete for buyers, finally creating a more balanced market."

Still, many buyers can't afford to buy a home that fits their needs with interest rates so high.

"Those who are currently home shopping will tell you that we're not there yet, as still-high interest rates and home prices are creating challenges in finding their ideal home," said Jones.