No, it's not.
And it depends on what is being examined. In one game, yes, all that matters is points. However, for an entire season, yards is more accurate than any one stat. And no, you should never use just one stat.
And for purposes of team evaluation, yards is very important. The team that gives up tons of yards but not so many points will see those yards turned into points in the playoffs. A good example of this is the 2017 season super bowl, where the Pats were 29th in yds allowed, but 5th in points allowed. That worked fine until the super bowl where Philly turned those yards allowed into points.
If you want to win a super bowl, allowing a pile of yards will keep you from it. Most super bowl winners are top 10ish in yards allowed. If you're in the bottom 5, that means you cannot stop the other team from driving up and down the field. That will always come back to haunt you.