I know, a hot take right off the bat, but I wouldn't be so adamant about it if I couldn't back it up. And let me be clear, when I refer to mods, I'm referring to mods that heavily impact the gameplay experience, either by changing mechanics or adding new content. Not client modifications that still allow you to connect to unmodified servers. Now, allow me to explain:
I've been playing Minecraft for 14 years now. A truly incredible amount of my time has been spent in this game, both alone and with friends. But for me to hold onto such a game as tightly as I do for that period of time, it needs something compelling. And Minecraft has that.
During my 14 years (and counting) of tenure with Minecraft, I've dabbled with mods more times than I can count. From Tekkit to GregTech, I've tried plenty. And while I have my fun poking and prodding around the new additions to the base game, one thing always comes up. It's a different thing, different from the core of Minecraft that compels me.
To many players (me included), vanilla Minecraft is not a game about survival. Frankly, it hasn't been for years at this point. Blame it on growing up, the ever-advancing meta, or the game simply getting easier; that's not what compels me to play the game. Nor is the implied end goal of the game, beating the Ender Dragon. Rather, that's viewed as the start of the game. Why? That's the point where everything becomes unlocked. Every tool, every enchantment, every block. Almost every restriction is removed, leaving the player free to do one thing: create.
Meanwhile, modded players have a different experience. Most of these players will be playing with a mod pack, and most mod packs contain one thing that reshapes how the game is played. It's the quest book, the one book every modded player treats as holier than the Bible. It's the lifeline, the instructions, the gospel. The quest book says to do X? The players do X. Make 15 of Y? Everyone rushes to do so. Little attention is paid to much else.
Vanilla players are motivated internally. They want to build something, create some vision that exists in their minds. So what do they do? They should best use the tools at their disposal to fulfill that goal. Grinding resources, building farms, and terraforming are all done with the explicit purpose of making that cool thing in their head.
Modded players are motivated externally. They chase after the steady drip feed of rewards the quest book provides. Their factories and facilities grow as the quest book demands more and more items, ever increasing in complexity. Time is not spent on the inconsequential; all that matters is the quest book. All the grinding and building of factories and machinery is done at the whim of the quest book and not themselves.
And I think that best outlines my point: modded Minecraft is this progression-based factory simulator, while vanilla Minecraft is a sandbox to express your ideas in. These have two completely separate purposes and thusly function as two separate games. Sure, overlap exists; both are still sandboxes, and the player can subvert these intended goals, but it fails to take into account that most players won't. Many a time, I have gone with my friends with the idea to play Minecraft, and many a time, they have agreed. But the ideas in our heads clash upon the question of "vanilla or modded?" because we're looking for different things under a game with the same name.