So the question is.
How To Design Difficulty Well?
To answer this question we need to check what makes something difficult in the first place. What are the elements we can use to raise the difficulty? How does the difficulty change over time? (difficulty curve) And why even bother to make content difficult? What's the gain?
What Makes Something Difficult?
Let's start with our first question. There are several ways to make something difficult. But to boil it down to make something difficult you need to make the player push their limits. Which limits? The limits I'm talking about include reaction time, hand-eye coordination, memory, fast thinking and several more.
What Elements Can Be Used?
One way to make things more difficult is to increase punishment when failing.
The most common way this is implemented is, taking higher amounts of damage from avoidable damage such as AoEs (Area of Effects) or from mechanics. The highest difficulty would be if everything one hits.
Another way would be to challenge the player's multitasking capability.
People might be able to do one boss mechanic and then another without problems. However, the layering of mechanics playing at the same time requires people to set a priority and solve multiple problems at the same time.
The reaction speed of the player can be tested as well.
Boss animations, marked areas, casts that need to be interrupted, turning away from the boss, regardless of the details if the player needs to react fast to a certain mechanic of the boss the faster he needs to react the harder it gets. However, this ability decreases with age. So when using this have older people in mind as well.
Or the player's ability to check their surroundings.
Again boss animations and marked areas, encounter timer, mini-map, changes in the boss arena, there's a lot you can focus on or watch.
Damage checks in games where there may not be better equipment require you to improve your play-style and/or movement.
Having new mechanics to learn.
As simple as it is, a new mechanic requires practice and time.
That's a bunch of them, there are more but that's something for now. Another thing we should look at is the difficulty curve.
The Change of Difficulty
The change in difficulty in games is usually represented by a curve. This curve starts slow and gets steeper over time with increasing progress. The easiest way to implement this is by starting off with a new mechanic and keep it simple having the focus on this new mechanic. After solving this mechanic you should create more complex systems that depend on this mechanic and on the next step mix it up with other already implemented and taught mechanics.