This is the single most important distinction, and it's the one that saves you a lot of worry: look at the stool, not the straining.
In infant dyschezia, the stool is soft. It might be the classic mustard-yellow breastfed stool, the tan paste of formula-fed stool, or anything in the normal range. When it comes out, it comes out without the stool itself being a problem. The difficulty was entirely in the coordination — the baby's muscles working against each other.
In actual constipation, the stool is hard. It looks like small dry pellets, like rabbit droppings. It may cause pain not from coordination difficulty but from the hard stool stretching the anal opening. Constipation is much less common in exclusively breastfed babies and more common after the introduction of solid foods or certain formulas.
Here's the framework: Screaming + soft stool = dyschezia (almost certainly fine). Screaming + hard stool = constipation (talk to your pediatrician). That's it. That's the whole decision tree.
It's worth noting that actual constipation in babies under 6 months who are exclusively breastfed is quite rare. Breast milk is very efficiently digested and produces soft stool. If your exclusively breastfed baby is straining and crying, dyschezia is by far the most likely explanation. Formula-fed babies can experience constipation more frequently, but even then, dyschezia is the more common cause of straining and crying in the under-3-month age group.