Let's say there are two attacks: one rolls a 10d10 and the other rolls a 5d20. Assuming the modifier was the same, which rolls would have better chances of rolling average, and which would have better chances of rolling the minimum or maximum?

To specify I am aware that more dice would have a higher avg and min roll but both of those would be by a slim margin and I'm not sure how those would affect your "odds" of hitting each number.

From what I can gather what you're asking, you want to know the probabilistic difference between rolling 10d10 and 5d20. You've rightly pointed out that each roll has the same maximum and that each has a better chance at rolling their given averages. The averages are different, which you already know. They obviously have different minimums (10 vs 5), and so you want to know precisely how different the rolls are.

Any time we look at two dice variations where the product of the number of dice (xdy) and the faces of the dice used (xdy) is equal between the variations, we usually want to know how they differ since the ranges are so similar. In our case, 10d10 versus 5d20 is very similar because 10*10 is equal to 5*20. The following answer can be used as an example for any comparison of dice rolls where x and y on both variations have the same product (2d10 versus 1d20, 2d6 versus 1d12, 3d8 versus 4d6 versus 2d12, etc).

Fewer Dice, More Variation

Using AnyDice.com we can calculate the probability very simply with the commands output 10d10 and output 5d20. And really that's all there is to it. The black line below represents 10d10, and the yellow line represents 5d20.

Generally speaking, when you have a greater number of smaller dice, your rolls are less "swingy". Meaning, there are better odds at rolling the "average". But, you have worse odds at rolling higher numbers. When you use fewer number of greater dice, your rolls are more "swingy", meaning you have less chance to roll the average and more chance to roll the extreme ends of the ranges.

Put another way: Look at this graph, it represents the odds that you will roll at least a given number. You can see in general it's better to roll 10d10 because you have greater odds at hitting a certain number until about 60, then 5d20 gives you better odds at hitting those values, but only slightly.

So with 5d20, you have higher odds at hitting a greater range of values, meaning that if you roll 5d20 often, you'll see more "swingy" results. But with 10d10, the odds are more stacked in the middle, meaning it should feel like you're hitting the "average" or the "middle" results more often.

Another Example

But let's simplify. Lets look at output 2d10 vs output 1d20. Same idea as 10d10 vs 5d20. With 2d10, the odds are much different than 1d20 because there are a greater number of rolls that represent the middle values (11). there's 10-1, 9-2, 8-3, 7-4, 6-5, 5-6, 4-7, 3-8, 2-9, and 1-10 representing 11. 10% of all the combinations are 11. But for higher values (20), there is only 10-10 representing that, which is only 1% of all possibilities. But for a 1d20, there is a 5% chance for every number. So 11 is represented by the same number of faces as 20, or 1.

Similarly, if you wanted to compare 1d100 to 5d20 and 10d10, you would see a flat probability: a 1% chance for each value between 1 and 100.

Conclusion, Final Thoughts

So we can see why certain combinations of damage die are used in RPGs, and more specifically D&D 5e (which you originally asked about). The more dice you can use for a given range, the more you, as a designer, can control the probable outcome of that roll. Whereas some rolls, like loot tables, rely on an equal probability of each result by using only 1 (or very few dice) such as 1d100 rolls. Simply put, if you want to design a system that uses dice, you can control probability more by adding more dice).

More dice lead to more average results

There are a number of dice calculators around the internet to illustrate the probabilities of whatever combinations you'd like to see, but a general statement is that rolling a combination of dice and summing them for a result will increase the probability of average results and reduce the probability of extreme results as compared with rolling fewer dice.

For a simple example consider 2d4-1 and 1d7 (this is for you @SevenSidedDie) both produce a number between 1 and 7. Your chance of a 1 with the 7-sider is 1 in 7 or about 14%. Your chance with the 2d4 is 1 in 16 (6.25%), because there are 16 different results possible, but only one of them is the 2 ones which give you a 1. On the other hand there are 3 ways to get 3: 1 & 3, 2 & 2, and 3 & 1, so 3 chances in 16 (18.75%), but still only 1 in 7 with the 7-sider.

Die rolls have mean equal to the average of the largest and smallest number so for a die with f faces (a "df"), the average is (1+f)/2 and the variance is equal to the mean times (f-1)/6; i.e. (f+1)(f-1)/12. The mean and variance of a sum of dice is the sum of the means and the sum of the variances respectively. If the dice are all the same (ndf), then the mean is n(f+1)/2 and the variance is n(f+1)(f-1)/12.

The chance of rolling the minimum (or the maximum) is (1/f)^n - if you halve the faces but double the number of dice, the second case has a larger minimum but the same maximum, but the maximum will be more likely with the larger die.

If you're adding more than a few dice, the probability of the nearest-to-average roll will be roughly 1.38/[f√n]; if you halve the number of faces and double the number of dice you move the average up by 1/2 for each die you started with, and the probability of rolling the nearest-to-average roll will increase by roughly 42%. (These numbers are a bit more accurate with more faces and more dice and not so accurate with few faces and few dice.)

Probability is often very accessible if you make the numbers very large/very small respectively, without knowing much about the maths behind it.

Compare $1,000,000text{d}3$ and $999,999 + 1text{d}2,000,001$. Both can roll any value between $1,000,000$ and $3,000,000$ respectively. But the numbers you would reasonably expect are widely different. For the second case, which is a single roll with a $2,000,001$-sided die, every one of those results has exactly the same probability (or at least we expect that from a non-weighted die, in general).

But for the first case, where you roll a 3-sided die a million times, it is intuitively extremely improbable that you roll, say, the result of 1 for a million times in succession. Hence, the outlier values (small or large ones) are incredibly improbable, and it is incredibly probable that the end result is relatively very close to the center of possible values.

Now, if you scale this down to manageable numbers, you see that there is a similar difference between 5 and 10 rolls. It certainly is not as big as between 1 and 1,000,000 rolls, but the principle applies.

You can use this effect to give flavour to certain features of your game. For example, in some systems the damage type "lightning" is represented with wildly fluctuating outcomes - you can simulate that with few dice rolls. But a damage type of "ice" could be much more regular and predictable => use many dice.

I'll just give a decent example. Rolling 1d6 is comepletely random. Every part has only one way to be rolled, and thus there are an equivalent amount of ways to roll each number. Given that the die is not weighted, then each roll is random. Rolling 2d6 means you suddenly can roll 6 in multiple different ways. The greatest example is that you can only roll "2" one way via 2d6, two "1s". However, 3+3 = 6, 2+4 = 6, 1+5 = 6, 4+2 = 6, and finally 5+1 = 6, so that means there are five ways to roll a six.

This is actually something relavent in the game Mekton. There is an optional rule people use to use 2d6 instead of 1d10. It was introduced in the liscensed spin off, (not created by tallorn games) Gundam Senki. I've been in a campaign where limited Senki rules were used instead of Metkon rules.

Also, it means a critical success, and thus exploding dice, was limited to a 1/36 vs a 1/10 chance. This is actually desirable. I'm playing in a campaign that's ironically Gundam themed, and we have quite the instance rate of critting and then disabling various famous mobile suits. Half the Gundams were captured by Zeon by killing the pilot, including the full armor 7th gundam, Netix, and a ton of interesting stuff. With a 2d12 system, the chance of that happening drops drastically, even if you still use the 1d10 table to roll for critical damage, to 1/180 vs 1/50 (there are two ways knock out enemy mobile suits without killing). It's slightly less than that, as there's more ways to disable a mekton without called shots, but there's my comparison. Still, we've had not quite good luck to die to a ball randomly, like had happened to Dozle, who took one right in the face.