i think ive cracked the dmg calculation after spending many moons on cabal.

if you have more attack than before, you will hit harder.
if you have more critical dmg than before, you will hit harder
if you have more amp than before, you will hit harder
if you have more penetration than before, you will hit harder

so basically the calculation seems to work in a way where if you increase attack, critical dmg, amp and penetration, you can expect your dmg to increase.

i hope this helps.