That is regulated by the item.sdata file.
Use
[Only registered and activated users can see links. Click Here To Register...] Tool to set it to your desired limit for the item in question.
The number you are looking for is the Bonif Max.
It is whatever you set it at depending on the stat if it is set to 40 then rerolls will get up to 4000 on HP MP and SP and up to 40 on Regular Stats...
Remember after updating the sdata you will need to patch the client with the updated sdata files and update the database to reflect the changes as well.