The race of the servants was the differentiating factor. With the white servants, it was common for a punishment to be much less severe than that given to black servants. Since blacks were considered to be "not a person" (note the 3/5 Compromise for evidence), it was common to treat them in such ways that only exacerbated the differences between whites and blacks.