If the area of the rectangle = xy we first have to find a relation between x and y.
If the angle opposite the side of length 5 is A then we have
tan A = 5/4
so we have the relation
5/4 = x / (4 - y)
giving
x = 5(4 - y) / 4
Substituting in the formula for the area ( A = xy), we have
A = 5y(4 - y) / 4
A = 5y - 1.25y^2
Differentiating:-
dA / dy = 5 - 2.5y = 0 for maximum area
y = 2
and x = 5(4 - 2) / 4 = 2.5
So area of largest rectangle = 2 * 2.5 = 5 cm^2