[tex]\vec F(x,y,z)=x^3\,\vec\imath+x^2y\,\vec\jmath+x^2e^y\,\vec k[/tex]
has divergence
[tex]\nabla\cdot\vec F(x,y,z)=\dfrac{\partial(x^3)}{\partial x}+\dfrac{\partial(x^2y)}{\partial y}+\dfrac{\partial(x^2e^y)}{\partial z}=4x^2[/tex]
so that by the divergence theorem, the flux of [tex]\vec F[/tex] through [tex]S[/tex] is the integral of [tex]\nabla\cdot\vec F[/tex] over the interior of [tex]S[/tex] (call it [tex]R[/tex]):
[tex]\displaystyle\iint_S\vec F\cdot\mathrm d\vec S=4\iiint_Rx^2\,\mathrm dV[/tex]
[tex]R[/tex] is a tetrahedron in the first quadrant defined by the set
[tex]R:=\left\{(x,y,z)\mid0\le x\le5,0\le y\le3,0\le z\le3-y\right\}[/tex]
(and this is by no means the only way to define [tex]R[/tex])
Then the integral over [tex]R[/tex] is
[tex]\displaystyle4\int_0^5\int_0^3\int_0^{3-y}x^2\,\mathrm dz\,\mathrm dy\,\mathrm dx=\boxed{750}[/tex]
