The mass density of a planet of radius R varies with the distance r from its centre as ρ ( r ) = ρ…

$\mathrm{dm}=\mathrm{\rho }\times 4{\mathrm{\pi x}}^2\mathrm{dx}={\mathrm{\rho }}_0\left(1-\frac{{\mathrm{x}}^2}{{\mathrm{r}}^2}\right)\times 4{\mathrm{\pi x}}^2\mathrm{dx}$

$\mathrm{m}=4\mathrm{\pi \rho }{\int }_0^{\mathrm{r}}\left({\mathrm{x}}^2-\frac{{\mathrm{x}}^4}{{\mathrm{R}}^2}\right) \mathrm{dx} \mathrm{s}$

$\mathrm{m}=4{\mathrm{\pi \rho }}_0\left|\frac{{\mathrm{r}}^3}{3}-\frac{{\mathrm{r}}^5}{5{\mathrm{R}}^2}\right|$

$\mathrm{E}=\frac{\mathrm{Gm}}{{\mathrm{r}}^2}=\frac{\mathrm{G}}{{\mathrm{r}}^2}\times 4{\mathrm{\pi \rho }}_0\left(\frac{{\mathrm{r}}^3}{3}-\frac{{\mathrm{r}}^5}{5{\mathrm{R}}^2}\right)$

$\mathrm{E}=4{\mathrm{\pi G\rho }}_0\left(\frac{\mathrm{r}}{3}-\frac{{\mathrm{r}}^3}{5{\mathrm{R}}^2}\right)$

$\mathrm{E}$ is maximum when $\frac{\mathrm{dE}}{\mathrm{dt}}=0 \Rightarrow \frac{\mathrm{dE}}{\mathrm{dr}}=4{\mathrm{\pi G\rho }}_0\left(\frac{1}{3}-\frac{3{\mathrm{r}}^2}{5{\mathrm{R}}^2}\right)=0$

$\Rightarrow \mathrm{r}=\frac{\sqrt{5}}{3}\mathrm{R}$

${\mathrm{E}}_{\max }=4{\mathrm{\pi G\rho }}_0\times \frac{\sqrt{5\mathrm{R}}}{3}\left[\frac{1}{3}-\frac{1}{5}\times \frac{5}{9}\right]$

${\mathrm{E}}_{\max }=\frac{8\sqrt{5}}{27}{\mathrm{\pi G\rho }}_0\mathrm{R}$