Let A be the set of all points α ,   β such that the area of triangle formed by the points 5 …

Given:Area of triangle formed by $\left(5,6\right),\left(3,2\right) \& \left(\alpha ,\beta \right)=12 \mathrm{square} \mathrm{units}$

$\therefore \frac{1}{2}\left|\begin{array}{lll}\alpha & \beta & 1\\ 5& 6& 1\\ 3& 2& 1\end{array}\right|=12$

$\Rightarrow \left|\begin{array}{lll}\alpha & \beta & 1\\ 5& 6& 1\\ 3& 2& 1\end{array}\right|=24$

$\Rightarrow \alpha \left(6-2\right)-\beta \left(5-3\right)+1\left(10-18\right)=\pm 24$

$\Rightarrow 4\alpha -2\beta -8=\pm 24$

$\Rightarrow 4\alpha -2\beta =32, 4\alpha -2\beta +16=0$

$\Rightarrow 2\alpha -\beta -16=0, 2\alpha -\beta +8=0$

So point will be either $\left(\alpha ,2\alpha +8\right) , \left(\alpha ,2\alpha -16\right)$

For the point $\left(\alpha ,2\alpha +8\right)$,

Distance from origin
$D=\sqrt{{\alpha }^2+(2\alpha +8{)}^2}=\sqrt{5{\alpha }^2+32\alpha +64} \left(\because \beta =2\alpha +8\right)$

$\Rightarrow D^2=5{\alpha }^2+32\alpha +64$

For maximum or minimum length,

$\frac{d\left(D^2\right)}{d\alpha }=10\alpha +32=0$

$\Rightarrow \alpha =-\frac{16}{5}$

$\therefore \beta =\frac{-32}{5}+8=\frac{8}{5}$

$\therefore D=\sqrt{{\left(-\frac{16}{5}\right)}^2+{\left(\frac{8}{5}\right)}^2}=\frac{8}{5}\sqrt{5}=\frac{8}{\sqrt{5}}$

Similarly if $\beta =2\alpha -16, D=\sqrt{{\alpha }^2+{\left(2\alpha -16\right)}^2}=\sqrt{5{\alpha }^2-64\alpha +256}$

$\therefore D^2=5{\alpha }^2-64\alpha +256$

For maximum or minimum length,

$\frac{d\left(D^2\right)}{d\alpha }=0$

$\Rightarrow 10\alpha -64=0$

$\therefore \alpha =\frac{32}{5}$

$\therefore D=\sqrt{{\left(\frac{32}{5}\right)}^2+{\left(\frac{64}{5}-16\right)}^2}=\frac{16}{\sqrt{5}}$at $\alpha =\frac{32}{5}$
So, least possible length of line segment is $\frac{8}{\sqrt{5}}$