The system with impulse response given by $h[n] = \cos(\pi\sqrt{n})u[n]$$\sum_{n=-\infty}^\infty |h[n]]$$k$$h[k^2]=\cos(\pi k)$$\pm 1$

$$\sum_{n=-\infty}^\infty |h[n]| = \sum_{n=0}^\infty |\cos((\pi\sqrt{n})|$$ $+1$$\cos(\pi r)=0$$r = k+\frac 12$$k$ is an integer, and there is no integer $n$$k+\frac 12$$\sum_{n=-\infty}^\infty |h[n]]$

The LTI system defined by the impulse response

$$h[n] = \cos(\pi \sqrt{n} ) u[n]$$

$$\sum_{n=-\infty}^{\infty} |h[n]| = \sum_{n=0}^{\infty} |\cos(\pi \sqrt{n}) u[n]| \longrightarrow \infty$$

The summation of $\cos(\pi (z^{n^(0.5)})) z^{-n}$ should converge for the $z$ transform to converge and it is stable if the ROC includes the unit circle. Since you got double poles at $|z|=1$, the causal ROC will lie outside the largest pole and hence it won't include $|z|=1$ and hence it cannot be stable.