Jtdylktuy

I want to know how to solve this type of questions. How can I find $ f(x)$ from $ f(f(x))$

Suppose, $ f(f(x)) = x$ , then $ f(x)=x$ or $ f(x)=dfrac{(x+1)}{(x-1)}$

how to find these solutions..

I have found one.. $ f(6) = sqrt{222} $

Is this correct?

Assume $f(x_1)=f(x_2)$. Then $x_1^2+1=f(f(x_1))=f(f(x_2))=x_2^2+1$ and so $x_2=pm x_1$.
Then from $f(f(-x))=f(f(x))$ we conclude that $f(-x)=pm f(x)$ for all $x$.

Define $gcolon [0,infty)to [0,infty)$ by $g(x)=|f(x)|$.
For $x$ with $g(x)=f(x)$ we have $$g(g(x))=|f(f(x))|=|x^2+1|=x^2+1.$$
And for $x$ with $g(x)=-f(x)$ we have $$g(g(x))=|f(-f(x))|=|pm f(f(x))|=|f(f(x))|=x^2+1$$
as well. Hence $g(g(x))=x^2+1$ for all $xge 0$.
Given a function $g$ with this property, we can easily construct a suitable $f$: Just let $f(x)=begin{cases}g(x)&xge0\g(-x)&x<0end{cases}$. If $g$ is additionally continuous then so is $f$.
We can find a lot of continuous $g$:

Pick $a_1in(0,1)$, let $a_0=0$ and recursively $a_n=a_{n-2}^2+1$ for $nge 2$. Then the sequence $(a_n)$ is strictly increasing towards $infty$.
Now let $g_1colon[0,a_1]to[a_1,1]$ be an arbitrary increasing bijection. We can work our way to infinity from there: Assume we have an increasing bijection $g_ncolon [0,a_n]to [a_1,a_{n+1}]$ such that $g(g(x))=x^2+1$ for $0le xle a_{n-1}$. Then define $g_{n+1}colon [0,a_{n+1}]to[a_1,a_{n+1}]$ by $$g_{n+1}(x)=begin{cases}g_n(x)&0le xle a_n\g_n^{-1}(x)^2+1&a_1< xle a_{n+1}end{cases}$$
(noting that the two cases agree for $a_1le xle a_n$)
and finally $g(x)=g_n(x)$ for any $n$ with $a_nge x$. One verifies that this is well-defined and indeed $g(g(x))=x^2+1$ for all $x$.

Note that the sequence of $a_n$ goes like this: $$0, a_1, 1, a_1^2+1, 2, (a_1^2+1)^2+1, 5, ((a_1^2+1)^2+1)^2+1, 26, ldots$$ and that it is quite clear that we can make our choices (even under the restriction of requiring continuous $f$) at least such that $f(x)$ is an arbitrary number between $6$ and $26$.

Here is a javascript function that conforms to the original poster's requirement that $f(f(x)) = x^2+1$. The original poster asked what is $f(6)$. This program computes that

$f(6)=12.813735153397387$

and

$f(12.813735153397387)=37$

QED we see $f(f(6)) = 6^2+1=37$

function f(x) {

  x<0 && (x=-x);

  return x==x+1 ? Math.pow(x, Math.sqrt(2)) : Math.sqrt(f(x*x+1)-1);

}

It is not known that the function $f(x)$ can be expressed in mathematical terms (closed-form). It may be the $f(x)$ implementation in javascript above could help in discovering such a closed-form solution. I only know that the above javascript code adheres to the OP's requirement that $f(f(x)) = x^2+1$ for all $-infty<x<infty$ insofar as javascript has finite precision.

I wrote the code, BTW. I am sharing it in the hopes that it will be useful.

Javascript can be executed in a browser sandbox, here is one (no affiliation):
https://jsconsole.com/

Here is the equivalent description of the function implemented above:
$f(x)=begin{cases}f(-x)&x<0\x^{sqrt{2}}&xapprox x+1\sqrt {f(x^2+1)-1}end{cases}$

I believe this function is the optimal solution for $f(f(x)) = x^2+1$ but I can't prove it.