One rather big surprise for me in my last little bit of undergraduate education is that both physicists and mathematicians often think of functions as being points in a vector space. This can be an incredibly powerful idea for instance the fourier transform is a projection onto a set of orthonormal basis vectors which are the appropriate family of complex exponentials. In the case of quantum physics these changes of basis take on actual physical meaning. For instance one can formulate wavefunctions as functions of position or momentum. These two wavefunctions are related to each other by a fourier transform or in other words by a change of basis. In fact the connection is even deeper than that. The heisenberg uncertainty principle is a side effect of the fact that compactly supported functions in one basis must have infinite support in the other basis. This is obviously not true of all bases we might choose (wavelet bases for instance) but the certain special bases that do have this sort of dual relationship seem to have a lot of interest for us. In fact an important part of the apparatus of quantum mechanics is using information about the commutativity of different operators. The fact that the position operator and the momentum operator do not commute implies that their bases are necessarily "inconsistent" in the sense that you can never have a wavefunction which has a finite representation in both.
But when we talk about these function spaces generally what we are talking about is L2(R) which is to say the space of square integrable functions on the reals. In general we could expand our horizons to say include L27(R) which is to say all functions for which the integral of the 27th power of that function is finite. But no matter which space you choose no integrable function space is going to include say the function x^2. It might seem odd at first that by far the most worked with function space L2(R) doesn't even include the polynomials (not any of the polynomials). But the reason is that we like dealing with functions which have a finite amount of area under their curves. This fact doesn't tend to ever be much of a problem because if you want a function which isn't in L2(R) then you just truncate it at some finite limit M and then let M go to infinity.
In fact it seems (at least for nice functions) that not only is the fourier basis a basis for functions in L2(R) but most any function on the reals. But there is of course a problem. I glossed over a little problem earlier, that is that the fourier transform of a function will not always perfectly reconstruct a function. If the function is continuous then all is well and the function is in fact exactly reconstructed. However at points of jump discontinuity the fourier transform fails to reconstruct the function and instead takes on the value of the midpoint of the discontinuity. The reason this isn't a problem is that we view functions in L2(R) to be "the same" if the "distance" between them is 0. Meaning basically that they differ only on a set of measure 0. Now when you move to functions which do not have a finite norm suddenly things become a whole lot more complicated. because the function is no longer bounded even if the function and its reconstruction from a fourier (or some other) basis differ only on a set of measure 0 there is no guarantee that the "distance" between them in the function space is 0.
Even more disturbing is that when we think about the "distance" between x and x^2 we come up with infinity. From the physical perspective it is actually a good thing that functions like x^2 are not part of the function space that is used to describe the real world. Otherwise we would allow infinite energy solutions to the wave equation. But I can't help but be deeply uneasy about the fact that there is no good way to incorporate even simple divergent functions into a nice function vector space like L2(R)