If you peruse the code in the xll library you might notice things being imported and exported. Libraries can be either static (.lib) or dynamic (.dll). Static libraries hook up functions at compile time. Dynamic libraries hook up functions at runtime. Import and Export controls what functions are available to your code.
The xll library is a static library. The .xll you create using it is a dynamic library, a .dll renamed with the .xll suffix so Excel can see it if you use File/Open… The /WHOLEARCHIVE flag is used to export all functions in xll.lib to create a self-contained add-in that does not require hooking up other dynamic libraries at runtime.

If you write an add-in that uses other shared libraries then it is on you to make sure they are available at runtime. If not, this is usually manifested by a bunch of gobbledy gook showing up in the spreadsheet after you open it. If your add-in uses a shared library then you have to make arrangements for that to be available at runtime. Use dependency walker to simulate what happens during runtime loading and identify the missing shared library. Charles Wright has reported there are newer versions of that.