Example: If in the course of a fight in say, a library a creature gets buried under a pile of books, blocking line of sight.

If an aoe spell is cast where the creature is, does it take damage?

Total cover blocks AoE unless the effect says otherwise, with the DM having the final say in any odd cases.

A pile of books generally isn't a totally solid barrier, so an AOE spell would get through.

It may depend on the spell. I know, for example, fireball and cloudkill both mention how they spread around corners. But if another spell doesn’t have that wording, for example moonbeam, you might be able to argue that means it doesn’t. 
Im not sure that would necessarily apply in this case, more just I think the specific spell would matter. 

Per the rules for areas of effect,

A spell's effect expands in straight lines from the point of origin. If no unblocked straight line extends from the point of origin to a location within the area of effect, that location isn't included in the spell's area. To block one of these imaginary lines, an obstruction must provide total cover.

Note, however, that certain spells specify otherwise, and what exactly counts as an obstacle that provides total cover is not well defined.