Information visualization is an increasingly widespread way to present and analyze complex data, but there is much we still do not know about how people understand visually presented information. Every visualization contains certain assumptions about the structure of its information: how the data can be broken down into pieces, how those pieces relate to one another, what actions can and cannot be performed with the data, and so forth. Yet information visualization still lacks the language and theory to analyze these properties of visual information structure. I propose that these structural properties can be thought of as visual metaphors that drive a visualization, analogous to the verbal metaphors that structure abstract information in speech and writing. In this model, people analyze visual relationships among shapes and patterns in a visualization in the same way that they analyze other kinds of visual scenes, then metaphorically interpret those visual relationships as conceptual relationships. I have grounded this proposed model through empirical studies showing how metaphors affect visualization use and how minor structural changes can have significant effects on the way people interpret visual information. I argue that this framework sheds new light on the importance of design and conceptual structure in visualization and can substantially improve future techniques and evaluation.