Matt Larkin (author of some rather spiffing riffs on Nordic mythology) posted an article recently suggesting fantasy novels can benefit from the lack of systematized magic. While I find his arguments engaging, for me the best answer lies in what is revealed rather than what is systematized.
Larkin’s thesis is that magic—at least magic that provides the experience fantasy readers seek—isn’t science; that magic which is explained is awful and terrible (in the modern sense of feeling a bit flat) rather than awful and terrible (in the archaic sense of invoking reverence and fear). As such, stories where the narrator and protagonists do not know how—or even whether—spells and rituals will work are more engaging.
As both a firm believer that fiction derives power from emotion not objective reportage and an avid reader of fantasy, I agree with Larkin that lectures and explanations remove the sense of otherness from magic. However, where I diverge—perhaps merely in stating rather than leaving unstated—is in whether the author should understand the magical system.
While I enjoy greatly the tension that comes from the narrator (and thus the reader) not knowing whether a spell will work or what the cost might be, I also enjoy deducing what the rules of magic might be from evidence in the same way that when reading detective fiction I seek to determine the perpetrator before the book reveals them. I-as-reader automatically assume there is a system behind the magic, and thus I-as-author foresee risks if I don’t have one in my works.
The most obvious is that of contradiction: no one would write a detective story in which the perpetrator was provably somewhere else when the crime occurred, and if that seemed to have happened, they would explain it; while magic, being a non-scientific force, does not need to adhere to so strict a degree of consistency, if a spell was useful or useless in a specific circumstance readers will expect that not to radically change without some evidence of a difference. Thus, it helps the author to know how magic works so they might seed differences (significant and not) into appearances of magic rather than leave readers with a sense that magic works or does not as the plot requires.
Humans are also pattern-seeking creatures, posing a second—more subtle—risk to magic without a plan: that of unconscious dissonance. Things that are part of the same system tend to have commonalities that are evident even to the layperson; for example, scatter parts from cars, steam engines, and computers in a room and most people would do a reasonable job of sorting many of them into rough piles even if they couldn’t do a perfect job of it. While “real” magic might be entirely without rules, the minds of readers are conditioned at the deepest level to expect fictional magic to obey the same rule of “similar feel”. Thus, it helps the author to know how the whole works so that the parts they do show all—underneath any in-world confusion—share a sense of fitting the whole.
So, even if the characters don’t know the rules of magic and the reader isn’t given enough evidence to deduce the truth, there being a truth beneath the mind-boggling occult events helps avoid the sense that the magic is just a series of ways of moving the plot on, freeing the magic to evoke awe and terror.
Of course, as a lover of Yog-Sothothery, I might be biased toward tales where fear comes from inability to comprehend rather than incomprehensibility itself.