So here are some of his arguments:
1. Islam doesn't have any real miracles attesting it. There are a couple of miracles attributed to Mohammed, but they're mostly private ones and of a somewhat dubious nature. For example, when he apparently cut the Moon in two with his fingers (I'm guessing it didn't stay cut in two in light of the view from my window at this point). Luther says that this is too absurd to believe. Most modern Islamic scholars think that what is being talked about is a metaphorical splitting of the prophet's heart.
The interesting point here (that could also be made about Joseph Smith) is that when someone tells us that they've spoken to God, we're just supposed to believe them without any proof. Interesting.
2. There's no real continuity with the Old and New Testaments. Now Luther's point here is that the Church since the time of Adam and Eve was always defined by the gospel, but now (according to Islam) somehow that's changed and it's supposed to be defined by the Law of Mohammed.
Now, this might not be a good argument to use against a Muslim, but the point works I think when discussing matter with Christians or even possibly Jews.
Part of the Muslim argument is that the OT and NT were real and legitimate revelations of God, and that they were corrupted by additions and then Mohammed then appeared and reformed everything and the Koran is then totally uncorrupted.
The first problem with this is historical. There's too much manuscript evidence that demonstrates that the texts of the OT and NT were not corrupted to take any those arguments seriously (for example, there only four minor difference between the Dead Sea version of Isaiah and the MS version). On the other hand, the Koran has a dubious history. It was only written down 20 years after Mohammed's death and one Hungarian scholar has shown that up to two-thirds of it has been borrowed from the OT, NT, Gnostic Gospels, Greek wisdom sayings and Persian religious texts. To make matters worse, recent archaeological evidence from Yemen of all places, has uncovered early Islamic graves with version of the Koran in them that don't match up with the current version. So, in actuality, the situation is the opposite of what Muslims claim it is.
The second difficulty is conceptual coherence. So, what we're supposed to believe is that instead of having a God who faithfully preserves his Word from paradise to the last judgment, is in a God who allows total corruption of his earlier revelations and a total apostasy of the people of God, and then in a kind of deus ex machina move, sends Mohammed? Even more strange, is that he was faithless in all those other situations, but now some how has become faithful in preserving the Koran. Of course, the Mormons would have us believe the same thing, which in my mind is just as unconvincing.
3. Luther notes Mohammed doesn't tell us anything reason doesn't already tell us about God. Meaning, if he only got that far, God didn't really talk to him.
In other words, Mohammed said murder and lying was bad, and that God existed and that there was one God. Anyone can figure this out on their own- so why a need for revelation? According to Luther, God's grace (as well as things like the Trinity and Incarnation), can only be known by is own revelatory act.
I think point two and three are particularly important. The ultimate thing that proves Christianity is God's grace and his faithfulness to his Word.