It is true that, at least with regard to certain executive officers, the Constitution grants the president the power to remove them at will. As the Supreme Court has said, “there are some ‘purely executive’ officials who must be removable by the President at will if he is to be able to accomplish his constitutional role.” Thus, the president is generally considered to hold the power to remove officers who assist him in exercising his inherent Article II power to engage in foreign affairs, such as the secretary of state and ambassadors who represent the United States abroad. But that fact alone doesn’t mean Trump is off the hook.

The scope of the president’s removal power was debated as far back as 1789, when the first Congress passed a bill establishing the Department of Foreign Affairs. As part of the consideration of that legislation, the House of Representatives engaged in a lengthy discussion of whether the Constitution’s separation of powers meant that the president had the power to remove the secretary of foreign affairs without the consent of the Senate. In that debate, James Madison—a prominent member of the Constitutional Convention and signatory to the Constitution—argued that the president could not be required to ask the Senate for consent to remove the secretary of foreign affairs. As he explained: “Vest this power in the Senate jointly with the President, and you abolish at once that great principle of unity and responsibility in the executive department, which was intended for the security of liberty and the public good.”

This is not the first time that Trump has been accused of misusing his removal power.

Madison’s defense of the president’s power to remove officers like the secretary of foreign affairs, however, came with an important caveat. At one point, Madison acknowledged the “danger” that the president could “displace from office a man whose merits require that he should be continued in it.” For such an “abuse of his power” and “mal-administration,” however, Madison responded that the Constitution built in a safety valve: impeachment. As he put it, “the wanton removal of meritorious officers would subject [the president] to impeachment and removal from his own high trust.”

The notion that the president might be impeached and removed for an improper exercise of his removal authority is not simply a theoretical matter. In 1868, the House impeached President Andrew Johnson based in part on the allegation that he violated a federal law that limited the president’s authority to remove certain officers, including the secretary of war. At his Senate trial, Johnson argued in part that he should not be convicted because that federal law was unconstitutional as applied to the secretary of war, whom he said he enjoyed the unfettered authority to remove. That argument didn’t persuade one of the House members prosecuting the case for removal—Rep. Thomas Williams—who argued that the president could be removed even if he did enjoy the power to remove the secretary of war, so long as his reason for doing so was improper.

According to Williams, “[i]t seems to have been assumed by [the President’s counsel] throughout … that if they shall be able to succeed in establishing a power of removal in the President, … he may exercise that power at his mere will and pleasure, without reason and without responsibility.” Responding to that argument, Williams cited Madison, arguing that the Framers “who asserted the power of removal to be in the President” never “supposed that its exercise was to be a question of mere caprice, or whim, or will.” Williams argued that Secretary of War Edwin Stanton was surely a “meritorious officer” and there was “no sufficient cause for his removal,” and so his removal was “wanton,” “wicked,” and cause for Johnson’s removal from office. Johnson ended up being acquitted by one vote in the Senate, but it is noteworthy that Williams—like Madison before him—took the position that some exercises of the president’s removal power could be grounds for impeachment.

To be sure, the president has broad discretion to remove the executive branch officers who serve at his pleasure. But even if Williams might have overstated things in suggesting that impeachment could be appropriate simply because there was “no sufficient cause” for an officer’s removal, the fact that he considered impeachment to be an appropriate response to some exercises of the president’s removal authority supports the view that misuses of the removal power—especially corrupt misuses—could justify impeachment. Thus, if the House investigation confirms that Yovanovitch’s removal was part of a corrupt scheme to get Ukraine to investigate the president’s political opponents, her removal could itself justify an article of impeachment against Trump.

Notably, this is not the first time that Trump has been accused of misusing his removal power. Volume II of special counsel Robert Mueller’s report describes the president’s removal of James Comey as director of the FBI as being an attempt to derail the investigation of Russian interference in the 2016 election. As James Madison teaches us, then, although the president likely has the power to remove the FBI director at will, a corrupt misuse of that authority could also be a basis for impeachment.

In short, while Pompeo is correct that Yovanovitch served at the pleasure of the president, that misses the point. Although the president has the constitutional authority to fire an ambassador, that authority does not insulate him from being impeached and removed from office if Congress finds that his exercise of that authority was an abuse of his power. Thus, if Congress finds that the president removed Yovanovitch for the corrupt purpose of seeking to persuade a foreign power to investigate a political rival, that removal could be yet another offense that finds its way into articles of impeachment brought against Trump.