It was soon becoming apparent that the Articles of Confederation were insufficient as a means of governing the nation. Important legislators such as Thomas Jefferson and James Madison quickly realized the importance of establishing national laws to prevent the states from becoming independent countries on their own. The new United States was on the verge of collapse due to a lack of monetary funds, and a series of rebellions. Shays' Rebellion, in Massachusetts, was an uprising of debtor farmers led by Daniel Shays that resulted in a failed attempt to seize a federal arsenal in Springfield.

The Constitutional (Philadelphia) Convention was called in 1786, and held at Independence Hall in 1787. 55 men from throughout the colonies convened for the purpose of strengthening the Articles of Confederation. George Washington was chosen to preside over the convention. From the start of the convention, controversy arose concerning voting on legislation. Larger states such as Pennsylvania wanted their votes to count more than smaller states because they represented larger populations. Smaller states such as Rhode Island and New Jersey feared that their interests would be ignored.

On May 29, 1787, Virginia governor Edmund Randolph presented the Virginia Plan, a compilation of proposals drafted by future president James Madison. Madison proposed a three-tiered government with a legislative branch consisting of two houses (Senate and House of Representatives) that would make laws, an executive branch to carry out the laws, and a judicial branch to enforce the laws. Madison's proposal also called for proportional representation in both houses of the legislative branch. This meant that states would be represented based on their populations or the amount of tax payments paid. Furthermore, the House of Representatives would be elected by the people, and the Senate would be elected by the representatives. To quell the rising tide of state sovereignty (independence), The Virginia Plan would authorize the national government to have direct authority over American citizens, as well as to negate any state laws that were not deemed in the best interest of the United States.

While the larger states seemed to support the Virginia Plan, the smaller states began to voice their opposition. William Paterson, from New Jersey, warned that his state would never go along with the plan, and Roger Sherman, from Connecticut opposed the popular election (by the people) of representatives. Others, such as Alexander Hamilton, claimed that the Virginia Plan was too democratic, and failed to protect the government against the passage of popular, but ultimately, harmful laws. Nevertheless, the Virginia Plan was voted (7 states to 3) as the convention's basis for deliberations. Thus, the Articles of Confederation would be effectively replaced rather than amended.

The issue of equal versus proportional representation, however, was the most contentious issue and threatened to destroy the deliberations, and perhaps, the new nation. The smaller states would not agree to any plan in which the larger ones had more votes. On July 5, 1787, a special committee was formed to try to come to a compromise regarding the issue of representation. The Great Compromise, as it came to be known, formed an alternative plan in which the House of Representatives would include one state delegate for every 40,000 citizens of a particular state, and the Senate would have the same number of delegates, regardless of population, for each state. On July 16, five states voted for the plan, and four (the larger states) voted against it. It was a victory for the smaller states.

On July 26, another committee was formed to begin drafting what would become the U.S. Constitution. On August 17, 1787, the Constitution was signed. The Constitution was first ratified by Delaware on December 7, 1787, and then by Pennsylvania on December 12. Although wealthy and powerful citizens in many states were reluctant to lend support to the Constitution because they would be relinquishing some powers, influential officials such as Benjamin Franklin and George Washington voiced their support for the Constitution which helped sway popular opinion. Alexander Hamilton and James Madison were among those who wrote The Federalist - a series of political essays written to promote ratification of the Constitution. By 1790, all thirteen colonies ratified the Constitution and became states.