The Cold War context cannot be forgotten here. The war was less about Vietnam specifically than it was about what it stood for symbolically. It was simply one piece in the global game being played with small nations suing for independence. The US was striving to prove democratic capitalism as the superior model in government and economy, while the Soviet Union (and China) were seeking to prove that communism was the superior model. To do so, each tried to gain influence in those small nations, supporting different factions.
Fearful of triggering a third world war (as well as wanting to reserve money for domestic programs) Johnson pursued a strategy of gradualism. There were plenty of military officers and civilian hawks (war supporters) who advocated for or believed that invading North Vietnam was necessary to win. Vietnam was unique in its proximity to China, one of the supporters of North Vietnam. Invading North Vietnam could also trigger a situation similar to the Korean War, or so some policy makers feared.
Some proponents of the gradualism plan genuinely believed it would be enough to achieve American objectives in Vietnam. This is generally put down to a fundamental misunderstanding of the Vietnamese. One way to put this is that the US was fighting a limited war while the Vietnamese communists (both North Vietnam and the Viet Cong) were fighting a total war.
TL;DR: American policy makers did not want to risk drawing Chinese or Soviet ground troops into the war.