The wisdom is: "don't worry Australia is only talking about developing nuclear propelled submarines, not nuclear armed submarines".
However, the 3 cities Morrison short-listed as possible choices to host an East Coast Nuclear Submarine Base have valid concerns, on at least 3 counts.
1. The reactors bought from the UK or US, to be used in these Australian submarines will contain nuclear weapons grade Highly Enriched Uranium (HEU). See page 20 of this publication, where it states:
"The nuclear submarines of the U.S. and the UK are propelled by nuclear reactors fuelled with the best nuclear weapon-usable material, HEU enriched to 93-97 percent or more."
In itself this may only be a low rated safety problem. However terrorists, insider sabotage or even conventionalChinese submarine launched missiles blowing up the reactors of these Australian submarines might spread radiation over a wide city area. See radiological warfare.
2. An Australian East Coast Nuclear Submarine Base is expected to host visits of UK and US nuclear submarines that carry cruise missiles. The UK and US will neither confirm nor deny their submarines are carrying nuclear weapons. There are increasing discussions that warheads of some Tomahawk or other cruise missiles carried by these submarines may be nuclear armed. See:
"Reports from early 2018 state that the U.S. Navy is considering (re)introducing a (yet unknown type of) nuclear-armed cruise missile into service."
3. Russian and Chinese plans for nuclear warfare, work on a list of targets based on priorities. An Australian East Coast Nuclear Submarine Base that may even host nuclear armed UK and US submarines, would be high on the Russian or Chinese nuclear missile hit list.