We cannot ignore it. It is a major factor. Attempts to dissociate the two are fraught with difficulty. No Australian city wants to wake to a nuclear disaster on its harbour front one morning because it ignored the issue. As much as you believe the two can be disassociated most Australians would disagree.
It is not a major factor except for those who try to conflate the issues to confuse people or are confused themselves. Also, these are not simple things operated by simple people. There are/will be care processes/systems etc put in place t ensure safe operation. By way of example, exactly how many so-called nuclear disasters have happened with nuclear powered vessels (be they submarines or otherwise) world wide since the dawn of the nuclear era? Apart from Soviet ones (and I hardly think that is a fair comparison) the last I am aware of would be the 1973 incident on the USS Guardfish (SSN-612) but even that was relatively minor. So we are talking about extremely small numbers - on fact, comparatively speaking one could argue that conventionally powered vessels are more dangerous.
People who attempt to disassociate the two do so because they try and hide the possible effects from the general population. A nuclear disaster isn't so particular, radiation doesn't say, "Wait a second, lets choose who we are going to affect and leave all those who like nuclear power alone..." A nuclear powered submarine patrols far from Australia ports most of the time but it has to return to a port, to exchange crew, revictual and refit upon occasion. We have had nuclear disasters in "well run systems" in the past, in Japan, Russia, France, the US, mostly "minor" but they did occur. No Australian city wants that to occur. Do you? Australia has come near at Lucas Heights apparently. I had a friend who was an Occ.Health and Safety Officer who had some hair-raising stories which circulated amongst that community about that facility. We are just lucky they were caught in time before disaster struck. Nuclear is as nuclear does and we have more than enough disaster stories from around the world.