I don’t know Canvas LMS.
… but there can hardly be a setting to open the course in a new window from inside the course, be it Lectora or Storyline. The course needs to be opened before any action can be executed.
The “Open in new tab” setting should be the way to go. Mobile devices, i.e. the respective browsers, don’t provide a “New window” and the default setting for most desktop browsers is also “Open in new tab instead of new window”. You can create a launch page in the course as Joe mentions, but ,opening “On: Show” is likely to activate the popup blocker of the browser. You may put a button to click on the launch page, but it will most likely also open the course in a new tab.