If for some reason you can’t put a button/link on the failure page to take the learner back to the beginning of the course, then the absolute best and easiest way to do this is via the LMS. However, not every LMS will let you configure options to the extent that you can have it reset a learner’s progress/status if they fail the course.
If that’s not an option, then from the SCO-side I would try setting the <span class=”rt_element”>cmi.core.exit </span><span class=”rt_datatype”>value to “logout” or alternatively “” (empty string). It’s the “suspend” status that tells the LMS the user intends to return later and saves their location, but in your case you want the LMS to treat the learner that has failed the course as though they’re taking it again for the first time in order to effectively reset their progress by “tricking” the LMS into not recording that value from the user.</span>
Note: I’m not 100% certain if this will apply to assessment questions, but as long as you’re not having the assessment answers retained between session in Lectora’s options you shouldn’t have to worry about it.
So basically, if the learner fails the quiz twice, they should reach the page telling them they’ll have to retake the course. Viewing that page should trigger your JS code to set the cmi.core.exit value. However, Lectora may have your content send an exit status after your code, effectively overriding it.
What I would do is have an HTML button on the failure page that the learner must click, which then triggers entirely through some JS code the setting of the cmi.core.exit status value and the actual ‘exit’ function for your content (instead of using Lectora’s actions). It’s possible you may not even need the step to set the cmi.core.exit value though if you trigger an exit from the course this way, as the LMS needs to explicitly be sent a cmi.core.exit value of “suspend” to save their progress in the first place.