NSQIP Calculator Yields Reasonable Risk Estimates, Study Finds

Study of Acute Care Surgical Patients; Underestimation of SSI Risk Skews Overall Results

By Monica J. Smith

Nashville, Tenn.—Past studies of the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator, developed in 2013, revealed that it tends to underestimate the risk for postsurgical complications. New research, however, shows that it is fairly accurate in gauging which acute care surgery patients are at higher risk for adverse outcomes.

“We know that acute care surgery patients are often sicker than their elective counterparts; they may be septic or in septic shock, under-resuscitated, and may not have had the opportunity for preoperative optimization,” said Jessica Burgess, MD, assistant professor of surgery and an acute care surgeon with the Eastern Virginia Medical School, in Norfolk.

“We hypothesized the ACS NSQIP Surgical Risk Calculator would underestimate postoperative complications and length of stay (LOS) in this patient population,” she said.

To test their hypothesis, Dr. Burgess and her colleagues conducted a retrospective study of all patients who were admitted to their institution’s acute care surgery service and who underwent laparotomy in the span of one year (N=98).

“The preoperative risk factors we looked at in the chart review included demographic data; functional status—whether they were living independently or came from a nursing facility; emergency status of the case; ASA [American Society of Anesthesiologists] status; wound class; and a number of patient risk factors, such as diabetes, hypertension, dyspnea and acute renal failure, at the time of admission,” Dr. Burgess said.

The risk calculator allows for a subjective assessment by a surgeon who can add a variable of risk: 1, 2 or 3 . To maintain uniformity, the researchers rated all patients with a risk adjustment of 1.

“The complications we looked at were pneumonia, pulmonary embolism, cardiac complications, infectious complication, renal failure, readmission, return to the OR [operating room], death, discharge to nursing facility and LOS,” Dr. Burgess said.

They entered the patients’ data into the calculator to obtain his or her predictive risk profile, and then compared the predictive rate of complications with the actual rate of complications documented.

“Overall, the calculator tended to underestimate the rate of serious complications and overall complications, and while this trend came close to statistical significance, it did not reach it,” Dr. Burgess said.

“The calculator was able to predict the majority of complications: pneumonia, cardiac events, urinary tract infection, renal failure, return to the OR, death, discharge to skilled nursing facility, but it did significantly underestimate the risk of SSI [surgical site infection], which is likely why there was an underestimation in the overall complications.”

The NSQIP Surgical Risk Calculator may help guide preoperative discussions.

In their initial analysis of hospital LOS, the researchers found a large discrepancy between predictive and actual stays (9.7 vs. 13 days). But when they excluded patients with an unusually long LOS, in excess of 30 days, they found the calculator to be fairly accurate at predicting LOS.

Dr. Burgess acknowledged the limitations of a single-center study with a short study period. “Certainly we would benefit by having a larger study with more patients. The documentation of preoperative risk factors was not always complete, and we were not able to include that individual surgeon risk adjustment,” she said.

Overall, she concluded that the ACS NSQIP Surgical Risk Calculator is a reliable tool for the acute care surgery population. “While it’s not perfect, it may be useful for preoperative discussions with patients and their families to not only set realistic expectations but to initiate extended care discussions with patients who we don’t think will do well after surgery,” Dr. Burgess said.

Michael R. Bard, MD, medical director of the surgical ICU and associate professor of surgery at East Carolina University, in Greenville, N.C., suggested that perhaps a better use of the risk calculator is as a quality metric, to compare one’s own institution with the larger sample from which the tool was derived to see whether there is room for improvement.

“We know that most risk calculators, including this one, are created based on population samples in the hope of helping us gain some insight into how individuals respond to an event,” he said. “The problem is that you either get the complication or you don’t. Our human nature likes to predict the possibility of an event based on percentages and to proceed with a recommended course of action, but this is fraught with unintended consequences.”

For instance, if a patient is told he or she has a 10% risk for a wound infection and winds up being among that 10% despite best efforts, “they are going to perceive that something has gone wrong,” Dr. Bard said. He asked Dr. Burgess whether she and her colleagues had changed their practice based on the findings of their study, and whether patients have decided not to undergo a procedure after being presented with the calculator’s estimate of their risk.

Dr. Burgess said they do not routinely use the risk calculator, especially when the decision seems relatively straightforward and when there is no reasonable nonsurgical alternative.

“The times I think this calculator is most useful is when patients have both surgical and nonsurgical alternatives, and in patients with significant comorbidities to discuss the possibility that proceeding with surgery isn’t the best thing for this patient,” she said.

Leave a Reply

Your email address will not be published. Required fields are marked *


three × 5 =