Simple Summary Inflammatory breast cancer (IBC) is a rare and aggressive form of breast cancer (BC) in which higher levels of stromal tumour-infiltrating lymphocytes (sTIL) before neo-adjuvant chemotherapy (NACT) are associated with a better outcome. The role of sTIL in patients with residual disease (RD) after NACT is not clearly established. In this study, we showed that a high number of sTIL after NACT was associated with a worse outcome. Furthermore, we also demonstrated that sTIL decreased more during NACT in IBC compared to subtype-matched non-IBC patients (nIBC) treated with NACT. We also looked at the effect of NACT on some peripheral immune markers. Unlike the sTIL, we could not demonstrate a prognostic effect of these markers after NACT and their change was not significantly different between IBC and nIBC, indicating that the effect of NACT on the peripheral immune response seems to be similar in IBC and nIBC. Introduction: Inflammatory breast cancer (IBC) is a rare but aggressive form of breast cancer (BC) in which the (prognostic) role of stromal tumour-infiltrating lymphocytes (sTIL) and the peripheral circulating immune cells in patients with residual disease (RD) after neo-adjuvant chemotherapy (NACT) is not clearly established. Methodology: To describe the evolution of sTIL and some peripheral inflammation markers (Neutrophil-to-lymphocyte ratio, Platelet-to-lymphocyte ratio and Lymphocyte-to-monocyte ratio) after NACT in IBC, we retrospectively collected clinicopathological variables for 125 stage III IBC patients. sTILs were scored by three different researchers on an H&E slide of the mastectomy specimen. A cohort of subtype-matched non-IBC breast cancer patients (nIBC) treated with NACT was included for comparison. Results: There was no significant difference in the pre- and posttreatment sTIL scores between IBC and nIBC and in both groups the number of sTIL was significantly lower after NACT. However, the IBC phenotype did correlate with a stronger decrease of sTIL after NACT (OR: 0.25, 95% CI: 0.073-0.76, p = 0.018). The change in the peripheral immune markers was not significantly different between IBC and nIBC. After NACT, 75 patients had residual disease. In this group, a high number of sTIL before NACT (HR: 0.23, 95% CI: 0.05-1.02, p = 0.05) was prognostic for a longer OS, while a low number of sTIL after NACT (HR: 0.33, 95% CI: 0.11-0.98, p = 0.046) and a low residual cancer cellularity (HR: 0.20, 95% CI: 0.08-0.52, p < 0.001) was associated with a longer DFS. Conclusions: IBC is associated with a significantly stronger decrease of sTIL after NACT compared to nIBC. Furthermore, a high number of sTIL after NACT was associated with a worse prognosis in IBC.