If you access your results with by_state it creates and empty list entry for the accessed value.
Example:
I use len(results.by_state[np.Critical]) > 0: in Summary.problem to print Critical first followed by Warnings. This though creates an empty record for Critical in my tests (where no critical results exist only warning)- making them fail:
pprint((check.results.most_significant_state))
pprint((check.results.most_significant))
Result:
Critical(code=2, text='critical')
[]
This results in the whole check return CRITICAL, despite having no results that are actually critical (only warnings are present). The issue is most likely due to self.by_state = collections.defaultdict(list)
Workaround:
Don't use results.by_state if you are not sure it contains elements, use results.most_significant_state == np.Critical instead. Still this seems rather critical problem to me.