Previously we have written about some of the challenges that occurred because of the CloudStrike outage. (CrowdStrike performed a root cause analysis of the problem and shared their analysis results and process changes to prevent it from happening again. Many of these are outside the typical audio video system.) This problem is not just a CloudStrike issue, it could just as easily happen to AWS, Google Workspace, or Microsoft Azure.
To paraphrase many management books the next sentence is, “Now that you have told me about this problem, how are you going to fix it? What are your solutions?” CloudStrike did outline the fixes and solutions in the analysis. From their statement, “Engage two independent third-party software security vendors to conduct further review of the Falcon sensor code and end-to-end quality control and release processes. This work has begun and will be ongoing as part of our focus on security and resilience by design.”
Advisist Group believes that this transparency is not only important but provides methods for all of us to improve our businesses. CloudStrike’s next steps are peer review, process-based quality checks, and release processes. These same principles can be applied to audio video control system designs, not all need to be implemented but each one independently definitely provides improvement.
Advisist’s first step would be to create a Predictive Failure Tree (PFT). Some may refer to it as a Predictive Failure Analysis (PFA), but they are not the same thing. A PFA is designed to monitor systems in use. It is a way to take measurements to find warning signs, such as a temperature being too high. A PFT is done during the design process. It is often a tree or a flowchart that has a unique problem per branch. For an AV system the branches could be, network disconnection, touch panel failure, software glitch, a piece of equipment over heating … etc. For each item, the approach is to define how it would impact the system and the mitigation options.
PFT studies are not the same for all systems nor will they result in the same situation. The idea of using similar faults as starting points is a great way to start. For example, a failure of a damaged loudspeaker is a good item to consider. The impact and mitigation would be different between a small huddle room and a large stadium. By evaluating the issues before a problem occurs the design team can determine a way to address the issue. Sometimes there is no solution, the system is offline until condition is resolved.
PFT’s is a good place to start collaborating with others. Everyone has their own unique experiences. Trying to list all the problems that can occur will never be complete. The PFT can be thought of as a living document for all projects, not just created for one. Decide which ones do not apply and prune them off the tree. Including user error is a valid thing as well that is often overlooked.
Only two things are infinite, the universe and human stupidity, and I’m not sure about the former.
– Albert Einstein
The process of peer review is listed as a preventative tool to combat future issues by CloudStrike. Peer reviews are a checkpoint for design accuracy and quality. After working with a project for an extended period, human nature and familiarity cause things to be overlooked [some refer to these as blind spots]. When someone not familiar with the project examines the design, they bring a different perspective that can uncover flaws or inconsistencies that might be missed by a single designer. This collaborative process is critical for ensuring that the AV system—hardware, software, and network infrastructure—work together as intended. Reviews minimize errors before they become costly issues. They can also be used to ensure compliance with industry standards. One can also involve experts from various fields like acoustics, lighting, and IT networking to create a sound design from every angle. (pun intended)
Beyond quality assurance, peer reviews are valuable for knowledge sharing. During the review process, people learn from each other, gaining insights into different approaches, new technologies, and best practices. Reviews might seem like an additional step, optimize project success. By identifying potential issues early, peer reviews prevent rework when problems are discovered during fabrication or installation phases. Even during procurement having someone check that all the accessories are included helps to improve the project. Finding and addressing errors during design phases is far less expensive than addressing them later.
Success of a project is measured by client satisfaction. Peer reviews help ensure that all client requirements are met by providing a fresh perspective. It also reduces the risk of a simple typographic error being found by a client which causes doubt that attention was paid to the details. This issue is not just for milestone submittals but even for items such as proposal documents and final closeout documents. I believe that all of us would like to avoid spelling a client’s name incorrectly on any document. If internally a bid or project is called USF, it would be bad to have University of Santa Fe in the documents delivered to Utah Shakespearean Festival.
Advisist Group does offer these services. We can create Predictive Failure Trees a various level. We can draw on our experience submitting and reviewing bids to review proposals before submittal to minimize things being overlooked. As some of Bradford’s former colleagues know, he has created a list of over 100 items to review for a design package submission. Please reach out to discuss this topic and our services more in depth.