Single points of failure: Are they your new business opportunity?
Some recent, and no-so-recent, bad news stories could be the basis of an electrical business opportunity. Peter Vandenheuvel looks at how enterprising electrical businesses might be able to grow their business.
A large cargo ship, the Dali, recently lost power and collided with Baltimore’s Francis Scott Keys bridge, bringing much of the USA’s Atlantic Ocean Seaboard to a halt. It got me thinking about how overlooked dealing with single points of failure appears to be across our world, and what opportunity this can offer us in our electrotechnology industry.
ADVERTISEMENT
In the above instance, it appears most likely to have been because of a total power failure on board the ship, making it impossible – amongst other things – to control the ship’s propulsion and steering. The result was a collision with a bridge pylon that brought down the entire three main sections of the 1.6-mile four-lane bridge.
Not only did this single point of failure (SPOF) most regrettably and unfortunately cause the death of six workers doing night-time road repairs on the bridge, but it could also have been a lot worse. Lucky for all, the ship was able to send an SOS in sufficient time to stop traffic from crossing it.
Additionally, the bridge will have to be entirely rebuilt at a huge expense, and the 35,000 or so cars and trucks using it daily, will have to find another way across the river as a work around until it’s back into service.
Isolated incidents?
No, we read about catastrophic incidents nearly every day. And what do these all have in common? With the hindsight of 20-20 vision, it always becomes obvious that these incidents could have been prevented.
There were even some comments in the press the Dali may have had a similar problem prior. If that was the case, the question going begging would be: “Why wasn’t something done about it before?”
Then, in many such instances, it will not be long before the blame game starts, with the claims for compensation following soon after. And that would be very difficult for those responsible (and their lawyers) to defend.
Even in the maritime industry, an industry that would be considered to be well regulated, there have been many serious, but potentially foreseeable, calamities.
Only one or two years ago, there was a Norwegian cruise liner, the Maud, stranded adrift and listing considerably in an extreme storm in Norway in the middle of a narrow fiord with no power.
This resulted in many of the passengers and crew having to be airlifted off by helicopters, a few at a time and in the middle of that fierce storm, all as the ship drifted helplessly toward the nearby cliffs because the waters were too rough and the list so steep it was not possible to use boats to evacuate them by sea.
So, if the very closely controlled maritime industry can be prone to SOPFs, given how everything in this modern world relies on electricity, our electrotechnology industry is even more so.
Recent history on real-world examples shows even the strictly-controlled nuclear industry is not immune. The tsunami that hit the Fukushima nuclear power station was another. There, it was the low-level location of the electric cooling pumps, with no diesel-driven units providing backup at a higher elevation.
Also, who can forget the September 2016 blacking-out of the entire South Australian power network for more than a day? That was caused by the collapse of many HV transmission pylons due to very strong winds – with that also not helped by the premature closing of the Port Augusta Power Station.
What is a SPOF?
There are several definitions, but they all mean the same thing: “A single point of failure of that system ‘part’, device, action or whatever, which – if it fails – will stop the entire system from working”.
In other words, if “that something” – whatever it is – goes wrong, everything stops.
It’s not surprising that SPOFs are seen by many in the know as probably the biggest risks not on the broader industries’ radar.
And SPOFs can take many forms. Amongst others they can be a; machine, machine part, process, procedure, software program, power supply, etc, to name but a few. Even a serious weather event or a person with very specific knowledge or skills could be considered as either a SPOF or a root cause of one.
So, technically SPOFs can exist as parts in many different electrotechnology sectors and activities, including;
- Power generation, distribution and user installations,
- Installation sensing, safety and protection devices,
- Installation main, sub-main and circuit switches and controls,
- Machines, and the wear in critical parts, devices and controls,
- Computers, data storage equipment, networks and components,
- Operating, monitoring and control software and other equipment,
- Systems, processes, and procedures that control a system’s workings,
- Bottlenecks and chain reactions, setting off others downstream.
But this list is almost endless, bearing in mind the many different situations where they can arise. In fact, it can easily also include key personnel where attendance by a specially qualified person is essential to their operation in certain situations.
An exceptional opportunity
It’s not often a new business opportunity presents itself in our electrotechnology industry. After all, typically, most businesses in our part of the industry are always waiting for that call to either price – or better still carry out – works or services.
That is because most of our traditional work is reactive, we have to wait for someone to call. But with this opportunity, however, we can be in the box seat and get proactive. You can use your customer base to generate extra work from them and at the same time find other new customers to add to the list.
This is why we, in the electrotechnology industry, are so extraordinarily well placed. Because, as well as finding other potential SPOFs, we already have all those customers that even now have SPOFs.
Why? Because every customer’s electrical installation has at least one single point of failure as a starting point – their point of supply. And there are potentially many more somewhere in their installation.
In addition, there are a number of other reasons we are so well placed including;
- The existence, identification and overcoming of SPOFs is now repeatedly overlooked,
- The skills needed are those already the foundation skills in your business now,
- Countless SPOFs involve electricity and electrical failures,
- We can identify and remedy SPOFs before failing, saving damage and repair costs,
- Their elimination can be readily integrated with solar and wind generation,
- The potential customers are already known to you as their dependable service provider,
- Many customers will want the risks eliminated and so avoid serious consequences,
- Many others will have to take action when their high-risk issues become evident,
- The awareness of their SOPFs can be introduced by you during regular customer callouts,
- This opportunity can combine SPOF elimination with solar and batteries,
- The staggering consequences are preventable.
Typical SPOF situations
Obvious situations would include, but certainly not be limited to, blackouts and any critical activities involving health, public safety and other situations with potentially catastrophic consequences, like in:
- Hospitals, nursing homes and other medical situations including homes,
- Hazardous or potentially hazardous machinery and chemical processes,
- Contamination and or waste of work in progress in food and pharmaceutical production,
- Absence of key specialist persons during high-risk activities or processes staff.
It should also be noted the bigger the installation, the bigger the SPOF risk, the more appealing its elimination potential.
The business opportunities
In our various parts of the electrotechnology industries, the opportunities are manifold. And they can be applied across most of the activities undertaken in your areas of that industry.
The activities can include all manner of simple to complex SOPF identification studies, mitigations, upgrades and eliminations. Also, they can involve all customers and prospects including those from the big end of town as well as rural-based high electricity users, to the smaller end of town for more simple solutions in domestic, residential and smaller industrial and commercial type installations,
At the larger end of town, it can:
- Overcome power blackouts in large enterprises and facilities,
- Reduce productivity losses from power outages,
- Reduce workplace hazards being created during blackouts,
- Prevent data loss from blackouts with data centres,
- Eliminate ‘work in progress’ waste in food and pharmaceuticals,
- Offer the biggest opportunities in installations with the biggest SPOF risk.
In the domestic and smaller ‘end of town’ businesses and other facilities such as schools it can reduce or overcome:
- Power blackouts in multistorey and high-rise residential buildings
- Serious consequences in facilities or homes where infirm persons reside,
- Power blackouts in bushfire and flood-prone areas,
- Power blackouts to home computers and or security systems,
- Productivity losses including for working from home employees.
The actual opportunities right across the whole industry are almost endless.
Identifying SPOFs
Here is the first challenge. No doubt identifying and eliminating higher-level SPOFs will become a specialist area of expertise if it’s not already done now.
There is unfortunately not really much outside help by way of identifying plans, charts or other tools specific to identifying undefined failures, so when you become more experienced, your expertise can be in high demand.
And you can develop as you go, especially if you (initially at least) keep it to what you are familiar with in the power-related SPOF issues. So, you can get a bit of a head start, as often you will be able to use a single-line diagram to see where the SPOFs ‘sit’ in the overall installation.
How? Here is a list of some of the issues which may need to be considered (so in other words a rough plan) that may be a guide for the SPOF identification process. But please remember, each situation must be identified on its own merit, as it will be different from the other various types of SPOFs being identified. Here are some of the suggested steps:
- Identify the actual failure risk, or different risks, to be overcome
- Evaluate and map the dependencies from the impact of failures
- Map any redundancies and failover equipment already in place
- Quantify the risk or different risks by risk analyses (see note below)
- Evaluate and prioritise the risk, or risks, to determine actions needed
- See if any similar risks can be grouped and dealt with together
- Determine if the SPOF affects the entire enterprise or only parts of it
- Design a plan or workaround to overcome the SPOF(s)
- Cost the plan(s) and any other requirements (eg. shutdowns, training etc.)
- Fine-tune the plan(s) for its optimal cost, practicality and cost-benefit
- Determine the possible all in cost and consequence of a failure that it will save
- Present the proposal, failure cost and any alternatives if helpful
- If approved, proceed with implementation
- Manage the implementation
- Confirm implementation has been successful
About note four above; in undertaking any risk analysis, wider risk factors will also have to be taken into account, such as what risk the failure may present to the:
- Business continuity, customer confidence and long-term sustainability
- Hazard to and safety of employees and the public at large
- Cost of unexpected repairs saved by preventing the failure
- Added cost also saved from any extended failure duration
- Saving the loss or waste of materials or products in progress
- Damage to your brand and or quality image
Eliminating SPOFs
Sure, there will be a number of cookie-cutter or textbook solutions like installing stand-by power supplies, integrating batteries within systems, use of uninterruptible power supplies (UPSs), installation of early warning systems, additional preventive maintenance and so on. This is especially true in our industry where the introduction of redundancy, duplication or failover mechanisms, either at the point of supply or nearer to the SOPF, can be cut in downstream of these if the failure only affects part of the installation. In those cases, this flexibility can also be utilised.
The SPOFs can be from very large systems for data centres and the like to much smaller units for small industries and domestic situations, including those at home where a medical situation exists.
Also, with the cost of small (and often portable) petrol-driven power supplies having remained reasonably affordable, these, hooked up with automatic starting and low fuel alarms, can bridge many power supply gaps.
Likewise, there may be opportunities to install new or extend existing, solar generation and/or new or upsized batteries as part of the SPOF mitigation as an informal UPS.
Such units could then even be used in bushfire-prone areas to provide some power during power failures (and through temporary network disconnection during fires) and for pumping water over the roof of a property in the event of failure, to protect properties at risk.
Although these remedies may cause a short transfer blackout, as they will need some form of switching to operate, a short blackout period of a few seconds is often quite acceptable.
Likewise, smaller UPSs can be used for short-term outages and with some smart interface connected to home computers, to shut systems down systematically if the blackout has not ended within a predetermined time. And thinking about UPSs, with the high level of take-up of solar energy, these systems (if coupled with a battery) could be set up to provide a more reliable power source for overcoming SOPSs 24/7 as opposed to only during daylight hours in many situations.
But there will be many situations where a high level of ingenuity will be required to develop cost-effective no-break solutions. However, this is not necessarily a problem because it will at least separate the specialist entrepreneurs from their me too competitors.
At the same time, there will be many obvious and previously proven remedies in many situations that will be quite easily implemented, especially in the more routine installations.
The N+1 solution
This is a commonly industry-adopted backup solution where reliability is paramount and often used as a means of failure mitigation. The term N+1 relates to building in permanent redundancy. In its most straightforward applications, it simply means that if the SPOF requires an ‘N’ capacity of something for it to work, then to eliminate the SPOF, you would have to install an added redundancy of at least one additional spare equal to the largest of the critical resource now in place.
For a simple example case, if you have, say, a plant or data centre that must operate 365-24-7 regardless, with a 500KW grid supply and a 500KW backup generator as a standby, this situation will require an additional (second) 500KW generator on automatic standby as back up in case the first standby unit failed.
Alternatively, however, if instead of the 500KW generator, a pair of 250KW stand-by units are already installed in tandem (so still providing 500KW), then the extra third unit can be 250KW as it is less likely for both smaller units to be out of service or fail at the same time.
Also, with a fuel-powered generating plant, the supply of fuel may itself become a SPOF if the volume of stored fuel or any delay in getting it topped is a potential issue. In those circumstances, where no standby generation start-up switching blackout period whatsoever can be tolerated – as in data centres and similar computers or other critical applications – where the power supply must never be interrupted, sufficient N+1 capacity UPS to bridge the switchover gap and a type suitable for the installed equipment, must also be permanently connected in the circuit.
Providing value for money
This will be a major challenge to those looking at this as a business opportunity. This is because unless you present a compelling case to show the combined costs of a failure will substantially outweigh the cost of any additional work, the proposal is unlikely to be successful. So, believe it or not, your ability to prove the saving can then of itself become a major SPOF for your new opportunity.
Also, in those situations where complete elimination of a SPOF is not practical or cost-effective, you may need to develop alternative part solutions that can at least lower or mitigate the risk, so a compromise workaround may need to be considered. Bear in mind as well that when adopting an interim short-term fix, a planned long-term solution can be a solid second-stage elimination strategy.
All this will certainly be a test of your ingenuity to develop practical a cost-effective elimination or SPOFs mitigation.
In all the above, it must of course be remembered that in situations where a SPOF identifies a potential catastrophic outcome or risk, such as involving potential serious injury, death or any other hazardous outcome, sufficient elimination or mitigation that reduces such hazards to an acceptable level should be strongly recommended to the owner or user of the system.
It’s further strongly suggested that any such recommendation be in writing so that the potential for you to later be called to account is substantially mitigated.
Also, as a plus, the results of the risk analyses can add weight to the need to remedy those SPOFs that are shown to create hazards regarded as high risk.
Risk analyses will be paramount
As part of the decision-making, concerning the overcoming (or not) of a SPOF, a risk analysis of the consequence of an identified potential SPOF failure using a recognised risk analysis tool should always be undertaken. Only in doing that can a well-informed decision be made for dealing with it.
This can then be used to inform the owner or user of the installation for them to make such a decision. The analyses will also assist in situations where total elimination of the SPOF may not be practical, but where keeping the status quo is also not acceptable. This could be by assessing other options that may at least reduce the consequence to a more manageable level.
Market entry strategy
It is suggested to not spend too much time initially on the detail as the situations will almost always be different. But this will at least to some degree depend on the:
- Understanding your existing typical customer profile – in terms of size, activities, types etc.
- Determining your preferred customer target market profiles – if different to the above
- Deciding your preferred market entry – as well as when and how quickly
- Listing the most obvious potential SPOF areas to look for
It is suggested to do this only as a back-of-the-envelope exercise, not wasting too much time on the details as part of an initial preliminary market test.
Suggested next steps
To complete the initial market testing for those wishing to explore this further the following is then suggested.
- If you don’t have a detailed existing customer list, start one,
- Identify those customers that may have a higher SPOF risk profile,
- From those, list the customers you have closer relationships with,
- From those, list especially those that are risk-averse or cautious,
- Do or have your key sales staff undertake SPOF identification training,
- Have a SOPF specialist trained in their identification and services,
- Training will possibly also need to include risk analysis,
- From that develop your initial basic SPOF introduction, audit and sales plans.
Initial customer introduction
Once confident you have done the initial preliminary introductions as above, at the end of a routine service call or other works, or by arranging a courtesy call to the customer if their work isn’t regular, just ask the owner or user something like: “What would be your biggest fear when having an unplanned power blackout?”
An explanation of what SPOFs and how they are overlooked until too late may then also be timely.
Then explore with them what that means in terms of costs, hazards, waste, inconvenience, loss of business etc. as earlier listed. You could then also further explore if there would be any interest in having the problem go away.
In addition, to explore if other SPOFs could also be reduced.
Also, closer to home
Practice what you preach! After all, if you are considering providing SPOF elimination services to others – regardless if you are taking up this opportunity or not – you should also be introspective.
How? By self-examining your own business to ensure you have no SPOFs! And this would be in any aspect of your business that is likely to cause it harm, such as:
- Having unexpected power network blackouts
- Your critical data not being regularly backed up
- Using known old or unreliable computer hardware
- Using outdated, unreliable, or unsupported software
- Using no or outdated network protection
- No one regularly tracking cash flow
- Having poor (or no) quality control
- Having poor (or no) OHS system adherence
- Having inexperienced key personnel or they leaving
- Having a lack of (or no) clear systems and processes
- Working from Insecure premises
- No regular fire protection testing
- Having no contingency planning
- Having outdated (or no) disaster recovery plans
So, are you ‘in’ and proactive, or ‘out’ and remaining reactive?
The ball is again in your court…
-
ADVERTISEMENT
-
ADVERTISEMENT