Tuesday, July 17, 2012

Keeping the lights on - Upgrade or Expand

Invensys supports a large number of SCADA systems products supporting the supervisory control of many large infrastructure projects. The products themselves have extended life-cycle support.  These products are supported through last time buys, re-design and other means of functional replacement to make sure end of life components are available to enable manufacture to proceed throughout the service life of the products. 


This post is a fictionalized story of the kind of issues that are considered when doing upgrades for systems installations. Any resemblance to a particular site, people or customer is purely coincidental and the story could be set anywhere and in reality might result in a totally different outcome.



Ahmed as an engineer for a large automation company and was proud of it's long life systems products.  He had been with them 10 years or so. Since joining he had worked on DCS and SAFETY systems on plant's strung out on oil and gas fields of various sorts in  the nether reaches of his large spread out country. With the increasing age of many of the components of the systems, Ahmed had developed a a friend in old Jack, an old time field service engineer who knew everything.

All the customers knew Jack, but loved Ahmed since he spoke the local language and was always available by mobile or email when needed. His biggest customer was a large corporation and had many offices across the country as well as research centres and links to international universities. Naturally with such a large customer, they had many touch points in his own company through relationships going back quite some time.

Just recently he met up with a young graduate engineer named Aron in a local cafe who was working through an idea for a SCADA project. As it happened Aron was attached to Ahmed's customer's research centre and was working through the design for an upgrade.

-I'm working on a substation job where we just want to upgrade, add some more bays, that kind of thing.  Apparently these old RTUs are something that your company supports, and maybe you could help me modernize the system.
-Sure! said Ahmed, a bit hesitantly [SCADA was a new thing for him]. We can do that I am sure. Perhaps we should consider upgrading by replacing the older equipment with the latest model and architecture. It would be backward compatible with your existing installation, which must be 20 years old by now.


-Sounds good, said Aron. 
-OK let me know how.


Naturally Ahmed could hardly curb his enthusiasm, not only was this opportunity with the large customer, but it was a bit out of the ordinary and involved upgrading older technology.  Upgrading older technology was a bit of a 'theme' with his company at the moment.


Ahmed called up Jack who quickly told him it's not a problem, we know all about doing cutover plans and can change out the system and we can migrate and re-engineer the user programs.


So Ahmed called up the product manager who quickly told him 'ok don't worry we can just upgrade the existing system, by replacing all of the older CPUs with newer CPUs and then expanding by adding new equipment to the network.  The newer CPUs are network compatible, have more memory and you can even use the old programs and user code on the newer CPUs.


Ahmed started mobilizing resources in the company to put a quotation together.  It was relatively straight forward, but naturally all the resources were blocked out doing other things and could not be allocated to this 'strange' project which did not fit the basic pattern of a DCS or SAFETY upgrade.


He appealed to his friend Jack, who had considerable influence in the organisation.  Jack said he would look into it and get back to him.

He was at his desk, and soon got a phone call, it was Azmy the senior manager. [Jack must have called him up]

- Ahmed, what is going on here.  I have major projects in play here and Jack tells me that you are trying to get a migration done for this ancient substation upgrading equipment we don't even understand ! The answer is NO ! and rang off.

- Jack said that ? So much for Jack then - some friend - ... Ahmed was getting upset now.  Perhaps he should call it off.  It was only a cafe conversation in any case, but he kind of liked that new kid Aron and his enthusiasm.

-Ahmed decided to go all in, and not even bother with Jack any more.  He called up the product manager again.

- Hello - ah - sorry to wake you in the middle of the night, but I have to talk about this upgrade - I have no support here and I need help. This is a major customer, it's our equipment out their whether we installed it or not, whether we even know about it or not.  Who would expect us to know it all in any case. It must have been 20 years ago.  But if the customer wants to upgrade it, we owe him at least some effort in trying to figure it out.

- Ahmed, take it easy.  OK so what about we just upgrade the old equipment instead of changing it out.  That way you can just buy some of the older parts and expand it as a service order.  I know it won't be as big or exciting an order, but perhaps you can do this and get on with life without losing your job or  the project into the bargain.  Let's give it a try.

-Hello Ahmed, this is Aron.  Aron - remember - we met in the cafe just recently.
-Oh Aron - I wanted to talk to you - I guess you've called about the substation job right ?

-Yes, can you do it ?

-Sort of - I mean I could do it, but I have a problem getting some quotation resources.  It's a big job replacing all of the old CPUs and changing them out just to add one or two bays, and it means we have to port all the old user programs, re-test them.. We are just short of engineering staff.  Do you mind if we just expand the existing system for now ?
-I'll have to get back to you - this stuff is really old.  Do you mean you still support it ?.. Aron rang off.

The next morning the product manager called back...
- Sorry Ahmed, those old parts you need, we have most of them but there are two parts we just can't make any more.  Some of the chips are end of life and we don't guarantee we can make those boards any longer.  This is going to mean you have to go back to your original plan to upgrade by replacement.

Ahmed thought it over.  On the one hand he could take the easier option and just replace the product, which would require resources yet on the other hand he could perhaps stand his ground.  Intuition said, why cause more problems when you could just for the cost of very few additional boards and virtually no re-engineering you could upgrade and expand the system and keep the customer happy.

Ahmed knew the product manager was right. The company theme was upgrade, the parts were old, the product was old, and they could not make the key expansion components anymore. He literally had nowhere to go.

Fortunately Aron rang back about an hour or so later...

-Ahmed, lets go for expanding the current system just by adding some more of the older technology.  You go ahead and give me a quotation for that. Aron said.

Ahmed didn't have the heart to tell him he couldn't do it, and in reality why should he not be able to do it ? The right decision was somehow curling around his brain, that something had to give.  The right thing was to just get that old component manufactured, get on with life, let the customer get on with life and then maybe go for an upgrade on a larger project basis later on.


Ahmed called up Jack.  The phone rang over until voicemail.  No answer. He rang again.  Jack answered.


- What is it Ahmed ? Jack asked, rather abruptly.


- Well it is about that upgrade Jack.  I have a plan, but don't ask questions, I need about a dozen old boards manufactured, but they don't want to do it. The factory that is.  The components are old, and they have probably got to get some of the chips on the grey market, whatever that is.  I really need your help.  You know people in the factory can you back me up on this with them at least.  It will go easier and I won't need a mass of engineering resource to quote it.


Jack was quiet for a bit and decided.
- Sure I'll give them a call.

Ahmed knew Jack would never say ok unless he thought he could pull it off. He called up Aron
- Aron we have a plan.  We are working it.  Wait for our quotation.






-




No comments:

Post a Comment

Popular Posts