Office Document Strategies Blog

The title of this blog is actually a quote from a client who brought his shredding and his old shredder into the Leppert office one day for shredding. His point was that his shredder was getting pretty long in the tooth and it didn't warrant repair so he thought he would just get it recycled along with his current need for shredding.  While this is kind of funny as shown in the attached picture it actually reflects a significant trend in office operations that has been underway for many years.

On March 9th Google announced its intention to build a Canadian Region Cloud Centre based in Montreal.  This is a significant announcement for Canadian customers and it means that all of the big three hosting services will have Canadian resident facilities to support cloud access from Canada.  AWS and Microsoft already offer this option for businesses who wish to connect their hosting and storage to a Canadian location. 

Anyone who spends any time watching news reports about technology issues is aware that there was a significant outage of Amazon Web Services (AWS) last week.  There were many major organizations who experienced outages as a result of the failure and it was a little time before news got out to tell us what happened. 

Amazon has been forthright in saying that it was an error in their system combined with some long time format issues which caused the problem.

"Removing a significant portion of the capacity caused each of these systems to require a full restart. While these subsystems were being restarted, S3 was unable to service requests. Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable."  

What started out as a fairly routine maintenance activity escalated into a systems delay which had a major impact on a large number of organizations.

How a single typo brought the internet to its knees

Fat finger: Typo caused Amazon's bigcloud-computing outage

Just a couple of the hundreds (maybe thousands) of articles which have been written to deal with the outage.

What Might We Learn From This Situation?

In spite of the overall reliability that such large IT systems offer there are a few things that any IT operator might learn from the situation that Amazon experienced and that led to some uncomfortable hours for many of their customers.

• No matter how rigorous your design and procedures are unforeseen actions can and will happen when you are dealing with IT systems.  Understanding how to deal with them by thinking ahead about what is most vulnerable in your operations is critical planning.
• Even large well resourced organizations can be caught out by not continually understanding how their systems have grown over time.  Amazon says in their explanation that one of the things which contributed to the problem was the growth of their systems built upon older components that had not been restarted for a long time.  When the restart was needed to deal with an unplanned occurrence it took much longer than expected. How vulnerable are your systems to this kind of delayed response?
• Sometimes it is hard to find out what is happening when your systems go down.  For many of the organizations and the world of users there was a break down of information about what was happening and where the problem was.  Isitdownrightnow.com one of the sites which is commonly used to identify individual site issues was itself affected.  It was through Twitter that many people learned about the outage and that the problem was an external one.  What information tools do you have to finding out if problems are localized or widespread?  Have you articulated a well planned checklist so resources are not wasted if the fault is not internal? 
  • Personally I was trying to use an online News aggregator which I check daily and got an "No Content" message from their page.  Unfortunately for them I thought the problem was caused by a push they had on to encourage users to upgrade to a new version of their service they were promoting.  I totally misread the situation and for a few hours was really frustrated by their service and support.  Wrong response from me as they were impacted by something that was beyond their control.  This should be a caution for us all as we can easily come to a wrong conclusion about what is happening in an IT situation when we don't have the facts.
• Know what systems, services and organizational needs are affected by each and all of your IT structures.  Is your VOIP affected by something you are not aware of due to its need for Internet connection?  Could you financial systems be affected if your bank connection was not available for an extended period?  How many people would be idled if you lost a 'cloud' connection to key software?  How much would the downtime cost/ What customer related activities might suffer?

Having answers to these types of questions is an important part of any business plan.  In our highly digitized world it may be necessary to maintain some analog systems which could keep you functioning if the digital infrastructure has to be shut down.

I know of one local company that actually maintains a paper based 'emergency' system for all of their order desks so that customers would not be totally inconvenienced if the electronic systems they normally rely on got shut off.  They have a formalized recovery procedure on how to capture the data from the paper system into the electronic one when it is again back on line, without creating duplication, or inconveniencing their customers.  Can your organization says they are this prepared for a failure?

Luckily, IT systems are very robust normally, and system administrators take the need for security, backup, redundancy and planning seriously.  This means that for most organizations they can operate each day without a threat of disaster hanging over their heads.

However, as the recent AWS, S3 issue has shown even the top professionals in the space can be caught out and even with a pretty quick recovery there can be a large amount of inconvenience, confusion and disruption.  

Lee K

Each new year brings with it an opportunity to sit back and think about what you are doing and what you might like to do differently in the coming months or years.  This applies to all aspects of a business but never more than to the technology needs which are integral to all businesses.

Canadian cloud services users have always had to be cautious in their choice of providers if they want to make sure that their data remains in Canada and subject to Canadian legislative jurisdiction.  

One of the great benefits of technology available to businesses today is being able to automate what was commonly a routine process which in a manual paper based world has taken extra time.  

Of course Microsoft has been selling cloud services and cloud based software in Canada for several years now.  Office 365 has been one of their hard pushes and is a cloud based app which is targeted at all sizes of business.

Every business leader has heard about cloud solutions for their computing needs.  What many don't know is when and where the cloud could be the best option or even a viable option for them.  Many IT managers take a cautious approach to the cloud because it includes many pitfalls that are not readily apparent.  

I read a really interesting post today by of all things the US Department of Homeland Security.  Now I have to admit that I am not a regular reader of their material but this one has some really good information that can affect millions of users of Windows Server 2003.