Loading...
HomeMy WebLinkAboutReso 58-19 Computer Disaster Recovery PlanRESOLUTION NO. 58-19 RESOLUTION OF THE CITY COUNCIL OF THE CITY OF MORRO BAY, CALIFORNIA, ESTABLISHING THE CITY OF MORRO BAYS COMPUTER DISASTER RECOVERY PLAN THE CITY COUNCIL City of Morro Bay, California WHEREAS, the City of Morro Bay desires to establish City Financial policies, so as to direct staff and provide transparency to the Council and Community; and WHEREAS, the City relies heavily on computers and other advance technologies to conduct its operations and provide essential services to its residents; and WHEREAS, Government Finance Officers Association (GFOA) recommends that every government formally establish written policies and procedures for minimizing disruptions resulting from failures in computers or other advanced technologies following a disaster; and WHEREAS, staff recommends the City Council adopt the proposed Computer Disaster Recovery Plan, which demonstrates the City's commitment to emergency response and fiscal responsibility and prudent management. NOW, THEREFORE, BE IT RESOLVED by the City Council of the City of Morro Bay, California: 1. The "Computer Disaster Recovery Plan," as set forth in Exhibit A, attached hereto and incorporated herein, is hereby approved 2. The City Manager in his/her reasonable judgment and as necessary is authorized to modify the approved "Computer Disaster Recovery Plan" for the sole purpose of minor updates concerning staffing and personnel changes, and such modifications shall have the full authority of Council approval unless otherwise disapproved by Council resolution. PASSED AND ADOPTED by the City Council of the City of Morro Bay at a regular meeting thereof held on the 25h day of June 2019, by the following vote: AYES: NOES: ABSENT: Headding, Addis, Davis, McPherson None Heller JQHN HEADDING, Mayor ATTEST: bwANA S ANSON, City Clerk 01181.0001/560858.1 Resolution No. 58-19 Exhibit A INFORMATION TECHNOLOGY DISASTER RECOVERY PLAN `1 CITY OF MORRO BAY Resolution No. 58-19 Exhibit A Table of Contents Introduction............................................................................................................................................ 3 Definition of a Disaster,"",',,,.,,,,, "I "1 3 Purpose.................................................................................................................................... 3 Scope....................................................................................................................................... 3 Version Information &Changes................................................................................................ 4 Disaster Recovery Teams &Responsibilities...................................................................................... 5 Disaster Recovery Lead........................................................................................................... 5 Disaster Management Team..................................................................................................... 6 FacilitiesTeam.......................................................................................................................... 7 NetworkTeam.......................................................................................................................... 8 Senior Management Team......................................................................................................10 CommunicationTeam..............................................................................................................11 FinanceTeam..................................................................................................................a.......12 Other Organization Specific Teams........","", "Mm .......... ........... 13 Disaster Recovery Call Tree................................................................................................................13 RecoveryFacilities..............................................................................................................................15 Description of Recovery Facilities............................................................................................15 Operational Considerations.....................................................................................................19 Dataand Backups...................................................................................................................21 Communicating During a Disaster..... mamomman go mommom as Monsoons ME EMMMME MMMMMKAKMKMMMMMM ME mommommoswoomon monsoons 22 Communicating with the Authorities...............................................................4.........................22 Communicatingwith Employees, W 1 1 6 d 1 9 1 a 1 0 P I I P I I I I I M M I I I I I I I I I I I I I I I I I I I I I I I I I I I I K I I I I I I I I I I I I I I I I I I I I I I I . . . . . . . . . . . . . 6 * , 0 * , 2 3 Communicating with Clients.,,,,,,.,,,,, .......... 0 , , 6 0 0 * 0 0 a a 0 4 1 N 4 0 0 a 0 P 4 F q ...................... 24 Communicating with Vendors..................................................................................................25 Communicating with the Media................................................................................................26 Communicating with <<Other group/stakeholders»...............................................................27 Dealingwith a Disaster........................................................................................................................ 28 Disaster Identification and Declaration.....................................................................................28 2 Resolution No. 58-19 Exhibit A DRPActivation.........................................................................................................................29 Communicating the Disaster....................................................................................................29 Assessment of Current and Prevention of Further Damage.....................................................29 Standby Facility Activation.......................................................................................................30 RestoringIT Functionality........................................................................................................30 Repair & Rebuilding of Primary Facility.......................................................0............................31 Other Organization Specific Steps Required............................................................................31 Restoring IT Functionality........... on off some as MENSWEAR oxmwwwwwommm MANN on NEAR MMMMMMMMMM MMUMME KNEW ME 31 Current System Architecture,,.,,,,,,,", 10# I'll, P032 ITSystems...............................................................................................................................32 PlanTesting & Maintenance................................................................................................................ 38 Maintenance",',,',',,"'',',, ....... ..... 38 Testing.....................................................................................................................................38 ij Resolution No. 58-19 Exhibit A Introduction This Disaster Recovery Plan (DRP) captures, in a single repository, all of the information that describes City of Morro Bay's ability to withstand a disaster as well as the processes that must be followed to achieve disaster recovery. Definition of a Disaster A disaster can be caused by man or nature and results in City of Morro Bay's IT department not being able to perform all or some of their regular roles and responsibilities for a period of time. City of Morro Bay defines disasters as the following: • One or more vital systems are non-functional • The building is not available for an extended period of time, but all systems are functional within it • The building is available, but all systems are non-functional • The building and all systems are non-functional The following events can result in a disaster, requiring this Disaster Recovery document to be activated: • Fire • Flash flood • Pandemic • Power Outage • War • Theft • Terrorist Attack Purpose The purpose of this DRP document is twofold: first to capture all of the information relevant to the City's ability to withstand a disaster, and second to document the steps that the City will follow if a disaster occurs. Note that in the event of a disaster the first priority of City of Morro Bay is to prevent the loss of life. Before any secondary measures are undertaken, City of Morro Bay will ensure that all employees, and any other individuals on the organization's premises, are safe and secure. After all individuals have been brought to safety, the next goal of City of Morro Bay will be to enact the steps outlined in this DRP to bring all of the organization's groups and departments back to business -as -usual as quickly as possible. This includes: • Preventing the loss of the organization's resources such as hardware, data and physical IT assets • Minimizing downtime related to IT • Keeping the business running in the event of a disaster This DRP document will also detail how this document is to be maintained and tested. Scope The City of Morro Bay DRP takes all of the following areas into consideration: • Network Infrastructure • Servers Infrastructure Resolution No. 58-19 Exhibit A • Telephony System • Data Storage and Backup Systems • Data Output Devices • End -user Computers • Organizational Software Systems • Database Systems • IT Documentation This DRP does not take into consideration any non -IT, personnel, Human Resources and real estate related disasters. Version Information &Changes Any changes, edits and updates made to the DRP will be recorded in here. It is the responsibility of the Disaster Recovery Lead to ensure that all existing copies of the DRP are up to date. Whenever there is an update to the DRP, City of Morro Bay requires that the version number be updated to indicate this. Name of Person Making Change Role of Person Making Change Date of Change Version Number Notes Steve Doerr IT Manager 03/11/19 1.0 Initial version of DR Plan Resolution No. 58-19 Exhibit A Disaster Recovery Teams &Responsibilities In the event of a disaster, different groups will be required to assist the IT department in their effort to restore normal functionality to the employees of City Morro Bay. The different groups and their responsibilities are a.c follows: • Disaster Recovery Leads) • Disaster Management Team • Facilities Team • Information Technology Team • Operations Team • ManagementTeam • Communications Team • Finance Team The lists of roles and responsibilities in this section have been created by City of Morro Bay and reflect the likely tasks that team members will have to perform. Disaster Recovery Team members will be responsible for performing all of the tasks below. In some disaster situations, Disaster Recovery Team members will be called upon to perform tasks not described in this section. Disaster Recovery Lead The Disaster Recovery Lead is responsible for making all decisions related to the Disaster Recovery efforts. This person's primary role will be to guide the disaster recovery process and all other individuals involved in the disaster recovery process will report to this person in the event that a disaster occurs at City of Morro Bay, regardless of their department and existing managers. All efforts will be made to ensure that this person be separate from the rest of the disaster management teams to keep his/her decisions unbiased; the Disaster Recovery Lead will not be a member of other Disaster Recovery groups in City of Morro Bay. Role and Responsibilities • Make the determination that a disaster has occurred and trigger the DRP and related processes. • Initiate the DR Call Tree. • Be the single point of contact for and oversee all of the DR Teams. • Organize and chair regular meetings of the DR Team leads throughout the disaster. • Present to the Management Team on the state of the disaster and the decisions that need to be made. • Organize, supervise and manage all DRP test and author all DRP updates. Contact Information Name Role/Title Work Phone Number Home Phone Number Mobile Phone Number 0 Resolution No. 58-19 Exhibit A ®isaster Management Team The Disaster Management Team that will oversee the entire disaster recovery process. They will be the first team that will need to take action in the event of a disaster. This team will evaluate the disaster and will determine what steps need to be taken to get the organization back to business as usual. Role &Responsibilities • Set the DRP into motion after the Disaster Recovery Lead has declared a disaster • Determine the magnitude and class of the disaster • Determine what systems and processes have been affected by the disaster • Communicate the disaster to the other disaster recovery teams • Determine what first steps need to be taken by the disaster recovery teams • Keep the disaster recovery teams on track with pre -determined expectations and goals • Keep a record of money spent during the disaster recovery process • Ensure that all decisions made abide by the DRP and policies set by City of Morro Bay • Get the secondary site ready to restore business operations • Ensure that the secondary site is fully functional and secure • Create a detailed report of all the steps undertaken in the disaster recovery process • Notify the relevant parties once the disaster is over and normal business functionality has been restored • After City of Morro Bay is back to business as usual, this team will be required to summarize any and all costs and will provide a report to the Disaster Recovery Lead summarizing their activities during the disaster Contact Information Name Role/Title Work Phone Number Home Phone Number Mobile Phone Number 7 Resolution No. 58-19 Exhibit A Facilities Team The Facilities Team will be responsible for all issues related to the physical facilities that house IT systems. They are the team that will be responsible for ensuring that the standby facilities (training room at the fire station) are maintained appropriately and for assessing the damage too and overseeing the repairs to the primary location in the event of the primary location's destruction or damage. Role � Responsibilities • Ensure that the standby facility is maintained in working order • Ensure that transportation is provided for all employees working out of the standby facility • Ensure that hotels or other sleeping are arranged for all employees working out of the standby facility • Ensure that sufficient food, drink, and other supplies are provided for all employees working out of the standby facility • Assess, or participate in the assessment of, any physical damage to the primary facility • Ensure that measures are taken to prevent further damage to the primary facility • Work with insurance company in the event of damage, destruction or losses to any assets owned by City of Morro Bay • Ensure that appropriate resources are provisioned to rebuild or repair the main facilities in the event that they are destroyed or damaged • After City of Morro Bay is back to business as usual, this team will be required to summarize any and all costs and will provide a report to the Disaster Recovery Lead summarizing their activities during the disaster Contact Information Name Role/Title Work Phone Number Home Phone Number Mobile Phone Number Rob Livick Director of Public Works Resolution No. 58-19 Exhibit A Information Technology Team The Network Team will be responsible for assessing damage specific to any network infrastructure and for provisioning data and voice network connectivity including WAN, LAN, and any telephony connections internally within the enterprise as well as telephony and data connections with the outside world. They will also be responsible for providing the physical server infrastructure required for the enterprise to run its IT operations and applications in the event of and during a disaster. Also responsible for ensuring that all enterprise applications operates as required to meet business objectives in the event of and during a disaster. This team will be responsible for providing employees with the tools they need to perform their roles as quickly and efficiently as possible. They will need to provision all City of Morro Bay employees in the standby facility and those working from home with the tools that their specific role requires. The primary responsibilities of this team are to provide baseline network functionality, provide baseline server functionality and to ensure and validate appropriate application performance. Role &Responsibilities • In the event of a disaster that does not require migration to standby facilities, the team will determine which network services are not functioning at the primary facility • If multiple network services are impacted, the team will prioritize the recovery of services in the manner and order that has the least business impact. • If network services are provided by third parties, the team will communicate and co-ordinate with these third parties to ensure recovery of connectivity. • In the event of a disaster that does require migration to standby facilities the team will ensure that all network services are brought online at the secondary facility • Once critical systems have been provided with connectivity, employees will be provided with connectivity in the following order: o All members of the DR Teams o All C-level and Executive Staff c All IT employees o All remaining employees • Install and implement any tools, hardware, software and systems required in the standby facility • Install and implement any tools, hardware, software and systems required in the primary facility • In the event of a disaster that does not require migration to standby facilities, the team will determine which servers are not functioning at the primary facility • If multiple servers are impacted, the team will prioritize the recovery of servers in the manner and order that has the least business impact. Recovery will include the following tasks: o Assess the damage to any servers o Restart and refresh servers if necessary • Ensure that secondary servers located in standby facilities are kept up-to-date with system patches • Ensure that secondary servers located in standby facilities are kept up-to-date with application patches • Ensure that secondary servers located in standby facilities are kept up-to-date with data copies • Ensure that the secondary servers located in the standby facility are backed up appropriately • Ensure that all of the servers in the standby facility abide by the City's server policy • Install and implement any tools, hardware, and systems required in the standby facility • Install and implement any tools, hardware, and systems required in the primary facility • In the event of a disaster that does not require migration to standby facilities, the team will determine which applications are not functioning at the primary facility • If multiple applications are impacted, the team will prioritize the recovery of applications in the manner and order that has the least business impact. Recovery will include the following tasks: o Assess the impact to application processes o Restart applications as required 7 Resolution No. 58-19 Exhibit A o Patch, recode or rewrite applications as required ® Ensure that secondary servers located in standby facilities are kept up-to-date with application patches ® Ensure that secondary servers located in standby facilities are kept up-to-date with data copies ® Install and implement any tools, software and patches required in the standby facility ® Install and implement any tools, software and patches required in the primary facility ® Maintain lists of all essential supplies that will be required in the event of a disaster ® Ensure that these supplies are provisioned appropriately in the event of a disaster ® Ensure sufficient spare computers and laptops are on hand so that work is not significantly disrupted in a disaster ® Ensure that spare computers and laptops have the required software and patches ® Ensure sufficient computer and laptop related supplies such as cables, wireless cards, laptop locks, mice, printers and docking stations are on hand so that work is not significantly disrupted in a disaster ® Ensure that all employees that require access to a computer/laptop and other related supplies are provisioned in an appropriate timeframe ® If insufficient computers/laptops or related supplies are not available the team will prioritize distribution in the manner and order that has the least business impact ® This team will be required to maintain a log of where all of the supplies and equipment were used ® After City of Morro Bay is back to business as usual, this team will be required to summarize any and all costs and will provide a report to the Disaster Recovery Lead summarizing their activities during the disaster Contact Information Name Role/Title Work Phone Number Home Phone Number Mobile Phone Number Steve Doerr IT Manager 805-772-6290 805-801-2948 Resolution No. 58-19 Exhibit A Senior Management Team The Senior Management Team will make any business decisions that are out of scope for the Disaster Recovery Lead. Decisions such as constructing a new data center, relocating the primary site etc. should be made by the Senior Management Team. The Disaster Recovery Lead will ultimately report to this team. Role � Responsibilities ® Ensure that the Disaster Recovery Team Lead is help accountable for his/her role ® Assist the Disaster Recovery Team Lead in his/her role as required ® Make decisions that will impact the company. This can include decisions concerning: o Rebuilding of the primary facilities o Rebuilding of data centers o Significant hardware and software investments and upgrades o Other financial and business decisions Contact Information Name Role/Title Work Phone Number Home Phone Number Mobile Phone Number Scott Collins City Manager 11 Resolution No. 58-19 Exhibit A Communication Team This will be the team responsible for all communication during a disaster. Specifically, they will communicate with City of Morro Bay's employees, clients, vendors and suppliers, banks, and even the media if required. Role &Responsibilities • Communicate the occurrence of a disaster and the impact of that disaster to all City of Morro Bays employees • Communicate the occurrence of a disaster and the impact of that disaster to authorities, as required • Communicate the occurrence of a disaster and the impact of that disaster to all City of Morro Bay's partners • Communicate the occurrence of a disaster and the impact of that disaster to all City of Morro Bays clients • Communicate the occurrence of a disaster and the impact of that disaster to all City of Morro Bays vendors • Communicate the occurrence of a disaster and the impact of that disaster to media contacts, as required • After City of Morro Bay is back to business as usual, this team will be required to summarize any and all costs and will provide a report to the Disaster Recovery Lead summarizing their activities during the disaster Contact Information Name Role/Title Work Phone Number Home Phone Number Mobile Phone Number 12 Resolution No. 58-19 Exhibit A Finance Team This team will be responsible for ensuring that all of City of Morro Bay's finances are dealt with in an appropriate and timely manner in the event of a disaster. The finance team will ensure that there is money available for necessary expenses that may result from a disaster as well as expenses from normal day-to-day business functions. Role &Responsibilities ® Ensure there is sufficient cash on -hand or accessible to deal with small-scale expenses caused by the disaster. These can include paying for accommodations and food for DR team members, incremental bills, etc. ® Ensure there is sufficient credit available or accessible to deal with large-scale expenses caused by the disaster. These can include paying for new equipment, repairs for primary facilities, etc. ® Review and approve Disaster Teams' finances and spending ® Ensure that payroll occurs and that employees are paid as normal, where possible ® Communicate with creditor to arrange suspension of extensions to scheduled payments, as required ® Communicate with banking partners to obtain any materials such as checks, bank books etc. that may need to be replaced as a result of the disaster Contact Information Name Role/Title Work Phone Number Home Phone Number Mobile Phone Number 13 Resolution No. 58-19 Exhibit A Disaster Recovery Call Tree In a disaster recovery or business continuity emergency, time is of the essence so City of Morro Bay will make use of a Call Tree to ensure that appropriate individuals are contacted in a timely manner. ® The Disaster Recovery Team Lead calls all Level 1 Members (Blue cells) ® Level 1 members call all Level 2 team members over whom they are responsible (Green cells) ® Level 1 members call all Level 3 team members over whom they are directly responsible (Beige cells) ® Level 2 Members call all Level 3 team members over whom they are responsible (Beige cells) ® In the event a team member is unavailable, the initial caller assumes responsibility for subsequent calls (i.e. if a Level 2 team member is inaccessible, the Level 1 team member directly contacts Level 3 team members). Contact Office Mobile Home DR Lead DR Management Team Lead DR Management Team 1 DR Management Team 2 Facilities Team Lead Facilities Team 1 Information Technology Team Lead Information Techonlogy Team Lead 1 Management Team Lead Management Team 1 Communications Team Lead 14 Resolution No. 58A 9 Exhibit A Communications Team 1 Finance Team Lead Finance Team 1 Disaster Recovery Call Tree Process Flow C 0 DR Lead Initiates Call Tree, Team Lead Contacts Team NO DR Lead Contacts Leads Sub Team Leads Yes a Sub (� Team Lead DR Lead or Team J Contacts Sub Team Lead NO Lead Contacts Team Lead Team Members Avail ble? (6 � Cc Sub Team Lead .Q J Contacts Team n Members Cn E a.) a) ETeam Members N LIrespond u DR > call -out 15 Resolution No. 58-19 Exhibit A Recovery Facilities In order to ensure that City of Morro Bay is able to withstand a significant outage caused by a disaster, it has provisioned separate dedicated standby facilities. This section of this document describes those facilities and includes operational information should those facilities have to be used. Description of Recovery Facilities The Disaster Command and Control Center or Standby facility will be used after the Disaster Recovery Lead has declared that a disaster has occurred. This location is a separate location to the primary facility. The current facility, located at 715 Harbor St is .50 miles away from the primary facility. The standby facility will be used by the IT department and the Disaster Recovery teams; it will function as a central location where all decisions during the disaster will be made. It will also function as a communications hub for City of Morro Bay. The standby facility must always have the following resources available: • Copies of this DRP document • Fully redundant server room • Sufficient servers and storage infrastructure to support enterprise business operations • Office space for DR teams and IT to use in the event of a disaster • External data and voice connectivity • Sleeping quarters for employees that may need to work multiple shifts • Kitchen facilities (including food, kitchen supplies and appliances) • Bathroom facilities (Including toilets, showers, sinks and appropriate supplies) • Parking spaces for employee vehicles 16 Resolution No. 58-19 Exhibit A Operational Considerations If employees are required to stay at the Standby Facility for extended periods of time and require hotel accommodations, they will be provided by the City of Morro Bay. The Facilities Team will be responsible for determining which employees require hotel accommodations and ensuring sufficient rooms are made available. If employees are required to stay at the Standby Facility for extended periods of time and require food, it will be provided by the City of Morro Bay. The Facilities Team will be responsible for determining which employees require food and ensuring sufficient food is made available via groceries, restaurants or caterers as appropriate. While in the Standby Facility, employees must work under appropriate, sanitary and safe conditions. The Facilities team will be responsible for ensuring that this facility is kept in proper working order. Include'. only those operations considerations'providers that are appropriate' given the facilities of the Standby Facility. Accommodations Hotel 1 Address Phone Number «Map of Hotel 1's Location» «Directions to get to !-lotel 1 from the standby facility» Hotel 2 Address Phone Number <<Map of Hotel 1's Location>> <<Directions to get to Hotel 2 from the standby facility>> Food, Beverages and Other Supplies Restaurant/Grocery 1 Address Phone Number <<Map of Restaurant/Grocery 1's Location>> <<Directions to get to Restaurant/Grocery 1 from the standby facility>> 17 Resolution No. 58-19 Exhibit A RestauralI mocery2 Address Resolution No. 58-19 Exhibit A Phone Number «/lap of Restaurant/Graaery 2's Location» <Directions to get to Restaurant/Grocery 2 from the standby facility>> Restaurant/Grocery 3 Address Phone Number <<Map of Restaurant/Grocery 3's Location>> «Lirections to get to Restaurant/Grocery 3 fr omi the standby facility>> Catering Caterer 1 Address Phone Number «Map of Caterer 1's Location>> «pirections to get to Caterer 1 from the standby facility» Caterer 2 Address Phone Number <<Map of Caterer 2's Location>> <<directions to get to Caterer 2 from the standby facility>> Standby Facility Maintenance Maintenance Company Address Phone Number 19 Resolution No. 58-19 Exhibit A Data and Backups This section explains where all of the organization's data resides as well as where it is backed up to. Use this information to locate and restore data in the event of a disaster. Data in Order of Criticality Rank Data Data Type Back-up Backup Location(s) Frequency 1 Documents and Files Confidential, Nightly Data is backed up to Synology sensitive, incremental Backup Appliance using Veeam personnel backup with Enterprise Software. The data is saved information, City revisions. Daily to the Fire Dept Server Room. business replication off site. 2 Financial Payroll, budgets, Nightly full Data is backed up to Synology city finances backup with Backup Appliance using Veeam revisions. Daily Enterprise Software. The data is saved replication offsite, to the Fire Dept Server Room. 3 Email Confidential, Nightly full Data is backed up to Synology sensitive, backup with Backup Appliance using Veeam personnel revisions. Daily Enterprise Software. The data is saved information, City replication offsite. to the Fire Dept Server Room. business 20 Resolution No. 58-19 Exhibit A Communicating ®wring a ®isaster In the event of a disaster City of Morro Bay will need to communicate with various parties to inform them of the effects on the business, surrounding areas and timelines. The Communications Team will be responsible for contacting all of City of Morro Bay's stakeholders. Communicating with the Authorities The Communications Team's first priority will be to ensure that the appropriate authorities have been notified of the disaster, providing the following information: • The location of the disaster • The nature of the disaster • The magnitude of the disaster • The impact of the disaster • Assistance required in overcoming the disaster • Anticipated timelines Authorities Contacts Authorities Point of Contact Phone Number E-mail Police Department Fire Department 21 Resolution No. 58-19 Exhibit A Communicating with Employees The Communications Team's second priority will be to ensure that the entire company has been notified of the disaster. The best and/or most practical means of contacting all of the employees will be used with preference on the following methods (in order): • E-mail (via corporate e-mail where that system still functions) • E-mail (via non -corporate or personal e-mail) • Telephone to employee home phone number • Telephone to employee mobile phone number The employees will need to be informed of the following: • Whether it is safe for them to come into the office • Where they should go if they cannot come into the office • Which services are still available to them • Work expectations of them during the disaster Employee Contacts Name Role/Title Home Phone Mobile Phone Personal E-mail Number Number Address 22 Resolution No. 58-19 Exhibit A Comrnunicating with Clients After all of the organization's employees have been informed of the disaster, the Communications Team will be responsible for informing clients of the disaster and the impact that it will have on the following: „ • Anticipated impact on service offerings • Anticipated impact on delivery schedules • Anticipated impact on security of client information • Anticipated timelines Crucial clients will be made aware of the disaster situation first. Crucial clients will be E-mailed first then called after to ensure that the message has been delivered. All other clients will be contacted only after all crucial clients have been contacted. Crucial Clients Company Name Point of Contact Phone Number E-mail Secondary Clients Company Name Point of Contact Phone Number E-mail 23 Resolution No. 58-19 Exhibit A Communicating with Vendors After all of the organization's employees have been informed of the disaster, the Communications Team will be responsible for informing vendors of the disaster and the impact that it will have on the following: ® Adjustments to service requirements ® Adjustments to delivery locations ® Adjustments to contact information ® Anticipated timelines Crucial vendors will be made aware of the disaster situation first. Crucial vendors will be t"-mailed first ti'�en called after to ensure that the message has been delivered, All other vendors will be contacted only after all crucial vendors have been contacted. Vendors encompass those organizations that provide everyday services to the enterprise, but also the hardware and software companies that supply the IT department. The Communications Team will act as a go-between between the DR Team leads and vendor contacts should additional IT infrastructure be required. Crucial Vendors Company Name Point of Contact Phone Number E-mail Secondary Vendors Company Name Point of Contact Phone Number E-mail 24 Resolution No. 58-19 Exhibit A Communicating with the Media After all of the organization's employees have been informed of the disaster, the Communications Team will be responsible for informing media outlets of the disaster, providing the following information: ® An official statement regarding the disaster ® The magnitude of the disaster ® The impact of the disaster ® Anticipated timelines Media Contacts Company Name Point of Contact Phone Number E-mail 25 Resolution No. 58-19 Exhibit A Communicating with Other group/stakeholders City Council Members Other Contacts Company Name Point of Contact Phone Number E-mail Resolution No. 58-19 Exhibit A Dealing with a Disaster If a disaster occurs in City of Morro Bay, the first priority is to ensure that all employees are safe and accounted for. After this, steps must be taken to mitigate any further damage to the facility and to reduce the impact of the disaster to the organization. Regardless of the category that the disaster falls into, dealing with a disaster can be broken down into the following steps: 1) Disaster identification and declaration 2) DRP activation 3) Communicating the disaster 4) Assessment of current and and prevention of further damage 5) Standby facility activation 6) Establish IT operations 7) Repair and rebuilding of primary facility Disaster Identification and Declaration Since it is almost impossible to predict when and how a disaster might occur, The City of Morro Bay must be prepared to find out about disasters from a variety of possible avenues. These can include: • First hand observation • System Alarms and Network Monitors • Environmental and Security Alarms in the Primary Facility • Security staff • Facilities staff • End users • 3rd Party Vendors • Media reports Once the Disaster Recovery Lead has determined that a disaster had occurred, s/he must officially declare that the company is in an official state of disaster. It is during this phase that the Disaster Recovery Lead must ensure that anyone that was in the primary facility at the time of the disaster has been accounted for and evacuated to safety according to the company's Evacuation Policy. While employees are being brought to safety, the Disaster Recovery Lead will instruct the Communications Team to begin contacting the Authorities and all employees not at the impacted facility that a disaster has occurred. 27 Resolution No. 58-19 Exhibit A DRP Activation Once the Disaster Recovery Lead has formally declared that a disaster has occurred s/he will initiate the activation of the DRP by triggering the Disaster Recovery Call Tree. The following information will be provided in the calls that the Disaster Recovery Lead makes and should be passed during subsequent calls: • That a disaster has occurred • The nature of the disaster (if known) • The initial estimation of the magnitude of the disaster (if known) • The initial estimation of the impact of the disaster (if known) • The initial estimation of the expected duration of the disaster (if known) • Actions that have been taken to this point • Actions that are to be taken prior to the meeting of Disaster Recovery Team Leads • Scheduled meeting place for the meeting of Disaster Recovery Team Leads • Scheduled meeting time for the meeting of Disaster Recovery Team Leads • Any other pertinent information If the Disaster Recovery Lead is unavailable to trigger the Disaster Recovery Call Tree, that responsibility shall fall to the Disaster Management Team Lead Communicating the Disaster Refer to the "Communicating During a Disaster" section of this document. Assessment of Current and Prevention of Further Damage Before any employees from City of Morro Bay can enter the primary facility after a disaster, appropriate authorities must first ensure that the premises are safe to enter. The first team that will be allowed to examine the primary facilities once it has been deemed safe to do so will be the Facilities Team. Once the Facilities Team has completed an examination of the building and submitted its report to the Disaster Recovery Lead, the Disaster Management, Networks, Servers, and Operations Teams will be allowed to examine the building. All teams will be required to create an initial report on the damage and provide this to the Disaster Recovery Lead within 4 hours of the initial disaster. During each team's review of their relevant areas, they must assess any areas where further damage can be prevented and take the necessary means to protect City of Morro Bay's assets. Any necessary repairs or preventative measures must be taken to protect the facilities; these costs must first be approved by the Disaster Recovery Team Lead. m Resolution No, 58-19 Exhibit A Standby Facility Activation The Standby Facility will be formally activated when the Disaster Recovery Lead determines that the nature of the disaster is such that the primary facility is no longer sufficiently functional or operational to sustain normal business operations. Once this determination has been made, the Facilities Team will be commissioned to bring the Standby Facility to functional status after which the Disaster Recovery Lead will convene a meeting of the various Disaster Recovery Team Leads at the Standby Facility to assess next steps. These next steps will include: 1. Determination of impacted systems 2. Criticality ranking of impacted systems 3. Recovery measures required for high criticality systems 4. Assignment of responsibilities for high criticality systems 5. Schedule for recovery of high criticality systems 6. Recovery measures required for medium criticality systems 7. Assignment of responsibilities for medium criticality systems 8. Schedule for recovery of medium criticality systems 9. Recovery measures required for low criticality systems 10. Assignment of responsibilities for recovery of low criticality systems 11. Schedule for recovery of low criticality systems 12. Determination of facilities tasks outstanding/required at Standby Facility 13. Determination of operations tasks outstanding/required at Standby Facility 14. Determination of communications tasks outstanding/required at Standby Facility 15. Determination of facilities tasks outstanding/required at Primary Facility 16. Determination of other tasks outstanding/required at Primary Facility 17. Determination of further actions to be taken During Standby Facility activation, the Facilities, Networks, Servers, Applications, and Operations teams will need to ensure that their responsibilities, as described in the "Disaster Recovery Teams and Responsibilities" section of this document are carried out quickly and efficiently so as not to negatively impact the other teams. Restoring IT Functionality Refer to the "Restoring IT Functionality" section of this document. Resolution No, 58-19 Exhibit A Repair & Rebuilding of Primary Facility Before the enterprise can return operations to Primary Facilities, those facilities must be returned to an operable condition. The tasks required to achieve that will be variable depending on the magnitude and severity of the damage. Specific tasks will be determined and assigned only after the damage to Primary Facilities has been assessed. ®ther ®rganization Specific Steps Required Restoring IT Functionality Should a disaster actually occur and City of Morro Bay needs to exercise this plan, this section will be referred to frequently as it will contain all of the information that describes the manner in which City of Morro Bay's information system will be recovered. 30 Current System Architecture x�a �y s 5 f p m m N Uxv 8 tpi oo$� Resolution No. 58-19 ExhibitA 31 pN NNO Ali O � IJNjVON � U ? 0 0 CD goo cacsoaN�N�yN�nA�$ ��WNNN,N•N�_N� tj O N N O N O to C N (O� O r Oree N N N K Nt� N tttt�� w ro N NNN N CNi+CUNT� NN m n- N N N N N= nj N NNtr +n Vs tT N qq tT tT Vt(71�N�c N N to IV aoN CNitYn,c"s�gv`� o a a® c o o a�� 000Q4��a-'_ NP�ACD oC>CD ..�® otppC)otpptatppsotpp}N}'' Cn 0 Resolution No. 58-19 Exhibit A N O V 32 IL Resolution No. 58-19 Exhibit A Rank IT System System Components (In order of importance) 1 Blade Servers — Virtual Servers 2 Data Storage Area Network 3 Backup Server 4 Internet Access 5 Firewalls 6 Switches & Routers 33 Resolution No. 58-19 Exhibit A Criticality Rank -One System In this sectionn you will be required to rank each system's components in order of criticality, supplying the information that each system will require to bring it back online. First, vendor and model information, serial numbers and other component specific information will be gathered. Next, you will be required to attach each component's runbooks or Standard Operating Procedure (SOP) documents. Each component must have a runbook or SOP document associated with it. If you do not have these documents for all components, please refer to the following Info -Tech Research Group notes for more information: • SOP Research: o SOP 101: Standard Operating Procedures o How to Write an SOP o How to Implement SOPs o Step -by -Step SOP Template o Hierarchical SOP Template o Flowchart SOP Template • Runbooks Research: o Don't Run without Runbooks o Free IT Staff Time: Implement Runbook Automation o How to Start Building Runbooks EXAMPLE: System Name «State the name of the IT System here» Component Name <<State the name of the specific IT Component here>> Vendor Name <<State the name of the IT Component's vendor here>> Model Number <<State the name of the IT Component's model number here>> Serial Number <<State the name of the IT Component's serial number here>> Recovery Time Objective <<State the IT Component's Recovery Time Objective here>> Recovery Point Objective <<State the IT Component's Recovery Point Objective here>> Title: Standard Operating Procedures for «Component Name» Document No.: <<Number of the SOP document>> 34 Resolution No. 58-19 Exhibit A Effective Dat <<The date Security Level: << Public, Restricted, or Departmental (the specific from which the SOP is to be department is named).>> nd followed>> implemented z SOP Author/Owner: SOP Approver: Review Date: «The date- on which the SOP must be submitted for review and revision>> a) Purpose This SOP outlines the steps required to restore operations of <<IT System Name». b) Scope This SOP applies to the following components of <<IT System Name>>: • Edit this list to include all included components of the system in question: • Web server • Web server software • Application server Application server storage system ffi Application server software Application serverbackup Database server ffi Database server storage systern Database serversoftware Database serverbackup Client hardware ffi Client software c) Responsibilities The following individuals are responsible for this SOP and for all aspects of the system to which this SOP pertains: • SOP Process: Network Connectivity: • ServerHardware: • Server Software: • Client Connectivity: • Client Hardware: • Client Software: «SOP Owner» «Appropriate Network Administrator» Appropriate Systems Administrator» «Appropriate Application Administrator» «Appropriate Network Administrator» «Appropriate Helpdesk Administrator>> «Appropriate Helpdesk Administrator>> For details of the actual tasks as with these responsibilities, refer to section h) of this SOP. 35 Resolution No. 58-19 Exhibit A d) Definitions This section defines acronyms and words not in common use: Document No.: Number of the SOP document as defined by [insert numbering scheme] Effective Date: The date from which the SOP is to be implemented and followed Review Date: The date on which the SOP must be submitted for review and revision Security Level: Levels of security are categorized as Public, Restricted, or Departmental SOP: Standard Operating Procedure e) Changes Since Last Revision • Add to this list as required. << Nature of change, date of change, individual making the change, individual authorizing the change>> f) Documents/Resources Needed for this SOP The following documents are required for this SOP: • 'Add to this list as required Document g) Related Documents The following documents are related to this SOP and may be useful in the event of an emergency. Their documents below are hyperlinked to their original locations and copies are also attached in the appendix of this document: • 'Add to #his list as required. • Document 3� Resolution No. 58-19 Exhibit A h) Procedure The following are the steps associated with bringing <<Component Name>> back online in the event of a disaster or system failure. Step Action Responsibility 1 «Step 1 Action>> <<erson(cgroup responsible>> 2 3 4 5 6 7 8 37 Resolution No. 58-19 Exhibit A Criticality Rank - Two System Repeat as above for as many systems as the enterprise makes use of. Resolution No. 58-19 Exhibit A Plan Testing &Maintenance While efforts will be made initially to construct this DRP is as complete and accurate a manner as possible, it is essentially impossible to address all possible problems at any one time. Additionally, over time the Disaster Recovery needs of the enterprise will change. As a result of these two factors this plan will need to be tested on a periodic basis to discover errors and omissions and will need to be maintained to address them. The DRP will be updated «indicate frequency» or any time a major system update or upgrade is performed, whichever is more often. The Disaster Recovery Lead will be responsible for updating the entire document, and so is permitted to request information and updates from other employees and departments within the organization in order to complete this task. Maintenance of the plan will include (but is not limited to) the following: 1. Ensuring that call trees are up to date 2. Ensuring that all team lists are up to date 3. Reviewing the plan to ensure that all of the instructions are still relevant to the organization 4. Making any major changes and revisions in the plan to reflect organizational shifts, changes and goals 5. Ensuring that the plan meets any requirements specified in new laws 6. Other organizational specific maintenance goals During the Maintenance periods, any changes to the Disaster Recovery Teams must be accounted for. If any member of a Disaster Recovery Team no longer works with the company, it is the responsibility of the Disaster Recovery Lead to appoint a new team member. Testing City of Morro may is committed to ensuring that this DRP is functional. The DRP should be tested every <<indicate frequency>> in order to ensure that it is still effective. Testing the plan will be carried out as follows: Resolution No, 58-19 Exhibit A 1) Walkthroughs-Team members verbally go through the specific steps as documented in the plan to confirm effectiveness, identify gaps, bottlenecks or other weaknesses. This test provides the opportunity to review a plan with a larger subset of people, allowing the DRP project manager to draw upon a correspondingly increased pool of knowledge and experiences. Staff should be familiar with procedures, equipment, and offsite facilities (if required). 2) Simulations- A disaster is simulated so normal operations will not be interrupted. Hardware, software, personnel, communications, procedures, supplies and forms, documentation, transportation, utilities, and alternate site processing should be thoroughly tested in a simulation test. However, validated checklists can provide a reasonable level of assurance for many of these scenarios. Analyze the output of the previous tests carefully before the proposed simulation to ensure the lessons learned during the previous phases of the cycle have been applied. 3) Parallel Testing- A parallel test can be performed in conjunction with the checklist test or simulation test. Under this scenario, historical transactions, such as the prior business day's transactions are processed against preceding day's backup files at the contingency processing site or hot site. All reports produced at the alternate site for the current business date should agree with those reports produced at the alternate processing site. 4) Full -Interruption Testing- Afull-interruption test activates the total DRP. The testis likely to be costly and could disrupt normal operations, and therefore should be approached with caution. The importance of due diligence with respect to previous DRP phases cannot be overstated. Any gaps in the DRP that are discovered during the testing phase will be addressed by the Disaster Recovery Lead as well as any resources that he/she will require. Call Tree Testing Call Trees are a major part of the DRP and the City of Morro Bay requires that it is tested every <<Enter time frame here>> in order to ensure that it is functional. Tests will be performed as follows: 1) Disaster Recovery Lead initiates call tree and gives the first round of employees called a code word. 2) The code word is passed from one caller to the next. 3) The next workday all Disaster Recovery Team members are asked for the code word. 4) Any issues with the call tree, contact information etc. will then be addressed accordingly.