Posted  by  admin

Download CompanyMOVE ShowPlanner For Mac 1.7.1

The Practice of System and Network Administration Second Edition
This page intentionally left blank
The Practice of System and Network Administration Second Edition Thomas A. Limoncelli Christina J. Hogan Strata R. Chalup Upper Saddle River, NJ • Boston • Indianapolis • San Francisco New York • Toronto • Montreal • London • Munich • Paris • Madrid Capetown • Sydney • Tokyo • Singapore • Mexico City
Many of the designations used by manufacturers and sellers to distinguish their products areclaimed as trademarks. Where those designations appear in this book, and the publisher wasaware of a trademark claim, the designations have been printed with initial capital letters orin all capitals.The authors and publisher have taken care in the preparation of this book, but make no ex-pressed or implied warranty of any kind and assume no responsibility for errors or omissions.No liability is assumed for incidental or consequential damages in connection with or arisingout of the use of the information or programs contained herein.The publisher offers excellent discounts on this book when ordered in quantity for bulk pur-chases or special sales, which may include electronic versions and/or custom covers and contentparticular to your business, training goals, marketing focus, and branding interests. For moreinformation, please contact: U.S. Corporate and Government Sales, (800) 382-3419, [email protected] sales outside the United States please contact: International Sales, [email protected] us on the Web: www.awprofessional.comLibrary of Congress Cataloging-in-Publication DataLimoncelli, Tom. The practice of system and network administration / Thomas A. Limoncelli, Christina J.Hogan, Strata R. Chalup.—2nd ed. p. cm. Includes bibliographical references and index. ISBN-13: 978-0-321-49266-1 (pbk. : alk. paper) 1. Computer networks—Management. 2. Computer systems. I. Hogan, Christine. II. Chalup, Strata R. III. Title. TK5105.5.L53 2007 004.6068–dc22 2007014507Copyright c 2007 Christine Hogan, Thomas A. Limoncelli, Virtual.NET Inc., and LumetaCorporation.All rights reserved. Printed in the United States of America. This publication is protectedby copyright, and permission must be obtained from the publisher prior to any prohibitedreproduction, storage in a retrieval system, or transmission in any form or by any means,electronic, mechanical, photocopying, recording, or likewise. For information regarding per-missions, write to: Pearson Education, Inc. Rights and Contracts Department 75 Arlington Street, Suite 300 Boston, MA 02116 Fax: (617) 848-7047ISBN 13: 978-0-321-49266-1ISBN 10: 0-321-49266-8Text printed in the United States on recycled paper at RR Donnelley in Crawfordsville, Indiana.First printing, June 2007
Contents at a GlancePart I Getting Started 1Chapter 1 What to Do When . . . 3Chapter 2 Climb Out of the Hole 27Part II Foundation Elements 39Chapter 3 Workstations 41Chapter 4 Servers 69Chapter 5 Services 95Chapter 6 Data Centers 129Chapter 7 Networks 187Chapter 8 Namespaces 223Chapter 9 Documentation 241Chapter 10 Disaster Recovery and Data Integrity 261Chapter 11 Security Policy 271Chapter 12 Ethics 323Chapter 13 Helpdesks 343Chapter 14 Customer Care 363Part III Change Processes 389Chapter 15 Debugging 391Chapter 16 Fixing Things Once 405Chapter 17 Change Management 415Chapter 18 Server Upgrades 435Chapter 19 Service Conversions 457Chapter 20 Maintenance Windows 473Chapter 21 Centralization and Decentralization 501 v
vi Contents at a Glance 521Part IV Providing Services 523 543Chapter 22 Service Monitoring 565Chapter 23 Email Service 583Chapter 24 Print Service 619Chapter 25 Data Storage 653Chapter 26 Backup and Restore 667Chapter 27 Remote Access Service 689Chapter 28 Software Depot ServiceChapter 29 Web Services 725Part V Management Practices 727 751Chapter 30 Organizational Structures 777Chapter 31 Perception and Visibility 819Chapter 32 Being Happy 853Chapter 33 A Guide for Technical Managers 871Chapter 34 A Guide for Nontechnical Managers 899Chapter 35 Hiring System Administrators 909Chapter 36 Firing System AdministratorsEpilogue 911Appendixes 913 939Appendix A The Many Roles of a System Administrator 945Appendix B Acronyms 955BibliographyIndex
ContentsPreface xxvAcknowledgments xxxvAbout the Authors xxxviiPart I Getting Started 1 1 What to Do When . . . 3 3 1.1 Building a Site from Scratch 4 1.2 Growing a Small Site 4 1.3 Going Global 4 1.4 Replacing Services 5 1.5 Moving a Data Center 5 1.6 Moving to/Opening a New Building 6 1.7 Handling a High Rate of Office Moves 7 1.8 Assessing a Site (Due Diligence) 8 1.9 Dealing with Mergers and Acquisitions 9 1.10 Coping with Machine Crashes 10 1.11 Surviving a Major Outage or Work Stoppage 11 1.12 What Tools Should Every Team Member Have? 12 1.13 Ensuring the Return of Tools 12 1.14 Why Document Systems and Procedures? 13 1.15 Why Document Policies? 1.16 Identifying the Fundamental Problems in 13 14 the Environment 14 1.17 Getting More Money for Projects 15 1.18 Getting Projects Done 1.19 Keeping Customers Happy vii
viii Contents1.20 Keeping Management Happy 151.21 Keeping SAs Happy 161.22 Keeping Systems from Being Too Slow 161.23 Coping with a Big Influx of Computers 161.24 Coping with a Big Influx of New Users 171.25 Coping with a Big Influx of New SAs 171.26 Handling a High SA Team Attrition Rate 181.27 Handling a High User-Base Attrition Rate 181.28 Being New to a Group 181.29 Being the New Manager of a Group 191.30 Looking for a New Job 191.31 Hiring Many New SAs Quickly 201.32 Increasing Total System Reliability 201.33 Decreasing Costs 211.34 Adding Features 211.35 Stopping the Hurt When Doing “This” 221.36 Building Customer Confidence 221.37 Building the Team’s Self-Confidence 221.38 Improving the Team’s Follow-Through 221.39 Handling Ethics Issues 231.40 My Dishwasher Leaves Spots on My Glasses 231.41 Protecting Your Job 231.42 Getting More Training 241.43 Setting Your Priorities 241.44 Getting All the Work Done 251.45 Avoiding Stress 251.46 What Should SAs Expect from Their Managers? 261.47 What Should SA Managers Expect from Their SAs? 261.48 What Should SA Managers Provide to Their Boss? 262 Climb Out of the Hole 27 2.1 Tips for Improving System Administration 28 2.1.1 Use a Trouble-Ticket System 28 2.1.2 Manage Quick Requests Right 29 2.1.3 Adopt Three Time-Saving Policies 30 2.1.4 Start Every New Host in a Known State 32 2.1.5 Follow Our Other Tips 33 36 2.2 Conclusion
Part II Foundation Elements Contents ix 3 Workstations 39 3.1 The Basics 41 3.1.1 Loading the OS 44 3.1.2 Updating the System Software and Applications 3.1.3 Network Configuration 46 3.1.4 Avoid Using Dynamic DNS with DHCP 54 3.2 The Icing 57 3.2.1 High Confidence in Completion 61 3.2.2 Involve Customers in the Standardization Process 65 3.2.3 A Variety of Standard Configurations 65 3.3 Conclusion 66 66 4 Servers 67 4.1 The Basics 69 4.1.1 Buy Server Hardware for Servers 69 4.1.2 Choose Vendors Known for Reliable Products 4.1.3 Understand the Cost of Server Hardware 69 4.1.4 Consider Maintenance Contracts and Spare Parts 72 4.1.5 Maintaining Data Integrity 72 4.1.6 Put Servers in the Data Center 74 4.1.7 Client Server OS Configuration 78 4.1.8 Provide Remote Console Access 78 4.1.9 Mirror Boot Disks 79 4.2 The Icing 80 4.2.1 Enhancing Reliability and Service Ability 83 4.2.2 An Alternative: Many Inexpensive Servers 84 4.3 Conclusion 84 89 5 Services 92 5.1 The Basics 95 5.1.1 Customer Requirements 96 5.1.2 Operational Requirements 5.1.3 Open Architecture 98 5.1.4 Simplicity 100 5.1.5 Vendor Relations 104 107 108
x Contents 109 110 5.1.6 Machine Independence 111 5.1.7 Environment 112 5.1.8 Restricted Access 115 5.1.9 Reliability 116 5.1.10 Single or Multiple Servers 116 5.1.11 Centralization and Standards 119 5.1.12 Performance 120 5.1.13 Monitoring 120 5.1.14 Service Rollout 120 5.2 The Icing 122 5.2.1 Dedicated Machines 124 5.2.2 Full Redundancy 126 5.2.3 Dataflow Analysis for Scaling 5.3 Conclusion 129 6 Data Centers 130 131 6.1 The Basics 134 6.1.1 Location 134 6.1.2 Access 136 6.1.3 Security 149 6.1.4 Power and Cooling 150 6.1.5 Fire Suppression 159 6.1.6 Racks 166 6.1.7 Wiring 170 6.1.8 Labeling 171 6.1.9 Communication 172 173 6.1.10 Console Access 175 6.1.11 Workbench 176 6.1.12 Tools and Supplies 176 6.1.13 Parking Spaces 179 6.2 The Icing 179 179 6.2.1 Greater Redundancy 183 6.2.2 More Space 185 6.3 Ideal Data Centers 6.3.1 Tom’s Dream Data Center 6.3.2 Christine’s Dream Data Center 6.4 Conclusion
7 Networks Contents xi 7.1 The Basics 187 7.1.1 The OSI Model 7.1.2 Clean Architecture 188 7.1.3 Network Topologies 188 7.1.4 Intermediate Distribution Frame 190 7.1.5 Main Distribution Frame 191 7.1.6 Demarcation Points 197 7.1.7 Documentation 203 7.1.8 Simple Host Routing 205 7.1.9 Network Devices 205 207 7.1.10 Overlay Networks 209 7.1.11 Number of Vendors 212 7.1.12 Standards-Based Protocols 213 7.1.13 Monitoring 214 7.1.14 Single Administrative Domain 214 7.2 The Icing 216 217 7.2.1 Leading Edge versus Reliability 217 7.2.2 Multiple Administrative Domains 219 7.3 Conclusion 219 7.3.1 Constants in Networking 219 7.3.2 Things That Change in Network Design 2208 Namespaces 223 8.1 The Basics 224 8.1.1 Namespace Policies 224 8.1.2 Namespace Change Procedures 236 8.1.3 Centralizing Namespace Management 236 237 8.2 The Icing 238 8.2.1 One Huge Database 238 8.2.2 Further Automation 239 8.2.3 Customer-Based Updating 239 8.2.4 Leveraging Namespaces 239 8.3 Conclusion 2419 Documentation 242 242 9.1 The Basics 9.1.1 What to Document
xii Contents 9.1.2 A Simple Template for Getting Started 243 9.1.3 Easy Sources for Documentation 244 9.1.4 The Power of Checklists 246 9.1.5 Storage Documentation 247 9.1.6 Wiki Systems 249 9.1.7 A Search Facility 250 9.1.8 Rollout Issues 251 9.1.9 Self-Management versus Explicit Management 2519.2 The Icing 252 9.2.1 A Dynamic Documentation Repository 252 9.2.2 A Content-Management System 253 9.2.3 A Culture of Respect 253 9.2.4 Taxonomy and Structure 254 9.2.5 Additional Documentation Uses 255 9.2.6 Off-Site Links 2589.3 Conclusion 25810 Disaster Recovery and Data Integrity 26110.1 The Basics 261 10.1.1 Definition of a Disaster 26210.2 10.1.2 Risk Analysis 26210.3 10.1.3 Legal Obligations 263 10.1.4 Damage Limitation 264 10.1.5 Preparation 265 10.1.6 Data Integrity 267 The Icing 268 10.2.1 Redundant Site 268 10.2.2 Security Disasters 268 10.2.3 Media Relations 269 Conclusion 26911 Security Policy 27111.1 The Basics 272 11.1.1 Ask the Right Questions 273 11.1.2 Document the Company’s Security Policies 276 11.1.3 Basics for the Technical Staff 283 11.1.4 Management and Organizational Issues 300
Contents xiii11.2 The Icing 31511.3 11.2.1 Make Security Pervasive 315 11.2.2 Stay Current: Contacts and Technologies 31611.4 11.2.3 Produce Metrics 317 Organization Profiles 317 11.3.1 Small Company 318 11.3.2 Medium-Size Company 318 11.3.3 Large Company 319 11.3.4 E-Commerce Site 319 11.3.5 University 320 Conclusion 32112 Ethics 32312.1 The Basics 323 12.1.1 Informed Consent 32412.2 12.1.2 Professional Code of Conduct 32412.3 12.1.3 Customer Usage Guidelines 326 12.1.4 Privileged-Access Code of Conduct 327 12.1.5 Copyright Adherence 330 12.1.6 Working with Law Enforcement 332 The Icing 336 12.2.1 Setting Expectations on Privacy and Monitoring 336 12.2.2 Being Told to Do Something Illegal/Unethical 338 Conclusion 34013 Helpdesks 343 13.1 The Basics 343 13.1.1 Have a Helpdesk 344 13.1.2 Offer a Friendly Face 346 13.1.3 Reflect Corporate Culture 346 13.1.4 Have Enough Staff 347 13.1.5 Define Scope of Support 348 13.1.6 Specify How to Get Help 351 13.1.7 Define Processes for Staff 352 13.1.8 Establish an Escalation Process 352 13.1.9 Define “Emergency” in Writing 353 354 13.1.10 Supply Request-Tracking Software
xiv Contents13.2 The Icing 35613.3 13.2.1 Statistical Improvements 356 13.2.2 Out-of-Hours and 24/7 Coverage 357 13.2.3 Better Advertising for the Helpdesk 358 13.2.4 Different Helpdesks for Service Provision and Problem Resolution 359 Conclusion 36014 Customer Care 36314.1 The Basics 36414.2 14.1.1 Phase A/Step 1: The Greeting 36614.3 14.1.2 Phase B: Problem Identification 367 14.1.3 Phase C: Planning and Execution 373 14.1.4 Phase D: Verification 376 14.1.5 Perils of Skipping a Step 378 14.1.6 Team of One 380 The Icing 380 14.2.1 Based Model-Training 380 14.2.2 Holistic Improvement 381 14.2.3 Increased Customer Familiarity 381 14.2.4 Special Announcements for Major Outages 382 14.2.5 Trend Analysis 382 14.2.6 Customers Who Know the Process 384 14.2.7 Architectural Decisions That Match the Process 384 Conclusion 385Part III Change Processes 38915 Debugging 39115.1 The Basics 39115.2 15.1.1 Learn the Customer’s Problem 39215.3 15.1.2 Fix the Cause, Not the Symptom 393 15.1.3 Be Systematic 394 15.1.4 Have the Right Tools 395 The Icing 399 15.2.1 Better Tools 399 15.2.2 Formal Training on the Tools 400 15.2.3 End-to-End Understanding of the System 400 Conclusion 402
Contents xv16 Fixing Things Once 40516.1 The Basics 405 16.1.1 Don’t Waste Time 40516.2 16.1.2 Avoid Temporary Fixes 40716.3 16.1.3 Learn from Carpenters 410 The Icing 412 Conclusion 41417 Change Management 41517.1 The Basics 416 17.1.1 Risk Management 41717.2 17.1.2 Communications Structure 41817.3 17.1.3 Scheduling 419 17.1.4 Process and Documentation 422 17.1.5 Technical Aspects 424 The Icing 428 17.2.1 Automated Front Ends 428 17.2.2 Change-Management Meetings 428 17.2.3 Streamline the Process 431 Conclusion 43218 Server Upgrades 435 18.1 The Basics 435 18.1.1 Step 1: Develop a Service Checklist 436 18.1.2 Step 2: Verify Software Compatibility 438 18.1.3 Step 3: Verification Tests 439 18.1.4 Step 4: Write a Back-Out Plan 443 18.1.5 Step 5: Select a Maintenance Window 443 18.1.6 Step 6: Announce the Upgrade as Appropriate 445 18.1.7 Step 7: Execute the Tests 446 18.1.8 Step 8: Lock out Customers 446 18.1.9 Step 9: Do the Upgrade with Someone Watching 447 447 18.1.10 Step 10: Test Your Work 448 18.1.11 Step 11: If All Else Fails, Rely on the Back-Out Plan 448 18.1.12 Step 12: Restore Access to Customers 448 18.1.13 Step 13: Communicate Completion/Back-Out
xvi Contents18.2 The Icing 44918.3 18.2.1 Add and Remove Services at the Same Time 450 18.2.2 Fresh Installs 450 18.2.3 Reuse of Tests 451 18.2.4 Logging System Changes 451 18.2.5 A Dress Rehearsal 451 18.2.6 Installation of Old and New Versions on the 452 Same Machine 452 18.2.7 Minimal Changes from the Base 454 Conclusion 45719 Service Conversions 45819.1 The Basics 458 19.1.1 Minimize Intrusiveness 46019.2 19.1.2 Layers versus Pillars 46119.3 19.1.3 Communication 462 19.1.4 Training 463 19.1.5 Small Groups First 463 19.1.6 Flash-Cuts: Doing It All at Once 465 19.1.7 Back-Out Plan 467 The Icing 467 19.2.1 Instant Rollback 468 19.2.2 Avoiding Conversions 469 19.2.3 Web Service Conversions 470 19.2.4 Vendor Support 470 Conclusion 47320 Maintenance Windows 47520.1 The Basics 475 20.1.1 Scheduling 477 20.1.2 Planning 478 20.1.3 Directing 479 20.1.4 Managing Change Proposals 481 20.1.5 Developing the Master Plan 482 20.1.6 Disabling Access 483 20.1.7 Ensuring Mechanics and Coordination 488 20.1.8 Deadlines for Change Completion 489 20.1.9 Comprehensive System Testing
20.1.10 Postmaintenance Communication Contents xvii 20.1.11 Reenable Remote Access 20.1.12 Be Visible the Next Morning 490 20.1.13 Postmortem 49120.2 The Icing 491 492 20.2.1 Mentoring a New Flight Director 492 20.2.2 Trending of Historical Data 492 20.2.3 Providing Limited Availability 49320.3 High-Availability Sites 493 20.3.1 The Similarities 495 20.3.2 The Differences 49520.4 Conclusion 496 49721 Centralization and Decentralization 50121.1 The Basics 50221.2 21.1.1 Guiding Principles21.3 21.1.2 Candidates for Centralization 502 21.1.3 Candidates for Decentralization 505 The Icing 510 21.2.1 Consolidate Purchasing 512 21.2.2 Outsourcing 513 Conclusion 515 519Part IV Providing Services 52122 Service Monitoring 52322.1 The Basics 52322.2 22.1.1 Historical Monitoring 22.1.2 Real-Time Monitoring 52522.3 The Icing 527 22.2.1 Accessibility 534 22.2.2 Pervasive Monitoring 534 22.2.3 Device Discovery 535 22.2.4 End-to-End Tests 535 22.2.5 Application Response Time Monitoring 536 22.2.6 Scaling 537 22.2.7 Metamonitoring 537 Conclusion 539 540
xviii Contents23 Email Service 54323.1 The Basics 543 23.1.1 Privacy Policy 544 23.1.2 Namespaces 544 23.1.3 Reliability 546 23.1.4 Simplicity 547 23.1.5 Spam and Virus Blocking 549 23.1.6 Generality 550 23.1.7 Automation 552 23.1.8 Basic Monitoring 552 23.1.9 Redundancy 553 554 23.1.10 Scaling 556 23.1.11 Security Issues 557 23.1.12 Communication 55823.2 The Icing 559 559 23.2.1 Encryption 560 23.2.2 Email Retention Policy 561 23.2.3 Advanced Monitoring 562 23.2.4 High-Volume List Processing23.3 Conclusion 56524 Print Service 566 56624.1 The Basics 56824.2 24.1.1 Level of Centralization 57224.3 24.1.2 Print Architecture Policy 573 24.1.3 System Design 574 24.1.4 Documentation 575 24.1.5 Monitoring 576 24.1.6 Environmental Issues 577 The Icing 578 24.2.1 Automatic Failover and Load Balancing 578 24.2.2 Dedicated Clerical Support 579 24.2.3 Shredding 580 24.2.4 Dealing with Printer Abuse Conclusion 58325 Data Storage 584 58425.1 The Basics 25.1.1 Terminology
25.2 25.1.2 Managing Storage Contents xix25.3 25.1.3 Storage as a Service 25.1.4 Performance 588 25.1.5 Evaluating New Storage Solutions 596 25.1.6 Common Problems 604 The Icing 608 25.2.1 Optimizing RAID Usage by Applications 609 25.2.2 Storage Limits: Disk Access Density Gap 611 25.2.3 Continuous Data Protection 611 Conclusion 613 61426 Backup and Restore 615 26.1 The Basics 619 26.1.1 Reasons for Restores 26.1.2 Types of Restores 620 26.1.3 Corporate Guidelines 621 26.1.4 A Data-Recovery SLA and Policy 624 26.1.5 The Backup Schedule 625 26.1.6 Time and Capacity Planning 626 26.1.7 Consumables Planning 627 26.1.8 Restore-Process Issues 633 26.1.9 Backup Automation 635 637 26.1.10 Centralization 639 26.1.11 Tape Inventory 641 26.2 The Icing 642 643 26.2.1 Fire Drills 643 26.2.2 Backup Media and Off-Site Storage 644 26.2.3 High-Availability Databases 647 26.2.4 Technology Changes 648 26.3 Conclusion 64927 Remote Access Service 65327.1 The Basics 654 27.1.1 Requirements for Remote Access 654 27.1.2 Policy for Remote Access 656 27.1.3 Definition of Service Levels 656 27.1.4 Centralization 658 27.1.5 Outsourcing 658
xx Contents27.2 27.1.6 Authentication 66127.3 27.1.7 Perimeter Security 661 The Icing 662 27.2.1 Home Office 662 27.2.2 Cost Analysis and Reduction 663 27.2.3 New Technologies 664 Conclusion 66528 Software Depot Service 66728.1 The Basics 66928.2 28.1.1 Understand the Justification 66928.3 28.1.2 Understand the Technical Expectations 670 28.1.3 Set the Policy 671 28.1.4 Select Depot Software 672 28.1.5 Create the Process Manual 672 28.1.6 Examples 673 The Icing 682 28.2.1 Different Configurations for Different Hosts 682 28.2.2 Local Replication 683 28.2.3 Commercial Software in the Depot 684 28.2.4 Second-Class Citizens 684 Conclusion 68629 Web Services 68929.1 The Basics 690 29.1.1 Web Service Building Blocks 69029.2 29.1.2 The Webmaster Role 69329.3 29.1.3 Service-Level Agreements 694 29.1.4 Web Service Architectures 694 29.1.5 Monitoring 698 29.1.6 Scaling for Web Services 699 29.1.7 Web Service Security 703 29.1.8 Content Management 710 29.1.9 Building the Manageable Generic Web Server 714 The Icing 718 29.2.1 Third-Party Web Hosting 718 29.2.2 Mashup Applications 721 Conclusion 722
Part V A Management Practices Contents xxi30 Organizational Structures 72530.1 The Basics 727 30.1.1 Sizing 72730.2 30.1.2 Funding Models30.3 30.1.3 Management Chain’s Influence 72830.4 30.1.4 Skill Selection 730 30.1.5 Infrastructure Teams 733 30.1.6 Customer Support 735 30.1.7 Helpdesk 737 30.1.8 Outsourcing 739 The Icing 741 30.2.1 Consultants and Contractors 741 Sample Organizational Structures 743 30.3.1 Small Company 743 30.3.2 Medium-Size Company 745 30.3.3 Large Company 745 30.3.4 E-Commerce Site 745 30.3.5 Universities and Nonprofit Organizations 746 Conclusion 746 74731 Perception and Visibility 74831.1 The Basics 75131.2 31.1.1 A Good First Impression 752 31.1.2 Attitude, Perception, and Customers31.3 31.1.3 Priorities Aligned with Customer Expectations 752 31.1.4 The System Advocate 756 The Icing 758 31.2.1 The System Status Web Page 760 31.2.2 Management Meetings 765 31.2.3 Physical Visibility 765 31.2.4 Town Hall Meetings 766 31.2.5 Newsletters 767 31.2.6 Mail to All Customers 768 31.2.7 Lunch 770 Conclusion 770 773 773
xxii Contents32 Being Happy 77732.1 The Basics 778 32.1.1 Follow-Through 77832.2 32.1.2 Time Management 78032.3 32.1.3 Communication Skills 79032.4 32.1.4 Professional Development 796 32.1.5 Staying Technical 797 The Icing 797 32.2.1 Learn to Negotiate 798 32.2.2 Love Your Job 804 32.2.3 Managing Your Manager 811 Further Reading 815 Conclusion 81533 A Guide for Technical Managers 81933.1 The Basics 81933.2 33.1.1 Responsibilities 82033.3 33.1.2 Working with Nontechnical Managers 835 33.1.3 Working with Your Employees 838 33.1.4 Decisions 843 The Icing 849 33.2.1 Make Your Team Even Stronger 849 33.2.2 Sell Your Department to Senior Management 849 33.2.3 Work on Your Own Career Growth 850 33.2.4 Do Something You Enjoy 850 Conclusion 85034 A Guide for Nontechnical Managers 85334.1 The Basics 853 34.1.1 Priorities and Resources 854 34.1.2 Morale 855 34.1.3 Communication 857 34.1.4 Staff Meetings 858 34.1.5 One-Year Plans 860 34.1.6 Technical Staff and the Budget Process 860 34.1.7 Professional Development 862
34.2 The Icing Contents xxiii34.3 34.2.1 A Five-Year Vision 34.2.2 Meetings with Single Point of Contact 863 34.2.3 Understanding the Technical Staff’s Work 864 Conclusion 866 86835 Hiring System Administrators 869 35.1 The Basics 871 35.1.1 Job Description 871 35.1.2 Skill Level 35.1.3 Recruiting 872 35.1.4 Timing 874 35.1.5 Team Considerations 875 35.1.6 The Interview Team 877 35.1.7 Interview Process 878 35.1.8 Technical Interviewing 882 35.1.9 Nontechnical Interviewing 884 886 35.1.10 Selling the Position 891 35.1.11 Employee Retention 892 35.2 The Icing 893 894 35.2.1 Get Noticed 894 35.3 Conclusion 89536 Firing System Administrators 899 90036.1 The Basics 36.1.1 Follow Your Corporate HR Policy 90036.2 36.1.2 Have a Termination Checklist 90036.3 36.1.3 Remove Physical Access 901 36.1.4 Remove Remote Access 901 36.1.5 Remove Service Access 902 36.1.6 Have Fewer Access Databases 904 The Icing 905 36.2.1 Have a Single Authentication Database 905 36.2.2 System File Changes 906 Conclusion 906
xxiv Contents 909Epilogue 911AppendixesAppendix A The Many Roles of a System Administrator 913Appendix B Acronyms 939Bibliography 945Index 955
PrefaceOur goal for this book has been to write down everything we’ve learned fromour mentors and to add our real-world experiences. These things are beyondwhat the manuals and the usual system administration books teach. This book was born from our experiences as SAs in a variety of organi-zations. We have started new companies. We have helped sites to grow. Wehave worked at small start-ups and universities, where lack of funding wasan issue. We have worked at midsize and large multinationals, where mergersand spin-offs gave rise to strange challenges. We have worked at fast-pacedcompanies that do business on the Internet and where high-availability, high-performance, and scaling issues were the norm. We’ve worked at slow-pacedcompanies at which high tech meant cordless phones. On the surface, theseare very different environments with diverse challenges; underneath, theyhave the same building blocks, and the same fundamental principles apply. This book gives you a framework—a way of thinking about system ad-ministration problems—rather than narrow how-to solutions to particularproblems. Given a solid framework, you can solve problems every time theyappear, regardless of the operating system (OS), brand of computer, or type ofenvironment. This book is unique because it looks at system administrationfrom this holistic point of view; whereas most other books for SAs focus onhow to maintain one particular product. With experience, however, all SAslearn that the big-picture problems and solutions are largely independent ofthe platform. This book will change the way you approach your work asan SA. The principles in this book apply to all environments. The approachesdescribed may need to be scaled up or down, depending on your environment,but the basic principles still apply. Where we felt that it might not be obvioushow to implement certain concepts, we have included sections that illustratehow to apply the principles at organizations of various sizes. xxv
xxvi Preface This book is not about how to configure or debug a particular OS andwill not tell you how to recover the shared libraries or DLLs when someoneaccidentally moves them. Some excellent books cover those topics, and werefer you to many of them throughout. Instead, we discuss the principles,both basic and advanced, of good system administration that we have learnedthrough our own and others’ experiences. These principles apply to all OSs.Following them well can make your life a lot easier. If you improve the wayyou approach problems, the benefit will be multiplied. Get the fundamentalsright, and everything else falls into place. If they aren’t done well, you willwaste time repeatedly fixing the same things, and your customers1 will beunhappy because they can’t work effectively with broken machines.Who Should Read This BookThis book is written for system administrators at all levels. It gives junior SAsinsight into the bigger picture of how sites work, their roles in the organiza-tions, and how their careers can progress. Intermediate SAs will learn how toapproach more complex problems and how to improve their sites and maketheir jobs easier and their customers happier. Whatever level you are at, thisbook will help you to understand what is behind your day-to-day work, tolearn the things that you can do now to save time in the future, to decidepolicy, to be architects and designers, to plan far into the future, to nego-tiate with vendors, and to interface with management. These are the thingsthat concern senior SAs. None of them are listed in an OS’s manual. Evensenior SAs and systems architects can learn from our experiences and thoseof our colleagues, just as we have learned from each other in writing thisbook. We also cover several management topics for SA trying to understandtheir managers, for SAs who aspire to move into management, and for SAsfinding themselves doing more and more management without the benefit ofthe title. Throughout the book, we use examples to illustrate our points. The ex-amples are mostly from medium or large sites, where scale adds its own prob-lems. Typically, the examples are generic rather than specific to a particularOS; where they are OS-specific, it is usually UNIX or Windows. One of the strongest motivations we had for writing this book is theunderstanding that the problems SAs face are the same across all OSs. A new 1. Throughout the book, we refer to the end users of our systems as customers rather than users. Adetailed explanation of why we do this is in Section 31.1.2.
Preface xxviiOS that is significantly different from what we are used to can seem likea black box, a nuisance, or even a threat. However, despite the unfamiliarinterface, as we get used to the new technology, we eventually realize thatwe face the same set of problems in deploying, scaling, and maintaining thenew OS. Recognizing that fact, knowing what problems need solving, andunderstanding how to approach the solutions by building on experience withother OSs lets us master the new challenges more easily. We want this book to change your life. We want you to become so suc-cessful that if you see us on the street, you’ll give us a great big hug.Basic PrinciplesIf we’ve learned anything over the years, it is the importance of simplicity,clarity, generality, automation, communication, and doing the basics first.These six principles are recurring themes in this book. 1. Simplicity means that the smallest solution that solves the entire problem is the best solution. It keeps the systems easy to understand and reduces complex component interactions that can cause debugging nightmares. 2. Clarity means that the solution is straightforward. It can be easily explained to someone on the project or even outside the project. Clarity makes it easier to change the system, as well as to maintain and debug it. In the system administration world, it’s better to write five lines of understandable code than one line that’s incomprehensible to anyone else. 3. Generality means that the solutions aren’t inherently limited to a particular case. Solutions can be reused. Using vendor-independent open standard protocols makes systems more flexible and makes it easier to link software packages together for better services. 4. Automation means using software to replace human effort. Automa- tion is critical. Automation improves repeatability and scalability, is key to easing the system administration burden, and eliminates tedious repetitive tasks, giving SAs more time to improve services. 5. Communication between the right people can solve more problems than hardware or software can. You need to communicate well with other SAs and with your customers. It is your responsibility to initiate communication. Communication ensures that everyone is working
xxviii Preface toward the same goals. Lack of communication leaves people con- cerned and annoyed. Communication also includes documentation. Documentation makes systems easier to support, maintain, and upgrade. Good communication and proper documentation also make it easier to hand off projects and maintenance when you leave or take on a new role. 6. Basics first means that you build the site on strong foundations by identifying and solving the basic problems before trying to attack more advanced ones. Doing the basics first makes adding advanced features considerably easier and makes services more robust. A good basic infrastructure can be repeatedly leveraged to improve the site with relatively little effort. Sometimes, we see SAs making a huge effort to solve a problem that wouldn’t exist or would be a simple enhancement if the site had a basic infrastructure in place. This book will help you identify what the basics are and show you how the other five principles apply. Each chapter looks at the basics of a given area. Get the fundamentals right, and everything else will fall into place. These principles are universal. They apply at all levels of the system.They apply to physical networks and to computer hardware. They applyto all operating systems running at a site, all protocols used, all software,and all services provided. They apply at universities, nonprofit institutions,government sites, businesses, and Internet service sites.What Is an SA?If you asked six system administrators to define their jobs, you would getseven different answers. The job is difficult to define because system admin-istrators do so many things. An SA looks after computers, networks, and thepeople who use them. An SA may look after hardware, operating systems,software, configurations, applications, or security. A system administratorinfluences how effectively other people can or do use their computers andnetworks. A system administrator sometimes needs to be a business-process con-sultant, corporate visionary, janitor, software engineer, electrical engineer,economist, psychiatrist, mindreader, and, occasionally, a bartender. As a result, companies calls SAs different names. Sometimes, they arecalled network administrators, system architects, system engineers, systemprogrammers, operators and so on.
Preface xxix This book is for “all of the above.” We have a very general definition of system administrator: one who man-ages computer and network systems on behalf of another, such as an employeror a client. SAs are the people who make things work and keep it all running. Explaining What System Administration Entails It’s difficult to define system administration, but trying to explain it to a nontechnical person is even more difficult, especially if that person is your mom. Moms have the right to know how their offspring are paying their rent. A friend of Christine Hogan’s always had trouble explaining to his mother what he did for a living and ended up giving a different answer every time she asked. Therefore, she kept repeating the question every couple of months, waiting for an answer that would be meaningful to her. Then he started working for WebTV. When the product became available, he bought one for his mom. From then on, he told her that he made sure that her WebTV service was working and was as fast as possible. She was very happy that she could now show her friends something and say, “That’s what my son does!”System Administration MattersSystem administration matters because computers and networks matter.Computers are a lot more important than they were years ago. Whathappened? The widespread use of the Internet, intranets, and the move to a web-centric world has redefined the way companies depend on computers. TheInternet is a 24/7 operation, and sloppy operations can no longer be tolerated.Paper purchase orders can be processed daily, in batches, with no one thewiser. However, there is an expectation that the web-based system that doesthe process will be available all the time, from anywhere. Nightly maintenancewindows have become an unheard-of luxury. That unreliable machine roompower system that caused occasional but bearable problems now preventssales from being recorded. Management now has a more realistic view of computers. Before they hadPCs on their desktops, most people’s impressions of computers were basedon how they were portrayed in film: big, all-knowing, self-sufficient, miraclemachines. The more people had direct contact with computers, the morerealistic people’s expectations became. Now even system administration itselfis portrayed in films. The 1993 classic Jurassic Park was the first mainstreammovie to portray the key role that system administrators play in large systems.
xxx PrefaceThe movie also showed how depending on one person is a disaster waitingto happen. IT is a team sport. If only Dennis Nedry had read this book. In business, nothing is important unless the CEO feels that it is im-portant. The CEO controls funding and sets priorities. CEOs now considerIT to be important. Email was previously for nerds; now CEOs depend onemail and notice even brief outages. The massive preparations for Y2K alsobrought home to CEOs how dependent their organizations have become oncomputers, how expensive it can be to maintain them, and how quickly apurely technical issue can become a serious threat. Most people do not thinkthat they simply “missed the bullet” during the Y2K change but that prob-lems were avoided thanks to tireless efforts by many people. A CBS Pollshows 63 percent of Americans believe that the time and effort spent fixingpotential problems was worth it. A look at the news lineups of all threemajor network news broadcasts from Monday, January 3, 2000, reflects thesame feeling. Previously, people did not grow up with computers and had to cautiouslylearn about them and their uses. Now more and more people grow up usingcomputers, which means that they have higher expectations of them whenthey are in positions of power. The CEOs who were impressed by automaticpayroll processing are soon to be replaced by people who grew up sendinginstant messages and want to know why they can’t do all their business viatext messaging. Computers matter more than ever. If computers are to work and workwell, system administration matters. We matter.Organization of This BookThis book has the following major parts: • Part I: Getting Started. This is a long book, so we start with an overview of what to expect (Chapter 1) and some tips to help you find enough time to read the rest of the book (Chapter 2). • Part II: Foundation Elements. Chapters 3–14 focus on the foundations of IT infrastructure, the hardware and software that everything else depends on. • Part III: Change Processes. Chapters 15–21 look at how to make changes to systems, starting with fixing the smallest bug to massive reorganizations.
Preface xxxi • Part IV: Providing Services. Chapters 22–29 offer our advice on building seven basic services, such as email, printing, storage, and web services. • Part V: Management Practices. Chapters 30–36 provide guidance— whether or not you have “manager” in your title. • The two appendixes provide an overview of the positive and negative roles that SAs play and a list of acronyms used in the book. Each chapter discusses a separate topic; some topics are technical, andsome are nontechnical. If one chapter doesn’t apply to you, feel free to skipit. The chapters are linked, so you may find yourself returning to a chapterthat you previously thought was boring. We won’t be offended. Each chapter has two major sections. The Basics discusses the essentialsthat you simply have to get right. Skipping any of these items will simplycreate more work for you in the future. Consider them investments that payoff in efficiency later on. The Icing deals with the cool things that you can doto be spectacular. Don’t spend your time with these things until you are donewith the basics. We have tried to drive the points home through anecdotes andcase studies from personal experience. We hope that this makes the advicehere more “real” for you. Never trust salespeople who don’t use their ownproducts.What’s New in the Second EditionWe received a lot of feedback from our readers about the first edition. Wespoke at conferences and computer user groups around the world. Wereceived a lot of email. We listened. We took a lot of notes. We’ve smoothedthe rough edges and filled some of the major holes. The first edition garnered a lot of positive reviews and buzz. We werevery honored. However, the passing of time made certain chapters look passe´. The first edition, in bookstores August 2001, was written mostly in 2000.Things were very different then. At the time, things were looking pretty grimas the dot-com boom had gone bust. Windows 2000 was still new, Solariswas king, and Linux was popular only with geeks. Spam was a nuisance,not an industry. Outsourcing had lost its luster and had gone from being thecorporate savior to a late-night comedy punch line. Wikis were a researchidea, not the basis for the world’s largest free encyclopedia. Google was nei-ther a household name nor a verb. Web farms were rare, and “big sites”served millions of hits per day, not per hour. In fact, we didn’t have a chapter
xxxii Prefaceon running web servers, because we felt that all one needed to know couldbe inferred by reading the right combination of the chapters: Data Centers,Servers, Services, and Service Monitoring. What more could people need? My, how things have changed! Linux is no longer considered a risky proposition, Google is on the rise,and offshoring is the new buzzword. The rise of India and China as economicsuperpowers has changed the way we think about the world. AJAX and otherWeb 2.0 technologies have made the web applications exciting again. Here’s what’s new in the book: • Updated chapters: Every chapter has been updated and modernized and new anecdotes added. We clarified many, many points. We’ve learned a lot in the past five years, and all the chapters reflect this. References to old technologies have been replaced with more relevant ones. • New chapters: – Chapter 9: Documentation – Chapter 25: Data Storage – Chapter 29: Web Services • Expanded chapters: – The first edition’s Appendix B, which had been missed by many read- ers who didn’t read to the end of the book, is now Chapter 1: What to Do When . . . . – The first edition’s Do These First section in the front matter has ex- panded to become Chapter 2: Climb Out of the Hole. • Reordered table of contents: – Part I: Getting Started: introductory and overview material – Part II: Foundation Elements: the foundations of any IT system – Part III: Change Processes: how to make changes from the smallest to the biggest – Part IV: Providing Services: a catalog of common service offerings – Part V: Management Practices: organizational issues
Preface xxxiiiWhat’s NextEach chapter is self-contained. Feel free to jump around. However, we havecarefully ordered the chapters so that they make the most sense if you readthe book from start to finish. Either way, we hope that you enjoy the book.We have learned a lot and had a lot of fun writing it. Let’s begin. Thomas A. Limoncelli Google, Inc. [email protected] Christina J. Hogan BMW Sauber F1 Team [email protected] Strata R. Chalup Virtual.Net, Inc. [email protected] Books, like software, always have bugs. For a list of updates, along withnews and notes, and even a mailing list you can join, please visit our web
This page intentionally left blank
AcknowledgmentsAcknowledgments for the First EditionWe can’t possibly thank everyone who helped us in some way or another,but that isn’t going to stop us from trying. Much of this book was inspiredby Kernighan and Pike’s The Practice of Programming (Kernighan and Pike1999) and John Bentley’s second edition of Programming Pearls (Bentley1999). We are grateful to Global Networking and Computing (GNAC), Synop-sys, and Eircom for permitting us to use photographs of their data center fa-cilities to illustrate real-life examples of the good practices that we talk about. We are indebted to the following people for their helpful editing: ValerieNatale, Anne Marie Quint, Josh Simon, and Amara Willey. The people we have met through USENIX and SAGE and the LISA con-ferences have been major influences in our lives and careers. We would notbe qualified to write this book if we hadn’t met the people we did and learnedso much from them. Dozens of people helped us as we wrote this book—some by supplyinganecdotes, some by reviewing parts of or the entire book, others by mentoringus during our careers. The only fair way to thank them all is alphabeticallyand to apologize in advance to anyone that we left out: Rajeev Agrawala, AlAho, Jeff Allen, Eric Anderson, Ann Benninger, Eric Berglund, Melissa Binde,Steven Branigan, Sheila Brown-Klinger, Brent Chapman, Bill Cheswick, LeeDamon, Tina Darmohray, Bach Thuoc (Daisy) Davis, R. Drew Davis, IngoDean, Arnold de Leon, Jim Dennis, Barbara Dijker, Viktor Dukhovni, Chelle-Marie Ehlers, Michael Erlinger, Paul Evans, Re´my Evard, Lookman Fazal,Robert Fulmer, Carson Gaspar, Paul Glick, David “Zonker” Harris, Kather-ine “Cappy” Harrison, Jim Hickstein, Sandra Henry-Stocker, Mark Horton,Bill “Whump” Humphries, Tim Hunter, Jeff Jensen, Jennifer Joy, Alan Judge,Christophe Kalt, Scott C. Kennedy, Brian Kernighan, Jim Lambert, Eliot Lear, xxxv
xxxvi AcknowledgmentsSteven Levine, Les Lloyd, Ralph Loura, Bryan MacDonald, Sherry McBride,Mark Mellis, Cliff Miller, Hal Miller, Ruth Milner, D. Toby Morrill, Joe Mor-ris, Timothy Murphy, Ravi Narayan, Nils-Peter Nelson, Evi Nemeth, WilliamNinke, Cat Okita, Jim Paradis, Pat Parseghian, David Parter, Rob Pike, HalPomeranz, David Presotto, Doug Reimer, Tommy Reingold, Mike Richichi,Matthew F. Ringel, Dennis Ritchie, Paul D. Rohrigstamper, Ben Rosengart,David Ross, Peter Salus, Scott Schultz, Darren Shaw, Glenn Sieb, Karl Siil,Cicely Smith, Bryan Stansell, Hal Stern, Jay Stiles, Kim Supsinkas, KenThompson, Greg Tusar, Kim Wallace, The Rabbit Warren, Dr. Geri Weitz-man, PhD, Glen Wiley, Pat Wilson, Jim Witthoff, Frank Wojcik, Jay Yu, andElizabeth Zwicky. Thanks also to Lumeta Corporation and Lucent Technologies/Bell Labsfor their support in writing this book. Last but not least, the people at Addison-Wesley made this a particularlygreat experience for us. In particular, our gratitude extends to Karen Gettman,Mary Hart, and Emily Frey.Acknowledgments for the Second EditionIn addition to everyone who helped us with the first edition, the second edi-tion could not have happened without the help and support of Lee Damon,Nathan Dietsch, Benjamin Feen, Stephen Harris, Christine E. Polk, Glenn E.Sieb, Juhani Tali, and many people at the League of Professional System Ad-ministrators (LOPSA). Special 73s and 88s to Mike Chalup for love, loyalty,and support, and especially for the mountains of laundry done and oceans ofdishes washed so Strata could write. And many cuddles and kisses for babyJoanna Lear for her patience. Thanks to Lumeta Corporation for giving us permission to publish asecond edition. Thanks to Wingfoot for letting us use its server for our bug-trackingdatabase. Thanks to Anne Marie Quint for data entry, copyediting, and a lot ofgreat suggestions. And last but not least, a big heaping bowl of “couldn’t have done itwithout you” to Mark Taub, Catherine Nolan, Raina Chrobak, and LaraWysong at Addison-Wesley.
About the AuthorsTom, Christine, and Strata know one another through attending USENIXconferences and being actively involved in the system administration com-munity. It was at one of these conferences that Tom and Christine first spokeabout collaborating on this book. Strata and Christine were coworkers atSynopsys and GNAC, and coauthored Chalup, Hogan et al. (1998).Thomas A. LimoncelliTom is an internationally recognized author and speaker on system adminis-tration, time management, and grass-roots political organizing techniques. Asystem administrator since 1988, he has worked for small and large compa-nies, including Google, Cibernet Corp, Dean for America, Lumeta, AT&T,Lucent/Bell Labs, and Mentor Graphics. At Google, he is involved in improv-ing how IT infrastructure is deployed at new offices. When AT&T trivestedinto AT&T, Lucent, and NCR, Tom led the team that split the Bell Labscomputing and network infrastructure into the three new companies. In addition to the first and second editions of this book, his publishedworks include Time Management for System Administration (2005), andpapers on security, networking, project management, and personal careermanagement. He travels to conferences and user groups frequently, oftenteaching tutorials, facilitating workshops, presenting papers, or giving invitedtalks and keynote speeches. Outside of work, Tom is a grassroots civil-rights activist who has re-ceived awards and recognition on both state and national levels. Tom’s firstpublished paper (Limoncelli 1997) extolled the lessons SAs can learn fromactivists. Tom doesn’t see much difference between his work and activismcareers—both are about helping people. He holds a B.A. in computer science from Drew University. He lives inBloomfield, New Jersey. xxxvii
xxxviii About the Authors For their community involvement, Tom and Christine shared the 2005Outstanding Achievement Award from USENIX/SAGE.Christina J. HoganChristine’s system administration career started at the Department of Mathe-matics in Trinity College, Dublin, where she worked for almost 5 years. Afterthat, she went in search of sunshine and moved to Sicily, working for a yearin a research company, and followed that with 5 years in California. She was the security architect at Synopsys for a couple of years beforejoining some friends at GNAC a few months after it was founded. Whilethere, she worked with start-ups, e-commerce sites, biotech companies, andlarge multinational hardware and software companies. On the technical side,she focused on security and networking, working with customers and helpingGNAC establish its data center and Internet connectivity. She also becameinvolved with project management, customer management, and people man-agement. After almost 3 years at GNAC, she went out on her own as anindependent security consultant, working primarily at e-commerce sites. Since then, she has become a mother and made a career change: shenow works as an aerodynamicist for the BMW Sauber Formula 1 RacingTeam. She has a Ph.D. in aeronautical engineering from Imperial College,London; a B.A. in mathematics and an M.Sc. in computer science from TrinityCollege, Dublin; and a Diploma in legal studies from the Dublin Institute ofTechnology.Strata R. ChalupStrata is the owner and senior consultant of Virtual.Net, Inc., a strategicand best-practices IT consulting firm specializing in helping small to midsizefirms scale their IT practices as they grow. During the first dot-com boom,Strata architected scalable infrastructures and managed some of the teamsthat built them for such projects as, the Palm VII, and as a sole proprietorship in 1993, Virtual.Net was incorporated in2005. Clients have included such firms as Apple, Sun, Cimflex Teknowledge,Cisco, McAfee, and Micronas USA. Strata joined the computing world on TOPS-20 on DEC mainframesin 1981, then got well and truly sidetracked onto administering UNIX by1983, with Ultrix on the VAX 11-780, Unisys on Motorola 68K micro sys-tems, and a dash of Minix on Intel thrown in for good measure. She has the
About the Authors xxxixunusual perspective of someone who has been both a user and an adminis-trator of Internet services since 1981 and has seen much of what we considerthe modern Net evolve, sometimes from a front-row seat. An early adopterand connector, she was involved with the early National Telecommunica-tions Infrastructure Administration (NTIA) hearings and grant reviews from1993–1995 and demonstrated the emerging possibilities of the Internet in1994, creating NTIA’s groundbreaking virtual conference. A committed fu-turist, Strata avidly tracks new technologies for collaboration and leveragesthem for IT and management. Always a New Englander at heart, but marooned in California witha snow-hating spouse, Strata is an active gardener, reader of science fic-tion/fantasy, and emergency services volunteer in amateur radio (KF6NBZ).She is SCUBA-certified but mostly free dives and snorkles. Strata has spenta couple of years as a technomad crossing the country by RV, first in 1990and again in 2002, consulting from the road. She has made a major hobby ofstudying energy-efficient building construction and design, including takingowner-builder classes, and really did grow up on a goat farm. Unlike her illustrious coauthors, she is an unrepentent college dropout,having left MIT during her sophmore year. She returned to manage the Cen-ter for Cognitive Science for several years, and to consult with the EECSComputing Services group, including a year as [email protected], beforeheading to Silicon Valley.
This page intentionally left blank
Part IGetting Started
This page intentionally left blank
1Chapter What to Do When . . .In this chapter, we pull together the various elements from the rest of thebook to provide an overview of how they can be used to deal with everydaysituations or to answer common questions system administrators (SAs) andmanagers often have.1.1 Building a Site from Scratch • Think about the organizational structure you need—Chapter 30. • Check in with management on the business priorities that will drive implementation priorities. • Plan your namespaces carefully—Chapter 8. • Build a rock-solid data center—Chapter 6. • Build a rock-solid network designed to grow—Chapter 7. • Build services that will scale—Chapter 5. • Build a software depot, or at least plan a small directory hierarchy that can grow into a software depot—Chapter 28. • Establish your initial core application services: – Authentication and authorization—Section 3.1.3 – Desktop life-cycle management—Chapter 3 – Email—Chapter 23 – File service, backups—Chapter 26 – Network configuration—Section 3.1.3 – Printing—Chapter 24 – Remote access—Chapter 27 3
4 Chapter 1 What to Do When . . .1.2 Growing a Small Site • Provide a helpdesk—Chapter 13. • Establish checklists for new hires, new desktops/laptops, and new servers—Section • Consider the benefits of a network operations center (NOC) dedicated to monitoring and coordinating network operations—Chapter 22. • Think about your organization and whom you need to hire, and provide service statistics showing open and resolved problems—Chapter 30. • Monitor services for both capacity and availability so that you can pre- dict when to scale them—Chapter 22. • Be ready for an influx of new computers, employees, and SAs—See Sections 1.23, 1.24, and Going Global • Design your wide area network (WAN) architecture—Chapter 7. • Follow three cardinal rules: scale, scale, and scale. • Standardize server times on Greenwich Mean Time (GMT) to maximize log analysis capabilities. • Make sure that your helpdesk really is 24/7. Look at ways to leverage SAs in other time zones—Chapter 13. • Architect services to take account of long-distance links—usually lower bandwidth and less reliable—Chapter 5. • Qualify applications for use over high-latency links—Section 5.1.2. • Ensure that your security and permissions structures are still adequate under global operations.1.4 Replacing Services • Be conscious of the process—Chapter 18. • Factor in both network dependencies and service dependencies in tran- sition planning. • Manage your Dynamic Host Configuration Protocol (DHCP) lease times to aid the transition—Section
1.6 Moving to/Opening a New Building 5 • Don’t hard-code server names into configurations, instead, hard-code aliases that move with the service—Section 5.1.6. • Manage your DNS time-to-live (TTL) values to switch to new servers— Section Moving a Data Center • Schedule windows unless everything is fully redundant and you can move first half of a redundant pair and then the other—Chapter 20. • Make sure that the new data center is properly designed for both current use and future expansion—Chapter 6. • Back up every file system of any machine before it is moved. • Perform a fire drill on your data backup system—Section 26.2.1. • Develop test cases before you move, and test, test, test everything after the move is complete—Chapter 18. • Label every cable before it is disconnected—Section 6.1.7. • Establish minimal services—redundant hardware—at a new location with new equipment. • Test the new environment—networking, power, uninterruptable power supply (UPS), heating, ventilation, air conditioning (HVAC), and so on—before the move begins—Chapter 6, especially Section 6.1.4. • Identify a small group of customers to test business operations with the newly moved minimal services, then test sample scenarios before moving everything else. • Run cooling for 48–72 hours, and then replace all filters before occu- pying the space. • Perform a dress rehearsal—Section Moving to/Opening a New Building • Four weeks or more in advance, get access to the new space to build the infrastructure. • Use radios or walkie-talkies for communicating inside the building— Chapter 6 and Section
6 Chapter 1 What to Do When . . . • Use a personal digital assistant (PDA) or nonelectronic organizer— Section 32.1.2. • Order WAN and Internet service provider (ISP) network connections 2–3 months in advance. • Communicate to the powers that be that WAN and ISP connections will take months to order and must be done soon. • Prewire the offices with network jacks during, not after, construction— Section 7.1.4. • Work with a moving company that can help plan the move. • Designate one person to keep and maintain a master list of everyone who is moving and his or her new office number, cubicle designation, or other location. • Pick a day on which to freeze the master list. Give copies of the frozen list to the moving company, use the list for printing labels, and so on. If someone’s location is to be changed after this date, don’t try to chase down and update all the list copies that have been distributed. Move the person as the master list dictates, and schedule a second move for that person after the main move. • Give each person a sheet of 12 labels preprinted with his or her name and new location for labeling boxes, bags, and personal computer (PC). (If you don’t want to do this, at least give people specific instructions as to what to write on each box so it reaches the right destination.) • Give each person a plastic bag big enough for all the PC cables. Technical people can decable and reconnect their PCs on arrival; technicians can do so for nontechnical people. • Always order more boxes than you think you’ll be moving. • Don’t use cardboard boxes; instead, use plastic crates that can be reused.1.7 Handling a High Rate of Office Moves • Work with facilities to allocate only one move day each week. Develop a routine around this schedule. • Establish a procedure and a form that will get you all the information you need about each person’s equipment, number of network and tele- phone connections, and special needs. Have SAs check out nonstandard equipment in advance and make notes.
1.8 Assessing a Site (Due Diligence) 7 • Connect and test network connections ahead of time. • Have customers power down their machines before the move and put all cables, mice, keyboards, and other bits that might get lost into a marked box. • Brainstorm all the ways that some of the work can be done by the people moving. Be careful to assess their skill level; maybe certain people shouldn’t do anything themselves. • Have a moving company move the equipment, and have a designated SA move team do the unpacking, reconnecting, and testing. Take care in selecting the moving company. • Train the helpdesk to check with customers who report problems to see whether they have just moved and didn’t have the problem before the move; then pass those requests to the move team rather than the usual escalation path. • Formalize the process, limiting it to one day a week, doing the prep work, and having a move team makes it go more smoothly with less downtime for the customers and fewer move-related problems for the SAs to check out.1.8 Assessing a Site (Due Diligence) • Use the chapters and subheadings in this book to create a preliminary list of areas to investigate, taking the items in the Basics section as a rough baseline for a well-run site. • Reassure existing SA staff and management that you are here not to pass judgment but to discover how this site works, in order to understand its similarities to and differences from sites with which you are already familiar. This is key in both consulting assignments and in potential acquisition due-diligence assessments. • Have a private document repository, such as a wiki, for your team. The amount of information you will collect will overwhelm your ability to remember it: document, document, document. • Create or request physical-equipment lists of workstations and servers, as well as network diagrams and service workflows. The goal is to gen- erate multiple views of the infrastructure. • Review domains of authentication, and pay attention to compartmen- talization and security of information.
8 Chapter 1 What to Do When . . . • Analyze the ticket-system statistics by opened-to-close ratios month to month. Watch for a growing gap between total opened and closed tickets, indicating an overloaded staff or an infrastructure system with chronic difficulties.1.9 Dealing with Mergers and Acquisitions • If mergers and acquisitions will be frequent, make arrangements to get information as early as possible, even if this means that designated peo- ple will have information that prevents them from being able to trade stock for certain windows of time. • Some mergers require instant connectivity to the new business unit. Others are forbidden from having full connectivity for a month or so until certain papers are signed. In the first case, set expectations that this will not be possible without some prior warning (see previous item). In the latter case, you have some breathing room, but act quickly! • If you are the chief executive officer (CEO), you should involve your chief information officer (CIO) before the merger is even announced. • If you are an SA, try to find out who at the other company has the authority to make the big decisions. • Establish clear, final decision processes. • Have one designated go-to lead per company. • Start a dialogue with the SAs at the other company. Understand their support structure, service levels, network architecture, security model, and policies. Determine what the new model is going to look like. • Have at least one initial face-to-face meeting with the SAs at the other company. It’s easier to get angry at someone you haven’t met. • Move on to technical details. Are there namespace conflicts? If so, determine how are you going to resolve them—Chapter 8. • Adopt the best processes of the two companies; don’t blindly select the processes of the bigger company. • Be sensitive to cultural differences between the two groups. Diverse opinions can be a good thing if people can learn to respect one another— Sections and 35.1.5. • Make sure that both SA teams have a high-level overview diagram of both networks, as well as a detailed map of each site’s local area network (LAN)—Chapter 7.
1.10 Coping with Frequent Machine Crashes 9 • Determine what the new network architecture should look like— Chapter 7. How will the two networks be connected? Are some remote offices likely to merge? What does the new security model or security perimeter look like?—Chapter 11. • Ask senior management about corporate-identity issues, such as account names, email address format, and domain name. Do the corporate iden- tities need to merge or stay separate? What implications does this have on the email infrastructure and Internet-facing services? • Learn whether any customers or business partners of either company will be sensitive to the merger and/or want their intellectual property protected from the other company—Chapter 7. • Compare the security policies, mentioned in Chapter 11—looking in particular for differences in privacy policy, security policy, and how they interconnect with business partners. • Check router tables of both companies, and verify that the Internet Protocol (IP) address space in use doesn’t overlap. (This is particularly a problem if you both use RFC 1918 address space [Lear et al. 1994, Rekhler et al. 1996].) • Consider putting a firewall between the two companies until both have compatible security policies—Chapter 11.1.10 Coping with Frequent Machine Crashes • Establish a temporary workaround, and communicate to customers that it is temporary. • Find the real cause—Chapter 15. • Fix the real cause, not the symptoms—Chapter 16. • If the root cause is hardware, buy better hardware—Chapter 4. • If the root cause is environmental, provide a better physical environment for your hardware—Chapter 6. • Replace the system—Chapter 18. • Give your SAs better training on diagnostic tools—Chapter 15. • Get production systems back into production quickly. Don’t play diag- nostic games on production systems. That’s what labs and preannounced maintenance windows—usually weekends or late nights—are for.

  1. Download Companymove Show Planner For Mac 1.7.10
  2. Download Company Move Show Planner For Mac 1.7.1 Download
Download companymove show planner for mac 1.7.10

Download Companymove Show Planner For Mac 1.7.10

Make the most of your 365. A collection of powerful productivity apps you can use anywhere. Organize your life, expand your creativity, and protect what’s important. Learn more about Microsoft 365 apps & features. Try Microsoft 365 for free. Over 100 of the best computer programs. Download Free for PC and Mac. Award-winning video, audio, business, utility, and graphics software programs.


Download Company Move Show Planner For Mac 1.7.1 Download