Proper Care & Feeding of Your IT Expert(s) is Essential

Working with small- & budding-businesses as an IT consultant for hire inspired me to write this. Indeed, way back in 1999 when I first started out, I recall I a fellow IT expert asked me "What do you do?" I said, "I'm doing on-call sys admin for small businesses." He looked at me shocked and said, "What are you doing prostituting yourself?!". I said "What?!", not getting at all why it would be so bad.  Indeed I had no clue: I had only started this line of work because a long-term sys-admin, a friend of a friend, who had developed some great regular customers who already knew how "to behave", handed them to me (just wanted someone to replace him) since he wanted to take-off to start some Internet venture in the .com boom.  They paid $75 and $85/hr, were delighted by my work, and never even questioned how I spent my time.  So naturally I thought all customers thought that way, there were plenty of customers that way.  But later, when the friend of a friend returned and took his clients back, causing me to look for other clients, Boy!, was I in for a surprise  -- it was such a shocker I had little clue what hit me.  Why when I just keep delivering good work would I now be treated so bad?  And not trusted and respected as I had been before.  Delivering excellent work, yet they not trusting me (unable to recognize it, and even wondering if I was trying to rip them off.  Or even have me do all sorts of work then not pay, perhaps because the fix wasn't instant as they naively imagined  --suddenly that imagery of "hoaring" oneself-out started making sense.   But I still didn't well understand why the disrespect was happening, most especially when I was delivering the same great work I had done for everyone, and even at for many of these at much reduced rates. The temptation was to just label everyone else "bad clients".  And it wasn't until a good number of such "bad clients" (or say "bad experiences") later until I finally started to get a clue as to what was making them bad.  Of course, having done no serious IT hiring before, these clients were hiring their IT help much the same way one might grab whatever convenient auto mechanic or handyman whenever a wheel started squeaking.   But of course in this mindset, they couldn't be more mistaken:

Bigger organizations, with at least one full-time IT person, seem to "get it" (get this) --or at least end up doing the right thing (though it could be by accident) as once they realize they have enough for IT work for a full-time person, they generally get ONE person (not many (anyone) on-call), and so treat him or her well & with respect (as they would all full-time staff), and so all goes well.  However smaller orgs, who just call in someone when they need IT help, tend to have a nightmare here, as they only call in an IT expert when they have a problem, and just to work on that one issue, but that IT person, not knowing the whole system & directions, easily breaks something and/or steers them wrong, loosing their trust in IT experts and making them even more likely to call in someone else next time, only making the whole matter worse, in a vicious deteriorating cycle for the org and IT experts alike. 

But why must an org always have such an trusted IT expert who knows all?  Because, even with just ONE computer, today's IT is so complex, customized (even inadvertently), and changing, that

• Your org's IT is NOT like your car (which are fully standardized today) so you can just call in whatever mechanic is convenient, and all s/he needs to look at is just it's flat tire, to slap on another and get it running again.  Rather
• Your org's IT is a delicate custom spider web of daily info flow between increasingly complex devices -- grown and customized in so many subtle ways according to you & your org's history, operations, and aspirations-- so that, in at least a few areas, one trip-up breaking some forgotten threads supporting it all and this whole spider web on which your org depends may not only collapse (grind your org to a halt) but also (say if your data got exposed, erased, or compromised) could instantly become a entangled sticky ball (sending your org belly-up) that even the best spider could never undo. 

So, even if it is painful to say, working their IT advisor, I would be remiss not to say it. And indeed, as another one of my IT expert friends agree, it puts a finger on a (or the?) subtle but leading problem when trying to do IT consulting on-call for orgs not yet up to the level of a full-time IT expert.  Breaking this key down further, here the key sub-rules:

1. Always have at least 1 ongoing IT expert (more if you can afford) who already knows & manages *everything* about your IT & your IT needs (much like always having a backup of your key data (and probably even more important) 

--In other words--

1. Have an ongoing relationship with ONE IT expert (and more if you can afford it) who EACH fully knows & understands your IT & your IT needs;
2. and don’t switch IT experts unless the switchover is very carefully reasoned & carried-out, including plenty of overlap to hand-off all that inside knowledge so to properly “carry on the torch” without that “flame” (on which your entire operation depends) accidentally going out;
3. and definitely don’t hand-out your IT work to different people –a nightmare in the making everyone!; rather, stick to the same IT expert(s) –unless you’re specifically such an IT expert.

1. This, however, couldn’t be a bigger mistake (and, my apologies: I am remiss for not realizing that I needed to make a bigger point of this).
   1. In fact it would be easier to switch one’s accountant, one’s attorney, or one’s PR firm – that switch could severely hurt a business, BUT, unlike switching your IT expert, it won’t likely instantly bring your biz to dead standstill: there, the wrong mistake could suddenly shut down all operations (& income) for hours or days, and in the most severe (but still quite possible) case could suddenly compromise or destroy all the business’ data.

Rather, a business’s IT expert is very much like their accounting software (and customer management database).

1. One wisely puts notable effort to pick accounting software (and customer-management database) to fill the perceived needs (or in a few cases, is just lucky), but then sticks with it --for at least a few months, if not years-- and sees how it goes.
2. And their IT expert is just the same: he or she each knows your business’ computers and networking and telephones and other hardware, plus the software & configurations on every device (all which are no simple matter as your business grows, representing a highly-customized delicate interdependent web of services which are needed for you to operate); in addition, hopefully your IT expert architects good improvements for you, but even with the best solutions the system inevitably involves subtle intricacies needed to keep the whole thing running which, in practicality, only he or she may understand (until you have multiple closely-duplicating IT staff, which due to cost, is generally much later).
3. Consequently, just as it would with switching out your accounting software or customer management database that you depend on every day, switching out your IT consultant is a serious decision with increasingly severe risks & costs as your business grows:
   1. the increasing cost to pay the incoming consultant to learn everything (else suffer the consequences) plus
   2.  the increasing cost to pay the outgoing one to share everything (and hope that he will, and even though he’s been put out); plus,
   3. the increasing risk that switchover will still break something, especially given the severe consequences described.

Consequently,

1. Unless working under some IT expert who already knows all this,
   Every IT expert who is new to your org, before making any changes, must
   * study & fully "get" all the IT & IT goals of your org
   * which usually means s/he must read&absorb and/or make a complete "assessment" of it
   1. This could take say up to 15 minutes per unique device and per key unique program install, and could easily be much longer.  Questions to answer include: to where exactly is it connected, under what protocols does it get is data, is it set up on DHCP, what problems does it have, who uses it, and more.
   2. This time should be budgeted, even though it seems no work is getting accomplished.  Rather, much work IS getting accomplished: it is getting familiar so the best suggestions can be seen, recommended, and implemented safely & efficiently.
2. And many other consequences.

Things which alerted me to this concern. Even though it would seem in my favor (well, on the short term) not to say anything (because you now choosing me), I was a bit worried to see

1. how it seemed my client (call them BB) so quickly:
   1. replaced their prior sys administrators (and even though BB later said they were pleased with their work: or at least how quickly they seemed to set them up) and
   2. when I asked `what setup on the LAN do I need to be cautious of (since clearly someone has put some work into all this)’, BB responded effectively `Don’t worry about that [all this work here]; just rearrange everything we’ve got here if you need to’
2. And did my client notice:
   1. the AT&T phone repair expert (Jimmy) refused to run the last bit of phone cable (from their IT closet to their offices), even though he was quite capable of doing it, and could charge you great money ($30/15min) doing it, because he said “that’s your phone vendor’s domain” –not wanting to get his company (AT&T) complaints that they intruded on the territory of another IT expert. As he noticed the computerized phone switch in your closet and all the wiring (seemingly 100 incoming phone lines) and could see it was someone else’s IT territory.
      1. (Fortunately, though, I figured it out and took care of it for my client –and even though I have never before (not even remotely) have had to rewire a phone system of this complexity! (the most I’ve dealt with (and built a manual switch & patch box for) is a phone system of 10 lines.)  Woha!)

How can such casual IT hand-off cause problems?

1. For one, especially with IT having so many options & parameters, plus yearly advances, IT admins easily do things differently; but without some overlap (knowing the other person and especially their work), one wouldn't always guess what the other would do (and often all it takes just one mistake).
   1. Here are two small but real & immediate examples in my client's (BB's) case –
      the first one which I spotted in advance,
      but then one more which I didn't and so causing devices to go offline.
      1.   I wouldn’t have (quick-and-dirty) piggy-backed one network on top of another (creating multiple subnets, indeed within the same office suite) --for the problems it would (and was) increasingly causing,-- unless there was real serious justification for it (which it seems there wasn’t – just the quick-and-dirty thing to do). Yet that’s what the prior admins did. Fortunately, though, I instantly spotted this.
      2. And I definitely wouldn’t have put any device on a static IP if ever it could have been set up to use dynamic IP (DHCP). 
         
         1. (Why? --for the inflexibility it could cause (and future mysterious problems, which it DID cause in the case of the Konica-Minolta copier & HP All-In-One). Just swapping out a bad router (or a change to the IP range or assignments) could render devices mysteriously offline, plus IP conflicts. –That practice should have been over and done with 15 or more years ago! –in the days before DHCP was so well supported as it is today (indeed, as of ~1996, you could even host an internet connection and even website on a dynamic IP, and indeed the vast, vast majority today now are). Sure, a fixed IP might be simpler to get things up & running THIS INSTANT, but in the long run it quickly can (and did) break things. I certainly didn’t anticipate this.)
         Yet 2 of the 3 of my client's printers, including the most critical Konica-Minolta, were set up on Static IP! --and even though every device offered Dynamic IP! And unfortunately I didn’t spot beforehand due to the very high cost to check for it and to it simply not being done today.  Result: when fixing the first problem, it triggered two more: the two primary printers now suddenly when offline for everyone, as they were messed up even more (and the Konica was no easy one to fix).  With healthy overlap (for instance me the new admin discussing any changes with the prior admin before making them, as I had suggested to the client before doing it), the outage could have been avoided.

2. Let your IT expert(s) upgrade & fix your IT at night (when biz closed else operations are lowest or none)

3. Let your IT expert work remotely.

4. (more rules to come)

About this document

1. 20081126112953 version copied from email sent 2008.11.26 03:47

2. 20081203013442: For point (at least 1 IT expert at all times), seems to capture the essentials (what, how, & why). I only wish I could say it in a few words “bumper-sticker style”, as a client (especially a new & prospective client) doesn’t want to read a lot, possibly most especially a lecture on how to s/he should pick his/her IT expert – my first sentence in bold is my best attempt though it’s still a lot of words (about 56).