He’s talking about phones — again. A cautionary tale of technical outcomes and incomplete information. Strap in…
Recently my team and I completed a long-overdue VoIP project, spurred on by building a new 60,000-square-foot addition and needing something more modern than a copper-based, expensive PRI system (from 2001 maybe?) with proprietary VoIP clients grafted on later.
Because phones are the lifeblood of business, VoIP projects are always dicey and fraught with potential issues. Even more so when no one on the premises knew much about how we got here. The previous IT leader apparently held every card close to his vest and didn’t leave hardly any documentation. I think he left the care and feeding of the phone system to a consultant, who’d regularly charge us full price for phones that would be EOL in early 2025. But that’s another story. There’d been many turnovers, leaving a team whose average IT tenure was about 2 years. Many activities had been outsourced to MSPs, but no external vendor seemed to have sufficient knowledge to tell us anything.
Having done several VoIP migrations, I knew who to contact for bids starting in May/June 2023 for cloud-based VoIP. I’ll not mention names, but they all had their pluses and minuses, and most were remote/national vendors. Eventually, we settled on one we liked (yes, one I’d worked with before, who works exclusively for nonprofit organizations), and offered us a great price.
Then came the period of discovery. I dug in deep, and found that:
- Our DID block, purchased over 20 years ago, was not big enough to provide contiguous numbering for our now-almost 200 phones. Another nonprofit in town had grabbed the adjacent DIDs, meaning that somewhere back in time we were forced to obtain another different DID, and confusing to the users.
- As a medical entity, we’re addicted to faxes. We apparently had a fax server (good), but it was tossed because the docs wanted things printed out to sign, then scanned back in (bad). From a VoIP standpoint, this meant flaky old backend VoIP-to-analog cards, LOTS of expensive desktop scanners for occasional use, and no one could tell me conclusively what was still in use.
- We could export lists of numbers linked to extensions, but no one had kept the system fully up to date. Names of people gone for years made it necessary to poll and survey departments to ask: Who is person X? Who do you have now in that role? Which of you share what phones? Eventually, we found old printouts that showed how almost every user was assigned a voice and fax number in years past — that didn’t help in determining what was still in use.
Next, we sent all this data, as clear as mud, to the vendor.
We also began talking about our current VoIP phones. All were made by Yealink, a respected brand. Could we flash the firmware on these phones to keep them? Most were relatively current.
This was red flag number one with the vendor. I stated repeatedly that this would be preferable, but they didn’t seem inclined to research this question conclusively, even after a month or so of badgering. Before Thanksgiving, we learned that these phones would be expensive to unlock with the old provider and maybe not work.
Next up: find new VoIP phones.
We learned enough to know that we had two distinct classes of users: 75% were hourly employees who just needed basic phones. Some with voicemail, most not, and the ability to do basic phone functions. Call transfer, call pickup, park, etc. The remaining 25% were administrative staff and departmental leaders.
We have been a Microsoft 365 shop since early 2022, but very little integration and full use of the stack. While I originally wanted something basic like Poly, our vendor asked us to consider Microsoft Teams phones (Yealink MP-56).
“Can we use them in hybrid mode, where the hourly users have basic phone features, and the execs have the Teams phone functions?”
“Yes, definitely. They have a ‘survivability mode” that lets them be regular SIP phones.”
I have to admit that we were seduced by the Teams phone. They sent us one, and we messed around with it. The ability of salaried staff to have phones change their presence/availability for calls based on one’s Outlook calendar is pretty cool. Knowing that the future of work is hybrid, it’s quite appealing for staff not to expose their personal cell numbers and be able to answer calls from anywhere. Or Doctors can return calls after hours from their office phone, even when they’re home.
We ordered 180 of them.
Red flag number two: while the vendor talked a good game about their familiarity of using Teams phones (and their various other clients who used them), they seemed intent on hand-crafting Teams phone documentation for us. This told me they hadn’t yet produced any documentation for their other Teams phone clients.
Knowing what our users required, we took their shoddy documentation and built our own. New images, everything. Not everyone understands that client service is everything.
We’d originally planned the phone migration the week before Thanksgiving 2023. When the vendor dragged their feet on determining the capabilities of the old phones, we now had phones arriving by the carton in December. With holidays and vacations, we set the new migration date for January: the weekend before Martin Luther King Day, giving us a 3-day weekend to deal with any issues and cleanup. All new phones were unboxed, plugged in, network connectivity to a new Voice VLAN verified and assigned to their new future users.
We’d upgraded our existing POE phone switches and tested everything with our vendor. Or so we thought.
Beginning on the Friday evening of migration weekend we visited every office, swapping old phones for new. Our remote vendor was online, pushing settings and getting phones linked to the cloud backend. We were switching jacks to the new Voice VLAN, and encountering issues that caused one of our biggest stacked switches to reboot.
Separately, the vendor encountered problems getting the SIP phones to stay online. Teams phones were fine.
We each worked our side of the issue. Rather than have the whole Team standing around while he puzzled, I sent everyone home and kept them updated as our vendor worked to figure things out. He tried things, and I drew new maps of our Voice VLAN network, learning intimately how everything connected in ways I had only surmised before.
I contacted our network architect vendor, who determined that our stack management had been misconfigured. Probably while we were doing some wire cleanup and extending our fiber backbone to these phone switches.
With the wire plant corrected, our VoIP vendor kept having problems making the SIP phones behave. We had a winter ice storm arrive that weekend, so our Monday holiday became a winter office closure on Tuesday.
Let me tell you there is nothing more gut-wrenching than being unable to do anything to fix your new phone system. It’s a nasty, punch-to-the-stomach feeling, knowing that users can’t do the basics of their job because of a phone.
On Monday, we got on a conference call with the vendor, who said: “Something is keeping the SIP phones from staying registered. How do you feel about switching the entire office to Teams phones?”
“Will it work 100% in all the ways that our users require?”
“Yes, definitely.”
I knew that was a lie (red flag number three), but I chose to guarantee our 176 users a dial tone and the ability to work on Wednesday morning over getting what I wanted. With the assurance that they would continue working on a solution, we bit the bullet.
As you can surmise, even with our best efforts at training, the tight integration of a Teams phone to Microsoft 365 proved problematic. For one thing, voicemails reside in Exchange Online. No way to forward a message from user to user, except as a WAV file attachment in Outlook. In a HIPAA-constrained organization, this also meant deploying headphones to group areas, so that users in the next cubicle over couldn’t hear privacy-protected messages.
Next up, we discovered problems with Call Forwarding. Calls could be forwarded once, maybe twice, before being dropped. Call Park was similarly affected. One of my team took on this challenge and eventually found that the default firmware on the Teams phones was two versions back. The vendor hadn’t caught this, so we spent a weekend pushing updates to all phones. Fixed, but still not perfect.
As you can guess, the VoIP vendor had not made any additional progress on solving the SIP phone problem (red flag number four). In talking with leadership, we agreed this was a breach of contract and began proceedings to get them to solve our issues as originally planned or exit the contract.
No organization is static and unchanging. Their team, product, and customer service were exemplary earlier, but things changed. People get promoted to new roles or move on. As I say (probably too often), you can’t walk into the same river twice.
Since we were starting the pursuit of a Baldrige award, we had access to our sister medical clinic in OKC and learned what VoIP vendor they used. We went down to talk to them and started meetings with their vendor. They heard our concerns, and we began the discovery audit process fresh, easier this time as we had already produced everything they required. We decided that nice as it was, we’d be dumping the Teams phones side, and going to traditional SIP phones everywhere. From a support standpoint, we liked identical hardware and software, and they had iOS/Android phone apps for the exec/admin folks. We shipped 3 of our MP-56 phones to them for testing and provisioning.
As we got in the weeds provisioning these test phones, they encountered similar SIP phone registration issues. Things that worked flawlessly on their test bench would fail when the phones were in Tulsa. They tried different approaches, and still no good.
After a few days of research and head scratching, we discovered that the problem from January on was our firewall. This unit had certain features that could only be accessed via command line. Someone back before any of us were there had disabled SIP. You can’t have SIP phones without the SIP protocol. Following that change, any factory reset phone, with it’s MAC address loaded into the Yealink cloud would power up, download a configuration, reboot itself a couple of times, and be a 100% SIP phone.
As a SIP phone, voicemail stays in the phone system for easy forwarding.
As a SIP phone, we can forward a call from phone to phone, etc.
When the weekend came for the migration, it worked almost flawlessly. The vendor came over on Monday to help with group staff training, and because SIP phones work and act like phones, everything worked. Few surprises.
The fact that we have a young staff who didn’t know where to look is not surprising. There are lots of folks all over who can keep the trains running but aren’t able to repair a train engine. See The Old Engineer and the Hammer.
As an IT leader, one has to put the resources into staff training. Give them test environments to break things and tinker in a controlled environment. I’ve been offering that to my teams forever. It will be exciting next year with a new MDF, where space can be carved out for such things.
As an IT Consultant, the cautionary tale for me and my ilk is to go the extra mile to understand the systems that were put in place before or seek to replace them. We had at least 4 different vendors, each looking at their corner of the elephant, and yet no one saw everything fully. Such is the nature of hourly consulting for some. Not for me.
