Tech Notes from the Field

When we made our move into new facilities, we had to change our automation/playback system from NexGen to WideOrbit. At first we thought that it may be a good change. Maybe even take care of or streamlining some of the processes. The plan was to continue with an IP based facility, so we needed to make sure there was not falling back to a ton of cabling and physical GPIO for this. Well, a year+ later and we continue to fight how to reliably and consistently run EAS tests from WideOrbit.

As we saw this was turning into and issue, we setup our Sage Endec to send a RWT on Sunday mornings while we worked out the details of how WideOrbit deals with them. This has turned out to be a good thing as various attempts and inconsistencies caused scheduled tests to be missed. We would make them up, or attempt to, as soon as we found out of such events, but some got through. This begged the question why? We stuck one of our guys on it. Week after week, months gone by, there was no rhyme or reason to why some worked and some didn’t. Unfortunately we had cutbacks and the person working on this was released. The void had to be filled and I took up the challenge.

I had, with a little work with NexGen support, the Endec and NexGen working well via Ethernet. Weekly tests were scheduled and they would run. Monthly tests or alerts came in, and NexGen would drop them into the log, fire them, and resume operation. This was 7+ years ago! I believe I even wrote about it in a previous blog. I began to dig in and find out what is so different here that we cannot do the same.

The WideOrbit documents show a log of information if I wired up GPIO, and they do have the EAS widget. I began questioning operators on how they used the system. What did they do? Who did what when it worked? Who did what when it didn’t? Yes, using the widget produced different results. I decided to dig deeper and pulled out the trusty Ethernet tap and placed that in front of the Endec. Data gathering time has begun. I learned a lot of what the Endec sends and what it wants to see. I sent notes to Sage support for clarification. I now know what the heartbeat is. I know what a query from the workstation is and the Endec response. Time to see what happens when they run their tests.

My packet captures were full of fun stuff. One forgets that the Endec, and all EAS units, communicate with the CAPS servers, so all that traffic I knew had to be ignored. Pretty cool though to see the chatter, but time to filter that out. I only cared about the talk between the workstation and the Endec. There could be much, but then I noticed the Endec sending a series of strings consistently timed. The heartbeat. The Endec sends a heartbeat to tell systems the unit is there waiting to play. Included in the heartbeat is the expected delay if a RWT is run. In our case it is 14.45s. I never knew that, so something new is learned. Then I see a packet sent from the workstation and the Endec immediately replied. I learned that this means the EAS widget on the workstation was definitely connected as this was a query to the Endec, “are you there?” The Endec sends a reply code, so all good there. Why are tests not being run?

After much digging, and observations, I still did not know why the inconsistency, so I reached out to our engineering community. With a couple of responses of positive results, I needed to chat. The main thing that came up and was consistent between the two engineers was the fact that a test run via the widget was dropped into the stack. From the information I received from our operators some would drop the test into the playlist/log in advance of the test, while a few would drop it into the stack. I took this information and decided to put it to the test. I went to each of 4 stations control rooms and asked, stack or playlist? The answers were mostly playlist insertion with a couple of stack if it was convenient. Each station we dropped the test into the stack. Each time it triggered. Part one of the mystery may be solved. Instructions went out to the air staffs’ to use the stack insertion.

All this fine if you have someone in the studio to run the tests. These days stations are unattended a good portion of a day to all day. How does a system like this deal with these situations. Turns out WideOrbit really does not have a system when using Ethernet or IP control. If you are wired for GPIO they have some provisions. So, I decided to experiment and see if I can at least fire a RWT using a workflow. I configured a workflow with the instructions to connect to the Endec’s IP address and port, and send the string to run a test. To make it testable, I put it on a hotkey so I can fire it at any time. The moment of truth, I click on the hotkey. I check out the Endec. Nothing. I do it a couple more times. Nothing. I create a telnet session from the workstation to the Endec and send the command. Works like a champ. I check the WideOrbit logs and they claim the workflow ran and sent the string, but the Endec did not respond nor did the packet get captured, but he telnet did.

At this time I am at a loss on how to make a completely working IP control of the Endec using WideOrbit. In today’s time and the proliferation of unattended stations one would think this would be of high importance. When asked, WideOrbit support responds with run the Endec in automatic and let it run. I hate to say it, but when you are in a top 20 market the last thing the PD, GM, and sales manager wants to hear is an EAS test stepping all over a commercial break or song. Sure it can be done with GPIO, but to me that is an 80’s, last resort solution. Leave a comment if you have any thoughts.

Cheers!