Archive
WideOrbit Automation & EAS
When we made our move into new facilities, we had to change our automation/playback system from NexGen to WideOrbit. At first we thought that it may be a good change. Maybe even take care of or streamlining some of the processes. The plan was to continue with an IP based facility, so we needed to make sure there was not falling back to a ton of cabling and physical GPIO for this. Well, a year+ later and we continue to fight how to reliably and consistently run EAS tests from WideOrbit.
As we saw this was turning into and issue, we setup our Sage Endec to send a RWT on Sunday mornings while we worked out the details of how WideOrbit deals with them. This has turned out to be a good thing as various attempts and inconsistencies caused scheduled tests to be missed. We would make them up, or attempt to, as soon as we found out of such events, but some got through. This begged the question why? We stuck one of our guys on it. Week after week, months gone by, there was no rhyme or reason to why some worked and some didn’t. Unfortunately we had cutbacks and the person working on this was released. The void had to be filled and I took up the challenge.
I had, with a little work with NexGen support, the Endec and NexGen working well via Ethernet. Weekly tests were scheduled and they would run. Monthly tests or alerts came in, and NexGen would drop them into the log, fire them, and resume operation. This was 7+ years ago! I believe I even wrote about it in a previous blog. I began to dig in and find out what is so different here that we cannot do the same.
The WideOrbit documents show a log of information if I wired up GPIO, and they do have the EAS widget. I began questioning operators on how they used the system. What did they do? Who did what when it worked? Who did what when it didn’t? Yes, using the widget produced different results. I decided to dig deeper and pulled out the trusty Ethernet tap and placed that in front of the Endec. Data gathering time has begun. I learned a lot of what the Endec sends and what it wants to see. I sent notes to Sage support for clarification. I now know what the heartbeat is. I know what a query from the workstation is and the Endec response. Time to see what happens when they run their tests.
My packet captures were full of fun stuff. One forgets that the Endec, and all EAS units, communicate with the CAPS servers, so all that traffic I knew had to be ignored. Pretty cool though to see the chatter, but time to filter that out. I only cared about the talk between the workstation and the Endec. There could be much, but then I noticed the Endec sending a series of strings consistently timed. The heartbeat. The Endec sends a heartbeat to tell systems the unit is there waiting to play. Included in the heartbeat is the expected delay if a RWT is run. In our case it is 14.45s. I never knew that, so something new is learned. Then I see a packet sent from the workstation and the Endec immediately replied. I learned that this means the EAS widget on the workstation was definitely connected as this was a query to the Endec, “are you there?” The Endec sends a reply code, so all good there. Why are tests not being run?
After much digging, and observations, I still did not know why the inconsistency, so I reached out to our engineering community. With a couple of responses of positive results, I needed to chat. The main thing that came up and was consistent between the two engineers was the fact that a test run via the widget was dropped into the stack. From the information I received from our operators some would drop the test into the playlist/log in advance of the test, while a few would drop it into the stack. I took this information and decided to put it to the test. I went to each of 4 stations control rooms and asked, stack or playlist? The answers were mostly playlist insertion with a couple of stack if it was convenient. Each station we dropped the test into the stack. Each time it triggered. Part one of the mystery may be solved. Instructions went out to the air staffs’ to use the stack insertion.
All this fine if you have someone in the studio to run the tests. These days stations are unattended a good portion of a day to all day. How does a system like this deal with these situations. Turns out WideOrbit really does not have a system when using Ethernet or IP control. If you are wired for GPIO they have some provisions. So, I decided to experiment and see if I can at least fire a RWT using a workflow. I configured a workflow with the instructions to connect to the Endec’s IP address and port, and send the string to run a test. To make it testable, I put it on a hotkey so I can fire it at any time. The moment of truth, I click on the hotkey. I check out the Endec. Nothing. I do it a couple more times. Nothing. I create a telnet session from the workstation to the Endec and send the command. Works like a champ. I check the WideOrbit logs and they claim the workflow ran and sent the string, but the Endec did not respond nor did the packet get captured, but he telnet did.
At this time I am at a loss on how to make a completely working IP control of the Endec using WideOrbit. In today’s time and the proliferation of unattended stations one would think this would be of high importance. When asked, WideOrbit support responds with run the Endec in automatic and let it run. I hate to say it, but when you are in a top 20 market the last thing the PD, GM, and sales manager wants to hear is an EAS test stepping all over a commercial break or song. Sure it can be done with GPIO, but to me that is an 80’s, last resort solution. Leave a comment if you have any thoughts.
Cheers!
Break-Through, Breaking, Broke, Fixed
That sums up this week. Break a few things, get good news on an old issue, and fixing stuff. That’s my life.
Cleaning up is my number one To-Do this week and I can walk into my office now. Hall cleared, office cleaned up, and Station Logs files for 2013 are made. (year, really.) As we progress I got to cleaning up my remote control situation even more and creating automated tasks with macros. This turned into a 2 day exercise as I had a chat with my pals at Burk Technology with some questions when I find out they posted updates for the ARC Plus firmware, now at 3.0.0.37, and AutoPilot 2010, officially 2.8.4. Not resisting something “new” I had to update. All went well except for 2 ARC Plus units. The configurations I saved prior to the update messed them up when loaded back in! Doh! To quote Homer. I figured that out and loaded configurations I had saved last week when I made a couple of macro changes. Lesson learned: Save that configuration as soon as you make a change! Now I can proceed with some more macro building and some JET flowchart learning. Yeah, I’m lagging on that.
The break-through is from RCS and our on-going NexGen v. Sage Endec IP control issue. WE CAPTURED AN EVENT! I received and email and RCS is working on code to install within NexGen to more closely evaluate why NexGen did not respond to the incoming alert. We await this little update to the A-serves. I ask again, anyone else out there doing IP control between NexGen and Sage? I wish I knew a bit more of the inner workings of this system. On the other hand I would not want to see that internal road map as the software in its basic form has been around for years!
Broke: AT&T. Nuff said? We’ve had a series of T1 failures. Some responded to quickly, others just lie in wait to annoy us. I would suspect our Moseley equipment, but history proofs otherwise. I do wish Moseley had a better self monitoring system, but the equipment is older and we are looking at upgrading to newer systems. Bottom line is it sucks to not have line-of-site to transmitter sites so I can use a reliable microwave STL, I want a new 11GHz system so bad with a large bandwidth! I can dream, can’t I?
As things settle down, other than AT&T, I am now going to sweep the studios and clean up some dust bunnies. I think I have a battery replacement on an UPS due, too. Love the holidays!
Cheers!
PS: Glossary for those new to this stuff:
STL = Studio Transmitter Link. Conversely TSL = Transmitter Studio Link, the opposite path. In radio we are mostly concerned with getting audio to the transmitter site. In this modern world with data, along with monitoring/metering, the return path is for this information.
11GHz = 11 Giga Hertz microwave systems, is the band that carries audio, data. “Classic” STLs are in the 950MHz band with limited bandwidth. For comparison, 802.11 WiFi routers run in the 2.4GHz band with wireless N also in the 5GHz band. Wireless mice and keyboards are also in the 2.4GHz band.
When The Economy Attacks
This was a tough week. The economy strikes again. I witnessed the layoff of 13 full-time employees and a handful of part-time employees. All good people. It seems every radio group has now seen what the economy has done, and we all must take up the slack as we move forward. Even with this tragic event, there is work to be done. In my case a lot.
This week I went live with our new Sage ENDECs. I have the air chain switching setup within the SAS router and all the handheld RC-1 controllers work. I now need to implement automation control, but that is up in limbo as we wait to see if and when we get our new system. I will have a temporary solution within the next couple of weeks.
On that note I am not impressed by the input controls of the ENDEC. There are 5 inputs to use, but you can’t use them the way you want. Sage gives you schemes to use them the way they think they should be used. I have called and expressed my concerns, but they go unheard. The main headache is their required 1 second closure for the box to work properly. Anything less and it acts different and creates a gotcha. The SAS closures are roughly 300ms. Using Scheme 1 instead of a RWT it starts a 15 minute hold timer before running the test. Not very intuitive. Why we cannot program how the inputs work is beyond me, especially since the box is network accessible. And the answer is yes, I am looking for ways to “hack” in and control the box via something other than their web GUI.
I also cleaned up a Harris/PR&E Impulse console. I had to order a handful of new switches. I must stress if coffee is spilt into a console, just tell us. I was able to clean things up on the PC board and luckily the solder pads were intact. I was suspect of the ribbon cable connector. I used Deoxit and made sure all the contacts got burnished. I sparyed air through to get any extra residue and cleaner out of the connector. The console is up and running for morning show production. Now if I could drop a little SAS Rubicon in there….
Now that this stress week is over I can attack details and do some digging. Some days I wish I could program so hacking wasn’t such a chore. At least it is fun to dig!
Have a good weekend.
Posted from WordPress on NookColor
NAB Convention Is Upon Us!
I sit here gathering up my “notes” and listening to my Blues channel on Pandora; NAB is around the corner. No way, it is here! I leave tomorrow! Wow. I have an agenda. The company has an agenda. The show has an agenda. My job is to make them all mesh. Of course it will not prevent me from doing some “vendor disruptions”.
I need to visit vendors regarding remote controls. Burk, Audemat, and Davicom come to mind. Any other suggestions? I’m familiar with Burk. I like the concept and white paper by Audemat, though some reports from the local boys across town will curl your hair. Davicom I’ve seen via web meeting and should be usable. How far do I want to go? This could be fun.
We will attempt to crash the NexGen, so be prepared RCS. We like to dig and we want to push buttons; simultaneously. We did this to WideOrbit and they passed. Others failed in an endless loop in the past, so maybe everyone is robust compared to then. Should be fun.
I will be at the NUG (Nautel Users Group) on Sunday. Hey, that app you posted in the Waves newsletter does not work through VPN. I look forward to seeing what’s new and how things are. I also want to see the advanced AUI presentation.
To bad Verizon won’t be there. Maybe even HTC and Samsung. No VPN through hotspots? NAT issue? Come on, now. How do we run our stuff securely, leave it on an open network? Security first, people. If WiFi is available on the floor or nearby, I will be happy to show you a VPN connection w/VNC to my Nautel transmitters on an eReader, a.k.a. cheap tablet.
I will be talking with SAS (Sierra Automated Systems) regarding my system and integration with our next automation system being mandated by the company. (I’ll elaborate later, and will be a subject of review) I have a studio upgrade coming.
Tieline will get a visit. I see a new box and they did not send me one to play with. Does that mean I can break it at the show? Also I noticed an application to manage the Tieline remote gear. Well, where is it? I know I have 8 boxes I could manage. I like pushing updates. I also like to add and subtract profiles as necessary. Let’s have at it! Oh, and where is the Android app?
I will visit Sage and ask to tear about the Digital Endec. EAS and CAPS is coming and we are to upgrade this year anyways. Just a few questions and a look at the box. This is the one we have decided on.
I’ll visit with Studer as they want to meet with me regarding consoles. This is not a formal thing, but would be cool if it led somewhere. I am quite happy with my SAS Rubicons and SLs. If it leads to anything, it is design, like I know anything about that; I install and make ’em work.
It just piles up. I notice a hard time for Audemat. Hopefully we are discussing remote control and not that stupid Golden Eagle box. I had to reboot that thing 3 times yesterday just to get diversity delay readings! Ouch. The multipath at the studios isn’t much help either.
I think I will swing by Harris and tell them that my Flexstar dropped off once again without any reason. Once in 1 year is a major improvement! Of course a week after that occurred I get an email talking about their new transmitters. WT>>>> I have a “young” transmitter, I cannot get another. There is other competition out there! 😉 (See my article in the Waves newsletter….) I want to see the STL solutions provided by these guys, too. This means I will be visiting Moseley, too. I have Moseley and Intraplex.
Gosh, I just remembered I wanted to look at some audio test gear and modulation monitors that work. The ones I have are in a word, worthless. They will make good air monitors for HD, though.
I will hit the usual suspects as time permits. I have a lot to do with only 2 days on the floor. I’m a bit disappointed I may not get to see interactive and video. I want to delve into some of that and see what ideas I can come away with.
Hope to see you all at the show! Walk up and smack me if you wish to talk.
Cheers!
Tech Notes from the Field 