“My Kingdom for a free and simple syslog server!” – Anonymous System Administrator
So I don’t know about you, but I can’t remember how many times I have got to the point of having to troubleshoot an issue with a Sonus gateway and suddenly remembering I need a Syslog server to get logging out of the box. At this point I usually go and ask Google politely “Google, can you please point me in the direction of a free, and simple, syslog server that I can run without installing a bunch of malware and other rubbish on this nice customer’s server?” At this point Google usually responds “No, I cannot. However, here is a syslog server that requires you install SQL, IIS, and fifteen other dependent services as well as being crippled unless you pay $14.99 per month to a Russian guy name Vlad via this popup window that displays in the middle of the screen every 5 minutes. Also, here’s a Yahoo browser search bar for your trouble.”
This is not an ideal situation… So as usual, I just decided to build it myself. In doing this I sat down and thought about the things I wanted in a simple syslog server, and came up with this list:
- It needs to have no installation process, and leave no trace once removed from a server, as it will be run on customers' servers in a lot of cases.
- It needs to have a display where I can see the messages coming in in real time.
- The messages being displayed must be able to be paused and reviewed, so I can check if a specific event has happened yet.
- The messages window must be able to be cleared so that I can start fresh when trying to troubleshoot a fault.
- The syslog server needs to be able to log to file. Ideally the files should be able to be opened in Sonus LX tool so that further message debugging can be done easily.
- The syslog server needs to be able to roll the log files once they get to a specific size (so they can be emailed, etc).
- The syslog server should only keep a specific number of these log files so that the server’s hard disk does not get filled with log files.
- Both the display and log files should be able to be filtered to display only information that I want to see. For example, only show lines with a specific phone number in them, or only show me SIP messages. These filters should be independent so that you can view the filtered information on screen whilst more detailed information is getting logged to file for further review and troubleshooting later.
Based on these requirements I figured it would be very cool to write the server in Powershell, as this allows for absolutely no installation and can be run on any Windows machine you are likely to run into. How hard could it be?
<Insert training montage>
SMASH CUT:
EXT. TRAINING MONTAGE - THE STAIRS AT THE FRONT OF THE PHILADELPHIA MUSEUM OF ART- DAY
A man in a sweaty hoody runs to the top of a large set of stairs carrying a tablet based productivity device that he is furiously typing on. A large group of the town’s population is also running after him in a large pack for no apparent reason. Upon reaching the top of the steps he punches the air and launches the tablet into the sky. The tablet hits the concrete and smashes into a million pieces. He falls to the ground and screams towards the sky.
MAN
Nooooooo! I should have backed up to the cloud, the cloud I tells ya.
Okay okay, let’s cut to the chase. I did it, and now you too can syslog with me into the sunset.
Power Syslog Server
Features:
- Zero installation.
- Signed Powershell Script.
- Real time log display (Approximately 1000 lines).
- Copy the displayed text with the Copy Text button. This is useful for more in depth analysis in your favourite notepad software.
- Rolling log files based on file size and number of files to keep.
- Clear display and Pause display functions.
- Filter real-time display logging with regular expression.
- Filter logging to file with regular expression.
- Log files can be opened in Sonus LX tool for further debugging.
- Open firewall for Syslog Server port with the click of a button. If you are not seeing any syslog output in the Power Syslog Server display log then try pressing the Open Firewall button.
- Server listening port can be changed by creating a config file (PowerSyslogServerSettings.cfg) in the same directory as the script. The config file needs to have text in it in the following format "SysLogPort=514". This allows you to maintain the integrity of the code signing by not directly editing the script file.
How to configure a Sonus Gateway for Syslog Output
Sonus makes some of the most popular Lync Gateways on the market, so I have chosen to use them as an example of how to set up a device to output syslog. Power Syslog Server will work with any other UDP based syslog client as well though, so feel free to use it with other devices too.
Remote Log Servers:
Setup your device to output syslog to the server you are running Power Syslog Server on.
Setup your device to output syslog to the server you are running Power Syslog Server on.
Global Log Level: If your subsystems are set to “Default” logging level then this setting will be applied to them. This is also the level it will log for all services that are not specified in Subsystems. You will usually set this to a low value like “Error” or “Warning” to avoid log flooding.
Log Destination: The server with the Power Syslog Server running on it.
Port: 514
Protocol: UDP
Log Facility: local0
Enabled: Yes
Important Note: When you're finished debugging remember to Disable the syslog output. Otherwise the device will continue to output syslog data over the network, which can be a significant amount of unnecessary overhead for your device, network and server.
Important Note: When you're finished debugging remember to Disable the syslog output. Otherwise the device will continue to output syslog data over the network, which can be a significant amount of unnecessary overhead for your device, network and server.
Subsystems:
Then enable the Subsystems as required:
Then enable the Subsystems as required:
Subsystem: Set the specific Subsystem that you would like to have logged to the syslog output. For troubleshooting call flows and SIP messaging the “SIP Stack Service”, “Common Call Control” (for ISDN translation tables), “Call Routing Service” (for SIP translation tables), and "ISDN Protocol" (for E1 integrations) are useful subsystems to configure here.
Log Level: Set the required Log Level.
Log Destination:The Remote Log Server we created in the first step.
Debugging Log Files in LX Tool
Once you have captured your syslog files using the Power Syslog Server on the server on site you may want to do further call flow debugging using the Sonus LX tool (which can offer you decoded call flows for both SIP and ISDN calls providing your syslog contrains "ISDN Protocol" DEBUG and "SIP Stack Service" DEBUG logging).
To import the file into the LX tool, simply take one of the log files that the Power Syslog Server created and drag it into the LX tool window (or use File->Open). When you do this the LX tool will break the syslog file down into the individual call flows that were captured in the log. Here is an example:
 |
| Sonus LX Tool |
By double clicking on a call in the "Calls" tab at the bottom of the screen you can get further details on each call flow (including ISDN decoding!):
 |
| Sonus LX Tool - Call Flow |
Note: The LX Tool is a tool orginally created by NET (which was subsequently aquired by Sonus). To get a copy of the software go to the Sonus Salesforce portal and select "Software Downloads" then select "LX" from the Products list. If you don't have access to the Portal, speak to your Sonus representative to get a copy of the software.
Example Display/Log Filters
Power Syslog Server includes a cool feature that allows you to filter (using regular expressions) what lines of syslog get displayed on the screen and logged to file. The reason for allowing for having a separate Display Filter and Log Filter is to help you when troubleshooting in real time. By this I mean that you can configure a very specific Display Filter to allow you to see only the messages you want to see for a specific issue and a more general Log File Filter so you can capture more detailed logs to review later in order to pinpoint the exact cause of the issue. Below are some examples of how you can use these filters when troubleshooting issues:
Show Only SIP Messaging
When you are running SIP Stack Service logging at a DEBUG level the Sonus gateway will output all of the SIP messaging that is traversing it. This can be very useful when you need to know what error messages are being sent by the Carrier SIP network or Lync when a call fails.
Example Filter (without quote marks):“sip:”
Example Output:
192.168.0.20 <135>[2014-09-16 00:57:02,709] 287 0002
OPTIONS sip:ux1000lab.mylynclab.com SIP/2.0
FROM: <sip:2013ENTFE003.mylynclab.com:5068;transport=Tcp;ms-opaque=152721d992435f69>;epid=B3F80C5FC7;tag=fb568a1fab
TO: <sip:ux1000lab.mylynclab.com>
CSEQ: 9993 OPTIONS
CALL-ID: 87a0bbd93e7f4e33a2c87ff8bbccd3d7
MAX-FORWARDS: 70
VIA: SIP/2.0/TCP 192.168.0.96:51823;branch=z9hG4bK96df5daa
CONTACT: <sip:2013ENTFE003.mylynclab.com:5068;transport=Tcp;maddr=192.168.0.96>
CONTENT-LENGTH: 0
USER-AGENT: RTCC/5.0.0.0 MediationServer
192.168.0.20 <135>[2014-09-16 00:57:02,718] 322 0001
SIP/2.0 200 OK
Allow: INVITE, ACK, CANCEL, BYE, UPDATE, NOTIFY, OPTIONS, REFER, REGISTER
Call-ID: 87a0bbd93e7f4e33a2c87ff8bbccd3d7
Content-Length: 0
CSeq: 9993 OPTIONS
From: <sip:2013ENTFE003.mylynclab.com:5068;transport=Tcp;ms-opaque=152721d992435f69>;epid=B3F80C5FC7;tag=fb568a1fab
Server: SONUS SBC1000 3.0.2v270 Sonus SBC
Supported: replaces,update,100rel
To: <sip:ux1000lab.mylynclab.com>;tag=aedb006-3ef64
Via: SIP/2.0/TCP 192.168.0.96:51823;branch=z9hG4bK96df5daa
192.168.0.20 <135>[2014-09-16 00:57:04,827] 393 0003
OPTIONS sip:siptrunk.aapt.com.au:5060 SIP/2.0
Allow: INVITE, ACK, CANCEL, BYE, UPDATE, NOTIFY, OPTIONS, REFER, REGISTER
Call-ID: call-71280200-0000-0010-1101-0@10.237.176.6
Content-Length: 0
CSeq: 132654 OPTIONS
From: <sip:Anonymous@10.237.176.6:5060>;tag=aedb006-1
Max-Forwards: 70
Supported: replaces,update,100rel
To: <sip:Anonymous@siptrunk.aapt.com.au:5060>
User-Agent: SONUS SBC1000 3.0.2v270 Sonus SBC
Via: SIP/2.0/UDP 10.237.176.6:5060;branch=z9hG4bK-UX-0aed-b006-40c88
Show Output Relating to Transformation and Route Rules
This can be extremely useful for troubleshooting what transformation rules a call is using and what routing rule it has chosen.
Example Filter (without quote marks):“regex match|transformation|route request”
Note: You need to be logging at DEBUG level for “Common Call Control” (for ISDN translation tables) and the “Call Routing Service” (for SIP translation tables) for this to work.
Example Output:
192.168.0.20 <134>[2014-09-16 00:51:13,126] 1160 0097 com.sonus.sbc.route INFO (callrouter.cpp:2193) - Handling route request.
192.168.0.20 <135>[2014-09-16 00:51:13,127] 1163 0094 com.sonus.sbc.route DEBUG (translation.cpp:1332) - Performing OPTIONAL transformation using entry Testing Calling Party Rule (13.1(4)).
192.168.0.20 <135>[2014-09-16 00:51:13,127] 1164 0093 com.sonus.sbc.route DEBUG (translation.cpp:649) - Failed regex match of "tfCallingSubNumber" field for "^(9999113\d{2})$" (updated "^(9999113\d{2})$") with input of ""
192.168.0.20 <135>[2014-09-16 00:51:13,127] 1165 0092 com.sonus.sbc.route DEBUG (translation.cpp:1332) - Performing OPTIONAL transformation using entry 4 digit to E.164 (13.2(1)).
192.168.0.20 <135>[2014-09-16 00:51:13,127] 1166 0091 com.sonus.sbc.route DEBUG (translation.cpp:653) - Successful regex match of "tfCalledNumber" field for "^(45\d{2})$" (updated "^(45\d{2})$") with input of "4501"
192.168.0.20 <135>[2014-09-16 00:51:13,127] 1168 008f com.sonus.sbc.route DEBUG (translation.cpp:1332) - Performing OPTIONAL transformation using entry Full National to Lync (13.3(2)).
192.168.0.20 <135>[2014-09-16 00:51:13,127] 1169 008e com.sonus.sbc.route DEBUG (translation.cpp:649) - Failed regex match of "tfCalledNumber" field for "^0(3958245\d{2})$" (updated "^0(3958245\d{2})$") with input of "+61395824501"
192.168.0.20 <135>[2014-09-16 00:51:13,127] 1170 008d com.sonus.sbc.route DEBUG (translation.cpp:1332) - Performing OPTIONAL transformation using entry Local to Lync (13.4(3)).
192.168.0.20 <135>[2014-09-16 00:51:13,127] 1171 008c com.sonus.sbc.route DEBUG (translation.cpp:649) - Failed regex match of "tfCalledNumber" field for "^(958245\d{2})$" (updated "^(958245\d{2})$") with input of "+61395824501"
192.168.0.20 <134>[2014-09-16 00:51:13,127] 1172 008b com.sonus.sbc.route INFO (callrouter.cpp:2396) - Successful route request with entry Analog to Lync (5.1(3))
Show Only Syslog Lines Related to a Specific Phone Number
This can be useful if you know a users telephone number and you only want to see messages that relate to them.
Example Filter (without quote marks):“+61399995555”
The Wrap Up
So there you have it, another tool for the kit bag. I hope you like it and find it useful, I know it’s already got me out of a few close calls. If you find any bugs or have any feature requests feel free to drop me a line.