ComCap5 Data Capture Utility Introduction COMM1

Top  Next

ComCap5 is a Windows application designed to capture data received on PC serial communications ports or using network TCP and UDP streams and write it to text files.  Captured data is shown in scrolling windows, and may be printed, written to SQL database tables or echoed to other PCs using network protocols or serial ports. Captured data can have text added such as date and time, a serial number and remote IP address. Data from up to 999 serial ports and 2,000 network streams can be captured simultaneously, in separate or mixed files, with various file rotation schemes to start new files periodically. ComCap will capture to files on two separate disk drives for redundancy and will send email and SMS alerts if problems occur. ComCap is both a system tray and background service application that can be set to start automatically when windows starts, and remain unobtrusive.  When using the background service, captured data may be still be viewed as it arrives.

 

Major changes between v4 and v5

 

Single channel will accept hundreds of simultaneous remote clients with 'TCP Multi Server', simplifies set-up and operation.
Zip compression of capture logs during rotation to save disk space.
Improved data Filtering including 'Required Phrases', one of which must exist for record to be captured.
Capture alerts to different email addresses or SMS numbers for differing phrases.
Searching for phrases including wildcard characters or complex regular expressions.
Capture of XML and Json data formats, more flexible CSV formats.
Capture of HTTP protocol POST and PUT requests.
Reformat captured data, for instance from Fixed Width Columns to CSV, saving in new format.
New SMS bureau, The SMS Works.
Automatic free SSL/TLS certificate acquisition and installation from Let's Encrypt.
Improved SSL/TLS certificate support, more flexible configuration.
Capture time format can now be UTC or local time without summer time.
Hexadecimal capture data converted to ASCII.
New Line or Record Start option, as well as Line End.
Manually close a remote TCP connection if stalled.
Windows Defender Firewall support.

 

ComCap features include:

 

Capture from hundreds of serial COM ports or TCP/IP Server, TCP/IP Client and UDP/IP network protocol streams simultaneously, with suitable hardware.  Many network appliances output log information, typically using the UDP/IP 'syslog' protocol, and as telephone switches become network aware they are offering network logging instead of the serial port. A TCP/IP Multiple Server channel will capture up to 2,000 simultaneous SSL streams saved to a single log file.
 
Serial COM ports are dynamically detected as they are installed and removed from Windows, so capture will automatically start if a USB serial device is plugged in, and stop if it's unplugged.
 
Capture from serial ethernet device servers that convert serial port data to network protocols, from Brainboxes, StarTek, Lantronix, Lavalink and RE Smith for instance, easing data capture distances, Audon in the UK sells various converters.
 
ComCap supports IPv4 and IPv6 network standards, and SSL/TLS network capture and echo, to provides encryption of data sent over the internet, and to confirm the identity of the other party with X509 SSL/TLS certificates.  It includes support for the latest TLSv1.3 protocol and automatic SSL/TLS certificate acquisition and installation from Let's Encrypt.
.
Email may be captured, such as the alert emails sent by internet aware appliances, such as firewalls, security monitors, power distribution units, uninterruptible power supplies, remote sensors, transponders, etc. The emails may be written to a SQL database or used to trigger alerts.
 
GPS NMEA 0183 sentences and various vehicle tracker outputs can be parsed into simple comma separated records, allowing location information received from a serial or network connected GPS sensor to be easily saved to a SQL database.  A map window shows a GPS track to ensure everything is working.
 
GPS location data may be captured on Windows tablets and high end laptops that have a GPS location sensor built-in.
 
Captured data is optionally shown in scrolling windows as it arrives, and earlier data can be viewed as well.  Coloured tabs indicate which channels have started capturing data and which are stopped.
 
Data from multiple serial or network capture channels may be merged or consolidating, allowing all data to be displayed in a single window, written to a single log file, and added to a database using a single connection.
 
Capture files may be in separate directories for each capture port and new files may be created daily or multiple per day (at specified times), weekly (Monday), monthly, hourly, every few minutes, after an inactivity period, when a new page character is received, each new record, or a fixed file name may be used.  Multiple channels may optionally capture data into a common file, to reduce the number of separate log files.
 
When a capture file is closed and rotated for a new file, it may be optionally zip compressed to save space, moved to another directory for further processing or emailed to multiple recipients.
 
Capture file names (and optionally paths) are automatically generated, with file name format customised with date and time in various ways, numeric or alphabetic.
 
Capture data may be written to a database, such as Microsoft SQL Server or MySQL.  Data formats may be created to identify fields within each line of data, as fixed width columns, character separated columns (CSV), variable named columns, Json or XML.  ComCap examines the SQL tables or stored procedures for column names and types, and allows mapping of which field of data is written to which SQL column.  ComCap protects data that can not be immediately written to the database due to network problems, and will write it once the database becomes available again.
 
Capture data may be reformatted and saved in a different format, for instance fixed width lines of data, Json or XML may be saved as comma separated quoted columns for easier processing, or CSV could be saved as Json.
 
ComCap can echo or proxy captured data directly to serial communications ports or parallel ports to drive printers, or echo to the network using UDP/IP (syslog), TCP/IP Server or TCP/IP Client protocols.  Network echo can be used to allow capture of the same data to a maximum of five PCs simultaneously for redundancy or for remote capture with one PC near the data source echoing data to a centralised location.
 
ComCap has options to safeguard captured data, closing the log after each line to force it to write to disk, or after an inactivity period or periodically every few seconds or minutes.
 
A separate information log file is maintained showing when capture starts and stops and other ComCap events, it will log capture status hourly with the number of lines received from each port or stream and to which logs it is being written.  The information log may be sent to a remote PC using network protocols, perhaps to another copy of ComCap, to ease central monitoring of remote capture.
 
A sound file may be played when each new line of data is captured (with a minimum gap between sounds, in case of frequent data).
 
Raw data may be captured unchanged from the COM port or network stream, or the data may be cleaned up with non-printing characters removed and trailing spaces removed. Hexadecimal capture data can be converted to ASCII
 
When capture is started or stopped, command strings may be optionally transmitted and periodically repeated, perhaps to trigger a remote appliance to start or stop. Optionally, extra commands may be transmitted on demand, perhaps to configure a remote capture device.
 
Captured lines may have text added at the start or end, that can include escape sequences to add a serial number (of specified length), date and time in various formats, PC name, local or remote IP address.
 
A comment may be added manually to any capture channel, using a pop-up dialog, this is intended to assist in documenting batch captures, perhaps from laboratory instruments.
 
Continuous data streams may be selectively captured by filtering to reduce the amount of data saved, for instance from GPS locators or environmental sensors. Alerts may be sent according to filtered data. Filters may discard records with specific phrases or only save records such records (ie specific mobile IMEI or IP address), including wildcard characters or complex regular expressions.
 
Alerts for problems are presented in a pop-up window and may be sent by email, by SMS to mobile telephones, using either a GSM modem or SMS internet gateway or to a remote PC using network protocols, perhaps to another copy of ComCap, to ease central monitoring of remote capture. Alerts may also be triggered for phrases in captured data, including wildcard characters or complex regular expressions.
 
Data loss checking, to detect if other windows applications caused ComCap to possibly lose data.  An alert may be triggered if new data is not captured after a period, configurable by time and day of week, or if the PC appears to hang for a short period.
 
For redundancy, ComCap will capture logs on two different disk drives at the same time, and continue logging if one of the disk drives is lost, the drives may be across a network with remote logon details specified.  ComCap makes multiple attempts to open files, in case of conflict problems such as backup and protects data until it can be written to the capture file.  If disk space runs low, ComCap will send an alert.
 
ComCap usually runs as a Windows service that starts immediately the PC starts without needing a user to log-on.  There is also a system tray application which can configure, monitor and control the service displaying data currently being captured, but which will also capture data if the service is not used.
 
ComCap is supported on Windows Vista, 7, 8, 10, 11, Servers 2008, 2012, 2016, 2019 and 2022, both 32-bit and 64-bit editions. Note Windows XP, 2003 and 2000 are no longer supported. ComCap may be run with a higher priority than other windows applications to ensure stable data capture, although a dedicated PC is recommended for valuable information.
 

 

ComCap was originally designed to capture telephone call logging data from the serial port provided on most telephone switching systems (PABXs), typically called Call Data Records (CDR).  The saved data may then be used as input to telephone call management applications that will cost calls and produce reports on telephone usage or for security audit purposes.

 

ComCap is application non-specific and will capture any data that arrives on a serial port or using network protocols.  It has been used for a wide variety of purposes, such as logging output from test, alarm, GPS and monitoring equipment and serial printer data (but it can not process printer control characters).

 

Please note that ComCap is not designed to monitor data between modems and PC applications, it requires exclusive access to the serial port so no other application can use the port at the same time.

 

The ComCap distribution includes the 'ComGen Data Stream Generator' application which is designed to generate various types of test streams using any or all of the PC serial COM ports and potentially dozens of network streams, UDP Client, TCP Client or TCP Server, and has been used extensively for testing ComCap capturing multiple channels.  There is also a small Test Serial Ports and Hardware Events utility that may be useful in testing communication port and cabling problems.  

 

If the requirement is to capture serial data from another application on the same PC, ComCap includes a Null Modem Emulator (com0com) from http://com0com.sourceforge.net/ that installs a linked pair of virtual serial ports, instead of needing to use a physical pair of COM ports and a null modem cable.

 

Web: https://www.magsys.co.uk/comcap/

 

Help File for ComCap Release 5.4, 18th October 2024.