ComCap v5 and v4
Release Notes |
|
ComCap Release 5.4 - October 2024 1 - Added a means to remove excessive captured data by ignoring duplicate lines. In Channel Settings, Records, new tick box 'Ignore Duplicate Lines' and 'Max Lines to Ignore' defaulting to 1 (max 9,999). During capture, ComCap checks if a newly captured line is the same as the previous line and then ignores it, unless the maximum lines to ignore is exceeded, when the line is captured and the counter reset. Duplicate checking is before any line processing, added text, etc. The total number of ignored lines is totalled and appears on the status bar and Information Log, similarly to other methods for ignoring lines. This is designed for instruments such as weigh scales or flow meters that output a continuous stream of data every second or so, even when idle. 2 - Added Text Replacement for captured data. In Channel Settings, there is a new tab 'Text Replacement' with a tick box 'Search and Replace Text in Record'. 'Search Method' is similar to Capture Filters and Capture Alerts, allowing search of a simple textual phrase, search with wildcards or a regular expression. The grid allows multiple 'Search Text' phrases to be entered, each with 'Replacement Text' if the search matches. More than one replacement may occur for a record if multiple searches are specified, but will also find any previous replacements. Text Replacement also allows found text to be replaced by a space or a blank, ie deleted. Both Search and Replacement text may include escaped characters, specifically \s for space, \_ for blank and \\ for backslash (\), similarly to 'Add Custom Text to Captured Lines'. When searching text, this also allows trailing spaces to be searched and replaced. The button bar allow loading captured text from a file or the capture window, similarly to Filters, clicking on a capture line will show any replacements that will occur. Text Replacement is processed after most record processing, but before adding Custom text to the record or a time stamp. 3 - ComCap5 and ComGen now use the latest OpenSSL 3.3 version with the latest security fixes, which is linked into the main application, avoiding separate DLL files. The latest SSL/TLS certificate root bundle is now also linked in the application so no separate file is needed. ComCap Release 5.3 - May 2024 1 - Fixed a bug when archiving capture logs on rotation to a different directory, and error was introduced when the feature to use custom sub-directory masks was added, so that the achieve directory always included an extra unwanted sub-directory that looked like a file name, although this did not effect the rotated capture log. This also effected zipping archived capture logs. 2 - If using 'Log Raw Data' for capture data and options that add extra lines like 'Add Date Daily' and 'Start/Stop in Capture Log', these lines now correctly have CRLF added. 3 - ComCap5 and ComGen now use the latest OpenSSL 3.0 version with the latest security fixes, which is linked into the main application, avoiding separate DLL files. The latest SSL/TLS certificate root bundle is now also linked in the application so no separate file is needed. ComCap Release 5.2 - July 2022 1 - The changes in this release are primarily related to SSL/TLS fixes and the validity checking of SSL/TLS certificates. There are no new features. 2 - When sending email and using TCP Client with SSL, the 'Check if SSL Certificate Revoked' options now works when using a PEM Bundle File, as well as Windows Cert Store, using OCSP. This helps checks SSL certificates are not faked. 3 - When using TCP Multi Server capture with SSL/TLS, when checking the certificate chain, OCSP is used to confirm server SSL/TLS certificates are legitimate and not revoked for security reasons, with a new certificate being ordered if necessary. OCSP Stapling is used during the handshake for efficiency. 4 - If Windows Explorer crashes and the Task Bar and System Tray are rebuild, ComCap5 and ComGen5 will now appear again. If not, pressing ALT-C will cause the tray icon to reappear. 5 - During an initial set-up, in Preferences, Network Options, it is no longer necessary to set Certificate Supplier Protocol to None to save settings. 6 - ComCap5 and ComGen now use the latest OpenSSL 3.0 version with the latest security fixes, which is linked into the main application, avoiding separate DLL files. The latest SSL/TLS certificate root bundle is included. 7 - The versions for Windows 11 and Windows Server 2022 are now displayed correctly. ComCap Release 5.1 - February 2021 1 - Capture Filters and Capture Alerts have been enhanced by adding searching for phrases including wildcard characters or complex regular expressions. The Capture Filters and Capture Alerts tabs each have a new 'Search Method' selection offering 'Simple Phrase', 'Simple Wildcard (!*)' and 'Regular Expression' options (see below). To make testing phrases easier, sample text can be loaded from the capture window by clicking the 'refresh' icon or from a file by clicking 'open', then click 'tick' or any line of sample data to see the results of searching the phrase on that line. With Capture Filters, note that 'Ignore Lines' are processed before 'Required Phrases', and then Capture Alerts after both, so a required line will not be found if already ignored for another phrase. In Capture Alerts, only Enabled phrases are tested. 2 - 'Simple Wildcard (!*)' searching is similar to file searching where ! matches any one character and * matches zero or more characters, each with multiples allowed in the same phrase. So phrase al!rm will match alarm or alxrm, and a*rm will match alarm, axxxxrm or arm. Search is case sensitive unless 'Case Insensitive Match' is ticked. 3 - 'Regular Expression' searching is more flexible but complicated to understand usually requiring several pages of help to explain, but fortunately there are numerous web sites with RegEx tutorials, such as https://regexone.com/, https://www.rexegg.com/ and https://www.regular-expressions.info/tutorial.html. Note that regular expressions reserve many symbols for commands ([]()?+.}^$|\*) and search for any of these reserved symbols needs to be escaped by preceding with backslash, ie \\ for one backslash. Wildcard is period, ^ is start of record anchor, $ is end of record anchor, \< is start is start of word, \> is end of word. So ^alarm would find that word at the start of the record only and the wildcard example al!rm above really searches for al.rm. Search is case sensitive unless 'Case Insensitive Match' is ticked. 4 - In Channel Settings, the Records tab has a new 'Line or Record Start' option defaulting to 'Anything'. If 'Multiple Tags' is selected, a list of 'Record Start Tags' may be specified similarly to 'Record End Tags'. The other options are CR, LF, CRLF or hex character. This option is processed after record end is found, causing the control character to be removed or any text before the start tag to be removed. This option may help cleanup records with unwanted data separating them. 5 - ComCap SMTP email now support OAuth2 authentication optionally used by Google Gmail and various Microsoft mail platforms like Outlook, Office365 and Live Mail. OAuth2 authentication does not use a locally saved account password which can potentially be compromised, but instead requires account login through a web browser window where account and password are specified, and internally ComCap saves a 'token' instead of the password which allow account access, usually for weeks or months. If account access has expired or the password changed, the browser window appears again. Because ComCap runs unattended, the email account login takes place when saving Common Settings. 6 - OAuth2 is set in Common Settings, Email. For Google Gmail, use SMTP server smtp.gmail.com, set Authentication to 'XOAuth2' and after clicking OK a browser window will ask for you account, the login must be the same as the User Name specified for the account. If your Google account has high security specified, only OAuth2 access is allowed. For Microsoft, OAuth2 does not seem to be required currently, but this could change in the future. Microsoft has a lot of different account types and mail servers, ComCap has been tested using personal rather than corporate accounts, using SMTP servers smtp-mail.outlook.com and smtp.office365.com, with Authentication set for 'XOAuth2'. 5 - A failure parsing Json records for a database or reformatting is now reported with the failure reason and position. 6 - Automatic SSL certificate ordering is working again without an error. 7 - SSL certificate errors or warnings they are about to expire are now displayed every six hours instead of hourly. 8 - If Windows Explorer crashes and the Task Bar and System Tray are rebuild, ComCap5 and ComGen5 will now appear again. If not, pressing ALT-C will cause the tray icon to reappear. 9 - ComCap5 and ComGen5 now use the new OpenSSL 1.1.1i version with the latest security fixes. The latest SSL/TLS certificate root bundle is included. 10 - Fixed a rare problem reading, writing and displaying blank lines in the capture log. 11 - Fixed a problem in Channel Settings sometimes trying to open SSL/TLS certificates for TCP Client channels that don't need such certificates. 12 - Fixed an internal validation error that prevented Data Format processing one or two character long records and therefore save them to a database. 13 - In ComGen5, it is now possible to specify SSL/TLS certificates with file extensions such as .PFX and .P12. ComCap Release 5.0 - February 2020 1 - In ComCap4, one TCP Server channel needs to be created to support each simultaneous remote client, usually with a few spare in case clients do not cleanly disconnect and block a channel. While adequate for a few remote clients, this is very tedious for hundreds of channels. To simplify all this, ComCap5 added a new channel type 'TCP Multi Server' which can accept hundreds of simultaneous remote clients, all capturing data to the same log file and optionally a database. 'TCP Multi Server' has been tested with 2,000 simultaneous SSL sessions, each sending one line per second. Although a large number of connections are supported, opening each new SSL connection does take a finite duration limiting the number of new connections per second. Testing seemed to show the SSL connection limit to be about 100 per second on a decent server, but this may vary significantly depending on hardware. Non-SSL connections have lower overhead, so many more per second. Most TCP clients will retry a refused connection, so should get connected when traffic is slower. 2 - Capturing from multiple remote clients to a single channel raises
issues of how to identify data from each client, and which remote clients
are connected. ComCap can currently add a sequential serial number for
each captured record, and a new sequential 'Session Id' has been added,
which is incremented for each new remote capture session, when ComCap
accepts a new incoming TCP connection or makes an outgoing TCP connection.
This Session Id is shown in the Information Log for all activities for all
TCP Multi Server remote connections, as follows (cut down a little): The above information is logged hourly is so specified in Common Settings, Log Files or when the Status button is clicked. The Session Id is also supported in Capture Settings, Logging, 'Add Custom Text to Captured Lines' using the escape \S, with a specified number of digits. It can also be added to a database by ticking 'Update Session_Id Column'. The other escape that will identify the remote client is \R for Remote IP Address. 3 - There is a new Sessions Window that allows easy viewing of remote TCP connections to TCP Multi Server channels. The main capture window right click menu has a new option 'Sessions Window' which will open a free floating resizable window listing remote sessions since capture was started. Currently sessions are identified only by remote IP address, so will not distinguish multiple connections from the same remote device. For each session, the remote IP address and port are shown, them the total lines captured, last line time, when the session started and ended (if over), session id, and amount of data captured, all the things shown in the main capture window for individual capture channels. Active remote sessions are coloured light green, closed sessions light red. There is a right click menu that allows control of individual remote sessions similarly to the main capture window, specifically Close Remote Session, Resend Start Command, Send Data, View Map Window and Log GPS Info, the last two for mapping channels only. The sessions window is automatically refreshed when a remote connections opens or closes, and also at an optional frequency for progress updates, between every 5 and 300 seconds. 4 - TCP Multi Server channels support Data Loss checking, but only cause disconnection of the remote client rather than restarting the channel. The normal 'Idle TCP Server Close Session Timeout' may also be used, but is only based on a simple timeout, rather than Data Loss which can have different timeouts at night if there is less data. Most ComCap features work with TCP Multi Server, except 'Remove Printer Control Sequences'. If Echo is used to forward data to another computer, any received data is ignored. Merging and Filtering from TCP Multi Server are not supported. 5 - When a capture log is rotated (updating completed), it may now be zipped to save disk space. In Capture Settings, Files, 'Archive Capture Log on Rotation' must be ticked and an archive directory specified, then also tick 'Zip Archive Logs' and optionally specify a password with which to encrypt the zip to prevent unauthorised access. Currently this zipping process temporarily blocks ComCap5 displaying more data, but this should be for less than a second unless the file is very large. If this delay becomes a problem, rotate capture logs more often to make them smaller. If the delay becomes a serious problem, zipping will be done using a background thread so capture continues. 6 - In Capture Settings, the Filters tab is now called Capture Filters, and includes a new option 'Required Phrases (case sensitive), any one'. This allows a list of phrases to be specified, at least one of which must be found in the record for it to be saved. This might be used for remote authentication so only records with a specific mobile IMEI or IP address are accepted, or perhaps including a month. Beware the phrase is not column specific and may be found as part of something unwanted, so the longer the better. Alert phrases have moved to a new tab Capture Alerts. 7 - In Capture Settings, a new Capture Alerts tab replaces the previous simple list of phrases to which the same alert be sent, instead allowing a different SMS number, mail address, subject and body to be specified for each phrase found. If Body is left blank, the whole line is sent as previously. To send to multiple SMS or emails, set-up duplicate phrases. All alerts still appear in the pop-up Window and sent remotely, if so configured. Beware any old alert phrases are lost with this beta and need to be set-up again in the new format with specific email addresses. 8 - In Capture Settings, General, Data Format has new Json and XML capture options. Json and XML are set-up on the Data Format tab similarly to Variable Named Columns. After a few lines of original data have been captured, the sample data should show a decoded line with names and values which can be selected in the grid to be added to database columns or output as reformatted data. Note the data format for Json and XML only handles top level objects and fields from a single record, it can not parse arrays or multiple records, nor nested objects which will be remain as Json (also for XML) objects or arrays. Also, ComCap needs to process the entire block of Json as a single record, so on the Records tab, 'Line or Record End' should be set to Multiple Tags, with the tag for Json generally being '}/n' and for XML '</lasttag>', assuming that the Json record is followed by a newline, and the XML tag name is that of the opening tag. These record end settings mean any embedded new line ends within the record are ignored so it is captured as a single line. 9 - In Capture Settings, General, when using Data Format 'Character Separated Columns (CSV)', it is now possible to set the 'Column Quoting Character' as well as the 'Column Separating or Delimiting Character'. Generally, the separator is a comma (,) and the optional quoting character a double quote (") which should appear after the separator if the column contains the separating character. If the quoting character appears in the column it needs to be escaped by doubling, ie "". But if the quoting character regularly appears in a column, then it should be changed to something else that does not appear. 10 - In Capture Settings, the General tab has a new 'Reformat Data' tick box that enables the 'Data Format' tab (similarly to Save to Database) and a new 'Reformatted Data Format' selection. These options now allow ComCap to save captured data in a different format, for instance fixed width lines of data, Json or XML may be saved as comma separated quoted columns for easier processing. Reformat options are Tab Separated, Comma Separated Quoted Variable, Variable Named, Json and XML. On the Data Format Tab, once some data has been captured in original format, Click the 'Sample Log Up' button to display a line of data, if it's CSV one format row will be created for each column found named Column 1, etc. For fixed length columns, a single format row is created and the 'Add New Row' button should be clicked to add extra rows with the required data position and data length. The Data Names for the columns only matter if the new format includes column names. Beware of editing the Data Format using the Reformatted Output, the sample data will be wrong. 11 - In Common Settings, SMS tab, added a second HTTP bureau The SMS Works at https://thesmsworks.co.uk/ for sending SMS, cheaper than Kapow at 2p to 3p each and allows the sender ID to be freely set as either a mobile number or text defaulting to 'ComCap Alert'. Once you have opened an account, generate an API Key and Secret which is a five lines of Json text that should be copied to the 'The SMS Works Json Login Lines' field instead of the login used by Kapow. Once an SMS has been sent, the number of account credits remaining will be shown in the Info Log. 12 - Various SSL improvements have been made. SSL certificates can now be specified separately for TCP Server capture and TCP Server echo, since these may be using different host names. Likewise, validation of certificates for TCP Client capture and TCP Client Echo can now be specified separately. When entering SSL certificate file names in Preferences and Capture Settings, it's now possible to specify a password if the private key is encrypted. When entering SSL server certificate, private key and optional intermediate chain are now validated to ensure the SSL server will start correctly, for instance the private key matches the certificate, the certificate is issued by an intermediate and/or by a root CA authority, and that certificates have not expired. These checks also take place when the SSL server is started and may prevent capture being started. 13 - Automatic free SSL/TLS certificate acquisition and installation from Let's Encrypt is now supported. Potentially commercial certificates can also be automatically bought and installed, but this requires account settings to be added and is not yet available. ComCap is only able to order certificates for channels available using public domain names on the open internet, not internal only servers. Again potentially ComCap can issue local certificates against a private certificate authority, but this also requires more account settings. Before issuing a certificate, Let's Encrypt will connect to a web server ComCap runs internally on port 80 of the same IP address used by the capture or echo channel, so public DNS must point to this IP address and there should not be any other web servers using it for validation will fail. The internal web server usually only runs for a few seconds during the certificate ordering process and while running ignores any requests other than from Let's Encrypt so is not a security risk. 14 - To configure SSL/TLS certificate ordering, in Common Settings, Network Options tab, select 'Certificate Supplier' as AcmeV2, 'Certificate Product' as 'Let's Encrypt 3 months', 'Certificate Challenge' as 'File - Local Web Server', 'Certificate Private Key Type' as 'RSA 2,048 bits' or 'Elliptic Curve secp256', 'Certificate Sign Digest' as SHA256 and 'Days before Expiry to Order' to 20 or 30 (Let's Encrypt certificates expire after 90 days). 'Certificate Ordering Work Directory' is where the new certificates, private keys and ordering database will be saved. If public internet access requires a proxy server, the 'Proxy URL' should be entered as http://server:port. These settings are common to certificate ordering for all ComCap channels, where domain specific information is specified. 15 - SSL/TLS certificates are only used by TCP Server capture channels, or TCP Server Echo to Remote. TCP Client and UDP do not need certificates. In Channel Setting, the Network Options tab has certificate settings for TCP Server and Multi TCP Server and HTTP Server, if SSL is enabled (in Common Settings). If a SSL Server Certificate file name is specified and exists, details will be shown half way down the tab, including the common name and expiry date. Public Certificates may be ordered from Let's Encrypt by clicking the 'Order Public Certificate Now' button, provided the correct Common Settings are configured. If the 'SSL Server Certificate' field is empty, the default certificates directory will be used. 'Certificate Domain Name' must be completed, which is the public domain pointing to the IP address configured for capture. The IP address may be a public IP range, or a local IP NAT forwarded from a public router. Before the certificate order process starts, ComCap checks the domain name is publicly available. If 'Private Key Password' is left blank, it will be automatically set to 'password' since P12/PFX certificates need a password. Once these details are completed, clicking the button will start the order progress with details appearing in the ComCap Information Log, it usually takes about 30 seconds to complete a Let's Encrypt order with no more interaction needed. Once settings are saved, the new SSL certificate will be loaded automatically. There is another button 'Create Local SSL Certificate' which will create a self signed certificate for the 'Certificate Domain Name' which is installed automatically, but there is no check this domain is publicly accessible. Finally, the check box 'SSL Certificate Automatic Public Ordering' allows Multi TCP Server channels only to re-order certificates without any manual intervention 30 days (or less) before they expire, provided the ComCap is capturing data. On the Echo tab, 'TCP Server Echo to Remote' has identical certificate and ordering settings to capture settings. 16 - When they start-up, ComCap5 and ComGen5 now check if the Windows Defender Firewall is running, that firewall rules are set-up and if not add them automatically. Note the tray versions can only add new rules if run with administrator rights (only needed once), the service version is always an administrator. Without firewall rules, ComCap5 and ComGen5 may be unable to use TCP/Server to listen for traffic. 17 - In Capture Settings, General, 'Capture Time Format' specifies how time is handled for each channel, specifically for date and time stamps and log rotation based on time. The options are 'Local Time with daylight time' (default), 'Local Time no daylight time' and 'UTC Time no daylight time'. These options may be used to match time to the format being used in the capture log, and the last two avoid summer time issues. These settings do not apply to the Information Log, which remains system format. 18 - In Capture Settings, Logging, 'Convert Hex to ASCII' specifies that captured data is being sent as hexadecimal text (only 0 to 9 and A to F) and should be converted into ASCII. No checks are made that the text really is hex, ComCap simply removes all spaces and tries to convert whatever arrives to ASCII. This processing takes place before other options so the text can be logged as Raw Data, filtered or any other features. 19 - ComGen5 now reports the SSL/TLS certificate chain on start-up, with a warning if it expires within the next 30 days. 20 - Improved the start-up of 'TCP Server Echo to Remote' logging any errors with SSL certificates. . 21 - In the main capture window and logs, any domain name specified for TCP Server are now shown as well as the IP address. 22 - If Windows Explorer crashes and the task bar disappears and is recreated, the ComCap icon is no longer lost. 23 - In Capture Settings, Records, the default minimum line length is now 2,000 characters, to avoid long lines being broken. 24 - In Common Settings, Capture Logging tab, 'Send Alert on Stop' has been changed to 'Send Alert on start and Stop' so ComCap sends an alert on start-up as well. 25 - Fixed a silly bug that caused the capture log window to clear if the line contained the letters FF. Undisplayable characters are now shown as space rather than being removed. 27 - Initial support for capture using the HTTP protocol has been implemented, for GET and PUT requests where the data is supplied URL encoded. POST requests will be added to the next release, as will more validation over page names. For HTTP capture, in Common Settings, Network tab, create a 'TCP Multi Server channel and set Service to HTTP. 28 - The right click menu in the main capture window has a new option 'Close Remote Session' for TCP channels that, after a prompt, allows closing of the remote connection, perhaps if it seems to have stalled, forcing the remote device or ComCap if TCP Client to attempt a reconnection. 29 - Fixed a capture log display issue when changing tab that caused the top line of the log to be sometimes lost. 30 - When archiving (and perhaps zipping) capture logs to a new directory on rotation, any customised capture sub-directory specified is now used as well. A customised capture sub-directory typically uses a partial date, so a new sub-directory is created monthly or daily, to avoid large numbers of files in the same directory. It is specified as part of the 'Custom Log Name Mask', ie 'yyyy-mm"\capture-"yyyymmdd".txt"' will cause a daily sub-directory to be created. 31 - The main window File menu has a new option 'View Mail Queue' that opens a new window showing any items pending in the Mail Queue, and allowing them to be cancelled if necessary. In general, email is sent within a few seconds assuming there is an internet connection, but sometimes email can get stuck in the queue and never sent. 32 - Reworked the way the capture and information log windows are refreshed so that much higher update rates are possible. Previously, only a few hundred lines per second could be displayed, now this is a few thousand per second. Viewing data at such rates is not really practical since the scrolling windows only show the last 5,000 lines (configurable), but this change means ComCap will no longer become non-responsive under heavy traffic. No data was ever lost, this merely relates to viewing live data. 33 - In Capture Settings, Data Loss, Check for Data Loss, if 'Send Alert' is ticked, it's now possible to enter a free format 'Alert Message' that will be emailed or sent as SMS, instead of the standard ComCap generated message, which may be more meaningful and useful. Currently the email address and subject are fixed.. 34 - ComCap5 and ComGen5 now use the latest OpenSSL 1.1.1d version with the latest security fixes. ComCap Release 4.20 - May 2019 1 - Support has been added for TLSv1.3, the first improvement in SSL for about 10 years, using OpenSSL 1.1.1c. 2 - In Common Settings, Common, 'SSL Client Security' now specifies the SSL security level for all TCP/IP Client (including email) to ensure that minimum security standards are enforced. The options are: SSLv3 Only If there are problems accessing servers using earlier beta versions of TLSv1.3, set the Security Level to TLSv1.2 Only. Updated SSL root certificates with more of those commonly used. 3 - In Common Settings, Common, 'SSL Server Security Level' now specifies the SSL security level for all TCP/IP Servers to ensure that minimum security standards are enforced. The options are: SSLv3 Only Ideally use TLSv1.1 minimum since TLSv1 has been deprecated by the PCI DSS council. It's no longer necessary to specify DHParams, SSL Server ECDH Key or SSL Allowed Host Names. 4 - Added support for two more GPS vehicle and personal trackers data formats: Xenun TK102/103 and WondeX/TK5000. The Xenun TK102/103 format is essentially the NMEA RMC sentance, preceded by date/time and mobile number, followed by useful stuff from other NMEA sentances like satellite count, mobile IMEI and cell station stuff. The WondeX/TK5000 format used by VT-10, VT300 and other devices is a simple format with IMEI, time, co-ordinates, speed and direction. Both these data formats are TCP/IP server only. They are also used by a Android application MyLiveTracker by Michael Skerwiderski available free from Google Store: https://play.google.com/store/apps/details?id=de.msk.mylivetracker.client.android. 5 - ComGen5 has been rewritten to support more functionality creating test data streams to exercise ComCap5. There is a new ComGen5 Settings window with the same four tab that were previously in the main window. The Network tab has a new protocol 'Multi TCP Client' which will generate the number of clients as specified in the new 'Clients' column, up to 2,000 at present. Each client session will terminate after 'Lines/Session' have been sent. To avoid starting all these sessions at once (which most servers will be unable to handle), the General tab has a new setting 'Maximum New TCP Client Sessions Per Second' to restrict that number, typically 100 per second, or less for slower servers. The tab also has 'Random Gap Between Sessions (secs)' with 'Minimum' and 'Maximum' being specified, defaulting to 5 and 60 seconds, so that repeat sessions are staggered. Any sessions that fail to connect first time will retry after 'Wait Before Retry (secs)' for 'TCP Client Retry Attempts'. 6 - ComGen5 will now create HTTP requests to test web servers. There are three new Data Type Formats, HTTP GET URL, HTTP POST URL, HTTTP PUR URL which should be used with TCP Client or Multi TCP Client. The Remote Host and Port are set in the Network grid, with the page and parameters specified as the Data Type Text, ie: HTTP GET URL and '/index.html' will get that URL, multiple URLs can be accessed by separating them with |, ie: '/page1.html|/page2.html|/page3.html'. For both POST and PUT, parameters follow the URL and ?, ie: /upload.html?Text test line from ComGen Id \? on \p at \t session \S serial \#6 with the escape dynamically processed identically to Simple Text format. ComGen5 does not support pipelining but waits for a response to each request, which is reported in the result column of the main progress grid. In Setting, General, the 'HTTP User Agent' for the requests may be specified. The request and first response will always be logged, but generally subsequent requests are not logged, see below, just counted. 7 - ComGen5 now include optional log files. The Settings, General tab has a tick box 'Write Log File' and 'Directory' which if completed cause daily log files to be created. 'Log All Connection Attempts' defaults to unticked to reduce the amount of logging with repeated connections and disconnections, so only the first and last are generally logged, but may be ticked for full logging of all attempts. 8 - ComGen5 includes the same 'SSL Client Security' and 'SSL Server Security Level' options as ComCap4 on the Network Options tab where the SSL server certificate is specified. ComCap Release 4.18 - July 2017 1 - Fixed a long term nasty logging bug when there was a problem
writing to the information log, that caused ComCap to exhaust memory and
crash. This would happen if Main and Alternate logging directories were
the same, so the second log files could not be opened and the usual
solution of using a different file name would fail since the name is the
same in both directories. The original bug is now fixed, and checks made
to ensure Main and Alternate logging directories are different. This issue
did not effect error handling Capture Logging. ComCap Release 4.17 - March 2017 1 - Fixed a bug that meant email being sent that failed all retries was not deleted from mail queue, but caused logging to go wild with errors. 2 - When starting an SSL server channel, the SSL server certificate, private key and optional intermediate chain are now validated to ensure the SSL server will start correctly, for instance the private key matches the certificate, the certificate is issued by an intermediate and/or by a root CA authority, and that certificates have not expired. Previously, only PEM certificates were supported, ComCap now also supports DER, PFX, P12, P7B, CER and CRT file formats. PFX/P12 (PKC12) is convenient because it can contain certificate, private key and intermediates so only a single file needs to be used. 3 - Fixed a validation issue introduced in ComCap 4.13 that prevented an IP address being specified to sent the Information Log remotely using UDP or TCP Client. 4 - Fixed two problems with Email channels, they no longer always listen on port 587 and listening now correctly ceases when a channel is stopped. 5 - ComCap4 and ComGen now use the new OpenSSL 1.1.0 version with the latest security fixes. The OpenSSL DLLs are now digitally signed and checked when being loaded, for improved security. 6 - If an unexpected error happens during capture, an alert is now sent and capture restarted. ComCap Release 4.16 - October 2016 1 - Fixed a problem introduced in the last release that meant
some lines of rapid data capture may have not appeared on the screen when
being captured by the service version. No data was lost and it was still
logged, this was purely a display issue. ComCap5, currently in beta,
improves the way data is displayed so is able to display data being
captured at the rate of thousands of lines per second, whereas ComCap4 is
limited to hundreds of lines. Viewing data at such rates is not really
practical since the 2 - ComCap4 and ComGen now use the new OpenSSL 1.1.0b version with the latest security fixes and features, including ChaCha20-Poly1305 encryption cipher suites, X25519 ecliptic curve for ECDH ciphers and OCB and CCM mode ciphers. 3 - Improved error messages when setting up Network channels so that trying to use a target host name instead of an IP address says this only works if Family is set to 'Any'. ComCap Release 4.15 - July 2016 1 - ComCap now supports capture files larger than 2 gigabytes in size. Previously errors were reported once this size was reached and a new file safely created without loss of data. 2 - In Channel Settings, Files, a new option has been added 'Max File Size (MBytes)' defaulting to 2,000 (which is 2 gigabytes) that restricts the size of capture files to a larger or smaller size, after which they will be rotated with a new file name. A similar new option in Common Settings, Log Files, specifies the maximum size for Common Capture Logs. 3 - In Common Settings, Capture Logging, a new option 'Format for Display of Large Numbers' has been added, which may be set to Bytes or KBytes/MBytes. This causes file size, lines captured, data rows written, etc, to be displayed and logged in KBytes or MBytes once they exceed 99,999, for easier reading. Fixed a problem displaying the size of captured data once it exceeded 2 gigabytes. 4 - In Channel Settings, General, a new option has been added 'Idle TCP Server Close Session Timeout, Zero None (secs)' which allows a TCP Server session to be closed if no data is received. Generally a remote TCP Client will reconnect when it has more data to send. This timeout is primarily for error conditions where a session remotely fails without a clean close down happening, so TCP Server waits for ever for new data, unless Data Loss checking is used to restart capture which is more complex. The timeout should vary depending on how frequently data is expected, and may be up to 99,999 seconds (69 days). This new setting also applies to Echo to TCP Server. 5 - Fixed a bug that prevented the tray version of ComCap automatically running when Windows 10 was booted, if ComCap had been given elevated access rights (run as administrator) in order to configure or control the ComCap Background Service. It will be necessary to untick 'Auto Run Tray Application', restart ComCap, then retick it to ensure the registry is updated correctly. Also, ComCap will now automatically restart if the ComCap Background Service setting is changed, rather than needing to be restarted manually. 6 - Fixed a bug introduced in the last release that prevented serial capture channels automatically restarting if a disconnected USB serial ports became available again. 7 - In Common Settings, Network, error messages relating to invalid IP address entry now show the Capture Name, to ease finding the failing row. 8 - When sending alert and capture file emails, the SSL server certificate is now checked, if so specified in Common Settings, Common. 9 - When starting TCP Server channels with SSL support, details of the SSL certificate specified are now logged and an alert sent if the certificate has expired or is about to expire. 10 - Fixed a problem that prevented ComCap being used after the year 2030. 11 - In Capture Settings, Logging, 'Add Escaped Text' is documented as \r adding a Remote IP Address but this accidentally got changed to address and port in Release 4.13 when IPv6 was added. So this release adds \R which adds the remote IP Address alone while the documentation has been corrected to show \r adds address:port. Likewise \I now adds Local IP Address only and \i adds address:port. 12 - Cleaned up some SSL issues with missing certificates. The OpenSSL DLLs will now only be loaded from the ComCap main directory to avoid conflict with older OpenSSL versions installed elsewhere. Updated OpenSSL libraries to the latest versions with all the latest security fixes ComCap Release 4.14 - September 2015 1 - This is a minor release of ComCap, with a few new features and bug fixes, including better GPS support. ComCap has been tested on Windows 10 for several months, with no unexpected issues, except that only this release will correctly report that it is running on Windows 10. Earlier versions will almost certainly work on Windows 10, but will report they are running Windows 8, so an upgrade for Windows 10 is not essential. 2 - In Windows 10, it is not possible to have desktop start-up programs with administrator rights, they simply don't run when you logon on (presumably a new anti-malware measure). But the ComCap tray application needs administrator rights in order to start and stop the ComCap Service. Normally, the ComCap Service will start automatically when Windows boots, and the ComCap Tray application can still monitor the service, change settings and pause/resume single channels, but it can not stop the ComCap Service or Start it again without being given administrator rights through Properties, Compatibility, 'Run this program as an administrator'. So the ComCap Service will need to be controlled by the Services Windows Administrator Tool, if you want the ComCap Tray to start automatically. Workarounds for this are still being investigated. Note that some Common Settings changes require the ComCap service to be restarted, for instance adding or removing channels, and unless this is done separately using the Services tool the ComCap tray may get very confused and show the wrong data for some channels (only a cosmetic issue). 3 - Added a feature to allow a new Capture Log file to be created on demand, rather than according to the normal settings. In the Capture Log window, select the channel, right click and click 'New Capture Log'. The Information Log should report the capture log has been rotated and show the old and new log names. The new log will have the current time, which means the Log Name Format must include a time mask, ideally including seconds otherwise this can only be used once a minute. 4 - In Capture Settings, Records, added a new 'Line or Record End' of 'CR / LF' as an alternative to CR or LF. Because ComCap normally strips non-ASCII characters one or other is normally fine, but CRLF is safer when capturing packet type data containing non-ASCII values. Also, a form feed character is lo longer treated as a line end unless 'Carriage Return' is specified. 5 - Added support for the GPS Tracker Communications Protocol GT02, that is used by Concox TR02 vehicle trackers that combine GPS, GPRS and GSM is a small 12V driven package designed for mounting in vehicles. This device is programmed by SMS messages and returns location and movement information to a TCP/IP Server. To keep mobile data cost low, it only send location during movement, although does return periodic handshakes while connected. ComCap needs one TCP/IP Server channel configured for each simultaneous device that will connect, all with the same local IP and port, then Channel Settings should have 'GPS Data Processing' ticked with input type of 'GTO2 GPS Tracker Protocol' selected, tick 'No Altitude', 'Report No Fix' and 'Report Distance Moved'. On the Records tab, set Line or Record End to CR/LF, on the Logging tab tick 'Log Raw Data'. Two new tick boxes 'Report Real Time' and 'Report Remote IP address' have been added to add two more columns to GPS capture. For GT02 GPS Tracker Protocol, the last fix is repeated every three
minutes for each heartbeat packet, to indicate the tracker is stationary
but still communicating. The packet sequence number is reported to ease
checking for lost packets. Note the GT02 reports the time a fix was taken,
not when it was transmitted. New fixes may be cached by the tracker if not
online and sent together when an internet connection is re-established, or
an old fix may be sent when a new connection is made. Fixed a problem that
meant the GPS Map was not updated for a merge channel. For the GT02
protocol, the captured columns are: 6 - Fixed some cosmetic issues when running ComCap without administrator rights. The hidden Start/Stop buttons now have a caption explaining this, and ComCap Tray will no longer try to restart the service if it stops. 7 - TCP/IP is often not a reliable protocol due to routing issues, sessions may drop expectedly because a router somewhere has been rebooted, re-cabled or many other reasons. ComCap therefore attempts to re-establish any TCP/IP Client connections that are unexpectedly terminated. In existing releases, there are two immediate attempts to reconnect, after which the number of further attempts and delay between them is defined in the grid in Common Settings, Network (zero attempts means keep trying for ever). This release adds a new tick box on the Common tab, 'TCP/IP Client, No Immediate Retry on Disconnect' which applies to all channels, capture and echo, and prevents those two immediate retries so the first retry is after 'Wait Seconds'. Some appliances may be unable to cope with an immediate reconnect. 8 - Fixed a problem with echo and SSL TCP/IP Client, that meant SSL certificate checking was still done even if specified to be ignored, which prevented a connection with a self signed certificate. 9 - Improved SSL support for DH and ECDH key exchange, which allows Forward Secrecy ciphers to be implemented. In Common Settings, Common, a new 'SSL Server DH Params' file name is specified, defaulting to comcap-dhparams1024.pem which is supplied with ComCap. In Channel Settings, Network, 'SSL Server DH Params' is also specified, with 'SSL Server ECDH Key' which should generally be set to Automatic. Note both client and server need to support these key exchanges for negotiation to work. Similar improvements are in ComGen. Updated OpenSSL libraries to the latest 1.0.2d versions with all the latest security fixes. If Forward Secrecy is important, you should create your own DH Params file using the OpenSSL tools. 10 - When SSL is configured for capture or echo, the channel descriptions now mention SSL, in case the channel name does not. 11 - When starting, ComCap now reports more information about the PC local IP addresses, including the network adaptor names. ComCap also now reports any changes to IP address configuration on the PC, such as IP addresses being added or removed, but does not yet stop any channels that will fail due to an IP address being removed. 12 - The 'Test RS232 Signals’ is now called 'Test Serial Ports and Hardware Events' and has been redesigned with a list view detailing all the properties of the installed serial RS232 ports, with greater detail about each port. Clicking on a port will attempt to start monitoring of that port. Hardware changes as USB ports are connected or removed will be shown. 13 - ComGen now uses the correct local IP address for added masked text, when there is more than one. ComCap Release 4.13 - July 2014 1 - This is a major release of ComCap, with several new important features: dynamic serial RS232 port detection, SSL/TLS encryption and certificates, IPv6 support, SMTP email capture, GPS NMEA 0183 parsing, GPS Location Sensor channel, new map window, and better support for tablets and touch screens. 2 - When ComCap was originally designed all PCs had fixed serial RS232 ports and these were detected only when ComCap started. Fixed serial ports with DB9 or DB25 connectors are now rare, with removable USB, virtual and Bluetooth serial ports common, so ComCap now dynamically detects serial ports as they come and go. The Serial Ports grid in Common Settings now shows all ports installed on the PC including those currently removed and unusable, which are typically USB serial ports that are unplugged. So capture can now be set-up and started for ports that are currently removed, and will start immediately the USB device is plugged into and becomes available to Windows. Likewise capture will be paused if a serial port disappears, and restart if it re-appears. This change also fixes a problem them caused some Bluetooth serial ports to be ignored. 3 - In Common Settings, Serial Ports, the grid now shows an extra column Hardware that displays the port Friendly Name as seen in Device Manager. If the serial port is not currently usable, 'REMOVED' precedes the name, but the port may still be configured. The Info Log reporting of serial ports has also been improved with more details about the ports, which may help identification, ie: COM1, Enabled=Y, Communications Port,
(Standard port types), Serial0 where COM1 is a motherboard port, COM3 is a removed USB Prolific port, COM11 is a USB modem, COM12 (and COM13 to COM19) are an 8-way MRI PCI card, COM4 to COM10 are the same card with older non-signed drivers so disabled. COM24 is a Bluetooth serial port, COM26 is a USB FTDI port. The VID_x strings may be used to Google search for unknown hardware device identification if looking for new drivers. Enabled=N means the port is removed, usually an unplugged USB device, but could be a removed PCI card. Port and hub may identify USB ports. Beware if a USB device is plugged into a different socket, it will often be installed as a new COM port. 4 - With the 64-bit editions of Windows 7, 8, 2008 R2, 2012, and later, Windows will no longer accept unsigned drivers for serial port expansion products such as USB, PCI or PCI Express cards. For manufacturers, getting signed drivers from Microsoft is expensive and they are usually only available for products still being manufactured and supported. Some RS232 cards and dongles used for testing ComCap under Windows XP no longer work under Windows 7 64-bit and later, but new signed drivers are now available from Microsoft Update or manufacturers sites for other newer hardware. Specific hardware still working includes: - MRI 8 way RS232 PCI card using Oxford
Semiconductor chips 5 - Added SSL/TLS support for ComCap network capture and echo. SSL/TLS provides encryption of data being sent over the internet, and also client and server to confirm the identity of the other party with X509 SSL/TLS certificates. Previously, ComCap only allowed IP addresses to be specified, but SSL/TLS certificates are mostly for domain host names, so a remote host name may now be specified for TCP/IP Client, instead of a remote IP address, provided family is set to 'Any'. Beware specifying a host name means a DNS look-up needs to happen before capture can start, which potentially can fail. Note that SSL/TLS is not supported for UDP, and that a certificate is required for TCP/IP Server to provide the private and public encryption keys. TCP/IP Client does not need a certificate, unless the server specifically want to check the identity of client, which ComCap does not yet support. 6 - In Common Settings, a new Common tab has various default SSL/TLS settings, some of which can be overridden in specific Channel Settings. Common Settings allows default SSL Server file names for X509 'SSL Server Certificate', 'SSL Server Private Key' and optionally 'SSL Certificate Authority' to be specified. 'SSL Client Verify Certificate Mode' may be set to None, PEM Bundle File or Windows Cert Store. The PEM Bundle File is a file supplied with ComCap with a few hundred root certificates: C:\ProgramData\Magenta-Systems\ComCap4\Certificates\RootCaCertsBundle.pem while Windows Certificate Store is a similar Microsoft store used by Internet Explorer and other applications. For the store only, certificate revocation can be checked, beware this requires internet access and can take several seconds, or longer. The checked certificates may be logged similarly to: telecom-tariffs:
3 SSL Certificates in the verify chain: Echo Send: 1
SSL Certificates in the verify chain: The first example is public certificate, the second is a test certificate supplied with ComCap. Note the Common Settings determine if and how remote server certificates are checked, each channel has a extra option 'Client Validate Server Certificate' which must also be ticked. 7 - In Channel Settings, Network, a new box 'SSL/TLS - Transport Layer Security' has been added. There are two tick boxes, 'SSL/TLS for Capture' and 'SSL/TLS for Echo' which specify when SSL/TLS should be used. For TCP/IP Server, the three default certificates from Common Settings will appear, but may be changed for individual certificates if necessary. For TCP/IP Client, 'Client Validate Server Certificate' should be ticked if this feature is needed, provided that 'SSL Client Verify Certificate Mode' is also specified in Common Settings. For TCP/IP Server only, SSL certificates are essential, since they control encryption as well as verification. Note that if capture and echo are both configured as TCP/IP Server, the same certificate is used for both, so the host name should be the same for both. Although an SSL certificate is generally issued to a domain host name, ComCap will be unaware of this host name, only the IP address can be specified. Note there is no automatic SSL/TLS negotiation, both ends of the connection need to support SSL/TLS (or not) for a connection to work, if one end does not support SSL/TLS, the connection will fail without any real error. ComCap now uses the latest 1.0.1h OpenSSL libraries with all the latest security fixes, it was never distributed with the earlier flawed 1.0.1 libraries. 8 - The SSL certificate files are X509 PEM files, that can be purchased from certificate authorities for a domain name that point to the IP address for the TCP/IP Server, alternatively, ComCap will create free self signed certificates. Clicking the 'Create SSL Certificate' button brings up a new window allowing either a certificate request for a certificate authority to be created, or a self signed certificate. For the latter, most of the certificate fields may be left as defaults since the certificate will not be used to identify identity, but is ordering creating a CA Request the details should be carefully entered, particularly the Common Name which is the host domain name to be secured. When 'Create' is clicking, two small text files will be created in the Certificates sub-directory, with the specified file prefix, and the fields on the Network tab updated for the new certificate files. For a CA Request, the file ending with 'certrequest.pem' should be opened in a text editor and the base64 block from '--BEGIN CERTIFICATE REQUEST--' copied to the certificate authority web site when requested. Once approved, the CA will return a file with a block '--BEGIN CERTIFICATE--' which should be specified as the certificate file. ComCap generates a second file ending with 'prvkey'pem' with '--BEGIN PRIVATE KEY--' which should not be sent anywhere. If a Self Signed Certificate is generated, ComCap creates a file with suffix 'cert.pem' with '--BEGIN CERTIFICATE--' instead of the CA request. With a CA certificate, there will be one or more certificates for the authority itself, that were used to sign the server certificate (a certificate chain or bundle) and the file containing one or more '--BEGIN CERTIFICATE--' blocks needs to be specified as 'SSL Certificate Authority' in ComCap. For an SSL connection, the information log will report 'Starting SSL Handshake' when the connection is made but won't start accepting data until 'SSL Connected OK with TLSv1, cipher AES256-SHA, 256 secret bits' is reported. If handshaking fails, the connection attempt will retry. 9 - Added IPv6 support for ComCap network capture and echo. IPv6 is the next generation of IP addresses that will increasingly become used as the present IPv4 ranges are depleted. Windows XP and later all support IPv6, and it's usually enabled as standard on recent PCs with Windows 7 and 8. In Common Settings, Common, there is a new tick box 'Support IPv6' must be set before any IPv6 addresses may be entered in the Network tab grid. A new network column 'Family' allows Any, IPv4, IPv6, Any IPv4 or Any IPv6. Any IPv4 will set the address to 0.0.0.0 while Any IPv6 sets the address to ::. Local IP Address offers a list of configured IPv4 and IPv6 addresses. If a remote host name is specified instead of an IP address, set Any and the family will be detected once the IP address has been looked up from a DNS server, if both IPv4 and IPv6 addresses are offered, the first is used. In Channel Settings, Network, the Local and Remote Echo IP Addresses or Host Name may be specified as IPv6 or IPv4, and the family will be set automatically. 10 - IPv6 addresses starting with fe80:: are LAN addresses usually automatically generated for each PC on the LAN, while addresses starting with 2 are public routable IPv6 addresses. Sometimes the fe80 addresses are shown with %xx at the end, which should be removed when entering them. For display and reporting, ComCap follows the convention of enclosing IPv6 addresses in square brackets, ie [fe80::136:4a6b:870b:e19] to distinguish them from dotted IPv4 addresses, but the brackets are never entered in fields. Each time ComCap starts, all the IPv4 and IPv6 addresses configured for the PC are reported in the information log. 11 - Added SMTP email capture, intended to capture alert emails sent by internet aware appliances, such as firewalls, security monitors, power distribution units, uninterruptible power supplies, remote sensors, transponders, etc. The emails may be written to a SQL database or used to trigger alerts. 12 - In Common Settings, Network, there is a new 'IP Protocol' of 'Email Server' for which a single local address should be selected, usually with port 25 or 587, with remote IP address left as 0.0.0.0. Internet appliances that will send email to ComCap should have their SMPT Mail Server changed to this local IP address, or set-up DNS for this address. 13 - In Channel Settings, Email, a new 'SMTP Email Server' box adds several more configuration options. 'Email Account Names Accepted by Server' is a list of email addresses for which the server will accept email, these don't need real domains so info@comcap.private is acceptable, or *@comcap.private would allow email from any address with that domain, ie xxx@comcap.private. 'Remote IP Addresses Accepted' allows restriction from which IP addresses email will be accepted. 'Server Requires Authentication' allows a single login name and password to be specified that will be required before email can be accepted, support AUTH PLAIN, LOGIN, CRAM-MD5 and CRAM-SHA1 methods. 14 - Accepted emails can be captured as multiple lines of plain text, or by ticking 'Save as Variable Named Columns' as a single record, ie each line of the email is formatted similarly to Subject="ComCap email testing" so ComCap Data Format parsing can separate the headers to be added to a SQL database. Three extra headers are always added, X-Envelope-From, X-Envelope-To and X-Originating-IP all from the SMTP envelope, in case the normal To: or From: headers are inadequate, and Date is converted to ISO format. The entire body becomes Body='xxx' with line endings replaced by \n or if 'Remove Body Line Endings' is ticked by spaces. Unfortunately ComCap can not currently process CRLF in a record, it breaks too many things, but \n can translated back to CRLF in a SQL stored procedure if necessary. Note that MIME encoded emails are automatically decoded, and only text-plain and text-html sections processed. The maximum email size that is accepted is 32,000 characters, and SQL will usually only handle a field 8,000 characters long so that is really the maximum body size. Ticking 'Show All Headers' will capture all the email headers, otherwise only From, To, Subject and Date are shown. 'Ignore Email Body' only saves the email headers and might be sufficient where the subject contains the alert information. 'One Log File Per Email' causes capture log rotation for each new email provided the file name format is suitable. 'Save Raw Email as EML File' causes each complete email to be saved separately to the capture file but in the same folder, with a unique file name, where it could be processed by another application. 'Relay Raw Email' causes the complete email to be forwarded to one or more email addresses specified as 'To Addresses' (same as emailing logs). Since email formats vary so wildly, conceptually saving them with ComCap can cause many issues. Hopefully the 'Variable Named Columns' format and other options described are a good start, but ComCap users are welcome to offer feedback on alternatives from the real life emails generated by various appliances. To demonstrate saving emails to a SQL database, a new Microsoft SQL Server table capture_email has been added to 'newdb-mssql.sql' and matching stored procedures to 'storedproc-mssql.sql'. 15 - Added GPS (global positioning satellite) support by parsing the NMEA 0183 sentences commonly generated by GPS receivers into simple comma separated records, allowing location information received from a serial or network connected GPS sensor to be easily saved to a database. For diagnostics, satellite information may also be saved to the information log on request. NMEA 0183 sentences processed are: GGA, GSA, GSV, RMC, GLL and VTG, others are ignored. In Channel Settings, General, a new box 'GPS Data Processing' should be ticked, on the new GPS tab the input and output types may be specified, also the minimum interval in seconds between reports, and the maximum interval before an old report will be repeated if nothing new, or a minimum movement in metres before a report will be made (all designed to reduce repeat data captured). A typical captured line may look like: "GPS COM20",51.38325,-0.08605,76,2,"2014-06-23T17:44:16" with the channel name, latitude and longitude in decimal degrees, altitude, distance in metres since last report, time stamp report captured as sent by remote device (might be UTC). For the UK only, degrees may be replaced by UK NGR Northings/Eastings in simple metres. Future releases may add support for parsing other data formats, including GPSD JSON and GPS Exchange XML tracks, and saving data in these formats. Most testing was with a GlobalSat BU-353-S4 USB GPS Receiver, a two inch diameter device with a roof magnet that presents as a Prolific serial port (a version with a real serial connector is also available). Also tested were a battery operated GlobalSat BT-359 Bluetooth CoPilot GPS device (but Bluetooth serial ports are not always very reliable) and NMEA 0183 streaming from a Nexus 7 Android tablet. To demonstrate saving GPS data to a SQL database, a new Microsoft SQL Server table capture_gps has been added to 'newdb-mssql.sql' and matching stored procedures to 'storedproc-mssql.sql'. 16 - Added a new GPS Location Sensor Channel to support Windows tablets and high end laptops that have a GPS location sensor built-in that supports the Windows Location Service allowing sharing location information between applications. In Common Settings, Common, tick Enable Location Sensor and enter a Capture Name. Note these settings may not be available if the service is not installed, and may give an error when started if there is no GPS sensor. Otherwise configuration is similar to GPS support detailed above, except that little diagnostic information may be available, the Asus VivoTab Note 8 tablet used for testing does not provide any satellite information. 17 - A new GPS Map window has been added for channels with GPS Data Processing enabled, that uses Google Maps to display the track taken as new GPS locations are processed, provided an internet connection is available to download the maps. This window is accessed from the right click menu for a channel and requires internet access to download the Google maps. 18 - Fixed a nasty bug when capturing to a database where, in some circumstance, capture got stuck in a loop trying to open the database and closing it again, introduced in 4.11. 19 - When the user clicks buttons and menu options to stop or pause Capture, these are now logged to make diagnostics easier. 20 - In Common Settings, a new option 'Stop PC Sleeping' may be ticked to try and stop Windows placing tablets and laptops in sleep mode, but testing shows Windows may ignore this setting, unfortunately. 21 - Fixed an initial set-up bug that meant sometimes there was no live display of successfully captured data. This happened if ComCap was initially accessed and exited without setting up any capture channels, and then only a single channel was set-up. The problem did not happen if two or more channels were configured. 22 - ComCap has been successfully tested on an Asus VivoTab Note 8in tablet running Windows 8.1. Using a stylus instead of mouse or fingers provides the precision to click all menus and buttons. 23 - ComCap now correctly displays Windows 8.1 as that version, rather than Window 8, likewise Windows 2012 R2 instead of Windows 2012. 24 - The Common and Channel Settings windows are now both resizable, both larger and smaller, larger to get more channels on the screen, if smaller scroll bars now appear to allow settings to be made on screen with small dimensions. All Windows have now been testing with 125% fonts and repositioning done where text was unreadable. When first accessed, particularly when using font sizing larger than 100%, it may be necessary to drag the bottom right corner of Windows down so all the items in the window become visible. 25 - To ease access, ComCap now supports an optional global hot key to display or hide the ComCap window, as an alternate to clicking the ComCap icon in the system tray. The exact key combination is configured in Common Settings, Capture Logging. If the PC has a touch screen, the hot key will default to ALT-C. Within ComCap, the F1 and F2 keys will now start and stop capture, as an alternate to the normal buttons. 26 - When setting up a new UDP Server channel, Line or Record End now defaults to 'Packet' which is common for that protocol. 27 - in Common Settings, after configuring serial ports, the serial mouse driver is now automatically disabled. This prevents a stream of serial data (such as GPS) being detected as a mouse during the boot process, and stopping it being captured. 28 - With TCP/IP Client channels, if the remote server is not available and ComCap is set-up to periodically attempt to reconnect, ComCap now correctly shows the channel as paused, and allows immediate resume to trigger another connect attempt, rather than waiting until for the next scheduled attempt. 29 - Fixed various bugs relating to the display and network echo of captured lines longer than about 800 characters, the data itself was always written to files safely. 30 - In Channel Settings, Records, there is a new 'Line or Record End' option 'Blank Line'. This is designed to allow multiple line non-blank records to be captured as a single long line when a blank line is reached. Specifically, Nortel telephone switches generate call data records (CDRs) comprising three lines of call data following by a blank line, and this new option allows such records to be captured as a single long line making subsequent processing such as adding to SQL much easier. Note a blank line is considered as CRLF CRLF. 31- In Channel Settings, Records, the 'Line or Record End' option 'Multiple Tags' now correctly supports escaped controls \n, \c and \l for new lines, CR and LF, previously these were removed before tags were checked. So a tag of '\n\n' would have the same effect as the new 'Blank Line' record end, '\n\n\n' would need two blank lines, etc. 32 - Fixed some problems correctly reloading settings when these were changed while capture was running. The service version no longer restarts capture if settings are changed while capture is paused. The Settings menu no longer disables entries unnecessarily. 33 - Fixed a problem introduced in 4.5 that stopped Send Data being able to Send a File from the interactive desktop version of ComCap only, this continued to work when using the service version. Note that Send File is only currently supported for TCP Client and TCP Server, not UDP or Serial. 34 - Fixed a long term problem with Echo to Windows Printer. If this option was somehow configured but Windows no Longer had any installed printers, it was not possible to access Settings to change Echo back to none. If a specific printer is removed, ComCap should revert to the default printer. 35 - When selecting a new table or stored procedure, the data format fields are now automatically reloaded, rather than needing the 'New Columns' button to be clicked. 36 -- In Capture and Common Settings, more fields which are disabled due to being unavailable or not configured are now shown greyed to make it more obvious they can not be accessed. 37 - The Test RS232 Signals and ComGen tools have been updated for the same new serial port detection capabilities as ComCap. 38 - ComGen has been updated to support
IPv6 and SSL/TLS, with similar features to ComCap for which it is the main
testing platform. On the Settings tab, a new tick box 'Enable IPv6' has
been added. On the Network tab has two new columns for 'Family' and 'SSL
Enable'. If SSL is enabled for TCP Server, the certificate files should
be specified on the Options tab, and a self signed certificate can be
created if needed. Note the same certificate will be used for all ComGen
TCP/Server channels. A remote host name may now be specified for instead
of a remote IP address, provided family is set to 'Any'. Note that TCP/IP
Client certificate is not support by ComGen, which is only a test tool.
ComCap Release 4.12 - April 20131 - Improved Capture Settings, Logging, 'Add Escaped Text' to be able to freely format dates and times, similarly to 'Time Stamp Each Line'. There is a new escape sequence \{} which allows the same time and date mask characters as supported by time stamps and capture file names to be placed between the two curly bracket. So \{hh:nn:ss.zzz} will add a time stamp as 12:59:06.720 and \{yyyymmdd"-"hhnnss} would give 20121101-124906. 2 - In Channel Settings, Logging, a new tick box 'Ignored Lines to Info Log' causes line ignored by 'Validate Line Length' or 'Ignore Lines with Phrases' to be logged to the information log instead. Note this is really for debugging, the info log might get large if a lot of lines are ignored. Lines ignored for 'too many' are not logged since this defeats the purpose. This option also logs printer control sequences removed. 3 - In Channel Settings, Records, 'Remove Printer Control Sequences' offers various means of cleaning up data captured from older computer printer output. 'HP PCL, PJL, GL/2 Escapes' removes the codes used by most Hewlett-Packard printers and many others emulating them, including fonts and raster graphics. 'Ansi/Epson Matrix Escapes' removes codes mostly used by old impact printers, note there are many permutations of these codes and only the most common codes are handled. 'Vertical Movement to Line End' converts PCL cursor movement to a line end, 'Horizontal Movement to Space' converts PCL cursor movement to a space, 'Horizontal Movement to Tab' converts PCL cursor movement to a tab. Ticking 'Ignored Lines to Info Log' causes any removed printer controls (except binary and graphics) to be logged for debugging purposes. Beware printer drivers can use massive amounts of cursor movement, even placing each word or character individually on a page, so captured data may not be as expected. 4 - In Channel Settings, Logging, a new tick box 'Remove Hi-Bit Characters' filters out any characters with the hi-bit set, above ASCII 127. This complements 'Remove Control Characters' and may help clean up corrupted modem data. 5 - In Channel Settings for Merge channels, more settings are now disabled since they are only effective on the original capture channel, before merge occurs. 6 - In Common and Channel Settings, any errors updating the ComCap configuration files are now correctly reported. Usually this will be 'access denied' due to file security modify permissions not allowing the current user to update settings. It is now also possible to cancel from this error. All settings errors now also appear in the Information Log for clarity. If the file access denied error occurs, use Windows Explorer to give modify permissions to the current to all files in the ComCap4 ProgramData directory shown in the info log window. 7 - In Common Settings, Capture Logging, if ComCap is being used by a logon without administrator privileges it is not possible to enable or disable the Background Service so the tick box is greyed out and a caption has been added to explain this. 8 - The Windows minimise button in the title bar now reliably hides ComCap, previously it worked only the first time it was used. 9 - The ComGen data generator tool is now able to send binary files as well as text files, specifically printer files containing fonts and raster graphics. It sends blocks of 100 bytes as a line. Also fixed a problem selecting file names. ComCap Release 4.11 - January 20121 - Fixed a bug introduced in release 4.10 that prevented Remote Address details being specified or changed for 'Echo Types' of TCP Client, TCP Server and UDP (Syslog). If previously set-up, these features still functioned correctly. 2 - Fixed a bug introduced in release 4.10 that stopped ComCap starting on some PCs with older operating systems that did not have recent Thawte root security certificates installed through Windows Update. 3 - Fixed a bug introduced in release 4.6 that prevented more than a single remote connection to Echo TCP/Server, when it should have allowed five remote connections. Also improved the error messages to provide a banner to the remote client when a connection is refused and log it locally as well. Fixed a bug with multiple remote connections, that caused echo to stop for all connections when just one connection dropped. ComCap Release 4.10 - October 20111 - Fixed a bug introduced in release 4.8 that meant the date and time added to a captured line with the 'Time Stamp Each Line' setting was the time the previous line was captured, or blank for the first line. This bug did not effect the date and time added by the more versatile option 'Add Custom Text to Captured Lines'. This bug unfortunately also effected database time stamps. 2 - Fixed a bug introduced in release 4.8 when changing Common Settings while capture is still running, such that disabling Background Service mode did not always stop the ComCap service first. If Channel Settings are changed while a channel is paused with the service version, the new settings are correctly used. This previously only worked if the channel was stopped or running, not paused. 3 - Fixed a database bug introduced in release 4.9 that sometimes caused an error to appear after successfully adding a database row using the 'Insert into Table' option, which then caused a second attempt to add the same row, causing further duplicate key errors. 4 - Fixed a possible database issue introduced in release 4.9 that may have caused excessive CPU use and/or large log files when repeatedly trying and failing to add bad records to a database. Retries are now slowed down with a maximum of about 1,000 attempts before a record is abandoned. Note the 'Ignore Records That Cause Error' always stopped these retries. 5 - Fixed a long term issue with the service version putting two meaningless entries in the Windows Event Log each time ComCap started. 6 - Fixed a bug with TCP Client that meant retry attempts could stop after certain unexpected socket connection errors, so capture stopped. 7 - In Capture Settings, Files, a new Log Name Format of 'Prompt on Start' has been added which allows a specific file name to be specified each time capture is started. Setting this option automatically sets 'Fixed File Name', and can only be done if ComCap is not specified for Auto Start or to run as a Background Service, since ComCap would never actually start. This feature is intended for applications where data is being captured from a single device for a specific purpose, such as a laboratory test. The 'Add Comment to Log' right menu option might also be useful to add information to the capture log. 8 - Added a new setting to improve performance when capturing high speed TCP and UDP traffic. In Capture Settings, Network, a new Network Performance box has 'Capture TCP/UDP Buffer Size (KB)' and 'Echo TCP/UDP Buffer Size (KB)', both which default to 8 which means 8KB (8,192 bytes). These buffers are where TCP/UDP temporarily saves received or sent data before ComCap is able to process it. With TCP, if data is not extracted from the buffer, the speed at which data is received will slow down, but with UDP received data is simply lost since there are no handshaking packets to confirm data needs to be delayed or resent. It should only be necessary to increase the capture buffer size if a lot of data is being received each second, maybe 16K/sec or more, or if the PC is very slow or has other CPU intensive applications so that ComCap can not get the CPU it needs. Note these new settings only appear for channels actually listening or sending data, not filter or merge channels. 9 - If using the 'Email Capture Log on Rotation' feature, there are now four new tick boxes to specify which title lines are added to the email body, to potentially keep the email simple. These four titles are 'Title and Date', 'From PC Name', 'Capture Name' and 'Log File Name'. 10 - Increased the maximum data loss checking period from 999 to 9,999 minutes, to allow for data arriving once a day (1,440 minutes), such as a new date in a telephone log. The logs now show the data loss period in hours or days, after one hour. Note that the period starts from when ComCap capture was last started, if later than the actual last line. 11 - When ComCap starts, it now checks the program's digital signing certificate to ensure the program file has not become corrupted in such a way that it might behave unpredictably, and instead gives a warning and refuses to start. If this happens, please inform Magenta Systems Ltd so the corruption cause can be identified. The Windows and ADO versions are now logged when ComCap starts, to ease problem finding. 12 - In the main Window, the 'Scroll Tabs' feature is now only automatically enabled if there are more than 100 capture channels, rather than 32 at present. This allows simpler single click tab channel selection for more channels, if the capture window is sufficiently large to display all the desired tabs. 13 - Fixed a possible issue with Windows 7 where dates may have been formatted in default USA format, when a different locale was specified during initial Windows installation and not changed subsequently. 14 - It is no longer possible to email capture logs larger than 10 megs in size. If the SMTP mail server gives an error with the email being too large, retry attempts are now skipped to avoid continual failures. Restored mail queue logging, lost in release 4.8. Serious SQL database issue with Windows 7 SP1 and Windows 2008 R2 SP1 Microsoft released service pack 1 for Windows 7 32-bit and 64-bit and Windows 2008 R2 during February 2011, and has introduced a serious SQL database issue in the Microsoft Data Access Components with all versions of ComCap prior to Release 4.9. If captured data is being saved to a database, for each written line three internal handles and some memory are no longer correctly released by a Microsoft database component, which causes a memory leak in ComCap. As more lines are written, more memory is leaked, causing a PC slow down and eventually ComCap, and perhaps the PC, will crash having run out of memory, probably once about 100,000 lines have been captured depending on PC resources. Note this database problem does not happen on the original versions of Windows 7 or Windows 2008 R2, or any other earlier versions of Windows, but was probably introduced by Microsoft hot fix KB983246 which fixed a different deadlock issue that could cause ComCap to hang, and which is available for Windows XP and later. So if this hot fix is installed, the same memory leak may be introduced. The bug has been identified and discussed in a Microsoft Blog, has been fixed in the first beta of Windows 8, and for for Windows 7 and 2008 R2 at KB2640696 with a hotfix in February 2012, which should also be included in service pack 2 in 2012 or 2013. If the hotfix is installed, when ComCap starts it will report 'Msado15.dll version 6.1.7601.17761 (win7sp1_gdr.120112-1636)' in the info log, the fault version says 'version 6.1.7601.17514'. ComCap Release 4.9 and later fix this problem by no longer using the faulty event and should also resolve possible longer term rare reports of ComCap locking up during database updates in Windows XP and 2003. Any ComCap database users with 4.8 or earlier are strong recommended to upgrade to 4.9 or later or install the KB2640696 fix. ComCap Release 4.9 - May 20111 - The unlimited version of ComCap now supports a maximum of 999 capture channels. This resolves an issue with PCs with more than 500 serial COM ports. For those wondering how any one PC can support so many COM ports when most how have just one, the answer is a rack of ethernet Serial Device Servers and hundreds of virtual COM ports. 2 - Merge Channels have been added as a way of combining or consolidating captured data from multiple Network and Serial channels. It is effectively a new form of capture data Echo. This will benefit applications capturing data from multiple sources, allowing all data to be displayed in a single window, written to a single log file, and added to a database using a single connection instead of one for each channel. One record at a time is merged, which may be one or more lines depending on the capture channel 'Line or Record End' setting (there is no record setting for merge channels). Most channel settings apply equally to capture and merge channels, but generally should not be duplicated. Some effort may be needed to avoid duplicate data being merged from different channels. With 'Add Custom Text to Captured Lines', the network or serial channel setting should be used to add a channel name, remote IP address, or device id, while the merge channel could add date and time and serial number so they are unique for the channel. Note that currently if a merge channel is paused, the capture channels continue but data is not merged. Merge Channels are set-up on the new 'Merging' tab in Common Settings, with a unique channel name, then a 'Merge From' name that is partially matched against Network or Serial Channel names, ie the partial name 'Weather' would match channel names 'Weather 1', 'Weather 2', 'Weather 3', etc. The 'Total' column shows how many channel's data will be merged. 3 - A 'New Log Interval' of Each Record/Line has been added, which will create lots of short log files each with a single line or record. This is really intended for capturing multiple line data such as remote alarm reports rather than high speed tabular type data. One disadvantage of this option is that when changing capture tabs to view previously captured data, only the data from the last file will be restored, which means just one record. 4 - A new 'Custom Log Name Mask' of \# will add the current serial number to the mask, which may be used instead of date and time for unique sequential file names. Previously custom masks had to include yy or yyyy for year, but this check has now been removed to allow more flexible file naming. Beware that sensible masks should still be used, to avoid duplicate file names. 5 - A new 'Line or Record End' of Timeout/Disconnect has been added. Previously the same effect could be achieved by specifying a special character that was never expected. This is intended for capturing multiple line data such as remote alarm reports, which arrive as a burst of data, separated by a gap from the next report or by serial lines dropping or TCP channel disconnecting. 6 - The main File menu has a new option 'Disable Alert Sound'. with a similar tick box in the Alert Window, which may be used to temporarily stop audible alerts driving you mad with repeated errors being reported. Fixed a long term problem that meant audible alerts from the service version of ComCap did not automatically repeat (if so specified). 7 - Totally rewrote the database code to avoid a serious SQL database memory leak issue introduced with Windows 7 SP1 and Windows 2008 R2 SP1, and not yet fixed by Microsoft. ComCap now performs all database processing in a new thread, to avoid using the Microsoft event that leaks memory. This has other performance benefits, since the Microsoft event started and stopped a thread for each row written, now there is one thread running continually. Separating the database code into a thread also avoids lock-up problems that have rarely been reported, since the thread is monitored and can be restarted if it stops responding after two minutes. Also, the maximum number of database rows that can be written each second will be increased, typically at least 50 rows per second (ie 20ms per row), but less if the stored procedure does a lot of processing or if ComCap is updated the database with more than a few thousand buffered rows when buffer processing slows down updates. 8 - When capturing to a database, there is a new 'Save/Restore Buffered Rows to Disk' tick box. Previously, ComCap only buffered rows that could not written to a database due to network errors in memory, and they were lost forever if ComCap was stopped before the database connection was successfully restored. With this new option, the buffered rows are saved to a file (in the same directory as the config files) when capture is stopped (but not paused) and restored to the buffer when capture is started again. The actual file name is reported in the info log file. Beware this may cause a problem if the database format is changed so the buffered rows are no longer valid, so just delete the file. 9 - The com0com serial port null modem emulation included with ComCap has been updated to version 2.2.2.0, and there is now a second signed version that may be installed on 64-bit versions of Windows. Note com0com is an open source project and has not been tested by Microsoft WHQL, so skip the security alerts that appear during 64-bit installation. Previously com0com could not be used on Windows 7 and 2008 R2 64-bit due to not having it's driver signed. ComCap Release 4.8 - January 2011 1 - It is now possible to change ComCap settings without first stopping capture. If settings are actually changed, the user is prompted to restart capture for a single channel or all channels, depending on what was changed, but this may be skipped and capture restarted later. Previously, settings tabs without any active settings were hidden until their features were enabled by various tick boxes (such as Save to Database, Check for Data Loss, etc). To make these setting more obvious, the tabs always appear but instead the options on them are disabled until needed. 2 - Email support has been improved to better ensure that alert emails are not lost due to internet problems or ComCap being stopped. A mail queue has been introduced, with extended retries potentially over 24 hours or more, so email alerts will not be lost if ComCap stops for any reason. The implication of the new mail queue is the use of temporary files to hold queued email, the root directory for which needs to be specified in Common Settings, Email. The previous simple retry attempts setting has been replaced by a comma list of 'Minutes Between Queue Attempts', ie 2,3,5,5,10,etc, which means retry attempts will take place 2, 5, 10, 15 and 25 minutes from when the email was queued, with one attempt for each of the two SMTP Servers, if both specified. Note that some email servers support grey listing and reject the first email attempt from a new sender but allow a retry 10 or 15 minutes later, something that is very effective in blocking spam emails (since they don't usually retry). Also note the email queue is only running while ComCap is running, but if ComCap is stopped it will make one attempt to send any pending emails (such as logs emailed on close, see below) within 30 seconds (but not for longer since Windows might be closing ComCap to reboot). 3 - Email support has further been improved by allowing separate authentication details to be specified for the two SMTP servers, which might be different ISPs, with new Authentication options such as NTLM, and by supporting secure email using SSL/TLS which is required by services such as Gogglemail. 'SSL/TLS Connection' should be used with SMTP Port 465 and forces an implicit TLS connection to this port. 'SSL/TLS Authentication' normally uses port 25, but will check if the server returns a STARTTLS response to indicate it supports SSL/TLS authentication at which point a secure connection is established instead. Note that the SMTP server SSL certificate is not currently checked. To support SSL/TLS, two new files are included with ComCap, libeay32.dll and ssleay32.dll. 4 - Many ComCap users further process capture logs, sometimes with difficulty due to the files being continually updated, so set short new log interval or rotations so new files are created regularly. To assist this, ComCap now has a new Capture Setting on the Files tab, 'Archive Capture Log on Rotation' which causes a closed log file to be moved to a specified archive directory on the same disk drive. This will potentially avoid conflicts since the other processing application will only find completed capture logs, and not those still open. A further option 'Archive/Email Capture Log on Stop Capture' causes the log to be archived even if the rotation time has not yet arrived when ComCap is stopped. If ComCap is restarted before the next rotation time, a new log with the previous name will be created, but when it is finally archived it will be renamed by the addition of -1, -2, etc, to the file name to avoid a conflict with previously archived log file names, if any. To FTP a capture log, use our DUN Manager application which offers various Scheduled Tasks including FTP Upload which will automatically FTP any files it finds in a specified directory and then move them elsewhere so they are not sent again. 5 - Capture logs may now be automatically emailed when closed and rotated. There is a new tab Capture Setting, Email, with 'Email Capture Log on Rotation' as 'Email as Body' or 'Email as Attachment, depending on the requirement. An 'Email Subject', 'From:' and multiple 'To': addresses may be specified. On the Files tab, 'Archive/Email Capture Log on Stop Capture' causes the log to be emailed even if the rotation time has not yet arrived when ComCap is stopped. 6 - The maximum captured line length has been increased from 2,048 to 20,000 characters, while the default length has been reduced to 256 characters. The Maximum Line Length is that beyond which a line is forcibly wrapped to the next line. There is also a Maximum Validated Line Length, which is used to ignore line which are too long, but without wrapping them. 7 - A new feature 'Ignore Too Many Lines' has been added, which may be used to reduce the amount of data captured from devices sending continuous streams, such as GPS locators or environmental sensors. A 'Line Time Gap' in fractions of a second may be specified during which any new data will be ignored. If the gap is set to 0.50 second, then only a maximum of two lines per second will be captured, or it may be one line every few seconds. The number of lines ignored are still counted and reported. This feature may be used to slow down database updates by ignoring data arriving too fast. 8 - When adding custom text to captured lines, a new escape \T has been added which gives a long time with milliseconds to three decimal places, ie 10:22:56:586. Beware Windows is not millisecond accurate, due to multi-tasking. 9 - The 'Info Messages in Capture Log' preference has been removed, due to recent testing showing potential sharing and reentrancy problems where an attempt is made to log error messages about capture logs to the capture log itself. 10 - In Capture Settings, Filters and Alerts, leading and trailing spaces are now removed from the filter phrases. ComCap Release 4.7 - August 2010 1 - Made some improvements to automatically restart the ComCap service if it stops unexpectedly due to an error or is manually stopped for some reason. When the service version is enabled through Settings, Common, the service properties now have Recovery set as 'Restart the Service' to avoid needing to set this manually. Note this change only happens if the existing service is disabled through Settings, Common, and then re-enabled. Also, provided the user is logged-on and the tray version of ComCap is monitoring the service, if the service is found to have stopped an alert will be sent and it will be automatically restarted after 20 seconds, provided the Stop button is not pressed meanwhile. An alert is now sent on start-up if ComCap is found to have previously stopped without a clean close down. Further ComCap monitoring is planned for a future release to ensure ComCap never stops. 2 - In Common Settings, Alerts, a new option 'Control Serial Port' has been added, which causes the RTS and DTR control lines to be raised on a specified RS232 serial port for one or more seconds, whenever an alert is triggered. If the serial port is wired to a low current relay, this output may be used to sound a bell or alarm to draw attention to ComCap. 3 - In Capture Settings, Filters and Alerts, there is a new option 'Ignore Same Alert for (mins)' which works in conjunction with 'Alert for Lines with Phrases' preventing the same alert being sent again until the specified period in minutes is reached. If more than one alert filter is specified, each filter will still trigger an alert the first time it is detected. This will avoid too many emails or SMS messages being sent when the same alert is continually repeated. 4 - In Capture Settings, Files, a new option 'Delete Empty Capture Log on Rotation' has been added, to avoid empty capture logs remaining on disk. Note this only applies to the main capture log, not the alternate. 5 - In Capture Settings General, there is a new tick box 'Log Only First Command Sent' that prevents repeated logging when the Start Command is sent every few seconds or minutes, filling up the info log. 6 - In Capture Settings, Database, a new option 'Escape Backslash (MySQL)' has been added to avoid a problem with old versions of MySQL that treat the backslash character as the first of an escape sequence (ie \f is form feed). This option sends \ as \\ so MySQL saves it as \ instead of reporting a syntax error. 7 - In Capture Settings, Database, a new option 'No Pause for Full Buffer (Ignore Data)' has been added so that capture to log files will always continue even if a database is unavailable, or if the capture rate is so fast that a database can not keep up. This works in conjunction with 'Maximum Rows to Buffer before Pausing Capture' so that once the limit is reached subsequent rows are ignored instead of being buffered, but are still written to the capture log file. This feature is primarily designed to support capture applications that regularly update the same information, such as global positioning satellite data where a vehicle position need not be recorded every second. The number of rows not written to the database is logged similarly to the following: Database Rows Added: 3,990, Database Errors 16, Skipped 113 and also reported in the tray application in brackets after the number of DB rows. Note the minimum number of rows that may be buffered is 50, which is required for normal operation where 10 or more rows may be captured as a burst which may be faster than they can be written to SQL. 8 - In Capture Settings, Database, a new option 'Ignore Records That Cause Error' has been added which prevents a row being buffered when a SQL error occurs and the database is closed and re-opened to try and clear the error. This is primarily intended to overcome syntax and duplicate key errors where the database can not write a specific row of data and will get stuck in a repeating loop trying to write the same row again and again. Unfortunately, due to the widely varying error responses from different SQL databases, this feature might cause a row to be lost if the network is lost or the SQL server simply stopped while a row is being written. 9 - Fixed a bug introduced in Release 4.6 for serial capture only that meant a Start Command was only send once and not repeated, instead giving an error each time. 10 - When ComCap starts, the PC local IP addresses are now listed in the info log. When the network configuration is loaded, and capture started, a warning is now given if a configured local IP no longer exists on the PC, due to networking having being changed. Currently, capture will still attempt to start, since other channels may still work, but those channels trying to use a non-existent IP address will give network errors. This mainly effects TCP and UDP server channels set-up to listen on a single IP, rather than 0.0.0.0 for all IP addresses on the PC. On the other hand, if the PC IP has changed perhaps due to DCHP being configured, data directed at the original IP address will not arrive anyway. ComCap Release 4.6 - November 2009 1 - Fixed a long term problem with TCP Server where multiple channels are being captured from the same IP and port, which caused a problem restarting capture after it was paused (usually for database problems). This was caused by only one channel being paused instead of all related channels, but a workaround has been found so capture now resumes correctly without an 'Address already in use' error. 2 - Channels configured for TCP Server now have tabs coloured blue rather than red when listening for incoming connections, changing to green when a connection is made. This also fixes a problem with the service version that meant the channel pop-up menu assumed the channel was stopped and allowed settings to be changed and capture resumed, rather than being paused. 3 - Tested ComCap support for IBM DB2 and Sun MySQL databases, fixing some minor issues. The IBM data link was previously unable to list table columns due to a slight difference in the schema arguments from Microsoft data links, it also returns all column names as upper case and sorted, which stopped the special database columns serial_nr and event_time being recognised. The error reporting listed database tables and columns has been improved so that data link errors are reported, rather than the lists just being left blank. Note that for a database to be supported by ComCap, the data link provider needs to support the ADO OpenSchema method, and this seems far from universal in third party providers or drivers. There is a new tick box 'Log More Information' on the Database tab that logs connection string and schema details, in case of problems in the future. 4 - If capture is to a database, previously when capture was started the database was opened before any data was accepted or logged. The default has now changed so that data capture starts immediately while the database is still being opened with data temporarily cached and written once the database is available, similarly to when database problem occur during capture. The old behaviour is retained if 'Immediate Pause for Database Problems' is ticked. Beware this may cause some confusion when initially testing database capture, unless the errors in the information log are seen. Reversed a database error handling change in 4.5 so that the first attempt to resume capture after it's been paused is again after 'Delay Before Restart', rather than immediately. This potentially avoids two or more restart attempts a second if restart immediately fails due to a database issue. 5 - Increased the number of database errors recognised as meaning the database server connection has been lost, each manufacturer returns it's own concept of errors, ie 'link failure', 'timeout', 'gone away', 'terminate', etc. The database is now closed after any error, since error reporting in usually better opening the database, however the last row is only saved if ComCap believes the error to be connectivity rather than a data format issue to avoid an endless loop of open and close. It is better to try and filter the data to avoid invalid data being written in the first place. 6 - ComCap was tested against IBM DB2 Express-C v9.7 for Windows, using the 'IBM OLE DB Provider for DB2' that is part of the client package that may be downloaded from: http://www-01.ibm.com/software/data/db2/express/ During install, you get an option to install just the client rather than the server, but beware it installs several windows services anyway. When setting up Data Link Properties, choose the IBM provider, then 'Direct Server Connection' with the server name as an IP or host, it should then allow a database to be selected once a logon has been entered and 'Allow Saving Password' ticked. A SQL script to create four ComCap tables may be found in newdb-ibmdb2.sql. Note that writing to stored procedures has not been tested. 7 - ComCap was tested against Sun MySQL v5.1 community edition, using the 'Connector/OBBC 3.51 driver' from http://dev.mysql.com/downloads/ When setting up Data Link Properties, choose the Microsoft OLE DB Provider for ODBC Drivers, 'Use Connection String', click Build, from the File Data Source tab click New, select 'MySQL ODBC 3.51 Driver;, click Next and specify a name for the link, a Driver Connect dialog will appear where the server IP or host, login and password, and database may be specified, after which the link appears in the list of DSNs and may be selected. Finally repeat the login and database selections on the Connection tab and tick 'Allow Saving Password'. Note the ODBC 5.1 driver does not seem to work with ComCap, nor does the 'MySQL OLEDB Provider'. A SQL script to create four ComCap tables may be found in newdb-mysql.sql. Note that writing to stored procedures has not been tested. 8 - Added a new 'Line or Record End' of 'Multiple Tags' which allows the end of a record to be determined by one or more short words or tags, instead of by CR or LF. This is primarily designed to ease parsing for database capture, so that multi line data can be processed, and also for multiple records sent without line endings. If these records have unwanted preceding or following tags, these may also be set as record endings and then filtered by phrases or minimum line length. All the line end settings have been moved from the General and Logging tabs, to a new Records tab. Record End Tags may include escaped characters similarly to 'Add Escaped Text', specifically \n new line, \c CR, \l LF, \\ backslash and \s space, so text only at the end of a line can be specified as a tag. 9 - Increased the maximum capture line length from 1,024 to 2,048. 10 - Made three improvements to the ComGen data generator, used to create data to test ComCap. Data Types now includes a column with a 'No CRLF' tick box to prevent CRLF being automatically added to each line. Network includes two new columns. 'Lines/Session' causes the TCP connection to be disconnected after that many lines of data have been sent, to simulate devices that periodically call home, send some data and then disconnect. Setting six lines per minute with one line/session will causes 10 sessions per minute. 'Device Id (first line)' causes some text to be sent once when a session connects, to simulate devices such as the Ecov that identify themselves when connected, note that CRLF is not added automatically so escapes should be used if a line ending is needed, ie \n for CRLF. 11 - Fixed a long term problem when running ComCap on Windows 2008 (and maybe Vista and 7) that meant, in some circumstances, the capture tabs in the main window were blank and not coloured. 12 - Fixed a problem introduced in the last release when Sending Data from the dialog using the interactive version only, where escape control sequences were ignored. 13 - Fixed a long term problem re-opening the info log that sometimes resulted in multiple 'file sharing violation' error messages and alerts. Also changed the mechanism for flushing the info log to disk, to reduce disk activity particularly with a lot of capture channels, which will be noticed when stopping capture. ComCap Release 4.5 - September 2009 1 - Fixed a long term problem with the ComCap Service where interactive tray monitoring and remote alerts and logging were lost if capture was restarted automatically due to a temporary PC freeze ore more than 15 seconds. Also added a periodic handshake to ensure tray monitoring has not stopped. 2 - It is now possible to set Data Loss checking separately for each capture channel, rather than all channels sharing the same settings, thus allowing for differing data flows and time zones for remote capture. In Capture Settings, on the General Tab if 'Check for Data Loss' is checked then new fields will appear on the Sounds/Data Loss tab. If 'Override Common Data Loss Settings' is checked, a grid identical to that in Common Settings appears, allowing different settings for each channel. These new settings are backward compatible, so no changes are needed if Data Loss is already set-up. Note that disabling Data Loss checking in Common Settings still disables it for all capture channels. 3 - Temporary database problems are almost inevitable, perhaps due to communication problems or the database server being rebooted, so ComCap has various means of coping with them. When a row can not be written to the database it is automatically buffered so it can be written later. This can happen because capture is happening faster than data can be written to the database, which may be limited to only 50 to 150 rows maximum per second. In previous versions, ComCap only buffered a maximum of 1,000 rows before automatically pausing capture, but this figure is now configurable from 50 to 99,999 rows on the Database tab. Note that rows are buffered in memory, so this must be sufficient for the data expected to be buffered. If ComCap is exited before the buffered rows are written, the rows are lost. Due to this, the 'Immediate Pause for Database Problems' will cause capture to be temporarily stopped so the source can continue to buffer data (is so capable). Attempts are made to re-open the database according to 'Delay Before Restart' seconds, and when successful any buffered rows are written. 4 - If 'Immediate Pause for Database Problems' is not specified, an alert is now sent when the database connection fails and another when it restarts. Improved error messages and alerts due to database problems, to avoid repeating the same alerts, and also added a new alert when capture restarts after being paused. 5 - Fixed a problem when attempts to reopen the database may have been skipped. Improved the error handling when there is a timeout writing to the database, usually due to communication problems. Fixed a problem when writing to databases using an ODBC connection when connection failure was not always detected correctly, thus stopping database re-open attempts 6 - The Information Log now has the date and time added when newly created, to avoid empty logs when there is nothing else to note. Fixed a problem with the ComCap service that meant small or empty log files may have been created in the program directory each time the services was started or stopped. 7 - The maximum captured line length may now be set to four characters (previously 64) before the line is broken. 8 - Fixed a bug sending data with UDP Server. Currently, UDP Send is only allowed if the channel is listening, not filtered from another channel. 9 - If echo captured data is configured, data received from the echo destination is now reverse echoed to the original capture source, with ComCap effectively behaving as a proxy. Currently, reverse echo data is handled a line at a time and echoed when each complete line is received, so can not be used for binary data. It is designed to send commands to remote capture devices. Reverse echo data is reported in the Information Log, not the capture logs. 10 - Fixed a validation problem on the Filters and Alerts tab, it's now possible to save settings with one of the filter lists blank. 11 - There is now no limit on the number of seconds that may be specified for the 'Repeat Start Command' allowing it to sent once a day, if required. ComCap Release 4.4 - January 2009 1 - In order to assist with waking up and configuring remote capture servers, the right click menu has three new options: Resend Start Command, Send Data, and Terminal Window. The first two are only available when capture has been started, the Terminal Window only when it's stopped or paused. Resend Start Command simply resends the same data sent when capture starts, but on demand. Send Data displays a new window allowing custom data to be sent on demand, with the last 50 command sent selectable from a drop down box, or allowing a file to be selected and sent (TCP/UDP only), typically to configure remote capture device, or load new firmware. 2 - A Telnet Terminal window has been added, to allow interaction with remote TCP/IP Servers. It may be selected from the File menu in the Main Window, or the right click menu when the remote address and port will be set according to the channel settings. Terminal window options may be specified such as font and size, function key behaviour, echo, auto LF and CR, etc. If Log is ticked, anything typed or received in the window is written to a session log with the file name telnet-(date)-(time).log. 3 - Added the ability to trigger alerts from captured text. The new Filters and Alerts tab includes a tick box 'Alert for Lines with Phrases (case sensitive)' which causes a captured lines to be checked against one or more phrases (entered one line at a time), with an alert being triggered if any are matched. The alert may be a pop-up window, email or SMS message, as configured in Settings, Common, and will include the channel name and the captured line. 4 - A number of remote TCP devices may be configured to identify themselves when a TCP session connects, by sending a device id as the first line of data. For instance, Tysso eCov serial to TCP/IP converter sends a five digit configurable number as the first line, while some GSM/3G modems send a device type and IMEI number. In Settings, Logging, 'First Line is Device Id' will cause the first captured line to be saved and not logged, and it may be added to subsequent captured lines using 'Add Escaped Text' with the \z command. So effectively the Device Id is added to each captured line to identify it, specifically with TCP Server where lots of different remote devices may be calling home. 5 - Added some more data validation and processing options, to clean up and reject unwanted captured data. In Settings, Logging, 'Remove Leading Spaces' ensures any control characters or blanks before real data are removed. 'Validate Line Length' allows lines shorter or longer than specified limits to be ignored. This check is done after space at the start and/or end of the line is removed, but before escaped text is added. On the Settings, General tab, ticking 'Filter and Alerts' will cause a new tab Filters and Alerts to appear, on which ticking 'Ignore Lines with Phrases (case sensitive)' causes a captured lines to be checked against one or more phrases (entered one line at a time), and ignored if any are found. ComCap now counts how many lines are ignored due to these various filters and checks, and shows the total in the status bar and hourly in the Information Log. 6 - It is now possible to add a pause into the Commands to Send Upon Start and Stop Capture, and the new Send Data command. \P (capital only) will cause a 50ms pause in the data being sent, with multiples allowing a longer delay. Note the pause may not necessarily be effective with TCP/IP, because packets may get combined at transport level, nor may the pause be exactly 50ms due to other activity within ComCap. 7 - Fixed a bug that meant the Log Files option Start/Stop in Capture Log did not always write the stop command, specifically if the capture file had closed due to inactivity. 8 - The Add Comment right click menu option is now effective when the ComCap service version is being used. Also, the Command window position is now restored correctly if moved. 9 - In Settings, Files, if Log Name Format is specified as Custom, the Custom Log Name Mask is now validated to ensure that it includes yy or yyyy mask characters for year. The custom mask has always been ignored unless yy was found, but without warning. 10 - If the Capture Log New Log Interval is set to Weekly, it's now also possible to set the New Log Open Time, so the log could rotate at 6am on Monday morning instead of midnight. 11 - When ComCap starts, the main window is only automatically minimised if Auto Start is specified. 12 - It's now possible to change channel settings during paused capture for both service and interactive versions, not just the latter. 13 - Fixed a bug introduced in Release 4.3 that meant the remote IP address and port could change line by line for TCP Client or Server, if data was being captured from other sessions at the same time. 14 - Fixed a possible bug that meant extra data might appear at the start of TCP session, if the connection was dropped and reconnected without capture being stopped and restarted. 15 - Fixed a bug that meant test email and SMS alerts could not be sent if ComCap was configured to use the service version and it was stopped. Also, the validation of Email To: addresses has been improved to stop a blank address preventing email being sent. ComCap Release 4.3 - July 2008 1 - The Unlimited version now supports capture from up to 500 serial COM ports, TCP/IP Server, TCP/IP Client and UDP/IP network protocols simultaneously, with suitable hardware. It has been tested on Windows XP with 260 TCP/IP sessions created by ComGen (which never had a limit), writing capture logs one line per second, with only 2-6% CPU usage on a old Pentium 4 and 12 megs memory usage. 2 - ComCap, ComGen and Signals now all support serial COM ports with names other than COM followed by a number, ie CNCA2. Virtual COM ports are sometimes installed with such strange names. 3 - When capturing data on the same channel from multiple UDP clients using different IP addresses, the correct IP of the last packet is now displayed and added to the capture log if the Remote IP Address is added as escaped text (\r), rather than the IP of the first packet only. 4 - It's now possible to use keyboard short cuts to copy text from the capture log windows, as well as the right click menu options. 5 - Improved the logging for license key failures, particularly when old version 3 keys are used by accident with ComCap version 4 (which needs newer keys). ComCap Release 4.2 - November 2007 1 - In order to support capture from appliances that only return data when triggered, in Capture Settings, General a new option 'Repeat Start Command Every X Seconds' has been added. This causes the Start command text to be repeatedly sent at the specified interval. The maximum interval is 999 seconds, with zero meaning don't repeat the command. 2 - In Capture Settings, General, a 'Line End Timeout' may now be specified in seconds, where zero means no timeout, up to 300 seconds. When the timeout expires, an incomplete captured line will be processed, saved and displayed. This is usually only necessary when non-ASCII data is being captured where there are no carriage returns or line feeds, but may also be useful when setting up ComCap to capture from a serial port with unknown speed, since it can be used to cause display of the 'corrupted' data caused by speed mismatch which will be missing line ends. 3 - In Capture Settings, Logging, 'Ignore Blank Lines' causes lines without any printable characters to be ignored. This will reduce the size of logs that contain far too many blank lines. This option should generally be used with 'Remove Control Characters'. 4 - In Capture Settings, Logging, 'Log Hex Data' causes all captured data to be converted into hexadecimal (doubling the size). This is primarily intended for capturing binary data, but may be used a debugging tool to find the exact format of data being captured, for instance the type of line endings. Normal line ending rules are applied when capturing in hex, so lines may be broken on linefeed, etc (also saved in hex). 5 - The actual information log file names are now logged hourly. This may be useful where an emergency log is opened due to an access conflict on the original file. 6 - When configuring ComCap to 'Capture Using Background Service', a warning is now displayed if any channels are configured as 'Save to Database' and a service account has not been specified, since database access needs logon credentials. 7 - It's now again possible to specify UNC network paths in Log Directories with 'Capture Using Background Service'. 8 - When starting the ComCap background service from the ComCap tray version, the channel currently selected now starts to update immediately all channels have started, rather than the next time it's clicked. 9 - When using the TCP Server protocol to Echo captured Data, or Send Remote the Information Log or Alerts, up to five remote TCP Clients can now access the TCP Server simultaneously (previously just one). This allows ComCap to be used to distribute captured data to five other PCs, perhaps also running ComCap. 10 - The highest numbered COM port recognised by ComCap has been increased from COM60 to COM200 to allow for virtual COM ports being installed with high numbers. Also, the Standard version now lists all installed COM ports, not just the first three, but still only allows capture from a maximum of three ports. 11 - If the requirement is to capture serial data from another application on the same PC, ComCap now includes a Null Modem Emulator (com0com) from http://com0com.sourceforge.net/ that installs a linked pair of virtual serial ports, instead of needing to use a physical pair of COM ports and a null modem cable. ComCap Release 4.1 - February 2007 1 - The ComCap system tray icon now shows a small red or green square to indicate whether capture is stopped or started. 2 - If ComCap is unregistered, an alert is now sent when capture starts and when it stops after one hour (24 hours with a v3 license). This will serve as a warning if the license key is lost or corrupted. 3 - Both the Information Log and Alerts may now be sent to a remote PC using UDP, TCP Client or TCP Server protocols, perhaps to another copy of ComCap. This will ease central monitoring of remote capture. 4 - The Alert window will now appear without also showing the tray capture window, which avoids unnecessary log scrolling on unattended PCs. 5 - Echo to Remote using UDP now pings the remote host before sending any data to establish it exists, and continues to ping at the 'Wait Seconds' interval during the capture session to ensure it's still available. Note this does not mean a UDP server is listening on the remote computer, just that the computer is running. This is disabled if Common Preferences 'Don't Check Connections with Ping Echo' is ticked. 6 - When using Echo to Remote with Syslog, the Severity and Facility Priorities are now selected from drop down lists, rather than needing to enter a numeric value (ie <14>). 7 - The ComCap application and set-up program are now both digitally signed as being from Magenta Systems Ltd. This will reduce the severity of the warnings that Windows Vista generates when programs are run. 8 - Fixed a bug with capture using TCP Server where an optional Start Command was only sent for the server remote connection, and not subsequent connections. 9 - Fixed a bug that meant the service version did not always log version and registration information on start-up. 10 - Fixed a bug that meant an unnecessary 'emergency' information log file might have been created in the program directory on start-up or close down. Such logs are generally only created when the correct log volume is not available or the log file can not be opened due to an error. 11 - In Common Settings, Serial Ports, a new column 'Control Lines' has been added. This option determines whether the three serial control lines CTS, DTS and DCD should be checked during capture, to make detection of connection or hardware problems easier since an alert can be sent if capture stops. If ticked, capture will only start when at least one of the three control lines goes high, and will stop if they all drop. Note this is really cosmetic only (with the tab colour changing red to green) and start/stop logging, and data will be still be captured even if all the control line are low. But unless the channel is seen to 'start', some functionality may not work correctly such as capture file name roll over. 12 - In Common Settings, Log Files, a new tick box 'Send Alert on File Errors' has been added. If ticked, any major errors opening or writing the information or capture logs will generate an alert. Currently there may be repeated alerts, since such errors often happen rapidly in multiple channels, but this should be fixed shortly. 13 - In Common Settings, Capture Logging, a new tick box 'Send Alert on Stop' has been added. If ticked, an alert is sent every time ComCap is stopped. This is intended for where ComCap should be running 24/7, and where alerts are sent via email, SMS or remotely to warn of accidental stops. Note this will only work for a 'clean' close down, if ComCap is crashed from Task Manager or by Windows close down, the alert may not be sent. 14 - Attempted to handle a situation with the service version where a serious problem that causes hundreds or thousands of error lines in the information log could potentially overflow the 'echo to tray' communication channel and lock-up ComCap. 15 - Made a few improvements in ComGen. It will now close down faster when Windows itself closes down. UDP Client now pings the remote host before sending any data to establish it exists, and continues to ping during the session to ensure it's still available. Corrected the default Data Types that still had \s8 for a serial number (from an earlier beta) instead of \#8 used in the final release. ComCap Release 4.0 - October 2006 New Release - Added TCP/IP and UDP. Added SQL Database capture. Added Echo of captured data. ComCap Release 3.0 - April 2003 New Release ComCap Release 2.0 - January 2001 New Release ComCap Release 1.0 - February 2000 First Release Return to ComCap Home Page Copyright © 2021 Magenta Systems Ltd, England. All Rights Reserved. |