Planet OpenNMS

December 09, 2019

OpenNMS On the Horizon – December 9th, 2019 – BMP, Graph Service, Time-Series APIs, Bug Fixes, and More!

It's time for OpenNMS On the Horizon!

In the last week we continued to work on BMP and improving graph and time-series APIs, as well as other bug fixing and enhancements.

Github Project Updates

  • Internals, APIs, and Documentation

    • Dustin continued to work on adding BMP support.
    • I fixed a bug where the ElasticSearch forwarder could modify events before they're passed on to other subsystems.
    • Chandra fixed a bug in opennms-snmp:remove-from-definition
    • Markus continued his work enhancing the new graph API.
    • Christian fixed an issue with leaked threads in JCIFS.
    • Antonio and Markus did more work on improvements to topology link provider(s).
    • Sean Torres worked on fixes to allow flows to balance across Kafka partitions.
    • Christian fixed processing of filter rules in poller-configuration.xml.
    • Patrick continued his work evaluating how to rework our time-series APIs.
    • Jeff worked on an absolute-value handler for the JRobin spike hunter.
    • Markus fixed an issue with classification rule definitions that could cause OOM.
    • Chandra worked on adding gRPC support for IPC between Minion and OpenNMS.
  • Web, ReST, and UI

    • Markus updated the topology UI to allow choosing a layer with a URL parameter.

December Horizon and Meridian Releases

In December we released updated versions of Horizon 25, Meridian 2018, and Meridian 2019.

Meridian 2018.1.14 was primarily a small bug fix release. Meridian 2019.1.1 and Horizon 25.1.1 both had a larger number of bug fixes mostly related to alarm and event processing and potential resource leaks, and a few other enhancements.

Calendar of Events

  • January Releases - January 7th, 2020

    The next OpenNMS release day is January 7th, 2020.

    So far we expect to be putting out the following releases:

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

- Ben

Resolved Issues Since Last OOH

  • NMS-12358: Create BMP listener and parser for telemetryd
  • NMS-12390: Elasticsearch event forwarder manipulates in-flight event
  • NMS-12400: Invalid poller filter rules can break service web UI page
  • NMS-12414: Create BMP Adapter for Telemetry
  • NMS-12415: Create BMP Adapter for Peer Up / Down Events
  • NMS-12425: Create graph definitions for BMP statistics
  • NMS-12429: Classification Engine reload causes OOM when defining a bunch of rules

by RangerRick at December 09, 2019 06:24 PM

December 03, 2019

OpenNMS Horizon 25.1.1 (Bánh bột lọc) Released

Release 25.1.1 is the third release in the Horizon 25 series.

It contains a number of bug fixes mostly related to alarm and event processing and potential resource leaks, as well as provisioning enhancements to SNMP profiles.

The codename for 25.1.1 is Bánh bột lọc.

Bug
  • Readiness probe with Minion in Kubernetes with health:check does not work (Issue NMS-12120)
  • AlarmLifecycleListenerManagerIT and AlarmdIT flapping (Issue NMS-12309)
  • Strange behavior on used threads and file descriptors on Minion (Issue NMS-12366)
  • Upstream Drools Bug: From with modify fires unexpected rule (Issue NMS-12367)
  • Build failure during release for 25.1.0 in CircleCI (Issue NMS-12380)
  • Discovery does not honor exclude-range inside a definition (Issue NMS-12385)
  • Discovery exclude-range is not location-aware (Issue NMS-12386)
  • Update OpenJDK 11.0.4 to 11.0.5 (Issue NMS-12387)
  • Elasticsearch event forwarder manipulates in-flight event (Issue NMS-12390)
  • send-event.pl is broken after OpenNMS 25.1.0 update (Issue NMS-12392)
  • SNMP profile fitting is not triggered in some cases when MINION is involved (Issue NMS-12399)
  • Alarmd fails intermittently and OOMs (Issue NMS-12412)
  • SNMP Remove from definitions fails for definitions with profile label (Issue NMS-12413)
Enhancement
  • Migrate guide to run with Docker from GitHub repo to Installation guide (Issue NMS-12375)
  • Enhance new snmp profiles to allow fitting to nodes inside requisitions without SNMP service associated to any IPs configured (Issue NMS-12396)
  • Use OpenJDK 8 JDK for build environment (Issue NMS-12398)

by RangerRick at December 03, 2019 10:10 PM

OpenNMS Meridian 2019.1.1 (Venus) Released

Release 2019.1.1 is the second release in the Meridian 2019 series.

It contains a number of bug fixes mostly related to alarm and event processing and potential resource leaks, as well as provisioning enhancements to SNMP profiles.

The codename for 2019.1.1 is Venus.

Bug
  • Readiness probe with Minion in Kubernetes with health:check does not work (Issue NMS-12120)
  • Cannot use poller:poll karaf command with WsManMonitor through Minions (Issue NMS-12365)
  • Strange behavior on used threads and file descriptors on Minion (Issue NMS-12366)
  • Upstream Drools Bug: From with modify fires unexpected rule (Issue NMS-12367)
  • "Page Not Found" in alarm-list when choosing number of alarms in dropdown-list (Issue NMS-12379)
  • Build failure during release for 25.1.0 in CircleCI (Issue NMS-12380)
  • backport missing patches from 25.1.0 to foundation-2019 (Issue NMS-12384)
  • Discovery does not honor exclude-range inside a definition (Issue NMS-12385)
  • Discovery exclude-range is not location-aware (Issue NMS-12386)
  • Update OpenJDK 11.0.4 to 11.0.5 (Issue NMS-12387)
  • Elasticsearch event forwarder manipulates in-flight event (Issue NMS-12390)
  • send-event.pl is broken after OpenNMS 25.1.0 update (Issue NMS-12392)
  • SNMP profile fitting is not triggered in some cases when MINION is involved (Issue NMS-12399)
  • Alarmd fails intermittently and OOMs (Issue NMS-12412)
  • SNMP Remove from definitions fails for definitions with profile label (Issue NMS-12413)
Enhancement
  • Create a step-by-step guide how to setup Kafka for Minions (Issue NMS-12368)
  • Enhance new snmp profiles to allow fitting to nodes inside requisitions without SNMP service associated to any IPs configured (Issue NMS-12396)

by RangerRick at December 03, 2019 10:01 PM

OpenNMS Meridian 2018.1.14 (Earthquake) Released

Release 2018.1.14 is a small update to 2018.1.13 that fixes a few bugs including a fix to syslog date parsing, Drools fixes, and a few other changes.

The codename for 2018.1.14 is Earthquake.

Bug
  • Syslog incorrect dates being parsed into database (Issue NMS-10605)
  • Cannot use poller:poll karaf command with WsManMonitor through Minions (Issue NMS-12365)
  • Upstream Drools Bug: From with modify fires unexpected rule (Issue NMS-12367)
  • "Page Not Found" in alarm-list when choosing number of alarms in dropdown-list (Issue NMS-12379)
  • Elasticsearch event forwarder manipulates in-flight event (Issue NMS-12390)

by RangerRick at December 03, 2019 08:46 PM

December 02, 2019

OpenNMS On the Horizon – December 2nd, 2019 – SNMP, Time Series Data, ALEC, BMP in Telemetryd, Linkd, and More!

It's time for OpenNMS On the Horizon!

In the last week we worked on SNMP profile matching, time series APIs, ALEC improvements, BMP in Telemetryd, the Linkd topology provider, and more!

Github Project Updates

  • Internals, APIs, and Documentation

    • Chandra continued to work on wrapping up improvements and fixes to profile fitting when provisioning.
    • Patrick did more work on evaluating refactoring our time series APIs.
    • Chandra worked on fixing exclude ranges inside discovery definitions.
    • Markus continued his work on improvements to the graph services API.
    • Jesse updated the ALEC API to use an IP->node cache.
    • Christian worked on fixing bugs in filter processing in poller-configuration.xml.
    • Chandra worked on fixing some issues with handling foreign sources in new suspect scans.
    • Chandra fixed a bug in the opennms-snmp:remove-from-definition Karaf command.
    • Dustin did more work on implementing BMP support in Telemetryd.
    • Antonio worked on improvements to the Linkd topology provider.
    • Ronny did more work updating documentation and links in a number of places.

Calendar of Events

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

- Ben

Resolved Issues Since Last OOH

  • NMS-12309: AlarmLifecycleListenerManagerIT and AlarmdIT flapping
  • NMS-12368: Create a step-by-step guide how to setup Kafka for Minions
  • NMS-12375: Migrate guide to run with Docker from GitHub repo to Installation guide
  • NMS-12385: Discovery does not honor exclude-range inside a definition
  • NMS-12386: Discovery exclude-range is not location-aware
  • NMS-12396: Enhance new snmp profiles to allow fitting to nodes inside requisitions without SNMP service associated to any IPs configured
  • NMS-12399: SNMP profile fitting is not triggered in some cases when MINION is involved
  • NMS-12408: Add possibility to listen for graph changes
  • NMS-12412: Alarmd fails intermittently and OOMs
  • NMS-12413: SNMP Remove from definitions fails for definitions with profile label

by RangerRick at December 02, 2019 07:23 PM

November 25, 2019

OpenNMS On the Horizon – November 25th, 2019 – Bug Fixes, BMP in Telemetryd, Time-Series APIs, Helm, and More!

It's time for OpenNMS On the Horizon!

In the last week we continued to work on fixing bugs and doing new work on BMP support in Telemetryd and time-series API improvements. Also, Helm got some fixes for Grafana 6.4 and a new release.

Github Project Updates

  • Internals, APIs, and Documentation

    • Ronny continued his work on expanding the documentation for setting up Kafka and Minions.
    • Dustin did more work on adding support for BMP to Telemetryd.
    • Chandra worked on wrapping up exclude-range fixes in SNMP definitions.
    • Jesse worked on fixing some issues in Drools and Alarmd.
    • Markus continued working on improvements to the graph API.
    • Dustin did more work on adding support for custom headers in PageSequenceMonitor and HttpPostMonitor.
    • Patrick worked on some proof-of-concept code for a new time-series API.
    • Chandra fixed SNMP profiles to be processed during provisioning node scan.
  • Web, ReST, and UI

    • I fixed an issue with the Flows datasource in Helm, as well as some other cosmetic fixes.
    • Markus did a little more work on his universal search changes.

Helm 4.0.2 Released

Last week, Helm -- the OpenNMS plugin for Grafana -- got updated to 4.0.2. RPM and Debian packages have been released, and we are waiting for Grafana to update their plugin registry with the change as well.

This release fixes a bug with the Flow datasource and Grafana 6.4, as well as some other cosmetic changes to the Flow query interface.

Calendar of Events

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

- Ben

Resolved Issues Since Last OOH

  • HELM-192: Rendering Bug in Flow Query Builder
  • HELM-193: Conversation Flow Histogram show "null" instead of Unknown/Other in legend
  • NMS-12347: Provide Central Search Function
  • NMS-12379: "Page Not Found" in alarm-list when choosing number of alarms in dropdown-list
  • NMS-12387: Update OpenJDK 11.0.4 to 11.0.5
  • NMS-12389: Implement caching strategies for graphs
  • NMS-12393: Add HTTP header injection in HTTP Monitors

by RangerRick at November 25, 2019 04:21 PM

November 18, 2019

OpenNMS On the Horizon – November 18th, 2019 – Bug Fixes, CircleCI, BMP Support, Helm, and More!

It's time for OpenNMS On the Horizon!

In the last week we fixed more bugs and worked on other infrastructure, as well as continuing work on BMP support and a Grafana 6.4 Helm fix.

Github Project Updates

  • Internals, APIs, and Documentation

    • Markus continued to work on graph API improvements.
    • I added support for building (and signing) Debian packages in the CircleCI workflow.
    • Christian worked on adding support for defining custom headers in the HttpPostMonitor and PageSequenceMonitor.
    • Ronny worked on a bunch of updated Docker and other container documentation.
    • Patrick worked on normalizing all our Karaf shell commands.
    • Ronny and I fixed the hour-rendering bug in send-event.pl introduced in Horizon 25.1.0.
    • Christian did more work on adding BMP support to Telemetryd.
    • Christian fixed an issue with WS-Man polling on Minion.
    • Chandra did more work on fixing exclude-range support in the discovery config.
  • Web, ReST, and UI

    • I fixed an issue with the Flow data source in the Helm Grafana plugin.

Calendar of Events

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

- Ben

Resolved Issues Since Last OOH

  • HELM-190: TypeError: Cannot read property 'toBits' of undefined
  • NMS-12365: Cannot use poller:poll karaf command with WsManMonitor through Minions
  • NMS-12367: Upstream Drools Bug: From with modify fires unexpected rule
  • NMS-12369: FlowStackKafkaIT hanging on CircleCI
  • NMS-12388: Allow persisting edges with references to at least one vertex which is not in the edge's namespace
  • NMS-12392: send-event.pl is broken after OpenNMS 25.1.0 update
  • NMS-12398: Use OpenJDK 8 JDK for build environment

by RangerRick at November 18, 2019 06:50 PM

November 12, 2019

OpenNMS On the Horizon – November 12th, 2019 – Bug Fixes, Docker and CI, Releases, and More!

It’s time for OpenNMS On the Horizon!

In the last week we fixed some bugs, worked on Docker and CI, and did the monthly Horizon and Meridian releases.

Github Project Updates

  • Internals, APIs, and Documentation
    • Patrick fixed a bug where the config tester didn’t catch a missing datacollection file.
    • Markus continued his work on enhancing graph querying to handle semantic zoom level and focus, as well as enriching graphs with additional node data.
    • Ronny and I worked on docker container updates.
    • Christian fixed the WS-Man poller.
    • I updated our CircleCI config to sign RPMs.
    • Chandra fixed discovery to honor <exclude-range> inside a definition.
    • Jesse worked on updating to the latest Drools engine.
  • Web, ReST, and UI
    • Alejandro fixed some issues in the requisition UI.
    • I worked on a Helm issue that could prevent adding a new flow histogram panel.

November Releases: Meridian 2016 through 2019, Horizon 25.1.0

Last week, we released Horizon 25.1.0 with a number of enhancements targeting our long-term-support release of Meridian 2019.

For a complete list of changes, see the release notes:

Calendar of Events

  • December Releases - December 3rd, 2019

    The next OpenNMS release day is December 3rd.

    So far we expect to be putting out the following releases:

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

- Ben

Resolved Issues Since Last OOH

  • NMS-10396: Config tester doesn’t detect missing xml datacollection file
  • NMS-12120: Readiness probe with Minion in Kubernetes with health:check does not work
  • NMS-12339: Clarify that LDAPS is supported for external authentication
  • NMS-12376: Allow querying a graph using vertices in focus and a szl
  • NMS-12377: Listing monitoring locations from the administrative WebUI is not working
  • NMS-12380: Build failure during release for 25.1.0 in CircleCI
  • NMS-12382: Implement Enrichment of vertices
  • NMS-12384: backport missing patches from 25.1.0 to foundation-2019

by RangerRick at November 12, 2019 08:24 PM

November 06, 2019

OpenNMS 2019.1.0 (Mercury) Released

Release 2019.1.0 is the first release in the Meridian 2019 series.

The codename for 2019.1.0 is Mercury.

 

What’s New in Meridian 2019

Since Meridian 2018, we have introduced a large number of features, most notably Telemetryd (for processing streaming telemetry like NetFlow and sFlow), the Sentinel (for horizontal scaling of telemetry and other processing), and ALEC (for alarm correlation).

On top of that, there have been many other improvements and bug fixes since Meridian 2018.

Meridian 2019 roughly matches the feature set available in Horizon 25.

Architecture for Learning Enabled Correlation

Horizon 23 introduced support for correlation of alarms into meta-alarms called "situations" using an engine called the Architecture for Learning Enabled Correlation.

Situations are OpenNMS alarms that contain one or more triggering alarms, which allows them to be browsed, acknowledged, and unacknowledged just like any other alarm.

A high-level overview of the goal and implementation of correlation can be seen on the ALEC web site.

Changes to the Alarm Lifecycle
Alarm Clearing

Traditionally, OpenNMS has created and resolved alarms in pairs, with one alarm representing the triggering event (or events), and then a second alarm representing the resolution. Horizon 23 changes this default behavior to use a single alarm to track the problem state, incrementing the alarm count when it occurs while in a problem state, or when moving from resolved back into a problem state. Additionally, you can configure OpenNMS to create a new alarm if a problem happens again.

These behaviors are controlled by the introduction of 2 new settings in the opennms.properties file:

org.opennms.alarmd.legacyAlarmState
This setting reverts to the old (pre-23) behavior of creating separate alarms for a problem and its resolution.
org.opennms.alarmd.newIfClearedAlarmExists
This setting forces Alarmd to create a new alarm if a problem reoccurs, rather than incrementing an existing alarm. (Note: this is ignored if legacyAlarmState is set to true.)

These improvements are covered in a lunch and learn video we published recently, if you would like to learn more.

Alarmd Architecture

To facilitate the implementation of ALEC, alarmd has been rearchitected to use Drools to manage the alarm lifecycle, rather than Vacuumd automations, triggers, and actions.

If are migrating changes to vacuumd-configuration.xml from an earlier Meridian release, it is strongly recommended you port them to the new Alarmd Drools context. The Drools rules are in the $OPENNMS_HOME/etc/alarmd/drools-rules.d/ directory.

Additionally, we no longer generate alarmCreated, alarmEscalated, alarmCleared, alarmUncleared, alarmUpdatedWithReducedEvent, and alarmDeleted events. Instead, it is recommended that you add Drools rules to react to alarm changes.

For more complicated integrations, we also have a new API — AlarmLifecycleListener — for reacting to alarm changes.

Kafka Data Collection Sync

In addition to publishing events, alarms, and node inventory to Kafka, we now publish collected time-series data to the Kafka bus as well.

Sentinel

In addition to the Minion, we have added a new container-based subsystem called "Sentinel." The Sentinel is a Karaf container that can be configured to run a subset of OpenNMS daemons as a standalone tool, to aid in horizontal scaling and/or high availability.

Sentinel is designed to run our Karaf/Camel/SQS-based messaging bus, syslog listener, telemetry receiver, and Newts and Elasticsearch persistence.

Node and Interface Metadata

There is now support for associating arbitrary metadata with nodes and interfaces, including configuring arbitrary metadata in the requisition UI.

For details on using the metadata APIs, see the Admin Guide and the Developer Guide.

Elasticsearch 7.x Support

All of the features that leverage integrations with Elasticsearch i.e. event & alarm history, flows & situation feedback have been updated to support Elasticsearch 7.x. Elasticsearch versions before 7.x are no longer supported.

Given the pace of changes and the number of breaking changes between major versions of Elastisearch, we will focus on supporting a single major version of Elasticsearch per release moving forward.

Other Improvements

Since Meridian 2019 is based on Horizon 25, it contains all the fixes and updates that have occurred since Meridian 2018 was created from the Horizon 21 codebase.

For a more complete list of changes included in this release, see the "What’s New" documentation for the following Horizon releases:

by RangerRick at November 06, 2019 02:17 AM

OpenNMS Horizon 25.1.0 (Momo) Released

Release 25.1.0 is the second release in the Horizon 25 series.

It contains a bunch of bug fixes and enhancements, including packaging fixes, BSM updates, alarmd improvements, and much more.

The codename for 25.1.0 is Momo.

  Note: This release contains a couple of breaking changes:

  1. Dates in event XML (in the ReST API and published to the event port) now must be formatted in standard ISO-8601 format (eg, 2019-11-05T14:36:00-04:00).
  2. Scriptd has been changed to execute scripts in a single read-only transaction by default. If you need the ability to write changes to the database or need to do your own transaction management for any other reason, you will need to add transactional="false" to the <scriptd-configuration …​> tag in the scriptd-configuration.xml file.
Bug
  • removed service will break BSM web ui (Issue NMS-9322)
  • Event parameters no longer preserve ordering (Issue NMS-9827)
  • The JMX-Cassandra service goes down for all the cluster when a single instance is down. (Issue NMS-10027)
  • deleting a BSM monitor while an alarm is active doesn’t clear the alarm (Issue NMS-10184)
  • default event description is incorrect (Issue NMS-10346)
  • Config tester doesn’t detect missing xml datacollection file (Issue NMS-10396)
  • Minion fails to start on Raspberry Pi (Issue NMS-10447)
  • BSM alarm severity is not being updated (Issue NMS-10578)
  • snmp authentication error traps with Enhanced Linkd / bridge discovery (Issue NMS-10582)
  • Elasticsearch forwarding fails to recover after outage (Issue NMS-10697)
  • Flow rest results for top N queries are not returned in the correct order (Issue NMS-12104)
  • Flow Classification not functioning as expected (Issue NMS-12259)
  • ReST API for meta-data doesn’t support JSON (Issue NMS-12272)
  • UI for meta-data is only present when using the horizontal layout (Issue NMS-12273)
  • Groups disappear in classification UI (Issue NMS-12291)
  • BSM simulation mode does not reset the last state (Issue NMS-12302)
  • Docker image for sentinel runs the process as root (Issue NMS-12311)
  • Docker image for Minion runs with UID 1001 but the minion user has UID 997 (Issue NMS-12312)
  • Web Assets Dependency Rollup 2019-09-24 (Issue NMS-12320)
  • Memory leak in Drools engine for alarmd (Issue NMS-12322)
  • Minion RPM upgrade deletes /opt/minion (Issue NMS-12324)
  • Threshold state keys do not incorporate the collected resource’s instance label (Issue NMS-12329)
  • Reportd generated reports cause: "No bean named '' is defined" in Persisted Reports (Issue NMS-12337)
  • InterfaceNodeCache doesn’t remove deleted nodes immediately (Issue NMS-12338)
  • Delivering a report with "-" in local part of email address is not working (Issue NMS-12342)
  • XMLMarshalException in ipc.log: Unexpected close tag </inst>; expected </instance> (Issue NMS-12343)
  • Update installation guide to CentOS 8 (Issue NMS-12348)
  • Install guide for R-core is broken for CentOS 8 (Issue NMS-12352)
  • Karaf feature install issue with opennms-core-tracing-jaeger (Issue NMS-12359)
  • Fix requisition cache when accessing the Requisitions UI via "Edit in Requisition" (Issue NMS-12360)
  • Listing monitoring locations from the administrative WebUI is not working (Issue NMS-12377)
Enhancement
  • Add more information into BSM alarms (Issue NMS-9352)
  • Refactor the compatibility matrix in the documentation (Issue NMS-9684)
  • Be able to change the number of rows for the pagination control on the Requisitions UI (Issue NMS-9793)
  • Documentation typo for /rest/ifservices on the developers guide (Issue NMS-9842)
  • Add Web-Hook as delivery option (Issue NMS-12153)
  • Refactor Event Timestamps to ISO-8601 Format (Breaking Change) (Issue NMS-12263)
  • Improve robustness of CassandraBlobStore for async operations (Issue NMS-12274)
  • Clearing threshold states via shell should take effect immediately and not require restart (Issue NMS-12277)
  • List Kafka RPC/Sink topics, Expose Metrics on Karaf shell (Issue NMS-12294)
  • Create proper systemd files for OpenNMS, Minion and Sentinel (Issue NMS-12299)
  • Add ability to update definitions when SNMP profile changes (Issue NMS-12307)
  • Fix security vulnerability with jackson-databind (Issue NMS-12308)
  • Availability boxes on node pages including sub pages differ (Issue NMS-12321)
  • OpenNMS 25 Dependency Still Allows Old PostgreSQL Versions (Issue NMS-12341)
  • Update base container image to use CentOS 8 (Issue NMS-12353)
  • Remove floating OpenJDK dependencies in OCI build (Issue NMS-12354)
  • Detect and help resolve Karaf bootstrap issues (Issue NMS-12356)
  • Update CISCO-ENTITY-SENSOR-MIB threshold trap events to include alarm-data (Issue NMS-12362)
  • switch core/web-assets from yarn to npm (Issue NMS-12363)
  • Collect and display file descriptor statistics via JMX for OpenNMS and Minion (Issue NMS-12364)

by RangerRick at November 06, 2019 01:24 AM

November 05, 2019

OpenNMS Meridian 2018.1.13 (Flood) Released

Release 2018.1.13 is a small update to 2018.1.12 that fixes a few UI bugs.

The codename for 2018.1.13 is Flood.

Bug
  • Listing monitoring locations from the administrative WebUI is not working (Issue NMS-12377)
Enhancement
  • Availability boxes on node pages including sub pages differ (Issue NMS-12321)

by RangerRick at November 05, 2019 09:31 PM

OpenNMS Meridian 2017.1.20 (Lisbon) Released

Release 2017.1.20 is a small update to 2017.1.19 that fixes an issue with the display of availability on the node page.

The codename for 2017.1.20 is Lisbon.

Enhancement
  • Availability boxes on node pages including sub pages differ (Issue NMS-12321)
  • Clarify that LDAPS is supported for external authentication (Issue NMS-12339)

by RangerRick at November 05, 2019 07:09 PM

OpenNMS Meridian 2016.1.23 (Balthasart) Released

Release 2016.1.23 is a small update to 2016.1.22 that fixes an issue with the display of availability on the node page.

The codename for 2016.1.23 is Balthasart.

Story
  • Add a command to show configuration diffs (Issue NMS-12129)
  • Availability boxes on node pages including sub pages differ (Issue NMS-12321)

by RangerRick at November 05, 2019 06:37 PM

November 04, 2019

OpenNMS On the Horizon – November 4th, 2019 – Bug Fixes, Debugging Tools, Documentation Updates, Helm, and More!

It’s time for OpenNMS On the Horizon!

In the last 2 weeks we fixed a bunch of bugs, improved tools for debugging running systems, updated docs, fixed Helm, and more!

Github Project Updates

  • Internals, APIs, and Documentation
    • Ronny worked on updates to our Docker images to use CentOS 8.
    • Markus did more work on supporting sending reports to a webhook.
    • I fixed the new unified Minion packages to properly upgrade from Horizon < 25.
    • Chandra continued to work on updating discovery to support detectors.
    • Chandra fixed a bug in the node-interface cache that could cause issues when nodes are deleted.
    • Chandra fixed an issue with duplicate chunks in large Karaf messages.
    • Chandra added support for listing Kafka RPC/sink topics and metrics in the Karaf shell.
    • Jeff updated the threshold trap events for CISCO-ENTITY-SENSOR-MIB to include alarm data.
    • I fixed the OpenNMS packages to depend on the correct PostgreSQL version(s).
    • Jesse fixed a bug in Drools alarm processing that could cause a memory leak.
    • Chandra changed Provisiond so it can run script policies at the end of the node scan.
    • Alejandro made JMX configuration updates for collecting file descriptor statistics.
    • Ronny updated the compatibility matrix in the docs, and moved it to the install guide.
    • Chandra updated the SNMP agent config handling to update definitions when profiles change.
    • Jesse and Patrick worked on a daemon to watch for common startup issues, log them, and shut down gracefully.
    • Will Keaney worked on an update to the embedded Drools to version 7.18.0 to fix rule-processing bugs.
    • Patrick finished up his work on fixing event parameter ordering.
    • Ronny updated the docs to describe setting up Kafka for Minions.
    • Markus updated the new graph API to support a default focus.
  • Web, ReST, and UI
    • Christian did more work on making BSM handle deleted alarms gracefully.
    • Alejandro fixed an issue with cache handling in the requisition UI.
    • I fixed the way metadata is passed in the Helm entities datasource so it works in Grafana 6.4.
    • Alejandro added support for changing the number of pagination rows in the requisition UI.

Helm 4.0.1

Helm, the OpenNMS plugin suite for Grafana, has had a small update to 4.0.1.
This release only bumps some dependencies, and fixes running under Grafana 6.4.

It is available in the OpenNMS yum and apt repositories, and will be available
for installation through the Grafana CLI
once our pull request is merged.

Calendar of Events

  • November Releases - November 5th, 2019

    The next OpenNMS release day is November 5th.

    We expect to be putting out the following releases:

    • Horizon 25.1.0
    • Meridian 2016.1.23
    • Meridian 2017.1.20
    • Meridian 2018.1.13
    • Meridian 2019.1.0

    With the release of Meridian 2019.1.0, Meridian 2016 will be end-of-lifed.
    Security patches will continue to be applied for Powered By OpenNMS customers, but no new releases will be made.

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

- Ben

Resolved Issues Since Last OOH

  • ALEC-80: The Kafka Topic edgesTopic from the configuration is ignored
  • ALEC-81: OpenNMS Kafka - Format dates in ISO-8601
  • HELM-184: Fix Helm for Grafana 6.4
  • IPL-34: Update RHEL/CentOS IPLIKE Packages To Be Co-Installable
  • JS-45: Panel data error: TypeError: Cannot read property ‘id’ of undefined
  • NMS-9352: Add more information into BSM alarms
  • NMS-9684: Refactor the compatibility matrix in the documentation
  • NMS-9793: Be able to change the number of rows for the pagination control on the Requisitions UI
  • NMS-9827: Event parameters no longer preserve ordering
  • NMS-10346: default event description is incorrect
  • NMS-10396: Config tester doesn’t detect missing xml datacollection file
  • NMS-11979: Provide DefaultFocus capabilities
  • NMS-12153: Add Web-Hook as delivery option
  • NMS-12197: Flapping KafkaOffsetIT
  • NMS-12288: BSM configuration breaks without being notifed
  • NMS-12294: List Kafka RPC/Sink topics, Expose Metrics on Karaf shell
  • NMS-12307: Add ability to update definitions when SNMP profile changes
  • NMS-12308: Fix security vulnerability with jackson-databind
  • NMS-12315: Update discoveryd to support detectors
  • NMS-12316: Provide criteria for determining the “management IP” in provisiond
  • NMS-12318: Process newSuspect events in a single threaded fashion
  • NMS-12322: Memory leak in Drools engine for alarmd
  • NMS-12324: Minion RPM upgrade deletes /opt/minion
  • NMS-12338: InterfaceNodeCache doesn’t remove deleted nodes immediately
  • NMS-12341: OpenNMS 25 Dependency Still Allows Old PostgreSQL Versions
  • NMS-12343: XMLMarshalException in ipc.log: Unexpected close tag ; expected
  • NMS-12352: Install guide for R-core is broken for CentOS 8
  • NMS-12353: Update base container image to use CentOS 8
  • NMS-12354: Remove floating OpenJDK dependencies in OCI build
  • NMS-12356: Detect and help resolve Karaf bootstrap issues
  • NMS-12359: Karaf feature install issue with opennms-core-tracing-jaeger
  • NMS-12360: Fix requisition cache when accessing the Requisitions UI via “Edit in Requisition”
  • NMS-12362: Update CISCO-ENTITY-SENSOR-MIB threshold trap events to include alarm-data
  • NMS-12363: switch core/web-assets from yarn to npm
  • NMS-12364: Collect and display file descriptor statistics via JMX for OpenNMS and Minion

by RangerRick at November 04, 2019 07:29 PM

October 25, 2019

OpenNMS.js v2.0.1

This is a small release with some dependency updates and a fix for handling of ReST data that is missing a response.

  • dao: handle .fromData when data is undefined (JS-45) (efe6858)
  • dao: warn if lastEvent is missing on an alarm (JS-45) (e4af27d)

by RangerRick at October 25, 2019 07:51 PM

October 21, 2019

OpenNMS On the Horizon – October 21st, 2019 – Bug Fixing, Packaging, Detectors and Discovery, Reporting, Helm, and More!

It’s time for OpenNMS On the Horizon!

In the last week we worked on lots of small bug fixes, packaging updates and improvements, adding detectors to discovery, reporting, and Helm.

Github Project Updates

  • Internals, APIs, and Documentation
    • Markus worked on fixing report delivery to some email addresses.
    • Patrick continued to work on fixing event parameter ordering.
    • I wrapped up my work fixing up the OpenNMS systemd service file as well as adding Minion and Sentinel native systemd services.
    • Ronny fixed up a bunch of compatibility and installation issues in the documentation.
    • I fixed an issue with upgrades when moving to the new monolithic Minion RPMs.
    • Chandra fixed an issue where kafka sync API chunks could be processed twice if a retry has occurred.
    • Christian fixed an issue with alarm clearing and BSM.
    • Chandra continued his work on being able to run detectors as part of the discovery process.
    • Alejandro fixed a configuration issue where the JMX-Cassandra service could be shown as “down” even if only one instance in the cluster is down.
    • Alejandro fixed a bug in the event description for interfaceAdminUp.
    • Patrick worked on cleaning up the rest of our Karaf shell commands to start with opennms for consistency.
    • Ronny worked on a bunch of doc updates to default to CentOS 8 for RPM installs.
  • Web, ReST, and UI
    • Markus worked on fixing Reportd-generated reports not being downloadable in the new UI.
    • Markus worked on adding support for sending a generated report to a URL as a webhook.
    • Markus continued work on his proof-of-concept universal search bar.
    • I worked on fixing Helm to work with Grafana 6.4.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

- Ben

Resolved Issues Since Last OOH

  • NMS-9842: Documentation typo for /rest/ifservices on the developers guide
  • NMS-10027: The JMX-Cassandra service goes down for all the cluster when a single instance is down.
  • NMS-10184: deleting a BSM monitor while an alarm is active doesn’t clear the alarm
  • NMS-10582: snmp authentication error traps with Enhanced Linkd / bridge discovery
  • NMS-12263: Refactor Event Timestamps to ISO-8601 Format (Breaking Change)
  • NMS-12299: Create proper systemd files for OpenNMS, Minion and Sentinel
  • NMS-12302: BSM simulation mode does not reset the last state
  • NMS-12337: Reportd generated reports cause: “No bean named ‘’ is defined” in Persisted Reports
  • NMS-12342: Delivering a report with “-” in local part of email address is not working
  • NMS-12348: Update installation guide to CentOS 8

by RangerRick at October 21, 2019 07:10 PM

October 14, 2019

OpenNMS On the Horizon – October 14th, 2019 – Minion, Sentinel, Date Parsing, SNMP Profiles, Discovery, Search, and More!

It’s time for OpenNMS On the Horizon!

In the last week we worked on a number of bugs mostly related to Minion and Sentinel, plus date parsing, SNMP profiles, discovery, universal search, and more!

Github Project Updates

  • Internals, APIs, and Documentation
    • Marcel and Markus worked on cleaning up the JMX configuration for monitoring JVMs.
    • Patrick continued to work on making sure event parameter order is preserved.
    • Chandra worked on fixing an issue where the Minion could stop sending data over Kafka.
    • Matt continued his work on improving async operations against Cassandra.
    • Chandra worked on adding detector support to discovery.
    • I worked on fixing some packaging issues with Minion and Sentinel, as well as improving systemd startup and shutdown.
    • Christian worked on a number of bugs in BSM’s backend.
    • Matt fixed an issue with clearing threshold states on the Karaf shell.
    • I updated startup to always use the (new) CLDR locale data, which includes a breaking change to event serialization to always use ISO-8601 dates.
    • Chandra did more work on his code to update SNMP definitions when the profile is updated.
    • Chandra worked on fixing some issues in the node/IP cache.
    • Matt fixed an issue with invalid key caching in thresholding state.
  • Web, ReST, and UI
    • Markus revived his work during Dev-Jam on a “universal” search for the OpenNMS web UI.

Release Day Moving

Moving to monthly releases has been a huge success.
It’s made it easier to anticipate upgrades, to tell users when a bug fix will make a release, and it’s (generally *cough*) taken the pressure off to get some specific feature into a release.

However, we originally picked Thursday as a release day because our weekly sprint meeting is Wednesday.
The idea was that we could go over what issues were wrapped up and make sure PRs got merged before the release Thursday.

In reality, Thursday only gave us just enough time to realize which important things wouldn’t make it in and not really have anything to do about it.

The new release day is (drumroll):

The First Tuesday of the Month

This means that we can encourage important PRs to get merged by Friday, which gives CI a chance to run over the weekend, including our weekly run of “Dependency Monkey” which goes through every Maven dependency in the OpenNMS source tree and builds it individually to be sure we don’t have any transient dependencies that can cause build problems.

That gives us Monday to fix any critical issues and get them merged before a Tuesday release.

The first Tuesday release will be November 5th, 2019.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

- Ben

Resolved Issues Since Last OOH

  • NMS-9322: removed service will break BSM web ui
  • NMS-10578: BSM alarm severity is not being updated
  • NMS-10697: Elasticsearch forwarding fails to recover after outage
  • NMS-12274: Improve robustness of CassandraBlobStore for async operations
  • NMS-12277: Clearing threshold states via shell should take effect immediately and not require restart
  • NMS-12307: Add ability to update definitions when SNMP profile changes
  • NMS-12317: instance/group ID is not being honored in some minion/sentinel code
  • NMS-12329: Threshold state keys do not incorporate the collected resource’s instance label

by RangerRick at October 14, 2019 01:36 PM

October 07, 2019

OpenNMS On the Horizon – October 7th, 2019 – Minion & Sentinel, Events and Alarms, Business Service Monitor, Flows, UI Improvements, and More!

It’s time for OpenNMS On the Horizon!

In the last week we worked on the Minion and Sentinel runtimes, event and alarm bugs, BSM, flow updates, UI improvements, and more!

Github Project Updates

  • Internals, APIs, and Documentation
    • Ronny worked on some fixes for running Minion and Sentinel as non-root.
    • Patrick did more work on making sure event parms are sorted correctly.
    • Chandra worked on fixing some issues with exception-handling when sending flows into Kafka.
    • Markus did more work on fixing Elasticsearch forwarding recovery when an outage occurs.
    • I worked on cleaning up some places in the code where the default instance ID wasn’t set up properly, in preparation for Meridian 2019.
    • Christian fixed a few issues with BSM, including alarm severities not updating properly and issues when services are deleted.
    • Markus did some work to wrap up Jeff’s fixes to the Slack notification strategy.
    • Jesse did some work on remote poller modernization.
    • I worked on changing event/alarm time serialization to use ISO-8601 rather than local-based timestamps. (including an update to send-event.pl)
    • I did more work on native systemd configs for Minion and Sentinel.
    • Matt made some improvements to async operations on the CassandraBlobStore.
    • I worked on fixing an upgrade issue with the minion packaging.
    • Marcel Fuhrmann created some configurations to enrich our node asset information using the SNMP Asset Adapter for the following products:
  • Web, ReST, and UI
    • Markus fixed an issue with groups disappearing in the classification UI.
    • I released OpenNMS.js 2.0.
    • Markus did more work on his refresh/improvements to the navbar.
    • Markus fixed an issue where severities were styled differently between the node page and sub-pages.
    • Markus fixed a problem where the meta-data editor wouldn’t show in virtical layout mode.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

- Ben

Resolved Issues Since Last OOH

  • NMS-10564: Shorten log message in SNMP trap event definitions
  • NMS-12104: Flow rest results for top N queries are not returned in the correct order
  • NMS-12128: Rework the current navbar
  • NMS-12259: Flow Classification not functioning as expected
  • NMS-12273: UI for meta-data is only present when using the horizontal layout
  • NMS-12311: Docker image for sentinel runs the process as root
  • NMS-12312: Docker image for Minion runs with UID 1001 but the minion user has UID 997
  • NMS-12320: Web Assets Dependency Rollup 2019-09-24
  • NMS-12321: Availability boxes on node pages including sub pages differ
  • PRIS-151: Allow node-level meta-data

by RangerRick at October 07, 2019 03:42 PM

September 30, 2019

OpenNMS.js v2.0.0

OpenNMS.js 2.0 adds a few new APIs, contains a ton of refactoring and build system updates, and has a number of breaking changes.

Notable Changes:

  • updated CLI libraries to fix/improve table output
  • better, faster build system to improve generated code
  • improved handling of authentication in HTTP implementations
  • support for HEAD requests has been added to the HTTP implementations
  • a number of metadata objects (notably OnmsServer) support .equals() for comparison now
  • property caching on v2 API calls is fixed when interacting with multiple servers/DAOs
  • filter API now supports orderBy and order (and the alarms CLI list command supports it)
  • HTTP parameters in OnmsHTTPOptions can now have multiple values -- for example a parameter with value [ 1, 2 ] will serialize to &foo=1&foo=2

Breaking Changes:

  • The api/Log module now only exports a single, simplified log object; typescript-logging was overly complicated and not really adding much in the way of value. Use .setDebug(), .setQuiet(), and .setSilent() to change the logging level instead.
  • A number of the TypeScript APIs have been clarified to be explicitly nullable (and/or undefined-able) to make strict null- and type-checking validation pass.
  • PropertiesCache and its associated interface, ISearchPropertyAccessor are gone. This only affects you if you have implemented custom DAOs, which is very unlikely. :)
  • The previously deprecated timeout property in AbstractHTTP (and sub-classes) has been removed. Access the AbstractHTTP.options.timeout property directly.
  • The Client no longer keeps a separate copy of the server object. Instead you should access the http.server sub-property directly.
  • A number of API objects are now immutable/read-only to reduce side-effects: OnmsAuthConfig, OnmsEnum, OnmsError, OnmsHTTPOptions, OnmsResult, OnmsServer, Operator, SearchPropertyType, ServerMetadata, TicketerConfig.
    The OnmsHTTPOptions and OnmsServer objects now have builders (use .newBuilder() to create) rather than constructors with a bunch of arguments.
  • The id property on OnmsServer is no longer generated, it is computed based on the contents of the server object and should be repeatably equal if the contents are equal.

by RangerRick at September 30, 2019 08:33 PM

A Low Bandwidth Camera Solution

My neighbor recently asked me for advice on security cameras. Lately when anyone asks me for tech recommendations, I just send them to The Wirecutter. However, in this case their suggestions won’t work because every option they recommend requires decent Internet access.

I live on a 21 acre farm 10 miles from the nearest gas station. I love where I live but it does suffer from a lack of Internet access options. Basically, there is satellite, which is slow, expensive and with high latency, or Centurylink DSL. I have the latter and get to bask in 10 Mbps down and about 750 Kbps up.

Envy me.

Unfortunately, with limited upstream all of The Wirecutter’s options are out. I found a bandwidth calculator that estimates a 1 megapixel camera encoding video using H.264 at 24 fps in low quality would still require nearly 2 Mbps and over 5 Mbps for high quality. Just not gonna happen with a 750 Kbps circuit. In addition, I have issues sending video to some third party server. Sure, it is easy but I’m not comfortable with it.

I get around this by using an application called Surveillance Station that is included on my Synology DS415+. Surveillance Station supports a huge number of camera manufacturers and all of the information is stored locally, so no need to send information to “the cloud”. There is also an available mobile application called DS-cam that can allow you to access your live cameras and recordings remotely. Due the the aforementioned bandwidth limitations, it isn’t a great experience on DSL but it can be useful. I use it, for example, to see if a package I’m expecting has been delivered.

DS-Cam Camera App

[DS-Cam showing the current view of my driveway. Note the recording underneath the main window where you can see the red truck of the HVAC repair people leaving]

Surveillance Station is not free software, and you only get two cameras included with the application. If you want more there is a pretty hefty license fee. Still, it was useful enough to me that I paid it in order to have two more cameras on my system (for a total of four).

I have the cameras set to record on motion, and it will store up to 10GB of video, per camera, on the Synology. For cameras that stay inside I’m partial to D-Link devices, but for outdoor cameras I use Wansview mainly due to price. Since these types of devices have been known to be easily hackable, they are only accessible on my LAN (the “LAN of things”) and as an added measure I set up firewall rules to block them from accessing the Internet unless I expressly allow it (mainly for software updates).

To access Surveillance Station remotely, you can map the port on the Synology to an external port on your router and the communication can be encrypted using SSL. No matter how many cameras you have you only need to open the one port.

The main thing that prevented me from recommending my solution to my neighbor is that the DS415+ loaded with four drives was not inexpensive. But then it dawned on me that Synology has a number of smaller products that still support Surveillance View. He could get one of those plus a camera like the Wansview for a little more than one of the cameras recommended by The Wirecutter.

The bargain basement choice would be the Synology DS118. It cost less than $200 and would still require a hard drive. I use WD RED drives which run around $50 for 1TB and $100 for 4TB. Throw in a $50 camera and you are looking at about $300 for a one camera solution.

However, if you are going to get a Synology I would strongly recommend at least a 2-bay device, like the DS218. It’s about $70 more than the DS118 and you also would need to get another hard drive, but now you will have a Network Attached Storage (NAS) solution in addition to security cameras. I’ve been extremely happy with my DS415+ and I use it to centralize all of my music, video and other data across all my devices. With two drives you can suffer the loss of one of them and still protect your data.

I won’t go in to all of the features the Synology offers, but I’m happy with my purchase and only use just a few of them.

It’s a shame that there isn’t an easy camera option that doesn’t involve sending your data off to a third party. Not only does that solution not work for a large number of people, you can never be certain what the camera vendor is going to do with your video. This solution, while not cheap, does add the usefulness of a NAS with the value of security cameras, and is worth considering if you need such things.

by Tarus at September 30, 2019 06:00 PM

OpenNMS On the Horizon – September 30th, 2019 – UI Updates, Flow Changes, and More!

It’s time for OpenNMS On the Horizon!

In the last week we did more cleanup in prep for Horizon 25, including UI fixes, flow changes, and more!

Github Project Updates

  • Internals, APIs, and Documentation
    • I continued to work on cleaning up systemd startup for Minion and Sentinel.
    • Chandra worked on making it possible to list RPC/sink topics from the Karaf shell.
    • Jesse fixed a potential ConcurrentModificationException in the event translator.
    • Patrick continued to work on making sure event parameters are always ordered correctly.
    • Markus worked on a set of flow classification Karaf shell commands, as well as a number of other flow classification bugs.
    • Dustin fixed an issue processing sFlow packets without IP information.
    • Chandra worked on making SNMP definitions update when the profile has changed.
    • Christian started working on adding Meta-Data support to PRIS.
    • Ronny did a bunch of updates to docker image publishing.
    • I did more work on updating events to use ISO-8601 dates when serializing (and parsing).
  • Web, ReST, and UI
    • Markus did more work exposing health:check info through ReST.
    • Jesse finished cleaning up the Jasper Studio integration.
    • Chandra moved the “Telephone PIN” in the notification UI with related relevant bits.
    • Jesse fixed an issue with timeouts in the new Grafana report client.
    • Christian fixed the Meta-Data ReST API to support JSON output.
    • Ronny updated Jetty to version 9.4.20.v20190813.

OpenNMS September Releases

In September we had Meridian 2017 and 2018 releases, as well as a new major Horizon release: Horizon 25.

OpenNMS Horizon 25

Horizon 25 contains a huge number of bug fixes and enhancements, most notably massive thresholding improvements, big changes to netflow and other streaming telemetry data, reporting and UI updates, plus much more.

For an overview of what has changed since Horizon 24, see What’s New in OpenNMS Horizon 25. For a complete list of changes, see the Horizon 25.0.0 release notes.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • HELM-182: Update icons entity and flow datasources
  • HELM-183: Custom panels fail to render as PNGs
  • NMS-10073: Telephone PIN Field in User Details is Misplaced
  • NMS-12190: Enable flow UDP listener by default on Minion and OpenNMS
  • NMS-12230: Verify Database Reporting Documentation is still up to date
  • NMS-12246: Documenting Thresholding for H25
  • NMS-12272: ReST API for meta-data doesn’t support JSON
  • NMS-12285: Poller take a long time to startup/initialize on large systems
  • NMS-12291: Groups disappear in classification UI
  • NMS-12297: Grafana report generation is timing out
  • NMS-12298: Expose health:check via ReST
  • NMS-12300: Update Minion packaging to use single RPM/Deb
  • NMS-12306: ConcurrentModificationException when cloning events
  • OIA-21: Tag and release OIA v0.2.1
  • PRIS-151: Allow node-level meta-data

by RangerRick at September 30, 2019 04:04 PM

September 26, 2019

OpenNMS Horizon 25.0.0 Released

Release 25.0.0 is the first release in the Horizon 25 series.

It contains a huge number of bug fixes and enhancements, most notably massive thresholding improvements, big changes to netflow and other streaming telemetry data, reporting and UI updates, plus much more.

The codename for 25.0.0 is Baozi.

For an overview of what has changed since Horizon 24, see What’s New in OpenNMS Horizon 25.  For a complete list of changes, see the Horizon 25.0.0 release notes.

by RangerRick at September 26, 2019 08:51 PM

OpenNMS Meridian 2018.1.12 Released

Release 2018.1.12 is a small update to 2018.1.11 that fixes a number of (mostly UI) bugs.

The codename for 2018.1.12 is Hurricane Force.

Bug
  • “Graph all” fails with ‘Request Header Fields Too Large’ error (Issue NMS-8712)
  • No class found exception in OSGi for org.osgi.service.jdbc.DataSourceFactory (Issue NMS-9341)
  • Updating external lists referenced by include-url requires restart OpenNMS (Issue NMS-10071)
  • Telephone PIN Field in User Details is Misplaced (Issue NMS-10073)
  • Cannot change the type of a scheduled outage (Issue NMS-12255)
  • Short outages are not visible in Availability-Bar (Issue NMS-12262)
  • vmwarereqtool freezes and never return to the prompt after is executed (Issue NMS-12265)
  • On the scheduled outage UI, the day is displayed as null for weekly schedule (Issue LTS-233)
  • Using special characters on the scheduled outages name breaks the UI (Issue LTS-234)

by RangerRick at September 26, 2019 05:14 PM

OpenNMS Meridian 2017.1.19 Released

Release 2017.1.19 is a small update to 2017.1.18 that fixes an exception and adds a utility for viewing config changes.

The codename for 2017.1.19 is Ujjain.

Bug
  • No class found exception in OSGi for org.osgi.service.jdbc.DataSourceFactory (Issue NMS-9341)
Enhancement
  • Add a command to show configuration diffs (Issue NMS-12129)

by RangerRick at September 26, 2019 04:12 PM

September 23, 2019

OpenNMS On the Horizon – September 23rd, 2019 – Bug Fixes, Drools, Kafka, Flows, Helm, and More!

It’s time for OpenNMS On the Horizon!

In the last week or so we worked on fixing a bunch of small bugs in Drools, Kafka, flow handling, and more, as well as reporting and Helm updates.

Github Project Updates

  • Internals, APIs, and Documentation
    • Jesse updated our embedded Drools engine to 7.26.0, and improved the session-handling in the DroolsAlarmContext to not require locking, which speeds up processing considerably.
    • Jesse fixed an issue with Kafka alarm synchronization.
    • Markus fixed an issue where health:check would respond that Elasticsearch in opennms-flows was failing when it was not configured.
    • Jesse enabled the flow UDP listener by default on Minion and OpenNMS.
    • Markus updated the flow enrichment code to use our session transaction management utility.
    • Chandra fixed the script provisioning policy and Scriptd to greedily initialize the model objects passed into scripts.
    • Jesse made some updates to Kafka statistics collection.
    • Dustin continued his work on making the Meta-Data DSL work with PSM.
    • Marcel worked on cleaning up some JMX collection names.
    • Dustin fixed flow processing to ignore SFlow data without IP data.
    • I fixed the issue where Minion and Sentinel logs can appear in the root of the filesystem.
    • Markus fixed a bug in flow priority management.
    • Ronny updated the default container images to use RRDtool rather than JRobin.
    • I continued to work on figuring out the best way to handle new DateTime locale processing in JDK11.
    • Jesse updated threshold processing to handle pushed metrics from OIA.
    • Dustin worked on improving Poller startup time.
    • I changed the Minion packages to be a single package like Sentinel.
    • Markus worked on fixing some issues with Elasticsearch forwarding after an outage.
    • David Schlenk added STARTTLS support to the SSLCertExpireMonitor.
    • Patrick made custom SNMP auth handling for Cisco routers configurable.
    • Patrick worked on making sure event parameter ordering is preserved.
  • Web, ReST, and UI
    • Patrick fixed an issue with rendering reports.
    • I added support for ordering entity queries in Helm.
    • I updated Helm to use Grafana’s built-in FontAwesome fonts rather than embedding Ionicons.
    • I fixed the regression where Backshift graphs would not properly restrict zooming.
    • Markus worked on a ReST service for retrieving health check data.
    • Jesse updated the Helm icons for the Flow and Entity datasources.

OpenNMS Releases This Week

Because we were so close to having all blockers closed for Horizon 25, we decided to hold off a week so we could get it out the door rather than just doing another 24 bump. The September releases will be happening this Thursday.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • HELM-130: Dynamic Filtering
  • HELM-172: Update Default Alarm Table Columns
  • HELM-174: Enhance query to support an order by clause
  • HELM-181: Severity icons are no longer displayed
  • JS-44: Fix parameter serialization in GrafanaHTTP
  • NMS-8294: Expose the complete OnmsNode object in Scriptd to avoid LazyInitializationExceptions
  • NMS-10498: HealthCheck should not fail for ElasticSearch if flows are not enabled
  • NMS-10605: Syslog incorrect dates being parsed into database
  • NMS-10635: Zooming with Backshift is broken
  • NMS-10662: Update default notification configuration to leverage alarm autoacknowledgements
  • NMS-10672: Redesign default threshold events
  • NMS-10704: Insecure cache-control for session cookies
  • NMS-10724: Kafka alarm producer synchronization fails due to missing event
  • NMS-11719: Refactoring Docker Container Images towards Horizon 25
  • NMS-11857: Improve visibility of IPCs (Monitor Sink/RPC)
  • NMS-12125: karaf.log appears on the root file system when running Minion/Sentinel on Ubuntu/Debian.
  • NMS-12185: sFlow adapter drops flows due to NPE
  • NMS-12229: Ensure the reporting jar for jasperstudio is still working and containing all the relevant jars
  • NMS-12244: Remove session lock from Drools in alarmd
  • NMS-12247: Add support for meta-data fields in thresholding expressions
  • NMS-12252: Add support for STARTTLS to TLS certificate monitor
  • NMS-12258: DefaultReportWrapperService: failed to run or render report
  • NMS-12268: ScriptPolicy scripts encounter LazyInitializationException on calling node.getSnmpInterface()
  • NMS-12271: allow using fetched RPMs in build_container_image.sh
  • NMS-12283: Be able to execute ICMP requests through Java when running as non-root
  • NMS-12286: Ordering of classification groups does not work
  • NMS-12287: Wrong decoding of hex string
  • NMS-12290: Default settings in container image don’t write RRD files
  • NMS-12293: The Debian APT repositories for branches looks incomplete
  • NMS-12301: Default “90% Interface Throughput” Threshold Doesn’t Work
  • OIA-14: Extend Threshold configuration

by RangerRick at September 23, 2019 09:04 PM

September 16, 2019

OpenNMS On the Horizon – September 16th, 2019 – Poller and Thresholding, UI Fixes, Provisioning, Startup, and More!

It’s time for OpenNMS On the Horizon!

In the last week or so we worked on poller and thresholding updates, more UI fixes, provisioning, startup fixes and enhancements, and more!

Github Project Updates

  • Internals, APIs, and Documentation
    • Chandra backported support for refreshing include-url when a reload event is sent to Pollerd.
    • Patrick did more work on the classification rules reordering backend.
    • Jeff added collection and graph support for Netscaler vServer.
    • Matt worked on performance improvements to thresholding.
    • Jesse did more work on making it so unit and integration tests run a subset based on changes, rather than the full suite.
    • Chandra fixed an initialization issue in ScriptPolicy when interacting with the node object.
    • Christian fixed a deadlock issue in vmwarereqtool
    • I increased the default heap size in Horizon 25 to 2GB.
    • Chandra fixed a bug in inconsistent ipHostName handling when (re-)provisioning a host.
    • Ronny updated the default notifd configuration to use default auto-acknowledgment.
    • Chandra worked on fixing WS-Man under JDK11.
    • Matt added a Karaf command-line tool for managing thresholding persistent state.
    • Jesse fixed an issue with wiring and Newts.
    • Ronny fixed an issue with an incorrect PID file when spawning OpenNMS.
    • Jesse worked on migrating the resourcecli tool to Karaf shell commands.
    • Ronny added a healthcheck script to the Minion and Sentinel containers.
    • Ronny fixed some issues with running as non-root in the OpenNMS containers.
  • Web, ReST, and UI
    • I did more work on supporting “order by” in Helm (and OpenNMS.js) entity queries.
    • Christian fixed an issue in the scheduled outage editor when using special characters in outage names.
    • Patrick fixed a bug in cache-control and session cookie headers.
    • Christian fixed a bug in deleting in the scheduled outage ReST API.
    • Matt fixed an issue in the scan reports ReST API.
    • I fixed a problem in OpenNMS.js with processing parameters with multiple values.

FYI: Mattermost Chat Security Update

If you are using the Mattermost desktop client to connect to chat.opennms.com, make sure you have updated to the latest version. They have released a security update for a remote code execution exploit.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • HELM-180: Flow Histogram – Not showing Legend
  • JS-43: Update Outdated Dependencies
  • NMS-10071: Updating external lists referenced by include-url requires restart OpenNMS
  • NMS-10278: Neutralise the docs stylesheet
  • NMS-11721: Replace custom bash templating with Confd and streamline entrypoint script
  • NMS-11830: CircleCI
  • NMS-11834: Thresholding for streaming metrics & more
  • NMS-11836: Project Drift Enhancements
  • NMS-11844: Grafana Integration
  • NMS-11861: Enterprise Reporting Enhancements
  • NMS-11975: Grouped flow classification rules
  • NMS-11998: Top N hosts (flows)
  • NMS-12084: Trapd & Syslogd Kafka Support
  • NMS-12212: Show configured rule groups
  • NMS-12213: Edit (CRUD) rule groups via rest & gui
  • NMS-12216: JMX Metrics for Newts disappeared
  • NMS-12226: Wrong PID in opennms.pid
  • NMS-12233: Dependency based testing
  • NMS-12235: WS-MAN doesn’t work with JDK 11
  • NMS-12238: Update minimum required version of PostgresSQL for H25
  • NMS-12243: Leverage telemetry sequencing for better thresholding performance
  • NMS-12248: Manage persisted threshold state via shell
  • NMS-12249: DNS resolved hostname should always be reflected in ‘iphostname’ field in ipinterface table
  • NMS-12255: Cannot change the type of a scheduled outage
  • NMS-12265: vmwarereqtool freezes and never return to the prompt after is executed
  • NMS-12267: EnhancedLinkd-Thread-x-of-x – Null Pointer Exception
  • NMS-12270: Remove old comment from data source configuration
  • NMS-12275: Fix usage description of vmwarereqtool
  • NMS-12276: Increase default heap size to 2GB
  • NMS-12278: Add documentation about classification rule & group sorting
  • NMS-12280: Migrate resourcecli to the Karaf shell
  • NMS-12281: Add Karaf shell command to view raw sample stored in Newts

by RangerRick at September 16, 2019 04:36 PM

September 09, 2019

OpenNMS On the Horizon – September 9th, 2019 – Flows, CI, Threshd, UI Fixes, and More!

It’s time for OpenNMS On the Horizon!

In the last week or so we worked on flows, speeding up tests, thresholding config sharing, a few web UI fixes, and more.

Github Project Updates

  • Internals, APIs, and Documentation
    • Matt updated the sFlow adapter to set the sequence number.
    • Jesse did more work on making it so CI tests only run the subset that have changed based on previous branches.
    • Dustin continued his work making the meta-data DSL work with the PageSequenceMonitor.
    • I worked on refactoring some common shell code to a separate project, and making it so Minion and Sentinel can launch differently depending on JVM.
    • Chandra fixed Pollerd to reload on a reloadDaemon event.
    • Matt continued his work on persisting and sharing Threshd configuration.
    • Marcel worked on splitting up the default JMX datacollection config into separate files.
  • Web, ReST, and UI
    • Chandra worked on wrapping up fixes to handle “Request Header Fields Too Large” errors.
    • Patrick worked on a caching/cookie issue with web sessions.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • NMS-9341: No class found exception in OSGi for org.osgi.service.jdbc.DataSourceFactory
  • NMS-12237: Ops wallboard hanging due to Vaadin session deadlock
  • NMS-12238: Update minimum required version of PostgresSQL for H25
  • NMS-12256: Resource Graphs : Graph Selection fails to render when many resources are selected
  • NMS-12264: Rewrite test in ScheduleOutageIT for new UI
  • NMS-12266: split shell utilities into a separate module

by RangerRick at September 09, 2019 04:10 PM

September 04, 2019

OpenNMS On the Horizon – September 4th, 2019 – Thresholding, JDK, PostgreSQL, Bugs, Helm, and Web UI!

It’s time for OpenNMS On the Horizon!

In the last week or so we worked on Thresholding, JDK and PostgreSQL updates, bug fixes, Helm and OpenNMS.js, and the web UI.

Github Project Updates

  • Internals, APIs, and Documentation
    • Matt did more work on enabling thresholding on Sentinel and persisting threshold state.
    • Matt wrapped up the core of the key/value store using PostgreSQL.
    • I worked on bumping Horizon 25’s minimum PostgreSQL version to 10.
    • I worked on fixing date serialization in OpenJDK 11 to match JDK 8’s.
    • Matt worked on a Karaf shell tool to benchmark performance of the blob store.
    • Chandra worked on fixing IP hostname getting clobbered during provisioning.
  • Web, ReST, and UI
    • I added orderBy support to OpenNMS.js filters, as well as support for repeating URL parameters that have multiple values.
    • I did a bunch of work on Helm, including bumping Helm 4’s minimum Grafana to 6.0, fixing CSV and Excel rendering, severity theme updates, a bunch of UI and bug cleanups, and more.
    • Chandra worked on making it so “graph all” doesn’t cause a long URL exception.
    • Christian fixed an issue with deleting scheduled outages.
    • Patrick did more work on grouping and reordering flow classification rules.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • HELM-161: Search Panel should never refresh
  • HELM-164: after first autocomplete, variables no longer show all options when first clicked in filter panel
  • HELM-165: allow customizable label in filter panel
  • HELM-170: The properties in the filter panel need a max width
  • HELM-175: Poor alarm query and rendering performance with 3000 alarms
  • HELM-178: Change minimum Grafana Dependency to 6 in Helm 4
  • HELM-179: add more severity themes for Helm
  • JS-42: improve navigation of generated typedoc
  • NMS-8712: “Graph all” fails with ‘Request Header Fields Too Large’ error
  • NMS-12181: Postgres KeyValueStore implementation via JDBC
  • NMS-12217: Can’t change Minion location in the Provisiond UI
  • NMS-12242: Invalid timestamps used for CollectionSets generated by telemetryd
  • NMS-12245: Karaf shell command for benchmarking performance of BlobStore

by RangerRick at September 04, 2019 03:04 PM

September 01, 2019

The OpenNMS Group Turns 15

Fifteen years ago today, on September 1, 2004, David Hustace, Matt Brozowski and I formed The OpenNMS Group, Inc.

This was the fourth business entity to steward the OpenNMS Project, and would turn out to be the one with staying power.

The original OpenNMS Group office was in a single 10 foot by 15 foot room with just enough space for three desks. The landlord provided Internet access. By adopting the business plan of “spend less money than you earn” we managed to survive and grow. Now the company has its main office in Apex, NC, USA as well as one in Ottawa, Ontario, CA, with a satellite office in Germany.

The OpenNMS platform is being used to monitor some of the largest networks in existence, many with millions of devices. With the introduction of ALEC the team is bringing artificial intelligence and machine learning technologies to network monitoring to provide the highest level of visibility to the most complex environments.

OpenNMS has always been lucky to have a wonderful community of users, contributors and customers. With their support the next fifteen years should be as great if not better than the first. I am humbled to have played a small part in its history.

by Tarus at September 01, 2019 05:10 PM

August 29, 2019

Crash

It’s been even longer than usual since I’ve updated this site. I’m missing a ton of stuff, including the last day of Dev-Jam as well as my trip to this year’s OSCON conference in Portland. I wouldn’t be surprised if I lose one if not all of my three readers.

But I do have an excuse. This happened.

Crashed F150 Pickup Truck

On Friday, July 26th, I left my farm in Chatham County, North Carolina, to head to town. I needed to get the oil changed in the F150 and I was planning on meeting some friends for lunch.

About three miles from my house, another driver crossed the centerline on Hwy 87 and hit my truck nearly head-on. I suffered a broken rib, a fractured C2 vertebrae, and a fractured right big toe, but the major damage was that my left ankle was shattered.

I’ve spent the last 33 days at the UNC Medical Center in Chapel Hill, where I underwent two surgeries and was taken care of by some amazing staff.

I’m home now and plan to return to work (remotely) next week. I still have many months to go before I can approach normality, but a journey of ten thousand miles begins with a single step.

Thanks for your kind thoughts. One good thing that has come out of this is that I’ve spent the last 17 years trying to build OpenNMS into something that can thrive even without me, and the team has been amazing in my absence. I can’t wait to be at full strength again.

by Tarus at August 29, 2019 05:47 PM

August 28, 2019

WinRM in OpenNMS

WSMAN support has been present in OpenNMS for a while. The main motivator for adding support of this was Dell iDRAC, but it is Microsoft Windows infrastructure the one that can have more benefits from it, as an alternative of SNMP to gather statistics from Windows.

Through WinRM, everything exported by WMI is accessible (if the user has the right permissions), which makes it a more attractive solution that using WMI directly.

The challenge is always, how to easily configure the Windows Infrastructure in terms of user authentication. Fortunately, thanks to Active Directory we have GPOs (Group Policy Objects), which can be used to configure a set of Windows machine from a central place.

Traditionally, WinRM (which stands for Windows Remote Management) is just a tool used by PowerShell to have remote access to a Windows machine to perform administrative tasks. For this reason, the protocol and the access to WMI statistics were meant for administrators.

That said, when speaking about monitoring Windows infrastructure, there is no need to have a privileged account, as the purpose of the monitoring tool will be only read statistics, and never perform changes.

Using an administrator account makes monitoring easy, but it can be seen as a security breach. This has been the reason why the WMI Tools in OpenNMS are rarely used, as the credentials have to be stored in plain text inside a configuration file.

Fortunately, to use WinRM, Kerberos has to be the algorithm to use when using Domain Accounts, as basic authentication is only allowed for local administrators. If AD and GPOs are going to be used to facilitate deployment, for sure a Domain Account will be used for WinRM, so Kerberos is going to be mandatory.

With Kerberos, OpenNMS and the WSMAN library will use GSS to negotiate authentication, and the only requirement is having an initialized KeyTab. That means, there is no need to store the credentials in plain text on a configuration file. Of course, the KeyTab file should be protected to make sure it won’t be misused to compromise the Windows infrastructure.

Using privileged Domain Account (or accounts with administrator access) can make the Windows Administrators nervous, so it is reasonable to consider using a non-domain account.

In terms of WinRM access, this is entirely possible, but the problem is going to be having access to WMI.

The changes required to allow a non-admin account to perform WMI queries cannot be done through a GPO directly, meaning that this GPO would have to distribute a PowerShell script that will be executed through the Task Manager, to perform all the changes required locally to allow non-users to have access to WMI, as explained in this article.

This article is not about explaining how to configure the Domain Account, is about configuring WinRM with Kerberos on Linux to monitor Windows Server using OpenNMS, so those details are not going to be explained here, and all the upcoming context will assume that there is an account with WinRM and WMI access (regardless if it is an administrative account or not).

DNS

To use Kerberos, DNS should be perfectly working on the Linux machine where OpenNMS is running; otherwise, it will be impossible to test connectivity, verify authentication and use WSMAN in general.

Usually, the same DNS used by the Domain Controller should be used on the OpenNMS server, and it should be able to perform forward and reserve lookups against any Windows Server that will be monitored by OpenNMS.

This is very important for the following reason:

Kerberos only work with FQDNs, not IP addresses, while OpenNMS only work with IP addresses not FQDNs

This might sound contradictory, but fortunately, that is not the case, and DNS plays an important role here.

So, as FQDNs will be specified when performing any Kerberos related operation, the DNS should be able to properly translate FQDNs to IP addresses the same way the Domain Controller would do it.

When configuring OpenNMS, usually through requisitions, the operator declares nodes and set at least one IP Address to the node. Usually, this IP is marked as primary if SNMP is going to be used, but with WSMAN, it is crucial that a reverse lookup against that IP resolves to the same FQDN used by the Domain Controller to recognize the Windows machine in question; otherwise Kerberos authentication is going to fail.

In other words, the WSMan library within OpenNMS will translate that IP into an FQDN, and if that operation succeeds, it will use that Hostname to perform the WinRM request, and Kerberos will work if the FQDN is valid from Active Directory’s perspective.

Here is a simple example:

Let’s say that the domain is called mynet.local, and the server that it is required to access is called winsrv01, its FQDN would be winsrv01.mynet.local.

This is how to validate if DNS works on the OpenNMS server:

Copy to Clipboard

As shown in the above output, both, a forward and a reverse lookup are working. That means, the requisitioned node should use that IP to detect and collect data through WSMAN using WinRM, for example:

Copy to Clipboard

NOTE: The node should only have the interface marked as primary if SNMP is intended to be used. Also, there is no need to force the WSMAN service. It was done to demonstrate how to define the node properly.

Verify connectivity using the wsman CLI

For simplicity, I’m going to assume that the Operating System where OpenNMS is running is CentOS/RHEL 7.

To install the wsman CLI:

yum install wsmancli

The command should be part of the core packages of the OS.

Then, we also need the Kerberos utilities:

yum install krb5-workstation

The command should be part of the core packages of the OS.

Now that we have DNS correctly configured, the required tools are installed, and there is a domain account with WinRM/WMI access, we can verify the installation.

Of course, before use Kerberos, the Kerberos Realm must be configured.

This can be done by overriding the main configuration file (i.e. etc/krb5.conf, or by extending it by adding the following to /etc/krb5.conf.d/winrm.conf

Copy to Clipboard

The configuration requires knowing the domain (i.e. mynet.local), the realm that Kerberos will be used (i.e. MYNET.LOCAL), and the FQDN of the Domain Controller (i.e. dc1.mynet.local).

Now, we’re ready to initialize the Ticket Cache.

We should use the username in Kerberos format, which is USERNAME@KERNEROS_REALM, let’s assume that the username we got from the Windows Team is operator, so the username will be operator@MYNET.LOCAL.

To initialize the cache:

kinit operator@MYNET.LOCAL

The above will ask for the user’s password. If everything went well, it should show something like this:

Copy to Clipboard

Now that we got the TGT (Ticket Granting Ticket), represented by the krbtgt service, we can use the wsman command:

Copy to Clipboard

Here is an explanation about the options:

a) -d 6

Enable DEBUG.

b) -h winsrv01.mynet.local

The FQDN of the target machine from which it is desired to get statistics. Keep in mind that this FQDN should be the same used when the server joined the domain in Active Directory.

c) -P 5985

The WinRM Port. For now, HTTP will be used. HTTPS is recommended, but that increases the level of complexity of the solution, which is why it is going to be omitted.

d) --username operator@MYNET.LOCAL

This is an optional parameter with the username in Kerberos format as mentioned before. If omitted, the tool will prompt for it.

f) --password "password_here"

This is an optional parameter with the user’s password. If omitted, the tool will prompt for it.

g) -y gss

To enable GSS/Kerberos authentication, or instruct the command to initiate GSS/Kerberos negotiation.

f) enumerate http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_OperatingSystem

To execute a WinRM enumerate command against the provided WMI resource.

If the above command works, a bunch of information about the server will be displayed. The output itself can be huge, so the following includes a partial content as an example:

Copy to Clipboard

The good news is that if the above output is displayed, that means that WinRM, as well as WMI, are properly configured. If not, the Windows Team should verify the permissions, depending on the error reported by the tool.

Remember, for a non-admin account, having access to WinRM doesn’t guarantee access to WMI. This is only guaranteed when using privileged users (or administrator accounts).

NOTE: Unfortunately, the command requires the username and the password even if the authentication is going to be GSS. Fortunately, this is just for testing.

Verify connectivity using the WSMAN Java Library

The idea here is to use the WSMAN library that OpenNMS uses directly from the command line to verify that the JDK can communicate with the server using WSMAN and Kerberos.

Most of the configuration files required for this CLI are going to be the same that OpenNMS will be using, so the location of these files will reflect this fact.

Previously, we’ve used the OS Kerberos Cache. Even if it is possible to do this with Java, it is advised to use a Kerberos KeyTab file instead.

To generate and initialize the KeyTab the credentials of the WinRM/WMI user are going to be required.

NOTE: If an administrator account is going to be used, and the Windows Team doesn’t want to share the credentials, they should be able to generate the KeyTab file and initialize it for you on any Windows server. That way, they can provide the KeyTab to you (without the credentials). Unfortunately, that also means that the wsman command won’t be used to verify the communication and the operator must rely on the Java tool for this purpose.

To generate the KeyTab file, use the following command:

kutil

After running the command, a special prompt will be displayed. When this happens, introduce the following command:

addent -password -p --username operator@MYNET.LOCAL -k 1 -e RC4-HMAC

Note that the username must be provided in Kerberos format. After hitting enter at the end, the tool will ask for the password.

Save the Keytab File by entering the following command and press enter:

wkt /opt/opennms/etc/winrm.keytab

Finally, exit the KUtil CLI by entering q and press enter.

Now that we have a KeyTab, we should initialize it to store the TGT on it:

kinit operator@MYNET.LOCAL -kt /opt/opennms/etc/winrm.keytab

The tool will request the user’s password.

Now that the KeyTab exist, we need to provide the Kerberos configuration. The following is similar to the initial one but includes additional information and should be stored at /opt/opennms/etc/krb5.conf (which is the equivalent of overriding /etc/krb5.conf with everything requried):

Copy to Clipboard

Finally, we need to instruct the WSMAN Client how to authenticate the requests. For this purpose, create a file called /opt/opennms/etc/winrm-login.conf with the following content:

Copy to Clipboard

Note that it has a reference to the principal (i.e. the username in Kerberos format), and more importantly, the location of the KeyTab created earlier.

debug=true is optional and can be used to understand what’s going on during the Kerberos negotiation and authentication phase (it helps for troubleshooting).

At this point, we have everything required to use the Java Client.

It is recommended to download the latest version of the WSMAN CLI for Java from the following link:

https://github.com/OpenNMS/wsman/releases/download/v1.2.3/org.opennms.core.wsman.cli-1.2.3.jar

Here is the syntax for the Java client to get the same content as the wsman CLI we’ve used before:

Copy to Clipboard

IMPORTANT: By the time this document was written, the above tool only works with Java 8. Java 11 is not supported due to JAXB issues, as reported here. That means, until that problem is fixed, WinRM/WSMAN can only be used with OpenNMS when using either OpenJDK 8 or Oracle JDK 8, even if Horizon 24+ or Meridian 2019 are being used.

Note that multiple JVM level settings have to be added when using Kerberos. The following explains each of them:

a) -Dsun.security.krb5.debug=true

This is for enabling DEBUG, for troubleshooting purposes.

b) -Djava.security.krb5.conf=/opt/opennms/etc/krb5.conf

This is to tell Java where is the Kerberos Configuration (with the domain and realm information).

If not specified, it defaults to /etc/krb5.conf

c) -Djava.security.krb5.realm=MYNET.LOCAL

To specify the Kerberos realm to be used.

If not specified, it defaults to the one declared on the kerberos configuration. If it is used, java.security.krb5.kdc must be specified as well.

d) -Djava.security.krb5.kdc=dc1.mynet.local

To specify the FQDN of the KDC or the Domain Controller.

If not specified, it defaults to the one declared on the kerberos configuration. If it is used, java.security.krb5.realm must be specified as well.

e) -Djava.security.auth.login.config=/opt/opennms/etc/winrm-login.conf

The location of the authentication handler we’ve created earlier.

For more information, follow this link.

The rest are arguments of the Java CLI

a) -jar /root/org.opennms.core.wsman.cli-1.2.3.jar

This is to specify the JAR that contains the CLI implementation.

b) -gssAuth

To enable GSS/Kerberos authentication via KeyTab

c) -r http://winsrv01.catsnet.local:5985/wsman

The Full URL of the WinRM location for the target host in question. Once again, note that HTTP is being used. For this reason, the port is the default for HTTP 5985, and the path has to be /wsman. Note that the FQDN of the target host is used (i.e. winsrv01.mynet.local).

d) -resourceUri http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_OperatingSystem

Similar to the wsman CLI, that’s the URL of the WMI information of interest.

e) -o enum

To specify to the Java CLI that we’re going to run a WSMAN enumeration

f) -w WSMAN_1_0

The version of WSMAN. This is very important as it seems that Microsoft only supports version 1.0, which is not the default of the Java CLI.

g) -vvv

To increase verbosity for troubleshooting purposes.

When executing the command, it shows a lot of information, even more than the wsman command (especially for the Kerberos authentication). In the end, the output is the same. It shows the SOAP response, but this client also enumerates it in plain text to facilitate reading.

Configure OpenNMS

Most of the required configuration files should be present:

  • /opt/opennms/etc/krb5.conf
  • /opt/opennms/etc/winrm-login.conf
  • /opt/opennms/etc/winrm.keytab

The only additional file would be /opt/opennms/etc/opennms.conf, and this file should include all the JVM level properties used on the Java CLI, for example:

Copy to Clipboard

As mentioned before, this link provides more information about the Kerberos settings at JVM level.

If all the tests were successful, and the inventory of the Windows machines have been added to be consistent with the DNS resolution, OpenNMS is now ready to collect WMI statistics through WinRM using Kerberos authentication against a Domain Account.

The default configuration contains basic metrics, but there are lots of room for improvements and use specialized metrics for Windows applications like Microsoft Exchange, Active Directory, etc.

by agalue at August 28, 2019 08:11 PM

August 26, 2019

OpenNMS On the Horizon – August 26th, 2019 – Kafka & JMS RPC, Reporting, Polling, SNMP, Horizon 25 Polishing, and More!

It’s time for OpenNMS On the Horizon!

Sorry about missing last week’s OOH. Just as I was feeling better post-surgery, I got knocked out with a bug and a persistent multi-day 101-degree fever. Starting to feel more human again.

In the last two weeks we did a lot more bug fixing and polishing in preparation for Horizon 25, including RPC fixes and improvements, reporting updates, and polling and SNMP agent config improvements.

Github Project Updates

  • Internals, APIs, and Documentation
    • Dustin did more work on the branch to fix flow data graphing.
    • Jesse made a few more small changes to his improvements to JMS RPC before release.
    • Jesse did some final wrapup for the high-performance name resolution backend.
    • Christian did more work on fixing the use of metadata in the PageSequenceMonitor.
    • Chandra did more work on supporting “profiles” for SNMP agent configuration.
    • Jesse worked on a tool to run only unit tests that have changed since an associated “base” branch.
    • Jesse fixed polling to actually update the lastgood and lastfail service poll columns.
    • Sean worked on cleaning up DNS dependencies.
    • Chandra updated Kafka RPC to process RPC responses asynchronously.
    • Jesse and Markus continued to work on speed and stability improvements to integration and smoke tests.
    • Christian added support for setting the Reply-To: field in notification emails.
    • Chandra improved the logging on elasticsearch bulk request failures.
    • Matt continued his work on persisting and sharing thresholding information.
    • Matt worked on a Karaf command to aid in testing and benchmarking the new key/value store.
  • Web, ReST, and UI
    • Patrick worked on wrapping up configurable rule ordering the flow classification UI.
    • I did more work on cleaning up code & documentation and prepping for OpenNMS.js 2.0 and Helm 4.0.
    • Christian fixed date handling in the ops board criteria component.
    • Markus did more work wrapping up his reporting UI changes plus a few more features, including time zone support, and multiple recipients for scheduled reports.
    • Chandra worked on fixing graphing (and bookmarking) large numbers of elements in one query.

Meridian and Horizon August Releases

August marked what is likely the last Horizon 24 release (24.1.3, WOPR), with 25 rapidly polishing up in preparation for next month. 24.1.3 got a bunch of bug fixes and a few small enhancements.

Additionally, Meridian 2018.1.11 was released, with a few small changes.

For a complete list of changes, see the release announcements:

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • ALEC-77: Upgrade Alec Kafka Components to 2.3.0 on Scala 2.12
  • HELM-167: Number values in Helm do not right align
  • HELM-171: Update Helm to use dependabot
  • JS-41: make API objects immutable
  • NMS-10708: Web Server HTTP Header Internal IP Disclosure – (CVE-2000-0649)
  • NMS-10721: Netflow 9 graphs do not line up with MIB2 data
  • NMS-12085: Initial support for Netflow in OpenNMS
  • NMS-12140: Store and read thresholding configuration from PostgreSQL
  • NMS-12148: Consider timezone when scheduling reports
  • NMS-12149: Allow multiple recipients when delivering/scheduling a report
  • NMS-12152: Allow for searching a dashboard
  • NMS-12167: User conceptualizes profile-fitting for SNMP across enterprise
  • NMS-12180: Create feature to install thresholding service on sentinel
  • NMS-12188: BulkRequests should log more details of why it failed
  • NMS-12203: Upgrade Kafka components to 2.3.0
  • NMS-12215: Add enhancements to DNS Resolver and Circuit Breaker
  • NMS-12222: Improve diagnostics and resiliency of JMS RPC (w/ embedded ActiveMQ)
  • NMS-12223: Date handling broken in OpsBoard’s criteria builder component
  • NMS-12224: Add reply-to field to notification emails
  • NMS-12225: Track and display timestamps for the last polls
  • NMS-12227: interfaceresolv event token is incorrect
  • NMS-12232: Kafka RPC: Slow callbacks cause lag on response topics
  • NMS-12236: Kafka RPC: StackOverflowError while unmarshaling causes processing to halt
  • NMS-12240: Report scheduler should be disableable
  • NMS-12241: Fix flapping DatabaseReportPageIT

by RangerRick at August 26, 2019 03:05 PM

August 15, 2019

OpenNMS Horizon 24.1.3 (WOPR) Released

Release 24.1.3 is a stable release of OpenNMS. It contains a few enhancements including JMS RPC improvements, as well as a bunch of bug fixes.

For a high-level overview of what’s changed in OpenNMS 24, see What’s New in OpenNMS 24.

The codename for 24.1.3 is WOPR

Bug
  • Possible Memory Leak in Ops Board (Issue NMS-8283)
  • Heatmap in CenterURL not functioning properly (Issue NMS-8884)
  • importFailed exposes the vcenter credentials (Issue NMS-9535)
  • Node label accepts ampersand (&) symbol, but it breaks graphing (Issue NMS-9904)
  • Default eventconf for clogMessageDiscarded has invalid snmphost mask (Issue NMS-10162)
  • Netflow 9 graphs do not line up with MIB2 data (Issue NMS-10721)
  • DNS Requisition import fails if location is specified (Issue NMS-11748)
  • Group Duty Schedules Formatting Issues (Issue NMS-12155)
  • Topology/Geographical map: Header not hidden when displayed inside iFrame (Issue NMS-12166)
  • Nodes and alarms disappear from Geo Map after few seconds (Issue NMS-12171)
  • Kafka RPC shouldn’t set default auto.offset.reset to earliest (Issue NMS-12172)
  • find-java.sh does not canonicalize all symlinks in java path (Issue NMS-12176)
  • Drools engine hangs on shutdown (Issue NMS-12201)
  • Minion can’t be started after installation from RPM/DEB with OpenJDK 11 (Issue NMS-12206)
  • Can’t connect to Karaf Shell in Minion on Debian/Ubuntu (Issue NMS-12211)
  • Date handling broken in OpsBoard’s criteria builder component (Issue NMS-12223)
Enhancement
  • %interface% & %interfaceresolve% variables should resolve nodes managed IP address (Issue NMS-10657)
  • Remove the /etc/default/opennms from Debian/Ubuntu installs (Issue NMS-12183)
  • Improve diagnostics and resiliency of JMS RPC (w/ embedded ActiveMQ) (Issue NMS-12222)
  • Track and display timestamps for the last polls (Issue NMS-12225)

by RangerRick at August 15, 2019 07:24 PM

OpenNMS Meridian 2018.1.11 Released

Release 2018.1.11 is a small update to 2018.1.10 that adds improved diagnostics for JMS RPC and fixes a few other small bugs.

The codename for 2018.1.11 is Violent Storm.

NOTE: This update modifies the custom.properties file in $OPENNMS_HOME/etc. This is generally not a user-modified file and should always be overwritten, but if you are in the unlikely situation of modifying custom.properties in your local environment, make sure you incorporate the update.

Bug
  • DNS Requisition import fails if location is specified (Issue NMS-11748)
  • Drools engine hangs on shutdown (Issue NMS-12201)
Enhancement
  • Improve diagnostics and resiliency of JMS RPC (w/ embedded ActiveMQ) (Issue NMS-12222)

by RangerRick at August 15, 2019 03:23 PM

August 12, 2019

OpenNMS On the Horizon – August 12th, 2019 – Sentinel, Flows, CI, Testing, Reporting, and Helm!

It’s time for OpenNMS On the Horizon!

In the last week we did more work on Sentinel and flow backend code, tests and CI, reporting, and Helm.

Github Project Updates

  • Internals, APIs, and Documentation
    • Jesse did more work stabilizing smoke tests.
    • Sean continued his work improving the CircleCI workflow.
    • Dustin did more work on validating Netflow 9 graphing/data.
    • Ron Roskens continued to work on getting builds/tests running under JDK 11.
    • Dustin fixed the metadata DSL to work with PageSequenceMonitor configs.
    • Ronny worked on converting container images to use Confd rather than custom shell scripting.
    • Jesse worked around a potential deadlock in Drools while shutting down.
    • Sean worked on updating our Kafka components to 2.3.0.
    • Matt continued his work persisting threshold config to PostgreSQL and syncing across Sentinel/OpenNMS.
    • Sean added some configurable enhancements to the new DNS resolver.
    • Jesse worked on improving diagnostics and performance of the JMS (ActiveMQ) RPC.
  • Web, ReST, and UI
    • Markus did more work on simplifying the UI for scheduling reports.
    • I worked on updating OpenNMS.js and Helm to use Dependabot to keep dependencies up-to-date.
    • I added support for customizing column centering in the Helm alarm table.
    • Jesse updated the Grafana reports to show the time range in the header.
    • Jesse fixed some Helm compatibility issues with Grafana 6.3.
    • Patrick continued his work on making filter priority customizable.
    • Christian fixed a memory leak in the ops board.

Tarus Balog Out On Medical Leave

On the morning of Friday, July 26th, a driver crossed the centerline and hit the Ford F-150 Tarus was driving nearly head on. He sustained several injuries included a fractured vertebrae and a crushed left ankle.

He has reluctantly gone on Medical leave and while we hope to have him back soon he will most likely be out until the end of September. He is still checking email but please understand his responses may be delayed while we wish him a full recovery.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • HELM-137: Create a filter panel
  • HELM-166: Grafana integration seems to be broken since v6.3.0
  • JS-36: Update to TypeScript 3 and Babel 7
  • JS-37: remove typescript-logging from OpenNMS.js
  • JS-40: Enable Dependabot on OpenNMS.js
  • NMS-8283: Possible Memory Leak in Ops Board
  • NMS-9535: importFailed exposes the vcenter credentials
  • NMS-10376: Document RPC TTL tuning
  • NMS-12058: Customize priority of flow classification rules
  • NMS-12065: Flow classification role
  • NMS-12150: Add ability to edit existing scheduled reports
  • NMS-12170: Grafana reporting: Render exceptions in images instead of aborting
  • NMS-12194: Update warmerge plugin dependency to 0.5
  • NMS-12195: Enhance Jaeger OpenTracing Logging with tag for thread name
  • NMS-12200: Grafana Reporting: Add time to date range in report header of template
  • NMS-12201: Drools engine hangs on shutdown
  • NMS-12206: Minion can’t be started after installation from RPM/DEB with OpenJDK 11
  • NMS-12207: Minion memory leak related to new DNS lookup functionality
  • NMS-12211: Can’t connect to Karaf Shell in Minion on Debian/Ubuntu

by RangerRick at August 12, 2019 02:20 PM

August 06, 2019

Dev-Jam 2019

Time flies and we have just finished our fourteenth Dev-Jam in fifteen years. We call it the fifth season- a group of full-time contributors, core community members, and power users get together for our annual OpenNMS Developers Conference. Our venue this year was the Yudof Hall on the University of Minnesota campus in Minneapolis. We are very happy and have to thank Mike Huot and his family, who make this happening for all of us very easy.

This year we got 24 people from Australia, the United States, Germany, the UK, and Canada for a whole week together – so what could go wrong?

The Dev-Jam concept is very easy, we use the time to learn, share experiences and experiment on things with others face to face. We work on stuff that we usually don’t have time for and it is not as fun to dig into new things alone. To get things kicked-off, we started this year on Monday with a short introductory tutorial. The idea is to give new attendees entry points to each topic, where to start with development, and how to get things running quickly.

At the end of the week, we make very short presentations to share what we have or haven’t achieved. We recorded these talks so if you are interested in certain topics, you can talk to the right people in our chat. We have added projects in Discourse and videos from results as a demo can be found on Youtube.

by mfuhrmann at August 06, 2019 10:27 PM

August 05, 2019

OpenNMS.js v1.2.0

This is a small release which adds support for a default timeout in GrafanaHTTP, as well as adding X-Requested-With headers to requests (NMS-9783).

Bug Fixes

  • http: add timeout to GrafanaHTTP (22bdd70)

Features

  • rest: NMS-9783: add X-Requested-With header to requests (e803726)

by RangerRick at August 05, 2019 07:57 PM

OpenNMS On the Horizon – August 5th, 2019 – Updating Dependencies, Fixing Bugs, Enhancing Flows and Reporting, Helm Improvements

It’s time for OpenNMS On the Horizon!

In the last week we did a ton of work modernizing some dependencies in OpenNMS and Helm, fixed a bunch of bugs, and continued work to enhance flows reporting, and UIs on the road to Horizon 25.

Github Project Updates

  • Internals, APIs, and Documentation
    • Matt continued his work on persisting threshold state to the database and sharing that state across Sentinels.
    • Jesse worked on wrapping up his high-performance DNS hostname resolution integration.
    • Chandra finished some changes to Kafka initialization settings.
    • Jesse updated the Elasticsearch integration so you can set a prefix, allowing multiple “namespaces” for data persisted to ES.
    • Ronny wrapped up his work to make our continuous integration containers run as non-root. He also made it so branch builds can be tagged in CI for release to dockerhub.
    • Jesse updated our Elasticsearch support to ES 7.2.0. Note that Horizon 25 (and Meridian 2019) will require ElasticSearch 7.
    • Chandra continued his work on an optimized off-heap storage on the Minion (to replace H2).
    • Christian added support for %interface% and %interfaceresolve% on node events, using the primary IP when available.
    • I fixed an issue with find-java.sh failing when java is behind multiple levels of symlinks.
    • Alejandro updated the NXOS GPB parser to provide more info on configuring the Nexus switch to work with it.
    • Jesse fixed the snmphost mask in clogMessageDiscarded in the default eventconf.
    • Sean did more work on improving our CircleCI workflow.
    • Ron Roskens continued to work on getting the build and tests running under JDK 11.
    • Justin Wood contributed a patch to fix provision:show-import when using the DNS importer.
    • Jesse updated our embedded Drools to 7.24.
    • Chandra worked on adding support for “profiles” when attempting to connect to new SNMP hosts.
    • Sean contributed an update to Jaeger tracing with current thread info in the Camel and Kafka IPC bits.
  • Web, ReST, and UI
    • Markus did more work on simplifying the UI for scheduling reports.
    • Patrick updated his branch to make flow classification rule priority customizable.
    • I fixed some issues with DAOs and property caching in OpenNMS.js.
    • I cleaned up some authorization-management code in OpenNMS.js after the recent refactor.
    • I updated OpenNMS.js to use dependabot to update dependencies semi-automatically.
    • Christian fixed a bug with displaying headers in Vaadin apps when embedded in an <iframe> tag.
    • I added support for configuring alignment (left, center, right) in Helm Alarm Table columns.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • HELM-163: Enhance CircleCI workflow to capture test metadata for Helm
  • JS-38: PropertiesCache implementation can return the wrong cache
  • JS-39: basic auth does not initialize properly in GrafanaHTTP
  • NMS-9318: Traps failing to be ingested by Elasticsearch
  • NMS-9702: GenericResourceType is altering index names on class object initialization
  • NMS-10657: %interface% & %interfaceresolve% variables should resolve nodes managed IP address
  • NMS-11720: Run container image as non-root
  • NMS-11748: DNS Requisition import fails if location is specified
  • NMS-11759: Improve the sample content for the NXOS GPB Parser Groovy Script.
  • NMS-11995: Run system tests using OpenJDK 11
  • NMS-12082: Configurable index name for Event & Alarm Forwarder
  • NMS-12099: Update Elasticsearch integrations to support ES 7.x
  • NMS-12154: Easier Schedule Definition
  • NMS-12157: Improve performance of hostname resolution for flows
  • NMS-12163: Add SonarQube Code Coverage to CircleCI Workflow
  • NMS-12166: Topology/Geographical map: Header not hidden when displayed inside iFrame
  • NMS-12172: Kafka RPC shouldn’t set default auto.offset.reset to earliest
  • NMS-12176: find-java.sh does not canonicalize all symlinks in java path
  • NMS-12183: Remove the /etc/default/opennms from Debian/Ubuntu installs
  • NMS-12187: Allow developer to push OCI’s to docker hub from arbitrary branches
  • NMS-12189: The Cron Expression is only shown for reports which are created with the new UI
  • NMS-12191: Upgrade Drools to latest 7.x release
  • NMS-12195: Enhance Jaeger OpenTracing Logging with tag for thread name
  • NMS-12196: Fix typo in sonar.sh
  • NMS-12199: Fix CircleCI Test Gathering for Smoke Tests

by RangerRick at August 05, 2019 04:37 PM

July 29, 2019

OpenNMS On the Horizon – July 29th, 2019 – Thresholding, Bug Fixing, Scheduling UI, and More!

It’s time for OpenNMS On the Horizon!

In the last week we continued to work on getting thresholding into Sentinel, fixed a bunch of minor bugs, worked on a new scheduling UI, and a whole lot more.

Github Project Updates

  • Internals, APIs, and Documentation
    • Patrick did more work on making graph API objects immutable.
    • David and Matt continued their work adapting thresholding to work on Sentinel.
    • Jesse worked on adding instrumentation to Telemetryd to so packet reception rates can be graphed.
    • Patrick worked on flow classification rule prioritization.
    • Christian did more work on fixing some flow persistence.
    • I fixed an issue with find-java.sh and multiple recursive JDK symlinks.
    • Dino fixed an issue with WsMan and ActiveMQ detectors handling configuration properties.
    • Chandra worked on making it possible to import a requisition from the Karaf CLI.
    • Dustin worked on fixing some graphing issues with flow data.
    • Sean did more fixes to his code coverage CircleCI changes.
    • Jesse continued his work on a high-performance DNS resolver for flow (and other) data.
  • Web, ReST, and UI
    • Chandra fixed some issues in the group duty schedule UI.
    • I continued to work on wrapping up my Helm entity and filter panel updates.
    • Christian fixed a potential NPE in the node maps.
    • Markus worked on an update to the report scheduling UI to support easier configuration.
    • Christian fixed an issue with embedding vaadin apps in iframes.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • NMS-12105: AS is stored as signed int in flow doc
  • NMS-12135: Add the ability to trigger a requisition import via the Karaf shell
  • NMS-12143: Share thresholding state across Sentinels
  • NMS-12155: Group Duty Schedules Formatting Issues
  • NMS-12171: Nodes and alarms disappear from Geo Map after few seconds
  • NMS-12175: WsManWQLDetector, ActiveMQDetector missing setBeanProperties
  • NMS-12182: Collect and graph packet reception rates for telemetryd listeners

by RangerRick at July 29, 2019 07:03 PM

July 22, 2019

OpenNMS On the Horizon – July 22nd, 2019 – Bug Fixes, Thresholding, Kafka, OpenNMS & Helm UI Improvements

It’s time for OpenNMS On the Horizon!

In the last week we fixed bugs in preparation for release day, and continued to work on thresholding and Kafka backend code, as well as a number of OpenNMS and Helm UI improvements.

Github Project Updates

  • Internals, APIs, and Documentation
    • Dustin did more work on supporting service wildcarding in the poller configuration.
    • Chandra made a few more fixes to his changes to allow importing requisitions from the Karaf shell.
    • David and Matt continued their work on refactoring thresholding state to be shared across OpenNMS and Sentinel.
    • Jesse worked on implementing a high-performance hostname resolution API.
    • Chandra updated some default Kafka RPC settings.
    • Christian made some updates to fix integer wraps in flow handling code.
    • Sean did more work on supporting Sonar code coverage in our CircleCI build.
    • Patrick worked on making flow classification rule priority customizable.
    • Chandra worked on updating offheap storage to use rocksdb instead of H2.
  • Web, ReST, and UI
    • Chandra fixed a bug in the KSC report search box.
    • Matt added a legend to the new flow histogram panel.
    • Markus did more work updating report UI code.
    • Jesse did more work speeding up the new Grafana report code.
    • I did a bunch of audit updates to JS dependencies in OpenNMS.js and Helm.
    • Markus fixed an issue editing Minion labels in the web UI.
    • I wrapped up my filter panel changes for Helm.

OpenNMS Horizon and Meridian July Releases

July gave us only minor bugfix updates, with the release of Horizon 24.1.2 and Meridian 2018.1.10.

For a list of what’s changed, see the release announcements:

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • NMS-10716: rancid-api version mismatch
  • NMS-11985: Make Graph immutable
  • NMS-12142: Stickiness for distributed processing in telemetryd
  • NMS-12144: Create Jasper report templates for Grafana dashboards
  • NMS-12145: Improve Grafana report rendering speed
  • NMS-12160: Outage timeline is empty
  • NMS-12161: Init script on Debian/Ubuntu does not start
  • NMS-12164: Minion not showing label correctly and label cannot be edited

by RangerRick at July 22, 2019 04:18 PM

July 18, 2019

OpenNMS Horizon 24.1.2 (Watson) Released

Release 24.1.2 is the latest stable release of OpenNMS. It contains bug fixes for flow processing, Debian package startup, the minion UI, and more.

For a high-level overview of what’s changed in OpenNMS 24, see What’s New in OpenNMS 24.

The codename for 24.1.2 is Watson.

Bug
  • HasFlows is not set when node was provisioned after first flow was received (Issue NMS-12106)
  • Wrong Wording for Empty Requisition (Issue NMS-12156)
  • Init script on Debian/Ubuntu does not start (Issue NMS-12161)
  • Minion not showing label correctly and label cannot be edited (Issue NMS-12164)
Enhancement
  • Add a command to show configuration diffs (Issue NMS-12129)

by RangerRick at July 18, 2019 08:24 PM

OpenNMS Meridian 2018.1.10 Released

Release 2018.1.10 is a small update to 2018.1.9 that fixes a KSC report search issue, and adds a tool for evaluating config changes.

The codename for 2018.1.10 is Storm.

Bug
  • KSC Reports search box on home page is case sensitive (Issue NMS-12137)
Enhancement
  • Add a command to show configuration diffs (Issue NMS-12129)

by RangerRick at July 18, 2019 04:16 PM

July 16, 2019

OpenNMS On the Horizon – July 16th, 2019 – CI Updates, Flows, UI and Reporting, and Bugfixes!

It’s time for OpenNMS On the Horizon!

Between the July 4th holiday in the US, and then personal vacation, this one’s getting out pretty late.
Luckily, no one did anything while I was out. :grimacing:

OK, fine, in the last couple of weeks we did a bunch more improvements to continuous integration, continued to work on flow fixes and updates, fixed a bunch of UI bugs and wrapped up some reporting updates, plus a bunch of other stuff.

Github Project Updates

  • Internals, APIs, and Documentation
    • Ronny finished up his scripts to compare configuration changes.
    • Ron Roskens made some changes to update core Maven plugins.
    • Dustin worked on some changes to how flow data is calculated based on session metadata.
    • Ronny did more work on the smoke tests and their Docker containers.
    • Sean finished his work speeding up CircleCI builds.
    • David worked on supporting thresholding of telemetry data on Sentinel.
    • Dustin and Jesse continued their work on supporting hostname lookups for flows.
    • Matt worked on refactoring OIA types out of OpenNMS and into the API.
    • Patrick worked on maging graph service model objects immutable.
    • Ronny worked on changes to the Docker containers to make them runnable on OpenShift.
    • Chandra continued working on his improvements to support large message sizes on Kafka.
    • Patrick did more work adding support for graph service searching.
    • Chandra worked on making it possible to change a node’s location within a requisition.
    • Chandra added support for sending events into OpenNMS from Sentinel.
    • Christian worked on detecting and reporting clock skew in flows.
    • Ron Roskens worked on fixing a regression in the ObjectNameStorageStrategy.
    • Jeff did more work on his project to enumerate system info for support use.
    • Matt worked on making it possible to share thresholding state across Sentinel instances.
    • Ronny fixed the Debian packages to properly find Java 11.
    • Chandra worked on refactoring some of the RPC code to make it more manageable.
    • Jesse fixed a number of flapping tests.
    • Chandra added support for triggering a requisition import from the Karaf command line.
  • Web, ReST, and UI
    • Markus continued his work on a universal Spotlight-like search box.
    • I worked on an OpenNMS.js HTTP plugin using cordova-plugin-advanced-http rather than AJAX.
    • Matt finished up and merged the flow histogram panel for Helm.
    • Chandra updated the time-series graph page to lazy-load graphs and support search-in-page.
    • Markus added an improved date/time chooser to the web UI.
    • Jesse did more work on supporting Graphana dashboards as reports.
    • Markus finished the work removing the old report UI in favor of the new JS-based one.
    • Christian fixed an issue that could cause the node detail outage timeline to be empty.
    • Christian added support for filtering alarms by surveillance category.
    • Markus fixed an issue with minion labels not showing/being editable.
    • Chandra fixed some issues with the duty schedule UI.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • ALEC-72: Deep learning engine vectorizer does not calculate distance correctly
  • ALEC-73: Performance improvements for DBScan engine
  • ALEC-75: Upgrade Alec Kafka Components to 2.2.0 on Scala 2.12
  • HELM-159: CSV / XLS export in alarm table panel
  • HELM-160: Move timestamps to top of alarm table details dialog
  • HELM-162: Add legend controls to the “flow histogram” panel
  • NMS-7188: Add a JMS broker monitor
  • NMS-10695: Filter alarm list by surveillance categories
  • NMS-11723: Replace current (Spring Webflow) Reporting UI with an angular implementation
  • NMS-11776: Design thresholding support for telemetryd on Sentinel
  • NMS-11797: Thresholding support for push metrics
  • NMS-11993: Skew detection for flow timestamps
  • NMS-12053: Dynamically change a node’s location
  • NMS-12080: Support wildcards in service names
  • NMS-12087: Hostname resolution for flows
  • NMS-12092: Provide search capabilities
  • NMS-12095: KPIs for resource graphs
  • NMS-12129: Add a command to show configuration diffs
  • NMS-12133: Allow tarball installation in Docker images for Horizon
  • NMS-12136: Enhance CircleCI workflow to reduce build times and tests
  • NMS-12137: KSC Reports search box on home page is case sensitive
  • NMS-12141: Add support for sending events from Sentinel
  • NMS-12147: Implement easier datetime selection
  • NMS-12156: Wrong Wording for Empty Requisition
  • NMS-12158: Remove Legacy Database Reports (Web Flow)
  • NMS-12159: UsersRestService: /users/whoami returns 404 on ldap authenticated user (not in users.xml)
  • OIA-17: Move implementations for API types from Opennms to integration API
  • OIA-19: Fix TicketerPlugin to return Ticket ID that is created

by RangerRick at July 16, 2019 08:51 PM

July 01, 2019

OpenNMS On the Horizon – July 1st, 2019 – Dev-Jam! Dev-Jam! Dev-Jam!

It’s time for OpenNMS On the Horizon!

Last week was Dev-Jam. There’s a lot to catch up on, so here we go!

Github Project Updates

  • Internals, APIs, and Documentation
    • Christian worked on some fixes to processing of Netflow v9 flows.
    • Dustin worked on hostname resolution when processing flows.
    • Jesse fixed a concurrent modification issue in the Karaf web bridge.
    • Jesse and David did more work on the Threshd refactor.
    • I fixed the Debian/Ubuntu package dependencies to mirror the way the work in RPM.
    • Markus added a ReST API for a “spotlight” search service.
    • Jeff did documentation updates for the RADIUS plugin, as well as some other doc cleanups.
    • Jesse did more work on smoke tests for the CircleCI builds.
    • Carlo Caminati contributed a fix to the Debian packages for PostgreSQL 11 support.
    • Matt fixed a bug where Sentinel would be missing the integration API on install.
    • Ronny added a health-checking wrapper to the Minion and Sentinel Docker images.
    • Chandra fixed the OIA ticketer plugin to return the ticket ID when it’s created.
  • Web, ReST, and UI
    • Markus worked on a single search box in the web UI for searching nodes, alarms,
      etc. in one place.
    • Chandra worked on adding searching/filtering to the graph results in the web UI.
    • Markus continued his work on the Grafana dashboard report UI.
    • Jesse made some fixes to the new metadata edit UI in develop.
    • Matt wrapped up his histogram panel enhancements for Helm.
    • I added CSV and Excel export support to the alarm table in Helm, as well
      as a few other alarm table UI tweaks.
    • Markus worked on an updated navbar for the web UI.
    • I worked on modernizing/cleaning up our web UI javascript build.

Dev-Jam 2019

Introducing Dev-Jam

Dev-Jam is our annual hackathon and developer conference where the OpenNMS Group, the Order of the Green Polo, and other OpenNMS contributors join each other in one place and work on whatever project tickles our fancy.

This Year at Dev-Jam

This year we were again at the University of Minnesota’s Minneapolis campus.

On Monday, some folks did presentations to launch Dev-Jam, including catching up on the last year of OpenNMS changes, tips on getting started with OpenNMS development, and more. Then we settled in for a week of hacking on code, socializing, and generally having a great time. On Friday, many folks gave presentations on what they worked on for the week.

If you want a feel for what the day-to-day is like during Dev-Jam, our CEO Tarus has been chronicling it on his blog starting at day 0 (Sunday night).

Projects

People worked on all kinds of projects, including but not limited to:

  • finishing a years-long refactor of the database infrastructure in our integration tests
  • modernizing OpenNMS Compass’s build infrastructure and backend code to use OpenNMS.js
  • sending Kubernetes events to OpenNMS
  • integrating PRIS into the OpenNMS core as a Karaf/OIA plugin
  • an Alexa skill for querying alarms and outages from OpenNMS
  • a demo of the MQTT plugin integration
  • a visualization framework for topology and alarms using GL and VR
  • a new “spotlight” search bar that searches nodes, alarms, categories, and more
  • massive performance improvements to ALEC processing
  • a tool for evaluating configuration changes to aid in upgrading
  • CircleCI performance improvements to bring the pipeline time down to a couple of hours
  • a Cisco APIC controller plugin for importing inventory and receiving faults as events
  • building and running tests under OpenJDK 11

Most of the projects had Friday video presentations, and we will work to get those out to you soon.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • HELM-133: Convert the fault datasource to a more general entity (node + alarm) datasource
  • HELM-145: Histogram panel for flows
  • NMS-10652: Failed to fetch store:/var/lib/apt/lists/partial/debian.opennms.org_dists_stable_main_Contents-amd64.gz Hash Sum mismatch
  • NMS-10682: WsMan Asset Adapter config contains an invalid resourceUri
  • NMS-10683: WsMan Asset Adapter should handle WQL filters returning more than one result
  • NMS-10713: WS-Man datacollection fail with OpenJDK11
  • NMS-11766: Sentinel container does not contain integration API in maven repository
  • NMS-12081: Refactor threshd out of opennms-services
  • NMS-12083: Add a UI to edit meta-data in requisition
  • NMS-12107: Migrate system tests to use TestContainers framework
  • NMS-12109: Hash Sum mismatch
  • NMS-12115: Provide a mechanism to collect the count of objects collected by a wsman enumeration
  • NMS-12116: CME in ProxyFilter (Karaf Web Bridge)
  • NMS-12117: Add local cache to reverse hostname resolution in flows
  • NMS-12119: Installing Minion on Debian with OpenJDK 11 installed tries to install Oracle JDK 8
    Wide character in print at scripts/twio-issues-list.pl line 85.
  • NMS-12121: Opennms 24.1.0 on Ubuntu 19.04 won’t install because of PostgresSQL 11
  • NMS-12122: Remove hack from opennms-spring-extender to expose ServiceRegistry
  • NMS-12123: exception adding data from snmp scan
  • NMS-12130: Update Webpack Build to Latest Babel and TypeScript

by RangerRick at July 01, 2019 07:41 PM

June 29, 2019

2019 Dev-Jam – Day 4

The next to the last day of Dev-Jam was pretty much like the one before it, except now it was quite clear that Dev-Jam was coming to a close (sniff).

I actually managed to get some of the work done that I wanted to do this week, namely to start working on the next version of my OpenNMS 101 video series. A lot changed in Horizon 24 and now the videos are a little off (especially when it comes to alarms) and I want to fix that soon.

2019 Dev-Jam: Group of People Hacking Away

I did make one bad decision when I purchased take-away sushi from the Union, but I was lucky that I got over it quickly (grin)

2019 Dev-Jam: Jesse Talking About ALEC

It’s so nice to be able to break out into little groups and share what is going on in OpenNMS. Jesse gave an in-depth talk on ALEC (and I’ll be presenting it at this year’s All Things Open conference).

It wasn’t all work, though.

2019 Dev-Jam: Table with Snacks and Ulf

A group of people had gone to the Mall of America on Sunday, and Markus bought a Rick and Morty card game that seemed pretty popular. Parasites!

For dinner I ordered some delicious pizza from Punch as many people wanted to stay in and finish up their projects in time for tomorrow’s “Show and Tell”.

It’s hard to believe Dev-Jam is almost over.

by Tarus at June 29, 2019 02:24 PM

June 27, 2019

2019 Dev-Jam – Day 3

Not much to add on Day 3 of Dev-Jam. By now the group has settled into a routine and there’s lots of hacking on OpenNMS.

As part of my role as “cruise director” Mike and I ran out for more snacks.

2019 Dev-Jam: Table with Snacks and Ulf

On the way we stopped by the Science Museum of Minnesota to pick up a hoodie for Dustin. As fans of Stranger Things we thought we should get our Dustin the same hoodie worn by Dustin in the show. The one in the show was apparently an actual hoodie sold by the museum in the 1980s, but it was so popular they brought it back.

2019 Dev-Jam: Dustin and Dustin in Brontosaurus Hoodie

While not exactly the “Upside Down” in the evening the gang descended on Up-Down, a barcade located a few miles away. Jessica organized the trip and folks seemed to have a great time.

2019 Dev-Jam: Selfie of Folks at Up-Down.

The combination bar and arcade features vintage video games

2019 Dev-Jam: People Playing Video Games at Up-Down.

as well as pinball machines

2019 Dev-Jam: Selfie of Folks at Up-Down.

Of course, there was also a bar

2019 Dev-Jam: People at the Bar at Up-Down.

Good times.

by Tarus at June 27, 2019 04:52 PM

June 26, 2019

2019 Dev-Jam – Day 2

While the OpenNMS team does a pretty good job working remotely, it is so nice to be able to work together on occasion. Here is an example.

I wanted to explore the current status of the OpenNMS Selenium monitor. My conclusion was that while this monitor can probably be made to work, it needs to be deprecated and probably shouldn’t be used.

I started off on the wiki page, and when I didn’t really understand it I just looked at the page’s history. I saw that it was last updated in 2017 by Marcel, and Marcel happened to be just across the room from me. After talking to him for awhile, I understood things much better and then made the decision to deprecate it.

The idea was that one could take the Selenium IDE, record a session and then export that to a JUnit test. Then that output would be minimally modified and added to OpenNMS so that it could periodically run the test.

The main issue is that the raw Selenium test *requires* Firefox, and Firefox requires an entire graphics stack, i.e. Xorg. Most servers don’t have that for a number of good reasons, and if you are trying to run Selenium tests on a large number of sites the memory resources could become prohibitive.

An attempt to address this was made using PhantomJS, another Javascript library that did not require a graphical interface. Unfortunately, it is no longer being maintained since March of 2018.

We’ve made a note of this with an internal OpenNMS issue. Moving forward the option looks like to use “headless Chrome” but neither OpenNMS nor Selenium support that at the moment.

We still have the Page Sequence Monitor. This is very efficient but can be difficult to set up.

Playing with that took up most of my morning. It was hard staying inside because it was a beautiful day in Minneapolis.

2019 Dev-Jam: Picture of Downtown Minneapolis from UMN

Most of my afternoon was spent working with OpenNMS customers (work doesn’t stop just because it is Dev-Jam) but I did wander around to see what other folks were doing.

2019 Dev-Jam: Jesse White with VR headset

Jesse was playing with a VR headset. The OpenNMS AI/Machine Learning module ALEC can create a visualization of the network, and he wrote a tool that lets you move through it in virtual reality (along with other people using other VR headsets). Not sure how useful it would be on a day to day basis, but it is pretty cool.

That evening most of us walked down the street to a pretty amazing Chinese restaurant. I always like bonding over food and we had discovered this place last year and were eager to return. I think the “bonding” continued after the meal at a bar across the street, but I ended up calling it a day.

2019 Dev-Jam: People at a table at a Chinese restaurant

2019 Dev-Jam: People at a table at a Chinese restaurant

by Tarus at June 26, 2019 01:54 PM

June 25, 2019

2019 Dev-Jam – Day 1

Dev-Jam officially got started Monday morning at 10:00.

I usually kick off the week with a welcome and some housekeeping information, and then I turn it over to Jesse White, our project CTO. We do a roundtable introduction and then folks break off into groups and start working on the projects they find interesting.

This year we did something a little different. The development team scheduled a series of talks about the various things that have been added since the last Dev-Jam, and I spent most of the day listening to them and learning a lot of details about the amazing platform that is OpenNMS. While we had some technical difficulties, most of these presentations were recorded and I’ll add links to the videos once they are available.

2019 Dev-Jam: Graph of Main Projects Over the Last Year

Jesse started with an overview of the main development projects over the last year. Sentinel is a project to use the Kafka messaging bus to distribute OpenNMS functionality over multiple instances. While only implemented for telemetry data at the moment (flows and metrics) the goal is to enable the ability to distribute all of the functionality, such as service assurance polling and data collection, across multiple machines for virtually unlimited scalability.

After the Sentinel work, focus was on both the OpenNMS Integration API (OIA) and the Architecture for Learning Enabled Correlation (ALEC).

The OIA is a Java API to make it easier to add functionality to OpenNMS. While it is used internally, the goal is to make it easier for third parties to integrate with the platform. ALEC is a framework for adding AI and machine learning functions to OpenNMS. It currently supports two methods for the correlation of alarms into situations: DBScan and TensorFlow, but is designed to allow for others to be added.

The current development focus is on the next version of Drift. Drift is the feature that does flow collection, and there are a number of improvements being worked on for “version 2”.

2019 Dev-Jam: Title Slide for the Contributing to OpenNMS talk

Markus von Rüden gave the next talk on contributing to OpenNMS. He covered a number of topics including dealing with our git repository, pull requests, test driven development and our continuous integration systems.

2019 Dev-Jam: Title Slide for the Karaf/OSGi talk

Matt Brooks presented an overview on how to leverage Karaf to add functionality to OpenNMS. Karaf is the OSGi container used by OpenNMS to manage features, and Matt used a simple example to show the process for adding functionality to the platform.

2019 Dev-Jam: Title Slide for the OIA talk

Extending on this was a talk by Chandra Gorantla about using the OIA with an example of creating a trouble ticketing integration. OpenNMS has had a ticketing API for some time but this talk leveraged the improvements added by the new API to make the process easier.

2019 Dev-Jam: Title Slide for the ALEC talk

Following this was a talk by David Smith on ALEC. He demonstrated how to add a simple time-based correlation to OpenNMS which covered a lot of the different pieces implemented by the architecture, including things like feedback.

That ended the development overview part of the presentation but there were two more talks on Docker and Kubernetes.

2019 Dev-Jam: Slide showing Useful Docker Commands for OpenNMS

Ronny Trommer gave a short overview of running OpenNMS in Docker, covering a lot of information about how to deal with the non-immutable (mutable?) aspects of the platform such as configuration.

2019 Dev-Jam: Kubernetes Diagram

This was followed by an in-depth talk by Alejandro Galue on Kubernetes, running OpenNMS using Kubernetes and how OpenNMS can be used to monitor services running in Kubernetes. While Prometheus is the main application people implement for monitoring Kubernetes, it is very temporal and OpenNMS can augment a lot of that information, especially at the services level.

These presentations took up most of the day. Since it is hard to find places where 30 people can eat together, we have a tradition of getting catering from Brasa, and we did that for Monday night’s meal.

2019 Dev-Jam: Table Filled with Food from Brasa

Jessica Hustace, who did the majority of the planning for Dev-Jam, handed out this year’s main swag gift: OpenNMS jackets.

2019 Dev-Jam: OpenNMS logo jacket

Yup, I make this look good.

by Tarus at June 25, 2019 03:08 PM

June 24, 2019

2019 Dev-Jam – Day 0

For the fourteenth time in fifteen years, a group of core community members and power users are getting together for our annual OpenNMS Developers Conference: Dev-Jam.

This is one of my favorite times of the year, probably second only to Thanksgiving. While we do a good job of working as a distributed team, there is nothing like getting together face-to-face once in awhile.

We’ve tried a number of venues including my house, Georgia Tech and Concordia University in Montréal, but we keep coming back to Yudof Hall on the University of Minnesota campus in Minneapolis. It just works out really well for us and after coming here so many times the whole process is pretty comfortable.

My role in Dev-Jam is pretty much just the “cruise director”. As is normal, other people do all the heavy lifting. I did go on a food and drink run which included getting “Hello Kitty” seaweed snacks.

2019 Dev-Jam: Hello Kitty Seaweed Snacks

Yudof Hall is a dorm. The rooms are pretty nice for dorm rooms and include a small refrigerator, two burner stove, furniture and a sink. You share a bathroom with one other person from the conference. On the ground floor there is a large room called the Club Room. On one side is a kitchen with tables and chairs. On the other side is a large TV/monitor and couches, and in the middle we set up tables. There is a large brick patio that overlooks the Mississippi River.

2019 Dev-Jam: Yudof Hall Club Room

The network access tends to be stellar, and with the Student Union just across the street people can easily take a break to get food.

We tend to eat dinner as a group, and traditionally the kickoff meal is held at Town Hall Brewery across the river.

2019 Dev-Jam: UMN Bridge Over the River

It was a pretty rainy day but it stopped enough for most of us to walk over the bridge to the restaurant. You could feel the excitement for the week start to build as old friends reunited and new friends were made.

2019 Dev-Jam: Town Hall Brewery

When we were setting up the Club Room tables, we found a whiteboard which is sure to be useful. I liked the fact that someone had written “Welcome Home” on it. Although I don’t live here, getting together with these people sure feels coming home.

2019 Dev-Jam: Welcome Home on Whiteboard

by Tarus at June 24, 2019 02:13 PM

June 20, 2019

OpenNMS Horizon 24.1.1 (GLaDOS) Released

Release 24.1.1 is the latest stable release of OpenNMS. It contains bug fixes to WS-Man, the web UI, flows, and more as well as a few small enhancements.

For a high-level overview of what’s changed in OpenNMS 24, see What’s New in OpenNMS 24.

The codename for 24.1.1 is GLaDOS.

Bug
  • WsMan Asset Adapter config contains an invalid resourceUri (Issue NMS-10682)
  • WsMan Asset Adapter should handle WQL filters returning more than one result (Issue NMS-10683)
  • WS-Man datacollection fail with OpenJDK11 (Issue NMS-10713)
  • Angular requests do not handle 401 responses from the OpenNMS backend (Issue NMS-10719)
  • sFlow: Type mismatch of VLAN value (Issue NMS-10775)
  • Timeline missing for service names including slashes (Issue NMS-12097)
  • CME in ProxyFilter (Karaf Web Bridge) (Issue NMS-12116)
  • Remove hack from opennms-spring-extender to expose ServiceRegistry (Issue NMS-12122)
  • Installing Minion on Debian with OpenJDK 11 installed tries to install Oracle JDK 8 (Issue NMS-12119)
  • Opennms 24.1.0 on Ubuntu 19.04 won’t install because of PostgresSQL 11 (Issue NMS-12121)
Enhancement
  • Increase logging clarity for flow processing (Issue NMS-10723)
  • Remove circleci build number as tags from container images (Issue NMS-12098)
  • Provide a mechanism to collect the count of objects collected by a wsman enumeration (Issue NMS-12115)

by RangerRick at June 20, 2019 09:12 PM

OpenNMS Meridian 2018.1.9 Released

Release 2018.1.9 is a small update to 2018.1.8 that fixes an issue with timeline graphs in the node details page.

The codename for 2018.1.9 is Strong/Severe Gale.

Bug
  • Timeline missing for service names including slashes (Issue NMS-12097)

by RangerRick at June 20, 2019 06:10 PM

OpenNMS Meridian 2017.1.18 Released

Release 2017.1.18 is a small update to 2017.1.17 that fixes an issue with timeline graphs in the node details page.

The codename for 2017.1.18 is Kyoto.

Bug
  • Timeline missing for service names including slashes (Issue NMS-12097)

by RangerRick at June 20, 2019 05:45 PM

OpenNMS Meridian 2016.1.22 Released

Release 2016.1.22 is a small update to 2016.1.21 that fixes an issue with timeline graphs in the node details page.

The codename for 2016.1.22 is Prof. Orlando Ferguson’s Map of the Square and Stationary Earth, 1893.

Bug
  • Timeline missing for service names including slashes (Issue NMS-12097)

by RangerRick at June 20, 2019 04:17 PM

June 17, 2019

OpenNMS On the Horizon – June 17th, 2019 – Flow Hostname Resolution, Helm and UI Enhancements, and More!

It’s time for OpenNMS On the Horizon!

Sorry I missed one, I am pretty sure I wrote it but left it in a tab and never hit “save” or something. ¯\_(ツ)_/¯

In the last two weeks we worked on a number of flow-related changes including bug fixing, resolving hostnames, and new Helm features. We also worked on a number of other UI enhancements relating to topology graphs, reporting, and authentication.

Github Project Updates

  • Internals, APIs, and Documentation
    • Christian worked on some fixes to Netflow 9 value scaling.
    • Chandra did more work on sending events from Minions.
    • Matt worked on adding search support to the new graph service.
    • Dustin and Christian continued to work on adding hostname resolution during flow processing.
    • Ronny and Jesse worked on CircleCI and test environment changes.
    • Chandra started work on supporting moving a (provisioned) node to a new location.
  • Web, ReST, and UI
    • Matt wrapped up his changes to add flow conversation, application, and host queries to OpenNMS.js and Helm.
    • Markus worked on rewriting the reporting UI in Angular.
    • Markus worked on adding support for generating reports from Grafana dashboards.
    • Markus fixed some issues with 401 authentication handling in Angular-based apps in the web UI.
    • Matt created a new histogram panel to Helm.
    • I updated a number of dependencies in OpenNMS.js and Helm to fix security alerts.
    • Markus worked on a UI to interact with Matt’s graph service search API.
    • Christian fixed a bug that would keep services with slashes in their name from showing timelines in the node detail page.
    • I worked on refactoring the experimental inventory datasource with the faults datasource into a new entity datasource.
    • Chandra worked on making the web UI graph page lazy-load graphs as well as support searching.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • HELM-148: Control resolution when using the flow datasource
  • HELM-155: Error on all panels and Grafana Home
  • NMS-10719: Angular requests do not handle 401 responses from the OpenNMS backend
  • NMS-10775: sFlow: Type mismatch of VLAN value
  • NMS-11760: Fix version range for maven-jaxb-schemagen-plugin version in root pom
  • NMS-12074: View traffic for specific conversations over time (flows) (frontend work)
  • NMS-12075: Provide possibility to add Grafana Instances
  • NMS-12076: View traffic for specific applications over time (flows) – front end work
  • NMS-12077: View traffic for specific hosts over time (flows) (frontend work)
  • NMS-12078: Add support for sending events from a Minion
  • NMS-12096: Custom TTLs for detectors, monitors and collectors
  • NMS-12097: Timeline missing for service names including slashes
  • NMS-12098: Remove circleci build number as tags from container images
  • NMS-12102: Windows is failing tests due to spurious cleanup errors
  • NMS-12107: Migrate system tests to use TestContainers framework

by RangerRick at June 17, 2019 04:05 PM

June 07, 2019

OpenNMS.js v1.5.0

This release includes support for some additional flow queries, as well as a few security updates and a fix for CLI formatting.

Bug Fixes

  • cli: fix broken table formatting in prod cli build (JS-33) (db50724)
  • security: update handlebars, js-yaml, and marked (f5f4ad2)
  • security: upgrade to Axios 0.19.0 (a731c60)

Features

by RangerRick at June 07, 2019 09:56 PM

June 03, 2019

OpenNMS On the Horizon – June 3rd, 2019 – Flows, Minion, Graph Services, UI Improvements, and More!

It’s time for OpenNMS On the Horizon!

In the last week we fixed some bugs, continued working on flow, Minion, and graph service backend improvements, and made a number of UI improvements.

Github Project Updates

  • Internals, APIs, and Documentation
    • Christian and Dustin continued their work on hostname resolution for flows.
    • Sean worked on flow logging to make diagnosing issues easier.
    • Chandra did more work on supporting originating events on the Minion.
    • Ron Roskens did a number of fixes and cleanups in the Maven build system.
    • Jesse did more work on updating smoke tests for the new CI infrastructure.
    • Chandra did more work on making TTLs configurable in detectors, monitors, and collectors.
    • Patrick worked on graph service search capabilities.
    • David continued his work refactoring Threshd out of opennms-services.
  • Web, ReST, and UI
    • Markus did more work on adding Grafana report support to the reporting UI.
    • Markus worked on fixing issues in Angular UIs when the backend returns authorization errors.
    • I did more work on refactoring entity datasource access in Helm.
    • I fixed a number of build and related bugs in Helm.

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • NMS-10281: We have a regression in the SNMP core API for get effective address (snmp-proxy)
  • NMS-10717: use sevntu-checkstyle-maven-plugin from maven-central
  • NMS-10718: use org.asteriskjava:asterisk-java from maven-central
  • NMS-10723: Increase logging clarity for flow processing
  • NMS-10775: sFlow: Type mismatch of VLAN value
  • NMS-10834: [circleci] Building container images fails intermittently
  • NMS-11153: View traffic for specific hosts over time (flows) (backend work)
  • NMS-11169: Update flow indicators for SNMP interfaces to link to deep dive tool
  • NMS-11372: Intermittent 404s from maven.opennms.org
  • NMS-11458: View traffic for specific applications over time (flows) – back end work
  • NMS-11722: Publish Docker images from CircleCI to DockerHub

by RangerRick at June 03, 2019 05:40 PM

May 28, 2019

OpenNMS On the Horizon – May 28th, 2019 – Flow Enhancements, Reporting, Bug Fixes, and More!

It’s time for OpenNMS On the Horizon!

In the last week we did more work on flow enhancements, reporting, and other bug fixes.

Github Project Updates

  • Internals, APIs, and Documentation
    • Sean did some more work on updating our Kafka components to 2.2.
    • Markus expanded some test coverage for SLM calculation.
    • Dustin and Christian continued to work on adding hostname resolution when processing flows.
    • Chandra did more fixes to memory leaks during Drools reloading.
    • Chandra added support for customizing TTLs in detectors/monitors/collectors
    • Patrick worked on fixing SNMP proxy issues.
    • Patrick continued his work on allowing vertexes from multiple namespaces to be in edges in the graph service.
    • Matt worked on flow API enhancements to allow querying specific conversations, hosts, or applications.
    • Jesse worked on porting smoke tests over to the new smoke test APIs.
    • I updated our pull request template to have a checklist of common tasks to complete.
    • Chandra worked on adding event API support to the Minion.
  • Web, ReST, and UI
    • I finished up adding support for severity color themes in Helm.
    • Jeff did some more input-sanitization fixes in the web UI.
    • Jeff fixed sorting of nodes when editing surveillance categories in the flow admin UI.
    • Jesse and Markus worked on reporting UI updates, including support for configuring accessing Grafana instances/dashboards.
    • Jesse changed the flow indicators in the UI to point to the deep dive tool.

OpenNMS May Releases

Last week we released OpenNMS Horizon 24.1.0, and new point releases for Meridian 2016, 2017, and 2018.

The Meridian releases contained primarily small UI fixes and a couple of security fixes. Horizon 24 got a decent number of bug fixes and enhancements including support for OpenTracing in the sink API and a rework of geocoding services.

For a complete list of changes, see the release notes:

Upcoming Events and Appearances

Until Next Week…

If there’s anything you’d like me to talk about in a future OOH, or you just have a comment or criticism you’d like to share, don’t hesitate to say hi.

– Ben

Resolved Issues Since Last OOH

  • HELM-147: Text fields on Alarm Details Modal Window do not wrap or scroll.
  • HELM-149: Add “theme” support for the alarm table severity colors
  • HZN-1521: Edges should allow to contain vertices of different namespaces
  • HZN-1551: View traffic for specific applications over time (flows) – back end work
  • HZN-1552: View traffic for specific hosts over time (flows) (backend work)
  • HZN-1554: View traffic for specific conversations over time (flows) (backend work)
  • HZN-1556: [circleci] Building container images fails intermittently
  • HZN-1566: Enforce VertexRef to be immutable
  • NMS-10281: We have a regression in the SNMP core API for get effective address (snmp-proxy)
  • NMS-10458: Wrong SLM 24hr availability calculation per node
  • NMS-10654: When editing a surveillance category from Admin flow, lists of nodes are not sorted by node label
  • NMS-10678: Memory Leak on Drools while reloading config
  • NMS-10679: Node detail page renders with no content when invalid node ID specified
  • NMS-10688: Install guide for Java versions is misleading
  • NMS-10698: EditInRequisitionIT flapping
  • NMS-10701: Jetty HTTPS selectors can become unresponsive following CancelledKeyException
  • NMS-10707: Reflected XSS vulnerability in notification/detail.jsp and outage/detail.htm

by RangerRick at May 28, 2019 05:54 PM

May 23, 2019

OpenNMS Horizon 24.1.0 Released

Release 24.1.0 is the latest stable release of OpenNMS. It contains a bunch of bug fixes and a few enhancements, including support for OpenTracing in the sink API and a rework of geocoding services.

For a high-level overview of what’s changed in OpenNMS 24, see What’s New in OpenNMS 24.

The codename for 24.1.0 is J.A.R.V.I.S.

Bug
  • Thresholds should work without restart when putting nodes into categories (Issue NMS-9811)
  • When editing a surveillance category from Admin flow, lists of nodes are not sorted by node label (Issue NMS-10654)
  • health:check command times out when a health check command gets stuck (Issue NMS-10667)
  • Vaadin bundles stuck intermittently stuck in “Waiting” state (Issue NMS-10668)
  • Table in “Manage Minions” page fails to load (Issue NMS-10670)
  • Default load threshold contains caluculation error (Issue NMS-10671)
  • Can not delete node due to database table constraint (Issue NMS-10674)
  • Missing Indication of Sync Needed for Requisitions (Issue NMS-10675)
  • .rpmnew, .rpmsave, and .dpkg-dist files not erroring out properly (Issue NMS-10676)
  • Some config files should be marked %config (rather than %config(noreplace)) in RPMs (Issue NMS-10677)
  • Memory Leak on Drools while reloading config (Issue NMS-10678)
  • Node detail page renders with no content when invalid node ID specified (Issue NMS-10679)
  • Installing the opennms package installs Oracle JDK 8 instead of OpenJDK 11 on Ubuntu (Issue NMS-10680)
  • Apparent memory leak in JMX collector, possibly restricted to “weird” JMX transports (Issue NMS-10684)
  • Install guide for Java versions is misleading (Issue NMS-10688)
  • Java configuration is ignored on Ubuntu/Debian (Issue NMS-10693)
  • CVE-2018-20433: XXE Vulnerability in c3p0 < 0.9.5.3 (Issue NMS-10694)
  • Memory leak in WS-Man (Issue NMS-10696)
  • EditInRequisitionIT flapping (Issue NMS-10698)
  • Jetty HTTPS selectors can become unresponsive following CancelledKeyException (Issue NMS-10701)
  • Reflected XSS vulnerability in notification/detail.jsp and outage/detail.htm (Issue NMS-10707)
Enhancement
  • Enable extraction of match groups from regex matches in Event.Mask.Varbind.Vbvalue (Issue NMS-10626)
  • Tag “root cause” alarm when providing feedback (Issue HZN-1492)
  • Rework the GeocoderService-Implementations (Issue HZN-1520)
  • Meta-Data Documentation Format Wrong (Issue HZN-1557)
  • Add OpenTracing support for Sink API (Issue HZN-1558)

by RangerRick at May 23, 2019 11:22 PM

OpenNMS Meridian 2018.1.8 Released

Release 2018.1.8 is an update to Meridian 2018.1.7. It contains a few UI fixes and security updates, as well as a fix for memory leaks in Drools config reloading, WS-Man monitoring, and the JMX collector.

The codename for 2018.1.8 is Gale.

Bug
  • Memory Leak on Drools while reloading config (Issue NMS-10678)
  • Node detail page renders with no content when invalid node ID specified (Issue NMS-10679)
  • Apparent memory leak in JMX collector, possibly restricted to “weird” JMX transports (Issue NMS-10684)
  • CVE-2018-20433: XXE Vulnerability in c3p0 < 0.9.5.3 (Issue NMS-10694)
  • Memory leak in WS-Man (Issue NMS-10696)
  • Jetty HTTPS selectors can become unresponsive following CancelledKeyException (Issue NMS-10701)
  • Reflected XSS vulnerability in notification/detail.jsp and outage/detail.htm (Issue NMS-10707)

by RangerRick at May 23, 2019 08:11 PM

OpenNMS Meridian 2017.1.17 Released

Release 2017.1.17 is a small update to 2017.1.16 that has a few UI fixes and security updates, as well as a fix for memory leaks in WS-Man monitoring and the JMX collector.

The codename for 2017.1.17 is Old Naval Observatory meridian.

Bug
  • Node detail page renders with no content when invalid node ID specified (Issue NMS-10679)
  • Apparent memory leak in JMX collector, possibly restricted to “weird” JMX transports (Issue NMS-10684)
  • CVE-2018-20433: XXE Vulnerability in c3p0 < 0.9.5.3 (Issue NMS-10694)
  • Memory leak in WS-Man (Issue NMS-10696)
  • Reflected XSS vulnerability in notification/detail.jsp and outage/detail.htm (Issue NMS-10707)

by RangerRick at May 23, 2019 08:09 PM