Data / Software

Sematext releases anomaly detector for Apache Spark

A major big-data tool within the Apache ecosystem now has its first anomaly detection system, thanks to a new product from the Gowanus-based Sematext.

A screenshot from Sematext's SPM software showing Apache Spark metrics. (Screenshot courtesy of Sematext)

Sematext is a company working deep within the core of the web.

As we’ve reported before, it has products that collect the logs from across servers; it has other products that intelligently detect anomalies in a site’s performance. The company just released a new version of its SPM product, making it the first anomaly and performance monitoring system for Apache Spark.

“By releasing the first Spark monitoring product to market with SPM, we have just filled a big hole in the Spark ecosystem,” Otis Gospodnetic, the company’s founder and CEO said.

In recent years, Spark, an open-source  software platform for data analytics, has grown in popularity within the Apache ecosystem. Apache is one of the main software systems used for running servers. It’s used by large companies like Yahoo and Intel, according to a 2013 article in Wired.

Gospodnetic explained, when we previously visited, that the core of SPM’s value is its ability to intelligently detect anomalies. That is, it establishes a baseline for metrics and redefines anomalies as those baselines change.

The easiest example to give is a spike in traffic. If a site normally gets 30,000 views per month, a 5,000-view day would be an anomaly; it would not, however, be an anomaly if the site gradually grew to 150,000 views per month. Rather than requiring development teams to define anomalies, SPM defines and redefines them based on activity. Sematext’s latest SPM release can now apply the same tools to any instances of Spark that a client is running across its servers.

Series: Brooklyn

Before you go...

Please consider supporting Technical.ly to keep our independent journalism strong. Unlike most business-focused media outlets, we don’t have a paywall. Instead, we count on your personal and organizational support.

3 ways to support our work:
  • Contribute to the Journalism Fund. Charitable giving ensures our information remains free and accessible for residents to discover workforce programs and entrepreneurship pathways. This includes philanthropic grants and individual tax-deductible donations from readers like you.
  • Use our Preferred Partners. Our directory of vetted providers offers high-quality recommendations for services our readers need, and each referral supports our journalism.
  • Use our services. If you need entrepreneurs and tech leaders to buy your services, are seeking technologists to hire or want more professionals to know about your ecosystem, Technical.ly has the biggest and most engaged audience in the mid-Atlantic. We help companies tell their stories and answer big questions to meet and serve our community.
The journalism fund Preferred partners Our services

Join our growing Slack community

Join 5,000 tech professionals and entrepreneurs in our community Slack today!

Technically Media