Coreset Technology: Unravelling The “Magic”

Much of software security platform’s ShieldIOT “secret sauce” and key differentiators are based on its unique Coreset technology. What are coresets and why can they act as a game changer in protecting mass scale IoT networks?

Current Challenge

Today there are many good and available (off-the-shelf) artificial intelligence / machine learning anomaly detection algorithms. The challenge is that as IoT networks become larger and larger (reaching dozens of thousands, hundreds of thousands (smart cities) and now even millions of devices (smart grid, smart transportation)), using these anomaly detection algorithms on large datasets can take hours and even days, which is not an option in case of an IoT cyber attack that can take down key physical services, cause significant financial damage and even loss of life.

The Coreset “Magic”

ShieldIOT’s Coreset “magic” (based on 12+ years of MIT and Haifa academic research, 40+ academic papers) is that it enables accurate analytics at scale. 

We did not invent a magical algorithm to “rule them all”. In fact, we do not directly solve the anomaly detection problem (considered by many as impossible) – instead we try to solve the data problem. We use Coresets.

Coresets are mathematical constructs that compress the data in real-time streaming mode, from 1 million to 20 data points (n to log(n)), with almost zero loss of energy. By running Coresets before applying the existing best of breed AI/ML anomaly detection algorithms, optimal results are reached in minutes (vs. hours/days) platform is running on much smaller data. These optimal results include significant lower false alarms levels (false positives) as well as improved detection rates (true positives).

Existing solutions, that are not using Coresets, are forced to perform various heuristics on the data (else they can’t analyze it as it’s too big) resulting in very high levels of false alarms (in some cases over 95% false). 

The Uniqueness

The 3 main ShieldIOT Coreset unique differentiators include:

• Unmatched accuracy: up to 20x lower false alarms vs. competitive solutions 
• Unlimited scalability: supporting networks of millions of devices today 
• Any device, application: the solution is agnostic across any IoT device/application (solution has been successfully used on meters, lighting, gateways, industrial sensors, car sensors, air quality sensors, HVAC….)

The Future of Coreset Anomaly Detection

ShieldIOT is the only solution today in the world that uses Coresets for anomaly detection in IoT. In fact there are only a handful of Coresets experts in the world today, with most of them focusing on pure academic research (and not on commercial applications).

Coresets is a new and complex field in mathematics/computer science, making it extremely difficult to leverage existing academic papers as “recipes” for commercial solutions. Having said that, we expect that as IoT networks continue to grow, coresets as an enabler of accurate analytics at scale, will become a main stream IoT technology applied across multiple IoT application and networks to ensure their protection.