StormLib++ - An Intro

StormLib++ is a monorepo of sorts containing Python and Storm code that makes my life as a Storm developer and Synapse user easier. It’s composed of two parts, a Python package stormlibpp and a collection of Storm Packages (Rapid Power-Ups). This blog post covers the evolution of the project and intends to highlight the main features it has to date. For more details on exactly what StormLib++ can do - including API docs, a User Guide, and a Contribution Guide - checkout the StormLib++ documentation.

StormLib++ started as a way to portably collect some Python code used to make developing Storm Services easier. In particular, it made developing a Storm Package alongside its Storm Service better. Instead of embedding the Storm code directly in Python files, users could write a package Yaml file in its own folder and define Storm code in .storm files next to the Yaml definition, the way Rapid Power-Ups are typically written. A class was created (see stormlibpp.stormpkg.StormPkg) to load the traditionally developed Storm Package definition into a Python object - the Storm Service - at runtime. It even handled syntax checking of the Storm code as loaded, so that users wouldn’t have to guess at where problems might be when their Storm Service is loaded by a Cortex. This feature was originally ported from the yarastorm project.

StormLib++ quickly grew into much more than a Storm Package loader. v0.3.0 brought a Python definition of a Synapse Node (StormNode) that was safe to read and modify within Python code before being “packed” and written back to a Cortex. v0.4.0 came with an HttpCortex object that implemented simple synapse.cortex.Cortex methods but over HTTP - something I needed in a network that wouldn’t allow Telepath connections between users and a Cortex. HttpCortex has grown considerably since it was first released. At first, it only had storm and callStorm methods. Today it supports addFeedData, exportStorm, and stormlist as well.

v0.5.x added an HTTP-based Storm CLI (stormlibpp.hstorm). It actually relies on the same Python objects as synapse.tools.storm, but it uses StormLib++’s HttpCortex implementation. At first, this tool could only execute Storm statements. But more recent versions allowed the CLI to import and export .node files. It also supports the onecmd option to run Storm code directly from the command line and exit. Unfortunately, this tool cannot upload/download files from a Synapse Axon like the out-of-the-box Storm CLI. But hopefully one day, once the Axon HTTP API endpoints are fronted by the Cortex HTTP API, this can happen.

The feature I’m most proud of so far, a Storm script importer (stormlibpp.import), came in v0.6.x and was perfected in subsequent versions. It works by recursively walking a directory, running each .storm file found, in alphabetical order, on a Cortex. It can read CSV, JSON, and text files of the same name as the Storm scripts as input for the given script. The idea was to improve upon the open source synapse.tools.csvtool features, and bring JSON and text file imports to the OSS Synapse community. This importer is feature rich and very powerful. A second blog post is in the works to cover it in depth - and even demonstrate how it can be used to manage more “static” Synapse data (user perms, tag definitions, taxonomies, etc.) as code.

v0.10.x was a big release for StormLib++. It added a new CLI tool, hfeed. Which, if you can’t already guess, is an HTTP-based version of synapse.tools.feed. It also made a change to StormLib++’s design, several Storm Packages (Rapid Power-Ups) were moved into the project’s repo under the src/pkgs/ folder. This was done for a few major reasons:

  • Primarily, I didn’t want to maintain multiple small repos for one off Synapse-related projects.

  • I eventually wanted to add Github Actions to all of these projects, including StormLib++, to build and release code as new features are developed. Bringing these Storm Packages into StormLib++ allows me to manage less GH Actions that would have ultimately been similar/the same.

  • Moving all of these Storm Packages into StormLib++ allowed me to namespace Storm Package commands and modules with an slib prefix. Something that’s largely cosmetic, but I prefer it.

What’s next: I’ve got 2 big ideas brewing that I’d like to get added to StormLib++. They’re both currently partially implemented, I just need to find the time to work on them.

  1. An improvement to the built-in Stix import features of Synapse (PR #37).

    • This is a custom import config, and a builder for the storm files that make up the config. There’s also a mostly complete, but useless without the custom config being finished, Storm Package for this already present in recent versions of StormLib++.

  2. stormgen a Markdown -> Synapse Node converter (PR #38). This idea is pretty wild, and I’ve pontificated on it quite a bit in a design proposal found in the stormgen branch. I encourage you to read this proposal doc if you’re interested in learning more about it.

    • The main idea is that it is a tool that will be able to convert structured Markdown documentation into Storm node tuples that can be used with a Synapse Cortex’s proxy methods to create or update nodes.

    • stormgen exists so that Synapse engineers/analysts don’t have to do the double work of both documenting their data model and creating the nodes, which often contain elements of the same documentation. This allows users to document the expected usage, explain any rules or restrictions, warn others of gotchas, and even provide examples for any part of their Synapse data model without storing them in the model itself.

    • My hope is that it can also be used to long form document nodes in Markdown reports. Something that’s very approachable for most analysts. The main use cases are reports that document threat actors, malware families, or observed activity and the IOCs seen along with it (like something observed by a security vendor or a report for a security event/incident at a company).

I don’t think I’ll consider StormLib++ 1 dot 0 ready until some housekeeping (read better documentation, greater test coverage, and CI/CD) is complete. But in the meantime, existing features are stable and ready for use by all who are interested! I hope this project has been/can be useful to some. Especially those that are relying solely on the OSS Synapse features and those that are contributing to the growing 3rd-party OSS Synapse tool base.