Frequently Asked Questions (FAQ)

General Questions

Why use streamparse?

To lay your Python code out in topologies which can be automatically parallelized in a Storm cluster of machines. This lets you scale your computation horizontally and avoid issues related to Python’s GIL. See Parallelism and Workers.

Is streamparse compatible with Python 3?

Yes, streamparse is fully compatible with Python 3 starting with version 3.3 which we use in our unit tests.

How can I contribute to streamparse?

Thanks for your interest in contributing to streamparse. We think you’ll find the core maintainers great to work with and will help you along the way when contributing pull requests.

If you already know what you’d like to add to streamparse then by all means, feel free to submit a pull request and we’ll review it.

If you’re unsure about how to contribute, check out our open issues and find one that looks interesting to you (we particularly need help on all issues marked with the “help wanted” label).

If you’re not sure how to start or have some questions, shoot us an email in the streamparse user group and we’ll give you a hand.

From there, get to work on your fix and submit a pull request when ready which we’ll review.

How do I trigger some code before or after I submit my topology?

After you create a streamparse project using sparse quickstart, you’ll have a fabfile.py in that directory. In that file, you can specify two functions (pre_submit and post_submit) which are expected to accept three arguments:

  • topology_name: the name of the topology being submitted
  • env_name: the name of the environment where the topology is being submitted (e.g. "prod")
  • env_config: the relevant config portion from the config.json file for the environment you are submitting the topology to

Here is a sample fabfile.py file that sends a message to IRC after a topology is successfully submitted to prod.

# my_project/fabfile.py
from __future__ import absolute_import, print_function, unicode_literals

from my_project import write_to_irc


def post_submit(topo_name, env_name, env_config):
    if env_name == "prod":
        write_to_irc("Deployed {} to {}".format(topo_name, env_name))

How should I install streamparse on the cluster nodes?

streamparse assumes your Storm servers have Python, pip, and virtualenv installed. After that, the installation of all required dependencies (including streamparse itself) is taken care of via the config.json file for the streamparse project and the sparse submit command.

Should I install Clojure?

No, the Java requirements for streamparse are identical to that of Storm itself. Storm requires Java and bundles Clojure as a requirement, so you do not need to do any separate installation of Clojure. You just need Java on all Storm servers.

How do I deploy into a VPC?

Update your ~/.ssh/config to use a bastion host inside your VPC for your commands:

Host *.internal.example.com
    ProxyCommand ssh bastion.example.com exec nc %h %p

If you don’t have a common subdomain you’ll have to list all of the hosts individually:

Host host1.example.com
    ProxyCommand ssh bastion.example.com exec nc %h %p
...

Set up your streamparse config to use all of the hosts normally (without bastion host).

How do I override SSH settings?

It is highly recommended that you just modify your ~/.ssh/config file if you need to tweak settings for setting up the SSH tunnel to your Nimbus server, but you can also set your SSH password or port in your config.json by setting the ssh_password or ssh_port environment settings.

{
    "topology_specs": "topologies/",
    "virtualenv_specs": "virtualenvs/",
    "envs": {
        "prod": {
            "user": "somebody",
            "ssh_password": "THIS IS A REALLY BAD IDEA",
            "ssh_port": 52,
            "nimbus": "streamparse-box",
            "workers": [
                "streamparse-box"
            ],
            "virtualenv_root": "/data/virtualenvs"
        }
    }
}

How do I dynamically generate the worker list?

In a small cluster it’s sufficient to specify the list of workers in config.json. However, if you have a large or complex environment where workers are numerous or short-lived, streamparse supports querying the nimbus server for a list of hosts.

An undefined list (empty or None) of workers will trigger the lookup. Explicitly defined hosts are preferred over a lookup.

Lookups are configured on a per-environment basis, so the prod environment below uses the dynamic lookup, while beta will not.

{
    "topology_specs": "topologies/",
    "virtualenv_specs": "virtualenvs/",
    "envs": {
        "prod": {
            "nimbus": "streamparse-prod",
            "virtualenv_root": "/data/virtualenvs"
        },
        "beta": {
            "nimbus": "streamparse-beta",
            "workers": [
                "streamparse-beta"
            ],
            "virtualenv_root": "/data/virtualenvs"
        }
    }
}