Welcome to Arsenal’s documentation!

Contents:

Arsenal - The Ironic image caching service

Build Status Documentation Status

About

A small, Openstack-y service designed to manage image caching to nodes in Ironic, written in Python.

Pluggable data-gathering and cache management strategy means Arsenal can be repurposed to work with other services.

Features

  • Pluggable data gathering.
  • Pluggable strategy/decisioning around caching images to nodes.
  • Built-in objects which provide client caching and API call retries to: Ironic, Nova, and Glance.

Documentation

Hosted HTML docs for Arsenal are available at http://arsenal.readthedocs.org/

You may also build a local copy of Arsenal’s documentation by using Sphinx:

$ sphinx-build $repo_root/docs/source $output_dir

Then you can read the local documentation by pointing a browser at $output_dir/index.html

Roadmap

See issues labeled ‘enhancements‘ on Arsenal’s Github project issues page.

Installation

Currently, the best way to obtain Arsenal is to clone the repository from Github:

$ git clone https://github.com/rackerlabs/arsenal.git

To perform a local installation, navigate to the repository’s root and run:

$ pip install .

Or, if you have virtualenvwrapper installed:

$ mkvirtualenv arsenal
$ pip install .

For a development installation:

$ pip install -e .

Should provide Arsenal to the system, while allowing development changes to your local repository to be reflected in the installed package.

Unforuntately, Arsenal is not yet available from PyPI. However, that should change soon.

Configuring Arsenal

arsenal.conf

Arsenal reads configuration variables from a single file, canonically called arsenal.conf. The location and name of this configuration file can be changed as long as the --config-file argument to arsenal-director is set accordingly.

Oslo/Config

Arsenal uses the oslo.config module to parse and load configuration. See oslo.config‘s documentation for detailed information on the supported syntax.

Basic Syntax

That said, the basic syntax of the configuration file is fairly straight-forward.

Sections

Arsenal’s configuration file is separated into sections. Each section begins with a bracket enclosed string. For example:

[director]

Begins the “director” configuration section. Each line following the section directive will populate that section’s configuration options, until another section directive is parsed, or the end of the file is reached.

Options

Each option has this basic format:

<option_name>=<value>

Where <option_name> is the option’s name, and <value> is the value to assign to the option.

A Short Example

The following:

[director]
dry_run=True

Would set the dry_run option to the boolean value ‘True’, which belongs to the [director] section.

arsenal.conf Sections

There are several sections which comprise arsenal.conf. You may not need to include every available section, nor set every option. Please read through the following section descriptions to get a sense for what functionality is made available through arsenal.conf.

[director] Section

The [director] section contains options which affect how arsenal-director gathers information using Scouts.

Note

See scheduler.py for all [director] configuration options.

Important Section Options

  • scout - Configures which Scout will be loaded by arsenal-director to gather data from services. The Scout also currently handles issuing directives to endpoints. The format is:

    <scout_module_name>.<ScoutClassName>

    For example, setting the scout option to:

    devstack_scout.DevstackScout

    Would cause arsenal-director to use the DevStack Scout, which is a Scout provided by Arsenal that is designed to work with DevStack.

  • dry_run - A boolean option. Setting this option to True will cause arsenal-director to run in dry run mode. Which means no directives generated by the configured Strategy will be issued.

Tip

dry_run is a great option to use while testing Arsenal without worrying about affecting outside services beyond requesting information.

  • directive_spacing - An integer option. Represents time in seconds. Determines how long the Director will wait between issuing new directives returned by the configured Strategy.
Cache Directive Rate Limiting

The next two options are related to limiting how many cache directives Arsenal will issue within a given period of time. They are tightly coupled and should be set together.

  • cache_directive_rate_limit - An integer option limiting how many cache directives Arsenal will issue within a period of time delimited by cache_directive_limiting_period. Defaults to 0, which indicates no rate limiting of cache directives will occur.
  • cache_directive_limiting_period - An integer option denoting the period of time, in seconds, to limit Arsenal issuing cache directives to the limit set by cache_directive_rate_limit. Once this period of time passes, Arsenal will again issue cache directives (if the configured Strategy is returning cache directives) until the rate limit is reached, or until the current time period again passes.

[strategy] Section

This section provides configuration options relevant to all Strategy objects.

module_class

The [strategy] section currently only has a single option: module_class. The module_class option controls which Strategy object is loaded and subsequently used to provide Arsenal’s cache decisions. The format of the module_class option is as follows:

<strategy_module_name>.<StrategyClassName>

For example, the default value for module_class is:

simple_proportional_strategy.SimpleProportionalStrategy

This causes the the class SimpleProportionalStrategy, which can be found in the simple_proportional_strategy module, to be instantiated and used by arsenal-director to provide cache decisions at run-time. The simple_proportional_strategy module is included as part of Arsenal.

Astute readers will notice the the syntax of this option matches that of scout from the [director] section.

[simple_proportional_strategy] Section

Currently, the SimpleProportionalStrategy class is the only concrete implementation of strategy.Strategy provided by Arsenal.

See the SimpleProportionalStrategy section for more information on this Strategy.

Important Section Options

percentage_to_cache - A floating point number. Valid values range from 0 to 1 inclusive. 0 corresponds to 0%, and 1 corresponds to 100%. Controls the percentage of unprovisioned/available nodes of a particular flavor to be cached at a particular time.

[client_wrapper] Section

The [client_wrapper] section contains options relevant to the Openstack client wrapper provided by Arsenal. Arsenal provides service-specific client wrappers for Ironic, Nova, and Glance.

The client wrappers provided by Arsenal all provide client caching and call-retry behavior. This section provides options to configure part of that behavior as well as provide credentials to all wrappers.

Note

Please see client_wrapper.py for all [client_wrapper] configuration options.

Important

Credential options defined in the client_wrapper section will be used by default by every derived instance of client wrapper unless the credential is overridden in the derived client wrapper’s section. For instance, if os_username is defined in the [client_wrapper] section, then the Nova client wrapper will use the client_wrapper.os_username value unless nova.admin_username is defined.

Important Section Options
  • call_max_retries - An integer value which determines how many times an individual client will be retried, until it is successful.
  • call_retry_interval - An integer value which Determines how long the client wrapper will wait before trying a call again.

[nova] Section

This section provides options mainly relating to credentials and the endpoint to use to communicate with Nova.

Note

Please see nova_client_wrapper.py for all [nova] configuration options.

[ironic] Section

This section provides options mainly relating to credentials and the endpoint to use to communicate with Ironic.

Note

Please see ironic_client_wrapper.py for all [ironic] configuration options.

[glance] Section

This section provides options mainly relating to credentials and the endpoint to use to communicate with Glance.

Note

Please see glance_client_wrapper.py for all [glance] configuration options.

A full example arsenal.conf file

See the example Arsenal configuration in the Arsenal source tree to see a full example configuration to use with arsenal-director.

Usage

arsenal-director

Arsenal is invoked by running:

arsenal-director

With various arguments. All of arsenal-director‘s supported arguments are documented on the command line. Run:

arsenal-director --help

To see them, and a brief explanation on each one.

A reasonable invocation for actual use looks something like:

arsenal-director --config-file /etc/arsenal/arsenal.conf --log-file /car/log/arsenal/arsenal-director.log

Which would start arsenal-director, and it would try to load the configuration file found at /etc/arsenal/arsenal.conf while logging to /var/log/arsenal/arsenal-director.log.

arsenal-director will periodically gather data using the configured Scout object, and issuing directives returned by the configured Strategy object. arsenal-director will continue in this way indefinitely, only stopping through program termination.

Important

It’s a good idea to set the dry_run option to True in order to prevent arsenal-director from issuing directives until you are confident that all the configuration settings appear to be correct, and the directives emitted by the configured Strategy are consistent with expected behavior.

Design

The core of Arsenal’s functionality consists of gathering data for input, through Scout objects, to send to Arsenal’s caching Strategy objects, which produce directives, which in turn are currently fulfilled by Scout objects.

Therefore, Scouts deal with the outside world, while Strategies provide introspection on data provided by Scouts to direct image caching on nodes in some meaningful way. The Scout and Strategy objects used by arsenal-director can be changed through configuration options.

Arsenal’s design philosophy can be summed up as: “Provide a way to do something, but make it easy to change or swap out.”

Scout

The responsibility of Scouts are to gather data from various outside sources, like Ironic, Nova, and Glance, convert that data to a form suitable for Strategy object consumption, as well as issue directives to endpoints, such as Ironic.

All of Arsenal’s Scout objects are derived from an abstract base class called Scout, which is defined in scout.py.

Tip

If you are thinking about defining your own Scout object, reading scout.py is a good place to start.

A couple of pre-made Scouts are currently included in Arsenal.

DevStack Scout

This Scout is designed to be used with the DevStack project, which provides a relatively easy way to setup an Openstack-based environment on a single machine, typically for testing purposes.

The DevStack Scout will communicate with Ironic, Nova, and Glance services, and filter for baremetal nodes. See Ironic documentation on how to configure virtual baremetal nodes for use with DevStack.

For more information see, devstack_scout.py.

OnMetal Scout

The OnMetal Scout is designed to work with Rackspace’s OnMetal product. While this specific Scout will probably not be directly useful to anyone outside of Rackspace, it can still be instructive to view a fully functional, concrete implementation of a Scout.

For more information, see onmetal_scout.py.

Strategy

A Strategy’s role lies in consuming data provided by Scouts, and then emitting directives to manage imaging caching on nodes.

Currently, two directives are supported. The first is CacheNode. CacheNode instructs the endpoint to cache a specific image onto a specific node. The second is EjectNode, which instructs the endpoint to do whatever is necessary to put a previously cached node back into an uncached state.

Tip

If you are thinking about defining your own Strategy object, reading strategy/base.py is a good place to start.

SimpleProportionalStrategy

Currently, SimpleProportionalStrategy is the only Strategy shipping with Arsenal.

This object implements a fairly straight-forward strategy: For each available flavor of node, use a constant proportion of available nodes for caching.

Image selection and node selection are currently completely random.

See the [simple_proportional_strategy] Section for information on how to configure this Strategy.

Contributing

Contributions are encouraged and welcome!

For any type of change, please follow this general workflow:

  1. Open an issue in Arsenal’s Github issue tracker. Describe the issue and tag it accordingly. That is, if the issue is a bug, please tag the issue as a bug. If an issue already exists, skip this step.
  2. Clone Arsenal’s repository locally.
  3. Create a topic branch for the changes you plan to make in regards to the issue you’re working on: git checkout -b your_branch_name
  4. Make your changes.
  5. Add appropriate unit-tests. If your change addresses a bug, please add a unit test that proves the bug is fixed by your change. For enhancements, try to thoroughly test all cases the new code will face.
  6. Make sure all unit tests pass. tox -epy27,pep8 should exercise all unit-tests and check for pep8 related style issues.
  7. Commit your changes to your local repository and reference the appropriate Github issue in your commit message, if appropriate.
  8. Push your topic branch: your_banch_name to Github.
  9. Create a pull request using the your_branch_name branch.

At that point, a repository maintainer will need to review and approve the pull-request. You may be asked to make additional changes to your pull-request before it is merged.

Please note that any contributions will fall under the Apache 2.0 license governing this project.

Thanks for contributing!

Indices and tables