JarCastingBreakerbox 
Breakerbox is a dashboard and dynamic configuration tool for Tenacity.
Running Breakerbox
Required: Breakerbox requires you to instrument your code with Tenacity which is a library to aid in resilient design of foreign dependencies.
Download a release or build it yourself with maven
Extract the archive and then run the following command
java -jar breakerbox-service-X.X.X.jar server breakerbox.yml
Then point your browser at http://localhost:8080. You should see a dashboard consisting of the local Breakerbox instance. By default it's backed by an in memory database, so any configuration changes you'll make won't be persisted anywhere until your configure it.
Configuring Breakerbox
Here's a very simple bare configuration of Breakerbox. Each section will be described in detail.
breakerbox:
urls: file:config.properties,http://localhost:8080/archaius/breakerbox
initialDelay: 0s
delay: 60s
breakerboxHostAndPort: localhost:8080 #default
database:
driverClass: org.h2.Driver
url: jdbc:h2:mem:inMemory;DB_CLOSE_DELAY=-1
user: breakerbox
password: breakerboxpass
maxSize: 10
minSize: 2
checkConnectionWhileIdle: true
checkConnectionHealthWhenIdleFor: 1s
tenacityClient:
connectionTimeout: 500ms
timeout: 1000ms
defaultDashboard: production
instanceDiscoveryClass: com.yammer.breakerbox.turbine.YamlInstanceDiscovery #default
hystrixStreamSuffix: /tenacity/metrics.stream #default
metaClusters:
- production
- stage
- staging
turbine:
urlSuffix: /tenacity/metrics.stream
clusters:
breakerbox:
instances:
- localhost:8080
production:
clusters:
- breakerbox
rancherDiscovery:
serviceApiUrl: http://localhost:8080/v1/projects/1a5/services
parameters:
kind: 'service'
accessKey: 3E0F7DB0A2B601981F1B
secretKey: fWWKGNvmuWpSngyVYHXFMSnE5cDhZWKNkVmQS8zn
marathonDiscovery:
- marathonApiUrl: http://localhost:8080
marathonAppPort: 12345
marathonAppNameSpace: /exampleNameSpace
cluster: exampleCluster
server:
applicationConnectors:
- type: http
port: 8080
adminConnectors:
- type: http
port: 8081
gzip:
enabled: false
requestLog:
appenders:
- type: file
currentLogFilename: /var/log/breakerbox/requests.log
archivedLogFilenamePattern: /var/log/breakerbox/requests-%d.log
archivedFileCount: 5
logging:
level: INFO
appenders:
- type: file
currentLogFilename: /var/log/breakerbox/breakerbox.log
archivedLogFilenamePattern: /var/log/breakerbox/breakerbox-%d.log
archivedFileCount: 5
Gzip Disabled
Javascript's EventSource doesn't support gzip encoded streams.
Persistence Storage
You can choose between using an in-memory database (h2), Postgresql, MySQL (potentially other RDBMSes) and Azure Table.
For Postgresql simply modify the database section to (assumes the database "breakerbox" is created):
database:
driverClass: org.postgresql.Driver
url: jdbc:postgresql://localhost/breakerbox
For MySQL:
database:
driverClass: com.mysql.jdbc.Driver
url: jdbc:mysql://localhost/breakerbox
If you wish to use Azure Table remove the database section entirely and add
azure:
accountName: your_test_account
accountKey: security_key
timeout: 2s
retryInterval: 500ms
retryAttempts: 1
If you specify both database and azure it will only leverage the database store.
Authentication
By default users that save configuration's will save it under the name anonymous.
At the moment it only supports authenticating users via LDAP and HTTP Basic-Auth which is configurable by adding:
ldap:
uri: ldaps://ldap.com
cachePolicy: maximumSize = 10000, expireAfterAccess = 15m
userFilter: ou=users,dc=company,dc=com
userNameAttribute: cn
connectTimeout: 500ms
readTimeout: 500ms
If you need group membership filters you can see the additional documentation on dropwizard-auth-ldap
Dashboard Configuration
turbine: /path/to/instances.yml
urlSuffix: /tenacity/metrics.stream
clusters:
breakerbox:
instances:
- localhost:8080
production:
clusters:
- breakerbox
turbineis a path to where ayamlconfiguration file containing the instances configuration is located (this is editable at runtime allowing for dynamic rediscovery).urlSuffixdefaults to/tenacity/metrics.streamyou can alter this if you have this resource at a different locationclustersis the top level item for your clusters or dashboards you wish to have available. Underneath clusters you can specify dashboard names such asbreakerboxorproductionin this example above. Underneath those you can specify eitherinstancesandclusters.clusterscan reference other dashboards and will include their instances. This can reference cyclic dashboards and will add instances from both.instanceslets you specify a single instance.
Instance Discovery Class
Specifies the Java canonical class name. It defaults to the YamlInstanceDiscovery implementation. You can also leverage the com.yammer.breakerbox.turbine.KubernetesInstanceDiscovery , com.yammer.breakerbox.turbine.RancherInstanceDiscovery , com.yammer.breakerbox.turbine.MarathonInstanceDiscoveryand classes.
To integrate with RancherInstanceDiscovery,
1. specify rancher services Api url, accessKey and secret key.
rancherDiscovery:
serviceApiUrl: http://localhost:8080/v1/projects/1a5/services
accessKey: 3E0F7DB0A2B601981F1B
secretKey: fWWKGNvmuWpSngyVYHXFMSnE5cDhZWKNkVmQS8zn
2. add labels in rancher service containers:
a. tenacity.metrics.stream.enabled: true
b. tenacity.metrics.stream.port: 8080
c. service.cluster.name: clusterName
3. Use hystrixStreamSuffix to provide custom path for hystrix metrics stream. It defaults to /tenacity/metrics.stream
4. RancherInstanceDiscovery will create dashboards per service-cluster with service.cluster.name label and one aggregated production dashboard. Dashboards can be created, enabled, disabled by updating labels at runtime.
To integrate with MarathonInstanceDiscovery, 1. specify marathon services Api url, marathonAppPort(your application port), marathonAppNameSpace and clusterName. marathonDiscovery: - marathonApiUrl: http://localhost:8080 marathonAppPort: 12345 marathonAppNameSpace: /exampleNameSpace cluster: exampleCluster
2. MarathonInstanceDiscovery will create dashboards per unique cluster specified in the config. MarathonDiscovery supports multiple marathon namespaces and cluster support.
Meta Clusters
These are clusters that should be in the dashboard dropdown but aren't necessarily configurable. This is used for dashboards that are made up of multiple clusters that are informational, but don't make much context in the sense of configuring one of them.
metaClusters:
- production
- stage
- staging
Adding Breakerbox to your Dropwizard+Tenacity Service
In any Tenacity service to have it poll Breakerbox for dynamic configurations add this to your service configuration.
urls: A comma-delimited list of breakerbox instances. The cluster you added in theconfig.propertiesis the identifier you want to supply here.initialDelay: How long it should wait before the first poll.delay: The interval at which to constantly checkBreakerboxfor configuration items. Here we specify every minute we'd like to check for new configurations.
breakerbox:
urls: http://breakerbox:8080/archaius/{the-cluster-you-setup-in-breakerbox}
initialDelay: 0s
delay: 60s
There is a configuration hierarchy priority and it's good to understand it especially when using dynamic configurations.
- Breakerbox
- Service configuration YAML
- Tenacity defaults
Additional Configuration
Here's a collection of things we have found very useful to make configurable.
hystrixMetricsStreamServletMaxConnections: The max number of connections anyDropwizard+Tenacityservice will allow from Breakerbox.turbineHostRetry: The interval at which to reconnect to a failed instance. 1s may be a bit fast as a default :)
archaiusOverride:
turbineHostRetry: 1s
hystrixMetricsStreamServletMaxConnections: 5
The client Breakerbox uses to fetch latest property keys and configurations is configurable by
tenacityClient:
connectionTimeout: 500ms
timeout: 1000ms
Building Breakerbox
mvn clean package
The artifact will then be available under breakerbox-service/target.
Running Tests
mvn test
breakerbox-azure doesn't have a mock for Azure Table as it wouldn't be that useful, so it requires a valid Azure Table account to run the tests. If you do not supply a valid Azure Table account, these tests will be ignored.
Docker
Justin Plock created a Docker for Breakerbox
Behind a Proxy
The dashboard leverages Javascript's EventSource which doesn't support CORs. There is a configurable parameter breakerboxHostAndPort which defaults to localhost:8080. You can change this to allow for your proxy to direct traffic appropriately.
