jrh3k5 / flume-agent-cookbook Goto Github PK
View Code? Open in Web Editor NEWA cookbook to set up a Flume agent
License: Apache License 2.0
A cookbook to set up a Flume agent
License: Apache License 2.0
To give the consumer more flexibility over how a Flume agent can be installed, it would be desirable to support an option to allow the user to override the default flume archive URL attribute in the resource such that if not specified, it would fallback to the default attribute.
Occasionally, the Flume agent will, to Chef's eye, appear to be running, even if it is not.
Chef's service
resource, at least on Chef 10.28.0, seems to ignore the return codes of the service status. Since we can't force Chef to ignore what it thinks is the operational status of the service, we should merely shell out the command to restart the agent.
To allow for the incorporation of things like health checks into the startup of a Flume agent, a consumer should be allowed to specify a script that gets executed following the startup of a Flume agent.
The service script for the installed Flume agent should return more meaningful status codes, such as those defined here.
There have been some issues that I've seen of cookbooks (especially ones that declare resource providers) working on Chef 12.5.1. This cookbook should be validated to work on Chef 12.5.1.
Hailing from its days as a Maven project, the provider for the Flume agent references a non-existent ${cookbookName} variable. This should be removed.
As the Flume agent does not provide the ability to isolate plugins from the classpath of the Flume agent itself, there can arise dependency conflicts (such as Flume's usage of Guava 10.0.1). It would be helpful to have a hook into the agent provider itself to remove these conflicting libraries.
We should add Vagrant tests
The shell script used to check the status of the Flume agent builds an array of running processes matching the service and then performs this check:
if [ ${#pids[@]} -ne 1 ]; then
echo "An instance of the Flume agent (see: $PID_FILE) that is not being tracked by this script was found."
return 4
else
In the event that the PID file contained only IDs for processes that are not running, this array will be empty and the script will falsely report that there is a running instance of the agent that is not being managed by the script.
Flume agent properties files can contain sensitive information (e.g., database passwords) which we would not want to show up in the chef-client logs. These files should be marked as "sensitive".
Presently, the resource defined in the {{flume_agent}} cookbook does not tell Chef that it supports the "status" command:
# Configure the service
service attributes["serviceName"] do
supports :start => true, :stop => true, :restart => true
action [ :enable ]
end
Chef can only try to make guesses about the status of the service without this.
We should only delete the Flume installation directory if the Flume installation version is actually changing.
Currently, the cookbook for the Flume agent is nested beneath a cookbooks/ folder. This is mostly to allow for the provisioning of a test cookbook, but there are ways that can be accomplished without complicated nesting like that.
When unpacking Flume plugin tarballs, the cookbook assumes that the resulting directory will follow the convention of either:
However, this does not account for when the directory is simply "my-plugin", and the cookbook will fail when trying to mv "my-plugin" to "my-plugin" because it expects to find "my-plugin-1.2.3".
As such, if the directory name matches the desired endstate, then don't error out (obviously).
If the directory (e.g., /opt/flume/my_agent
) does not exist but /opt/flume
does, the mv
command on Centos apparently does not create the missing directory.
Presently, this is what the flume_agent
cookbook does to install Flume:
tar zxf #{Chef::Config[:file_cache_path]}/#{tarFilename}
mv #{tarBasename}/* #{attributes["installDir"]}
rm -rf #{tarBasename}
This is all well-and-good for a fresh install, but the mv
command will not overwrite an existing directory with the contents of the tarball (say, for example, when upgrading from Flume 1.4.0 to 1.5.0).
To rectify this, we should make sure that directory is empty before attempting to mv
the contents of the TAR to the Flume installation:
tar zxf #{Chef::Config[:file_cache_path]}/#{tarFilename}
rm #{attributes["installDir]}/*
mv #{tarBasename}/* #{attributes["installDir"]}
rm -rf #{tarBasename}
We occasionally see "blips" of process IDs where the dump of the pgrep
command to the PID file results in multiple PIDs in the file, though only one is still alive. The service script should tolerate these cases and only return an "unknown" state if no PIDs are found at all.
Right now, the cookbook uses Vagrant (and not kitchen) to run its tests. Lives would generally be made easier if the cookbook used kitchen.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.