mount and systemd

Had a task: double the size of a volume on amazon AWS EC2 instance. The process is yet manual and it is roughly as follows:

  • Create a new volume on AWS with double the size of the the old one
  • Attach it to the instance
  • Create partition and filesystem on the new volume
  • Mount the new volume somewhere next to the old volume mount point
  • Rsync data from the old volume to the new volume
  • Adjust /etc/fstab to point to the new volume for the corresponding mount point
  • Unmount both volumes
  • Mount the new volume to the old mount point
  • Detach the old volume from instance
  • Delete the old volume

All pretty simple and strait-forward. BUT! The new volume is not mounting to the old mount point, while mount command is silent about it!!!

Syslog gives a hint:

systemd: Unit var-whatever.mount is bound to inactive unit dev-xvdg1.device. Stopping, too.
lb1 systemd: Unmounting /var/whatever...
lb1 systemd: Unmounted /var/whatever.

That’s interesting. Why it is still bounded to inactive device (which I have already detached) and how I can unbound it?

Apparently all records in /etc/fstab are converted to systemd units and all mounting is (these ugly days) done via systemd. So when I changed /etc/fstab, the systemd didn’t update the the related unit and was still trying to mount the old device. To fix the problem you need to run:

systemctl daemon-reload

I am too old for this shit… Why are simple things getting more and more complicated (firewalld? ;-))

Robo.li: reducing code base and other tricks

Last time I showed some basic tricks on how to use Robo.li with ease and that was a big post (compared to my other ones), but still didn’t cover some of the very basic things that can save a lot of time and efforts.

Abstract Base Task

To avoid doing same thing over and over again, let’s start with the abstract base task that will extend \Robo\Task\BaseTask. From the last time example you saw that my tasks were extending the Foo\Robo\AbstractTask, so let see three main things that are done there:

<?php

namespace Foo\Robo;

use \Robo\Common\ConfigAwareTrait;
use Robo\Task\BaseTask;
use Robo\Result;

/**
 * Foo base task.
 */
abstract class AbstractTask extends BaseTask
{
    use ConfigAwareTrait;

    /**
     * @var array $data Task data fields
     */
    protected $data = [];

    /**
     * @var array $requiredData List of required data fields keys
     */
    protected $requiredData = [];

    /**
     * @var string $configPrefix Config path prefix
     */
    protected static $configPrefix = "task.";

    /**
     * @var string $configClassRegexPattern Regex to extract class name for config
     */
    protected static $configClassRegexPattern = "/^.*Tasks?\.(.*)\.[^\.]+$/";

    /**
     * @var string $configClassRegexReplacement Regex match to use as extracted class name for config
     */
    protected static $configClassRegexReplacement = '${1}';

    public function __construct($params)
    {
    }

    /**
     * {inheritdoc}
     */
    public function run()
    {
        // for any key defind in data
        // set it to config value, if available
        foreach ($this->data as $k => $v) {
            $default = $this->getConfigValue($k);

            // specifically check for null to avoid problems with false and 0
            // being overwriten
            if ($this->data[$k] === null && $default !== null) {
                $this->data[$k] = $default;
                continue;
            }
            // if key value is an array, merge the config value to it
            if (is_array($this->data[$k]) && is_array($default)) {
                $this->data[$k] = array_merge_recursive($this->data[$k], $default);
            }
            continue;
        }        

        // check if we have all required data fields
        $res = $this->checkRequiredData();
        if (!$res->wasSuccessful()) {
            return $res;
        }

        // general success, as will be overriden by child classes
        return Result::success($this, "Task completed successfully", $this->data);
    }


    /**
     * Magic setters via __call
     * Make sure only valid data passes thtough
     *
     * @param string $name data key name
     * @param mixed $value data value name
     */
    public function __call($name, $value)
    {
        // we use snake_case field keys
        // but camelCase setters
        $name = $this->decamelize($name);

        // only set values for predefined data keys
        if (array_key_exists($name, $this->data)) {
            $this->data[$name] = $value[0];
        }

        return $this;
    }

    /**
     * Check that all required data present
     *
     * @return \Robo\Result
     */
    protected function checkRequiredData()
    {
        $missing = [];
        foreach ($this->requiredData as $key) {
            if (!isset($this->data[$key]) or empty($this->data[$key])) {
                $missing []= $key;
            }
        }

        return count($missing)
            ? Result::error($this, "Missing required data field(s) [" . implode(",", array_map([$this,"camelize"], $missing)) . "].", $this->data)
            : Result::success($this, "All required data fields preset", $this->data);
    }


    /**
     * Ported from Ruby's String#decamelize
     *
     * @param string $word String to convert
     * @return string
     */
    protected function decamelize($word)
    {
        return preg_replace_callback(
            '/(^|[a-z])([A-Z])/',
            function ($matches) {
                return strtolower(strlen($matches[1]) ? $matches[1] . "_" . $matches[2] : $matches[2]);
            },
            $word
        );
    }

    /**
     * Ported from Ruby's String#camelize
     *
     * @param string $word String to convert
     * @return string
     */
    protected function camelize($word)
    {
        return preg_replace_callback(
            '/(^|[a-z])([A-Z])/',
            function ($matches) {
                return strtoupper($matches[2]);
            },
            $word
        );
    }

    /**
     * Override of Robo\Common\ConfigAwareTrait configPrefix()
     */
    protected static function configPrefix()
    {
        return static::$configPrefix;
    }

    /**
     * Override of Robo\Common\ConfigAwareTrait configClassIdentifier($classname)
     */
    protected static function configClassIdentifier($classname)
    {
        return preg_replace(
            static::$configClassRegexPattern,
            static::$configClassRegexReplacement,
            str_replace(
                '\\',
                '.',
                $classname
            )
        );
    }

    /**
     * Override of Robo\Common\ConfigAwareTrait getClassKey()
     *
     * makes method protected instead of private
     */
    protected static function getClassKey($key)
    {
        return sprintf(
            "%s%s.%s", 
            static::configPrefix(),
            static::configClassIdentifier(get_called_class()),
            $key
        );
    }

    /**
     * A quick fix on printInfo, as it is not very friendly
     * when you use 'name' placeholders or even just have 'name'
     * set in the data
     */
    protected function printInfo($msg, $data = null)
    {
        // pass-through when no 'name' found in data
        if ($data == null || !isset($data['name'])) {
            return $this->printTaskInfo($msg, $data);
        }

        // doubt someone will use this ever in data
        $key = 'print_task_info_name_replacement_macro';

        // replace 'name' with above key both in data
        // and in msg placeholders
        $data[$key] = $data['name'];
        unset($data['name']);
        $msg = str_replace('{name}','{' . $key . '}', $msg);

        // print nice message
        $result = $this->printTaskInfo($msg, $data);

        return $result;
    }
}

Configuration

The first thing to note is that we use Robo’s config aware trait, that gives all our tasks access to the configuration (that we have covered in the previous post about Robo). This is very handy, as we don’t need hard-code anything and have couple of interesting tricks, but that’s a bit later. Obviously we had to override couple of methods from configAwareTrait to make thing work for us (and actually took me some time to figure out how all of this works in Robo).

Dynamic Data

The second trick here is to avoid using plain class properties and end up writing a hell amount of getters/setters/validators/etc. Instead we use $data array property with magic __call method that will do a job for us. The only restriction we put in here is that __call will set data only if key is already defined in the array. This restriction is there to avoid rubbish in our data, since we use to pass data as a whole set in many places, and need to make sure we know what’s in there. Another sub-trick here is that we user $requiredData property with a list of keys that are essential to us on run. While this is not something very tricky, it really helps in child classes run() method when called like

public function run() 
{
   $result = parent::run();
   if (!$result->wasSuccessful()) {
       return $result;
   }

   ...
}

So all base validation is done in parent class as long as you have your $data and $requiredData populated.

Please note that we also have to utility methods camelize() and decamelize() that are handy, as magic methods for setters are in camelCase, while actual data keys are in snake_case and those two methods are used to convert between the two.

Dynamic Data Configuration

The third trick, which is actually happens even before the second one, is that since we have access to configuration, whenever something is not set in $data during the run, but we have it in the corresponding config path – apply it to the data, but be careful apply only those configuration variables, that are present as our $data key. This is very handy and very flexible. First of all, many of the defaults are defined in the config and are easy to change. The second advantage is that we can have some other custom things in the config that can be used by any other custom command or component, that will not conflict with our tasks. For example configuration of an API with keys/secrets. Moreover, since tasks can also write to the config, same tasks can share things between them if required (for example some runtime cache data).

Task Information Output

Another handy Robo’s Task thing is the ability to inform about task activity via

$this->printTaskInfo("some message with {some_var} variable", ['some_var' => $this->some_var]);

This is a very cool feature and it looks awesome in the console, as some_var will be highlighted and so on. The above example is exactly of the format I found it on Robo.li documentation, but while using it, I found two problems:

Coming up with different {some_var} macros and using the second arg as above is just ugly and unproductive. That was one of the things that pushed me to using $this->data array instead, so in my case, when I need to print something, I will just do like this:

$this->printTaskInfo("This guy's name is {name}", $this->data);

Assuming we have $this->data[‘name’] of course. The trick with the data property saved me a day, but later on I found out one drawback: the output has a defined format, and if the above would run from \Foo\Robo\Task\Guy\PrintName, the output I would expect would be:

[\Foo\Robo\Task\Guy\PrintName] This guys name is Some Guy

But instead I was getting

[Some Guy] This guys name is Some Guy

This is due to the way Robo loggers are trying to find their context and finally end up using name supplied as their name. To fix the problem, I simply wrote a printInfo() method that would replace name with something else during the debug.

Task Return Value

Make sure your task always returns \Robo\Result instance as it will make your life much easier when you will be going to use your tasks in the commands. Make sure you always specify the reason/message that clearly describes why you return this type of result both for success and errors. Finally make sure you return some kind of supporting data (passed as a third argument to result constructor) to make commands life even easier. Normally I would put $this->data  in most of the cases, for example:

public function run()
{
    try {
        $res = $this->tryToValidataDataExampleMethod();
        if (!$res) {
            return Result::error($this, "Failed to validate data", $this->data);
        }
        return Result::success($this, "Data successfully validated", $this->data);
    } catch (\Exception $e) {
        return Result::fromException($this, $e);
    }
}

There few interesting things you can do with the data from the commands, but that I will probably cover in a later posts.

Asterisk: Initiate a call from extension by PHP script

If you have an asterisk PBX and some kind of internal web-based system like intranet or CRM with a contacts that your team need to call from time to time, there is an easy way to allow users fast dial those contact by linking a telephone number on a web page with a script that will call Asterisk and instruct it to dial first the user extension, and then connect him to the contact number.

The example code, that was originally taken from here and slightly modified looks like this:

# ip address that asterisk is on.
$strHost = "127.0.0.1"; 

# asterisk manager username and password
$strUser = "admin";
$strSecret = "secret_password"; 

# specify the channel (extension) you want to receive the call requests with
# e.g. SIP/XXX, IAX2/XXXX, ZAP/XXXX, etc
$strChannel = $_REQUEST['exten'];
$strContext = "from-internal";

$number = strtolower($_REQUEST['number']);
$strCallerId = $number;

#specify the amount of time you want to try calling the specified channel before hangin up
$strWaitTime = "30";

#specify the priority you wish to place on making this call
$strPriority = "1";

# validation
$valNumber = '/^\d+$/';
$valExt = '/^(SIP|IAX2|ZAP)\/\d+$/';

if (!preg_match($valNumber, $number)) {
    print "The number is incorrect, should match '$valNumber' pattern\n";
    exit();
}
if (!preg_match($valExt, $strChannel)) {
    print "The extension is incorrect, should match '$valExt' pattern\n";
    exit;
}

$errno=0 ;
$errstr=0 ;
$oSocket = fsockopen ($strHost, 5038, $errno, $errstr, 20);

if (!$oSocket) {
    echo "$errstr ($errno)<br>\n";
    exit();
}

fputs($oSocket, "Action: login\r\n");
fputs($oSocket, "Events: off\r\n");
fputs($oSocket, "Username: $strUser\r\n");
fputs($oSocket, "Secret: $strSecret\r\n\r\n");
fputs($oSocket, "Action: originate\r\n");
fputs($oSocket, "Channel: $strChannel\r\n");
fputs($oSocket, "WaitTime: $strWaitTime\r\n");
fputs($oSocket, "CallerId: $strCallerId\r\n");
fputs($oSocket, "Exten: $number\r\n");
fputs($oSocket, "Context: $strContext\r\n");
fputs($oSocket, "Priority: $strPriority\r\n\r\n");
fputs($oSocket, "Action: Logoff\r\n\r\n");
sleep(2);
fclose($oSocket);

echo "Extension $strChannel should be calling $number." ;

So, for example, if you put this code on the asterisk server in a web root as call.php and then call http://<your_asterisk_ip_address>/call.php?exten=SIP/737&number=77777777, your asterisk will attempt to connect extension 737 with number 77777777.

Robo.li and how to cook it

Introduction

One of the things I am heavily involved to is doing all sorts of infrastructure automation, and while ansible is a perfect tool for system administration automation, when it comes to more complex things like interacting with different APIs, and making all kind of more hardcore things, Robo.li (robo) is what can really save you.

I have started learning robo few weeks ago, but the problem is that they have pretty short documentation, and few basic examples, so it was not that easy at all and I am still in the process of finding best ways to go around, but I already have quite few things that I want to share here for anyone to use and for me have it saves in a safe place.

Ok, this one gonna be long enough and even probably a series of posts, but I have to share my experience.

While you can use single RoboFile.php with all commands you need in the project directory and call it via ./vendor/bin/robo, this is not a very comfortable approach for complex things, as this file get huge and messy very-very fast. For a proper use of robo, you should write a set of tasks, a set of command classes and then use all of it from your stand-alone robo app. For a sake of example, lets assume that all robo related commands and tasks are under src/Robo directory with Foo\Robo namespace prefix and you have a robo.php file with the actual app that you call.

To make it clear, here is a directory structure I have with some real tasks and commands already in place:

.
├── composer.json
├── composer.lock
├── .env
├── README.md
├── robo.php
└── src
    └── Robo
        ├── AbstractApiTask.php
        ├── AbstractCommand.php
        ├── AbstractTask.php
        ├── Command
        │   ├── BitbucketCommand.php
        │   ├── HipchatCommand.php
        │   └── RedmineCommand.php
        └── Task
            ├── Bitbucket
            │   ├── BranchRestrict.php
            │   ├── BranchUnrestrict.php
            │   ├── RepoCreate.php
            │   └── RepoDelete.php
            ├── Hipchat
            │   ├── RoomCreate.php
            │   └── RoomDelete.php
            └── Redmine
                └── ProjectCreate.php

(vendor directory is excluded on purpose to make tree clear)

Configuration

So now we have all in place and first thing first – we need to configure our app with a variety of different parameters. Robo does have Config implemented and configuration part is covered in a Robo as a Framework section of their site, but that mostly rely on storing your configuration in YML files, while I already have all configuration in a $config array and need to supply it to robo. To do that, we need to create a default robo container and pass our config to it, so in robo.php we have:

\Robo\Robo::createDefaultContainer(null, null, null, $config); 
$statusCode = \Robo\Robo::run(
    $_SERVER['argv'],
    $commandClasses,
    "Foo Robo",
    "v0.0.1"
);

Moreover, as we are passing the $commandClasses array with a list of the command classes, we want that part to be auto-generated as well, so above the code that creates a container we have:

$cmdPath = "src/Robo/Command";
$cmdNamespace = "\\Foo\\Robo\\Command";
$commandClasses = [];
foreach (glob("$cmdPath/*.php") as $file) {
    if (preg_match('/^.*\/(.*)\.php$/', $file, $matches) and class_exists($cmdNamespace . "\\" . $matches[1])) {
        $commandClasses []= $cmdNamespace . "\\" . $matches[1];
    }
}

And that’s pretty much all we need to have in robo.php apart of the code that fill the $config array, which is irrelevant to this blog post.

Task auto-loading

One thing that I found annoying in Extending robo how-to is the way they load tasks to commands. Basically for each collection of tasks you would have a loadTasks trait that describes each task and then you use it in the command class. While traits are questionable by nature, writing lots of same functions for a purpose of describing tasks is boring, time consuming and no way flexible. So instead of doing trait bullshit, we go with magic __call method in the AbstractCommand class that is extended by all actual tasks:

<?php
/**
 * Base command class for Foo Robo.li
 *
 * @see http://robo.li/
 */
namespace Foo\Robo;

use \Robo\Common\ConfigAwareTrait as configTrait;

abstract class AbstractCommand extends \Robo\Tasks
{
    use configTrait;

    /**
     * @var string $taskDir path to Tasks dir relative to our namespace
     */
    protected $taskDir = 'Task';

    /**
     * Magic __call that tries to find and execute a correct task based
     * on called method name that must start with 'task'
     *
     * @param string $method Method name that was called
     * @param array $args Arguments that were passed to the method
     *
     * @return 
     */
    public function __call($method, $args = null)
    {
        if (preg_match('/^task([A-Z]+.*?)([A-Z]+.*)$/', $method, $matches)) {
            $className = __NAMESPACE__ . "\\" . $this->taskDir . "\\" . $matches[1] . "\\" . $matches[2];
            if (!class_exists($className)) {
                throw new \RuntimeException("Failed to find class '$className' for '$method' task");
            }
            return $this->task($className, $args);
        }
        throw new \RuntimeException("Called to undefined method '$method' of '" . get_called_class() . "'");
    }
}

This class follows the naming convention and directory structure to auto-load tasks based on the method names and in out example the following code will work as expected from inside any command:

$this->taskHipchatRoomCreate()
     ->name("foobar")
     ->run();

$this->taskBitBucketRepoCreate()
     ->name("foobar.com")
     ->run();

So now we are focused only on writing tasks (example):

<?php

namespace Foo\Robo\Task\Hipchat;

use Robo\Result;
use Foo\Utility\Hash;

/**
 * Creates a HipChat room.
 *
 * 
 * <?php
 * $this->taskRoomCreate()
 * ->name('testroomname')
 * ->topic('This is a test room. Feel free to chat')
 * ->privacy('public')
 * ->run();
 * ?>
 * 
 */
class RoomCreate extends \Foo\Robo\AbstractApiTask
{
    /**
     * @var array data
     * @see https://www.hipchat.com/docs/apiv2/method/create_room
     */
    protected $data = [
        'name'                      => null,
        'privacy'                   => 'public',
        'delegate_admin_visibility' => null,
        'topic'                     => null,
        'guest_access'              => false
    ];

    /**
     * @var array requiredData
     * @see https://www.hipchat.com/docs/apiv2/method/create_room
     */
    protected $requiredData = [
        'name'
    ];

    /**
     * {@inheritdoc}
     */
    public function run()
    {
        
        $result = parent::run();
        if (!$result->wasSuccessful()) {
            return $result;
        }

        $this->printInfo("Creating {name} room", $this->data);

        try {
            $data = $this->api->post('room', $this->data);
        } catch (\Exception $e) {
            return Result::fromException($this, $e);
        }

        if (!$data || !is_array($data)) {
            return Result::error($this, "Failed to create room", $this->data);
        }

        // Because the create call returns only id, we'll make a new call to get
        // complete info about new room

        $this->printInfo("Retrieving {name} room info", $this->data['name']);
        
        try {
            $data = $this->api->get('room/' . $data['id']);
        } catch (\Exception $e) {
            return Result::fromException($this, $e);
        }

        if (!$data || !is_array($data)) {
            return Result::error($this, "Failed to retrieve room info", $this->data);
        }

        $data = array_filter(Hash::flatten($data));
        return Result::success($this, "Room successfully created", $data);
    }
}

and commands (example):

<?php

/**
 * Foo HipChat Robo.li commands file
 *
 * @see http://robo.li
 */

namespace Foo\Robo\Command;

use \Consolidation\OutputFormatters\StructuredData\PropertyList;
use \Foo\Robo\AbstractCommand;

class HipchatCommand extends AbstractCommand
{

    /**
     * Create a HipChat room
     *
     * @param string $name Room name
     * @param string $topic Room topic
     * @param string $privacy Room privacy
     *
     * @option string $format Output format (table, list, csv, json, xml)
     * @option string $fields Limit output to given fields, comma-separated
     *
     * @return PropertyList Room info
     *
     * @field-labels:
     *   id: id
     *   name: name
     *   topic: topic
     *   privacy: privacy
     *   xmpp_jid: xmpp_jid
     *   is_archived: is_archived
     *   is_guest_accessible: is_guest_accessible
     *   owner.id: owner_id
     *   owner.name: owner_name
     *   created: created
     *   last_active: last_active
     */
    public function hipchatRoomCreate($name, $topic = "", $privacy = 'public', $opts = ['format' => 'table', 'fields' => ''])
    {
        $result = $this->taskHipchatRoomCreate()
            ->name($name)
            ->topic($topic)
            ->privacy($privacy)
            ->run();

        return new PropertyList($result->getData());
    }

    /**
     * Delete HipChat room
     *
     * @param string $name Room name
     *
     * @option bool $force Force deletion
     *
     * @return bool true on success and false on failure
     */
    public function hipchatRoomDelete($name, $opts = ['force' => false ])
    {
        if (!$opts['force']) {
            $this->say("Won't delete anything unless you force me to");
            return false;
        }

        return $this->taskHipchatRoomDelete()
            ->name($name)
            ->run();
    }
}

I hope this post will be useful for someone (at least for me). There are few things in the examples above that are also tricks and not really standard to robo, including printInfo() method, data and requiredData properties, etc, but that will be covered later in another post.

AWS net.ipv4.tcp_tw_recycle follow-up

Yesterday I wrote a post on AWS EC2 instance networking problem that I was pretty surprised to find out. And while yesterday I was focusing on fixing the problem, today my first task was to find out what actually sets the flag, and quick grep on /etc of the instance revealed that the settings were applied by /etc/sysctl.d/net.ipv4.tcp_tw_recycle.

Very strange to find it there along with net.ipv4.tcp_tw_reuse which is also something that you should not touch. Anyhow, the problem identified, fixed and is about to be added to monitoring…