Architectural diagram of App - using serverless

If you do a lot of DevOps deployments - let's say with terraform for example - or train machine learning models, one slightly minor hit on productivity is that they can take a while to run and you're not sure when they're done. While not knowing exactly when a task is done is pretty minor, it can add up over time and account for a good chunk of time.

For example, you run a terraform deployment that includes creating some RDS instances, which might take up to ten minutes. In the meantime you go to code review pull requests, and next thing you know, it's been 15 minutes, 5 past when your terraform script finished. Of course that's not always a bad thing, but often times when running tasks via the CLI or prototyping, you have serial dependencies, and the faster you can kick off the next step, the faster you can switch back to more code review, and not introduce unnecessary delays to your manual pipeline.

This is where Vigil comes in. It is a simple and free managed service (so no SLA, don't use for production systems!!) intended to speed up your "manual pipelines" or iterative development by letting you know when a script/CLI task has finished. Check out the application architecture in the above diagram. The app is powered by serverless which lets us run on-demand at low cost and scale up when needed. The other services used are postgres via RDS, SES for email notifications, and SNS for SMS notifications.

Note again this is more for "manual pipelines" or iterative development, running tasks from the shell. For automated pipelines in CI/CD tools or taskflow tools like Airflow/Luigi, there is usually a notification hook built-in or via an extension.

And this brings up another good point; in these automated pipelines, we're typically only interested in a notification when a task fails. Another benefit of using Vigil for wrapping tasks, compared to running tasks naked, is that if we move to some other work while the task runs, and the task unexpectely fails early, we may not check back until a while later (our expected successful runtime) and we lose a lot of time. The notifications also tell you the status of the task (whether it failed/succeeded).

Example usage of Vigil CLI to notify when a ML model has finished training.

This command generates the following email notification.

Email notification

And the following sms notification

SMS notification

The cli can also be run with the following options:

vigil -help

Usage of vigil:
 -r, --result-all Show all output in notifications
 -e, --result-error Show output in notifications of command fails

vigil -r echo "testing"

Elapsed time: 134ms
Forwarded task status to notifier

You can sign up for a free Vigil account here and find the installation / usage instructions on the home page when you log in. Installation is just downloading the cli binary wrapper for your system, configuring your token for authentication, and putting your email or phone number into the web interface. The reason that this is a managed service rather than a completely stand-alone binary is that it's providing you with a free, pre-configured gateway for sending our email or SMS.

Well that's it! Please let me know if you find this tool useful so I know to put in more effort updating it. I plan to add some more useful features like push notifications on desktop/mobile and logging of past commands.