Introducing the Scalr Scheduler

March 5th, 2010

Today Scalr released the Scheduler, a Cron Job task manager designed for Cloud environments.

The problem is that Cron jobs, which are scripts that are executed periodically, were up until now tied to individual servers. When that individual server failed, the Cron job wouldn’t be executed. When that server was scaled, it would be executed once per server. This means that the tasks you set to be executed once per hour, could be executed 5 times in the hour, or not at all – not very reliable, to say the least.

Scalr based the Scheduler off Apache ZooKeeper, which is a part of Hadoop. It is a centralized service for providing group services, especially distributed lock services. The Scheduler extends Scalr’s scripting interface, and allows you to write scripts and set their execution schedule and scope: once on all servers, every hour on a subset of servers, daily on a single server, and any other combination!

As always, the code is available on Google Code and delivered as a service at scalr.net.

***

You can find the Scheduler under the Tools menu, found at the top when logged in:

Tasks Scheduler drop-down

You can schedule tasks to be executed periodically using the Scalr Scheduler

To add a Cron job or schedule a script, click the + icon next to the Search box:

Click the + icon to add a new cron job or to set a schedule for a script

You can then select the script to be executed, choose the parameters for it, schedule a time or interval, and run!

Configure the cron job's interval, parameters, and more

New Scalr release featuring the Scalr Scheduler

February 25th, 2010

Hi all,

We pushed a new release out this week, a few of you might have seen this already.

This is a rather important update, as it brings us closer to Scalr 2.0, which will extend Scalr to all Operating Systems and environments. You can find the complete list of features added below, and I’d like to highlight the Scheduler, the improved Monitoring system, as well as the shortcuts to your farms added to the top menu.

  • Added the Scheduler (Ability to execute script, launch farm or terminate farm at specified date/time)
  • Improved the Monitoring system, including an API call to get the graph’s URL
  • Monitoring now available on mobile.scalr.net, too
  • Added automatic snapshots for RDS instances
  • Added ability to update SOA refresh
  • Added AMI filter during Spot instances request
  • Added ability to download logs as csv file.
  • Improved navigation menu.
  • Various bugfixes and improvements.

Updated scalr.net, development.scalr.net and trunk repo for scalr o/s (http://code.google.com/p/scalr/source/browse/#svn/trunk).

2010 Resolution: keep Scalr on google code up to date!

January 14th, 2010

Hi everyone, happy new year!

As part of our 2010 resolutions, we decided to keep in sync the source code on Scalr.net and that on Google Code. This is good news for everyone that deployed Scalr on their own servers, but also to everyone else as the power of open source lies in that other people can contribute to it. If somebody wants support for Eucalyptus, they can modify the source code to add capability for it, and send us a patch. We then merge it to the trunk, and you and everybody else then gets it. Patches should be sent to patches@scalr.com.

If you have suggestions for other resolutions, let us know!

Below is a complete changelog:

Events system improvements:

  • Added new events: OnDNSZoneUpdated, OnEBSVolumeAttached

Scripting engine improvements:

  • Added the zone_name variable for scripts executed on DNSZoneUpdate event
  • Added the new_ip_address variable for scripts executed on IPAddressChanged event
  • Added the volume_id and mountpoint variables for scripts executed on EBSVolumeMounted event
  • The ability to execute scripts from the Scripts view page.

Snapshots manager improvements:

  • Remove multiple snapshots in one time
  • Share snapshots!

API improvements:

  • Added methods: LaunchInstance, TerminateInstance, GetFarmDetails, GetScriptDetails, RebootInstance, GetEvents, GetLogs
  • Added methods for working with DNS zones: ListDNSZones, ListDNSZoneRecords, AddDNSZoneRecord and RemoveDNSZoneRecord
  • Improved the ExecuteScript method.
  • Added LA for each instance to the GetFarmDetails method
  • Added the ability to execute scripts with specified revision and parameters

Core improvements:

  • Amazon RDS support
  • Amazon Spot instances support.
  • Amazon CloudWatch support (fine grained monitoring)
  • Amazon VPC support (enterprise feature, deploy on non-shared servers)
  • Rewrote the “Synchronize to all” feature. Cleaner and more reliable now.
  • Same goes for AutoEBS and AutoEIP. Cleaner and more reliable.
  • When you create a new AMI for a custom role, you can now switch over to it immediately.
  • And the long awaited “Keep me logged in” checkbox on the login page!
  • Filters for instances on the Servers view page.
  • CloudFront distributions for domains not managed by Scalr.
  • The ability to remove Elastic Load Balancers.
  • Support for new region: us-west-1 (deploy on the west coast!)
  • A new page with more details on the instance.
  • Increased page load speed. Optimized js code. (faster, better interface!)
  • An improved MySQL status page for your Farms
  • The ability to set whether Scalr should terminate or reboot instances that fail to respond to SNMP calls.
  • The ability to slowdown the scaling process
  • Support for new instance types, the high memory instances (32 and 68GB of memory)
  • The ability to add Google Apps MX records in the Zone Edit page in a single click.
  • The ability to edit system DNS records. For advanced clients.
  • The ability to set both size and snapshot for Role auto EBS (previously just one)
  • Added ability to edit farm role specified security group
  • Added Hide terminated instances checkbox on instances list.
  • Added ability to view all instances (include non-scalr ones)
  • Added ability to set system timezone for clients (Logs, Events, API logs)
  • Fixed bug in garbage.php with “select all” checkbox
  • Fixed bug with default SSH port (see thread)
  • More than 200 bugs was fixed and tons of other internal improvements.

To update from 1.1 to 1.2:

  1. Backup your database and all files.
  2. Disable cronjobs.
  3. Copy app/* to your directory with scalr.
  4. Execute in shell /path_to_your_scalr/bin/upgrade-to-1.2.0.php
  5. Enable cronjobs.

One more release squeezed in 2009!

December 30th, 2009

Scalr now supports Spot Instances from AWS! You can find the official documentation here: http://aws.amazon.com/ec2/spot-instances/

Spot instances is theoretically a great way to lower your web infrastructure costs: since the hourly cost of a Spot instance will likely not go above the rate of an On Demand instance, then if you set your Spot price above that, you will always keep your instances running. You’ll then be saving the difference between the current Spot price and that of a Reserved or  On Demand instance. Let us know if you try this and have any success with it.

To get started, from the menu: Tools -> Amazon Web Services -> Amazon EC2 -> Spot instances.

I hope everyone has a happy new year’s celebration!

Cheers,
Sebastian on behalf of all of Scalr

Our gift for the holidays: December release!

December 15th, 2009

Lots of Cloud goodness for everyone! The December release brings:

* Amazon CloudWatch support (fine grained monitoring)
* Amazon VPC support (enterprise feature, deploy on non-shared servers)
* Filters for instances on the Servers view page.
* CloudFront distributions for domains not managed by Scalr.
* The ability to remove Elastic Load Balancers.
* The ability to execute scripts from the Scripts view page.
* Support for new region: us-west-1 (deploy on the west coast!)
* A new page with more details on the instance.
* Various bugfixes and improvements.
* Increased page load speed. Optimized js code. (faster, better interface!)

On behalf of everyone at Scalr, we wish you happy holidays!

Cheers,
Sebastian, on behalf of the Scalr Team

New Scaling Algorithm

November 30th, 2009

We are pleased to announce a new algorithm to base your application / website scaling on, scaling based on URL response time.

If response time is critical to you, you can now set this parameter and Scalr will add more nodes when response time on a URL increases above a threshold, or remove nodes when the response time decreases below another.

To access this algorithm, simply edit the Farm from the options menu, select the Role for which to change the scaling algorithm, click the scaling tab, and select ‘Enable scaling based on HTTP URL Response time’. Then add your parameters and url.

Cheers,
Sebastian

More Scalr Goodness

November 24th, 2009

More Scalr goodness for Thanksgiving!

You can now terminate and reboot instances in mobile.scalr.net, and add comments/description for Farms in scalr.net.

There’s also an improved EBS snapshots manager; you can now

  • Remove multiple snapshots in one time
  • Share snapshots!

For those using the free development.scalr.net, it has been updated with the latest Production bugfixes, too.

November release

November 19th, 2009

Hi all!

We’re proud to bring you November’s release, as you’ll see filled with useful new features.

In scalr.net, you’ll find:

  1. A new event: OnEBSVolumeAttached
  2. An improved MySQL status page for your Farms
  3. Initial Amazon RDS support
  4. The ability to set whether Scalr should terminate or reboot instances that fail to respond to SNMP calls.
  5. The ability to slowdown the scaling process
  6. Support for new instance types, the high memory instances (32 and 68GB of memory)
  7. The ability to add Google Apps MX records in the Zone Edit page in a single click.
  8. The ability to edit system records. For advanced clients.
  9. API changes: Added LA for each instance to the GetFarmDetails method
  10. The ability to set both size and snapshot for Role auto EBS (previously just one)
  11. Internal improvements.

For Scalr Remote (mobile.scalr.net), you’ll find:

  1. Better layout when images are disabled
  2. Added version number in footer
  3. Added LA and Uptime on instance details page
  4. Added View instances from farm details page

Drop us a note if you have any feedback!

Database browsing in Scalr!

October 22nd, 2009

We integrated phpmyadmin, a database administration tool, into Scalr today. It allows you to see the data you have on your database, update individual records, alter tables, and do lots of other stuff.

This is another step into making Scalr a great tool for sysadmin work, and making great sysadmin more accessible to non-specialists.

You can find it under Farms > View, and from options, select MySQL status. Click ‘Setup PHPMyAdmin access’, wait a minute, refresh, and you’re done!

Let us know what you think!

Scalr.net Performance Improvements

October 14th, 2009

If you used Scalr.net these last few days, you probably experienced some mysql connection errors. These were due to a very large amount of concurrent connections, typically when a large farm (100+ instances) is launched, as it triggers too many requests too quickly, before Scalr can react to load.

This was more of an architectural flaw, so we worked to reduce the amount of requests every instance makes. In many cases we got it down to a single request. For example, when an instance requests a list of instances of a role, it now gets all the information for all roles in the single initial request. Same goes for the config_opts queries from instances, equally optimized (we brought down the amount of requests to rebuild /etc/aws/hosts from 5 requests to a single one).

The next thing we did is tune mysql to handle thousands of connections, and over 100 other settings and sysctl options, then optimized our db structure (incl. added new indexes).

We also moved some stuff higher up in the stack to nginx, to be served faster.

We took the occasion to rewrite the client dashboard so that logs load instantly. You’ll notice this when you first log in.

Bottom line is that things are faster for you, and put less load on us.