Cloud, Big Data, and the Internet of Things

Automic Blog

Subscribe to Automic Blog: eMailAlertsEmail Alerts
Get Automic Blog via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Related Topics: Big Data on Ulitzer, DevOps Journal

Blog Feed Post

Creative Data Automation and Orchestration By @Automic | @BigDataExpo #BigData

Big data, automation, social media and other technologies combine to give us unprecedented convenience today

How Creative Data Automation and Orchestration Give Stunning End Results
By Larry Salomon

Data automation and orchestration have given us free two-day shipping via Amazon Prime; next day shipping via the major postal companies; delivery via Uber or drone; and so on. These are all concepts that we are either intimately familiar with as consumers of these services or at least are cognizant of them because of news reports, TV commercials or YouTube videos.

Convenience and digital transformation
The value of convenience cannot be overstated. However, our need for instant gratification dictates that there will be times when these methods of delivery, unthinkable just a few years ago, are not enough. For example, a freak snow blizzard in my area wasn't a big deal for me since I have snow removal equipment. But as I drove past kids sledding in the field at the end of the street I wanted to buy a sled for my children also. Next day shipping wouldn't have satisfied my need to have a sled now.

In spite of this, technology has afforded us a level of agility that didn't exist ten years ago. For example:

  • Netflix uses data automation to perform real-time analysis of their customers' viewing habits to suggest new movies they may want to watch. This starts with an ETL process that gathers the data and normalizes it into a format that is more easily digestible by the balance of the algorithmic analysis engine, which results in bespoke recommendations for the viewer.
  • LinkedIn uses data automation to perform real-time analysis of their users' network to suggest people that they may know based on degrees of separation. This starts with several ETL processes that run on Hadoop, Teradata, and other data sources to enable this functionality to work.
  • MC Theaters uses data automation to perform real-time analysis of ticket and concession sales, which automatically updates their social media feeds. This starts with an ETL process on Hadoop to gather, normalize, and process information to post updates to various social media sites with no user intervention.


Digital growth
Anything that defines "a quality experience" for a business is quantifiable and should be measured. Why not? After all, this need exists whether you are a brick and mortar business, an online business, or a hybrid of the two like AMC. The process of measuring, gathering, analyzing and reporting unearths what is and is not working so success can be capitalized on in real time.

As you can see from the examples above, the results can be stunning. But the quantity of data that these companies generate means none of this would be possible without data automation, which is geared specifically at real-time execution of jobs to gather and then process large amounts of data in real-time.

Orchestration backbone
There are a number of platforms to process large amounts of data (Hadoop, Hortonworks, and Cloudera, to name a few). However, the orchestration to coordinate the activities around gathering and reporting, is critical. Without it, you would be stuck with the task of building a custom set of solutions to do these two very important activities. It is this orchestration that distinguishes a large data processing platform like Hadoop from a data automation platform that enables you to effect a change in your company based on what the data contains.

The other thing all the above examples have in common? They all chose Automic to provide the orchestration capabilities to allow them to transform their businesses into something that responds in real time to the needs of their customers and users.

As Vijay Aruswamy, Big Data Operations at LinkedIn Corporation, put it, "We're using Automic to automate data transformation on Hadoop, Teradata, User Input and External sources of information. It supports our data driven products like ‘People You May Know' or ‘Jobs You May Be Interested In'."

Automation, when properly harnessed, means the difference between being able to continuously innovate and rapidly becoming irrelevant. And though large scale adoption of automation is not an endeavor to be undertaken on a whim, examples of data automation like those listed above show the outstanding benefits that come as a result of this investment that you make for your company's future.

Read the original blog entry...

More Stories By Automic Blog

Automic, a leader in business automation, helps enterprises drive competitive advantage by automating their IT factory - from on-premise to the Cloud, Big Data and the Internet of Things.

With offices across North America, Europe and Asia-Pacific, Automic powers over 2,600 customers including Bosch, PSA, BT, Carphone Warehouse, Deutsche Post, Societe Generale, TUI and Swisscom. The company is privately held by EQT. More information can be found at www.automic.com.