Cloud, Big Data, and the Internet of Things

Automic Blog

Subscribe to Automic Blog: eMailAlertsEmail Alerts
Get Automic Blog via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Related Topics: Agile Software Development, Big Data on Ulitzer, DevOps Journal


Agile Analytics | @DevOpsSummit #DevOps #BigData #ContinuousDelivery

DevOps + Continuous Delivery + Big Data = Continuous Analytics

Five Key Requirements for Enabling Agile Analytics
By Yann Guernion

In today's digital economy, companies are faced with a fast data challenge as well as a Big Data one. As a result they are under pressure to adapt their analytics processes and data flows at pace to move beyond traditional data warehouse silos.

Big Data projects are either too big or too complex to handle the traditional way. That's why most projects by companies at the start of their Big Data initiative have no process at all. Waterfall approaches are notably inefficient as you probably won't have access to proper staging environment and only limited time and scale for qualification.

Big Data and DevOps
Big Data implicitly promotes DevOps because there is no ability to separate Operations from Development when you ultimately discover the relevance of your algorithms at the production stage. It is inevitable that Big Data will either promote a DevOps initiative or feed in to your current DevOps model if a transformation has already occurred in your IT department.

Continuous Analytics is an extension of DevOps and Continuous Delivery when applied to Big Data. It provides ways to incorporate analytics into the agile model where building, testing, provisioning and deploying are all run as automated processes.

Of course putting this to work requires you to address some specific organizational and automation challenges:

  1. Data Scientists and Data Analysts that usually work in isolation from the Big Data Engineers and Architects need to be onboarded to impose higher levels of process standardization and make handoffs easier.
  2. The complexity of underlying Big Data technology must be hidden from Data Scientists by exposing clean high-level APIs to them, and predefined components that can be used in simple graphical user interfaces.
  3. Data Scientists have to store their code, workflows and associated artifacts to the same repository that developers and other Big Data team members are using, so the application objects can be easily pulled out and orchestrated during the deployment process.
  4. Big Data infrastructure has to be abstracted so that it can be pushed out in the manner of a continuous release and deployed to any type of environment, whether is it private, public or hybrid cloud. The ideal state would look like a ‘Hadoop-as-a-Service' consumption model.
  5. Release coordination needs to be combined with deployment automation across the delivery pipeline, whether open-source or commercial tools are used.

Access to the right information at the right time is essential to make informed business decisions, get ahead of the competition and drive innovation. Heavy processes simply do not fit into our fast-changing environment, which is why Continuous Analytics leverages what has been learned from Agile and DevOps and applies these methodologies to Big Data development.

In five years enterprises will be dealing with even larger volumes of structured and unstructured data and using it to make even smarter decisions. To be successful, they need to allow Data Scientists and Big Data developers to get to the data and analytics as quickly as possible. So now is a good time to consider looking into Continuous Analytics, as more than ever, speed is of the essence.

Read the original blog entry...

More Stories By Automic Blog

Automic, a leader in business automation, helps enterprises drive competitive advantage by automating their IT factory - from on-premise to the Cloud, Big Data and the Internet of Things.

With offices across North America, Europe and Asia-Pacific, Automic powers over 2,600 customers including Bosch, PSA, BT, Carphone Warehouse, Deutsche Post, Societe Generale, TUI and Swisscom. The company is privately held by EQT. More information can be found at www.automic.com.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.