Using DevOps to help increase Systems of Engagement

What Is DevOps needed for Systems of Engagement?

Making any kind of change in business is often hard and typically requires an investment. Anytime an organization adopts any kind of new technology, methodology, or approach, that adoption ought to be driven by a business need. To build a business case for adopting DevOps, one must understand the business need for it, which includes the challenges that it addresses.

The Business Need

Organizations strive to create innovative applications or services to solve business problems. This might be either to address internal business needs or provide services that reach out to their clients or end users. A majority of the organizations have challenges undertaking software projects successfully, and their failures are often associated with challenges in software development and delivery. Though the majority of enterprises consider software development and delivery are critical a tiny percent feel that their teams are effective. This execution gap leads to missed business opportunities.

This problem has been further amplified by a significant shift in the kinds of applications that businesses are expected to deliver, from systems of record to systems of engagement:

Systems of record:

Conventional software applications are often large systems that function as systems of record, which include things like massive amounts of data and/or transactions and are intended to be highly reliable and stable. As these applications don’t need to change often, organizations will often meet the needs of their customers and their own business needs by delivering only one or two significant new releases a year.

Systems of engagement:

With the advent of mobile communications and the maturity of web applications, systems of record are being supplemented by systems of engagement, which customers can access directly and use to interact with the business. Such applications must be easy to use, high performing, and capable of rapid change to address customers’ changing behavior and evolving market forces.

Because systems of engagement are utilized directly by customers, they demand intense focus on user experience, speed of delivery, and agility – in other words, a DevOps approach.

Systems of engagement aren’t isolated islands and are often tied to systems of record, so rapid changes to systems of engagement bring about changes to systems of record. Any kind of system that needs rapid delivery of innovation requires DevOps. Such innovation is driven primarily by emerging technology trends such as cloud computing, mobile applications, Big Data, and social media, which may affect all types of systems.

Recognizing the Business Value

DevOps applies agile and lean principles across the entire software supply chain. It enables a business to maximize the speed of its delivery of a product or service, from initial idea to production release to customer feedback to enhancements based on that feedback.

Because DevOps improves the way that a business delivers value to its customers, suppliers, and partners, it’s an essential business process, not just an IT capability.

DevOps provides significant return on investment in three areas:

  • Enhanced customer experience
  • Increased capacity to innovate
  • Faster time to value

In conclusion, DevOps helps deliver better systems of engagement and help business reach their customers by adapting to their changing behavior and keep them engaged

Service Oriented Architecture vs Microservices Architecture

What is Service Oriented Architecture (SOA)?

​Service Oriented Architecture (SOA) which is a software structuring principle, the premise is that software systems are built around the concept of services these can be consumed by applications that can be independently be built around these services. Service providers published a service description that is available for consumers, consumers are able to interact with these using these descriptions (for example WSDL).

In a nutshell the goal of SOA is to decouple systems by allowing the service and client to evolve independently. By following proper versioning Services can provide newer capabilites for clients while supporting legacy clients to be operational and upgrade their capabilities based on their schedules. The follows are the attributes that SOA services adhere to

  • Boundaries are explicit
  • Service compatibility is based on policy
  • Services are autonomous
  • Services share schema and contract

SOA or Tiered Architecture

Microservices Architecture (MSA)

While SOA defines traits that Services need to adhere to the definition for Microservices are not well defined. Microservices are an evolution of SOA with the aim or creating modular services, the primary goal of Mircroservices are to evolve independently with a single application focus. An application can be composed of 100’s of Microservices, each driving an independent feature that allow them to be built on different platforms and deployed independently. Scalability is improved by allowing services to span multiple instances if the demand increases along with fault tolerance as a single service not loading will not cause the application to fail.

Microservices have gained in popularity with teams adopting Continuous Integration and Continuous Delivery, using rapid deployment techniques an application have 100’s or Microservices can have multiple deployments during the day. This is a big shift from SOA where large scale deployments were prevalent. The downsides of Microservices are that though they can be easily scaled they do have drawbacks as an in-memory function call is always less resource intensive than out-of-process component requests.

Microservice Architecture

Conclusion

SOA came about to decouple monolith applications and help usage of contracts to drive application development. This brought about the use of WSDL as a service description along with communication protocols that were standardized. This allowed for client to consume the service and also evolve at their own pace. The downsides for SOA were that deployments took longer as the size of the services grew and with a larger footprint scalability had challenges. Enter Microservices which takes advantage of the Continuous Integration and Continuous Delivery pipelines and also since they are feature focused can be scaled easier and also more fault tolerant.

Microservices vs SOA

See also

https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-overview-microservices

Agile Scrum Estimation Techniques

There are wide verity of estimation techniques available. Some of the prominent examples are

  • Counting
  • Expert judgment (individual and/or group)
  • Decomposition
  • Analogy
  • Proxy estimation
  • Planning poker
  • Triangulation
  • Wall estimation

And the typical units of measurements are

  • Hours
  • Days
  • Weeks
  • Months
  • Ideal Days
  • T-shirt sizing
  • Story Points

Typically, Scrum recommends use of Story Points measurement techniques.

Scrum Estimation  Sizing – Story Points

Story points are a unit of measure for expressing the overall size of a user story, feature or other piece of work – Mike Cohn

Story points tell us how big a story is, relative to others, either in terms of size or complexity. Story points are relative values, not fixed. There is no direct correlation between hours and points. For example, try to use animals instead of, like “dog points” for a relative sizing:, then Chihuahua would suggest a 2 points estimate, Beagle would be 5 points, Labrador would be 8 points and Great Dane 13 points.

We use Fibonacci sequence (1, 2, 3, 5, 8, 13…) as its easy to visualize estimates as a base ten value. Also, it is advisable that each team establish a fixed comparison points for estimation. Teams choose a Story from the backlog that is small and another one that is huge to establish this comparative estimation.

Scrum Estimation Negotiation Practices:

Planning Poker:

  • The Scrum Master presents the top item in the product backlog to the team.
  • The team discusses the details  related to the story
  • The product owner clarifies questions, assumptions, and unknowns—as well as acceptance criteria.
  • Each team member privately decides how big this story is relative to a reference story, a series of reference stories, or all of the stories on the product backlog
  • At the count of three, everyone shows his or her chosen card simultaneously.

If everyone played the same card, the team can log the estimate and move on to the next story, else have the low bidder and the high bidder both explain their reason estimates then repeat steps 4 and 5.

This is mostly done in a couple of iterations, for example…

Estimator Round 1 Round 2
Team member 1 3 5
Team member 2 8 5
Team member 3 5 8
Team member 4 2 5

Wall Estimation:

Designed to allow teams to eliminate discussions of 2 versus 3 and 5 versus 8 and instead group things in a purely relative manner along a continuum, at least initially. It also allows stakeholders to give a general prioritization to a large group of stories without getting hung up on whether one story is slightly more important than another. Good for a grooming session, so you can quickly estimate the raw backlog

  • Print User Stories into cards
  • Join the team and the stakeholders in a room with a big empty wall
  • Explain the rules:
  • Height determines priorities (top items are more critical)
  • Width determines size (left items are smaller)

As with planning poker, we are using relative sizing, but instead of 2 reference stories for comparison, the wall becomes the constant:

Small story? Move to the left. Big story? Move to the right. Important story? Place it high. A story that we can live without for now? Place it low.

est-1   est-2

Ref: https://msdn.microsoft.com/en-us/library/hh765979(v=vs.120).aspx

Retrospective

Retrospective is one of the key aspects of engagement of an Agile Scrum team facilitated by the Scrum Master, attended by implementation team and product owner. These meetings are conducted on the last day of the sprint after demo. Identify start, stop, and continue doing, things to try practices and learnings selecting one or two goals to work on during the next sprint.

Retrospectives is perhaps the most important Scrum meeting which facilitates the principles of evolving an adaptable framework, continuous improvement and inspect and adapt.

It is often observed that Retrospectives can become stale so it’s imperative to keep it engaging and participative and focus on coming up with action items.

Retrospective Guidelines:

The Goal: Create actionable items to improve any aspect of the development process after each sprint.

A Retrospective should:

  • Happen after every sprint.
  • Be long enough to allow team members to reflect and discuss the previous sprint (45mins – 1 hour.)
  • Produce actionable and measurable goals or tasks that can be done by team members.
  • Involve all team level developers, PM’s, Product Owners, and anyone else that was involved with the sprint.
  • Call out things done well and make a point of continuing to follow the same process that produced those results.

A Retrospective is not:

  • A code review
  • A blame game (We all should own our mistakes as a team)
  • Just another meeting

Retrospective Agenda

Gather Data

  • Record a shared set of data
  • Consider Objective and subjective data
  • Understand facts vs opinions
  • The retrospective focus guides what data is relevant for the retrospective

Generate insights

  • Move beyond habitual thinking
  • Observe patterns
  • Understand root causes and influences
  • Develop a shared awareness
  • Observe / understand systemic effects

Consolidate and prioritize ideas, Decide what to do

  • Focus on what the team can accomplish
  • Determine one or two actions or experiments
  • Don’t necessarily focus on what is ‘most important

Appreciations

“A simple thank you can make a difference; appreciation builds good will, and reminds people that they are valued as human beings, not just as CPUs (Code Producing Units) or FTEs (Full Time Equivalents).”

— Esther Derby

Close

  • Recap agreed-upon actions, actors (if needed), and follow up
  • Identify ways to improve the next retrospective (retrospect the retrospective)

Agile Methodology

Methodology – a set of methods, rules, or ideas that are important in a science or art: a particular procedure or set of procedures (Ref: merriam-webster). So, let’s step back a little, Agile is not a mere methodology but a new paradigm a mindset. It is an alternate to orthodox project management practices which prescribed sequential execution like with waterfall. Agile advocates a response mechanism to unpredictable changes through progressive delivery and feedback cycles.

The sequential execution model was inspired by the manufacturing sector which evolved during early industrialization of the world. The approach was phased with predefined gates, moving from one phase to another only after fully completing the current phase. For seven years now software has also been developed in the fashion moving from envisioning, planning, design, development, testing and release to production with clear gate checks at each phase entry & exit. This approach adored several inherent challenges including longer delivery time, rework identification late in the life cycle, money spent on unproductive activities, and lack of effective communication across teams due to compartmentalization.

Agile provides an opportunity to address these limitations through short iterative delivery cycles with endorsed work items at the end of each cycle.  The delivery cycles are iterations of 2-4 weeks duration leading to delivery of potentially shippable product increments. This approach accentuates continuous revisit of every aspect of software development in each of the iteration resulting in enduring course corrections of the projects.

This ‘inspect-and-adapt’ approach exemplify the spirit of Agile resulting in reduced time-to-marked, cost to deliver and improved acceptance. Additionally, the active involvement of users through the delivery cycle creates much better business engagement and customer satisfaction and finally the agile teams are empowered through active involvement, cooperation and collaboration resulting in highly motivated, high performance teams.

Scrum is the most widely used Agile practice due to its simplicity and flexibility. Scrum is characterizes by small cross functional self-managed teams delivering evolving requirements in short durations which are well verified. However, organizations face several challenges while adapting Scrum due to their org structure and practices.

Scrum has three main roles: Product Owner, Team, and Scrum Master. The responsibilities of the traditional project manager role are split up among these three Scrum roles. Scrum has five meetings: Backlog Grooming (aka Backlog Refinement), Sprint Planning, Daily Scrum (aka 15-minute standup), the Sprint Review Meeting, and the Sprint Retrospective Meeting. A more difficult and principled approach to agility in a multi-team organization is called Large Scale Scrum.

scrum

Agile Project Life Cycle

Agile life cycle is an iterative and incremental development process comprising four phases to deliver Business
Value in an incremental time boxed cross-discipline iterations. The four phases are

  • Inception
  • Elaboration
  • Construction
  • Transition

Detailed descriptions of each phase

Inception:

  • Establish a business case for the project – On request from Business Users ((IT Head)
  • Identify Program Manager, PO, Architect and TPM (IT Head)
  • Establish the project scope and conditions ((IT Head /Program Manager))
  • Outline the key requirements that will drive the design tradeoffs (IT Head /Program Manager)
  • Outline one or more candidate architectures (Architect/ Program Manager/PO)
  • Identify risks and resource needs (IT Head/Program Manager)

Outcome:

  • Project Charter – vision, goal, preliminary project schedule and cost estimate
  • Solution design
  • Identified Product Owner (PO), Solution Architect and Scrum Master (TPM)

Elaboration:

  • Address known risk factors (IT Head)
  • Capture a healthy majority of system requirements (BRD) – Development of a Product Backlog list (BA’s)
  • System Conceptualization and analysis – Establish and validate system architecture (Architect)
  • Estimate Cost and Schedule (IT Head/PO/SM)
  • For each item on Product Backlog list (PO)
    • Division of items into Sprints (after Prioritizing)
    • Assessment of risk and mitigation strategies
  • Conduct a Project Kick-off Workshop with the Scrum Team – Share Project Vision, Goal, Release Plan, Requirements Elaboration,
    Technical Solution Elaboration (PO)
  • Conduct Sprint 0 – Grooming, Prioritizing, Sizing, Tasking and Estimating (upcoming 2 to 3 sprints) (Scrum Team)

Outcome:

  • Product Backlog (Well defined User Stories)
  • Stabilized system architecture – that supports the key system functionality and ensures the right behavior in terms of performance,
    scalability, usability etc
  • Release Plan
  • Scrum Team

Construction:

  • Sprint Planning (Scrum Team)
  • User Stories/Product Backlog items are implemented (Coded and Tested) in ‘Sprints’ to result in an executable release of
    the software (Development Team)
  • Product backlog Grooming (Scrum Team)
  • Other Scrum Ceremonies – Daily Scrum, Review, Retrospective (Scrum Team)
  • Capture definite Metrics (Scrum Master/PO)

Outcome:

  • Sprint Plan (Commitment based/Capacity based/Velocity based)
  • Working Software
  • Metrics
  • Key Learnings and Continuous Improvement measures

Transition:

  • Project Readiness to ‘Go live’ – System deployment to the target users – UAT/Production
  • Further Refinements
  • Standardization
  • E-Learning, User Training, Knowledge Transfers etc

Outcome:

  • Release Notes
  • Final Build
  • Documentation

Understanding Covariance & Contravariance

What is Covariance & Contravariance?

Covariance – Enables you to use a more derived type than originally specified.
You can assign an instance of IEnumerable<Derived> to a variable of type IEnumerable<Base>.

Covariant type parameters enable you to make assignments that look much like ordinary Polymorphism, as shown in the following code.

IEnumerable<Derived> d = new List<Derived>();
IEnumerable<Base> b = d;

Contravariance – Enables you to use a more generic (less derived) type than originally specified.
You can assign an instance of IEnumerable<Base> to a variable of type IEnumerable<Derived>.

Action<Base> b = (target) => {Console.WriteLine(target.GetType().Name);};
Action<Derived> d = b;
d(new Derived());

Invariance – Means that you can use only the type originally specified; so an invariant generic type parameter is neither covariant nor contravariant.

You cannot assign an instance of IEnumerable<Base> to a variable of type IEnumerable<Derived> or vice versa.

Arrays/Delegates/Generics

Arrays – Are covariant

Bear[] bears = new Bear[3];
Animal[] animals = bears; // OK

The downside of this reusability is that element assignments can fail at runtime:

animals[ 0 ] = new Camel(); // Runtime error

Delegates – Allows for methods to have less specific input parameters (covariance) and more specific return types (contravariance)

Covariance – Parameter Compatibility

  • A method target can define a parameter that is more generic than described by the delegate OR
  • A delegate can have more specific parameter types than its methods target
delegate void StringAction (string s);

static void Main ()
{
   StringAction sa = new StringAction (ActOnObject);
   sa ( "hello" );
}
static void ActOnObject (object o)
{
   Console.WriteLine (o); // hello
}

Contravariance – Return Type Compatibility

  • A method target can return a more specific type than described by the delegate OR
  • A delegate can define a return type that is more generic than the method signature
delegate object ObjectRetriever();
static void Main()
{
    ObjectRetriever o = new ObjectRetriever (RetriveString);
    object result = o();
    Console.WriteLine (result);      // hello
}

static string RetriveString() { return "hello"; }

Generics

Covariance – Allowed for parameters marked with “out” modifier

// Covariant interface.
interface ICovariant<out R> { }

// Extending covariant interface.
interface IExtCovariant<out R> : ICovariant<R> { }

// Implementing covariant interface.
class Sample<R> : ICovariant<R> { }

class Program
{
    static void Test()
    {
        ICovariant<Object> iobj = new Sample<Object>();
        ICovariant<String> istr = new Sample<String>();

        // You can assign istr to iobj because
        // the ICovariant interface is covariant.
        iobj = istr;
    }
}

Contravariance – Allowed for parameters marked with the “in” modifier

// Contravariant interface.
interface IContravariant<in A> { }

// Extending contravariant interface.
interface IExtContravariant<in A> : IContravariant<A> { }

// Implementing contravariant interface.
class Sample<A> : IContravariant<A> { }

class Program
{
    static void Test()
    {
        IContravariant<Object> iobj = new Sample<Object>();
        IContravariant<String> istr = new Sample<String>();

        // You can assign iobj to istr because
        // the IContravariant interface is contravariant.
        istr = iobj;
    }
}

Invariance – Means that you can use only the type originally specified

Generic classes are not covariant to ensure static type safety

class Animal {}

class Bear : Animal {}

class Camel : Animal {}

public class Stack<T>  // A simple Stack implementation
{
      int position;
      T[] data = new T[100];

      public void Push (T obj) { data [ position ++ ] = obj; }
      public T Pop() { return data [ -- position ]; }
}

The following fails to compile:

Stack<Bear> bears = new Stack<Bear>();
Stack<Animal> animals = bears; // Compile-time error

That restriction prevents the possibility of runtime failure with the following code:

animals.Push ( new Camel());

Dependency Injection

What is Dependency Injection?

This is a pattern used for decoupling components and layers in the system. The pattern is implemented through injecting dependencies into a component when it is constructed. These dependences are usually provided as interfaces for further decoupling and to support testability.

Pure Dependency Injection (Poor man’s DI)

Pure Dependency Injection uses constructor or property Injection where lower level components are passed using constructors or properties.

DI container

Simplifies the creation of dependencies and also provides benefits like Interception (for Aspect oriented programming) for cross-cutting concerns like logging, caching security etc.

DI style Advantages Disadvantages
Poor Man’s DI Easy to learn
Strongly typed
Rapid Feedback
High maintenance
DI Container Low maintenance

Convention/xml registration

Weakly typed
Harder to learn
Slower feedback

Without DI container

var svc = new ShippingService(new ProductLocator(), new PricingService(), new InventoryService(), new TrackingRepository(new ConfigProvider()), new
 Logger(new EmailLogger(new ConfigProvider())));

With DI Container (Autowiring)

var svc = IoC.Resolve();

Comparing DI Containers

Autofac DryIoc SimpleInjector Unity
Configuration XML/Auto Auto XML/Auto
Custom Lifetimes Yes Yes Yes
Interception Yes Yes Yes
Auto Diagnostics Yes
Speed Average Fast Fast Average

Container compatibility for generics vary especially for the following scenarios and a more detailed analysis is needed with regards to capabilities required by our system

  • Open Generics
  • Covariance & Contravariance
  • Optional parameters
  • Func & Lazy support