AWS Whitepaper Series: Practicing CI and CD on AWS

Title: Practicing Continuous Integration and Continuous Delivery on AWS Accelerating Software Delivery with DevOps - June 2017

Emphasis on the Summary of Best Practices

• Treat your infrastructure as Code
• Use version control for your infrastructure code.
• Make use of bug tracking/ticketing systems.
• Have peers review changes before applying them.
• Establish infrastructure code patterns/designs.
• Test infrastructure changes like code changes.
• Put developers into integrated teams of no more than 12 self-sustaining members.
• Have all developers commit code to the main trunk frequently, with no long-running feature branches.
• Consistently adopt a build system such as Maven or Gradle across your organization and standardize builds.
• Have developers build unit tests toward 100% coverage of the code base.
• Ensure that unit tests are 70% of the overall testing in duration, number,and scope.
• Ensure that unit tests are up-to-date and not neglected. Unit test failures should be fixed, not bypassed.
• Treat your continuous delivery configuration as code.
• Establish role-based security controls(that is,who can do what and when)
• Monitor/track every resource possible.
• Alert on services, availability, and response times.
• Capture, learn, and improve.
• Share access with everyone on the team.
• Plan metrics and monitoring into the lifecycle.
• Keep and track standard metrics
• Number of builds.
• Number of deployments.
• Average time for changes to reach production.
• Average time from first pipeline stage to each stage.
• Number of changes reaching production.
• Average build time.
• Use multiple distinct pipelines for each branch and team.

Don’ts

• Have long-running branches with large complicated merges.
• Have manual tests.
• Have manual approval processes, gates, code reviews, and security reviews.

Read More

Best Practices: CI Patterns and Anti-Patterns

Here is the summary of all CI patterns and anti-patterns as mentioned in this article. Refer to the link for a more detailed explanation of these.

Build Software at every change

Pattern	Run a software build with every change applied to the Repository.
Anti-patterns	Scheduled builds, nightly builds, building periodically, building exclusively on developer's machines, not building at all.

Task level commit

Pattern	Organize source code changes by task-oriented units of work and submit changes as a Task-Level Commit.
Anti-Patterns	Keeping changes local to development for several days and stacking up changes until committing all changes. This often causes build failures or requires complex troubleshooting.

Label build

Pattern	Tag or Label the build with unique name so that you can refer to run the same build at another time.
Anti-Patterns	Not labeling builds, using revisions or branches as "labels."

Automated build

Pattern	Automate all activities to build software from a source without manual configuration. Create build scripts that will be executed by a CI system so that software is built at every change.
Anti-Patterns	Continually repeating the same processes with manual builds or partially automated builds requiring numerous manual configuration activities.

Pre-Merge build

Pattern	Verify that your changes will not break the integration build by performing a pre-merge build—either locally or using Continuous Integration.
Anti-Patterns	Checking in changes to a version-control repository without running a build on a developer's workstation.

Continuous feedback

Pattern	Send automated feedback from the CI server to development team members involved in the build.
Anti-Patterns	Sending minimal feedback that provides no insight into the build failure or is non-actionable. Sending too much feedback, including to team members uninvolved with the build. This is eventually treated like spam, which causes people to ignore messages.

Expeditious fixes

Pattern	Fix build errors as soon as they occur.
Anti-Patterns	Allowing problems to stack up (build entropy) or waiting for them to be fixed in future builds.

Developer documentation

Pattern	Generate developer documentation with builds based on checked-in source code.
Anti-Patterns	Manually generating developer documentation. This is both a burdensome process and one in which the information becomes useless quickly because it does not reflect the checked-in source code.

Independent build

Pattern	Create build scripts that are decoupled from IDEs, but can be invoked by an IDE. These build scripts will be executed by a CI system as well so that software is built at every change.
Anti-Patterns	Relying on IDE settings for Automated Build. Build cannot run from the command line.

Single command

Pattern	Ensure all build processes can be run through a single command.
Anti-Patterns	Requiring people or servers to enter multiple commands and procedures in the deployment process, such as copying files, modifying configuration files, restarting a server, setting passwords, and other repetitive, error-prone actions.

Dedicated resources

Pattern	Run master builds on a separate dedicated machine or cloud service.
Anti-Patterns	Relying on existing environmental and configuration assumptions (can lead to the "but it works on my machine problem").

Externalize and tokenize configuration

Pattern	Externalize all variable values from the application configuration into build-time properties. Use tokens so the build process knows where to add variable values.
Anti-Patterns	Hardcoding values in configuration files or using GUI tools to do the same.

Scripting database changes

Pattern	All changes made to a database during development should be recorded into database scripts that can be run on every database on every platform hosting the project (including developer machines, see below).
Anti-Patterns	Expecting database administrators to manually compare databases between platforms for changes, or to create on-off scripts that are only good for updating a single platform.

Database sandbox

Pattern	Create a lightweight version of your database (only enough records to test functionality) Use a command line interface to populate the local database sandboxes for each developer, tester, and build server Use this data in development environments to expedite test execution
Anti-Patterns	Sharing development database.

Update Scripts Stored In The VCS   

Pattern	Store the scripts for updating the database and its data in the version control system with the code and annotate appropriately
Anti-Patterns	Storing update scripts in an alternative location, i.e. a shared file server

Automated tests

Pattern	Write automated tests for each code path, both success testing and failure testing.
Anti-Patterns	Not running tests, no regression testing, manual testing

Build quality threshhold

Pattern	Notify team members of code aberrations such as low code coverage or the use of coding anti-patterns. Fail a build when a project rule is violated. Use continuous feedback mechanisms to notify team members.
Anti-Patterns	Deep dive reviews of every code change. Manually calculating or guesstimating code coverage.

Automated smoke test

Pattern	Create smoke tests that can be used by CI servers, developers, QA, and testing as a pre-check to confirm the most important functionality as they work, or before committing resources to a full build.
Anti-Patterns	Manually running functional tests. Forcing QA to run the full suite before every session Manually checking deployment sensitive sections of the project

Read More

Best Practices: Security Concerns with Microservices

Microservices architecture is more dispersed in nature hence, there will be a larger surface area for attacks and vulnerabilities. But because of this dispersion where services being independent at one another, this architecture allows for a more granular restrictions. 

• At the infrastructure, the concern is coming from multiple network entry-points, rate limits and traffic throttling configuration, need for multiple firewalls, defense in depth, network structures and keys and tokens management.
• Ensure regular updates taking note of the following:
• updates must not break the system
• updates should not interrupt the application
• updates must not require full deployments
  

• Keep control 
• never assume infinite scalability 
• rate limits protect from DDoS 
• always test your load
  

• Keep your guard up 
• multiple firewalls 
• minimum exposure (expose only what is needed)
• keep sensitive services private
  

• Keep/protect your keys 
• do not store in the servers 
• encrypt everything 
• use internal and changeable tokens like OAuth
  

• Application concerns
• repeatable deployments 
• encryption algorithms 
• protection methods depending on data types 
• security configuration 
• testing (white, black and penetration)

• Data concerns
• input validation 
• output encoding 
• not obvious ID's 
• end-to-end encryption 
• authentication and authorization even among services

• To mitigate both application and data concerns
• don't make the data structures obvious
• keep IDs out of API endpoints
• it is recommended to use OAuth for authentication and authorization
• usually done in API gateway which enables service discovery
• enables: role based access, access control and signed requests 
• helps with orchestration 
• provides caching layer
  

Reference: Packt's Hands-On Microservices with Python by Peter Fisher

Read More