DevOps in Legacy Systems

I had a discussion with one of my then-manager-colleague about ensuring the movement of a planned item even if the only available team member has no expertise to take it on. The simple answer is to get that team member to a starting point with the help of the experienced ones, through a non-heavy pair work format, just enough to give them momentum. It is possible that the task won't be completed on time, but at least they got it moving.

In practice this is always easier said than done since we always fall to the mindset of either: "It is going to be faster if the expert will do it" or "I'll just take this non priority item which I have expertise on rather than mess up that very critical one". This somewhat silo mentality is especially prevalent in legacy systems, adding to the hurdles on its already challenging DevOps adoption.

In attempts to assume DevOps practices, most of the failures are caused by the teams' resistance to change because of the leaders oversimplifying the problem. That is why at the initial stages of DevOps assessment, dialogue between leaders and legacy teams is required so that visions and goals are personally communicated, uncertainties are addressed, and negotiations and agreements are formulated, which in turn will help earn the people's confidence and acceptance towards it. DevOps is a culture after all. It is also worth noting that a DevOps team/person is not necessary where creating one turned out to be problematic for some because the bottleneck just got transferred to another entity.

An attitude of shared responsibility is an aspect of DevOps culture that encourages closer collaboration. It’s easy for a development team to become disinterested in the operation and maintenance of a system if it is handed over to another team to look after. If a development team shares the responsibility of looking after a system over the course of its lifetime, they are able to share the operations staff’s pain and so identify ways to simplify deployment and maintenance (e.g. by automating deployments and improving logging).  - Martin Fowler

Here are some of what needs to be resolved and considered when deliberating on adopting DevOps:

Architecture and technology audit. In DevOps POV, it is important to include: list of supported systems (document declared and client-deployed), SCM branch strategy and end-to-end delivery process flow definition. In a decade old legacy systems, small automated processes and tools are most likely to be existing already and should also be accounted.

Assessment of your system against automation tool-sets, taking note of benchmark and baseline data to help define realistic targets for the metrics. And speaking of metrics, selecting the right kind that can be translated in business value terms and is understood by all is key to continuous improvement. Examples: deployment speed, failure rates, delivery frequency, bug recurrence rate, etc.

Interface agreements. When teams are at their exploratory work, be it in automation or trying out new tools, establishing standards on how they will send their data across the entire delivery process will promote parallel development and help in traceability. Most importantly, these data interfaces will protect their autonomy where implementation is being left to their discretion, as long as the ending output adheres to the standard format that is compatible to the next phases of the pipeline. Given that, a part of these interface agreements must include counter-checking mechanisms.
But, whether a DevOps initiative will push through or not, it is always beneficial for an organization maintaining a legacy system to already start investing on the following:

1. Standards and best practices.
2. Unit test coverage including codes involved in operations, to get the earliest feedback at the most fundamental level.
3. Automation and development of specialized tooling that will cater to the system's automation needs.
4. Technical debt repayment.
5. Exploring the possibility of cloud and containerized solutions.

Porting legacy systems to DevOps is especially hard because of the associated practices and attitude that came with it. Just like in any acclimatization process, small adaptations with demonstrative progress is more effective rather than having a strict top-down approach. People are inherently motivated if the progress is a result of their own volition to the improvement of self or otherwise. 100% in full DevOps may not be achieved, but at least there will be a motivation to get it moving.

----------------------------------------------------------------------------------------------------------
This article was originally published in LinkedIn

Read More

Quantum Computing: Drivers of the Hype

Last October, I attended PSIA's SoftCon.ph 2019, where one of the plenary hall speakers is a Managing Director from Accenture who discussed a new set of emerging technologies following the SMAC (Social, Mobile, Analytics, Cloud) technologies from 6 years ago. Abbreviated as the DARQ Power which stands for Distributed Ledgers, AI, Reality Extended, and Quantum Computing.

The first time I heard of a Quantum Computer from BBC's 2013 documentary, "Defeating the Hackers", was with a raised eyebrow due to the fact that Quantum Mechanics, from how I recall it in college around 10 years ago, defies the reality as we know it. And upon pursuing this topic now, this branch of physics is still regarded as 'weird' even by physicists themselves.

Despite the uncertainties and being at an early phase of development, people are heeding and already considered Quantum Computing/Computer (which I will refer to as QC from here on) as if it is the only way to move forward.

'Quantum Supremacy'

Just last October of this year, Google announced claiming that they have achieved 'quantum supremacy' with their QC system named Sycamore. The task involved, somewhat like a 'hello world' program for QCs, is just a random set of instructions which they estimated will take 10,000 years for the world's fastest supercomputer to complete. Sycamore made it in 3.33 seconds.

At this point, there are no practical applications yet but they compared the feat to the first successful flight of the Wright brothers. It is the beginning of something that surely could become very significant.

The Early Buy-ins

Even before Google's announcement, Accenture had already made their move. Last July, they acquired a patent for a Machine Learning module that would help business decision-makers determine if QC would be beneficial to them.

Accenture, starting in 2015, already did their research work on the feasibility of QC as a business solution. In 2017, partnering with 1QBit Information Technologies Inc., they were able to map out more than 150 use cases for the technology. Most notably out of these is for speeding up drug discovery.

With QC's power becoming more evident, more companies and investors are taking interest, including the Trump administration, whose support is mainly coming from cybersecurity reasons. The US government made Quantum research development a priority.

 The Ultimate Driver

Obviously, most of QC's potential applications are coming from the areas where supercomputers are having difficulties still. Molecular simulations, (which is important for chemistry research) and forecasting (for meteorology, financials, logistics etc.) are exponentially complex where classical computers are generally unable to overcome, except through approximations. Although, there could still be ways to improve our current supercomputers, we are nearly approaching the physical limits. Expanding the infrastructure will eventually become impractical unless we make every component smaller to maximize the use of space.

But here is the problem...
The heart of a supercomputer is its CPUs (IBM's Summit, the world's fastest supercomputer has 200,000 of them) and the "building-blocks" of a CPU are its transistors. Today, transistors are around 10-20 nanometers (the most recently launched AMD Zen 2 is at 7nm), and we are still going for 5, even 3nm, on the next couple of years, but as transistors get smaller, it will reach a point where it will experience quantum tunnelling. We are nearing the end of Moore's law.

If one is to imagine a transistor being a switch where it has an 'on' and 'off' state, hence how a computer communicates through the binary language of 1s and 0s respectively. At an 'off' state, there is a barrier that stops the flow of electrons, but in quantum tunnelling, the electrons are now able to get to the other side regardless of that barrier, likening to a ghost that can go through a wall. This means, the transistor will cease to function as intended.

In closing...

Still, we do not expect QC to replace classical computers any moment now or even in the next couple of decades. There is still a long way down the road, and it is reserved to specific purposes especially given its expensive setup. It is fundamentally different against classical systems, like how pencil is to a pen.

----------------------------------------------------------------------------------------------------------
This article was originally published in LinkedIn

Read More

My notes on setting up PyCharm with Gerrit

PyCharm Notes1. Install Pycharm
2. Install Git
3. Install Python interpreter
4. Install required Python modules

In Pycharm
1.    Clone the projects
2.    For multiple projects in one window – do an ‘attach’
3.    Configure Python interpreter per project
5.    install packages/plugins  via Settings - e.g. Gerrit plugin
6.    Set the project dependencies in Settings - e.g. Project1 and Project2 projects are dependent to COMMON
7.    Set COMMON project as a source directory

Gerrit - new change
1. Pull to latest
2. In committing: Supply 'Author'
3. In Pushing: make sure 'Push to Gerrit' is checked
4. Check that a new Gerrit record has been created by refreshing the Gerrit pane below.

Gerrit - amending
5. Pull to latest. The Gerrit pane should also refresh.
6. Checkout the change (right-click - checkout)
7. In committing: Make sure of the following
  - Commit Message, "Change-Id: I34e32........" line exists
  - Author is supplied
  - "Amend commit" is checked
8. In pushing: by default all Projects are checked, uncheck project that has no changes
  - Make sure "Push to Gerrit" is checked

Read More

Best Practices: CI Patterns and Anti-Patterns

Here is the summary of all CI patterns and anti-patterns as mentioned in this article. Refer to the link for a more detailed explanation of these.

Build Software at every change

Pattern	Run a software build with every change applied to the Repository.
Anti-patterns	Scheduled builds, nightly builds, building periodically, building exclusively on developer's machines, not building at all.

Task level commit

Pattern	Organize source code changes by task-oriented units of work and submit changes as a Task-Level Commit.
Anti-Patterns	Keeping changes local to development for several days and stacking up changes until committing all changes. This often causes build failures or requires complex troubleshooting.

Label build

Pattern	Tag or Label the build with unique name so that you can refer to run the same build at another time.
Anti-Patterns	Not labeling builds, using revisions or branches as "labels."

Automated build

Pattern	Automate all activities to build software from a source without manual configuration. Create build scripts that will be executed by a CI system so that software is built at every change.
Anti-Patterns	Continually repeating the same processes with manual builds or partially automated builds requiring numerous manual configuration activities.

Pre-Merge build

Pattern	Verify that your changes will not break the integration build by performing a pre-merge build—either locally or using Continuous Integration.
Anti-Patterns	Checking in changes to a version-control repository without running a build on a developer's workstation.

Continuous feedback

Pattern	Send automated feedback from the CI server to development team members involved in the build.
Anti-Patterns	Sending minimal feedback that provides no insight into the build failure or is non-actionable. Sending too much feedback, including to team members uninvolved with the build. This is eventually treated like spam, which causes people to ignore messages.

Expeditious fixes

Pattern	Fix build errors as soon as they occur.
Anti-Patterns	Allowing problems to stack up (build entropy) or waiting for them to be fixed in future builds.

Developer documentation

Pattern	Generate developer documentation with builds based on checked-in source code.
Anti-Patterns	Manually generating developer documentation. This is both a burdensome process and one in which the information becomes useless quickly because it does not reflect the checked-in source code.

Independent build

Pattern	Create build scripts that are decoupled from IDEs, but can be invoked by an IDE. These build scripts will be executed by a CI system as well so that software is built at every change.
Anti-Patterns	Relying on IDE settings for Automated Build. Build cannot run from the command line.

Single command

Pattern	Ensure all build processes can be run through a single command.
Anti-Patterns	Requiring people or servers to enter multiple commands and procedures in the deployment process, such as copying files, modifying configuration files, restarting a server, setting passwords, and other repetitive, error-prone actions.

Dedicated resources

Pattern	Run master builds on a separate dedicated machine or cloud service.
Anti-Patterns	Relying on existing environmental and configuration assumptions (can lead to the "but it works on my machine problem").

Externalize and tokenize configuration

Pattern	Externalize all variable values from the application configuration into build-time properties. Use tokens so the build process knows where to add variable values.
Anti-Patterns	Hardcoding values in configuration files or using GUI tools to do the same.

Scripting database changes

Pattern	All changes made to a database during development should be recorded into database scripts that can be run on every database on every platform hosting the project (including developer machines, see below).
Anti-Patterns	Expecting database administrators to manually compare databases between platforms for changes, or to create on-off scripts that are only good for updating a single platform.

Database sandbox

Pattern	Create a lightweight version of your database (only enough records to test functionality) Use a command line interface to populate the local database sandboxes for each developer, tester, and build server Use this data in development environments to expedite test execution
Anti-Patterns	Sharing development database.

Update Scripts Stored In The VCS   

Pattern	Store the scripts for updating the database and its data in the version control system with the code and annotate appropriately
Anti-Patterns	Storing update scripts in an alternative location, i.e. a shared file server

Automated tests

Pattern	Write automated tests for each code path, both success testing and failure testing.
Anti-Patterns	Not running tests, no regression testing, manual testing

Build quality threshhold

Pattern	Notify team members of code aberrations such as low code coverage or the use of coding anti-patterns. Fail a build when a project rule is violated. Use continuous feedback mechanisms to notify team members.
Anti-Patterns	Deep dive reviews of every code change. Manually calculating or guesstimating code coverage.

Automated smoke test

Pattern	Create smoke tests that can be used by CI servers, developers, QA, and testing as a pre-check to confirm the most important functionality as they work, or before committing resources to a full build.
Anti-Patterns	Manually running functional tests. Forcing QA to run the full suite before every session Manually checking deployment sensitive sections of the project

Read More

Best Practices: Security Concerns with Microservices

Microservices architecture is more dispersed in nature hence, there will be a larger surface area for attacks and vulnerabilities. But because of this dispersion where services being independent at one another, this architecture allows for a more granular restrictions. 

• At the infrastructure, the concern is coming from multiple network entry-points, rate limits and traffic throttling configuration, need for multiple firewalls, defense in depth, network structures and keys and tokens management.
• Ensure regular updates taking note of the following:
• updates must not break the system
• updates should not interrupt the application
• updates must not require full deployments
  

• Keep control 
• never assume infinite scalability 
• rate limits protect from DDoS 
• always test your load
  

• Keep your guard up 
• multiple firewalls 
• minimum exposure (expose only what is needed)
• keep sensitive services private
  

• Keep/protect your keys 
• do not store in the servers 
• encrypt everything 
• use internal and changeable tokens like OAuth
  

• Application concerns
• repeatable deployments 
• encryption algorithms 
• protection methods depending on data types 
• security configuration 
• testing (white, black and penetration)

• Data concerns
• input validation 
• output encoding 
• not obvious ID's 
• end-to-end encryption 
• authentication and authorization even among services

• To mitigate both application and data concerns
• don't make the data structures obvious
• keep IDs out of API endpoints
• it is recommended to use OAuth for authentication and authorization
• usually done in API gateway which enables service discovery
• enables: role based access, access control and signed requests 
• helps with orchestration 
• provides caching layer
  

Reference: Packt's Hands-On Microservices with Python by Peter Fisher

Read More

CKAD Learning Series 01: Installing minikube

1. Enable CPU virtualization support which can be done through the BIOS of your laptop/PC. To check, run the command:
grep vmx /proc/cpuinfo


2. Add Kubernetes to your repository
create this file: /etc/yum.repos.d/kubernetes.repo
to contain the snippet below:
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

3. It is adviced to install kubectl first so that minikube will detect it and automatically attach itself with its configuration.
sudo dnf install -y kubectl

4. Install VirtualBox or KVM (KVM in this example)
sudo dnf clean all
sudo dnf -y upgrade
sudo dnf install @virtualization -y 

Then start the libvirtd service and add your user to the libvirt group
sudo systemctl enable --now libvirtd
sudo usermod -aG libvirt $USER
 
5. Installing minikube
Download minikube:
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
Set it as an executable and move it to bin right away
chmod +x minikube
mv minikube /usr/local/bin
 
6. Reboot the machine

7. After reboot, but before running minikube, you may have to add the user that you'll be using with to run minikube to the libvirt group
usermod -aG libvirt $USER

Run minikube
minikube start --memory 4096 --vm-driver=kvm2
followed by: 'kubectl cluster-info' and 'kubectl get nodes'

minikube ssh

Read More

Distributing Test Cases to a set of Jenkins slave for execution

This groovy script will distribute a set of testcases to a set of slaves under a certain label.

https://github.com/elogs/jenkins/blob/master/groovy/testcaseSlaveDistributor.groovy

Prerequisites:

1. The slaves are part of a label
2. The Jenkins job which executes the test cases should have the following capabilities:
1. It has "test case list" as a parameter. Therefore this job should be able to execute 1 or more test cases upon trigger by looping its execution through that list.
2. It should be able to identify already running (or already passed) test cases as a first step of the execution loop and skip these cases already.

What this script does:

1. Create a pool of online slaves. While creating this, the free slaves will have priority over the busy slaves once test-case distribution happens. 
2. If the number of slaves in the pool is greater than the number of test cases, trim the pool to create a matching number of slaves but taking only from the priority side.   
3. Pair the test-case list (list as a string, because this will be passed later on as a job parameter for the test execution job) to each of the slaves in the pool. But for every next slave, the test case list will rotate its list. Ex:
• slaveA:TC001,TC002,TC003
• slaveB:TC002,TC003,TC001 
• slaveC:TC003,TC001,TC002
4. Trigger parallel builds of the "test execution job", passing the pairs as Node and testcase list parameters of that target job. Because this step will trigger parallel execution of test cases, the rotating test case list will prevent a scenario where multiple slaves are executing the same test case at the initial run.

Read More

#Throwback: A batch script to clone or update Project from a Git repository

Background:The use-case behind this was that we could not checkout our project from Git repository in Jenkins for reasons I could not remember. I'm just publishing this here as an archive for future reference in case I encounter something of the same nature.

https://github.com/elogs/jenkins/blob/master/batch/gitCloneOrPull.bat

1. There should be curl in the system to verify the URLs
2. Set the variables, set proxy depending on IP of the Jenkins slave. We had slaves within a certain subnet which needs proxy for them to be able to connect to our Git repositories.
3. Perform connection checks 
4. Proceed with clone if project is not yet existing in the machine, update if already existing 
5. Create backup of successful clone/update

Read More