Hello Every One in this blog I would like to explain about basics of what are Jenkins agents how to connect EC2 instance as a permanent agent to Jenkins controller running locally.
Agents
Jenkins agents (also referred to as nodes) are separate machines (physical or virtual) that run jobs assigned by the Jenkins master (controller). Master can share workload on these machines. By default Jenkins comes with single node where jobs are run on the same master node .
An agent can run on a different machine (remote or local), on a container, or in the cloud.
Use of Jenkins Agents
Agents help distribute the load of running jobs, which is especially useful when the master is under heavy load.
With agents, multiple jobs can run in parallel on different machines, which reduces build times.
Sometimes jobs need to run in specific environments (e.g., testing in both Linux and Windows). Agents allow you to run jobs in these various environments.
Type of Agents
Permanent Agents
Dynamic Agents
Permanent Agents
These are dedicated agents manually connected to Jenkins controller for specific jobs. These type of agents stay connected until they explicitly stopped.
Typically used for long-running, stable agents in environments that require specific configuration
Dynamic Agents
These agents are created on-demand when jobs are triggered and are terminated when the job is finished.
Example: AWS EC2 instances , Containers, Kubernetes pods.
How Jenkins Connects to Agents
SSH:
Jenkins uses Secure Shell (SSH) to connect to a remote agent. Jenkins pushes jobs over SSH to the agent machine.
JNLP (Java Network Launch Protocol):
Agents initiate the JNLP connection to the master. This is particularly useful if the agent is behind a firewall or NAT.
Setting EC2 instance as a permanent agent
After getting a basics of agents and how does it work. Lets set up a virtual machine as an agent to the Jenkins controller running locally.
On the local machine :
By clicking Build Executor Status we will be able to create nodes, select new node , select Permanent Agent as this is our agenda today.
executors refer to the number of concurrent jobs (or builds) that a Jenkins agent can run at the same time.
Suppose if we mention 4 executors. Jenkins can assign 4 jobs to run in parallel on that Linux machine. If you try to assign more than 4 jobs, the additional jobs will wait in the queue until one of the executors is free.
The Remote Root Directory in Jenkins refers to the directory on an agent machine where all the files related to the Jenkins jobs are stored.
Labels are used by the controller to detect on which agent jobs has to run.
Launch Method here we are going to look at
Launch Agent via SSH
Launch agents by connecting to controller.
Launch Agent via SSH
In this blog I will try to explain
Non Verifying Verification Strategy
Known Host files verification Strategy.
Under Host Key Verification Strategy.
SSH Host Key Verification: Normally, when we connect to a server via SSH, our SSH client checks if the server's SSH host key matches a known value. This helps us ensure that we are connecting to the right server and not an imposter.
Non-Verifying Strategy: When using the Non-Verifying Verification Strategy, Jenkins skips this verification step. It means Jenkins will connect to the agent without checking if the agent's SSH host key is valid or known.
Non Verifying Verification Strategy
Put the IP of instance in the host section.
In the Credentials Section we can select Credential Kind as Username and Password or Username Private Key
sudo passwd ec2-user we can set password for ec2-user. and we can use in credentials.
Now we can use these credentials.
We can even select Username and Private Key Option. Lets find how does this work.
Using ssh-keygen command on master node we get public and private keys.
Copy the Public Key: Take the file
~/.ssh/id_rsa.pubfrom the master machine and put its contents into the file~/.ssh/authorized_keyson the agent machine.Use the Private Key: On the master machine, use the private key file
~/.ssh/id_rsato set up the connection in the credentials section of Jenkins.
By following any of the method master node could be able to connect to Agent node.
Known Host files verification Strategy.
Just simply follow above Username & private key method and add agents IP in known host file like this on master node.
ssh-keyscan -H agent-ip >> ~/.ssh/known_hosts
By this way agents would be able to connect vias SSH.
Launch agents by connecting to controller.
If we simply select this option Jenkins provides us commands to run on the agent node. Simply run those commands on agent node. Agent node is now authenticated and will be connected to master node.
But there is a catch here as our Jenkins master is running locally agent could not be able to find the master node. In such cases we have to use Reverse Proxy like Nginx, Ngrok, PageKite, serveo.net etc.. through this we can transfer traffic from these reverse proxies to our Local Jenkins Master node.
I have used Ngrok which is so simple to setup just navigate to ngrok official website they have provided clear guide to walk us through.
This Simple command generates a secure URL and http traffic which hits this URL will be transferred to localhost:8080.
ngrok http http://localhost:8080
Simple right.
Simple Pipeline Job using Jenkins agents.
pipeline {
agent none // Pipeline will not run on the master node
stages {
stage('Build and Test on All Platforms') {
parallel {
stage('Linux Build and Test') {
agent { label 'linux' } // Assign the job to the Linux agent
steps {
// Commands to build and test on Linux
sh 'make build-linux'
sh 'make test-linux'
}
}
stage('Windows Build and Test') {
agent { label 'windows' } // Assign the job to the Windows agent
steps {
// Commands to build and test on Windows
bat 'make build-windows'
bat 'make test-windows'
}
}
}
}
By using agents and the labels attached to them, we can specify on which node a pipeline job should run. Similarly, as in the example above, we can define which stage should run on a particular node..
Thanks for reading my blog. Have a great day🎉🎉🎉.