• Download Java 8 from Oracle and install it on your machine.
• Configure environmental variables using the following steps:
  • Copy the JDK path (e.g., <C:\Program Files\Java\jdk1.8.0_241>).
  • Search for environment variable setup in the search bar.
  • Proceed to set up the necessary variables.

• Navigate to the “User variables” section and click on “New” to proceed

• In the “Variable name” field, enter “JAVA_HOME”, and in the “Variable value” field, provide the JDK path (<C:\Program Files\Java\jdk1.8.0_241>). Then, close the window.

• Next, in the “System variables” section, select “Path” and click “Edit”.

• Here, click on “New”.

• Next, provide the path to the JDK bin folder (<C:\Program Files\Java\jdk1.8.0_241\bin>), then click “OK” and close the window.
• To verify if the environment variables are correctly set up, use the following commands in the command prompt:
  • java -version
  • javac -version

Setting up the Tomcat server in the Local Machine:

• Download Apache Tomcat 9 from the official website

• Then zip folder will downloads, so you can unzip it in a separate folder

• After downloading, copy the path of the Tomcat’s bin folder. Then, open the command prompt and navigate to the Tomcat bin directory.

• Initiate the Tomcat server by executing the command “startup.bat”.

• Access the application in Chrome by entering “localhost:8080” in the address bar

• Subsequently, click on the “Manager App” option.
• It will prompt you to enter the User ID and password. To set up the user and password, navigate to the “conf” folder in the Apache Tomcat unzip directory.
• Find the “tomcat-users.xml” file, right-click, and open it with Notepad. Enter the desired username and password.

• Before making any changes to the username and password:

• After updating the username and password:

• Afterward, select the “Manager App” option. This will display the following screen

Then, proceed to the “Deploy War File” section and select “Deploy Your War File”. Choose your war file from your system and click on “Deploy”.

• Once the war file is deployed successfully, you will see JPetstore accessible at the specified version path.

• Click on the JPetstore link displayed in the path. This will open the Petstore application in a new Chrome window.

• Click on “Enter the Store” and proceed to log in to the application using the following credentials: Username: j2ee Password: j2ee

The post Setting Up JPetstore Application on Your Local System appeared first on Isha.

  No one will disagree that customer experience is now of paramount importance in an enterprise. All business processes, workflows, and technologies in modern businesses are adopted and implemented, keeping the customer in mind. The tacit belief is that the better the customer experience, the better the business. And this level of customer experience is directly related to the ease of user interaction experienced by the customer while interacting with enterprise applications, software, and IT infrastructure that is interfacing with the customer. The speed, responsiveness, uninterrupted service availability, and the ease of use are critical factors that customer experiences are rated on. To that end, digital transformation is being adopted across the entire enterprise to enhance the customer experience being facilitated. This means that all applications being used across the enterprise must maintain a level of performance and service prescribed as a “minimum” by the enterprise. It’s here that Observability and Application Performance Monitoring are introduced into the enterprise to ensure that all digital infrastructure is tuned towards the best performance at all times, thus ensuring the required customer experience. Application Performance Monitoring (APM) – A Primer Application performance monitoring or APM is the process where tools and technologies are used to ensure that the applications deployed in an enterprise function as per expectations in terms of performance,reliability, and user experience. These expectations are predefined norms laid down by the enterprise and applied to employees, vendors, and customers that are considered users of the software in the enterprise. This monitoring focuses on application performance, unit and component performance, user interactivity, transaction speeds, etc. APM can be done in-house or by forming strategic partnerships with to implement the same. Observability – A Primer Observability is the process of collecting data, including that from APM systems, endpoint performance tracking tools, and other such digital assets in an enterprise, to deduce the state of an entire system. Observability aims to get a holistic view of the whole system and its functioning across the entire enterprise. This view enables teams to monitor solutions in their entirety and make decisions that will enhance customer experience. Observability also includes tracking system event management, cybersecurity, and the software development and deployment cycles within the enterprise. Log files, metrics, and trace data are collected for this process of observability from every possible IT asset across the enterprise. With the process of observability, data on performance, management, fault detection, and mitigation is collected across distributed computing environments regardless of complexity. This overall observability process enhances root cause analysis undertaken in an organization, thus, making failures and anomalies easier to detect and mitigate. The importance of observability is not restricted to IT asset management and performance. The process of observability has a direct impact on business decisions and workflow management as well. Connecting Monitoring and Observability APM and Observability are similar in the sense that for both, data is collected to get a better picture of the performance of applications and a system. Data collected for both processes can be similar and be used for either process. The difference, however, lies in the scope of the process of observability and application performance management. Monitoring Is a Process Within Observability Different data is monitored and collected by monitoring systems across the enterprise. This data can then be analyzed for performance and error detection at the individual entity or component level or used for observability of the bigger picture at the enterprise level. Metrics, logs, and trace data collected by APMs are one of the sources of analysis of observability. Monitoring vs. Observability Monitoring is a process within the framework of observability. Though both processes look at how to increase performance and productivity, the scope of each is different. Observability looks at a system in its entirety in a top-down approach, implementing root cause analysis for detection, mitigation, and enhancement of an enterprise as a whole. Observability is about the state and health of a system as a whole. Monitoring is more at the entity level and individual components within the IT enterprise. Monitoring will give performance data of an application, and performance management will ensure its enhancement, but it will take observability to view the state of the entire complex system where the application is just one of the entities. Why is Observability needed even when Monitoring is enabled 1:- Complexity overrides monitoring Adopting digital transformation usually entails the increasing complexity of the IT infrastructure. The fusion of legacy and newly developed software, adoption of a multi-cloud environment along with the native cloud and edge computing, and their interdependencies are just some of the complexities of an ever-growing digital transformation process. Organizations tend to use a variety of monitoring tools to attempt to gain a better view of the enterprise as a whole. This strategy however often ends up in failure as the monitoring systems are of individual entities of the system and do not observe the interdependencies within the entities nor treat the system as a whole. This is where observability tools, techniques, and strategies are adopted to look at components, services, applications, and systems at an enterprise level. 2:- Plugging into AI and ML While monitoring and its dashboard are at the individual IT entity level, Observability with its top-down approach has a better view of the enterprise systems in its entirety. Since observability adopts a holistic view of a system rather than a collection of individual entities or microservices, the models that are built of performance data across a life cycle are tuned in for more accurate insights and analytics, root cause analysis, event monitoring, and pattern recognition with AI adoption with ML. This allows for early detection of unpredictable losses and failures with proactive AI-based Observability strategies and processes. 3:- Root Cause Analysis and Context Propagation It is not possible to work out a real and actual root analysis based only on monitoring data. Monitoring will give data on individual entities or microservices. Observability is required for accurate Root Cause Analysis as it has models based on data across all the microservices or systems in a distributed system. Since in Observability data assimilated is across the microservice, context propagation is possible along a path that might traverse across multiple microservices in the form of a transaction or a user request. This context propagation can be correlated to the transaction behavior and can help in further root causes analysis of application performance. 4:- The Bottom Line Observability looks at a system in its entirety in a top-down approach, whereas Monitoring is a process within the framework of observability. Furter, observability brings-in higher capability to the enterprise to deliver better performance and reliability.   Source: https://www.avekshaa.com/blogs/avekshaa_insights/Observability-and-Monitoring/

The post Observability and Monitoring – Same or Different? appeared first on Isha.

• • {tomcat-directory}/conf tomcat-users.xml file configuration

• • Login to the application by clicking Manager App button, by giving tomcat and tomcat as the user credentials, which we already configured @ /conf

• • Navigate to WAR file to deploy section and browse the .war file and click deploy

• • Once the deployment is successful, the application will be listed here

• • Launch the application by clicking on /jpetstore URI, ensure the application is running as below

• • Login to the application using the credentials – username (j2ee) and password(j2ee)

• • Lets understand on the two applications those comes bundled with JDK to monitor the application’s metrics (which internally uses JVM)
  • Java Visual VM (jvisualvm)
  • JConsole

• JVisual VM

  • Navigate to JDK/bin and open jvisualvm to view that the JPetStore application is already being listed to be monitored, as the application utilizes the backend java runtime(JVM)

• Analyze the behavior of Heap, GC, CPU, Memory metrics via Monitor tab
• We can perform the below kinds of analysis using the Java VisualVM
  • Heap Analysis
  • Thread Analysis
  • Garbage Collection Analysis
• Access your already listed application (Tomcat being the one referred to here), Navigate to Monitor tab

• Make sure the CPU Utilization never crosses 80%, as well the Garbage Collection activity

• Discuss with DEV and Product teams to understand the results by comparing the already discussed SLA’s, and take necessary steps to resolve these issues
• Perform GC
  • Click the Perform GC button for manually performing the Garbage Collection(here on referenced as GC)
    • While you manually induce the GC, is there any impact in CPU Usage, GC Activity and Memory Usage, is the intention of performing manual GC by clicking Perform GC button
      • BTW, What is GC? – The activity of clearing the un-referenced objects in memory
• Heap Dump
• • It’s a process of taking the memory snapshot while issue, to analyze the reason of the issue

• • Capture multiple heap dump and analyze and come to an understanding and do the RCA of the issue
• Analyze the behavior of Threads via Threads tab
• We can validate various Thread status like – Running, Sleeping, Waiting, Parked, Monitored Threads

• Analyze the behavior of currently running application’s CPU and Memory data using Sampler and Profiler tabs
  • Sampler and Profiler
    • Observe the CPU metrics of the backend application’s API(methods)

• • • Observe the Memory metrics of the backend application’s API(methods)

• • • Also, Leverage the benefits of analyzing the garbage collection metrics with available plugin that comes bundled with Java VisualVM – Visual GC

• • • Once the plugin gets installed, you can monitor the GC metrics produced by Visual GC by clicking on the application name

• JConsole

  • Select the process that I want to attach/enable JConsole to monitor

• • The below were the various metrics associated with the Tomcat process that we attached above

• • Both JVisualVM and JConsole were intended to monitor JVM, but an added advantage of JConsole is that it’s metrics were analyzed and fetched for a desired time range as below

• • Memory metrics can be analyzed as below

• • Any Spikes, Exponential Increase in the graph, Memory related errors like OutOfMemory error will be a red flag for you while measuring the performance metrics
  • Observe and Analyze the Thread related metrics and issues under the Threads tab

• • How to take the heap dump?

• • • Navigate to MBeans tab, expand com.sun.management\Operations\dumpHeap, and browse a new file {someFileName.hprof }, and the heap dump gets created

-By  Saravanan & Sai Krishna

The post Performance Engineering – JVM – Performance Monitoring (Open Source Tools) appeared first on Isha.

• Inside JVM, there exist separate memory spaces (Heap, Non-Heap, Cache) in order to store runtime data and compiled code.
• Heap Memory
  • Heap is divided into 2 parts
    • Young Generation
    • Old Generation
  • Heap is allocated when JVM starts up and Heap size increases/decreases while the application is running

• • Young Generation
    • A reserved memory for newly allocated objects which consists of
      • One Eden Memory

• • • • • Most of newly instantiated entities gets into Eden Memory. When Eden space is filled with objects, Minor GC is performed and all the survivor objects are moved to one of the survivor spaces.
        • Minor GC also checks the survivor objects and move them to the other survivor space. So at a time, one of the survivor space is always empty.
      • Two Survivor Memory areas (S0, S1)
      • Objects that are survived after many cycles of GC, are moved to the Old generation memory space.
  • Old Generation
    • This memory is reserved for storing long lived objects which could survive after many rounds of Minor GC
    • When Old Gen space is full, Major GC is performed
• Non-Heap Memory
  • Non-Heap Memory includes Permanent Generation (Replaced by Meta space since Java 8)
  • Permanent Generation stores per-class structures such as runtime constant pool, field and method data, and the code for methods and constructors, as well as interned Strings

• Cache Memory
  •  Stores compiled code (i.e. native code) generated by JIT compiler, JVM internal structures, loaded profiler agent code and data, etc.
  • When Code Cache exceeds a threshold, it gets flushed (and objects are not relocated by the GC).

Configuration of Heap Size in JVM

• Access any application that’s designed to run on JVM / a Java based application, and view its configuration(Eg: Tomcat)
  • {Tomcat Home Directory}/bin/catalina.bat

• • • set CATALINA_OPTS = -Xloggc : ”{tomcat\logs\filename.log}” -XX : +PrintHeapAtGC -XX : +PrintGCDetails – XX : + PrintGCTimeStamps -XX : +HeapDumpOnOutOfMemoryError -XX : HeapDumpPath ={tomcat\logs\heapdump.bin} -Xms128m -Xmx256m -XX : + UseSerialGC
• Observe the heap configuration -Xms which is minimum value and -Xmx which is maximum value. It is suggested for heap to have 1/4th of RAM size
• Depending on the application’s performance, adjust the minimum and maximum size values. Each application has its own performance constraints and accordingly adjust the minimum and maximum size values.
• Understand that by increasing the heap size value do not resolve the bottom line problem.
• The below heap related switches are for quick reference

Memory Related Issues

• When there is a critical memory issue, the JVM gets crashed and throws an error indication in your program output like below.
  • java.lang.StackOverFlowError — indicates that Stack Memory is full
  • java.lang.OutOfMemoryError: Java heap space — indicates that Heap Memory is full
  • java.lang.OutOfMemoryError: GC Overhead limit exceeded — indicates that GC has reached its overhead limit
  • java.lang.OutOfMemoryError: Permgen space — indicates that Permanent Generation space is full
  • java.lang.OutOfMemoryError: Metaspace — indicates that Metaspace is full (since Java 8)
  • java.lang.OutOfMemoryError: Unable to create new native thread — indicates that JVM native code can no longer create a new native thread from the underlying operating system because so many threads have been already created and they consume all the available memory for the JVM
  • java.lang.OutOfMemoryError: request size bytes for reason — indicates that swap memory space is fully consumed by application
  • java.lang.OutOfMemoryError: Requested array size exceeds VM limit– indicates that our application uses an array size more than the allowed size for the underlying platform

-By  Saravanan & Sai Krishna

The post Performance Engineering – JVM Heap Structure appeared first on Isha.

• Performance Monitoring – JVM

  • JVM Architecture

    • JVM acts as a run-time engine to run Java applications
    • JVM converts java byte code (the generated information as a .class file once you compile a .java) into machine language code

• Important memory areas, individual modules and activities within a JVM
  • Class Loader Subsystem
    • Loading
      • Modules and Activities
        • Bootstrap Class Loader
          • Loads the rt.jar to the classpath

• • • • • Extension Class Loader
          • Loads the internal .jars from /ext folder to the classpath

• • • • • Application Class Loader
          • Loads the environment variables to the classpath

• • • • Linking

• • • • • Modules and Activities
          • Verify

• • • • • • • A Byte Code Verifier will validate if the generated byte code is proper
          • Prepare

• • • • • • • Memory gets allocated and assigned with default values for static variables in source code
        • Resolve
          • The previous default value will be replaced with original value
      • Initialization
  • Runtime Data Areas
    • Modules and Activities
      • Method Area
        • This is a shared resource (only 1 method area per JVM). All JVM threads share this same Method area, so the access to the Method data and the process of dynamic linking must be thread safe.
        • Method area stores class level data(including static variables) such as:
          • Class loader reference
          • Run time constant pool Field data
          • Method data
          • Method code
        • Heap Area (Shared among Threads)
          • This is also a shared resource (only 1 heap area per JVM). Information of all objects and their corresponding instance variables and arrays are stored in the Heap area. Since the Method and Heap areas share memory for multiple threads, the data stored in Method & Heap areas are not thread safe. Heap area is a great target for GC.
        • Stack Area (per Thread)
          • This is not a shared resource. For every JVM thread, when the thread starts, a separate runtime stack gets created in order to store method calls. For every such method call, one entry will be created and added (pushed) into the top of runtime stack and such entry. It is called a Stack Frame.

• • • PC Registers (per Thread)
      • For each JVM thread, when the thread starts, a separate PC (Program Counter) Register gets created in order to hold the address of currently-executing instruction (memory address in the Method area). If the current method is native then the PC is undefined. Once the execution finishes, the PC register gets updated with the address of next instruction.
    • Native Method Stack (per thread)
      • There is a direct mapping between a Java thread and a native operating system thread. After preparing all the state for a Java thread, a separate native stack also gets created in order to store native method information (often written in C/C++) invoked through JNI (Java Native Interface).
  • Execution Engine
    • The actual execution of the byte code occurs here. Execution Engine executes the instructions in the byte code line-by-line by reading the data assigned to above runtime data areas.
    • Modules and Activities
      • Interpreter
        • The interpreter interprets the byte code and executes the instructions one-by-one. Hence, it can interpret one byte code line quickly, but executing the interpreted result is a slower task. The disadvantage is that when one method is called multiple times, each time a new interpretation and a slower execution are required.
    • Just-In-Time (JIT) Compiler
      • If only the interpreter is available, when one method is called multiple times, each time the interpretation will also occur, which is a redundant operation if handled efficiently. This has become possible with JIT compiler. First, it compiles the entire byte code to native code (machine code). Then for repeated method calls, it directly provides the native code and the execution using native code is much faster than interpreting instructions one by one. The native code is stored in the cache, thus the compiled code can be executed quicker.
      • Execution Engine qualifies to become a key subsystem when introducing performance optimizations by JVM vendors. Among such efforts, the following 4 components can largely improve its performance.
      • Intermediate Code Generator produces intermediate code.
      • Code Optimizer is responsible for optimizing the intermediate code generated above.
      • Target Code Generator is responsible for generating Native Code (i.e. Machine Code). Profiler is a special component, responsible for finding performance bottlenecks a.k.a.
      • hotspots (e.g. instances where one method is called multiple times)
    • Garbage Collector (GC)
      • As long as an object is being referenced, the JVM considers it alive. Once an object is no longer referenced and therefore is not reachable by the application code, the garbage collector removes it and reclaims the unused memory. In general, garbage collection happens under the hood, however we can trigger it by calling System.gc() method (Again the execution is not guaranteed. Hence, call Thread.sleep(1000) and wait for GC to complete).
    • Java Native Interface (JNI)
      • This interface is used to interact with Native Method Libraries required for the execution and provide the capabilities of such Native Libraries (often written in C/C++). This enables JVM to call C/C++ libraries and to be called by C/C++ libraries which may be specific to hardware.
    • Native Method Libraries
      • This is a collection of C/C++ Native Libraries which is required for the Execution Engine and can be accessed through the provided Native Interface.

-By  Saravanan & Sai Krishna

The post Performance Engineering – Related Java Concepts appeared first on Isha.

• Threads allows a program to work parallelly on different tasks at the same time. In a single threaded application, only one thread can execute, meaning only one task can run at any point in time.
• If the application simply creates a new thread for every new HTTP request, and the system receives more requests than it can handle — immediately, the application will suddenly stop responding to all requests when the overhead of all those threads exceeds the capacity of the system. With a limit on the number of the threads that can be created, the application will not be servicing HTTP requests as quickly as they come in, but it will be servicing them as quickly as the system can sustain. That is the reason we have a thread pool so that we can minimize the management of threads and focus on our business logic.
• Thread Pool is a collection of threads which can directly work on tasks assigned by us. Threads are also reused to work on tasks waiting in the queue.
• Types of Thread Pools
  • Fixed Thread Pool
    • Number of Threads will be Fixed. Eg: Using ExecutorService in Java
    • Whenever you are sure on the number of threads that might be used, go for Fixed Thread Pool
  • Cached Thread Pool
    • We will create the thread whenever there is a need, and keep on creating these threads until an extent.
    • Cached Thread Pool is not to be used for long running tasks.
  • Single-Threaded Thread Pool
    • A single thread is being created to perform all the activities
  • Scheduled Thread Pool
    • Tasks to be taken based on the time delay via Scheduled Thread Pool

• Core Pool Size

  • Defines as “How many threads to be created per pool?”
  • When the incoming tasks goes beyond the Core Pool Size, the tasks will be assigned to the queue
    • • Queuing – Bounded Queue, Un-Bounded Queue
      • Tasks can be queued until a certain limit in the Bounded Queue, which is frequently leveraged
      • Tasks can be queued without having any limit in the Un-Bounded Queue
      • Once the bounded queue has been set with value as pool maximum size, once the inbound task count reach this pool maximum size, that task gets rejected

• Multi-threading concepts

  • Thread Life Cycle

• • • Refer below to the Thread Life Cycle

• • • New
      • This the state where the thread is created.
    • Ready/Runnable

• • • • When the start() method is invoked to a particular thread, it will switch state from New to Ready or Runnable state
    • Running
      • • The Thread will switch to Running state when run() method is invoked. which means when the process is executing. But it may go back to Ready/Runnable state and come back to Running state and this can happen again and again also.
    • Blocked
      •  This is the state where one thread blocked on the lock because other thread has already acquired this lock.
      • For performance engineers, this state is very important to monitor
    • Waiting
      • In this state, the thread will wait forever until if there is any interruption. Usually the invocation of  join() or wait() method will put the thread in waiting state.
      • For performance engineers, this state is very important to monitor
    • Timed Waiting
      • When sleep() method or join() or wait() methods with timeout are invoked, that state is known as Timed Waiting. Name itself explains that the thread will wait for a certain given time.
    • Dead
      • This state represents the completion of process.

• JVM Thread Dump Analysis

• • Whenever your application have issues(crashing/taking more time to respond/etc), we will either take a Heap Dump or Thread Dump that gives us the current snapshot to analyze if the issue is with memory or threads
  • Thread dumps are vital artifacts to diagnose CPU spikes, deadlocks, memory problems, unresponsive applications, poor response times, and other system problems.
  • Ways to take the Thread Dumps
    • jstack

• • • • ‘jstack’ is an effective command line tool to capture thread dumps. The jstack tool is shipped in JDK_HOME\bin folder. Here is the command that you need to issue to capture thread dump: jstack -l <pid> > <file-path>
      • Jstack tool is included in JDK since Java 5. If you are running in older version of java, consider using other options
    • Kill -3
      • In major enterprises for security reasons only JREs are installed in production machines. Since jstack and other tools are only part of JDK, you wouldn’t be able to use jstack. In such circumstances, ‘kill -3’ option can be used
      • kill -3 <pid>

• • • JVisualVM
      • Java VisualVM is a graphical user interface tool that provides detailed information about the applications while they are running on a specified Java Virtual Machine (JVM). It’s located in JDK_HOME\bin\jvisualvm.exe. It’s part of Sun’s JDK distribution since JDK 6 update 7.
      • Launch the jvisualvm. On the left panel, you will notice all the java applications that are running on your machine. You need to select your application from the list (see the red color highlight in the below diagram). This tool also has the capability to capture thread dumps from the java processes that are running in remote host as well.

• • • JMC
      • Java Mission Control (JMC) is a tool that collects and analyze data from Java applications running locally or deployed in production environments. This tool has been packaged into JDK since Oracle JDK 7Update 40. This tool also provides an option to take thread dumps from the JVM. JMC tool is present in JDK_HOME\bin\jmc.exe
      • Once you launch the tool, you will see all the Java processes that are running on your local host. Note: JMC also has the ability to connect with java processes running on a remote host. Now on the left panel click on the “Flight Recorder” option that is listed below the Java process for which you want to take thread dumps. Now you will see the “Start Flight Recording” wizard, as shown in the below figure.

• • • • Here in the “Thread Dump” field, you can select the interval in which you want to capture thread dump. As per the above example, every 60 seconds thread dump will be captured. After the selection is complete start the Flight recorder. Once recording is complete, you will see the thread dumps in the “Threads” panel, as shown in the figure below.

• • • JCMD
      • The jcmd tool was introduced with Oracle’s Java 7. It’s useful in troubleshooting issues with JVM applications. It has various capabilities such as identifying java process Ids, acquiring heap dumps, acquiring thread dumps, acquiring garbage collection statistics
      • Using the below JCMD command you can generate thread dump:
        • jcmd <pid> Thread.print > <file-path>
    • APM Tool – App Dynamics
      •  Few Application Performance Monitoring tools provide options to generate thread dumps. If you are monitoring your application through App Dynamics (APM tool), below are the instructions to capture thread dump:
        • Create an action, selecting Diagnostics->Take a thread dump in the Create Action window.
        • Enter a name for the action, the number of samples to take, and the interval between the thread dumps in milliseconds.
        • If you want to require approval before the thread dump action can be started, check the Require approval before this Action checkbox and enter the email address of the individual or group that is authorized to approve the action. See Actions Requiring Approval for more information
        • Click OK.

• Analyze the generated Thread Dump

  • Let’s have a look at the generated thread dump. The first line displays the timestamp while the second line informs about the JVM:

• • Next section shows the Safe Memory Reclamation (SMR) and non-JVM internal threads:

• • Then, the dump displays the list of threads. Each thread contains the following information:
    • Name: it can provide useful information if developers include a meaningful thread name
    • Whether the thread is a daemon or a non-daemon thread
    • Priority (prior): the priority of the thread
    • Java ID (tid): the unique ID given by the JVM
    • Native ID (nid): the unique ID given by the OS, useful to extract correlation with CPU or memory processing
    • State: the actual state of the thread
    • Stack trace: the most important source of information to decipher what is happening with our application
    • We can see from top to bottom what the different threads are doing at the time of the snapshot. Let’s focus only on the interesting bits of the stack waiting to consume the message:

• • • At a first glance, we see that the main stack trace is executing java.io.BufferedReader.readLine
    • which is the expected behavior. If we look further down we’ll see all the JVM methods executed by our application behind the scenes. Therefore, we are able to identify the root of the problem by looking at the source code or other internal JVM processing.
    • At the end of the dump, we’ll notice there are several additional threads performing background operations such as Garbage Collection (GC) or object termination:

• • • Finally, the dump displays the Java Native Interface (JNI) references. We should pay special attention to this when memory leak occurs because they aren’t automatically garbage collected:

• • • Thread dumps are fairly similar in their structure, but we’ll want to get rid of the non-important data generated for our use case. On the other hand, we’ll need to keep and group the important information from the tons of logs produced by the stack trace.
    • Better to have multiple thread dumps, and search the blocked thread id from these multiple thread dumps, and if we happen to find the thread id in multiple thread dumps, meaning that this thread is still in blocked state. Similarly if we find any thread that is in waiting state/running statein multiple thread dumps, then these too can be considered as the error scenarios and have to be avoided.
      • Report all these issues and problematic line of code details to dev team and ensure resolution and retest for the issue resolution

-By  Saravanan & Sai Krishna

The post Performance Engineering – JVM – Thread Pooling, Multi-threading concepts appeared first on Isha.

• Development – Code Optimization

  • Along with development tasks by development team, performance team can contribute towards Code Optimization methods[How much ever to the extent possible] like
    • Check if redundant code exists, avoid it
    • Check if the existing code is utilizing minimum resources [DB, File System, Threads, etc]
    • Code Review – Leverage static code analysis tools [SonarQube, etc] to ensure existing code base/repository is meeting coding standards
    • Check if any code optimization can be made possible like Caching – Redis, Reducing back end DB calls – Stored procedures, etc

• Production Deployment

  • While development team is working on deploying the changes to Production Environment, the Performance Engineer have to think of monitoring and capacity planning
    • Thus the performance engineer will have the leverage towards forecasting the production problems
    • Plan for capacity of the underlying infrastructure
    • Analyze the infra metrics(CPU and Memory Utilization, etc) and bottlenecks across various layers of application

• Performance Testing / Tuning

• • Ensure to cover various performance test kinds – Load Test, Stress Test, Endurance Test, Scalability Test, Volume Test
    • Load Test – Testing the application with decided virtual users with expected concurrency
      • How to decide on this expected concurrency for an existing application?
        • Leverage your project’s observability platforms like Coralogix, New Relic, Splunk, etc and calculate the daily, weekly, monthly metrics and arrive at an understanding to proceed further

• How to decide on this expected concurrency for a new application?
  • We have to baseline the metrics like response time, transactions/second with single user and calculate the metrics /hour, /day, /week, /month etc
  • Also, we will discuss, review these results with project’s stake holders and arrive at expected concurrency
    • Load Testing focuses on arriving at (x – number of expected) users per (y – duration of performance run) time
• Stress Test
  • Test the AUT for checking the break point (Until which point the application is performing well)
    • Eg: Testing the back-end APIs of the AUT until a point where we get Server Errors (500, 503, etc
• Endurance Test (a.k.a Soak Test)
  • To test the AUT by injecting x users for a longer period of time to check the memory leak
  • General time duration for Endurance test is approximately 4 hours
• Scalability Test
  • Testing the application with the infrastructure that’s available at hand/currently using infrastructure for a specified time, predicting a future load
  • The objective of this test is to understand if the current infrastructure can handle future
• load Volume Test
  • Volume test stresses more on testing those specific application scenarios having more DB(Database – SQL/NO-SQL) interactions
• It is expected to work on all these type of tests, and we might land up on testing a few as per project timelines, limitations, requirements

• Arriving at performance workload

• Use Little’s Law
  • Little’s Law defines as
    • Number of concurrent users a.k.a Concurrency = ( Transactions/Sec a.k.a Throughput ) * ( Response Time + Think Time + Pacing )
      • Pacing = Duration between various iterations
      • All the Transactions/Sec a.k.a Throughput ) * ( Response Time + Think Time + Pacing are having seconds as their unit of measure
    • Do not arrive at a random user concurrency, rather applying Little’s Law is suggested.
• Simulate a production like scenario
  • Analyze the user scenarios and arrive at an understanding on which scenarios contribute to maximum load, average load, and less load, and accordingly run the performance runs to simulate a production scenario
• Baseline/Benchmark the report
  • Baseline

• • • Running the performance script with a single user/desired user percentage, and multiply as per the desired outcome.
    • If we happen to arrive at a particular number for a previous application release, we will use that number as a baseline for the upcoming releases
• Benchmark
  • • To compare the existing number with that of the industry standard that’s been performing well
  • Performance Engineer have to focus on the above mentioned value adds and leverage the benefit towards getting desired results in Performance Testing

-By  Saravanan & Sai Krishna

The post Performance Engineering – Life Cycle – Code Optimization, Production Deployment, Tuning appeared first on Isha.

Types of Garbage Collectors

• Garbage Collection tracks each and every object available in the JVM heap space, and removes the unused ones.
• Basically, GC works in two simple steps, known as Mark and Sweep:
  • Mark – this is where the garbage collector identifies which pieces of memory are in use and which aren’t.
  • Sweep – this step removes objects identified during the “mark” phase.
• JVM has five types of GC implementations

Serial Garbage Collector

• This is the simplest GC implementation, as it basically works with a single thread. As a result, this GC implementation freezes all application threads when it runs. Therefore, it’s not a good idea to use it in multi-threaded applications, like server environments.
• To enable Serial Garbage Collector, we can use the following argument:
  • java -XX:+UseSerialGC -jar Application.java

Parallel Garbage Collector

• It’s the default GC of the JVM, and sometimes called Throughput Collectors. Unlike Serial Garbage Collector, it uses multiple threads for managing heap space, but it also freezes other application threads while performing GC.
• If we use this GC, we can specify maximum garbage collection threads and pause time, throughput, and footprint (heap size).
• To enable parallel Garbage Collector, we can use the following argument:
  • java -XX:+UseParallelGC -jar Application.java
• If we use this GC, we can specify maximum garbage collection threads and pause time, throughput and footprint (heap size). The numbers of garbage collector threads can be controlled with the command-line option
  • -XX:ParallelGCThreads=<N>
• The maximum pause time goal (gap [in milliseconds] between two GC)is specified with the command-line option
  • -XX:MaxGCPauseMillis=<N>
• The maximum throughput target (measured regarding the time spent doing garbage collection versus the time spent outside of garbage collection) is specified by the command-line option
  • -XX:GCTimeRatio=<N>

• A maximum heap footprint (the amount of heap memory that a program requires while running) is specified using the option -Xmx<N>.

CMS Garbage Collector

• The Concurrent Mark Sweep (CMS) implementation uses multiple garbage collector threads for garbage collection. It’s designed for applications that prefer shorter garbage collection pauses, and can afford to share processor resources with the garbage collector while the application is running.
• Simply put, applications using this type of GC respond slower on average, but don’t stop responding to perform garbage collection.
• CMS garbage collector holds all the application threads in the following two scenarios only
  • During marking the referenced objects in the old-generation space.
  • Any change in heap memory in parallel with doing the garbage collection

• In comparison with the parallel garbage collector, CMS collector uses more CPU to ensure better application throughput. If we can allocate more CPU for better performance then CMS garbage collector is the preferred choice over the parallel collector.
• To enable CMS Garbage Collector, we can use the following argument:
  • java -XX:+USeParNewGC -jar Application.java

G1 Garbage Collector

• G1 (Garbage First) Garbage Collector is designed for applications running on multi-processor machines with large memory space. It’s available from the JDK7 Update 4 and in later releases.
• G1 collector will replace the CMS collector, since it’s more performance efficient.
• In G1 collector contains two phases;
  • Marking
  • Sweeping
• Unlike other collectors, G1 collector partitions the heap into a set of equal-sized heap regions, each a contiguous range of virtual memory. When performing garbage collections, G1 shows a concurrent global marking phase to determine the liveliness of objects throughout the heap.
• After the mark phase is completed, G1 knows which regions are mostly empty. It collects in these areas first, which usually yields a significant amount of free space. It is why this method of garbage collection is called Garbage-First.
• To enable G1 Garbage Collector, we can use the following argument:
  • java -XX:+UseG1GC -jar Application.java

Epsilon Garbage Collector

• Epsilon is a non-operational or a passive garbage collector. It allocates the memory for the application, but it doesn’t collect the unused objects. When the application exhausts the Java heap, the JVM shuts down. It means Epsilon garbage collector allows, applications to run out of memory and crash.
• We can use this argument to get a heap dump when the application is crashing due to out of memory.
  • XX:HeapDumpOnOutOfMemoryError

• If we need to squeeze every bit of performance out of our application, Epsilon might be your best option for a GC. But we need to have a complete understanding of how our code uses memory. If it creates almost no garbage or you know exactly how much memory it uses for the period it runs in, Epsilon is a viable option.
• To enable Epsilon Garbage Collector, we can use the following argument:
  • java -XX:+UseEpsilonGC -jar Application.java

Z Garbage Collector

• ZGC (Z Garbage Collector) is a scalable low-latency garbage collector that debuted in Java 11 as an experimental option for Linux. JDK 14 introduced ZGC under the Windows and macOS operating systems. ZGC has obtained the production status from Java 15 onwards.
• ZGC performs all expensive work concurrently, without stopping the execution of application threads for more than 10 ms, which makes it suitable for applications that require low latency. It uses load barriers with colored pointers to perform concurrent operations when the threads are running, and they’re used to keep track of heap usage.
• To enable Z Garbage Collector, we can use the following argument:
  • java -XX:+UseZGC -jar Application.java

• jMeter a java application uses the below Garbage Collector

• Below information is for reference, on performance metrics comparison of various Garbage Collectors

• Use the garbage collectors for the below use cases
  • G1 G.C – While focusing on the latency
  • CMS G.C – While working with common applications
  • Parallel G.C – While working with batch applications
  • Serial G.C – While working with small devices or when the garbage collection doesn’t affect other applications or CPUs

-By  Saravanan & Sai Krishna

The post Performance Engineering – GC – Types appeared first on Isha.

Common problems in Production environment

• When CPU reaches its 100%, we will get RequestTimedOut Errors, by which the response time increases much
• High memory utilization – Elastic Search / Redis caching or API deployed in EC2 instance has generated much logs thereby consuming more memory
• Other issues could be
  • In-Efficient source code that could result in
    • Memory Leaks
    • Dead Lock issues
    • Not using proper data structures that could lead to more CPU an memory usage, etc
  • DB issues
    •  Issues with indexes, etc
  • Not setting up of infrastructure to suite to future growth rate(users, transactions, etc) of data
  • Peak application access on specific days – Thanks Giving, etc
  • Improper load distribution
  • Issues with DNS, Firewall, Network connectivity
  • Issues with a dependent 3rd party application
  • Issues with shared resources
  • Issues with threads
  • Issues with Garbage Collection Memory issues
    • Memory Leak
    • Not leveraging caching mechanism
    • Not allocating proper memory
  • Domino Effect
    • An issue existing at one place creates issues in other places, and we have to properly backtrack the issue for RCA
  • Other miscellaneous issues
    • Network issues
    • Application doesn’t respond as usual, due to scheduled job
  • It will be better if the performance engineer focuses on upskilling on the below aspects
    • Learning programming languages
    • JVM internals
    • Garbage Collection internals
    • Memory Leak issues
    • Source Code tuning
    • Performance Tuning
    • Performance Testing Tools – Load Runner, Neo Load, jMeter, Gatling, etc
    • Understanding the workload – Little’s Law, etc
    • Analyzing the application’s / platform’s logs
    • Identify critical business flows
    • Test data management
    • Basics of script development
    • Understand the performance metrics, parameters, bottlenecks of the Performance Test Report
    • Clear understanding of their application’s architecture
    • Database tuning
    • Analyze the metrics, information from Application’s monitoring tools – Dynatrace, App Dynamics, New Relic, Coralogix, Data Dog, CA APM, Splunk, etc
    • Good grasp on operating system internals
    • Understanding on Load Balancer internals
    • Awareness towards good understanding on Mobile, Cloud, Containerization, Microservices concepts
    • Understanding on browser side metrics – Single user metrics
    • Code Profiling Tools

-By  Saravanan & Sai Krishna

The post Performance Engineering – Performance Engineer – Activities appeared first on Isha.

1. It is mandatory (as well as recommended) that the two Checkboxes in all the Transaction Controllers used in Test Script must NOT be checked (also, by default they are not checked as well).
2. Its mandatory (as well as recommended) to perform “Apply Naming Policy” action for each group of Samplers inside each respective Transaction Controllers by “right clicking” on each Transaction Controllers.
3. Always, remember to Restart the JMeter whenever we do any changes in any of the six“properties” file or any “bat” file in the “bin” folder so that changes can take place in JMeter GUI. Also, remember to Restart JMeter if we do any changes in “lib” folder or “lib -> ext” folder in JMeter directory for the changes to take place.
4. Also remember, while running a Test in JMeter, either in Non-GUI mode or GUI mode, it’s always recommended to disable the unnecessary Elements of that Test Plan/Script.
5. Always remember to use “Backslash” (i.e. “/ ”) while giving File path or Directory path everytimein JMeter wherever required, because JMeter generally recognise the “Forward slash” (i.e. “ \ ”) as Escape Character.

The post JMeter HTML Dashboard Report Generation appeared first on Isha.