Archive for the ‘Java’ Category
Integration testing of a legacy code
Wouldn’t it be nice to work only on greenfield projects with the team that shares the same vision and style? Unfortunately, we have to spend a significant portion of our professional life dealing with the messy legacy code. Such code is very hard to comprehend. It often consists of tangled classes, with some arbitrary division of responsibilities. Unit test coverage is often very low. Sometimes unit tests are formally there, but you can clearly see that they have been written after the fact, simply repeating the production code mess. These tests are very fragile and don’t provide adequate “safety net”.
Before trying to refactor such legacy code we must first create good end-to-end tests. Tests that are, as Kent Beck says, “sensitive to changes in system behavior, but insensitive to changes in code structure”.
Let’s assume your project is a typical SpringBoot application. We have some complex functionality, that works, but we don’t fully understand how. We want to do some “forensic analysis” of its implementation and cover it with a stable integration e2e test. We know that this functionality starts with externally exposed REST API. We know that after going through layers of services, managers, helpers, utility classes, repositories, etc. it stores some data in the database and also makes some calls to external systems using REST and/or JMS.
Here is the plan:
- Assuming JUnit 5, create test class “XyzComponentIT” and annotate it with @ExtendWith(SpringExtension.class) and @ContextConfiguration(classes=XyzConfig.class). This XyzConfig class should be annotated with @TestConfiguration and will be used to create Spring beans using @Bean instead of relying on a component scan. This way we will know exactly which beans participate in our workflow.
- Create the first bean for the entry point (REST controller) and inject it into the test using @Autowired
- Create a test case method that invokes this entry point passing some typical payload into it.
- If you try to run the test at this point, it will fail to initialize the Spring context – this is expected, we didn’t provide any controller’s dependencies yet. Let’s start creating them one by one.
- Since we are writing end-to-end integration tests, we want to create beans using a real implementation of every dependency, except “edge components”. Edge components are those that encapsulate communication with external systems – e.g. database repositories, HTTP and JMS clients, etc.
- Beans of edge components should be mocked. One very useful technique here is to use “strict mocks”. “Strict mocks” fail on every call that is not explicitly stabbed. This way we can identify the exact external system communication and crosscheck it with the business requirements. Below I included a snippet of Mockito based implementation of such strict mocks. For better readability, consider encapsulating such mock beans in properly designed test doubles.
- Next, implement the actual test case method, stubbing calls that read external data and verifying essential interactions with the external systems. For example, if our workflow updates the customer profile in the database and sends notifications via email service, then we should verify all interactions with test doubles that encapsulate DAO and email server communication.
Integration tests described here are quite stable since they have minimal knowledge of the internal structure and implementation details of our components. Since they consider the whole system as a black box, they are an essential tool that gives us confidence while doing code refactoring.
Here is the GIST of a very simplified example:
package com.jpragma.jtest; | |
import org.mockito.Mockito; | |
import org.mockito.stubbing.Answer; | |
import java.util.Arrays; | |
public class MockitoExtensions { | |
private static Answer<?> alwaysThrowingAnswer = (invocation –> { | |
throw new IllegalArgumentException("Unexpected call to " + invocation.getMethod() + | |
" with " + Arrays.toString(invocation.getArguments())); | |
}); | |
public static <T> T strictMock(Class<T> clazz) { | |
return Mockito.mock(clazz, alwaysThrowingAnswer); | |
} | |
} |
package com.jpragma.jtest.xyz; | |
import com.jpragma.jtest.MockitoExtensions; | |
import org.junit.jupiter.api.Test; | |
import org.junit.jupiter.api.extension.ExtendWith; | |
import org.springframework.beans.factory.annotation.Autowired; | |
import org.springframework.boot.test.context.TestConfiguration; | |
import org.springframework.context.annotation.Bean; | |
import org.springframework.test.context.ContextConfiguration; | |
import org.springframework.test.context.junit.jupiter.SpringExtension; | |
import static org.mockito.ArgumentMatchers.any; | |
import static org.mockito.Mockito.*; | |
@ExtendWith(SpringExtension.class) | |
@ContextConfiguration(classes = XyzComponentIT.Config.class) | |
public class XyzComponentIT { | |
@Autowired | |
XyzController controller; | |
@Autowired | |
AddressValidator addressValidator; | |
@Autowired | |
CustomerRepository customerRepository; | |
@Autowired | |
EmailServiceClient emailServiceClient; | |
@Test | |
void updateCustomerProfile() { | |
CustomerProfile customerProfile = new CustomerProfile() | |
.setId("001") | |
.setFirstName("John") | |
.setLastName("Doe") | |
.setAddress( | |
new PrimaryAddress() | |
.setStreet("1 Main st") | |
.setCity("Toronto") | |
.setRegion("ON") | |
.setCountry("Canada") | |
); | |
doReturn(true).when(addressValidator).validateAddress(any(PrimaryAddress.class)); | |
doNothing().when(customerRepository).persistProfile(any(CustomerProfile.class)); | |
String email = "jdoe@email.com"; | |
doReturn(email).when(customerRepository).getCustomerEmail(customerProfile.getId()); | |
doNothing().when(emailServiceClient).sendEmail(anyString(), anyString()); | |
controller.updateCustomerProfile(customerProfile); | |
verify(customerRepository).persistProfile(customerProfile); | |
verify(emailServiceClient).sendEmail(email, "Profile Update"); | |
} | |
@TestConfiguration | |
static class Config { | |
@Bean | |
XyzController controller() { | |
return new XyzController(profileUpdateService()); | |
} | |
@Bean | |
ProfileUpdateService profileUpdateService() { | |
return new ProfileUpdateServiceImpl(addressValidator(), customerRepository(), emailServiceClient()); | |
} | |
@Bean | |
AddressValidator addressValidator() { | |
return MockitoExtensions.strictMock(AddressValidator.class); | |
} | |
@Bean | |
CustomerRepository customerRepository() { | |
return MockitoExtensions.strictMock(CustomerRepository.class); | |
} | |
@Bean | |
EmailServiceClient emailServiceClient() { | |
return MockitoExtensions.strictMock(EmailServiceClient.class); | |
} | |
} | |
} |
Identifying technical debt
Today’s very short post is about ways to identify technical debt and how to find components/classes that require immediate attention.
- Files that have a large number of git commits (they are changed often). These files are dependency magnets, they most likely violate “Open-close principle” and “Single responsibility principle”. Create a report by running:
# list repo files with their total number of commits git log --name-only --pretty=format: | sort | uniq -c | sort -nr > git_stats.txt
- SonarQube provides pretty good reports showing the complexity and test coverage of your classes
Properties of good tests
I recently read a very interesting article called “Test Desiderata” published by Kent Beck. In this article, Kent describes the properties of good tests. He also mentions that tests of different levels (unit, integration, acceptance, etc.) should focus on different properties.
Here I’d like to discuss some of these properties and try to find some practical steps to improve our tests in regard to these properties.
Kent Beck says:
“Programmer (aka “unit” tests). Give up tests being predictive and inspiring for being writable, fast, and specific.”
Writable — tests should be cheap to write relative to the cost of the code being tested.
So, unit tests are our first line of defense. They should be extremely cheap to write, they should be easy to read and they must be very fast to execute.
Tools that help to increase the development speed of such tests are Spock Framework and Groovy in general. Even if the production code is in Java, I still prefer writing my unit tests in Groovy. “Optional typing”, “Multiline strings“, “Safe Navigation“, “Native syntax for data structures” and especially “Initializing beans with named parameters” are huge productivity boosters. Check out my presentation deck for a very basic introduction to Groovy. Spock takes productivity even further allowing you to write very concise and expressive tests. My favorite feature is Spock’s own mock/stub framework. It is way more readable than more commonly used Mockito. Enabling Groovy and Spock in Java projects is simply a matter of adding an appropriate Gradle or Maven dependency.
Here is somewhat controversial thought… If our tests are very cheap to write, maybe we should treat them as immutable. I recently saw a tweet suggesting that we should be allowed to write new unit tests or delete existing ones, never modify them. This might sound a bit extreme, but there is a rationale in it. Quite often our tests look good initially, but their quality and readability degrade with time. We refactor production code, which leads to tests failures, we realize that the test is outdated and we start quickly “patching” it.
Kent Beck says:
Inspiring — passing the tests should inspire confidence.
Structure-insensitive — tests should not change their result if the structure of the code changes.
Predictive — if the tests all pass, then the code under test should be suitable for production.
These properties are very important for integration/end-to-end tests. I discussed this in my previous post. When writing “stable” e2e tests, I use real components/collaborators for the entire system and mock only “edge” dependencies such as DAO Repositories, JMS abstraction interfaces, REST clients, etc. I prefer creating dedicated mock implementations of such “edge” components rather than using raw Mockito, JMock, Spock. This helps to make such tests readable for non-programmers. Typical e2e test starts specially configured application context, invokes input component and validates expected behavior at all “edges”. The value of such tests is enormous, they give us the confidence that the entire system is working properly. Being structure-insensitive, e2e tests serve us as safety-net when doing code refactoring.
In one of the future posts, I’ll cover my experience working with legacy code and how to efficiently create e2e tests there.
As usual comments, opinions, and criticism are more than welcome!
Don’t over spec your tests
-
Validate correctness of the system under test
-
Document usage and expectations of the tested module
-
Help designing component’s API and interactions (when practicing TDD)
-
Provide a safety net that enables fearless refactoring
Useful options running “mvn test”
There are some very useful command line options you can specify when executing maven tests:
-Dtest=[testname], this allows selective execution of unit tests, instead of running them all. [testname] might be full class name, class name without package, or even wildcard.
-Dsurefire.useFile=false, this will force surefire plugin to write failing test info to stdout instead of file in target/surefire-reports/. Can be a huge time saver.
Control Log4j message timezone
If your server is running in one timezone, but you want to have log messages be printed using different timezone, here is a simple solution:
- Add log4j extras to project classpath (maven GAV is log4j:apache-log4j-extras:1.0)
- In log4j.xml use EnhancedPatternLayout, specifying required timezone for logging purposes (see example below for EST TZ)
<appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender"> <param name="Threshold" value="TRACE" /> <layout class="org.apache.log4j.EnhancedPatternLayout"> <param name="ConversionPattern" value="%d{ISO8601}{EST} %-5p [%t][%c:%M(%L)] %m%n" /> </layout> </appender>
In this example, log messages will be printed using EST regardless of your server timezone.
Setting timeout on Axis WebService calls
In several projects we participated, there was a need to call external Web Services using Java classes generated by wsdl2java Axis tool (or wsdl2java goal of axistools-maven-plugin).
Sometimes these external systems that exposed Web Services experienced high load, slow database and other factors, which caused our calls to them to take outrageously long time before the response have been received.
So, we had to introduce timeouts and handle such calls as errors.
Looking into Axis documentation we found that there is a wsdl2java “timeout” parameter available in both command line tool and via maven plugin. However setting this parameter had no effect, our calls were still hanging indefinitely. It’s not even clear from the documentation if this timeout is applied to webservice call or to execution of wsdl2java itself.
Anyways to have a timeout on webservice call the one must be set on instance of org.apache.axis.client.Call prior to calling invoke(). But creation of Call instance and setting of all its properties is done from Axis generated code and all your changes there will be overwritten on next wsdl2java run.
The solution is to set timeout property on the class that extends org.apache.axis.client.Stub, it’s Axis generated class and would be called something like MyWebServicePort_PortType. If timeout on Call is not set it uses the value from Stub.
This solution is also much more flexible, because now you can set different timeouts on various calls depending on your business logic.
Finding unused code in Java projects
In many projects, especially big ones, developed over several years, there is a lot unused code. Cleaning up projects and deleting such unused code is important.
- Eclipse can detect unused methods and variables.
- PMD (and PMD maven plugin) can also do this and even be integarated into CI process.
- Eclipse Core Tools plugin can find unused classes (see description here).
Unfortunately all above methods will mark code invoked by reflection (e.g. using IoC container such as Spring) as unused. At the same time garbage code that was left behind together with its unit test won’t be detected.
I see only one way to detect such obsolete code – run the system for a while through all typical scenarios and collect statistics on loaded classes, invoked methods, etc. Of course report generated using this technique can be only considered as a hint and manual verification is required.
Proper solution should probably be done using AOP, instrumenting code either during compile or load time. There is however quick and dirty way that archives similar results.
- Run your app with -verbose:class option (send standard output to file in addition to console using | tee <filename>)
Example:
$ export MAVEN_OPTS="$MAVEN_OPTS -verbose:class"
$ mvn clean jetty:run | tee jetty.log - Execute all typical scenarios of your application
- From generated log file create sorted list of your project classes used in executed scenarios:
$ grep "\[Loaded.*com.jpragma" jetty.log | awk '{print $2}' | sort | uniq -u > loaded_classes.txt
- Create sorted list of your compiled project classes:
$ find target/classes/ -name *.class | sed 's/target\/classes\/\///g' \
| sed 's/.class//g' | tr -s '/' '.' \
| sort > declared_classes.txt - Compare two txt files finding all classes that declared but not used:
$ diff loaded_classes.txt declared_classes.txt | grep ">"
Combined Java+Flex project with hot redeployment
In small applications it makes sense to combine Java and Flex code under same project. I’m a big fan of maven and one of my favorite maven plugins is embedded jetty container. This plugin runs jetty directly from maven and supports automatic monitoring and redeployment of java classes and resources. I thought it would be nice to run Java+Flex projects in the same way and have updated SWF file instantly available in the browser immediately after Flex Builder compiles it.
You can find step-by-step instructions on such project set up on my wiki: http://wiki.jpragma.com/creating-new-javaflex-project
Using Groovy in JMeter
Note: This is a copy from my old blog posted in 2007. Some things mentioned here have been changed since then.
Groovy is a great scripting language intimately integrated with Java. Being able to leverage any code written in Java and at the same time having all the advantages of dynamic programming language, makes it an excellent choice for tasks such as file management, build systems and testing.
JMeter is an open source load test tool. It supports many protocols (HTTP, FTP, JMS and others) out-of-the-box.
Many times I found myself in situation when I wanted to run load test against stand-alone Java components or against an application that uses proprietary binary TCP based protocol. Groovy would be an excellent choice for this task. Some work on integrating JMeter and Groovy was done by Alexandru Popescu, however the project is not available to the public yet. Then I discovered that there is an indirect support for groovy in JMeter itself. JMeter supports BSF, which is an abstraction layer on top of many scripting languages including groovy. BSF distribution that comes with JMeter actually has some glitch in groovy support, but after upgrading to the latest version of bsf.jar it was very easy to start writing JMeter scripts in groovy.
Here the steps I followed:
- download latest groovy binaries from http://groovy.codehaus.org/
- copy groovy-all-1.0.jar to jmeter/lib directory
- download latest bsf binaries from http://jakarta.apache.org/bsf/ (I was using version 2.4.0)
- replace bsf-2.3.0.jar in jmeter/lib with bsf.jar from the version you just downloaded
- start jmeter
- create BSF Sampler and specify “groovy” as a scripting language (see screenshot below)
- type your groovy script and run it
This method is far from being perfect. It actually monitors how much time it takes to execute the script itself, but in most cases you’d like to specify exactly when to start counting execution time [results.sampleStart(); results.samplePause(); results.sampleResume(); results.sampleEnd();]
I hope Alexandru will make his project available soon, or may be I’ll find some time to create more sophisticated JMeter Groovy plug-in.
Another issue is how to transfer data between several scripts. Let’s say you’d like to create 1st script that establishes TCP connection; 2nd that sends proprietary login message and gets back some security token; 3rd that sends some business message (using the same TCP connection and providing the token) and 4th that just disconnects. How to transfer Socket and Token objects between scripts? Well, you can use ThreadLocal. Following code snippet defines ThreadLocalStorage class that can be used from the scripts (isn’t it great that you can just call Java from Groovy?) to save and retrieve any data:
package com.jpragma.utils; import java.util.HashMap; public class ThreadLocalStorage { private static ThreadLocal storage = new ThreadLocal() { protected Object initialValue() { return new HashMap(); } }; public static Object get(String name) { HashMap vars = (HashMap) storage.get(); return vars.get(name); } public static void set(String name, Object value) { HashMap vars = (HashMap) storage.get(); vars.put(name, value); } }
Just package it as a jar file and put into jmeter/lib directory. Now you can write your scripts like this:
Connect:
import com.jpragma.utils.ThreadLocalStorage def socket = new Socket("localhost", 18012); def iStream = new DataInputStream(socket.getInputStream()); def oStream = new DataOutputStream(socket.getOutputStream()); ThreadLocalStorage.set('mySocket', socket) ThreadLocalStorage.set('myInStream', iStream) ThreadLocalStorage.set('myOutStream', oStream) return "success"
Echo Message
import com.jpragma.utils.ThreadLocalStorage def iStream = ThreadLocalStorage.get('myInStream') def oStream = ThreadLocalStorage.get('myOutStream') def str = "My String" oStream.writeInt(str.length()) oStream.write(str.bytes) def msgSize = iStream.readInt() def buf = new byte[msgSize] iStream.readFully(buf) def responseStr = new String(buf) if (str == responseStr) return "success" else return "failure"
Disconnect
import com.jpragma.utils.ThreadLocalStorage def socket = ThreadLocalStorage.get('mySocket') if (socket != null) { socket.close() } return "success"