How agile helped Cambridge Assessment to scale

In 2008 the media were reporting on mistakes and delays in marking examination scripts. To ensure they never made the front page of newspapers for the wrong reasons, Cambridge Assessment decided to rethink how they electronically marked examination scripts for millions of students worldwide.

Their existing solution, developed using BizTalk, processed tasks in a single long-running transaction. While it was able to process the current number of scripts, Cambridge Assessment wasn't confident it would be able to handle the ever-growing increase in demand as it had not been designed to work at such a scale.

I was hired along with two other engineers, Richard Miller and Ian Johnson, to create a solution that would scale. We joined a team alongside a project manager, test team and a member of the existing development team, whose experience and knowledge of the business domain gave us invaluable insight.

Our Engineering Manager

Adam Straughan was a true engineers manager.

He was open, honest and had the courage to voice his opinions which some of the management team didn't like.

He trusted and empowered us to get on with it. He was passionate about creating a development culture to create innovative solutions using practices such as test-driven development and continuous deployment, a first for a project at Cambridge Assessment.

It wouldn't be an understatement to say he played a significant role in revolutionising software development at Cambridge Assessment.

The agile approach

As far as I know, this was one of Cambridge Assessments first agile projects at this scale.

I had already worked on so many "agile" projects, but this was the best I had experienced.

All thanks to Richard, who did both a scrum master and engineer role at the highest level.

He was able to explain why we were doing what we did and made everything ridiculously simple. I still use his ideas in the teams I lead today.

Stand-ups were short, and to the point, planning sessions focused, and our retrospectives made a difference as we improved as a team throughout the project.

Gaining trust and confidence from the business

Our agile approach was perfect for gaining trust and confidence from the business. We needed to show outcomes rather than just output.

We had access to a stakeholder who had in-depth knowledge and experience of the complexities and rules in marking examinations.

After every sprint, we did demos to the business to show we were applying the same marking algorithms and able to compute the same result as the existing solution. We started with simple user stories at the start of the project before moving on more complex scenarios such as remarks.

Forming, Storming, Norming, and Performing

The four of us had never worked together before. Some of our skills and approaches overlapped others didn't. We were all passionate about software development and went to the same user groups outside of work.

Our team grew over time with more engineers joining.

We learnt a lot from each other, probably no more than when we had conflicting views. It was my first time working with people who were passionate about the product and had to learn how to present my ideas, so they were simple to understand.

It was a psychologically safe environment to disagree. We trusted each other and ensured discussions remained constructive and not personal.

We knew how vital collaboration was. All members of the team put forward ideas and paired together when we needed to. We used whiteboards for discussions which helped to ensure all team members had a shared understanding.

To this day, we have no doubt the application would never have been as good as it was without us being passionate, empowered and having a feeling of ownership which gave us the energy to voice our opinions.

The architecture

As well as being able to process exam scripts correctly, the scalability of our solution would ultimately determine the success of our solution.

It didn't take long for us to work out our approach. As we attended the same talks and read the same resources from blog posts to books, we had similar ideas about what to do.

We split the application into microservices each having a single responsibility. There was a service for:

• storing examination scripts sent from a third party
• for processing examination scripts
• uploading results to an internal service

Transactions boundaries were small using queues to communicate between the services, which enabled additional benefits such as replayability.

So now we could deploy and scale components independently as required. Not bad in the days before auto-scaling!

Developing the solution

We worked closely with the test team to learn about the business domain, which helped us develop a ubiquitous language.

We added instrumentation which enabled us to develop dashboards and build tools to replay scripts which helped to debug and troubleshoot.

We were able to call on architects and database experts to ensure we were on the right path and give advice on best practices and what to look out for.

Our shared philosophy enabled us to build a culture in which engineers could easily understand the vision, collaborate, share ideas, innovate and contribute.

Small details such as being able to reproduce a bug with a unit test before fixing ensured we built a high-quality application.

As our team grew, we learned and adapted to ensure people were happy and enjoying working on the project. For example, we dedicated a different engineer every sprint to take on a 'Batman' role to answer questions from the business and the test team, allowing other engineers to focus without being distracted and having to switch context.

Being in a support role helped engineers learn how crucial logging was for them to understand why something was happening. Now when the engineer returned to regular duties, they understood how critical it was to write tests, handle errors and instrument their code.

Making the grade

Our agile approach of having short feedback loops and incrementally adding features helped us keep focus.

The application was launched on time and successfully processed millions of examination scripts for years to come allowing Cambridge Assessment to build upon its' excellent reputation as a world-class qualification provider.