How the Singapore Government turned to its experimental tech team to solve COVID-19 problems quickly
SINGAPORE: On Dec 21, 2020, a Singapore Airlines cargo plane landed at Changi carrying Singapore's first batch of Pfizer-BioNTech COVID-19 vaccines.
Back then, local transmission of coronavirus cases had dropped to zero on most days. The maximum group size for social gatherings was about to go up from five to eight. Capacity limits at public places would also be raised.
There was cause for optimism and the arrival of the vaccines – with early results indicating they were highly effective against COVID-19 – was another shot in the arm.
But as the crates of vaccines were slid out through the nose of the Boeing 747 and whisked away to a cold storage facility, authorities did not yet say how they would be rolled out to the public.
What was clear was that healthcare and essential workers, as well as vulnerable patients and the elderly, would get priority.
Behind the scenes, Open Government Products (OGP) – an experimental tech development unit under the Government Technology Agency (GovTech) – got to work. The mission: Build an appointment system in two weeks so everyone could start getting vaccinated against COVID-19.
Before the pandemic, OGP was already in the business of rolling out simple yet effective tech solutions quickly. It came up with now-ubiquitous products like the government form building tool FormSG and parking payment app Parking.sg.
OGP would go on to create at least five COVID-19-related products, like a stay-home notice reporting tool, a case management system and an informational website on all things COVID-19. Some of them are still in use today.
Still, a vaccination appointment system presented its own challenges.
"At that time, it was fairly uncertain what was going to happen because there wasn't a service model around vaccination roll-outs," OGP deputy director Chan Chi Ling told CNA.
Before there was an appointment system, the Ministry of Health (MOH) had planned the roll-out to be done in phases with more manual labour involved, including individually calling up those who were eligible, she said.
"Our team’s diagnosis was that we really needed a system that could do this at scale so it reaches millions of people within months," she added.
First, the team had to deal with multiple requests from stakeholders across the Government, ranging from whether people should book a time slot or location first, to more complicated features like whether people could book as a family.
For the latter, engineers would have needed to work with extra data sets like birth and marriage records, adding weeks to the project timeline.
"For people who have not done software development before, it's very hard to imagine how certain requests that seem innocuous can actually dramatically increase the timeline for a project," OGP director Li Hongyi told CNA.
"One thing that we've demonstrated that we can do is that if you need a solution quick, we can make something for you and we can put something together literally within like one or two weeks," he said.
So, Mr Li entered meetings with high-level government officials with clear aims: Strip away what he called "nice-to-have" features and ensure everyone was pulling in the same direction.
"MOH especially is quite a big ministry and there's a lot of stakeholders, and even within MOH it takes time to get consensus," he said.
"I think a very big part of my role wasn't so much about figuring out tech solutions, but explaining what the time trade-off is for any of these things. Because for COVID, even if you make a perfect system but it's six months late – it's useless."
Back in the project rooms, the OGP team brainstormed a system robust enough to handle millions of bookings and intelligent enough to prevent overcrowding at the vaccination centres. A system that crashes coupled with long lines at the centres would undermine public trust.
"At the time, not everyone wanted to take the vaccine. There was a lot of vaccine hesitancy so the pressure also came from building a system that people trusted," Ms Chan said.
The additional time pressure made it "nerve-wracking", she said, pointing out that the system also had to be flexible to cater to MOH's policies, including giving vulnerable groups priority.
One important factor was getting the triage page right to ensure only those who were medically eligible could book an appointment. These conditions were set by doctors, and included not being immunocompromised or having a history of anaphylaxis.
"Usually physicians have very precise terms, but they may not necessarily be understandable by members of the public," Ms Chan said, admitting she did not know what anaphylaxis was before building the system.
"We have to find easy ways to explain what the terms are and make sure that the translations are there so that people of all languages can understand the medical triage."
Still, Ms Chan recognised the medical importance of this. "Let's say somebody gets cardiac arrest after taking the vaccine and our system didn't filter them out accurately at the point of booking, then that’s not going to build public trust," she said.
The OGP team had several exchanges with the doctors, and they eventually settled on a triage page that was more readable and "quite different from the first version", she said.
The team also had to ensure the system was understood by the average user, so they headed to the vaccination centres before it was launched to get feedback.
Ms Chan found some common questions among the elderly, including whether they could take the vaccine if they had other medical conditions, like high cholesterol or heart problems.
"So, we realised that we had to develop a negative list – if you have this condition, it’s still okay to take the vaccine," she added.
The team also developed a separate system for social workers and volunteers to book appointment slots for seniors. These proxies could knock on seniors' doors and answer any queries they might have.
OGP rolled out the appointment system at the start of January 2021, meeting the deadline. Mr Li was pleased with what the team came up with.
"A lot of other countries also had appointment systems, but because they didn't design it quite right, when they opened their vaccine appointment system everyone charged in and it crashed," he said.
"Whereas ours from day one has been pretty solid. Maybe there's a couple of hiccups but overall, you get your SMS, you book your appointment, you show up, no fuss. There's none of this anxiety about refreshing the page and trying to charge in.
"I think that was a really big deal and it gave people a lot of confidence."
DELTA VARIANT HITS
As much as anyone would like vaccines to be the pandemic's panacea, reality proved otherwise.
"We sort of thought the vaccination roll-out was going to be the end of it, but as with most COVID cases, it doesn't quite work that way," Ms Chan said. "Post-vaccination, there were other new problems that emerged."
When antigen rapid testing became the choice for routine testing among the public, OGP built a registration system that could also capture the results. Soon, the more infectious and deadly Delta variant became dominant, leading to about 5,000 new cases a day at its peak.
Before this, when authorities were expecting only a handful of new cases each day, case management was done using Excel spreadsheets.
Once someone tested positive, an MOH officer recorded their details on the spreadsheet and passed it on to another officer who assessed their risk of developing severe disease and whether they should isolate at home or be sent to a facility. After this information was inserted, the sheet would be handed to another officer in charge of arranging the facility.
"And then they have to call you up, get your contacts and then they do the same thing for contacts in case they are quarantining them," said Ms Sonjia Yan, a senior product manager at OGP.
"So it's as much as like 10 stages to this life cycle, and they will just pass the Excel sheets from one team to another."
As cases spiked, it quickly became clear that this practice would be unsustainable. In the middle of last year, OGP was roped in again.
"I think it was about 400 cases a day when we were brought in, and (case operations) were outfitted only for like 50 to 100 cases at most," Ms Yan said.
"The staff were really working overtime at 400 cases a day. So we really had to automate things very quickly so that they could free up staff to do the other duties."
Some said they could not get through to MOH for official advice, while others who were told they would be taken to a facility complained about the long waiting times.
"That definitely was the impetus for us to jump in. One is just that we couldn't process all the cases fast enough. And then the second one was that communications to the public were not very clear," Ms Yan said.
"The call centre was swamped with queries, and all their load populated down to the ops teams, because they had to ask the ops teams what is the status and get back."
Ms Yan said the OGP team had to "dive in" and learn about case operations from scratch. They had to ensure that officers could keep up with the sheer number of cases, so they "attacked" the step that was taking the most time.
The team decided to use a customer service software called Zendesk – OGP's "experimental" nature means it will use proven tools on the market to save time – and customise it for case management.
The system, eventually named Zendemic, automates and tracks case operations. The system automatically updates itself when a patient completes each step of the journey, from filling up a form with their personal details to being registered at a facility.
This means that MOH officers, including those at the call centre, could all log into the same interface and get an instant status update on a patient. Zendemic also automatically sends a health risk warning SMS to close contacts, further automating the contact tracing process.
"Our goal was that whenever someone calls into the call centre with a query that can be resolved very quickly, like where is my grandmother now, the call centre should be able to respond immediately," Ms Yan said.
When everything was on Excel sheets, call centre officers would have to take down the question and call back because they had to go to the ops teams first, a process that could take a day or more, she said.
"I think what really made a big difference was the call centre being able to just log into Zendesk and look at all the status updates of where this grandma is at and immediately just say, 'Okay, she's in the facility, don't need to worry.' That saves so much work for everybody involved."
WEBSITE FOR ALL THINGS COVID
Beyond helping the call centre, OGP also decided to produce an informational COVID-19 website, that broke down complicated policies, presented them simply and with infographics that were easily shareable on social media.
This would be especially for those who tested positive and were unsure what to do next. The idea come after a meeting with a Singapore Armed Forces team – which had been helping out with COVID-19 operations – in October 2021.
OGP software engineer Lam Kee Wei remembers that day, which was a Monday. He had recently got married and was fresh off a week of leave before being thrown straight into the action.
"It just shows OGP's (way of working). Rather than put together a complete proposal, we just started working from day one and see what physical thing we can put out for approval," he said, adding that his colleagues spent several days at the call centre working on it.
OGP immediately set out to find what exactly people wanted to know. The team sent out surveys, interviewed parents and chatted with MOH officers in the call centre. They even embedded themselves in a Telegram chat group for those on quarantine.
"We talked to people on the Telegram channel, we talked to the admins, we read through the chats, we monitored the chat to see what people were saying as well," OGP product designer Natalie Tan said.
"We tried to cover as wide a range of personas as we could so that we will make sure that our solution was all-encompassing for all walks of life, essentially."
This meant that they knew which phrases on the website needed tweaking, and were able to include other information like what family members of COVID-19 patients should do.
OGP built the website, covid.gov.sg, using its Isomer product – used widely across the Government to quickly build standardised web pages – and launched it less than a week later on Friday night. Shortly after, it started getting good feedback on the Telegram channel.
The team started a thread in their Slack channel compiling the good reviews, including positive messages from strangers as well as family and friends, which Ms Tan said was the incentive they needed.
"At the point in time, everything was pretty hectic and the team was pretty high on stress and very low on sleep," she said.
"It just helped with the team's motivation as well, to know that the work that we were actually doing was kind of paying off. We were really helping to make things a little bit easier for people, which was just very fulfilling for all of us."
Mr Li said he was proud of what the team had achieved in Singapore's response to COVID-19, highlighting that he had "too many people trying to work too hard". "I didn't have to push a single person," he said.
In fact, he had to remind colleagues to take a breather after working full tilt for two weeks straight, stressing that they would not be working efficiently otherwise.
This includes rejecting requests from government officials for new features in the OGP products, like integrating them with a hospital system. "It's a really hard conversation to have, but it needs to happen because we just don't have time to do everything," he added.
The need to save time also applied to the question of how much security should be put into their products, especially as they were dealing with things like vaccination records. Mr Li said making these products fully secure could take another two months.
"It's not saying we don't do things securely, but instead of going through the full (process), actually let's just do a penetration test. Check to make sure that all our bases are covered – seems more or less right. We need to get it out," he said.
"And if something goes wrong, we'll be ready to respond, but we need to go. After you get it out, then you go back and reinforce some of the mechanisms, but sometimes you do need to make trade-offs in terms of feature capability."
These time savings were valuable considering the work OGP was already putting in to ensure its products reflected the constantly changing MOH policies that evolved with the dynamic COVID-19 situation.
For instance, the team had to overhaul the covid.gov.sg website only a few days after launch when MOH stopped issuing quarantine orders for close contacts.
OGP also had to modify its back-end data exchange link, called the COVID-19 management backbone, when MOH changed the definition of "fully vaccinated" to include the Sinovac vaccine, or when the same definition required a booster shot.
This is because the public's digital vaccination records pass through this backbone, which then applies what is called business logic to update their vaccination status in other products for travelling or contact tracing.
While altering it might seem like just changing a few lines of code, Mr Lam said the process was complex due to the many permutations involved with different brands and doses of vaccine.
"It was difficult to a point where we did consider making it an interview question for joining OGP, because it was just hard to get it out," he said.
Ultimately, Mr Li hopes to get more budget to hire more people for OGP, noting that COVID-19 has taken up most of its manpower, leaving its other products with little to no support.
"To give you an idea of the order of magnitude, an average engineering team in most other places, in GovTech or whatever, there are 20 to 30 people for each module, like the Moments of Life or Singpass app," he said.
"Our team, the average team is like maybe two to three engineers, a product manager and like half a designer's worth of time. And so it's super lean."
Mr Li said the ability to "build good things quickly" is something the Government has not traditionally invested in, and that IT projects have always been thought of as multi-month or multi-year endeavours.
"This idea of having a responsive task force that can set up solutions in days or weeks as opposed to months or years, was just not a thing that we really had," he added.
"We've demonstrated that it can work. And not just us, but a few other teams in Government have done so as well."