Shaun Chuah

Building the Research Operating System We Needed

Thu, 04 Jun 2026 09:00:00 GMT

A while ago I wrote about the challenge of orchestrating multi-omic data in translational research. That post described the problem. This one is about what we did next - we built Foundry120, a research operating system for translational science.

The Problem, Restated

In modern translational research, the hard part is no longer generating data. Sequencing is cheap. Assays are routine. You can now do 6000 genes at subcellular resolution and pin their x,y coordinates?! Most groups are drowning in data, not starved of it.

The hard part is knowing what you actually have. What samples exist? What clinical data do they link to? What clinical phenotype/treatment outcome are you looking for? Who is allowed to use them? Are they ready for the next research question, or do they need cleaning first, or do we need to recruit more participants?

In practice, the answers to these questions live in too many places. Samples are tracked in one spreadsheet, clinical data sits in RedCap or even more excel files, omics outputs are scattered across different computers and hard drives, and the protocols and consent documents are in a shared drive somewhere. As I said in the earlier post, these files tend to be spread over multiple computers and held by different individuals. When you want to ask a simple question - ‘how many UC patients with anti-TNF failure have both microbiome data and biopsy samples?’ - you end up hunting multiple individuals down.

This is the gap Foundry120 was built to close. It came out of years of running multi-site studies and the pain of experiencing the same problems over and over again.

Three Layers

The earlier post ended on a principle I keep coming back to: embrace the complexity rather than fight it. You cannot force genomics, clinical records, proteomics, and histology into one tidy schema without losing something important. So instead of flattening everything, Foundry120 just accepts the complexity.

Organise. Sample and study tracking across multiple sites. This is foundational - knowing what physically exists. We now track over 30,000 samples, supporting six clinical studies as of writing, and seventeen sample types. Get this layer wrong and nothing above it can be trusted.

Centralise. This layer, which I have spent years thinking about, links clinical datasets, omics outputs, and documents together while preserving the data structures that already exist. Let’s be honest, nobody has the time to make every file/folder structure the same.

Accelerate. This is the layer that turns a well-organised data estate into answers. This is where the agentic AI unlock is.

Agentic AI as a Unifying Force

This is the part I find most exciting, and it changes how I think about the whole problem. Helix is our AI agent, and building it is where my thinking about agentic AI in medicine stopped being a thought experiment.

The idea is simple to state and hard to do well: a researcher asks a question in plain English, and Helix turns it into an approved query over governed data, returning charts and cohort summaries. ‘Show me samples belonging to patients with a particular genotype we are investigating’ should be a sentence, not a data request that takes a fortnight.

In my agentic AI post I mentioned that the real unlock is tool calling - connecting probabilistic models to deterministic systems so an LLM can perform specific, reliable tasks. Helix is exactly that pattern applied to research data. It calls into a governed query layer that knows what data exists, what is allowed, and what links to what. The intelligence is in translating intent into the right query; the safety is in the deterministic layer underneath.

We implemented a huge amount of context engineering/tool design given current model performance, which is extremely interesting and I have no doubt that as more capable models arrive, we can hopefully simplify some of these things we do to optimise LLM context windows.

For decades, the only way to integrate inconsistent datasets was to standardise them first. You agreed on a schema (if you even manage to), then forced every source to conform to it (almost impossible) - the classic data warehouse approach. In the orchestration post I explained why that breaks down for research: the data is dynamic and unstructured, the analysis requirements keep changing, and you cannot even define the ‘grain’ the way a retailer can for sales data. So you were left with a bad choice. Either expend an enormous amount of effort to harmonise the data, or leave everything fragmented and pay the cost every time you ask a question. In fact, it was totally unworkable.

Agentic AI offers a third option, and this is the genuinely new thing. An LLM agent is comfortable with messiness in a way that traditional programming is not. It can all sorts of data (within certain limits and design choices), and reason about how they relate - even when one study calls a field dob, another calls it date_of_birth, and a third buried it in free text. The reconciliation that used to demand a fixed, upfront schema can now happen at query time, semantically, on demand.

In other words, the agent becomes the integration layer. Not by forcing every dataset into one shape, but by understanding many shapes at once and bridging between them. This is exactly what lets the Centralise layer keep its promise of preserving existing structures. You no longer have to choose between standardisation and flexibility - the AI absorbs the inconsistency so the humans do not have to.

I want to be careful not to oversell this. This is still extremely early. The agent still needs the deterministic layer underneath to ground it (and for heavier bioinformatic workflows we have to specify the parameters properly), and it still needs evals - a flexible reasoner over wrong data is just a faster way to be confidently wrong. But the direction is clear. For the first time, heterogeneity is not purely a tax to be paid. Handled well, agentic AI turns a pile of disparate, inconsistent datasets into something you can actually interrogate as a whole.

What 30,000 Samples Taught Me

A few lessons stand out from building this for real rather than in theory.

The boring layer is the most important one. Sample tracking is unglamorous, but it is the bedrock - an AI query layer sitting on messy provenance just produces confident wrong answers faster. Most of the value of Helix comes from the Organise and Centralise layers being right first.

Meeting people where they are beats forcing a migration. The fastest way to kill adoption is to tell a research group that their existing workflow was wrong. Preserving existing structures and linking across them, rather than replacing them, is what made this usable.

And governance is critical. In clinical research the audit trails, queries, tool calls and reasoning chain are needed to analyse how an answer was generated and importantly, to debug and improve the system when the agent gets it wrong.

Where This Goes

In the agentic AI post I imagined ‘software-defined’ hospitals - data-sharing, AI integration, and rapid iteration as core principles rather than afterthoughts. Research infrastructure is the same story one step upstream. If we want AI agents that can reason over clinical and molecular data, they need a governed, well-organised substrate to reason over. That substrate does not build itself and is probably the most challenging issue preventing AI from achieving its promised benefits.

Foundry120 is my attempt at building it. It is still early, but we are gaining huge value, time savings, and we see our team members spend more time focusing on fun and interesting research questions instead of trudging through tedious spreadsheets.

And a sign of how fast this all moves: in my last post I was discussing Grok Code Fast 1 and GPT-5-Codex. Both have already aged like milk - at this point of writing, we’re on Opus 4.8 and GPT-5.5.

Update: I talked through how Foundry120 came about on Talk Python To Me, episode 560, and presented it at the Microsoft Academic Research Exchange.

Agentic AI and the Future of Medicine

Mon, 06 Oct 2025 09:54:00 GMT

I didn’t expect AI to change the way I code this much. This year alone, two models — Grok Code Fast 1 and GPT-5-Codex — have transformed how I work. Grok handles the small stuff, like fixing my poorly named variables. GPT-5-Codex, on the other hand, can build entire features on its own — for example, adding speech input with the Web Speech API.

These tools show what’s now possible with “agentic AI” — models that can think and act for themselves. What once took weeks now takes hours, freeing up time to write posts like this (a rarity lately!). It’s hard to describe how revolutionary this feels. After years of managing several software projects, I finally have real help.

When I return to an old project, I can just ask these models for a quick summary or even to add new features, instead of spending hours catching up. Boring but necessary jobs — like writing tests, cleaning up code, and updating dependencies — are now handled automatically. That’s made me far more productive and lets me focus on the creative work: architecture and design.

Thinking about Large Language Models (LLMs) in Medicine

Watching these AI agents evolve makes me think about the future of LLMs in medicine, especially in inflammatory bowel disease. Two ideas stand out: tool calling and agentic architectures. Tool calling connects probabilistic and deterministic systems, so LLMs can perform specific tasks. Agentic architectures allow LLMs to decide which tools to use. This is a real game changer - it means we can automate complex workflows.

But before that, the first step for medical applications is learning to ground an LLM in up-to-date and reliable medical knowledge. Just a few years ago, when ChatGPT first appeared, many criticised its hallucinations and inaccuracies. Fast-forward to today, and things are very different. I thought it’d be interesting to see if we can ground an LLM like ChatGPT in robust medical knowledge, which led to the launch of ChatIBD - a fun project to test how well this tech works in practice. Although it is not perfect, I think it is surprisingly good. It’s also a glimpse into what future clinician-AI partnerships might look like.

Once LLMs are grounded in reliable medical knowledge, we can then apply tool calling and agentic architectures.

Thought Experiment - The IBD Surveillance AI Agent

Let’s take a concrete example: an AI agent that manages IBD surveillance scopes. This feels technically feasible today. Here’s how it would work:

The trigger - for example, in an outpatient IBD clinic, opening the EPR of a patient triggers the agent to begin.
The AI agent uses the patient ID to lookup endoscopy/pathology reports and clinic letters, understanding the patient context - things like disease duration, extent, presence of PSC etc.
Next, we add the relevant/up-to-date guidelines into its reasoning chain.
Using all the right context, the agent then works out the appropriate interval for the next colonoscopy.
It interfaces with the endoscopy booking system to find suitable slots.
It presents a summary to the clinician and patient for review - allowing humans to account for things like holidays etc.
Once confirmed, the appointment is booked.

Done right, this can improve adherence to surveillance guidelines, reduce the workload on clinicians and administrative staff, and improve patient outcomes. In addition, when guidelines change, updating the AI agent’s information will immediately send the change everywhere, ensuring that all patients receive care based on the latest evidence.

The biggest challenge here isn’t AI, it’s the hospital’s existing software setup. Hospitals need sandbox environments to test and connect with existing EPRs (electronic patient records), which most large systems simply don’t have or are hidden behind a wall of bureaucracy. Hospitals will also need to build specific APIs (application programming interfaces) that allow AI systems to retrieve data from current databases. EPR systems have to become open and interoperable, but current vendor incentives rarely align with that.

The ‘Software-Defined’ Hospital

Smaller, more agile hospital systems will likely be the first to adopt these AI workflows. Large hospital systems are likely to be hampered by legacy systems and procurement processes that are not agile enough to keep up with the rapid pace of AI development. Smaller hospitals can try and test new AI tools faster - much like how startups move quicker than large corporations. These changes could improve patient care and cut costs.

Just like how Tesla, BYD and other new car manufacturers came up with the idea of the ‘software-defined’ vehicle, we’ll likely see ‘software-defined’ hospitals too. Data sharing, AI integration, and rapid iteration are going to be the core principles that enable future AI workflows.

Interestingly, I think developing countries may actually leapfrog here. With fewer legacy constraints, bureaucracy, and stronger incentives for cost efficiency, they can iterate and adopt next-gen EPR systems purpose-built for AI workflows much faster.

A quick word on regulation - too much could slow progress here. These systems are complex and too much bureaucracy could push the best builders elsewhere. We need people who understand the tech to strike the right balance - encouraging innovation while maintaining robust systems for safety and oversight.

Conclusion

That said, we’re in an exciting time for medicine. Scalable tools and smarter workflows will take away tedious tasks and make being a doctor much more enjoyable. These technologies will boost every doctor’s productivity, letting us focus on the many other parts of the job that matter most. I look forward to seeing how this space develops over the next few years.

Multi-omic Data Orchestration

Fri, 13 Dec 2024 09:02:45 GMT

Over the last few years, one of the key challenges we have faced in our multi-omic translational studies is the orchestration of multi-omic datasets. In this blog post, I summarise some of the strategies we have employed to manage this complexity, after dedicating significant time to considering this problem.

Industry Data Warehousing

I recommend the Kimball book on data warehousing (The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modelling) as a starting guide to see what other people have done in tackling ‘big data’.

The concept of dimensional modelling is also interesting because, in IBD studies, one of the big challenges is modelling drug data properly. I have personally seen all sorts of techniques used to model drug data, but each has its own shortcomings when performing complex analyses on them.

The Challenge

Having said that, the data warehouse approach is not entirely applicable to research data workflows. It tends to be more suited to high-volume transactional data, where the data is relatively static and well-structured. In research, the data is often dynamic and unstructured, and the analysis requirements are constantly changing. This makes it difficult to design a data warehouse that can accommodate all possible analysis scenarios. It is hard to define the ‘grain’ compared to, for example, sales data for a retailer.

Multi-omic data is complex and often requires a lot of data wrangling to get it into a format that is suitable for analysis. This is especially true when dealing with data from multiple sources, such as genomics, transcriptomics, proteomics, metabolomics, and clinical data. Each of these data types has its own unique characteristics and processing pipelines.

Another major issue, particularly when working in a group, is that these data files tend to be spread over multiple computers and held by different individuals. How do we approach this complexity?

Embrace the Complexity

Initially, I spent a lot of time thinking about how to simplify things, but with experience, I have learned that we must embrace the complexity of this data landscape. This means that we must be prepared to integrate data from multiple sources, in multiple formats, and at multiple scales. This is not a trivial task and requires a lot of careful planning and coordination.

I like to break down data into the following model:

Structured data:
- RedCap/SQL databases.
- CSV/excel files.
- Easy to extract and transform.
Unstructured data:
- Raw -omic files such as fastq.
- Processed -omic files such as gene expression matrices, variant call files.
- Other types of unstructured file data such as video, images, histology slides etc.

The Strategy

Structured data should be extracted, transformed, and loaded into a downstream database.
Unstructured data should be stored in cloud object storage such as AWS S3 or Azure Blob Storage. This allows for easy programmatic access and therefore the ability to integrate these data files with structured data.
Team members should be able to access both raw and processed files at certain stages necessary for their work. For example the seurat object used in single-cell analysis pipelines is a very specific data structure that needs accommodation.
Use a data orchestrator to coordinate and manage the complexity of the data landscape.

Data Orchestrator

A data orchestrator is a tool that allows you to define data pipelines in code and automates the transformation, integration, and loading of data from multiple sources. This unifies and streamlines data processing workflows and can save team members a lot of time, providing easy access for someone with microbiome expertise to access genomic data and integrate the two.

Another benefit of defining data transformations in code is the ability to version control your data pipelines, ensuring they are reproducible. This is particularly important for addressing the replicability issue in research, especially with increasingly complex data processing workflows.

Another significant advantage is the ability to standardize the data dictionary across different studies. Often, researchers use slightly varied names or coding schemes for their variables. A data orchestrator enables you to define transformations and centralize the data dictionary, facilitating the integration of datasets and the reuse of data collected at considerable time and expense.

There are a number of data orchestrators, and the most established is Apache Airflow, widely used in industry. However, it is a complex piece of software that takes significant expertise to set up and maintain. Managed services are offered by cloud providers, but they are expensive.

Dagster

Having tried a number of these tools, we have settled on Dagster. Dagster adopts a declarative approach to data assets rather than a functional programming approach seen in other orchestrators. It also allows for easy scheduling and testing of data pipelines. More importantly, it is relatively easy for a small team to set up and run. We run it using Docker Compose and secure it with Caddy as a reverse proxy.

We use Dagster to extract data from multiple sources and present it within G-Trac (our Django app) as a unified view. G-Trac maintains access control and audit logs for all data access. Team members can now visit a single point for all their data needs.

Conclusion

Investing time in solving this issue is worth it, as it enables data re-use and scalability, which are essential for comprehending the high-dimensional biological datasets produced by the latest biotechnologies. We’ll monitor our progress, and I may provide updates in the future.

If you face a similar challenge, I recommend you look into data orchestrators, particularly Dagster. They can help tame the complexity of multi-omic data and provide a unified view of your data landscape. Having said that, none of this is trivial, so feel free to connect on X/Twitter if you have any questions.

How to Handle Big Data Uploads in Django

Sun, 10 Dec 2023 18:59:15 GMT

Traditional File Uploads in Django

Handling non-trivial big data file uploads (think >5gb/file) in Django can be challenging. A typical file upload storage strategy in Django is to use django-storages and an Amazon S3 backend. In the traditional way, your browser uploads the file directly to Django which then transmits the data to S3.

This creates two problems:

Django needs to have enough memory to hold your uploaded file for retransmission.
The Django worker receiving the file upload is blocked until the upload is finished.
Streaming the upload onwards to S3 doubles the amount of bandwidth used.

What if we could instead upload directly into AWS S3 while registering the upload with Django?

I stumbled across this excellent article by Radoslav Georgiev of HackSoft: https://www.hacksoft.io/blog/direct-to-s3-file-upload-with-django

It’s a slightly complicated set-up but in essence, when the browser wants to perform an upload, it tells Django so and Django gives it a pre-signed S3 URL which will accept the upload.

The upload is then performed directly by the browser to S3. On completion, S3 lets the browser know it was successful and the browser then informs Django that the upload is successful.

This forms the basis of big data handling in Django and allows browsers to connect directly to S3 which in itself is an amazing capability. I highly recommend you check out the linked tutorial above!

Launching Nextflow Pipelines From the Cloud

Thu, 28 Apr 2022 18:29:04 GMT

Why bother setting up another server to manage your Nextflow pipeline?

Avoid premature termination of the pipeline. When you run a Nextflow pipeline from your local computer, your local computer is managing the tasks and communicating with Azure as jobs are completed. Depending on the complexity of your pipeline, this may be a long time (couple of days!). If your wi-fi router accidentally comes unplug or the connection is broken, the pipeline will terminate prematurely. While you could recover from this using -resume, you can avoid the risks of disconnection by using a cloud server, which is designed to run 24/7, to orchestrate this.
Location independence. You can start a pipeline from home (wfh being the consequence of covid) and perhaps go elsewhere and check in later.
Free up your local computer for other things. You do not have to keep your local computer running Nextflow constantly - just start the pipeline in the cloud and continue on with your usual day.

Obtaining a cloud server

Plenty of hosting providers provide you with access to a VPS (Virtual Private Server) running Ubuntu 20.04. My recommendation is DigitalOcean - fantastic documentation, much cheaper compared to AWS/Azure/GCP, good performance and reliability in my experience. The $5/month server is what I personally use and is more than capable of handling Nextflow.

Setting up an Ubuntu server on DigitalOcean

Server authentication

For authentication, select SSH key to access your server from the terminal (mac/linux) or using PuTTY (windows):

Logging in to the server

Once your server is set up, log in to it.

Mac/Linux:

ssh -i <private_key.pem> root@<server_ip>

For Windows I suggest using the PuTTy client. Depending on your choice of VPS and platform, you may wish to consult this guide: https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys-on-ubuntu-20-04

Installation

Install Java and screen first.

sudo apt update
sudo apt install default-jre screen

We will install Nextflow in /nextflow. You can change this location if you wish.

cd /nextflow
curl -s https://get.nextflow.io | bash

We then need to add this directory to PATH. Doing this allows you to type nextflow directly into the terminal. Open .bashrc file to begin.

nano ~/.bashrc

Add the bottom, add the following line:

export PATH=“/nextflow:$PATH”

# this line is optional
export NXF_DEFAULT_DSL=1 
# I am using DSL1 as per the docs when I was setting up my pipeline.
# If you are using DSL1 you will need to add this to .bashrc.

Save and exit with Ctrl+X and yes. Next, we need to source the file.

source ~/.bashrc

Check Nextflow works

If everything is installed correctly, typing nextflow into the terminal should show the program help screen.

nextflow

Clone and run the pipeline

Now it’s time to bring in your pipeline using the power of git.

cd ~
git clone <your_nextflow_pipeline_repository>

For example, I would put in

git clone https://github.com/shaunchuah/cfdna_nextflow/

and this will bring in my pipeline at ~/cfdna_nextflow.

For my pipeline I would need to clone sample_credentials.json to credentials.json and fill it in with my Azure credentials.

cd ~/cfdna_nextflow
cp sample_credentials.json credentials.json
nano credentials.json

Now you can launch the pipeline!

nextflow pipeline.nf -resume -profile test

Now your cloud server is orchestrating the cloud cluster! But let’s take this one step further…

Use screen to launch long-running pipelines

Reference: https://www.howtogeek.com/662422/how-to-use-linuxs-screen-command/

Here’s what I do when launching pipelines that may take days to weeks to complete. Create a screen session:

screen

If this is the first time, there will be some text. Hit enter and it all disappears but you are now in a screen session. Try any command - eg. nextflow.

To detach from the screen session, hit Ctrl+A and D.

To list all screen sessions type screen -ls.

To reattach to the screen session type screen -r.

I launch my pipelines from within screen and hit Ctrl+A and D to detach. Now I can safely close my SSH connection to the server and the pipeline will remain running. Reconnect to the server and type screen -r and you will see that the pipeline remains running. This nifty trick will help avoid pipelines terminating because of flaky connections and give you true mobility to perform bioinformatics from anywhere with an internet connection.

It also means that in 2022, you no longer need powerful desktop computers to run bioinformatics anymore. All you need is a credit card and you can access vast computing power way beyond any single machine is capable of.

I hope you’ve found this series helpful. If this has helped you please let me know on Twitter!

Bonus: If you want to take this even further, you could have a powerful cloud server to develop on using VSCode Remote Containers. Check out GitHub Codespaces as an example of this.

Setting Up Azure With Nextflow

Wed, 18 Aug 2021 18:27:01 GMT

We will cover getting your nextflow pipeline up and running in the cloud using Azure. First thing to note, this process will take some time so sit back, grab a coffee and take your time working through each section.

Useful References

Nextflow documentation here. Nextflow blog post here.

Context

Let’s start with some context. I’m working with a small team of immunology researchers who are following their scientific questions which has led into genomics. We don’t have any bioinformatics setup and are starting from nothing - which is probably going to be the case for many newcomers as NGS technology continues to become more accessible (for example Nanopore sequencing).

The University has a compute cluster but it is equally challenging to get up and running there as on the cloud. In 2021, if you are starting from scratch, I would recommend a cloud-first approach as the cloud allows scalability way beyond any high performance compute cluster at a fraction of the cost. Also, you can access the cloud from anywhere and do not have to setup VPNs etc. to use a university network.

Why Azure over GCP/AWS?

When I first attempted to get Nextflow up and running in the cloud, naturally AWS (Amazon Web Services) was the first contender. For those of you new to cloud computing, AWS is the most established player having been around the longest.

However, following the Nextflow documentation requires you to create a custom Amazon Machine Image that has awscli installed and no matter how many different attempts I tried, I just couldn’t get it to work properly. I suspect updates to Amazon’s infrastructure have changed the previous instructions available on the internet.

After Amazon, I tried Google Cloud using their Cloud Life Sciences API and Nextflow. It was incredibly easy to get things up and running however I soon ran into limitations on vCPU quota. Google limits new accounts to 8 vCPUs even though you have put your credit card on and upgraded your account.

When I contacted Google Support about increasing the quota, they replied and suggested contacting one of their partners and sent a link with a list of maybe a hundred companies, of which you would have no idea who would do bioinformatics. Also, I’m really not interested in going through a third party to get my account running. I might also mention my slight worry that Google does not have a great track record of supporting their non-advertising projects. See Here (I will never forget Google Reader!)

Finally, after much time spent on the above two, I had nothing to lose giving Azure a try. Initially, Azure’s interface was almost as complex as AWS. However, once you set it up, Azure support were the most responsive in getting your vCPU quota up. Within 30 minutes of setting up my account I obtained 48 vCPUs and subsequently had it increased to 128 with very little hassle at all.

Therefore, if you’re going to do what I did, I would highly recommend you try Azure first and this is my documentation of what worked in the end after about 2 weeks of effort.

Setting Up Your Azure Account

Let’s kick things off. First, sign up for an Azure account (that pretty much goes without saying).

Azure Cloud Dashboard

Here’s the important bit.

Once you have your Azure account, within Azure, you need to create two accounts: one Azure Batch account and one Azure Storage account.

Batch and Storage Account Creation

Login into Azure Portal and search for ‘batch’ in the top bar. You should then click on Batch accounts. Create a batch account name and the location where you want to run your compute. When you are creating your batch account, it will ask you to associate a storage account - now is a good time to create your storage account as well.

Create A Storage Container

Once you have your storage account, enter it and click on containers in the left menubar. Then click ‘+ container’ and set up a ‘blob’ - this is equivalent to Amazon S3 and will function as the filesystem Azure Batch will read and write files to.

Get Your Account Keys

Batch account: Enter your batch account and click ‘Keys’ in the left menubar. Take a note of the name and key1.

Storage account: Enter your storage account and click ‘Access Keys’ in the left menubar. Again, take note of the storage account name and key1.

That’s it for this step. Now it’s time to setup Nextflow.

Configuring Nextflow

I’m going to show you an example configuration here and explain it in detail. The details of your configuration will depend on your pipeline and compute requirements. I hope this serves as an easy template to follow. See the previous post for the nextflow pipeline.

profiles {
  az {
      params.reads = 'az://scgenomics/test_data/*/*_{R1,R2}_*.fastq.gz'
      params.outdir = 'az://scgenomics/test_reports'
      params.bowtie2_reference_index = "az://scgenomics/bt2_index.tar.gz"
      params.kraken2_db = "az://scgenomics/k2_standard_8gb_20210517.tar.gz"
      params.metaphlan_db = "az://scgenomics/metaphlan_db.tar.gz"
      params.cpus = 16
      process.executor = 'azurebatch'
      workDir = 'az://scgenomics/work/'
      azure {
          storage {
              accountName = azure_config["storageAccountName"]
              accountKey = azure_config["storageAccountKey"]
          }
          batch {
              location = 'uksouth'
              accountName = azure_config["batchAccountName"]
              accountKey = azure_config["batchAccountKey"]
              autoPoolMode = true
              pools {
                  auto {
                      vmType = 'Standard_D16_v3'
                      maxVmCount = 8
                      autoScale = true
                  }
              }
          }
      }
  }
  az_prod {
      params.reads = 'az://scgenomics/data/*/*_{R1,R2}_*.fastq.gz'
      params.outdir = 'az://scgenomics/reports'
      params.bowtie2_reference_index = "az://scgenomics/bt2_index.tar.gz"
      params.kraken2_db = "az://scgenomics/k2_standard_20210517.tar.gz"
      params.metaphlan_db = "az://scgenomics/metaphlan_db.tar.gz"
      params.cpus = 16
      process.executor = 'azurebatch'
      workDir = 'az://scgenomics/production_work/'
      azure {
          storage {
              accountName = azure_config["storageAccountName"]
              accountKey = azure_config["storageAccountKey"]
          }
          batch {
              location = 'uksouth'
              accountName = azure_config["batchAccountName"]
              accountKey = azure_config["batchAccountKey"]
              autoPoolMode = true
              pools {
                  auto {
                      vmType = 'Standard_D16_v3'
                      maxVmCount = 8
                      autoScale = true
                  }
              }
          }
      }
  }
}

Create test and production profiles

In your nextflow.config file - set up profiles. I’ve set up two profiles which are almost identical. One is for testing and the other is when you actually want to run the real workflow. In the following example my test profile is named ‘az’ and the production profile is ‘az_prod’. Feel free to change it to what you want.

Let’s break it down

First, let’s start with pipeline parameters. This will probably be different to your application. I’ve identified where my data resides and where I want my reports to be placed.

  params.reads = 'az://scgenomics/data/*/*_{R1,R2}_*.fastq.gz'
  params.outdir = 'az://scgenomics/reports'

2 Major Points to Note Here

  params.bowtie2_reference_index = "az://scgenomics/bt2_index.tar.gz"
  params.kraken2_db = "az://scgenomics/k2_standard_20210517.tar.gz"
  params.metaphlan_db = "az://scgenomics/metaphlan_db.tar.gz"

Important Azure’s containers doesn’t work like a real filesystem. When using reference databases I found that the pipeline doesn’t read folders. What I’ve done is zipped up the reference DBs and unzipped them in the process so that the database is copied over to the VM.

Second Point I would upload a copy of the reference database to your storage account. This is because pulling files from your storage account to the VM is really fast whereas if you asked the VM to download the database from the original external servers you are going to cause two major problems:

It’s much slower downloading a database from S3 to Azure.
You’re going to incur high bandwidth transfer costs both for yourself and for the team hosting the database. Not a great idea to download 20GB everytime you run a single process.
Be nice and considerate to the kind folks who host the databases

Therefore, download a single copy of the reference database, zip it up and put it in your storage container.

  params.cpus = 16

This just allows me to tweak the number of cores and is completely optional. My local machine has only 4 while I use 16 core VMs on Azure. On the process side of Nextflow I declare the cpus option as a variable that allows me to tweak the entire pipeline with a change of a single number.

Here’s a brief snippet to give you an idea (pipeline.nf file):

process bowtie2 {
    container 'biocontainers/bowtie2:v2.4.1_cv1'
    cpus "$params.cpus".toInteger()

Finally the magic

  process.executor = 'azurebatch'
  workDir = 'az://scgenomics/production_work/'
  azure {
      storage {
          accountName = azure_config["storageAccountName"]
          accountKey = azure_config["storageAccountKey"]
      }
      batch {
          location = 'uksouth'
          accountName = azure_config["batchAccountName"]
          accountKey = azure_config["batchAccountKey"]
          autoPoolMode = true
          pools {
              auto {
                  vmType = 'Standard_D16_v3'
                  maxVmCount = 8
                  autoScale = true
              }
          }
      }
  }

This block configures Azure Batch settings. You will see that I run Standard_D16_v3 machines and I use up to 8 VMs for my pipeline. Have a look at Azure’s VMs to have an idea of what machines are available. Your settings for this will be entirely dependent on your quotas and what you’re trying to run. Since kraken2 requires 64GB of RAM those are the VMs I’ve selected.

Separating Credentials Out of Nextflow Config

I’m assuming you are using git to version track your code. There are other tutorials on how to do that so look it up if you don’t. Personally I commit my code to GitHub and would suggest you do something similar as it will allow further cool automations that I will cover in the future.

In the above section you will notice this block of code which is different from what Nextflow’s documentation is.

accountName = azure_config["batchAccountName"]
accountKey = azure_config["batchAccountKey"]

I’m going to explain how to separate out your credentials from your nextflow.config file. You will often hear people scream ‘Don’t commit your credentials to git!’ They are right of course - if people get hold of your credentials they could run their pipelines on your credit card!

However, how to avoid doing this is often unclear and leaves beginners wondering what they are supposed to do instead..

How I Separate Credentials Out

I have my credentials in a separate JSON file named credentials.json that is added to the .gitignore file to avoid committing that to the repo.

Here is the json file which is literally 6 lines that you can fill in and rename to credentials.json

{
    "storageAccountName": "",
    "storageAccountKey": "",
    "batchAccountName": "",
    "batchAccountKey": ""
}

Make sure you add the line credentials.json to your .gitignore file. If you don’t have .gitignore, create one.

Now open nextflow.config and at the top here’s what I’ve got:

import groovy.json.JsonSlurper
def JsonSlurper = new JsonSlurper()
azure_config = JsonSlurper.parse(new File("./credentials.json"))

This allows nextflow.config to read your credentials.json file which is in the same directory as the config file.

Side note: due to relative file naming you do need to run nextflow in the root directory of your repo.

That’s it, 9 lines of code and a .gitignore file to protect you from exposing your sensitive information on the internet!

Running Test and Production Pipelines

To run my test pipeline (my pipeline file is named pipeline.nf, other common names include main.nf):

nextflow pipeline.nf -resume -profile az

To run production:

nextflow pipeline.nf -resume -profile az_prod

You can watch Azure Batch spin up VMs in Azure Portal.

Requesting A Quota Increase

Go to your batch account, click ‘quotas’ in the sidebar and then there is a button ‘Request quota increase’. Fill that in for your desired VM and Microsoft Support will hopefully quickly approve that.

What’s next?

The above steps should get you up and running initially on your own cloud supercomputer. However, you will quickly realise that this launches the pipeline from your local machine and as you know, real production pipelines can run for hours. During this time, your computer needs to remain active as it is the main controller for scheduling and arranging jobs.

In a future article, I will cover launching pipelines from a cloud server or ‘head node’ and using screen to keep long running processes active. This will let you launch pipelines from anywhere and turn off your main computer. You can even code directly on your cloud machine using VSCode Remote. Once you’ve tried this you will never go back!

How to Create Your Bioinformatics Pipeline with Nextflow

Thu, 12 Aug 2021 18:24:45 GMT

Now that you know how to run bioinformatics software in Docker containers, it’s time to connect them up. If you’ve missed the last post the link is here: Getting started with Docker for bioinformatics.

Content Overview

What is a pipeline?
Nextflow vs Snakemake
Using Nextflow and Docker containers to create your pipeline
Summary

What is a pipeline anyway?

The term ‘pipeline’ is thrown around a lot in bioinformatics. In simple terms, it refers to the programs that have to be run in a certain order to complete the analysis. Some of these programs take the outputs of earlier programs and process them in order to achieve a specific objective.

This can be as simple as a couple of programs or it can become a messy spider web. A basic example would be: raw fastq files from the sequencer –> passing it through an aligner against the human genome –> variant calling.

NF core have a nice set of pipelines here including much more complex ones.

How to get started

There are two key frameworks that I will point you to: nextflow and snakemake. These are the programs that will run your programs for you and orchestrate the input and output files.

Nextflow vs Snakemake

Disclaimer: this is my personal opinion on this topic. Try both and make up your mind as to which suits your thought process better.

1. The programming language difference doesn’t matter

Snakemake is based on python whereas nextflow uses groovy (which is like python for java). Initially when I first came across these comparisons I immediately jumped to snakemake as I’m pretty comfortable with python.

Having tried both now, I would say that since you need to learn the pipeline syntax anyway, the actual difference between python and groovy is pretty minimal.

2. The biggest difference is the ‘thinking direction’ of the pipeline

Let me explain what I mean. In snakemake you have to ‘think backwards’ and work your way from the desired output towards the input. I find nextflow way more intuitive in that you think about your pipeline in sequence of how you run your programs.

Here’s how you have to think in snakemake: I need output C which comes from program B and to get program B to run you need the output of program A.

Compared to nextflow: With my input files I run it in program A and then take it to program B. Output C will result.

3. Why I switched to Nextflow: The separation of how the pipeline is run from what the pipeline is

The key reason why I switched from snakemake to nextflow is ‘the decoupling of pipeline execution’ or in plain terms, the separation of where and how your pipeline is run and what the pipeline actually does. The series of steps is described in the main file while the execution can be controlled by the configuration. We will cover pipeline execution in a separate article.

Nextflow Pipeline

It is now time to go into more detail on setting up your pipeline in nextflow. Here’s an example pipeline which takes in Illumina paired reads, runs them through fastqc and bowtie2 and perform some simple statistics on the aligned file.

Prerequisites

Install Nextflow
Install Docker

1. Set up and define your input and outputs

First we set up a couple of parameters that can be easily changed depending on whether you are running test files vs actual data. These parameters can be modified in your config profiles.

// main.nf file
#!/usr/bin/env nextflow

// PIPELINE PARAMETERS HERE

// Input Files
params.reads = "$baseDir/data/*/*_{R1,R2}_*.fastq.gz"

// Report Directory
params.outdir = 'reports'

// Reference Genomes
params.bowtie2_reference_index = "$baseDir/reference_db/bowtie2/bt2_index.tar.gz"
bowtie2_db_ch = Channel.value(file("${params.bowtie2_reference_index}"))

2. Create ‘channels’ and feed your input into them

In nextflow, there is this concept of ‘channels’. The basic premise is simple - each channel represents one file that can only be consumed once (unless it’s a ‘value channel’). Think of channels as pipes that you can feed files into which can be used only once.

In this example, we generate the channels using the fromFilePairs method and create 2 channels - fastqc_reads and reads_for_alignment. *side note this is using the standard nextflow language rather than their newer DSL2.

reads = Channel.fromFilePairs(params.reads)
reads.into { fastqc_reads; reads_for_alignment }

3. Now let’s have a look at our first process and break it down

Here is the process block - this process is named fastqc_run but you can change this to whatever name suits.

process fastqc_run {
    publishDir "$params.outdir/fastqc/$sample_id/", mode: 'copy'
    container 'biocontainers/fastqc:v0.11.9_cv8'
    cpus 16

    input:
    tuple val(sample_id), file(reads_file) from fastqc_reads

    output:
    file "*_fastqc.{zip,html}"

    script:
    """
    fastqc $reads_file -o . --threads ${task.cpus}
    """
}

In general, there are 4 components to each process:

A configuration block
Process inputs
Process outputs
Script that runs the desired program

Let’s break down the above process codeblock into the 4 components.

Configuration

  publishDir "$params.outdir/fastqc/$sample_id/", mode: 'copy'
  container 'biocontainers/fastqc:v0.11.9_cv8'
  cpus 16

publishDir This allows you to copy the output files of this process to a desired location for easy access. Note that by default it creates a shortcut link to the actual file location and you have to explicitly specify for it to copy and give you the actual file.

container This is where the real magic happens. If you specify a publicly available Docker container, Nextflow will seamlessly pull the container in, run the script and generate the output that you’re looking for. *side note on the config you will need to specify docker enabled as true.

cpus This specifies the amount of CPUs you wish to run for this process. You can also specify memory options and will help in autoscaling your cloud computation needs. In general this option is useful if you are using VMs of varying size and works quite well with Google Cloud.

Inputs

  input:
  tuple val(sample_id), file(reads_file) from fastqc_reads

The Channel.fromFilePairs method generates a tuple [wildcard value, [the pair of files for processing]]. Read here for full documentation.

So when we call it into the input, we assign the variable name sample_id to the wildcard value and call the files reads_file. As a side note, if you need to access the first file you can call reads_file[0] to do so and the second file is reads_file[1].

Outputs

  output:
  file "*_fastqc.{zip,html}"

Here we are telling nextflow to expect a zip file and a html file from the output of fastqc. Since we declared publishDir above, nextflow will then copy these files to the report directory.

Script

Finally, let’s tell nextflow how to run fastqc!

  script:
  """
  fastqc $reads_file -o . --threads ${task.cpus}
  """

A couple of comments. -o . tells fastqc to place the output in the current working directory - this will be located in nextflow’s work directory. --threads ${task.cpus} is how we parallelise fastqc to take advantage of the available cpus for this process.

4. Here is how the pipeline continues

The code below is hopefully pretty straightforward. The output from bowtie2 is split into two channels aligned_ch and stats_ch. stat_ch is sent to samtools to run flagstat and aligned_ch continues on for onward processing in the next steps of the pipeline.

process bowtie2 {
    container 'biocontainers/bowtie2:v2.4.1_cv1'
    cpus 16

    input:
    tuple val(sample_id), file(reads_file) from reads_for_alignment
    file db from bowtie2_db_ch

    output:
    tuple val(sample_id), file('*.sam') into aligned_ch, stats_ch

    script:
    """
    tar -xvf $db
    bowtie2 -t -p ${task.cpus} -x bowtie2/GRCh38_bowtie2 -1 ${reads_file[0]} -2 ${reads_file[1]} -S ${sample_id}.sam
    """
}

process samtools_flagstat {
    publishDir "$params.outdir/samtools_flagstat/"
    container 'biocontainers/samtools:v1.9-4-deb_cv1'
    cpus 16
    tag "$sample_id"

    input:
    tuple val(sample_id), file(sam_file) from stats_ch

    output:
    path "${sample_id}_flagstat.txt"

    script:
    """
    samtools flagstat -@ ${task.cpus} $sam_file > ${sample_id}_flagstat.txt
    """
}

Summary

The above is a simple introduction to using nextflow to pull in containers and orchestrate a pipeline. It is well worth investing some time to automate your pipeline - write your code once and use it forever!

More importantly, this is one step closer to creating your cloud genomics supercomputer. We will cover this next.

Getting Started With Docker for Bioinformatics

Thu, 05 Aug 2021 18:23:00 GMT

Next generation sequencing is becoming much more accessible to researchers in 2021. As you stare at the freshly minted .fastq files, you’re wondering - how do I go about analysing this?

After a stint on Google, you decide that you want to run bwa-mem/bowtie2 and then send the output into samtools. Next thing you know, you’re trying to install half a dozen bioinformatic programs on your new ubuntu machine. You run into dependency hell or else conda seems to be stuck solving god knows what and this quickly eats up half your day.

Stop wasting your time installing software - there is a secret to bypass all of this - Docker!

What is Docker? Here’s a quick 100 second introduction.

There are Docker containers for most bioinformatic tools. Instead of installing software, all you have to do is specify the Docker image and you’re ready to run the program!

Here’s the secret

docker run -it -v '<path to your files>:/data' <container of your choice>

This immediately spins up the shell version of the bioinformatic tool and you can immediately run the commands needed to execute the program.

Bowtie2 Example

Assuming your directory structure looks like this:

sequencing_data/
├─ reads/
│  ├─ sample_id/
│  │  ├─ read_1.fastq.gz
│  │  ├─ read_2.fastq.gz
├─ bowtie2/
│  ├─ GRCh38_bowtie2.fa

docker run -it -v '~/sequencing_data:/data' biocontainers/bowtie2:v2.4.1_cv1

bowtie2 -t -p 4 -x bowtie2/GRCh38_bowtie2 -1 reads/sample_id/read_1.fastq.gz -2 reads/sample_id/read_2.fastq.gz -S output.sam

Here’s the breakdown of the magic

-it: This opens an interactive shell into the running container

-v ‘path-you-want-to-access:/data’: This is the key command that binds your computer’s files into the container. For most biocontainers the target is /data but this may need tweaked depending on the container. This allows the program in the container to access and compute the files on your local computer. Understanding this was the key bit that unlocked the power of Docker in my mind.

How to find Docker containers

For most things, just google the tool you want and add Docker eg. ‘bwa docker’, ‘metaphlan docker’

Here are a list of public container registries with amazing software:

Give this a try - 10-15 mins of experimenting will save you days of time and you will never have to manage dependencies anymore!

What next?

Once you have got your head round using Docker for your bioinformatics, the next thing that comes to mind is to automate the containers locally on your computer so that you can easily run 100 read files at the touch of a button. I will cover this in a future article.

Architectural Approaches to Building Websites

Thu, 01 Jul 2021 18:19:49 GMT

This article is for people who are new to web development, lost in the myriad of web frameworks and are asking themselves:

What are the real differences in the frameworks?

Having spent the last year getting up to speed with the state of the web, I’m going to summarise the 4 major architectural approaches to launching a new website in 2021. Although I describe them as discrete categories, in reality they exist on a continuum and some frameworks will blur the lines between these categories (eg. NextJS). For the beginner though, it is helpful to have a rough overview of the major choices available to you.

#1 Traditional HTML/CSS/Javascript

The original way websites were build. Just pure html, css and maybe a bit of javascript. Throw in a css framework or sprinkle a bit of javascript. Ideal for one-page brochure websites which do not need constant updates of content, at a stretch maybe up to 2-3 pages. If you’re planning anything bigger, try one of the other options below.

Deployment Methods

Grab index.html and all associated files and SFTP it to apache or nginx or CDN of your choice

A better way - Setup a git repo on github and use git clone into your apache/nginx server. Makes updating website a matter of running git pull.

Advantages: Fast to develop, extremely easy to customise, easy to deploy, easy to scale.

Disadvantages: Does not suit websites which have more than one page or content that needs updated regularly.

#2 Full Stack CMS

Examples: Wordpress, Django, Ruby on Rails, Laravel

These are the traditional server side frameworks which still run most of the web today. The magic here is that the website is connected to a backend database which can hold different content and render the content into html based on templates. These frameworks also allow for user authentication and storing things like contact form data.

Deployment Methods

Usually custom to the framework. Generally requires installing a few pieces of software on your server. Think of the traditional LAMP stack (Linux Apache MySql PHP).

Use docker to automate the framework deployment steps.

Some webhost may offer one-click deployment eg of Wordpress

Platform as a service eg. Heroku

Advantages: Well used, well documented, tried and tested way of having a website hosting content which can be updated easily.

Disadvantages: Need to deploy, monitor and look after database backend.

#3 Decoupled Javascript Frontend/API Backend

Examples: React/Vue + Django Rest Framework/Laravel/Ruby on Rails, React/Vue/Mobile Frontend + AWS Amplify/Google Firebase (Backend as a service)

The trendiest way to build websites in 2021. Pick a frontend javascript framework and communicate with databases via an API gateway. Makes for very slick websites and web applications on par with desktop apps.

Deployment Methods

More involved than #2. Deploy both the frontend and the backend.

Docker is highly recommended here.

Advantages: Amazingly slick and interactive websites. Easy to split work between multiple developers.

Disadvantages: Need to deploy, monitor and look after two projects. Double the work of the traditional frameworks.

#4 Full JAMStack Static Site Generation

Examples: Hugo, Jekyll, GatsbyJS, Docusaurus, VuePress, Gridsome

Number 3 and 4 are similar in concepts at first glance. The slight difference is that these full on static site generators are more suited for webpages where the content can be preloaded (eg blogs, articles) and less suited for web applications which have users interacting with the database (such as chat apps/social media).

The key concept is that before the site is deployed, the engine pulls all the content and all the code and spits out html/javascript/css files. These files are then deployed like #1 to a server or to a CDN for global scaling.

Advantages Can create single page websites (like this blog) which do not reload the browser when navigating between pages. Makes the website feel really slick.

Disadvantages: This is for websites rather than web applications (use #3 instead). Initial learning curve on the generator of choice.

Conclusion

The right architectural choice will depend on the scope of what you’re building and the expected scale/audience. Brochure websites are different to blogs which are different to web applications.

Something important to note is that web projects are often unpredictable and tend to grow in size and complexity with time. You may reach a point where you need to consider a rework of the architecture (usually this is a good problem to have if your site traffic is growing!).

Choosing the right approach and avoiding premature optimisation will avoid wasted effort and speed your time to deployment.

The State of Web Development

Tue, 29 Jun 2021 18:01:06 GMT

I’ve had to dabble in a bit of web development over the past year as part of my research fellowship and having not done this for a while, I have accrued a couple of thoughts I want to share.

My background

First, a bit of background. Growing up, I’ve had the privilege of witnessing the birth and evolution of the modern internet. From the days of 56kbps dial-up modems which gets taken out by lightning(!) occasionally and back when Yahoo was the default home page for everybody (remember those days?). This was before MySpace and Friendster.

As I got through secondary school, that was my first real encounter with web development. We had to do group projects which involved making a webpage around a particular topic. Something like a wiki in the days when Wikipedia was barely around. I handcoded webpages with notepad. Yes, those days.

Let’s talk code editors

In 2021, I am glad to report I no longer use notepad. Now VSCode seems to be the defacto IDE and certainly is the one I would recommend. There is AI code autocompletion. What?! Just incredibly amazing. In fact, this blog is written purely in VSCode.

Nowhere has this evolution in IDEs been more impressive to me than in the world of Javascript coding. I used to waste countless hours hunting for that missing curly brace in Notepad which screwed up the entire program. And in fact, I hated JavaScript for that very reason. Now with eslint and autoformatting, JavaScript is a complete breeze and a joy to write. Let’s not go into its idiosyncracies.

How about CSS frameworks?

CSS is still probably one of the toughest things to get right. And then Bootstrap by Twitter came along and allowed everybody to stop handwriting CSS from scratch every single time. After that, the mobile revolution happened and websites had to accommodate a ridiculous number of screen sizes.

Now in 2021, it seems that all the jazz is around tailwindcss and rightly so. Initially I struggled to get my head around it. Why write things like:

<div class="bg-blue-100 mb-4 rounded shadow"></div>

How is this different from html in the pre-CSS days?

<div style="background: blue;"></div>

and that brings me onto:

Javascript frameworks

The biggest difference in 2021 vs say 2010 are javascript frameworks - ReactJS is number one today, followed by Vue, Svelte, Angular, Alpine JS. When you make the conceptual switch from building whole web pages to building components, that’s when the logic of tailwindcss really shines.

These frameworks allow you to build amazingly interactive websites that pretty much outstrip desktop applications in every way. The only desktop applications that are better installed in your machine are things like games which use tremendous amounts of horsepower.

Backend frameworks

Backend frameworks in 2021 have now transformed into APIs that serve a variety of clients - whether web or mobile app. Again another major conceptual shift. In the pre-Ruby on Rails days you had to be a php wizard or know some cgi to interact with a database. In fact, the reason why Wordpress was so popular was because it allowed you to quickly set up a website with a database on what we used to call a LAMP stack.

Deploying to the cloud

Deploying a website in 2021 has never been easier. Gone are the days of having to actually tinker with a physical machine and hook it up to your ISP’s network in order to serve your first static web page ever. In 2021, a solo developer can single handedly build a web application that can scale across the entire world as long as you have a big enough credit card limit!

The web is still the future

If there is one thing you can bet on, it’s that the web is still the future. Although some may feel that the space has matured and there are a ton of tech giants, when you go into actual industries you will see that there are still a ton of things in the real world that haven’t made the technological shift.

As someone who has been doing more real world stuff than being in the tech space, seeing the state of the web in 2021 has pretty much blown my mind. Companies that manage to get their heads round the above changes will rise while those who ignore them will have an increasingly tough time.

If you need any further convincing, I will just say one word - covid.