We explain what bioinformatics is, the purpose of a bioinformatics pipeline, and how GPU acceleration and other techniques can help speed up the processing time.

What is a bioinformatics pipeline? Bioinformatics is the intersection of biology and computer science, using software programs on biological data for various applications. A bioinformatics pipeline is a series of software algorithms that process raw sequencing data and generate interpretations from this data.

What Is Bioinformatics?

Bioinformatics is an interdisciplinary field focused on developing software and hardware tools and methods to support biological data storage, organization, and analysis, particularly related to genetic sequencing.

Biological research has, in recent decades, turned to the complex task of investigating the makeup of DNA. DNA is, in essence, a building block of cellular components like proteins. Outside of the general knowledge surrounding DNA as a hereditary component of living beings, DNA is also understood as a sort of template from which simple and then more complex organisms emerge. In short, DNA is an information sequence that determines the building blocks of life.

Then it stands to reason that mapping DNA could provide significant knowledge about how life emerges from simpler chemicals and proteins. Modeling DNA, often called genomic sequencing or genomics, is also incredibly difficult and computationally intensive—so much so that effectively doing so requires extensive input from data scientists and high-end computation.

Enter bioinformatics. This combination of biochemistry, mathematics, computer science, data science, and modern data analytics leverages modern cloud computing and expertise to support the analysis of DNA. In fact, the volume of data associated with modern bioinformatics is estimated to grow anywhere from 2 to 40 exabytes per year.

What Is the Bioinformatics Pipeline and Why Is it Important?

As modern technologies around bioinformatics are changing how scientists engage the human genome, they also shape how these scientists create data—namely, they create incredible amounts of data.

Modern cloud computing has provided several key technologies to support bioinformatics:

  • Hardware-Accelerated Parallel Processing: Utilizing modern graphical processing units, high-performance computing environments provide the capacity to process high volumes of data in parallel. This speeds up processing for tasks that call for massive but identical processes. GPU acceleration has led to the advent of next generation sequencing.
  • Cloud-Based File Systems: The expansion of the cloud has supported bioinformatics. As the sheer volume of data increases, however, these environments must have a way to scale readily, which often means supporting flexible and heterogeneous systems. High-performance systems with cloud file management will often allow admins to link disparate clouds, whether public or private, to scale with demand.
  • Fast-Access Memory and Storage: Bioinformatics call for significant speed and responsiveness, which means large RAM caches and access to Non-Volatile Memory Express storage.

What’s important to note here is that the next generation sequencing has evolved from this hardware and, at the same time, added to the demand for high-end processing requirements. As bioinformatics becomes more complex, systems must add more labels, parameters, metadata, and analytics to an already-significant volume of data.

This is where a bioinformatics pipeline comes into play. A pipeline is a series of processing, storage, and compute elements connected into a series that creates a pipeline through which data travels. The output of one process provides direct input for the next, and so on, using parallel buffers to maintain performance.

Why is this important? Because, as data scientists create more complex data models from genomic information, it becomes necessary to have a way to map and control analysis in a step-by-step fashion. This helps streamline information management, minimize errors from manual processing, and so on.

What Are Some Bioinformatics Pipeline Frameworks?

Even pipelines can get complex or suffer failures. Scientists must have a way to automate pipelines so that more complex processes can be implemented and linked. To support that kind of automation, computer scientists have created what are known as bioinformatics pipeline frameworks.

Pipeline frameworks control critical pipeline management features, including all functionality for scripting, versioning, reporting, security, and so on.

There are three basic approaches to a pipeline framework:

  • Class-Based: Class-based frameworks use existing programming languages, libraries, and expansions to combine functionality into a single framework. These pipelines are primarily defined as a domain-specific application of one or more programming languages which, while sometimes unwieldy, can also provide system-agnostic frameworks through code-specific APIs.
  • Server-Based: These models often use web servers and interfaces through which users can log in, create pipelines, and feed data. The web interface will usually provide preconfigured modules that users can manipulate through the web or drag-and-drop graphical interface. These are typically much easier to use than a class-based framework but lack robust controls or configuration operations.
  • Cloud-Based: Cloud-based solutions combine the usability of a server workbench with scalable, flexible, and adjustable cloud technology. Cloud-based solutions provide more rapid pipeline creation and management and better performance for large-scale analytics and APIs that support automation without working through a web interface.

In all cases, one of the most critical issues with an informatics pipeline is iteration. The system (and the pipeline logic) should support rapid reiteration of the pipeline structure over new inputs.

Best Practices in Developing a Bioinformatics Pipeline

In some of these cases, the user doesn’t control some best practices to make pipelines faster. Smaller, more efficient modules on a cloud or server workbench are often directly under the control of a third party. However, there are several best practices you can implement regardless of the framework:

  • Reuse Computations: The strength of any parallel processing system is its ability to pump data through identical computations. The more you can use those exact computations to get the desired result, the better. As such, you are relying on the minimal number of processing approaches to make your pipelines more efficient.
  • Use Modeling Tools: Tools like modeling software with Workflow Description Language can help you and your team map out processes and dependencies. This capability can, in turn, support optimized pipelines.
  • Utilize Modern Could Technology: Most pipelines are developed for or on the cloud in modern bioinformatics. Use a cloud environment that supports high-performance workloads in genomics or life sciences, especially those that boast GPU acceleration, fast memory access, and modern data architecture.

Mobilize Bioinformatics Pipelines on WEKA

Bioinformatics pipelines are best built on the best parallel computing technology available, which means high-performance cloud computing supports genomics and life sciences workloads.

WEKA has purpose-built a cloud system that supports these workloads with the latest in hardware-accelerated computing and rapid-access storage. Features included with WEKA are as follows:

  • Streamlined and fast cloud file systems to combine multiple sources into a single high-performance computing system
  • Industry-best, GPUDirect performance (113 Gbps for a single DGX-2 and 162 Gbps for a single DGX A100)
  • In-flight and at-rest encryption for governance, risk, and compliance requirements
  • Agile access and management for edge, core, and cloud development
  • Scalability up to exabytes of storage across billions of files

Reach out to the WEKA team to learn more about how our cloud platform can power your bioinformatics pipelines.

Additional Resources