Security in Your DevOps Pipeline

In software development and your DevOps processes—from design to deployment—security must be at the forefront of your mind. Make it an afterthought, and security incidents will come back to bite you.

In this post, we’re going to tackle a common but critical oversight committed by developers of GraphQL APIs—the enabling of introspection features in production environments. Introspection is an incredible feature for understanding and testing your API capabilities when you’re in the development stage. But if you don’t disable it before going live, you open up your application to significant vulnerabilities. We’ll discuss how this can escalate into a major security risk, along with preventative steps you can take to tighten up security in your DevOps pipeline.

The Scenario

Imagine a development team spearheading the implementation of their organization’s GraphQL API. GraphQL is a powerful query language designed to let clients request exactly what they need in a single query. This is unlike traditional REST APIs that might require multiple requests to piece together complex data and relationships. The structure of the data in GraphQL is defined in a schema, which outlines data types, queries, and mutations available.

The dev team has engaged in a massive effort and built something quite impressive. This API isn’t just robust; it scales, capable of handling millions of transactions per minute, all thanks to the auto-scaling capabilities that the team has designed and built. With their focus on performance and scalability, they’ve made sure that this API can handle a huge load without any hiccups.

However, in their rush to deploy and impress with this shiny new GraphQL API, they overlooked a small detail—introspection is still enabled in the production environment. Introspection is a powerful tool for developers, giving them a way to query which resources are available in the API. It’s just not something that should be opened up for access in production.

What are the consequences?

Well, as the API scales out to meet demand, each new instance multiplies the potential risk. Now, the significant security vulnerability has spread across an increasing number of nodes. With that, we’ve set the stage for potential exploitation by bad actors. They have everything they need to discover detailed information about the structure and capability of this shiny new API.

By forgetting a simple toggle, the team has turned a well-intentioned feature into a gaping security flaw.

Understanding the Vulnerability

With introspection enabled, a developer can query the GraphQL API to see the full schema—with all its data types, fields, queries, and mutations. This is like having a detailed map of all the operational endpoints and data structures for an API. That’s a great resource to have in the hands of a developer. But it’s not something you want to put in the hands of a cyberattacker.

Consider how an attacker might exploit introspection capabilities to learn about your GraphQL API. First, they can define Fragments to make their query results friendlier for reading.

fragment FullType on __Type {     kind     name     description     fields(includeDeprecated: true) {       name       description       args {         ...InputValue       }       type {         ...TypeRef       }       isDeprecated       deprecationReason     }     inputFields {       ...InputValue     }     interfaces {       ...TypeRef     }     enumValues(includeDeprecated: true) {       name       description       isDeprecated       deprecationReason     }     possibleTypes {       ...TypeRef     }   }   fragment InputValue on __InputValue {     name     description     type { ...TypeRef }     defaultValue   }   fragment TypeRef on __Type {     kind     name     ofType {       kind       name       ofType {         kind         name         ofType {           kind           name           ofType {             kind             name             ofType {               kind               name               ofType {                 kind                 name                 ofType {                   kind                   name                 }               }             }           }         }       }     }   }

Then, they can send a query to introspect the __schema field, which is available on the root type of a query. The GraphQL query might look like this:

{   __schema {     queryType {       name     }     mutationType {       name     }     subscriptionType {       name     }     types {       ...FullType     }     directives {       name       description       locations       args {         ...InputValue       }     }   } }

By sending that query, along with the Fragment definitions, to a GraphQL API with introspection enabled, an attacker will receive a response with a comprehensive JSON object with all your API details. Then, they can use a tool like GraphQL Voyager to show the GraphQL API as an interactive graph.

Here’s an example visualization of a GraphQL API based on the response to the introspection query we showed above:

As you can see, this reveals a lot of the structure and details related to sensitive payment operations. This is extremely useful information for an attacker attempting to penetrate your API. Again—great for developers, and unfortunately also great for attackers.

When you leave introspection enabled in production, here are the risks you face:

• Information exposure: Detailed schema information can arm attackers with insights into your backend systems, data models, and business logic. By handing them the blueprints to your castle, you make it easy for them to craft targeted attacks.
• Facilitation of attacks: Armed with these details about your API, attackers can create queries that exploit vulnerabilities. This can lead to data breaches or service disruptions.
• Resource drain: Attackers can also use this knowledge to craft malicious queries as denial of service (DoS) attacks. These attacks will drain resources and potentially bring your system down completely.

But don’t despair! The steps to mitigate this risk are simple and straightforward. As a first step toward protecting ourselves, let’s see what OWASP has to say.

Guidance from the OWASP GraphQL Cheat Sheet

The OWASP GraphQL Cheat Sheet is a valuable resource for securing your GraphQL APIs. It provides guidance to help you with the following areas:

• Validating inputs
• Limiting or preventing expensive queries
• Ensuring proper access control checks
• Disabling insecure configurations, such as introspection

The cheat sheet has a section devoted to Introspection and GraphiQL. GraphiQL is an in-browser IDE for exploring GraphQL. Similar to introspection capabilities, GraphiQL can expose detailed information about the inner workings and design of your API.

If you’re building a GraphQL API, we recommend familiarizing yourself with the OWASP cheat sheet.

How to Mitigate the Introspection Security Vulnerability

Taking our cues from the OWASP cheat sheet, we’ll heed its instruction to disable introspection queries in production or publicly accessible environments. This is pivotal in preventing unauthorized access to detailed schema information that could be exploited by attackers.

How do we do this?

Here is a simple code snippet in JavaScript that demonstrates how to turn off introspection in production if you use Apollo Server:

const { ApolloServer } = require('apollo-server'); const server = new ApolloServer({   typeDefs,   resolvers,  // Enable introspection only in development or test environments   introspection: process.env.NODE_ENV === 'development' ||                 process.env.NODE_ENV === 'test', }); server.listen().then(({ url }) => {   console.log(`Server ready at ${url}`); });

Yes, that’s basically all there is to it. This simple step (setting introspection to false when configuring a new instance of ApolloServer) will prevent users from accessing detailed schema information through introspection queries. This will reduce your risk of leaking sensitive information.

Making this a part of your Secure DevOps pipeline

If the step is so simple, why do developers so often forget it? Because they’re focused on other things and busy trying to meet critical deadlines. It’s no surprise that seemingly minor configuration settings get overlooked occasionally.

That’s why we have automated testing and the DevOps pipeline—to guard against human error and forgetfulness.

How might you integrate this security measure into your DevOps pipeline? Here are two suggestions:

• Write a pre-deployment test that explicitly checks your GraphQL server configuration to ensure that introspection is disabled upon production deployment.
• Write a post-deployment test that attempts an introspection query on the production environment. When your attempt is met by an error response (such as Cannot query field '__schema' on type 'Query'), then your test should pass.

Take advantage of CI/CD pipeline tools like Jenkins, GitHub Actions, or CircleCI to run scripts that will check your GraphQL API settings as part of the CI/CD process.

Conclusion

Building GraphQL APIs can be a great way for your company to innovate, but it requires you to be proactive about your security practices. One little slipup could expose your API to huge vulnerabilities. In this post, we’ve covered a prime example of this: disabling introspection for production deployments, so that attackers don’t have access to detailed information on your GraphQL schema.

For further guidance and resources, check out Linode’s comprehensive library of guides related to security, GraphQL development, and https://www.linode.com/docs/guides/platform/.