How we stopped vulnerable code from landing in production

July 23, 2019 5 min read
Hasnae Rehioui
Hasnae Rehioui
Reading Time: 5 minutes

Around a year ago, my team shipped one of the most highly voted features for Confluence Server—and no, it's not rename space key.

We were looking into Team Calendars and wanted to show our customers that we are still very much invested in the product by finally making support for CalDAV a reality. I joined the project mid-way, but little did I know that the third party library the original team used came with a serious vulnerability.

What put you on the right track?

I attended BugBash Sydney, a two-day event that brought together Atlassian engineers and security researchers to try and break into our products and uncover potential code vulnerabilities or security bugs.

In particular, I spent the first day watching some hackers study one of our open source repositories; they were identifying code patterns that display known XML vulnerabilities in our endpoints, and exploiting them by constructing malicious requests.

Security researchers and engineers look at code on a laptop

After a memorable moment of "OMG what is this sorcery!" I found myself suddenly immersed in reading up about XML attacks and joining forces with the hackers to help them craft the best exploits, to the point where some of them thought I was a security researcher myself…

"What? No, actually, I work on the product you're trying to break into."

How did the security research unfold?

We were looking at an open-source app that enriches Confluence with the Web Distributed Authoring and Versioning protocol. The plugin came with a reverse proxy class that does some XML processing:

Code showing XML processing

The pattern we were worried about was the TransformerFactory#newInstance() which makes it seem as if we could be subject to an XXE (XML external entity) attack; however, after a more thorough study of the rest of the app code, we eventually came across a security patch our developers put in place a long time ago:

Code showing security fix

So once we established that the webdav plugin is actually safe when it comes to XML processing, we started scrutinizing the same code except this time we were after symptoms of SSRF (server side request forgery).

Code showing vulnerability

Staring long enough at the code above we realized that it was indeed vulnerable to SSRF by injecting a malicious host header onto any of the requests targeting the reverse proxy.

Well, after a whole afternoon improving on my ability to think like a hacker, I spent my commute back home reminiscing about all the features I contributed to during my tenure at Atlassian, tapping myself on the shoulder "good job Viqueen, that was a secure one" …all up until CalDAV… "oh no, I think we messed up."

Needless to say, it kept me up all night.

Turns out we built our feature on top of a third party library that was carrying XML processing vulnerabilities through one of its dependencies.

Build information showing a dependency
Build information showing a dependency

After regrouping with the security researcher on the second day of BugBash, I showed them the way to the doors of Team Calendars and ask them to make their way in.

We managed to bring down a Confluence test instance by feeding it an exponential entity expansion request, commonly known as the billion laughs attack. I am quite a cheerful person, but I wasn't laughing then.

1 <?xml version="1.0"?>
2 <!DOCTYPE propfind [
3  <!ENTITY lol "lol">
4  <!ELEMENT lolz (#PCDATA)>
5  <!ENTITY lol1 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
  <!ENTITY lol2 6"&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;">
  <!ENTITY lol3 7"&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;">
 <!ENTITY lol4 8"&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;">
 <!ENTITY lol5 9"&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;">
 <!ENTITY lol6 10"&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;">
 <!ENTITY lol7 11"&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;">
 <!ENTITY lol8 12"&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;">
 <!ENTITY lol9 13"&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;">
14]>
15<d:propfind xmlns:d="DAV:" xmlns:cs="http://calendarserver.org/ns/">
16  &lol9;
17  <d:prop>
18     <d:displayname></d:displayname>
19     <cs:getctag />
20  </d:prop>
21</d:propfind>

Since entity expansion was possible, we thought to take it a step further and attempt to expose files from the host instance. It was not easy, but we managed to upload files from a Confluence host instance to an external ftp server through a rather elaborate scheme:

  1. Exploit XXE to reach out to an external server
  2. External server responds with more entities to expand
  3. One of those entities is a file system query, the other entity is an ftp upload back to the external server
Whiteboard showing how a security attack would be executed

In retrospect, I am sure there was an easier way to achieve that exploit, but to be honest I always dreamed to have a complex terminal setup like the following:

Command line interface showing a security attack

We were able to report that the CalDAV feature was vulnerable to XXE and SSRF after which the researchers finally called it a day, leaving me with the tedious task of showcasing the issue to my fellow colleagues and patching Team Calendars. I also reported the vulnerability to the authors of the third party library and contributed a fix.

Pull request showing a security fix

Awesome, so any lessons learned?

Glad you asked.

At Atlassian, we are very serious about learning from our mistakes in order to make sure they never happen again; so after resolving any incident we ran an Incident Postmortem, a session designed to determine and understand the root cause of what happened.

In the case of Confluence Server, we do have tools such as SourceClear put in place to analyze and scan our code for security issues; however, those only catch "known and already reported" vulnerabilities.

So we first proceeded by reporting and fixing the issue in order for it to appear in security scanners; we then introduced security champions across all of our sub teams to make sure we catch what our tools cannot and challenge all of our solutions from a security perspective.

Ideally you're learning your code vulnerability lessons through articles like this, rather than the hard way. If you still have further questions or feedback, feel free to reach out to us on twitter @atlassiandev or get connected to Atlassian’s developer community.

Pro tip!
If you are building an app that requires XML processing, you should use Atlassian's secure xml library. It will protect you from various XML attacks such as Billion Laughs and XML external entity expansion.

1 <dependency>
2    <groupId>com.atlassian.security</groupId>
3    <artifactId>atlassian-secure-xml</artifactId>
4    <version>3.2.11</version>
5 </dependency>
6
7 SecureXmlParserFactory.newDocumentBuilder();
8 SecureXmlParserFactory.newXmlInputFactory();
9 SecureXmlParserFactory.newXmlReader();

Pro tip!
If you’re building an app that requires a connection to an external service, you should use Atlassian's whitelist plugin:

1   <dependency>
2      <groupId>com.atlassian.plugins</groupId>
3      <artifactId>atlassian-whitelist-api-plugin</artifactId>
4      <version>3.0.2</version>
5  </dependency>
6
7  ...
8
9  @Component("communityOutboundService")
10 public class CommunityOutboundService {
11
12      private final OutboundWhitelist outboundWhitelist;
13
14      public CommunityOutboundService(
15        @ComponentImport OutboundWhitelist outboundWhitelist) {
16          this.outboundWhitelist = outboundWhitelist;
17      }
18
19      public Optional<URI> validateURI(final URI uri) {
20          return Optional.ofNullable(uri)
21                  .filter(outboundWhitelist::isAllowed);        
22      }
23  }

Resources