With Connect realm migration reaching GA this quarter, Forge data residency on the roadmap, and the app data residency UI approaching general availability, you can be sure app data residency is top of mind here at Atlassian. As regional privacy regulations become more prominent, customers are often looking to keep data in a region they trust, whether they’re using an Atlassian product or a Marketplace app.
We know these customer requests don't always make it all the way to Marketplace Partners, so we do our best to share survey data with you to give you an idea of the demand. If you're a regular reader, you've heard that 80% of enterprise cloud customers surveyed say apps meeting data residency requirements is "very or extremely important." This is the second most common need expressed by the cohort of customers surveyed (after more control over app access to data).
For this reason, we've been thrilled to see more partners implementing data residency (both realm pinning and migration). This journey can look different from partner to partner and app to app depending on your setup. Fortunately, we're sharing examples of varying scenarios to help you prepare.
Learn how three Marketplace Partners implemented Data Residency as they share their individual journeys in this candid interview.
Meet the Marketplace Partners
Hannes Obweger and Daniel Franz, Co-founders at JXL
JXL for Jira is an all-in-one issue editor and organizer that combines the power of Jira with the simplicity of spreadsheets. Inline-edit issues in highly customizable tables, copy-paste fields, sum up, rank, group and structure issues in custom hierarchy, apply conditional formatting, and much more. JXL performs at any scale to save countless clicks and hours.
Jon Bevan, Engineering Team Lead at ScriptRunner, part of The Adaptavist Group
The Adaptavist Group is a global family of companies with one common goal: to make business work better. They combine the best talent, technology, and processes to make it easier for their customers to excel. ScriptRunner is a market-leader when it comes to stretching Jira, Confluence and Bitbucket beyond their limits; for teams of any size, in any industry. Their must-have suite of super-apps provide new and unlimited possibilities for automation, customisation and integration, condensed into a simple, nimble toolkit.
Markus Doll, Operations Director at codefortynine
codefortynine‘s mission is to enhance Jira and Confluence instances, remove redundant work and make day-to-day work easier. They are known for popular apps like Deep Clone for Jira, and the Atlassian Partner of the Year Award-winning app, External Data for Jira Fields. As an Atlassian Platinum Marketplace Partner, codefortynine serves 19,000+ customers, 1+ million users, in 100+ countries. codefortynine recognizes the importance of data compliance and data residency, actively incorporating these principles into its solutions.
Why did you decide to implement data residency?
Most of our interviewees were inspired to support data residency to improve customer experience, meet customer data protection needs, or keep their products in line with Atlassian's.
Hannes & Daniel from JXL:
Data protection, privacy, and security are important to our customers and therefore to us. They are built into the fabric of our app, infrastructure, processes, and services, so customers can rest assured that their data is always safe.
JXL for Jira is architected in a way that exclusively stores all data inside Jira. Your data never leaves your site. This made it pretty straightforward for us to implement data residency capabilities, as all we needed to do is provide app installation lifecycle data and gateway services, e.g. our Sheet API, in the same realm as the customers’ Jira sites. As these services can now process data in the same AWS region that stores the data, it also results in even better performance of JXL.
Jon from ScriptRunner:
We implemented data residency because it was the latest big technological initiative coming out of Atlassian to bring value and satisfaction to our customers. We were trying to keep up with Atlassian announcements and product changes.
Markus from codefortynine:
As a European-based company, we’ve felt firsthand the growing emphasis on keeping data closer to its origin. We sensed this trend, especially among larger enterprises, and decided it was crucial to adapt. That’s why we took the leap and started supporting data residency in February 2022.
Which regions do you support and why?
While some partners may have a setup that keeps data residency costs low, others may need to prioritize one region at a time to maximize return on investment. When it comes to prioritization, you may want to select regions based on where your customer base is located, where your company is located, where you’re currently storing data, or you might want to simply follow Atlassian's roadmap. Let’s see what our interviewees had to say…
Hannes & Daniel from JXL:
We support all regions Atlassian supports. The incremental cost is marginal for us due to our infrastructural setup. At this point, Atlassian supports the USA, Canada, Europe, Germany, Singapore, Australia, and so do we. With Japan, UK, Brazil, India, Switzerland, South Korea, and more coming soon.
Jon from ScriptRunner:
Before Data Residency was announced by Atlassian we were already hosting the app in the US. We added support for the new EU region first as this was the first region added by Atlassian, but we also support Germany since that's where we decided to host. Since releasing support for the EU we've not pursued supporting any other regions because it was a bigger investment than we anticipated (see our response to the question of implementation). Going forward, customers can ask us directly for region support to help us understand demand and prioritize new regions.
Markus from codefortynine:
We began by supporting the United States and Europe. Our decision for the US was influenced by the substantial portion of our customers residing there. As for Europe, it’s our home continent, and given its stringent data protection laws, it was important for us to prioritize data residency there.
How did you implement realm pinning?
Hannes & Daniel from JXL:
We have defined our stack via AWS CloudFormation and can easily spin up and operate it in different regions. The region-specific baseUrl is essentially the entrypoint to the whole stack of a particular region. As Atlassian provides the appropriate baseUrl depending on the customer's region choice, customers always end up in the correct stack.
Jon from ScriptRunner:
We host both storage and compute in each region. When we receive a Connect install lifecycle request we keep a record of the region that request is sent to in our "customers" database table.
Confusingly, we chose AWS Frankfurt as our EU region, which made supporting Germany as a dedicated region easier in some ways, because we already host there, and harder when it came to migration due to how our migration service works (I'll explain this when we get into migration).
One of the big challenges with ScriptRunner was reorganizing and rearchitecting our infrastructure as code so it knew about multi-region support and to get us into a place where we are able to define new regions in our build pipelines and deploy to more places.
For us, there is a significant extra burden of running the application in more than one place. Centralized logging, and easy visibility of which region any given customer is in, is really important so you don't need to check all regions in order to find logs or problems.
Our testing infrastructure had to scale up as well, so we can validate changes against each region before they are deployed to those regions. We currently don't deploy anything to customers unless it's working in all regions in our staging or UAT environment. Another approach would be to decouple each region but then you've got to be able to handle situations where different regions are running on different versions of your code, which makes support investigations and release planning more challenging.
Markus from codefortynine:
Following the guidelines in the Developer Guide, we paid special attention to the descriptor modifications. Within AWS, we established a mirrored environment in the eu-central-1 region, ensuring it was an exact replica of our initial us-east-1 setup. The new environment is dedicated to Europe, including Germany, while our primary environment caters to the United States and the broader Global Region. We then ensured that region-specific URLs were appropriately routed to their respective regional environments
How did you implement realm migration?
Hannes & Daniel from JXL
Since we don't store any customer data external to Jira, realm migration is almost a non-operation and completed within seconds. We're basically just signaling to Atlassian that we're okay with using a different baseUrl for services for the relevant customer now.
Jon from ScriptRunner
Before the official Connect realm migration integration was published we built a mechanism that depended on the Connect lifecycle installation requests to identify which region a customer had migrated to (by comparing where the request was sent to and where the customer used to be hosted).
Now we have integrated with the official Connect webhooks and REST APIs for realm migration, so we no longer rely on the Connect lifecycle installation requests.
Internally, realm migration involves moving some files (customer scripts and configurations) and some database entries (saved queries) from one region to another within AWS which takes a few minutes or less. After that we run a longer migration of historical script logs data which can take several hours.
Our realm-to-realm migration code assumes the source and target hosting regions for each Atlassian region are different. In the case of the EU and Germany, these regions are actually the same. As a result, we couldn’t just update our app descriptor with support for DE, we needed to modify our realm migration code so that it does nothing when migrations between DE and EU occur.
Markus from codefortynine
From our perspective and setup it was easier to support realm pinning compared to realm migration.
For realm migration, we chose to create internal APIs that can send and retrieve data between realms. Given that realm migration is a very asynchronous process with many permutations and edge cases, we placed a strong focus on unit and integration testing to tackle its complexity.
Rather than trying to test the entire migration process all at once, we broke down the integration testing. We divided it into more digestible parts, like testing the data sending or retrieval separately, which simplified the testing process significantly.
However, it’s worth noting that the development process took longer than we anticipated. We had underestimated the multitude of states this process could encompass, leading to some unexpected challenges. Still, we encountered and addressed issues in production as we are still monitoring the system closely.
After our initial tests, we collaborated with Atlassian on real test migrations. We fine-tuned the code with each iteration, making improvements until we were confident in its stability and went live.
Since then, we’ve successfully migrated several customers.
Thank you to Hannes and Daniel from JXL, Jon from ScriptRunner, and Markus from codefortynine for supporting customers with data residency needs, and for sharing your stories! With more and more partners adding data residency support to their Connect apps, it's great to learn from each others’ experiences firsthand.
We’re also working to bring you data residency for Forge hosted storage to take some of the effort of realm pinning and migration off your plate for apps that use Forge hosted storage. Stay tuned for more on this in the new year!
If you'd like to learn more about implementing data residency for your Connect apps, please see the documentation here.