Hi all. Wanted to make an official post about what's been going on for the last week or so with the site and performance issues.
TL;DR: We've deployed a number of potential fixes. Let me know in the comments if you were experiencing problems and are now seeing performance improvements, or not.
Long version: We had a number of unrelated, but overlapping, issues last week. But one big one was a performance issue with the new Shared Dice feature causing massive browser slowdowns and sometimes crashes for a number of users.
We've been working on this feature for months, had over 70 people using it regularly, and done hundreds of hours of testing across multiple teams, devices, platforms, browsers, and even continents. While this is clearly a real issue, we've struggled to replicate it ourselves. This is the most frustrating sort of problem for everyone involved - for our players, who had their games and prep disrupted. For us, who did a huge amount of testing but something this significant somehow slipped through to production undetected.
To make matters worse, the other unrelated site issues made it difficult to understand what was causing the various problems users were reporting and led to a longer response time from our side as we were trying to make sense of what we were seeing.
(Shout-out to the folk in our discord, especially, who stayed up late helping me troubleshoot and learn more about the shape of what was going on)
As of a few minutes ago, we deployed a number of potential fixes. Since it's been difficult for us to replicate, we're asking y'all to help us get a sense of whether these performance issues have been fully resolved, are better but there are still problems, or are unchanged.
In the mean time, we're adding bullets to a list of things we can do in the future to avoid situations like this. Three examples are: - pulling old devices out of closets to further increase the diversity of hardware we're testing on - Figuring out how to get access to a wider variety of devices for performance testing - We piloted an informal internal beta program for this project. That did work very well - we identified and addressed dozens of issues that way. We recently formalized that, and intend to continue expanding it.
I know it doesn't uncancel games, or give you your prep time back. It's important to us to be transparent about what we've done, what's happening, and what we're trying to do.
@WOTC_Zac, I appreciate you and your team responding to this quickly. I hear what you're saying about it being difficult to test every possible combination of devices etc. I like your idea of pulling old devices out of the cupboard for testing.
I've just tested the website 1.70.56 on my iPad 6th Gen in Safari. I'm still getting page reloads on about every 3rd dice roll.
Another symptom. 3 separate custom rolls of D20. The first 2 didn't appear in the game log and the D20 image stayed on the screen. I've now got 2 D20 that won't disappear. 3rd roll went into the game log and then faded away as per normal. A few seconds later the page reloaded.
Was experiencing lag in browser moving to VTT tab, 3D dice not really rolling, and results not appearing in the log. Had to enable Hardware Acceleration in settings to resolve the issue. Didn't need it before, hope updates can resolve the issue without requiring it.
I am no longer experiencing immediate slowdown when opening Maps. Before it took forever to load and was constantly stuttering. Now it seems to work fine until I roll 3D dice, which causes it to slow down dramatically. Maps also becomes extremely laggy when I return to the tab after being in another tab for awhile. So still some issues but definitely an improvement for me. Big thanks to the DnD Beyond team, I know how tough it is to deal with an issue in prod like this.
Zac, it's great how transparent you are about this.
I don't know if my problem has been mentioned anywhere since the update; at least I haven't found anything about it.
But when I already have a round of Maps running (even if no dice have been rolled yet), the maps area turns completely white, and when I reload the page, it tells me to turn on hardware acceleration, even though it's already active. It then takes between 5 and 10 minutes for the page to calm down again and for me to continue playing normally.
Rollback Post to RevisionRollBack
To post a comment, please login or register a new account.
-
View User Profile
-
View Posts
-
Send Message
D&D StaffHi all. Wanted to make an official post about what's been going on for the last week or so with the site and performance issues.
TL;DR:
We've deployed a number of potential fixes. Let me know in the comments if you were experiencing problems and are now seeing performance improvements, or not.
Long version:
We had a number of unrelated, but overlapping, issues last week. But one big one was a performance issue with the new Shared Dice feature causing massive browser slowdowns and sometimes crashes for a number of users.
We've been working on this feature for months, had over 70 people using it regularly, and done hundreds of hours of testing across multiple teams, devices, platforms, browsers, and even continents. While this is clearly a real issue, we've struggled to replicate it ourselves. This is the most frustrating sort of problem for everyone involved - for our players, who had their games and prep disrupted. For us, who did a huge amount of testing but something this significant somehow slipped through to production undetected.
To make matters worse, the other unrelated site issues made it difficult to understand what was causing the various problems users were reporting and led to a longer response time from our side as we were trying to make sense of what we were seeing.
(Shout-out to the folk in our discord, especially, who stayed up late helping me troubleshoot and learn more about the shape of what was going on)
As of a few minutes ago, we deployed a number of potential fixes. Since it's been difficult for us to replicate, we're asking y'all to help us get a sense of whether these performance issues have been fully resolved, are better but there are still problems, or are unchanged.
In the mean time, we're adding bullets to a list of things we can do in the future to avoid situations like this. Three examples are:
- pulling old devices out of closets to further increase the diversity of hardware we're testing on
- Figuring out how to get access to a wider variety of devices for performance testing
- We piloted an informal internal beta program for this project. That did work very well - we identified and addressed dozens of issues that way. We recently formalized that, and intend to continue expanding it.
I know it doesn't uncancel games, or give you your prep time back. It's important to us to be transparent about what we've done, what's happening, and what we're trying to do.
More updates to come.
@WOTC_Zac, I appreciate you and your team responding to this quickly. I hear what you're saying about it being difficult to test every possible combination of devices etc. I like your idea of pulling old devices out of the cupboard for testing.
I've just tested the website 1.70.56 on my iPad 6th Gen in Safari. I'm still getting page reloads on about every 3rd dice roll.
Another symptom. 3 separate custom rolls of D20. The first 2 didn't appear in the game log and the D20 image stayed on the screen. I've now got 2 D20 that won't disappear. 3rd roll went into the game log and then faded away as per normal. A few seconds later the page reloaded.
Was experiencing lag in browser moving to VTT tab, 3D dice not really rolling, and results not appearing in the log. Had to enable Hardware Acceleration in settings to resolve the issue. Didn't need it before, hope updates can resolve the issue without requiring it.
I am no longer experiencing immediate slowdown when opening Maps. Before it took forever to load and was constantly stuttering. Now it seems to work fine until I roll 3D dice, which causes it to slow down dramatically. Maps also becomes extremely laggy when I return to the tab after being in another tab for awhile. So still some issues but definitely an improvement for me. Big thanks to the DnD Beyond team, I know how tough it is to deal with an issue in prod like this.
Problem unchanged on iPad Pro.
iPad Pro still crashes and logs me out.
Zac, it's great how transparent you are about this.
I don't know if my problem has been mentioned anywhere since the update; at least I haven't found anything about it.
But when I already have a round of Maps running (even if no dice have been rolled yet), the maps area turns completely white, and when I reload the page, it tells me to turn on hardware acceleration, even though it's already active. It then takes between 5 and 10 minutes for the page to calm down again and for me to continue playing normally.