tag:status.chord.co,2005:/historyChord Commerce Status - Incident History2024-03-28T18:40:24ZChord Commercetag:status.chord.co,2005:Incident/187926502023-10-14T14:55:58Z2023-10-14T14:55:58ZChord API Unavailable<p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>14:55</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>14:51</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>14:51</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>14:45</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.chord.co,2005:Incident/177927812023-07-07T20:07:55Z2023-07-07T20:07:55ZGitlab outage delaying some data jobs<p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>20:07</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>20:07</var> UTC</small><br><strong>Update</strong> - Gitlab services have been restored and all data jobs will resume normal schedules.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>18:32</var> UTC</small><br><strong>Investigating</strong> - While Gitlab is down some data jobs will be delayed. Those jobs will resume as soon as Gitlab service is restored, and no data will be lost.</p>tag:status.chord.co,2005:Incident/175631262023-06-13T21:04:39Z2023-06-13T21:04:39ZChord.co marketing site outage<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>21:04</var> UTC</small><br><strong>Resolved</strong> - AWS has instituted fixes, and Netlify and other integration partners are reporting improvements/fixes on their end as well. Chord.co is up and running again, and we will continue to monitor our products.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>19:37</var> UTC</small><br><strong>Update</strong> - So far, API and store behavior appear to be unaffected. We will update if this changes.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>19:24</var> UTC</small><br><strong>Identified</strong> - The main chord.co marketing site is currently not responding. Two major upstream providers (AWS and Netlify) are reporting outages. We will keep an eye on the situation, as well as assess if this affects our main product.</p>tag:status.chord.co,2005:Incident/174232442023-05-30T17:56:46Z2023-05-30T17:56:46ZChord Hub Timeouts<p><small>May <var data-var='date'>30</var>, <var data-var='time'>17:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>17:19</var> UTC</small><br><strong>Identified</strong> - Our engineers are currently investigating root cause, and requests to a specific Redis cache are timing out, causing a cascade of timeouts downstream. We are currently working on a fix for this.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>15:30</var> UTC</small><br><strong>Investigating</strong> - Some analytics requests from the Chord Hub are currently timing out for some customers.</p>tag:status.chord.co,2005:Incident/171209682023-05-17T07:05:15Z2023-05-17T07:05:15ZChord OMS Database Upgrade<p><small>May <var data-var='date'>17</var>, <var data-var='time'>07:05</var> UTC</small><br><strong>Completed</strong> - The maintenance has concluded and all Chord OMS powered stores are working normally once again. Once again customers may notice a delay of up to 24 hours in their data refresh due to this maintenance. If you have any questions or concerns please reach out to help@chord.co and we're happy to address them.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>06:53</var> UTC</small><br><strong>Verifying</strong> - Chord OMS and storefronts are now available and full customer functionality is available. We are monitoring for unexpected behavior before declaring the all-clear.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>06:23</var> UTC</small><br><strong>Update</strong> - This maintenance is proceeding as expected. The upgraded database has passed our initial preflight checks. We're preparing to begin deeper QA of the upgrade.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>05:36</var> UTC</small><br><strong>Update</strong> - Our maintenance is proceeding as planned. All Chord OMS admin functionality and cart/checkout functionality on stores is currently non-functional. We'll continue to keep you posted.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>05:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>14:32</var> UTC</small><br><strong>Scheduled</strong> - Chord OMS will undergo a database upgrade on Wednesday, May 17. The upgrade will start at 1 AM ET and we expect to finish by 5 AM ET at latest.<br /><br />During this maintenance window, stores powered by Chord OMS will not process customer actions like adding to cart and the Chord OMS Admin panel will be unreachable. Background tasks will also be suspended during this time frame. One the database upgrade is complete, we will restore access to stores immediately.<br /><br />After the upgrade concludes, we will need to reconnect all data pipelines powering Chord’s analytics. This will result in a one-time delay to ChordOMS analytics data of up to 24 hours. Normal data refreshes will resume the following day, May 18.<br /><br />There is minimal risk of data loss or corruption during this process in either the Chord OMS itself or in the Chord Analytics Stack. This upgrade and the associated downtime is necessary to take advantage of new features, performance improvements, bug fixes, and operational cost improvements in the database that backs Chord OMS.<br /><br />We do not take downtime lightly and will endeavor to get your stores back online as quickly as possible. We will keep this status page updated throughout the event with our progress. As always, you can reach us at help@chord.co if you have any questions or concerns about this maintenance window.</p>tag:status.chord.co,2005:Incident/172289722023-05-12T16:11:26Z2023-05-12T16:11:26ZData Pipeline Delays for Some Customers<p><small>May <var data-var='date'>12</var>, <var data-var='time'>16:11</var> UTC</small><br><strong>Resolved</strong> - We're considering this incident fully resolved.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>15:04</var> UTC</small><br><strong>Monitoring</strong> - The upstream issue at the vendor appears to have been resolved. We'll leave this incident in Monitoring for the next hour or so to ensure we don't see any rebound.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>14:29</var> UTC</small><br><strong>Update</strong> - The vendor is making progress on a fix and reporting a decrease in issues they're seeing, but are not declaring an all-clear yet.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>14:05</var> UTC</small><br><strong>Identified</strong> - Due to issues with a third party vendor, some data pipelines are being delayed from their normal schedules. We're tracking the incident at the vendor and will update here when we believe we're all clear.</p>tag:status.chord.co,2005:Incident/171441932023-05-04T17:46:06Z2023-05-04T17:46:06Zdocs.chord.co is unavailable<p><small>May <var data-var='date'> 4</var>, <var data-var='time'>17:46</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>Monitoring</strong> - The outage appears to be resolved. We will continue to monitor.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>14:40</var> UTC</small><br><strong>Update</strong> - Chord's documentation, docs.chord.co, is currently unavailable due to an outage of our website provider.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>14:38</var> UTC</small><br><strong>Identified</strong> - Chord's documentation, docs.chord.co, is currently unavailable due to an outage of our website provider.</p>tag:status.chord.co,2005:Incident/171151472023-05-02T03:13:54Z2023-05-02T03:14:01ZIntermittent Issues in Chord OMS<p><small>May <var data-var='date'> 2</var>, <var data-var='time'>03:13</var> UTC</small><br><strong>Resolved</strong> - Heroku is back, and scheduled jobs are operating again.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>02:20</var> UTC</small><br><strong>Monitoring</strong> - Heroku is currently experiencing downtime to their API and dashboard. This is impacting scheduled batch jobs, so some scheduled processes will experience delays until they get back online.<br /><br />Our OMS and APIs are still online and operational. We will provide updates if anything changes.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>01:38</var> UTC</small><br><strong>Investigating</strong> - We've received alerts of a partial outage throughout Chord OMS systems. So far this has affected some background tasks and may intermittently affect the availability of stores. We're working to get confirmation of suspected issues with a vendor upstream.</p>tag:status.chord.co,2005:Incident/169331692023-04-26T15:00:16Z2023-04-26T15:00:16ZStaging: Major Database Upgrade<p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>13:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>19:51</var> UTC</small><br><strong>Scheduled</strong> - We will be performing a major database upgrade on the Chord Staging Environment. During this time, the Staging Admin, OMS, and Data Pipelines will be fully unavailable to allow us to perform the upgrade. This maintenance is expected to last up to 2 hours, but we will restore access sooner if we can.</p>tag:status.chord.co,2005:Incident/168840482023-04-17T15:49:23Z2023-04-17T16:04:32ZIntermittent connectivity issues in the Chord Staging OMS admin interface<p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>15:49</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>16</var>, <var data-var='time'>19:50</var> UTC</small><br><strong>Identified</strong> - The issue is rooted in the importer worker, so it has been quarantined. Aside from that functionality, Staging should be functioning.</p><p><small>Apr <var data-var='date'>16</var>, <var data-var='time'>19:23</var> UTC</small><br><strong>Investigating</strong> - The staging OMS administrative interface is experiencing intermittent downtime. We are currently investigating this issue.</p>tag:status.chord.co,2005:Incident/165340162023-03-17T03:00:56Z2023-03-17T03:00:56ZService Update<p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>03:00</var> UTC</small><br><strong>Resolved</strong> - All affected jobs are now recovered.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>02:46</var> UTC</small><br><strong>Update</strong> - Service is being restored and scheduled jobs have recovered. We are continuing to monitor to ensure these jobs complete successfully.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>00:27</var> UTC</small><br><strong>Monitoring</strong> - We are still monitoring the outage and awaiting updates from our service provider. Our applications remain operational, except that scheduled background jobs are currently affected and will resume when service is restored.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>21:36</var> UTC</small><br><strong>Investigating</strong> - We are currently monitoring an outage at one of our service providers, which is not affecting our applications at this time. We will continue to monitor the situation closely and will update this page if our services are impacted.</p>tag:status.chord.co,2005:Incident/162357062023-02-23T11:15:14Z2023-02-23T11:15:14ZSegment maintenance<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>11:15</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>15:09</var> UTC</small><br><strong>Scheduled</strong> - As our Segment instance undergoes scheduled maintenance, users may be unable to log into or otherwise access their Segment accounts for up to 15 minutes.</p>tag:status.chord.co,2005:Incident/160798952023-02-14T12:00:10Z2023-02-14T12:00:10ZDatabase maintenance window<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>12:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>11:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>18:48</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled, routine maintenance at this time. The Chord platform will be impacted until maintenance is completed.</p>tag:status.chord.co,2005:Incident/158453262023-01-16T16:42:32Z2023-01-16T16:42:32ZIntermittent connectivity issues in the Chord Staging OMS admin interface<p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>16:42</var> UTC</small><br><strong>Resolved</strong> - The issue has been resolved. All OMS staging services should be operating normally.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>23:49</var> UTC</small><br><strong>Update</strong> - We are still investigating the issue, but can confirm the Staging OMS API should still be operating as expected.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>23:32</var> UTC</small><br><strong>Investigating</strong> - The staging OMS administrative interface is experiencing intermittent downtime. We are currently investigating this issue.</p>tag:status.chord.co,2005:Incident/157786292023-01-05T15:14:22Z2023-01-05T15:15:29ZCircleCI Security Breach<p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>15:14</var> UTC</small><br><strong>Resolved</strong> - CircleCI disclosed their service was subject to a significant security incident. They are advising all users to rotate their secrets as soon as possible. Chord is resolving any risks to our platform and urgently advises its users to change all of their impacted keys. <br /><br />https://circleci.com/blog/january-4-2023-security-alert/</p>tag:status.chord.co,2005:Incident/138898422022-11-28T22:52:07Z2022-11-28T22:52:07ZOMS Outage<p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>22:52</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>22:46</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>22:39</var> UTC</small><br><strong>Identified</strong> - We believe we have a fix. We are seeing less errors and will continue to monitor.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>22:31</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>22:23</var> UTC</small><br><strong>Update</strong> - We have increased the database performance, but you may still see intermittent errors.</p><p><small>Nov <var data-var='date'>28</var>, <var data-var='time'>22:04</var> UTC</small><br><strong>Investigating</strong> - We are experiencing an outage in our OMS</p>tag:status.chord.co,2005:Incident/135721262022-11-21T16:00:00Z2022-11-21T17:46:06ZElevated API Errors<p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Resolved</strong> - There was an issue with elevated level of API errors at 11am ET which resulted in some customers seeing 500 errors. This issue was resolved a few minutes after the hour.</p>