@wix/site-assets

Welcome to Site Assets Server

This is a node artifacts that generates assets (like fixed page jsons) for the viewer. The results are cached in a CDN and in Ngnix cache.

Getting Started

$ npm install

Running tests:

$ npm test

Monitoring

• Newrelic
• Grafana
• Asset Module Grafana

Related repos

• Data Fixer Runner
• Santa Module Fetcher
• Santa Data Fixer
• SSR

Slack

• #site-assets-server - site-assets Slack channel
• #site-assets-urgent - site-assets Urgent Slack channel

Managing servers in production

This artifact is using AWS Spot Fleet, which means that production machines are not managed via Fryingpan. Instead, we have our own dashboard

Overview

site-assets is a server, exposing get endpoints which are all cached (meaning that the request url is the cache key). It has no logic but running npm modules in specific versions, while providing them fixed jsons as inputs (according to the parameters, passed in the request url), and returning the results as a response. The name of the module and the version is provided as part of the request url, and should also be added to Frying pan (for security reasons). For each request, site-assets will download the requested module version from npm (unless it was already downloaded to the machine), and will run its exported function - execute, with the requests fixed pages.

At the moment, site-assets supports 2 controllers:

• Pages controller - requests handled by this controller should have the following structure: https://siteassets.parastorage.com/pages/?pageCompId&module&moduleVersion&siteRevision&<data-fixers-params....> while pageCompId is the id of the page on which you wanna run your module, module is the module name and moduleVersion is its version. the endpoint-name should be chosen by you, and it's important for monitoring reasons.

A module compatible with pages controller should follow these conventions:

1. have an index.js file, exporting the following function: module.exports = ({metricsReporter}) => ({ execute: ({pages: , ...params}) => { //your module logic... } }) metricsReporter allows you to report metrics to grafana if you need. pages map will contain 2 entries - 1 is a promise to the page specified in the request url, and the other is a promise to the masterPage. params are all query params from the request url

2. bundle the module code to a file called -server.js (you can use this example of webpack.config.js file). an example of a pages request: http://siteassets.parastorage.com/pages/pages/siteMetaData?dfCk=1&dfVersion=1.366.0&experiments=sv_fixedMobileHeader&isHttps=true&isUrlMigrated=true&metaSiteId=7e97936e-819e-4897-8112-df4b3048397e&module=santa-site-metadata&moduleVersion=1.71.0&pageCompId=c1dmp&quickActionsMenuEnabled=true&revision=11128&siteId=3ff1895d-e77e-4537-9c63-5dbf528e2685&siteRevision=31&v=3&version=1.71.0 and the module: https://github.com/wix-private/santa-core/tree/master/santa-site-metadata

• Sites controller - requests handled by this controller should have the following structure: https://siteassets.parastorage.com/sites/?module&moduleVersion&siteRevision&<data-fixers-params....> This controller is almost the same as pages, but no pageCompId should be passed in this case, and the pages map that will be passed to execute will include all site page promises.

Module version: As described above, module version should be specified so site-assets will be able to run the correct code version + for cache busting. So how do different clients know which module versions to use? For this purpose we have a static project called site-assets-versions, which exports a json that maps each module name to its production version.
In order to add/update a module version, you'll need to edit site-assets-versions input.json file, build the project (using yarn), push & GA.

Monitoring: you'll be able to get all sorts of metrics for your endpoint such as RPM, median response time, error rate etc. Just take a look here, and pick your endpoint and module name.

Architecture

You can find all the details here

API

Get a fixed page

Fetches a site page json and returns it fixed (after running santa-data-fixer.fix)

• Method:

  GET

• URL

  /fixedPage ?pageId=:pageId&v=:v&siteId=:siteId&metaSiteId=:metaSiteId&version=:version&experiments=:experiments&isHttps=:isHttps &isUrlMigrated=:isUrlMigrated&quickActionsMenuEnabled=:quickActionsMenuEnabled

• URL Params

  Required:

  pageId=[guid]

  AKA jsonFileName. A unique id representing the page in the statics servers

  v=[integer]

  a version parameter that is used to build a page json url. Should be taken from publicModel.pageList.topology

  siteId=[guid]

  Should be taken from rendererModel.siteInfo.siteId

  metaSiteId=[guid]

  Should be taken from rendererModel.metaSiteId

  version=[string]

  santa-data-fixer module version

  isHttps=[boolean]

  isUrlMigrated=[boolean]

  true iff url was migrated to the new "slash" format

  quickActionsMenuEnabled=[boolean]

  true iff rendererModel.siteMetaData.quickActions.configuration.quickActionsMenuEnabled is true

  Optional:

  experiments=[list[string]]

  a list of experiments (separated by commas), that are open and used in santa-data-fixer

  ck=[integer]

  a cache killer param, whose value comes from rendererModel.siteAssets.cacheVersion.dataFixer (this param should be added to the url only if this value exists!)

• Success Response:

  The fixed page json

  • Code: 200
    Content: { data : {}, structure: {} }

• Error Response:

  This endpoint will always return 503 upon failure, and will have the real status code in the error message.

  • Code: 503 Service Unavailable
    Content: { error : <error_message_with_status_code> }

  Common errors you might see in newrelic:

  • PageFetchError - statics returned 404 upon request to page json
  • PageFetchSystemError - when a non 404 error code was returned upon request to page json
  • ClientSpecMapFetchError - metasite-public could not build clientSpecMap for the passed metasiteId
  • ClientSpecMapFetchSystemError - an exception was thrown upon request to get clientSpecMap (in most cases the request doesn't even reach metasite-public, and the exception is thrown due to a timeout or some other system issue)
  • PagesInfoFetchError - html-info could not build pagesInfo for the passed siteId
  • PagesInfoFetchSystemError - an exception was thrown upon request to get pagesInfo (in most cases the request doesn't even reach html-info, and the exception is thrown due to a timeout or some other system issue)
  • DataFixerMissingParamError - an invalid request (missing required query params)
  • DataFixerFetchError -a request was performed with an invalid version param (santa-data-fixer version can't be fetched from npm)
  • DataFixerError - santa-data-fixer fix method throw an exception

• Usages:

  This service is currently being called from:

  1. ssr - using siteassets.*.wixprod.net/pages, on first site render flow
  2. browser - using siteassets.parastorage.com/pages (via cdn), on site navigation flow

Viewing the error message

To see the error message for your request, add the Accept: application/json header.

For example, curl -H "Accept:json" "http://siteassets.parastorage.com/pages/pages/..."

Known issues and mitigations:

Issues:

1. statics request timeouts (PageFetchSystemError) - the most common error, usually caused by statics latency or due to cross DC requests (statics run in US only).
2. RPC call timeouts (ClientSpecMapFetchSystemError / PagesInfoFetchSystemError) - happens way less than it used to, but still happens sometimes when calling meta-site-public or html-info. These requests usually don't even reach the servic, but get a timeout somewhere along the way (dispatchers/app-routers). *Note that these errors are not the same as ClientSpecMapFetchError (when clientSpecMap for the given metasiteId wasn't found) and PagesInfoFetchError (when pagesInfo for the given siteId wasn't found).
3. Page navigation success rate drop - in case you notice a decrease here, while the service is healthy (not many errors or high latency), there might be an issue with the cdn (system often change providers/configuration)
4. cdn/ngnix cache hit ratio drop - usually happens when Santa is being GA'd or when they're rollouting a version using the Gradual Rollout mechanism (by opening a FT called "OverrideReactSourceVersion").

Mitigations:

1. check if these are timeouts or 403s. wait 10 minutes to see if it's resoved by itself and if not, contact media team (SlavaG/Golan)
2. Filter the errors by host to understand if it's an issue with a specific host/DC. Wait 10 minutes and if it's not resolved by itself, contact systen (you may also try tracking a request that timed out by its requestId). *note that in order to see the full error you'll need to ssh the machine and look for it in stderr.log file
3. Try running one of the failing requests from the browser (you can use Sentry to find failed requests), and contact system (Barak Yakir/Andrey Makarov).
4. This is actually not an issue in most cases, as that's how this system behaves (santa-data-fixer version is included in the request url, so when its version changes it causes cache busting). When hit ratio drops without any correlation to santa rollout, try executing 2 identical requests and check if you're get cache miss in the second one). Try understanding if it's an issue with ngnix cache/cdn/both and contact system (Barak Yakir/Andrey Makarov).