modular-server

modular_srm_server

Overview

The Modular SRM Server project is a Sinatra server which serves as the backend for the modular_srm UI project. The majority of the Sinatra/Ruby code for the project is in the app/ and the lib/ folders.

The UI code for the project is primarily written in javascript using reactjs. The UI code can be compiled, minified, and packaged for the Sinatra server using the Gulp task runner and the provided gulp commands listed in this Readme

Getting Started (MAC OSX local development)

This guide assumes you are running things on your local mac and your machine has Homebrew, Node, NPM, & Ruby installed

Prerequisites

Postgres

brew install postgres

initdb /usr/local/var/postgres -E utf8

cp /usr/local/Cellar/postgresql/9.2.3/homebrew.mxcl.postgresql.plist ~/Library/LaunchAgents/

launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist

createuser -s -r postgres

If you already have Postgres but don't have 'postgres' user and command above doesn't help try this:

psql template1
CREATE ROLE postgres LOGIN
SUPERUSER INHERIT CREATEDB CREATEROLE REPLICATION;
\q

Step 1)

Clone the Repository

git clone [e-mail]:vitrue/modular_srm_server.git

cd modular_srm_server

Step 2)

bundle install to get the gem dependencies

bundle install --path=.bundle

Step 3)

install the node js, and gem dependencies with npm and bundler

npm install
bundle install

Step 4)

Preparing foreman:

1. Copy the .env.example in the base directory to .env.

cp .env.example .env

After copying, you will need to replace the values in the new .env file for all keys related to sessions, tokens, and secrets as we do not check those values into source code. Please contact the Modular team if you have problems locating valid values. If you are running locally, replace redis://redis:6379 with redis://localhost:6379

Step 5)

1. Copy over all relevant config files. You will need to copy over the database.yml.example file separately.

bundle exec rake app:configure

Step 6)

Drop the DB (Just in case you have a old copy lying around)

rake db:drop

Step 7)

Create the databases

rake db:create

Step 8)

Run the db setup script

rake db:schema:load

Step 9)

Populate the asset packs in the database

rake db:assetpacks

Step 10)

Run the server

bundle exec foreman start

Step 11)

Setting up the Testing Environment's database

bundle exec rake db:test:clone

Developing For The Project (External Module Development)

One of the main purposes of the modular_srm_server architecture was to provide a Module Ecosystem for adding features built externally by disparate teams. These external projects are called "Asset Packs".

In the most simple terms, an asset-pack is simply a collection of urls, available at a url, which is registered into the AssetPack Model's DB table in the Modular Database.

server-side inclusion of asset packs

Asset Packs are registered into the system by adding them to the project's AssetPack database table.

Currently, all asset packs entries are created via a database seed file located at db/seeds/all.rb

In the future, we will develop an admin panel UI for OPs to use, and for development purposes

Requirements of an asset pack

An npm asset pack should distribute a lib of static asset files alongside a main.js manifest file.

An example asset pack properties:

{ 
  name => "modular-engage",
  style_url => "css/bundle-cbe.css",
  script_url => "js/bundle-cbe.js",
  locale_base_url => "js/i18n/"
}

explanation of AssetPack properties

name (String - required)

the module-name. Should match the github repo name

local (Boolean)

NOTE: This is a temporary STOP GAP: it should only be entered into the modular system by a modular admin.

A boolean value which indicates whether the asset_pack asset files should be copied into the modular_srm_server dist folder and served internally by the app

While we are building out the system, we are allowing an asset-pack's static files to be served from the modular_srm_server. When local is true, the asset pack urls will be relative urls, and will be added to the Modular server at build time and served by the Modular app server

script_url (String - required)

The url to reach a single, concatenated and minified javascript file built from all of your app's js

style_url (String - required)

The url to reach a single, concatenated and minified css file built from all of your app's css

locale_urls (Array of Strings - required)

An array of all the js paths of locale culture files for translations needed by i18n to translate your UI's strings

Developing For The Project (Internal)

This guide assumes you are running things on your local mac and your machine has Homebrew, Node, NPM, & Ruby installed

We have added a few helpful features for developers working on client code in the project. The project has a strict build process which lints all javascript and scss files. In order to facilitate faster development we have included a

Git Pre-commit Hooks

We register pre-commit hooks in Git during the npm install process. Any hooks are pulled in which exist in config/git-hooks Currently, we run our lint script in a hook to lint all js and scss files before allowing a git commit.

Gulp Build process

You can build the project using the gulp task

gulp build

The process will compile react jsx files, scss stylesheets, minify images, and move tranlation files and font files into a dist/ folder ready which will be served via the sinatra server

Note that the process will fail if you have any linting errors or other compilation problems. The console output provides a lot of debugging info in these cases.

Gulp watch process

Live JSHint linting and SCSS linting with gulp watch

Running the command (from the main project directory)

gulp watch

in a terminal window will enable a watch process which will run jshint and gulp-scss-lint on your files and inform you of any liniting errors in real-time inside the console. It is recommended that you always keep a terminal window open with this task running while you are developing client code

Running Javascript Tests

We are using the React team's Jest testing framework to write client side tests

Running the command

npm test

will run your test suite in a terminal window and report any errors to your console

Alternatively, you can run a single test file or directory with the following:

$(npm bin)/jest {file or directory name}

where {file or directory name} is replaced with a valid path in your project directory

The Jest convention is to name files with the src file name, appending -spec, prior to the file extenion Test files should be stored in a directory named tests in the same parent directory as the file you are testing. ex tree:

/components
  __tests__/
    my-component-spec.js
  my-component.js

Gulp - The Nitty Gritty Gulp is a javascript task-runner/build-management tool which is powered by node-js under the hood. It has a unix-like piping syntax that makes it easy to run tasks across many files. . For notes on installing gulp see: https://github.com/gulpjs/gulp/blob/master/docs/getting-started.md#getting-started

We have set up a simple gulpfile.js in the main project directory that sets up a watch task. It will look for any javascript and scss stylesheet changes in the assets directory. When it detects a change, it will run jshint and scss-lint with the configuration options specified in .jshintrc and .scsslintrc respectively. If you see errors, fix them, and get your builds passing!

Getting Started (Ubuntu VM)

This guide assumes you are running things on your local mac and your machine has Homebrew, Node, NPM, & Ruby installed

Prerequisites

Redis

You need to have redis installed and running on the VM

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install redis-server
sudo redis-server & #start redis-server 

Postgres

The project uses postgres as its database: To intall postgres on Ubuntu:

sudo su
add-apt-repository ppa:pitti/postgresql
apt-get update
apt-get install postgresql-9.1 postgresql-contrib-9.1 libpq-dev
su postgres -c "createuser -s vitrue"

Make the following edits to the file: /etc/postgresql/9.1/main/postgresql.conf Uncomment the line

listen_addresses='localhost'

Make the following edits to the file: /etc/postgresql/9.1/main/pg_hba.conf change the line

local   all             all                                     peer

to

local   all             all                                     trust

and change the line

# IPv4 local connections: 
host    all             all             127.0.0.1/32            md5

to

# IPv4 local connections: 
host    all             all             127.0.0.1/32            trust

Restart Postgres

service postgresql restart

Note: You can access the postgres client with the following command:

sudo su - postgres -c psql postgres

Step 1)

Instal libcurl openssl

sudo apt-get install libcurl4-openssl-dev

Step 2)

Clone the Repository (we typically install at ~/projects/)

git clone [e-mail]:vitrue/modular_srm_server.git

cd modular_srm_server

Step 3)

install the bundler gem & intall bundle

gem install bundler

bundle install

Step 4)

Install nodejs & npm

sudo apt-add-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs
sudo apt-get install npm
 
sudo npm install

Step 5)

Install the gulp task manager globally from npm

sudo npm install gulp -g

Step 6)

Preparing foreman:

1. Copy the .env.example in the base directory to .env

cp .env.example .env

After copying, you will need to replace the values in the new .env file for all keys related to sessions, tokens, and secrets as we do not check those values into source code. Please contact the Modular team if you have problems locating valid values.

Step 7)

1. Copy over all relevant config files

bundle exec rake app:confgure

Step 8)

Drop the DB (Just in case you have a old copy lying around)

bundle exec rake db:drop

Step 9)

Create the databases

bundle exec rake db:create

Step 10)

Run the db setup script

bundle exec rake db:schema:load

Step 11)

seed the db

bundle exec rake db:seed

Step 12)

Setting up the Testing Environment's database

bundle exec rake db:test:clone

Step 13)

Build the javascript, css, font, and image asset directories with gulp

gulp build

or, alternatively with npm

npm run-script build

Step 14)

The server should assign a random session token for added security. This is obtained via the RACK_SESSION_SECRET environment variable. You can set this in your .env file or export it directly. For example:

export RACK_SESSION_SECRET=`openssl rand -base64 32`

Run the server

bundle exec foreman start

Getting Started (Docker)

There is a Modular_SRM docker container available which points to the standard SRMA Staging environment.

Prerequisites

You'll need a running Docker environment configured with docker-compose (ODC VM's work great for this).

Docker only runs on linux, so under OS X or Windows, you'll need Docker Machine (formally boot2docker)

brew install docker-machine docker-compose
docker-machine create dev
docker-machine start dev
eval $(docker-machine env dev)

How To (container only)

Obtain the modular-srm.tgz container tarball from someone on the dev team and run docker load -i modular-srm.tgz to install the container image. (only until we have a secure docker repository)

Create a docker-compose.yml file containing the following:

web:
    image: modular-srm
    ports:
        - 8000:8000
    links:
        - db
        - redis
db:
    image: postgres
    ports:
        - 5432
redis:
    image: redis
    ports:
        - 6379

Start the environment

1. Start the environment: docker-compose up -d

2. Create and seed the database (only necessary when you create a new environment)

docker-compose run web rake db:create docker-compose run web db:schema:load docker-compose run web db:seed

#### Connect to your running environment

Your container will be running a web server, listening on port 8000.

If you're running on OS X or Windows, you'll need to obtain the IP address of your docker VM with `docker-machine ip dev`.
Connect to either localhost:8000 or http://YOUR-VM-IP:8000


## How To (development environment)

Clone the repo, if you don't already have a local copy
`git clone https://github.com/vitrue/modular_srm_server.git`

We've created a couple of helper scripts to build and run a complete development environment.

- `script/docker-build` will build the environment
- `script/docker-start` will start it and seed the database if necessary.

Or, if you prefer...

##### Manually Build

1. Package the gems into `vender/cache` so that they can be accessed inside the container (which has no github credentials):

`bundle install && bundle  package --all`

1. Do the same for node modules:

`npm install && npm shrinkwrap —dev`

1. Finally, build the environment:

`docker-compose build`

Start the environment either with the helper script or follow the instructions under `Start the environment` in the container-only section.

### Note on Development Environment

The `docker-compose.yml` file included within the repo is slightly different from the one we created in the container-only section above. When started, it will mount the current directory into the container at `/data/modular_srm`.

Any changes within your local source directory will be reflected within the running container, so you can develop normally and see your changes reflected within the running Docker environment.

## Troubleshooting

###OSX
OSX users may run into a browserify related error on OSX due to a low operating system limit on the number of files which can be required.

The errors tend to surface in the form of
```node
Error: EMFILE, open '{path_on_your_system}'

You can work around this problem by setting the ulimit in OSX by typing the following to set the number to a higher limit (2560 is usually sufficient)

ulimit -n 2560

Please reference the issue here: OSX Error:EMFILE browserify issue #431