secc

secc - Distributed compiler with modern web technology.

a project of 'Second Compiler'.

Key Features

• RESTful API in Daemon/Scheduler
• Preprocessed and Pump Mode
• Memory Cache(works like remote Ccache)
• Debug fission(-gsplit-dwarf option) supports.
• Monitoring by Browser(Navigate to http://SCHEDULER:PORT)
• Cross-compile(Various version/machine of compiler in network) supports
• Gcc / Clang support

Quick Start

• use DOCKER to start a scheduler and daemons

How to use

install by git clone (then npm install)

git clone https://github.com/ivere27/secc.git
cd secc
npm install

make sure you've already installed node.js, npm and (optionally) redis

• Ubuntu $ sudo apt-get install nodejs npm nodejs-legacy redis-server
• Mac $ brew install nodejs redis

Client - your PC

set PATH and NUMBER_OF_PROCESSORS(-jX)

edit "client" part in 'settings.json' file. set scheduler's address and port.

$ export PATH=/path/to/secc/bin:$PATH
$ export NUMBER_OF_PROCESSORS="8"
$ # then, 
$ clang -c test.c

or just use CC, CXX and NUMBER_OF_PROCESSORS env
```sh
$ SECC_ADDRESS=172.17.42.1 SECC_CACHE=1 SECC_CROSS=0 \
CC=/path/to/secc/bin/clang CXX=/path/to/secc/bin/clang++ ./configure
$ SECC_ADDRESS=172.17.42.1 SECC_CACHE=1 SECC_CROSS=0 \
CC=/path/to/secc/bin/clang CXX=/path/to/secc/bin/clang++ make -j8

upload your compiler archive by ./tool/secc-upload-archive.js

$ # for clang 
$ node secc-upload-archive.js --clang /path/to/clang /path/to/clang++ archivetool.js http://SCHEDULER:PORT
$ # for gcc 
$ node secc-upload-archive.js --gcc /path/to/gcc /path/to/g++ archivetool.js http://SCHEDULER:PORT

in linux case(you can use the specific compiler version),

 node secc-upload-archive.js --gcc /usr/bin/gcc-5 /usr/bin/g++-5 ./secc-create-archive-linux.js http://172.17.42.1:10509

 node secc-upload-archive.js --clang /usr/bin/clang /usr/bin/clang++ ./secc-create-archive-linux.js http://172.17.42.1:10509

Daemon - n PC

edit "daemon" part in 'settings.json' file. set scheduler's address and port.

if you want to use cache, go #Caches section.

run 'node secc-daemon.js' as root for chroot-jail.

$ sudo DEBUG=secc* node secc-daemon.js

Scheduler - 1 PC

$ DEBUG=secc* node secc-scheduler.js

Modes

MODE 1 - Preprocessed Mode(default) send one preprocessed source to daemon. (cause CPU load)

MODE 2 - Pump Mode send headers and source to daemon. (cause Memory/Bandwidth load)

MODE 3 - Git Mode(not yet supported.)

Debug

• DEBUG=* to watch every verbose logs.
• DEBUG=secc* to watch only SECC's log.
• SECC_LOG=/path/to/log.txt - redirect logs to a file

$ DEBUG=* SECC_MODE=1 SECC_CACHE=1 /path/to/secc/bin/gcc -c test.c

Caches

install REDIS in a daemon computer. then, enable "cache" in "daemon" part of 'settings.json' file.

Environment Variable

ENV	secc-client	secc-daemon	secc-scheduler
SECC_ADDRESS	scheduler address	scheduler address	-
SECC_PORT	scheduler port	scheduler port	listening port
DEBUG	log level	log level	log level
SECC_LOG	log file path	log file path	log file path
SECC_CMDLINE	log command line	-	-
SECC_MODE	processed or pump	-	-
SECC_CACHE	cache prefer	cache enable	-
SECC_CROSS	cross-compiling prefer	-	-
SECC_CC	c compiler path	-	-
SECC_CXX	c++ compiler path	-	-
REDIS_ADDRESS	-	redis address	-
REDIS_PORT	-	redis port	-
SECC_EXPOSE_ADDRESS	-	address for client	-
SECC_EXPOSE_PORT	-	port for client	-

How It Works

• Scheduler - RESTful/WebSocket/Monitoring WebServer
• Daemon - RESTful WebServer + WebSocket Client + Cache Server(redis)
• Client - http client

• (once) Client uploads a Compiler Archive to Scheduler
• Scheduler and Daemons are connected by WebSocket
• Client asks to Scheduler which daemon is available by REST API
• (Optional) Client tries to get caches if possible
• Client sends a source or/and dependencies to a Daemon by REST API
• (once) Daemon downloads the archive from Scheduler if not exists
• Daemon compiles the sources by Client's Compiler Archive
• Daemon responds to Client with a object(+@)
• (Optional) Objects are stored in Daemon's MemoryDB(redis)

License

MIT