epochcore

Epochtalk Core

EpochCore

####Warning this project is under active development, design is subject to change

Next generation forum backend designed with performance in mind, using leveldb for speed and reduced database size. EpochCore is a forum database backend designed to be paired with the EpochTalk Frontend. Epochcore provides a forum specific interface for the EpochTalk Frontend to manage boards, posts, threads and users.

##Database Design Overview

Using LevelUp and SubLevel, the EpochCore database is split into six subsections including one toplevel db (content) and five sublevels dbs (indexes, metadata, legacy, deleted, and messages).

  • Content - Contains the actual forum content such as boards, posts, and users.
  • Indexes - Contains all indexing of content for fast and efficient lookups.
  • Metadata - Contains all generated information in the DB such as view count and post count.
  • Legacy - Contains legacy indexes that allow older urls to point to their new pages.
  • Deleted - Contains data that has been purged from the content db. Purged data is stored in case it is needed in the future, no content data is ever completely deleted.
  • Messages - Contains message type content such as PMs.

The content database is the main database which stores boards, threads, posts, and users.

PropertyTypeDescription
idstringThe board's unique identifier
namestringThe name of the board
descriptionstringA brief description of the board
created_atdateCreated time in epoch format
updated_atdateUpdated time in epoch format
imported_at*dateImported time in epoch format
parent_idstringThe parent board's id if applicable
chilren_idsstring arrayAn array of the board's child boards if applicable
deletedbooleanTrue if board is deleted
smf.ID_BOARD*numberSMF legacy board id
smf.ID_PARENT*numberSMF legacy parent board id
PropertyTypeDescription
idstringThe thread's unique identifier
board_idstringThe id of the thread's parent board
created_atdateCreated time in epoch format
updated_atdateUpdated time in epoch format
imported_at*dateImported time in epoch format
deletedbooleanTrue if thread is deleted
smf.ID_MEMBER*numberSMF legacy user id
smf.ID_TOPIC*numberSMF legacy thread id
smf.ID_FIRST_MSG*numberSMF legacy first post id

Post Schema

PropertyTypeDescription
idstringThe post's unique identifier
titlestringThe title of the post
bodystringThe body content of the post
user_idstringThe id of the post's author
thread_idstringThe id of the post's parent thread
created_atdateCreated time in epoch format
updated_atdateUpdated time in epoch format
imported_at*dateImported time in epoch format
deletedbooleanTrue if post is deleted
versionnumberThe version of the post
smf.ID_MEMBER*numberSMF legacy user id
smf.ID_TOPIC*numberSMF legacy thread id
smf.ID_MSG*numberSMF legacy post id
PropertyTypeDescription
idstringThe user's unique identifier
usernamestringThe user's screenname
emailstringThe user's contact email
passhashstringStores hashed version of user's password
created_atdateCreated time in epoch format
updated_atdateUpdated time in epoch format
imported_at*dateImported time in epoch format
deletedbooleanTrue if user is deleted
smf.ID_MEMBER*numberSMF legacy user id

* Note: imported_at and smf object properties are used for imported legacy SMF forum data. Currently there are future plans to support importing data from other widely used forum platforms. The smf data is stored to maintain relationships between boards, threads and posts after importing.

Stores indexes for threads, posts, and users for quick data lookups. Current indexes include the following:

  • boardThread - an index which maps thread ids to their parent boards
  • threadPostOrder - an index of post ordering within a thread
  • username - an index that maps usernames to user ids
  • email - an index that maps user emails to the user ids

This database is used to store metadata which can be derived using boards, posts, threads, and users. This allows for quick lookups of useful metadata without having to bring back complete objects.

Boards Metadata

  • postCount - the number of posts excluding childboards
  • threadCount - the number of threads excluding childboards
  • totalPostCount - the total number of posts including childboards
  • totalThreadCount - the total number of threads including childboards
  • lastPostUsername - the username of the author of the last post within the board
  • lastPostCreatedAt - the lasts post's creation time within the board
  • lastThreadTitle - the last updated thread's title within the board
  • lastThreadId - the last updated thread's id within the board

Thread Metadata

  • firstPostId - the id of the threads first post
  • title - the title of the threads first post
  • username - the username of the thread's author
  • lastPostCreatedAt - the time the last post was created
  • lastPostUsername - the username of the last post's author
  • viewCount - the view count of the thread
  • postCount - the post count of the thread

Post Metadata

  • postOrder - the order number of a post within a thread

The legacy database is used to store a mapping of legacy forum ids to their new EpochTalk ids. Currently only SMF is supported.

Storing legacy id's is useful for maintaining relationships between imported data. For example: if a post's body contains a link to another thread/board/post/user using the old forums url schema. Using the mappings in the legacy db we can ensure that legacy hyperlinks will still work within EpochTalk.

Board Legacy Data

  • ID_BOARD - legacy SMF board id
  • ID_PARENT - legacy SMF parent board id

Thread Legacy Data

  • ID_MEMBER - legacy SMF user id
  • ID_TOPIC - legacy SMF thread id
  • ID_FIRST_MSG - legacy SMF first post id

Post Legacy Data

  • ID_MEMBER - legacy SMF user id
  • ID_TOPIC - legacy SMF thread id
  • ID_MSG - legacy SMF post id

User Legacy Data

  • ID_MEMBER - legacy SMF user id

The deleted database is used to store data that has been purged from the content database. The format of the data remains the same, it is simply moved to the deleted database upon purging.

Upon purging data, all data that can be derived (metadata) is removed, everything else is moved to the deleted db.

The messages database is used to store private message data.

TODO...

##Data Interfaces EpochCore provides multiple interfaces to access forum related data such as boards, threads, posts, and users.

To interface with EpochCore install it as a dependency using npm or manually by cloning this repository.

var core = require('epochcore')();
// Optionally include a path to where the database should be created. Defaults to ./epoch.db 
// var core = require('epochcore')(path.join('path', 'to', 'database')); 
 
var boards = core.boards; // Allows access to board interface methods 
var threads = core.threads; // Allows access to thread interface methods 
var posts = core.posts; // Allows access to post interface methods 
var users = core.users; // Allows access to user interface methods 

The following interfaces are currently supported:

###Boards

NameParametersReturnsDescription
boards.createBoard objectCreated board objectUsed to create a new board object
boards.findBoard IDBoard objectAllows lookup of a board by its id
boards.byOldIdSMF board IDBoard objectAllows lookup of a board by its legacy SMF ID
boards.allNoneArray of all boardsReturns an array of all parent/child boards
boards.updateModified board objectModfied board objectUsed to update a board
boards.deleteBoard IDDeleted board objectFlags a board as deleted
boards.purgeBoard IDPurged board objectMoves board to the deleted database
boards.importBoard object including SMF embedded objectImported board objectUsed to import legacy board objects

###Threads

NameParametersReturnsDescription
threads.createThread objectCreated thread objectUsed to create a new thread object
threads.findThread IDThread objectAllows lookup of a thread by its id
threads.byBoardBoard ID, OptsThreads for a boardAllows lookup of threads by board id, opts can be passed in to grab a specific page or limit
threads.threadByOldIdSMF thread IDThread objectAllows for lookup of a thread by its legacy SMF ID
threads.updateModified thread objectModified thread objectUsed to update a thread
threads.deleteThread IDDeleted thread objectFlags a thread as deleted
threads.purgeThread IDPurged thread objectMoves thread to the deleted database
threads.importThread object including SMF embedded objectThread objectUsed to import legacy thread objects

###Posts

NameParametersReturnsDescription
posts.insertPost objectCreated post objectUsed to create a new post object
posts.findPost IDPost objectAllows lookup of a post by its id
posts.byThreadThread ID, OptsPosts for a threadAllows lookup of posts by thread id, opts can be passed in to grab a specific page or limit
posts.postByOldIdSMF post IDPost objectAllows for lookup of a post by its legacy SMF ID
posts.updateModified post objectModified post objectUsed to update a post
posts.deletePost IDDeleted post objectFlags a post as deleted
posts.purgePost IDPurged post objectMoves post to the deleted database
posts.importPost object including SMF embedded objectPost objectUsed to import legacy post objects

###Users

NameParametersReturnsDescription
users.insertUser objectCreated user objectUsed to create a new user object
users.findUser IDUser objectAllows lookup of a user by their id
users.findByLegacyIdSMF user IDUser objectAllows for lookup of a user by their legacy SMF id
users.findByUsernameUsernameUser objectAllows for lookup of a user by their username
users.findByEmailEmailUser objectAllows for lookup of a user by their email
users.updateModified user objectModified user objectUsed to update a user
users.deleteUser IDDeleted user objectFlags a user as deleted
users.purgeUser IDPurged user objectMoves user to the deleted database
users.importUser object including SMF embedded objectUser objectUsed to import legacy user objects

##TODO (Planned Changes) See our github issues flagged with the TODO Label

##License The MIT License (MIT)

Copyright (c) 2014 EpochTalk

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.