hack-click

Migrate Click to Mongo Library

hack-click

Click Commerce to Mongodb using Mongoose

##Roadmap

  • Phase 1: Syndication replacement & enhancements
    • Managed pub/sub messaging, elimination of repeated submission of failed requests
    • User can perform manual invocation of failed requests
    • Failed requests are never added back to the queue
  • Phase 2: Click to EPIC Integration via syndication replacement
    • Close to realtime updates
  • Phase 3: Bulk data migration & near realtime synchronization thereafter
    • Store consolidation & removal of data duplication
    • Single authorization scheme
      • Study based, removal of duplication and store dependencies
    • Backend replacement depending on use cases
      • Mongodb is preferred for web front-ends dealing with CRUD
      • Elasticsearch is preferred for search and dashboards
      • mysql is preferred for business analysts and OLAP
  • Phase 4: Corpus to Elasticsearch Migration
    • Enable fuzzy/heuristic search of binary documents (pdf, docx, xlsx, etc)
  • Phase 5: Hadoop integration with Elasticsearch & HBase
    • All Click data loaded into HBase for Map/Reduce processing
    • Binary documents for Map/Reduce will be part of pipeline
    • Hadoop workflow will be managed by Luigi

###Syndication Replacement

Status: completed

###EPIC Integration

Status: testing

  • A datasource agnostic proxy server acts as the bridge to EPIC Interconnect. Please check out this REPO for project status.

###Click to MongoDB Migration

Status: in progress

###Click to mysql Migration

Status: in progress

###Corpus to Elasticsearch Migration

Status: started

  • Initially documents will be uploaded to MongoDB GridFS. Please check out this REPO for project status.

####Installation Please refer to this page for complete instructions.

###Structure of library

  hack-click/
    lib/
      common/
      worker/
        attempt/
    schemas/

####apps Modules that use the libraries found in the apps folder hack-click-apps REPO to perform specific tasks. ####lib Shared modules that can be used by app modules to perform CRUD, filesystem, SOAP requests. ####schemas Mongoose schema used for Click objects.

####Test Results Can be found here