Novice, Paragon, Master

    marketing-data-file-uploader

    1.1.4 • Public • Published

    FB Marketing Data File Uploader

    FB Marketing Data File Uploader (referred as 'MDFU' hereinafter) is a command line tool and node.js module that helps Facebook advertisers and marketing partners upload offline transactions and customer lists to the FB marketing API without building their own application for API integration.

    Why use MDFU?

    • Building API integration will require engineering resources. Typically, an engineer without experience working with our FB marketing API will need about 3 weeks for development and testing.
    • In order to achieve the best possible match between your customers and FB users, the data needs to be normalized and hashed correctly. MDFU tool uses the libraries written by FB to ensure the best possible match rate.
    • For any issues with this tool, you will get support from Facebook.
    • This tool will be updated periodically to support more features.

    Requirements

    FB Marketing Data File Uploader requires or works with

    • Mac OS X, Linux, or Windows
    • FB App used for API calls. See this guide for instructions.
    • FB system user token that has access to the asset you are uploading to: offline event set or custom audience. See instructions here. (Regular user access token can be used, but using system user token is highly recommended)

    To upload offline conversions, you will also need

    • Offline conversions data in CSV format. Download sample file Here
    • User configured settings file. See sample file here
    • User configured column mapping file. See sample file here

    To upload customer lists to update your FB custom audience, you will instead need

    • Customer profile data in CSV format. Download sample file Here
    • User configured settings file. See example here
    • User configured column mapping file. See example here

    Step-by-Step instructions

    0. Prepare data for upload

    The input for upload is the CSV file with either offline transactions or customer list.

    • Offline conversions: Download sample file Here

    • Custom Audiences: Download sample file Here

    • If you have not tried yet, it's highly recommended to try uploading in our UI.

      • Offline Conversions: Please see this guide for upload instructions
      • Custom Audiences: Please see this guide for upload instructions.

    1. Set up upload destinations

    • Offline Conversions: Create offline event in the business manager and have the offline event set id ready. Please see this guide for instructions
    • Custom Audience:
      • Create custom audience in ads manager and have the audience id ready. Please see this guide for instructions.
      • Alternatively, if not given an audience id, we will create a new custom audience with each customer file you upload. The file name will be used as the audience name. You will need to provide your ad account id in this case.

    2. Install MDFU

    MDFU comes both as a pre-built binary and as a node.js program. If your organization does not support node.js, please see answer to question #2 in FAQ section below to learn how to obtain pre-built binary from Facebook. You can then run the pre-built binary from the command line or a script:

    $ marketing-data-file-uploader offline-conversions --accessToken YOUR_ACCESS_TOKEN... --uploadTagPrefix "Offline Sales xx/xx/xxxx"...
    
    (Standard output)
    2017-05-15T15:52:52.265Z INFO Posting events 1 - 500 to http://graph.facebook.com/v2.9/00000000000000/events
    ...
    

    For the node.js program, you have three options depending on your environment and need:

    Option 1: Run as node.js script - Install globally and use as command line tool

    $ npm install -g marketing-data-file-uploader
    $ marketing-data-file-uploader offline-conversions --accessToken YOUR_ACCESS_TOKEN... --uploadTagPrefix "Offline Sales xx/xx/xxxx"...
    
    (Standard output)
    2017-05-15T15:52:52.265Z INFO Posting events 1 - 500 to http://graph.facebook.com/v2.9/00000000000000/events
    ...
    

    Option2: Import into your own node.js app - Install locally and require the module from your app

    $ npm install marketing-data-file-uploader --save
    
    (In your app's .js file)
    const MDFU = require('marketing-data-file-uploader');
    MDFU.upload();
    
    $ node your_app.js offline-conversions --accessToken YOUR_ACCESS_TOKEN... --uploadTagPrefix "Offline Sales xx/xx/xxxx"...
    
    (Standard output)
    2017-05-15T15:52:52.265Z INFO Posting events 1 - 500 to http://graph.facebook.com/v2.9/00000000000000/events
    ...
    

    Options 3. Build binary deployable to windows/linux/mac

    $ git clone https://github.com/facebookincubator/marketing-data-file-uploader
    
    $ npm install
    $ npm run build-binary (or build-binary-exe for Windows)
    $./build/marketing-data-file-uploader offline-conversions --accessToken YOUR_ACCESS_TOKEN... --uploadTagPrefix "Offline Sales xx/xx/xxxx"...
    
    (Standard output)
    2017-05-15T15:52:52.265Z INFO Posting events 1 - 500 to http://graph.facebook.com/v2.9/00000000000000/events
    ...
    

    Options 4. Download pre-build binary deployable for windows/mac

    3. Set up Facebook app

    You need to create a FB app to make requests to FB Marketing API.

    • Create FB app. See this guide for instructions.
    • Add the app to your business manager. See this guide for instructions.

    4. Set up system user and generate system user token

    System user is a special type of user for doing API integration without the need to use someone's personal FB account to generate access token. System user token, unlike regular access token, never expires.

    • Create system user in your business manager and generate new token with the app in #3 above and give 'ads_management' scope. See instructions here. (Ask the admin of business manager to do this)
    • (Offline Conversions only) Add system user to offline event Set.
      • Go to 'offline event set' tab in business manager and select an event set
      • Click 'Add People', select system user, then click 'Save Changes'
    • (Custom Audience only) Assign ad accounts to system user
      • Go to 'System Users' tab and select the system user
      • Click 'Assign Assets', then select 'Ad Accounts'
      • Choose ad accounts that owns the audiences you will upload to, then click 'Save Changes'

    5. Set up configuration file

    Add configuration file that contains static settings.

    • Offline Conversions: Add static settings such as 'dataSetId' (ID of the offline event set) or uploadTagPrefix (prefix for the label of each upload). By default a file named oca_file_uploader.conf.yml in the same directory is used. See an example file
    • Custom Audience: Add static settings such as 'adAccountId' or 'customAudienceId' (if updating audience) By default a file named ca_file_uploader.conf.yml in the same directory is used. See an example file

    Available Configuration options

    Configuration options can either be stored in a file (.yml format) or passed in as command line arguments. Values provided on command line always overwrites settings from the config file. See following for available options:

    Option Description default
    accessToken* Access token for API call
    columnMappingFilePath File containing column mapping info. For more info see Column Mapping File section below. oca_column_mapping.json (if command is offline-conversions) or ca_column_mapping.json (if command is custom-audiences) in current directory
    configFilePath* File containing configurations oca_file_uploader.conf.yml (if command is offline-conversions) or ca_file_uploader.conf.yml (if command is custom-audiences) in current directory.
    dataSetId ID of your offline event data set if you are uploading offline conversions
    customAudienceId ID of your custom audience if you are uploading to an existing audience
    adAccountId ID of your ad account if you are creating a new audience with this upload
    inputFilePath File containing offline conversions data
    logging Control the logging level of program (available options: silly, debug, info, warn, error) info
    uploadTag Tag to identify the events uploaded. Should use unique string for each distinct file uploaded. Offline Conversions
    uploadTagPrefix Instead of providing uploadTag, you can also define prefix (ex: Offline Conversions), then the tool will append filename/timestamp and use it as the uploadTag. If uploadTag is set, uploadTagPrefix is ignored. ex) Offline Conversions (example_events_big_100k.csv@1493837377000)

    * These options MUST be passed in as command line arguments and cannot be set in configuration file.

    6. Set up column mapping

    For each file you upload you need to provide a corresponding column mapping file which defines what each column in the file is for. See an example for the column mapping for an offline conversion file, and another for the mapping for a customer PII file. The description of each key in the JSON file is below:

    Fields in column mapping file

    Field Description Required?
    header Whether the file has header row or not Yes
    delimiter The delimiter for column. comma for CSV, tab for TSV, etc... Yes
    mapping The mapping for columns in the file. key-value pair represents "column index": "column type". For more detailed description for each column type, please see Column Types section below Yes
    format Format of your dob field to help with the normalization. See foramt section below for more info No
    customTypeInfo Info for the key-value pairs of custom_data fields. See customTypeInfo section below for more info. No

    Column Types for 'mapping' field

    Column Type Required by offline-conversions? Required by custome-audiences Description
    event_time Yes No Use ISO8601 format or unixtime timestamp
    event_name Yes No See event_time row in the data parameters table
    currency Yes No Three-letter ISO currency for this conversion event. Required for Purchase events.
    value Yes No Value of conversion event. Required for Purchase event. ex) 16.00
    match_keys.xxxxx Yes Yes The identifier info used to match people. xxxxx needs to be replaced with the match key type such as email, phone, etc... For list of available match key types, please see 'Key name' column in this table
    custom_data.xxxxx No No Additional information about the conversion event. For example, send store location ID as custom_data.location_id or product category as custom_data.category

    format

    Provide more information about the formatting of certain types of columns

    For dob choose one of following format:

    • MM/DD/YYYY
    • DD/MM/YYYY
    • YYYY/MM/DD
    • MM/DD/YY
    • DD/MM/YY
    • YY/MM/DD

    Note: '/' can be skipped (MMDDYYYY) or replaced with '-' (MM-DD-YYYY)

    customTypeInfo

    Applies to offline conversion uploading only. For each custom data column, add following JSON object to customTypeInfo.

    If you added custom_data.store_num as one of the mapped columns in "mapping", you should add following:

    "customTypeInfo": {
      "store_num" : {
        "key": "store_id",    // key used in upload
        "baseType": "number"  // number or string
      }
    ...
    }
    

    7. Upload!

    Available Commands

    You need to supply one of the following commands to the tool to specify action.

    • offline-conversions: Upload offline conversion event data
    • custom-audiences: Upload customer PII data
    • version: Print the version of the tool

    For example:

    $ marketing-data-file-uploader offline-conversions [configs...]
    $ marketing-data-file-uploader custom-audiences [configs...]
    $ marketing-data-file-uploader version
    

    Command Line Examples

    This will upload offline transactions from /data/offline_transactions_20170701.csv file using oca_file_uploader.conf.yml and oca_column_mapping.json in the current directory and use 'Offline Sales for July 1st' as the upload tag.

    marketing-data-file-uploader offline-conversions --accessToken XXXXAAAA... --inputFilePath /data/offline_transactions_20170701.csv --uploadTag 'Offline Sales for July 1st'
    

    This will upload offline transactions from /data/offline_transactions_20170701.csv file using oca_file_uploader.conf.yml in the current directory and /home/custom_mapping_file.json

    marketing-data-file-uploader offline-conversions --accessToken XXXXAAAA... --inputFilePath /data/offline_transactions_20170701.csv --columnMappingFilePath /home/custom_mapping_file.json
    

    This will add customer profiles from /data/email_list_for_brand_XYZ_201707.csv file to audience 12345 using ca_file_uploader.conf.yml and ca_column_mapping.json in the current directory.

    marketing-data-file-uploader custom-audiences --accessToken XXXXAAAA... --inputFilePath /data/email_list_for_brand_XYZ_201707.csv --customAudienceId 12345
    

    This will create a new audience under the ad account 98765, and add customer profiles from /data/email_list_for_brand_XYZ_201707.csv file to the newly created audience (using ca_file_uploader.conf.yml and ca_column_mapping.json in the current directory). The file name will be used as the audience name

    marketing-data-file-uploader custom-audiences --accessToken XXXXAAAA... --inputFilePath /data/email_list_for_brand_XYZ_201707.csv --adAccountId 98765
    

    Scheduling MDFU

    The main benefit of using MDFU is the ability to automate the uploads. Although the way the upload is scheduled can vary based on the environment and different use cases, please see following for a couple of examples of using cron and windows schedule/power shell

    Using crontab: This will schedule the tool to run every day @ 2am with the filename formatted with the date string, ex) offline_conversions_YYYY_MM_DD.csv (offline_conversions_2017-04-07.csv)

    0 2 * * * cd <Path to the tool and config file> && ./marketing-data-file-uploader offline-conversions —accessToken <Your_Access_token> —inputFilePath <Path to your data file>/offline_conversions_$(date +%F).csv —uploadTag offline_conversions_$(date +%F)
    

    On Windows using Powershell and Task Scheduler: This will schedule the tool to run every day @ 2am with the filename formatted with the date string, ex) offline_conversions_YYYY_MM_DD.csv (offline_conversions_2017-04-07.csv)

    1. Create a Powershell script.
    # offline-coversions-upload.ps1
    cd c:\marketing-data-file-uploader
    marketing-data-file-uploader.exe offline-conversions --accessToken xxxxxx --inputFilePath ('c:\data\offline_conversions_' + $(get-date -f yyyy_MM_dd) + '.csv')
    
    1. Schedule execution of the script
    Schtasks /create /tn "Offline Conversions Daily Upload" /sc daily /st 02:00 /tr "PowerShell -noexit & c:\Users\Ryan\scripts\offline-coversions-upload.ps1"
    

    Code organization

    • /lib: Source code Transpiled to ES5 for npm
    • /src: Original source code written in ES6 with FB's flow type system
    • /cli.js: For when using MDFU as command line tool
    • /index.js: For when requiring MDFU as a module
    • /nexe.js: For building binary file with nexe

    FAQ

    1. My company has firewall which will block API calls to Facebook. What are my options?
    • Whitelist FB IP's: Contact your security team to whitelist IP addresses returned by this command:
    whois -h whois.radb.net -- '-i origin AS32934' | grep ^route
    

    For more information, please refer to this guide which explains whitelisting for FB crawlers, but the same set of IP's are used for API servers.

    • Request your security team to create DMZ where outbound HTTP request is allowed.
    1. My company does not support running node.js. Can we still use file uploader?
    • Pre-built binary executable files are available. Please contact FB team for access to these files. If your company has node.js install on any one of the machines, you could also try building these binary files using Option 3 above in the Installing FB Marketing Data File Uploader section.l
    1. How MDFU works

    This is a node.js application that will go through following steps to upload your offline conversions to FB's marketing API.

    1. Read command which specifies whether to upload offline events or customer lists
    2. Read configurations and column mappings
    3. Read input file in stream
    4. For each line read, columns are normalized and hashed for upload.
    5. Collect normalized and hashed data into batches (per batch size configured. Default: 500)
    6. POST each batch to the API endpoint
    7. How the binary is built: Binary file containing node runtime is built using an open source utility called nexe.

    License

    FB Marketing Data File Uploader is BSD-licensed. We also provide an additional patent grant.

    Install

    npm i marketing-data-file-uploader

    DownloadsWeekly Downloads

    1

    Version

    1.1.4

    License

    BSD-3-Clause

    Last publish

    Collaborators

    • klei87
    • ryanher