# via Firehose - Realtime Messages ## Motivation ### Why Firehose? A common need among Sentiance customers is to react to *their* user's activities. You might want to enhance their experience in a travel app based on their mode of transport, or show them relevant offers when they plan to visit a store. Receiving updates of a user's event stream in real-time is an invaluable tool. Enter the **Firehose**: our way of sending you updates on the activity stream of your users. ## Webhooks Delivery of messages in Firehose is done via Webhooks. Webhooks are a well-known and much used integration pattern and quite popular in linking applications for [message](https://zapier.com/blog/what-are-webhooks/) [delivery](https://sendgrid.com/blog/whats-webhook/). ## Setup ### Things We Need from You In order to receive events over Firehose we need you to set up an endpoint capable of receiving **HTTP POST** requests in **JSON format**, compressed with **gzip,** secured by [BasicAuth](https://www.httpwatch.com/httpgallery/authentication/). Messages are based on events that take place on the Sentiance platform. You tell us which events you want to listen to and the details of your endpoint through the Insights [webhooks dashboard](https://controltower.sentiance.com/) and we start sending you messages whenever we have a new event for a user of your app. {% hint style="info" %} Other portals are available for specific regions. For our US customers, please use [https://controltower.p15.sentiance.com](https://controltower.p15.sentiance.com/). For our Indian customers, please use {% endhint %} We also need you to create a **HTTP GET** version of the webhook endpoint. This should be secured with the same BasicAuth credentials as the POST version and should respond with a JSON in the following format. ```javascript { "app_id": "" } ``` This ensures that the data gets sent to the correct Application ID. In case incorrect credentials are received by either POST or GET endpoint, a status **401** is expected as response. ### Message Format All messages will have a general envelope format and a `data` field with a unique format per event type. ```javascript { "data": [ { "meta": { "message_type": , "message_timestamp": }, "data": { "user_id": } } ] } ``` The **POST body** will always be a **JSON object** with a `data` field which is an array of multiple events. These events could be of [different types](#event-reference) and from different users, batched into one request. Each item will have a `meta` JSON object with fields `message_type` and `message_timestamp`. On webhooks configured to listen to only one type of event, the `message_type` will always be the same. If you wish to receive more than one type of event on the same webhook, you will need to check the `message_type` field to determine which event you are looking at. The `message_timestamp` field will tell you when the event was generated in our system. These timestamps are in ISO 8601 format. Based on the value of `message_type` you'll need to parse the `data` field. All messages are gzipped before sending over the wire. ### Batching To save on network bandwidth we batch messages sent over the webhook. Batching is done over both time and space, the defaults are **5 seconds** and **1 MB.** That is to say that once *we have 1 MB worth of data* or *5 seconds have passed*, we will create a batch of data and send a request. These values can be configured when creating or updating a webhook. The ranges are **1 - 300 seconds** and **23kb - 4MB**. **Note:** Batching by size is done before gzip compression. ### Delivery On each successful delivery we expect a **200 OK** Status Code. If we don't get one, we will keep retrying (see below). We try our best to guarantee **at least** **once** delivery. This means we might sometimes send multiples of the same message if our server fails to recognise a successful acceptance of our POST. *We don't read the body when the response is a 200 OK.* ### Security To ensure that your messages originate from Sentiance and not from a malicious third party, we will set a **BasicAuth** header on every request as per the requested configuration in the [webhooks dashboard](https://controltower.sentiance.com/). {% hint style="info" %} Other portals are available for specific regions. For our US customers, please use [https://controltower.p15.sentiance.com](https://controltower.p15.sentiance.com/). For our Indian customers, please use {% endhint %} Your connection ***must*** be secured with TLS. is a great place to inspect your endpoint and ensure it meets ongoing security standards. We ***require*** a B grade or above to ensure all data is transmitted securely. Furthermore all our calls originate from the following dedicated IPs, if you wish to protect your endpoint with IP whitelisting please add these to your whitelist. * 52.213.134.71 * 34.252.131.81 We perform a list of verifications before activating webhooks. * We check for a valid SSL certificate present on your endpoint * Valid BasicAuth credentials * Correct implementation of BasicAuth security * Intended AppID to which messages are being delivered * The ability to successfully receive messages We will be calling the **GET** endpoint to ensure the data gets sent to the correct Application ID. We will also call the endpoints with *incorrect* credentials to verify they are being rejected. We will be sending test payloads to the **POST** endpoint to ensure that the endpoint can handle our data formats. These test payloads can be identified by looking for the HTTP header `sentiance-payload-type: test` Once automated verifications have passed, your request will be forwarded to a member of the Sentiance Client team for final validation. If everything looks good your Webhook will be made active and you will be informed over email. In case there is a problem a Sentiance Delivery Team member will reach out to you. ## Errors ### Handling them like a Champ If your endpoint is unreachable, whether it be network fluctuations or a temporary server outage, we aim to still attempt delivery. Firehose expects a **200 OK** response for every message sent. If it gets back anything else, it will keep **retrying with exponential backoff starting from 100 ms up to 5 minutes**. ### Retention Every webhook has a **Message TTL**. This is the **amount of time we will keep messages** for that webhook before discarding them. This ensures that we avoid sending stale information. For example, if your **Message TTL** has been set to 30 minutes and your endpoint has been down for 40 minutes, on resuming you will only receive messages that are up to 30 minutes old. Messages from the first 10 minutes of downtime will have been dropped. You can request a specific **Message TTL** during Webhook setup. ## Testing To test out the viability of your newly created endpoint, you can try the following curl: ``` curl -X POST \ https://example.com/webhook \ -H 'Authorization: Basic c2VudGlhbmNlOnNlY3VyZXBhc3N3b3Jk' \ -H 'Content-Encoding: gzip' \ -H 'Content-Type: application/json; charset=utf-8' \ --data-binary @firehose-transport-example.gzip ``` {% file src="" %} firehose-transport-example {% endfile %} You'll have to change the url being targeted, but for the rest you can use as is. The encoded basic auth credentials are `sentiance` and `securepassword`. The provided example file is a [Transports](##transports) event as shown at the end of the page. Additionally example server implementations can be found [here](https://github.com/sentiance/firehose-receiver-example). ## FAQ #### Do you support multiple endpoints per webhook? Unfortunately we don't. We have tried to keep the design of the Firehose as simple and fast as possible which means keeping the core feature-set minimal and maintainable. #### Can I receive messages from multiple apps on the same endpoint? While this is possible, it comes with the catch that you won't know which messages are from which app. One way to remedy that is to use a dynamic route parameter in your endpoint that encodes the appId. For example: * * With `https://example.com/webhook/:appId` being the route that handles and parse `appId`. Check our [example server implementation](https://github.com/sentiance/firehose-receiver-example) for how such an endpoint might be written. ## Message Reference Below is a list of the supported Firehose messages with a brief description and an example payload ### Transport Complete This type of message is sent when a transport has been completely processed by the Sentiance platform. It includes only a `transport_id` and the `user_id` of the Sentiance user to whom the transport belongs. ```json { "meta": { "updated_attributes": [], "message_timestamp": "2022-12-07T09:10:07.000+02:00", "message_type": "transport_processing_complete", "app_id": "00000000000000000000000a" }, "data": { "user_id": "5984483fa3b15f0700000288", "transport_id": "55efa8422fdf4b4c1ac3b795c70a1e59dc68c3bc0d7492e423cab683197c4768" } } ``` ### Transport Details Similar to the Transport Complete message, this type includes extra details about the transport such as `mode`, `duration` and the start and end time of the transport. > Note: This message is sent for both the driver and the passenger. Do not assume that the user of this transport is necessarily the driver. ```json { "meta": { "updated_attributes": [], "message_timestamp": "2022-12-07T10:10:07.000+02:00", "message_type": "transport_processing_complete_with_transport_details", "app_id": "00000000000000000000000a" }, "data": { "user_id": "5984483fa3b15f0700000288", "transport_id": "55efa8422fdf4b4c1ac3b795c70a1e59dc68c3bc0d7492e423cab683197c4768", "mode": "CAR", "start_at": "2022-12-07T09:08:07.000+02:00", "end_at": "2022-12-07T10:08:07.000+02:00", "duration": "3600" } } ``` ### Transport with Scores Similar to the Transport Details message, this message also includes scores associated with the transport. ```json { "meta": { "updated_attributes": [], "message_timestamp": "2022-12-07T10:10:07.000+02:00", "message_type": "transport_processing_complete_with_transport_details_and_scores", "app_id": "00000000000000000000000a" }, "data": { "user_id": "5984483fa3b15f0700000288", "transport_id": "55efa8422fdf4b4c1ac3b795c70a1e59dc68c3bc0d7492e423cab683197c4768", "mode": "CAR", "start_at": "2022-12-07T09:08:07.000+02:00", "end_at": "2022-12-07T10:08:07.000+02:00", "duration": "3600", "overall": 0.0, "legal": 0.0, "smooth": 0.0, "attention": 0.0 } } ``` ### Off The Grid Start This type of event is sent when the SDK move into an off the grid state. It contains an identifier of the event, a reason describing why it went off the grid, the user id and a start time in iso format\\ ```json { "meta": { "updated_attributes":[], "message_timestamp":"2023-01-27T07:15:41.875+00:00", "message_type":"off_the_grid_start", "app_id":"00000000000000000000000a" }, "data": { "reason":"OFF_THE_GRID_LOCATION_MODE_DEVICE_ONLY", "start_iso":"2023-01-26T13:24:58.125+01:00", "otg_id":"4b3ea6b2-9697-4174-b54f-99cbb494f02f", "user_id":"5984483fa3b15f0700000288", "app_id":"00000000000000000000000a" } } ``` ### Off The Grid End This event is sent when the off the grid ends. It contains the identifier of the event (allowing you to couple it to the start event), the end time in iso format and the user id. ```json { "meta": { "updated_attributes":[], "message_timestamp":"2023-01-27T07:15:41.743+00:00", "message_type":"off_the_grid_end", "app_id":"00000000000000000000000a" }, "data": { "stop_iso":"2023-01-26T13:24:58.572+01:00", "otg_id":"4b3ea6b2-9697-4174-b54f-99cbb494f02f", "user_id":"5984483fa3b15f0700000288", "app_id":"00000000000000000000000a" } } ``` ### Off the Grid Reasons
ReasonDescriptionAndroidiOS
OFF_THE_GRID_LOCATION_PERMISSIONThe app has not been granted the location permission.truetrue
OFF_THE_GRID_MOTION_ACTIVITY_PERMISSIONThe app has not been granted the motion activity permission.truetrue
OFF_THE_GRID_AIRPLANE_MODEThe device is in Airplane mode.truefalse
OFF_THE_GRID_EXTERNAL_EVENTSDK detections have been disabled. i.e. the SDK has been stopped by the app.truefalse
OFF_THE_GRID_LOCATION_MODE_BATTERY_SAVINGLocation provider is set to network-based only at the device level.truefalse
OFF_THE_GRID_LOCATION_MODE_DEVICE_ONLYLocation provider is set to GPS-based only at the device level.truefalse
OFF_THE_GRID_LOCATION_ACCESS_ALWAYSLocation permission is not set to “always allow.” It is set to “while-in-use.”truetrue
OFF_THE_GRID_LOCATION_MODE_OFFLocation service has been disabled at the device level.truefalse
OFF_THE_GRID_NO_LOCATION_FIXESAccurate locations weren’t available during a trip, for a period of at least 60 minutes.truefalse
OFF_THE_GRID_DISK_QUOTA_EXCEEDEDThe SDK has exceeded its assigned disk quota.truefalse
OFF_THE_GRID_BG_EXECUTION_RESTRICTEDThe user/OS has restricted the app from running in the background.truefalse
OFF_THE_GRID_LOCATION_ACCURACY_REDUCEDThe app has not been granted the precise location permission.truetrue
### Engagement These are triggered by engagement actions, like challenges, badges, etc. Can be used to deliver real-time push notifications to users or as general updates from users. They contain the event type and a context map that's custom for each type. All context attributes are of type String. #### CHALLENGE Triggered when a user completes or fails a challenge. 
{
    "meta":{
        "app_id":"00000000000000000000000a",
        "message_timestamp":"2025-11-19T11:10:08.382+00:00",
        "message_type":"engagement",
        "updated_attributes":[]
    },
    "data":{
        "user_id":"5984483fa3b15f0700000288",
        "context":{
            "challenge_id":"challenge.driving.focused.easy-001",
            "challenge_state":"COMPLETED",
            "progress":"100.0",
            "evaluation_id":"695bb322-c95a-4a9b-963e-bb28c58a58f2",
            "notification_id":"notification-challenge-complete-00",
            "title":"Challenge completed",
            "message":"Challenge completed! Congrats.",
            "url":"",
            "challenge_description":"The next trip with your car will be without using your phone. Deal?",
            "challenge_category":"driving",
            "challenge_subcategory":"focused",
            "challenge_difficulty":"easy",
            "challenge_image_url":""
        },
        "event_type":"CHALLENGE"
    }
}
Possible values for challenge state: **COMPLETED** and **FAILED.** #### BADGE Triggered when a user achieves a badge. 
{
    "meta":{
        "app_id":"00000000000000000000000a",
        "message_timestamp":"2025-11-19T11:10:08.382+00:00",
        "message_type":"engagement",
        "updated_attributes":[]
    },
    "data":{
        "user_id":"5984483fa3b15f0700000288",
        "context":{
            "badge_id":"badge.no-speeding.level-1",
            "badge_state":"COMPLETED",
            "progress":"100.0",
            "evaluation_id":"70cd50c6-5eb2-494a-a0f6-463c1aba45f2",
            "notification_id":"notification-badge-complete-00",
            "title":"Badge earned!",
            "message":"Awesome! You earned a badge: check it out in the app.",
            "url":"",
            "badge_name":"Not making tracks",
            "badge_description":"Comply with the speed limit for 4.45 km",
            "badge_category":"No Speeding",
            "badge_reward_text":"4.45 km is exactly the length of the Phillip Island Formula 1 circuit.",
            "badge_image_url":""
        },
        "event_type":"BADGE"
    }
}
Possible values for badge state: **COMPLETED** #### TRIP Triggered when a new trip is recorded for a user. 
{
    "meta":{
        "app_id":"00000000000000000000000a",
        "message_timestamp":"2025-11-19T11:10:08.382+00:00",
        "message_type":"engagement",
        "updated_attributes":[]
    },
    "data":{
        "user_id":"5984483fa3b15f0700000288",
        "context":{
            "evaluation_id":"70cd50c6-5eb2-494a-a0f6-463c1aba45f2",
            "notification_id":"notification-trip-complete-00",
            "title":"Trip completed!",
            "message":"We recorded a new trip for you, CHeck it out in the app!",
            "url":"",
            "event_id": "",
            "total_distance_m": "10000",
            "duration_minutes": "60"
        },
        "event_type":"TRIP"
    }
}
#### STREAK Triggered when a user breaks a streak. 
{
    "meta":{
        "app_id":"00000000000000000000000a",
        "message_timestamp":"2025-11-19T11:10:08.382+00:00",
        "message_type":"engagement",
        "updated_attributes":[]
    },
    "data":{
        "user_id":"5984483fa3b15f0700000288",
        "context":{
            "evaluation_id":"70cd50c6-5eb2-494a-a0f6-463c1aba45f2",
            "notification_id":"notification-streak-broken-00",
            "title":"Sreak broken!",
            "message":"Oh no! Your last trip broke your streak of perfect driving!",
            "url":"",
            "streak_type": "STRICT",
            "score_type": "bae-mffs-score",
            "streak_state": "BREAK",
            "streak_count": "21"
        },
        "event_type":"STREAK"
    }
}
Possible values for streaks state: **BREAK.** Possible values for streak type: **STRICT** and **SELF\_COMPETING** #### REPORT Used as trigger for delivering reports to users via email or other channels. 
{
    "meta":{
        "app_id":"00000000000000000000000a",
        "message_timestamp":"2025-11-19T11:10:08.382+00:00",
        "message_type":"engagement",
        "updated_attributes":[]
    },
    "data":{
        "user_id":"5984483fa3b15f0700000288",
        "context":{
            "language":"en",
            "evaluation_id":"13674fa0-73d8-4513-877f-bc08810d9e9a",
            "notification_id":"notification-email-sign-up-00",
            "title":"",
            "message":"",
            "url":"",
        },
        "event_type":"REPORT"
    }
}
#### MESSAGE This is a generic category for events that do not fit any of the categories above. For example, an event with type MESSAGE can be triggered for reminders. 
{
    "meta":{
        "app_id":"00000000000000000000000a",
        "message_timestamp":"2025-11-19T11:10:08.382+00:00",
        "message_type":"engagement",
        "updated_attributes":[]
    },
    "data":{
        "user_id":"5984483fa3b15f0700000288",
        "context":{
            "evaluation_id":"e266a94e-9dee-4bf4-8990-ee69ec029ddb",
            "notification_id":"notification-app-reminder-02",
            "title":"We miss you",
            "message":"Please don't give up on safer driving.",
            "url":""
        },
        "event_type":"MESSAGE"
    }
}
#### SCHEDULED Triggered for messages that are scheduled to be delivered at a specific time and date. 
{
    "meta":{
        "app_id":"00000000000000000000000a",
        "message_timestamp":"2025-11-19T11:10:08.382+00:00",
        "message_type":"engagement",
        "updated_attributes":[]
    },
    "data":{
        "user_id":"5984483fa3b15f0700000288",
        "context":{
            "evaluation_id":"4bf4fefa-90dc-4aa5-ad98-1372e9b0306a",
            "notification_id":"2c260af0-22d4-4731-a9cc-4f3d02f23861",
            "title":"You are the best driver!",
            "message":"Keep it going!",
            "url":""
        },
        "event_type":"SCHEDULED"
    }
}
#### CRASH\_EVENT Triggered whenever a crash event is detected by our sdk. 
{
    "meta":{
        "app_id":"00000000000000000000000a",
        "message_timestamp":"2025-11-19T11:10:08.382+00:00",
        "message_type":"engagement",
        "updated_attributes":[]
    },
    "data":{
        "user_id":"5984483fa3b15f0700000288",
        "context":{
            "severity":"UNAVAILABLE",
            "time_iso":"2021-09-13T21:17:52.000Z",
            "detector_mode":"UNKNOWN",
            "speed_at_impact":"50",
            "confidence":"0.95",
            "location":"[37.7749,-122.4194]",
            "magnitude":"4.5",
            "time":"1763557910000",
            "preceding_locations":"[[37.775,-122.4195],[37.7752,-122.4196]]",
            "delta_v":"15"
        },
        "event_type":"CRASH_EVENT"
    }
}
Possible values for severity and detector\_mode can be found in the [sdk API docs](https://docs.sentiance.com/important-topics/sdk/api-reference/android/crashdetection/vehiclecrashevent).